From roybryant at seventwentyfour.com Sun May 1 04:43:36 2005 From: roybryant at seventwentyfour.com (Roy at SEVENtwentyfour) Date: Sun, 01 May 2005 07:43:36 -0400 Subject: [openib-general] Broken link in www.openib.org Message-ID: <20050501114959.0ABC322834D@openib.ca.sandia.gov> An embedded and charset-unspecified text was scrubbed... Name: not available URL: From gdror at mellanox.co.il Sun May 1 06:11:36 2005 From: gdror at mellanox.co.il (Dror Goldenberg) Date: Sun, 1 May 2005 16:11:36 +0300 Subject: [openib-general] FMR and how they work Message-ID: <506C3D7B14CDD411A52C00025558DED6078F38D6@mtlex01.yok.mtl.com> > -----Original Message----- > From: Sean Hefty [mailto:mshefty at ichips.intel.com] > Sent: Friday, April 29, 2005 1:33 AM > Based on my understanding of Mellanox FMRs, I thought that this true. > The buffers are still available to the remote host even after > deregistration occurs. Of course my understanding could be wrong... Yes and no. You can explicitly make them unavailable to remote if you invalidate the caches. If you don't invalidate the caches, then they may be available and may not be available depending on whether the old entries are still hanging out there in the cache. > > - Sean -------------- next part -------------- An HTML attachment was scrubbed... URL: From gdror at mellanox.co.il Sun May 1 06:11:36 2005 From: gdror at mellanox.co.il (Dror Goldenberg) Date: Sun, 1 May 2005 16:11:36 +0300 Subject: [openib-general] FMR and how they work Message-ID: <506C3D7B14CDD411A52C00025558DED6078F38D5@mtlex01.yok.mtl.com> > -----Original Message----- > From: Fab Tillier [mailto:ftillier at infiniconsys.com] > Sent: Thursday, April 28, 2005 6:00 PM > > > When you deregister a FMR, what information from that FMR can > the HCA cache > - just the MTTs or also the MPTs? Both. I assume that by "deregister FMR" you refer to ib_fmr_pool_unmap(), which only takes the FMR and puts it back into a pool. If you fully destroy an FMR object then the mapping is disabled. > Can an incoming RDMA > transfer access the pages previously referenced by that FMR? > If so, for how long? When is it safe to unpin the pages? As long as the entries are cached, then the answer is yes. Obviously according to the FMR access rights and PD which are always validated. The entries get out of the cache either when cache is trashed or when you explicitly flush it. If you want explicit cache flush, then you can use SYNC_TPT command. Destruction of the object (HW2SW_MPT) also updates the caches but is an expensive operation. > > What about when you modify the FMR? How long are previous > mappings "at risk" of transfers? Same answer. > > I'm kinda confused because the Mellanox docs indicate that > the HCA can cache stuff and that the pages can't be unpinned > until the cache is flushed. If that's true, I don't see how > FMRs can be used because they could be suspect to malicious > use by a remote host. It's a tradeoff. If you want bulletproof against malicious user then you'd need to invalidate the FMR by destroying the object in each IO operation. If you want high performance, then you need to trust your peer to some extent. Looking into the future, then Extended verbs will enable a light operation to invalidate such objects, which will facilitate improvements of security. > > Help? > > - Fab > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To > unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > -------------- next part -------------- An HTML attachment was scrubbed... URL: From halr at voltaire.com Sun May 1 07:38:24 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 01 May 2005 10:38:24 -0400 Subject: [openib-general] Another RMPP related question Message-ID: <1114958304.4477.4937.camel@localhost.localdomain> Hi Sean, Is a MAD agent which registers for RMPP use restricted to using ib_create_send_mad (even when not sending a RMPP MAD) ? That may be a current restriction despite the comments for tha routing indicating "Users are not required to allocate outbound MADs using this call". If that is the case, either the comment should be changed to indicate this or the restriction removed. -- Hal From ftillier at infiniconsys.com Sun May 1 18:50:27 2005 From: ftillier at infiniconsys.com (Fab Tillier) Date: Sun, 1 May 2005 18:50:27 -0700 Subject: [openib-general] FMR and how they work In-Reply-To: <506C3D7B14CDD411A52C00025558DED6078F38D5@mtlex01.yok.mtl.com> Message-ID: <000601c54eb9$51d9ed40$fede1142@infiniconsys.com> > From: Dror Goldenberg [mailto:gdror at mellanox.co.il] > Sent: Sunday, May 01, 2005 6:12 AM > > > -----Original Message----- > > From: Fab Tillier [mailto:ftillier at infiniconsys.com] > > Sent: Thursday, April 28, 2005 6:00 PM > > > > Can an incoming RDMA > > transfer access the pages previously referenced by that FMR? > > If so, for how long? When is it safe to unpin the pages? > > As long as the entries are cached, then the answer is yes. Obviously > according to the FMR access rights and PD which are always validated. > The entries get out of the cache either when cache is trashed or when > you explicitly flush it. If you want explicit cache flush, then you can > use SYNC_TPT command. > Destruction of the object (HW2SW_MPT) also updates the caches > but is an expensive operation. > Is the HW2SW_MPT command more expensive than the SYNC_TPT command? Does it just flush the MPT out of the cache? What happens to that MPT's MTTs - do they get flushed out too (assuming the MPT still references them)? Why aren't FMR's bindable? It seems that if the HW2SW_MPT flushes the MPT out of the cache, then one could use it for normal memory registrations and avoid the WRITE_MTT command, no? That is, during registration, do posted writes for the MTTs and then a SW2HW_MPT command for an MPT entry that references those MTTs. Would this work? How much slower are memory windows compared to FMRs (assuming the underlying MR is already registered)? Thanks, - Fab From taniguchi at make-love.cx Sun May 1 21:51:23 2005 From: taniguchi at make-love.cx (taniguchi at make-love.cx) Date: Mon, 02 May 2005 13:51:23 +0900 Subject: [openib-general] =?iso-2022-jp?b?GyRCT04ySkh+Q1I7UiRHJDkbKEI=?= =?iso-2022-jp?b?GyRCISMkNEw1OjtCQSQ3JEYkKiRqJF4kOSEjGyhC?= Message-ID: <200505021351875.SM00672@192.168.1.10> $B$*5W$7$V$j$G$4$6$$$^$9!#;d$O=w at ->R2p%5!<%/%k!X(BMake-up Love$B!Y$r(B $B1?1D$7$F$*$j$^$9!"ON2JH~CR;R$H?=$7$^$9!#(B $B:#G/$b$h$j0lAX$NH/E8$HHtLv$rK>$_!"AO6H$KCQ$8$L$h$&!"(B $B$h$jNI$$=P2q$$$r5.J}MM$KDs6!$G$-$k$h$&$K@:?J$7$F;2$j$^$9!#:#8e$H$b(B $B$h$m$7$/$*4j$$CW$7$^$9!#(B $B$5$F!"$3$N$h$&$J?4Ev$?$j$O$4$6$$$^$;$s$+!)(B $B!Z;E;v>l$G=P2q$$$,$J$/!"5$$,IU$/$H0l?M$\$C$A!D![(B $B!Z7k:'@83h$bD9$/!"%^%s%M%j$,?J$s$G$$$k!D![(B $B!ZNx$r$7$?$$$,!"$J$+$J$+Nx0&$r$9$k5!2q$,$J$$!D![(B $B!Zp$b$4$6$$$^$9!#$=$N$h$&$J%K!<%:$K$*Ez$($9$k$N$,;dC#(B $B!X(BMake-up Love$B!Y$G$4$6$$$^$9!#(B $B0J2<$N#2$D$N$4R2pNA6&$KL5NA$G$4>R2p$5$;$FD:$-$^$9!#(B ------------------------------------------------------------------ $B"'$*=;$^$$$NCO0h(B($BETF;I\8)$N$_$G7k9=$G$9(B) $B!!!!!!!V!!!!!!!W(B $B"'$44uK>$N%8%c%s%k$rHV9f$G$*Ez$($/$@$5$$!#(B $B!!-!Nx?MJg=8!J??LLL\$J8r:]!&7k:'A0Ds!K(B $B!!-"%5%]!<%H4uK>!J;v6H;q6b$J$I%5%]!<%H$7$FM_$7$$!K(B $B!!-#%;%C%/%9%U%l%s%I!J3d$j at Z$C$?Bg?M$N$*IU$-9g$$!&0&?M!&?M:J!K(B $B!!-$B?e$N$*IU$-9g$$(B $B!!!!!!!V!!!!!!!W(B ----------------------------------------------------------------- $B@^$jJV$7$4JV?.2<$5$$$^$;!#!X(BMake-up Love$B!Y%+%9%?%^!<%5%]!<%H%;%s%?!<(B $B$h$j!"%3%s%?%/%HJ}K!!"F1$8CO0h$K$*$$$F$N$4>R2p$,2DG=$J=w at -2q0wMM$r(B $B$A$K$4O"Mm:9$7>e$2$^$9!#(B $B!THw9M!U(B $B!X(BMake-up Love$B!Y$O>R2p$rL\E*$H$7$?=w at -$@$1$N%0%k!<%W$G$9!#(B $B!X(BMake-up Love$B!Y$O=w at -2q0w$N2q0wHq$H9-9p<}F~$G1?1D$5$l$F$$$^$9!#(B $BCK at -$O>R2pNA!&EPO?NA6&$KD:$$$F$*$j$^$;$s!#(B $B$3$N%a!<%k$O2a5n$KEPO?D:$$$?$*5RMM$+!"%0%k!<%W%5%$%H$N#M#L$KEPO?(B $B$7$FD:$$$?J}$K8~$1$FAw?.$7$F$*$j$^$9!#(B $BEPO?$K$D$$$F0l at Z3P$($NL5$$J}!"Kt$OBh;0l9g$O$4LLE]$G$9$,!"$3$N%a!<%k$K!VG[?.ITMW!W$HE:$($F$4JV?.$/$@$5$$!#(B $B $B!!!!!!!!!!!!"v!&!yL5NAEPO?%-%c%s%Z!<%sCf!y!&"v(B $B!!!!!!$d$C$Q$j=P0)$&$J$i$46a=j$G2q$($kAjR2pCW$7$^$9!#(B $B!!!!!!!!!!!!!!!!!!!!$^$:$O$*;n$7L5NAEPO?$+$i(B $B!!!!!!!!!!!!!!!!!!!!!!!!!!"-"-"-"-"-"-(B $B!!!!!!!!!!!!!!!!(Bhttp://www.jumpb2.net/?imasugu $B!!!!!!!!!!!!!!!!(B $B!!!!!!!!!!!!!!!!!!(B $B%U%j!<%a!<%kBP1~$G$9!*(B From mst at mellanox.co.il Mon May 2 01:40:13 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Mon, 2 May 2005 11:40:13 +0300 Subject: [openib-general] slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects Message-ID: <20050502084013.GR16918@mellanox.co.il> Hi! I have this script to unload all modules: killall opensm sleep 3 killall -9 opensm modprobe -r ib_ipoib modprobe -r ib_umad modprobe -r ib_mthca So, I try to unload the modules while opensm may still be dying. Every now and then I see this crash (below), which seems to indicate a race condition or leak somewhere around ib_mad or ib_umad. My guess is a mad may still be outstanding. Ideas, anyone? Further, I'm looking at the mad agent logic and it seems a bit weird that ./core/agent.c does kmem_cache_free in agent_send_handler, while all allocs are in mad.c. What prevents an agent from deregistering while a send is outstanding? What'll free the mad_priv then? log dump below. Thanks, MST This is with 2.6.11 + rev 2235 (latest bits as of now), x86_64 (Intel Nocona). May 2 10:36:12 swlab156 kernel: slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects May 2 10:36:12 swlab156 kernel: May 2 10:36:12 swlab156 kernel: Call Trace:{kmem_cache_destroy+184} {:ib_mad:ib_mad_cleanup_module+28} May 2 10:36:12 swlab156 kernel: {sys_delete_module+487} {__up_write+28} May 2 10:36:12 swlab156 kernel: {sys_munmap+74} {system_call+126} May 2 10:36:12 swlab156 kernel: May 2 10:36:12 swlab156 kernel: ib_mad: Failed to destroy ib_mad cache Any attempt to load ib_mad after that fails: May 2 10:36:25 swlab156 kernel: kmem_cache_create: duplicate cache ib_mad May 2 10:36:25 swlab156 kernel: ----------- [cut here ] --------- [please bite here ] --------- May 2 10:36:25 swlab156 kernel: Kernel BUG at slab:1472 May 2 10:36:25 swlab156 kernel: invalid operand: 0000 [1] SMP May 2 10:36:25 swlab156 kernel: CPU 1 May 2 10:36:25 swlab156 kernel: Modules linked in: ib_mad ib_core May 2 10:36:25 swlab156 kernel: Pid: 14102, comm: modprobe Not tainted 2.6.11-openib May 2 10:36:25 swlab156 kernel: RIP: 0010:[kmem_cache_create+1384/1539] {kmem_cache_create+1384} May 2 10:36:25 swlab156 kernel: RIP: 0010:[] {kmem_cache_create+1384} May 2 10:36:25 swlab156 kernel: RSP: 0018:ffff81015d8c7ee8 EFLAGS: 00010202 May 2 10:36:25 swlab156 kernel: RAX: 000000000000002a RBX: ffff81015fd69670 RCX: ffffffff804572a8 May 2 10:36:25 swlab156 kernel: RDX: ffffffff804572a8 RSI: 0000000000000296 RDI: ffffffff8055f0c0 May 2 10:36:25 swlab156 kernel: RBP: ffff81015fd69480 R08: ffff81015e0976c0 R09: 0000000000000000 May 2 10:36:25 swlab156 kernel: R10: 0000000000000000 R11: 0000000000000080 R12: ffffffff8055f0c0 May 2 10:36:25 swlab156 kernel: R13: 0000000000002000 R14: ffff810000000000 R15: 0000000000000080 May 2 10:36:25 swlab156 kernel: FS: 00002aaaaade26e0(0000) GS:ffffffff80583180(0000) knlGS:0000000000000000 May 2 10:36:25 swlab156 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b May 2 10:36:25 swlab156 kernel: CR2: 00002aaaaaacc000 CR3: 000000013f67a000 CR4: 00000000000006e0 May 2 10:36:25 swlab156 kernel: Process modprobe (pid: 14102, threadinfo ffff81015d8c6000, task ffff81015dcc57f0) May 2 10:36:25 swlab156 kernel: Stack: ffffffffffffff80 0000000000000000 0000000000000000 ffffffff88010951 May 2 10:36:25 swlab156 kernel: 0000000000000180 ffffffff8045a000 ffffffff88013000 ffffffff80459fc0 May 2 10:36:25 swlab156 kernel: ffffffff80459fc0 00007ffffffff408 May 2 10:36:25 swlab156 kernel: Call Trace:{:ib_mad:ib_mad_init_module+51} {sys_init_module+298} May 2 10:36:25 swlab156 kernel: {system_call+126} May 2 10:36:25 swlab156 kernel: May 2 10:36:25 swlab156 kernel: Code: 0f 0b e5 be 3e 80 ff ff ff ff c0 05 48 8b 1b 48 8b 03 0f 18 May 2 10:36:25 swlab156 kernel: RIP {kmem_cache_create+1384} RSP -- MST - Michael S. Tsirkin From halr at voltaire.com Mon May 2 05:14:19 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 02 May 2005 08:14:19 -0400 Subject: [openib-general] slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects In-Reply-To: <20050502084013.GR16918@mellanox.co.il> References: <20050502084013.GR16918@mellanox.co.il> Message-ID: <1115036059.4477.5703.camel@localhost.localdomain> On Mon, 2005-05-02 at 04:40, Michael S. Tsirkin wrote: > Hi! > I have this script to unload all modules: > > killall opensm > sleep 3 > killall -9 opensm > modprobe -r ib_ipoib > modprobe -r ib_umad > modprobe -r ib_mthca > > So, I try to unload the modules while opensm may still be dying. > Every now and then I see this crash (below), which seems to indicate > a race condition or leak somewhere around ib_mad or ib_umad. > My guess is a mad may still be outstanding. > > Ideas, anyone? > > Further, I'm looking at the mad agent logic and it seems a bit weird that > ./core/agent.c does kmem_cache_free in agent_send_handler, > while all allocs are in mad.c. This has been brought up before on the list. I think much of this could (and should) be rewritten now using Sean's helper functions. > What prevents an agent from deregistering while a send is outstanding? > What'll free the mad_priv then? Nothing. I think there is some missing code in ib_agent_port_close to handle this scenario. However, unless that MAD from the SM were directed locally (and was pending), that would not cause the problem where the ib_mad cache could not be destroyed. I will see if I can recreate this and work up a patch for this. > log dump below. > > Thanks, > > MST > > This is with 2.6.11 + rev 2235 (latest bits as of now), x86_64 (Intel Nocona). > > > May 2 10:36:12 swlab156 kernel: slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects > May 2 10:36:12 swlab156 kernel: > May 2 10:36:12 swlab156 kernel: Call Trace:{kmem_cache_destroy+184} {:ib_mad:ib_mad_cleanup_module+28} > May 2 10:36:12 swlab156 kernel: {sys_delete_module+487} {__up_write+28} > May 2 10:36:12 swlab156 kernel: {sys_munmap+74} {system_call+126} > May 2 10:36:12 swlab156 kernel: > May 2 10:36:12 swlab156 kernel: ib_mad: Failed to destroy ib_mad cache Has this been occuring for a while or is this new (with the recent changes to mad handling) ? > Any attempt to load ib_mad after that fails: > > > May 2 10:36:25 swlab156 kernel: kmem_cache_create: duplicate cache ib_mad > May 2 10:36:25 swlab156 kernel: ----------- [cut here ] --------- [please bite here ] --------- > May 2 10:36:25 swlab156 kernel: Kernel BUG at slab:1472 > May 2 10:36:25 swlab156 kernel: invalid operand: 0000 [1] SMP > May 2 10:36:25 swlab156 kernel: CPU 1 > May 2 10:36:25 swlab156 kernel: Modules linked in: ib_mad ib_core > May 2 10:36:25 swlab156 kernel: Pid: 14102, comm: modprobe Not tainted 2.6.11-openib > May 2 10:36:25 swlab156 kernel: RIP: 0010:[kmem_cache_create+1384/1539] {kmem_cache_create+1384} > May 2 10:36:25 swlab156 kernel: RIP: 0010:[] {kmem_cache_create+1384} > May 2 10:36:25 swlab156 kernel: RSP: 0018:ffff81015d8c7ee8 EFLAGS: 00010202 > May 2 10:36:25 swlab156 kernel: RAX: 000000000000002a RBX: ffff81015fd69670 RCX: ffffffff804572a8 > May 2 10:36:25 swlab156 kernel: RDX: ffffffff804572a8 RSI: 0000000000000296 RDI: ffffffff8055f0c0 > May 2 10:36:25 swlab156 kernel: RBP: ffff81015fd69480 R08: ffff81015e0976c0 R09: 0000000000000000 > May 2 10:36:25 swlab156 kernel: R10: 0000000000000000 R11: 0000000000000080 R12: ffffffff8055f0c0 > May 2 10:36:25 swlab156 kernel: R13: 0000000000002000 R14: ffff810000000000 R15: 0000000000000080 > May 2 10:36:25 swlab156 kernel: FS: 00002aaaaade26e0(0000) GS:ffffffff80583180(0000) knlGS:0000000000000000 > May 2 10:36:25 swlab156 kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b > May 2 10:36:25 swlab156 kernel: CR2: 00002aaaaaacc000 CR3: 000000013f67a000 CR4: 00000000000006e0 > May 2 10:36:25 swlab156 kernel: Process modprobe (pid: 14102, threadinfo ffff81015d8c6000, task ffff81015dcc57f0) > May 2 10:36:25 swlab156 kernel: Stack: ffffffffffffff80 0000000000000000 0000000000000000 ffffffff88010951 > May 2 10:36:25 swlab156 kernel: 0000000000000180 ffffffff8045a000 ffffffff88013000 ffffffff80459fc0 > May 2 10:36:25 swlab156 kernel: ffffffff80459fc0 00007ffffffff408 > May 2 10:36:25 swlab156 kernel: Call Trace:{:ib_mad:ib_mad_init_module+51} {sys_init_module+298} > May 2 10:36:25 swlab156 kernel: {system_call+126} > May 2 10:36:25 swlab156 kernel: > May 2 10:36:25 swlab156 kernel: Code: 0f 0b e5 be 3e 80 ff ff ff ff c0 05 48 8b 1b 48 8b 03 0f 18 > May 2 10:36:25 swlab156 kernel: RIP {kmem_cache_create+1384} RSP That is because the destruction didn't work. I'm not sure it should be expected to. (This is a second level issue which will go away when the first level one is fixed). -- Hal From mst at mellanox.co.il Mon May 2 05:27:34 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Mon, 2 May 2005 15:27:34 +0300 Subject: [openib-general] slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects In-Reply-To: <1115036059.4477.5703.camel@localhost.localdomain> References: <20050502084013.GR16918@mellanox.co.il> <1115036059.4477.5703.camel@localhost.localdomain> Message-ID: <20050502122734.GY16918@mellanox.co.il> Quoting r. Hal Rosenstock : > Subject: Re: [openib-general] slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects > > On Mon, 2005-05-02 at 04:40, Michael S. Tsirkin wrote: > > Hi! > > I have this script to unload all modules: > > > > killall opensm > > sleep 3 > > killall -9 opensm > > modprobe -r ib_ipoib > > modprobe -r ib_umad > > modprobe -r ib_mthca > > > > So, I try to unload the modules while opensm may still be dying. > > Every now and then I see this crash (below), which seems to indicate > > a race condition or leak somewhere around ib_mad or ib_umad. > > My guess is a mad may still be outstanding. > > > > Ideas, anyone? > > > > Further, I'm looking at the mad agent logic and it seems a bit weird that > > ./core/agent.c does kmem_cache_free in agent_send_handler, > > while all allocs are in mad.c. > > This has been brought up before on the list. I think much of this could > (and should) be rewritten now using Sean's helper functions. > > > What prevents an agent from deregistering while a send is outstanding? > > What'll free the mad_priv then? > > Nothing. I think there is some missing code in ib_agent_port_close to > handle this scenario. > > However, unless that MAD from the SM were directed locally (and was > pending), that would not cause the problem where the ib_mad cache could > not be destroyed. I will see if I can recreate this and work up a patch > for this. Interestingly, sometimes the error occurs even if I wait seconds after killing opensm. > > log dump below. > > > > Thanks, > > > > MST > > > > This is with 2.6.11 + rev 2235 (latest bits as of now), x86_64 (Intel Nocona). > > > > > > May 2 10:36:12 swlab156 kernel: slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects > > May 2 10:36:12 swlab156 kernel: > > May 2 10:36:12 swlab156 kernel: Call Trace:{kmem_cache_destroy+184} {:ib_mad:ib_mad_cleanup_module+28} > > May 2 10:36:12 swlab156 kernel: {sys_delete_module+487} {__up_write+28} > > May 2 10:36:12 swlab156 kernel: {sys_munmap+74} {system_call+126} > > May 2 10:36:12 swlab156 kernel: > > May 2 10:36:12 swlab156 kernel: ib_mad: Failed to destroy ib_mad cache > > Has this been occuring for a while or is this new (with the recent > changes to mad handling) ? > I think I remember seeing this a week and a half ago. -- MST - Michael S. Tsirkin From halr at voltaire.com Mon May 2 05:38:41 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 02 May 2005 08:38:41 -0400 Subject: [openib-general] slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects In-Reply-To: <20050502122734.GY16918@mellanox.co.il> References: <20050502084013.GR16918@mellanox.co.il> <1115036059.4477.5703.camel@localhost.localdomain> <20050502122734.GY16918@mellanox.co.il> Message-ID: <1115037521.4477.5719.camel@localhost.localdomain> On Mon, 2005-05-02 at 08:27, Michael S. Tsirkin wrote: > Interestingly, sometimes the error occurs even if I wait seconds > after killing opensm. And I presume opensm doesn't actually terminate ? Also, do you see a message referring to some number of outstanding mads ? > > Has this been occuring for a while or is this new (with the recent > > changes to mad handling) ? > > > > I think I remember seeing this a week and a half ago. OK. Thanks. -- Hal From mst at mellanox.co.il Mon May 2 05:49:01 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Mon, 2 May 2005 15:49:01 +0300 Subject: [openib-general] slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects In-Reply-To: <1115037521.4477.5719.camel@localhost.localdomain> References: <20050502084013.GR16918@mellanox.co.il> <1115036059.4477.5703.camel@localhost.localdomain> <20050502122734.GY16918@mellanox.co.il> <1115037521.4477.5719.camel@localhost.localdomain> Message-ID: <20050502124901.GD16918@mellanox.co.il> Quoting r. Hal Rosenstock : > Subject: Re: [openib-general] slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects > > On Mon, 2005-05-02 at 08:27, Michael S. Tsirkin wrote: > > Interestingly, sometimes the error occurs even if I wait seconds > > after killing opensm. > > And I presume opensm doesn't actually terminate ? Also, do you see a > message referring to some number of outstanding mads ? I think I used to, but not anymore. Now I just see # stopsdp OpenSM: Got signal 15 - exiting... ------- OpenSM Exiting (in 3 seconds) ------- [1]+ Done taskset 1 opensm > > > Has this been occuring for a while or is this new (with the recent > > > changes to mad handling) ? > > > > > > > I think I remember seeing this a week and a half ago. > > OK. Thanks. > > -- Hal > -- MST - Michael S. Tsirkin From mst at mellanox.co.il Mon May 2 07:02:46 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Mon, 2 May 2005 17:02:46 +0300 Subject: [openib-general] SDP: unexpected conn state Message-ID: <20050502140246.GL16918@mellanox.co.il> Hi, Libor! With ttcp, I sometimes see the following messages from sdp_rcvd_disconnect: WARN: : Unexpected conn state. conn <1> state <0404:2480> After this, I cant bind to the same port without unloading mod_sdp. Why could that be? -- MST - Michael S. Tsirkin From gdror at mellanox.co.il Mon May 2 07:18:19 2005 From: gdror at mellanox.co.il (Dror Goldenberg) Date: Mon, 2 May 2005 17:18:19 +0300 Subject: [openib-general] FMR and how they work Message-ID: <506C3D7B14CDD411A52C00025558DED6078F3995@mtlex01.yok.mtl.com> > -----Original Message----- > From: Fab Tillier [mailto:ftillier at infiniconsys.com] > Sent: Monday, May 02, 2005 4:50 AM > > Is the HW2SW_MPT command more expensive than the SYNC_TPT > command? SYNC_TPT wipes out the caches. HW2SW_MPT cleans caches for the specific MPT. You're welcome to compare the two execution time. I believe that the main impact on CPU is by the fact that you need to submit a command and wait for it to complete (interrupt->EQE). > Does it just flush the MPT out of the cache? What > happens to that MPT's MTTs - do they get flushed out too > (assuming the MPT still references them)? For correctness you'd have to flush MPTs and MTTs. > > Why aren't FMR's bindable? They are not. Check out the PRM. > It seems that if the HW2SW_MPT > flushes the MPT out of the cache, then one could use it for > normal memory registrations and avoid the WRITE_MTT command, > no? Please explain. Maybe you meant to use HW2SW_MPT for Deregistration. If this is what you mean, then you're right. But it is against the spirit of "lazy deregistration". This will only speed the IO operations halfway. On the creation of mapping it'll be fast - just posted writes to the MPT/MTT, on the destruction of the mapping, it'll be slow - because you need HW2SW_MPT and you'd need probably to create a new blank MR for the next reuse. The intent was to use bulk deregistrations... > That is, during registration, do posted writes for the > MTTs and then a SW2HW_MPT command for an MPT entry that > references those MTTs. Would this work? > > How much slower are memory windows compared to FMRs (assuming > the underlying MR is already registered)? You're welcomed to measure. I think that FMRs will be the fastest way to create mapping. Faster than MWs. Also note that FMRs are not bindable. So, there are certain applications that can benefit from FMRs (e.g. SRP), but can not leverage MWs, at least I can't think of a model for that... > > Thanks, > > - Fab > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mst at mellanox.co.il Mon May 2 07:21:48 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Mon, 2 May 2005 17:21:48 +0300 Subject: [openib-general] Re: SDP: unexpected conn state In-Reply-To: <20050502140246.GL16918@mellanox.co.il> References: <20050502140246.GL16918@mellanox.co.il> Message-ID: <20050502142148.GA9674@mellanox.co.il> Quoting r. Michael S. Tsirkin : > Subject: SDP: unexpected conn state > > Hi, Libor! > > With ttcp, I sometimes see the following messages from sdp_rcvd_disconnect: > > WARN: : Unexpected conn state. conn <1> state <0404:2480> > > After this, I cant bind to the same port without unloading mod_sdp. > Why could that be? > > -- > MST - Michael S. Tsirkin This happened on the ttcp.aio.x server side when the client had the ulimit set too low. It goes away if I set the limit high enough. -- MST - Michael S. Tsirkin From steve at wooding.uklinux.net Mon May 2 07:48:51 2005 From: steve at wooding.uklinux.net (Steven Wooding) Date: Mon, 2 May 2005 15:48:51 +0100 Subject: [openib-general] [PATCH][SDP] AIO buffer corruption Message-ID: <200505021548.52053.steve@wooding.uklinux.net> Hello Libor, I've tried your patch, but unfortunately it made no difference to the -32/-104 errors I get. I have observed the following features, which may help you diagnose my problem: 1. Sometimes the receiver reports -32 and the transmitter -104, but other times the errors are reversed. 2. The number of ping-pongs seems to be the main factor. However, the value of -n at which ttcp.aio fails is not fixed, if all other parameters are the same. 3. -32/-104 error occurs for a much smaller value of -n if -a is 1 to 3. 4. This maybe unrelated, but I see very poor performance for -a = 1 (50 - 75 MB/s). If -a is larger that 4 then the performance comes back (400 - 900 MB/s). Is there are other information I can supply to help you? Hope this finds you well. Cheers, Steve. Platform: 64 bit OS: RHEL 4 Kernel: 2.6.11.6 OpenIB gen2: 2225 SM: IO5000 switch From jlentini at netapp.com Mon May 2 09:07:02 2005 From: jlentini at netapp.com (James Lentini) Date: Mon, 2 May 2005 12:07:02 -0400 (EDT) Subject: [openib-general] Re: [PATCH][DAPL] make dapl build outside of kernel tree In-Reply-To: <1114795779.24949.3.camel@duffman> References: <1114633684.20016.8.camel@duffman> <1114795779.24949.3.camel@duffman> Message-ID: Tom, I just committed the IBAT Makefile. For now, I'd like to stick with our current DAT and DAT Provider Makefiles. I've added your makefiles as patches to ours. NFS needs to be built as part of the kernel, so we need the option of building the DAT modules that way as well. james On Fri, 29 Apr 2005, Tom Duffy wrote: > On Fri, 2005-04-29 at 12:43 -0400, James Lentini wrote: >> Tom, >> >> Did you try building using the method described in the README file? > > No. > >> svn/gen2/users/jlentini/linux-kernel/README >> >> The procedure there has been working for me. I'd rather continue using >> it than change. I like the idea that the build setup is exactly as it >> would be if it were part of the trunk. > > Once the code is integrated into trunk, it should follow the in kernel > method. Until then, it should build outside of the tree. > > I think the Makefiles in your branch should build outside the tree. If > you want Makefiles that build inside the tree, provide those as a patch. > > -tduffy > From mshefty at ichips.intel.com Mon May 2 09:07:03 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Mon, 02 May 2005 09:07:03 -0700 Subject: [openib-general] slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects In-Reply-To: <1115036059.4477.5703.camel@localhost.localdomain> References: <20050502084013.GR16918@mellanox.co.il> <1115036059.4477.5703.camel@localhost.localdomain> Message-ID: <42765027.10605@ichips.intel.com> Hal Rosenstock wrote: >>What prevents an agent from deregistering while a send is outstanding? >>What'll free the mad_priv then? > > > Nothing. I think there is some missing code in ib_agent_port_close to > handle this scenario. > > However, unless that MAD from the SM were directed locally (and was > pending), that would not cause the problem where the ib_mad cache could > not be destroyed. I will see if I can recreate this and work up a patch > for this. Sending a MAD will result in taking a reference on the corresponding MAD agent. If unregister is called, the call will first cancel all outstanding sends, and then wait for the reference count to go to zero. It should be noted that until unregister returns MADs can be received by the user, and the MADs must be freed. - Sean From mshefty at ichips.intel.com Mon May 2 09:09:35 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Mon, 02 May 2005 09:09:35 -0700 Subject: [openib-general] Re: Some Initial RMPP Comments and Questions In-Reply-To: <1114860788.4477.1779.camel@localhost.localdomain> References: <1114802602.4477.931.camel@localhost.localdomain> <4272B0AC.1050405@ichips.intel.com> <1114860788.4477.1779.camel@localhost.localdomain> Message-ID: <427650BF.3090004@ichips.intel.com> Hal Rosenstock wrote: >>Since you have to do a data >>copy for usermode anyway, I think it makes sense to just return the >>coalesced buffer. This makes me think that I should implement >>ib_coalesce_recv_mad() now. > > > That would make the receive side of user space much easier. Shall I > defer implementing this part until coalesce is implemented ? If you want to go ahead and do this part of the implementation, you can just call coalesce and assume that it'll be there shortly... - Sean From mst at mellanox.co.il Mon May 2 09:33:27 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Mon, 2 May 2005 19:33:27 +0300 Subject: [openib-general] slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects In-Reply-To: <42765027.10605@ichips.intel.com> References: <20050502084013.GR16918@mellanox.co.il> <1115036059.4477.5703.camel@localhost.localdomain> <42765027.10605@ichips.intel.com> Message-ID: <20050502163327.GB9939@mellanox.co.il> Quoting r. Sean Hefty : > Subject: Re: [openib-general] slab error in kmem_cache_destroy(): cache?`ib_mad': Can't free all objects > > Hal Rosenstock wrote: > >>What prevents an agent from deregistering while a send is outstanding? > >>What'll free the mad_priv then? > > > > > >Nothing. I think there is some missing code in ib_agent_port_close to > >handle this scenario. > > > >However, unless that MAD from the SM were directed locally (and was > >pending), that would not cause the problem where the ib_mad cache could > >not be destroyed. I will see if I can recreate this and work up a patch > >for this. > > Sending a MAD will result in taking a reference on the corresponding > MAD agent. If unregister is called, the call will first cancel all > outstanding sends, and then wait for the reference count to go to zero. > > It should be noted that until unregister returns MADs can be received > by the user, and the MADs must be freed. > > - Sean > Do you have a theory on why are MADs leaking? -- MST - Michael S. Tsirkin From halr at voltaire.com Mon May 2 09:29:26 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 02 May 2005 12:29:26 -0400 Subject: [openib-general] slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects In-Reply-To: <42765027.10605@ichips.intel.com> References: <20050502084013.GR16918@mellanox.co.il> <1115036059.4477.5703.camel@localhost.localdomain> <42765027.10605@ichips.intel.com> Message-ID: <1115051091.4477.6033.camel@localhost.localdomain> On Mon, 2005-05-02 at 12:07, Sean Hefty wrote: > Hal Rosenstock wrote: > >>What prevents an agent from deregistering while a send is outstanding? > >>What'll free the mad_priv then? > > > > > > Nothing. I think there is some missing code in ib_agent_port_close to > > handle this scenario. > > > > However, unless that MAD from the SM were directed locally (and was > > pending), that would not cause the problem where the ib_mad cache could > > not be destroyed. I will see if I can recreate this and work up a patch > > for this. > > Sending a MAD will result in taking a reference on the corresponding > MAD agent. If unregister is called, the call will first cancel all > outstanding sends, and then wait for the reference count to go to zero. > > It should be noted that until unregister returns MADs can be received > by the user, and the MADs must be freed. I stand corrected. This is already taken care of. I forgot about this. -- Hal From halr at voltaire.com Mon May 2 09:32:39 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 02 May 2005 12:32:39 -0400 Subject: [openib-general] Re: Some Initial RMPP Comments and Questions In-Reply-To: <427650BF.3090004@ichips.intel.com> References: <1114802602.4477.931.camel@localhost.localdomain> <4272B0AC.1050405@ichips.intel.com> <1114860788.4477.1779.camel@localhost.localdomain> <427650BF.3090004@ichips.intel.com> Message-ID: <1115051127.4477.6035.camel@localhost.localdomain> On Mon, 2005-05-02 at 12:09, Sean Hefty wrote: > Hal Rosenstock wrote: > >>Since you have to do a data > >>copy for usermode anyway, I think it makes sense to just return the > >>coalesced buffer. This makes me think that I should implement > >>ib_coalesce_recv_mad() now. > > > > > > That would make the receive side of user space much easier. Shall I > > defer implementing this part until coalesce is implemented ? > > If you want to go ahead and do this part of the implementation, you can > just call coalesce and assume that it'll be there shortly... Sounds good. Thanks. -- Hal From mshefty at ichips.intel.com Mon May 2 09:55:24 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Mon, 02 May 2005 09:55:24 -0700 Subject: [openib-general] slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects In-Reply-To: <20050502163327.GB9939@mellanox.co.il> References: <20050502084013.GR16918@mellanox.co.il> <1115036059.4477.5703.camel@localhost.localdomain> <42765027.10605@ichips.intel.com> <20050502163327.GB9939@mellanox.co.il> Message-ID: <42765B7C.9000903@ichips.intel.com> Michael S. Tsirkin wrote: > Do you have a theory on why are MADs leaking? No, but I don't see where the user_mad code would cause it. It also doesn't sound like anything invoked the RMPP code, the CM, or SA query code, which narrows the scope of the problem. Is this the case? - Sean From ashah at xsigo.com Mon May 2 10:04:07 2005 From: ashah at xsigo.com (Ashit Shah) Date: Mon, 2 May 2005 10:04:07 -0700 Subject: [openib-general] Broken link in www.openib.org Message-ID: The link works ok. May be acrobat reader is not installed on your node. Ashit Shah -----Original Message----- From: openib-general-bounces at openib.org [mailto:openib-general-bounces at openib.org] On Behalf Of Roy at SEVENtwentyfour Sent: Sunday, May 01, 2005 4:44 AM To: openib-general at openib.org Subject: [openib-general] Broken link in www.openib.org There appears to be a problem on this page of your site. On page http://www.openib.org/workshop.html when you click on "pdf", the link to http://www.openib.org/docs/oib_wkshp_022005/openmpi-sandia-sukalski.pdf gives the error: Not found. As recommended by the Robot Guidelines, this email is to explain our robot's visit to your site, and to let you know about one of the problems we found. We don't store or publish the content of your pages, but rather use the link information to update our map of the World Wide Web. Are these reports helpful? I'd love some feedback. If you prefer not to receive these occasional error notices please let me know. Roy Bryant ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Roy Bryant, roybryant at seventwentyfour.com President SEVENtwentyfour Inc. http://www.seventwentyfour.com ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ _______________________________________________ openib-general mailing list openib-general at openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From mst at mellanox.co.il Mon May 2 10:08:52 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Mon, 2 May 2005 20:08:52 +0300 Subject: [openib-general] Re: slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects In-Reply-To: <42765B7C.9000903@ichips.intel.com> References: <20050502084013.GR16918@mellanox.co.il> <1115036059.4477.5703.camel@localhost.localdomain> <42765027.10605@ichips.intel.com> <20050502163327.GB9939@mellanox.co.il> <42765B7C.9000903@ichips.intel.com> Message-ID: <20050502170852.GA10291@mellanox.co.il> Quoting r. Sean Hefty : > Subject: Re: slab error in kmem_cache_destroy(): cache?`ib_mad': Can't free all objects > > Michael S. Tsirkin wrote: > > >Do you have a theory on why are MADs leaking? > > No, but I don't see where the user_mad code would cause it. It also > doesn't sound like anything invoked the RMPP code, the CM, or SA query > code, which narrows the scope of the problem. Is this the case? > > - Sean > I only see it with opensm. -- MST - Michael S. Tsirkin From halr at voltaire.com Mon May 2 10:32:37 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 02 May 2005 13:32:37 -0400 Subject: [openib-general] Re: slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects In-Reply-To: <20050502172542.GA10506@mellanox.co.il> References: <20050502084013.GR16918@mellanox.co.il> <1115036059.4477.5703.camel@localhost.localdomain> <42765027.10605@ichips.intel.com> <20050502163327.GB9939@mellanox.co.il> <42765B7C.9000903@ichips.intel.com> <20050502170852.GA10291@mellanox.co.il> <1115053681.4477.6073.camel@localhost.localdomain> <20050502172542.GA10506@mellanox.co.il> Message-ID: <1115055004.4477.6100.camel@localhost.localdomain> On Mon, 2005-05-02 at 13:25, Michael S. Tsirkin wrote: > > I only see it with opensm. > > > > This may be something with the OpenIB vendor layer of OpenSM. I will > > look at it to see if I can see something. > > > > -- Hal > > > Even if it is, umad must prevent buggy app from crashing the kernel. Sure. I wasn't suggesting otherwise. If it is opensm, it somehow has to be umad related as the vendor layer works through this. -- Hal From mst at mellanox.co.il Mon May 2 10:49:58 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Mon, 2 May 2005 20:49:58 +0300 Subject: [openib-general] Re: slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects In-Reply-To: <1115055004.4477.6100.camel@localhost.localdomain> References: <20050502084013.GR16918@mellanox.co.il> <1115036059.4477.5703.camel@localhost.localdomain> <42765027.10605@ichips.intel.com> <20050502163327.GB9939@mellanox.co.il> <42765B7C.9000903@ichips.intel.com> <20050502170852.GA10291@mellanox.co.il> <1115053681.4477.6073.camel@localhost.localdomain> <20050502172542.GA10506@mellanox.co.il> <1115055004.4477.6100.camel@localhost.localdomain> Message-ID: <20050502174958.GA10586@mellanox.co.il> Quoting r. Hal Rosenstock : > Subject: Re: slab error in kmem_cache_destroy(): cache?`ib_mad': Can't free all objects > > On Mon, 2005-05-02 at 13:25, Michael S. Tsirkin wrote: > > > I only see it with opensm. > > > > > > This may be something with the OpenIB vendor layer of OpenSM. I will > > > look at it to see if I can see something. > > > > > > -- Hal > > > > > > Even if it is, umad must prevent buggy app from crashing the kernel. > > Sure. I wasn't suggesting otherwise. If it is opensm, it somehow has to > be umad related as the vendor layer works through this. > > -- Hal > I now can do it as easy as: udevstart modprobe ib_mthca modprobe ib_umad opensm& killall opensm sleep 4 modprobe -r ib_umad modprobe -r ib_mthca error in dmesg: slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects can anyone preorduce this? -- MST - Michael S. Tsirkin From halr at voltaire.com Mon May 2 11:09:57 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 02 May 2005 14:09:57 -0400 Subject: [openib-general] Some more RMPP Questions and Comments (mad_rmpp.c) Message-ID: <1115057396.4477.6120.camel@localhost.localdomain> Hi again Sean, from rmpp.c::ib_create_send_mad ... if (mad_agent->rmpp_version) { struct ib_rmpp_mad *rmpp_mad; rmpp_mad = (struct ib_rmpp_mad *)send_buf->mad; rmpp_mad->rmpp_hdr.paylen_newwin = cpu_to_be32(hdr_len - offsetof(struct ib_rmpp_mad, data) + data_len); } Should the above also be based on whether RMPP is active in the MAD to be sent ? I guess this depends on whether one MAD agent can be shared for RMPP and non RMPP sends. Also from rmpp.c::ib_create_send_mad: send_buf->send_wr.wr.ud.remote_qkey = IB_QP_SET_QKEY; Should the remote_qkey be a passed in parameter to this routine ? from mad_rmpp.c: static int data_offset(u8 mgmt_class) { if (mgmt_class == IB_MGMT_CLASS_SUBN_ADM) return offsetof(struct ib_sa_mad, data); else if ((mgmt_class >= IB_MGMT_CLASS_VENDOR_RANGE2_START) && (mgmt_class <= IB_MGMT_CLASS_VENDOR_RANGE2_END)) return offsetof(struct ib_vendor_mad, data); else return offsetof(struct ib_rmpp_mad, data); } Seems like there are a number of places where vendor range 2 is checked. There already is a (static) function for this in mad.c. Should this be used (and made global) ? -- Hal From halr at voltaire.com Mon May 2 11:14:49 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 02 May 2005 14:14:49 -0400 Subject: [openib-general] Re: slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects In-Reply-To: <20050502174958.GA10586@mellanox.co.il> References: <20050502084013.GR16918@mellanox.co.il> <1115036059.4477.5703.camel@localhost.localdomain> <42765027.10605@ichips.intel.com> <20050502163327.GB9939@mellanox.co.il> <42765B7C.9000903@ichips.intel.com> <20050502170852.GA10291@mellanox.co.il> <1115053681.4477.6073.camel@localhost.localdomain> <20050502172542.GA10506@mellanox.co.il> <1115055004.4477.6100.camel@localhost.localdomain> <20050502174958.GA10586@mellanox.co.il> Message-ID: <1115057689.4477.6122.camel@localhost.localdomain> On Mon, 2005-05-02 at 13:49, Michael S. Tsirkin wrote: > I now can do it as easy as: > > udevstart > modprobe ib_mthca > modprobe ib_umad > > opensm& > > killall opensm > > sleep 4 > > modprobe -r ib_umad > modprobe -r ib_mthca > > error in dmesg: > slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects > > can anyone preorduce this? I couldn't reproduce this on either x86 or x86_64. -- Hal From mlleinin at hpcn.ca.sandia.gov Mon May 2 11:42:43 2005 From: mlleinin at hpcn.ca.sandia.gov (Matt Leininger) Date: Mon, 02 May 2005 11:42:43 -0700 Subject: [openib-general] Broken link in www.openib.org In-Reply-To: References: Message-ID: <1115059363.12288.677.camel@localhost> I fixed the link that's why it works now. Thanks for finding the problem. - Matt On Mon, 2005-05-02 at 10:04 -0700, Ashit Shah wrote: > The link works ok. May be acrobat reader is not installed on your > node. > > Ashit Shah > > -----Original Message----- > From: openib-general-bounces at openib.org > [mailto:openib-general-bounces at openib.org] On Behalf Of Roy at > SEVENtwentyfour > Sent: Sunday, May 01, 2005 4:44 AM > To: openib-general at openib.org > Subject: [openib-general] Broken link in www.openib.org > > There appears to be a problem on this page of your site. > > On page http://www.openib.org/workshop.html > when you click on "pdf", > the link to > http://www.openib.org/docs/oib_wkshp_022005/openmpi-sandia-sukalski.pdf > gives the error: Not found. > > As recommended by the Robot Guidelines, this email is to explain > our robot's visit to your site, and to let you know about one of > the problems we found. We don't store or publish the content of > your pages, but rather use the link information to update our map > of the World Wide Web. > > Are these reports helpful? I'd love some feedback. If you prefer > not to receive these occasional error notices please let me know. > > Roy Bryant > > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > Roy Bryant, roybryant at seventwentyfour.com > President > SEVENtwentyfour Inc. > http://www.seventwentyfour.com > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From mshefty at ichips.intel.com Mon May 2 11:46:16 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Mon, 02 May 2005 11:46:16 -0700 Subject: [openib-general] Re: Some more RMPP Questions and Comments (mad_rmpp.c) In-Reply-To: <1115057396.4477.6120.camel@localhost.localdomain> References: <1115057396.4477.6120.camel@localhost.localdomain> Message-ID: <42767578.10908@ichips.intel.com> Hal Rosenstock wrote: > from rmpp.c::ib_create_send_mad > ... > if (mad_agent->rmpp_version) { > struct ib_rmpp_mad *rmpp_mad; > rmpp_mad = (struct ib_rmpp_mad *)send_buf->mad; > rmpp_mad->rmpp_hdr.paylen_newwin = cpu_to_be32(hdr_len - > offsetof(struct ib_rmpp_mad, data) + data_len); > } > > Should the above also be based on whether RMPP is active in the MAD to > be sent ? I guess this depends on whether one MAD agent can be shared > for RMPP and non RMPP sends. ib_create_send_mad doesn't have the information to know if the MAD will active RMPP or not, just that the header is present. Hmm... I was originally thinking that it wouldn't matter if this field was set or not, but if RMPP is not active, the field is supposed to be set to 0. I can think of a couple of options here. We can extend this API to indicate if RMPP is active or not. Have the user clear this field if RMPP is not active. Or let the RMPP code clear the field when sending the MAD... It might even make sense to have calls to initialize the MAD and RMPP header information given a list of parameters. > Also from rmpp.c::ib_create_send_mad: > > send_buf->send_wr.wr.ud.remote_qkey = IB_QP_SET_QKEY; > > Should the remote_qkey be a passed in parameter to this routine ? The user could override this once the MAD has been returned, but I don't object to adding this... > from mad_rmpp.c: > static int data_offset(u8 mgmt_class) > { > if (mgmt_class == IB_MGMT_CLASS_SUBN_ADM) > return offsetof(struct ib_sa_mad, data); > else if ((mgmt_class >= IB_MGMT_CLASS_VENDOR_RANGE2_START) && > (mgmt_class <= IB_MGMT_CLASS_VENDOR_RANGE2_END)) > return offsetof(struct ib_vendor_mad, data); > else > return offsetof(struct ib_rmpp_mad, data); > } > > Seems like there are a number of places where vendor range 2 is checked. > There already is a (static) function for this in mad.c. Should this be used > (and made global) ? I thought about this too, but didn't think that this check was complex or long enough to create a function for it... I doubt many clients outside of the MAD layer itself would need such a check however, so sharing it between mad.c and mad_rmpp.c seems fine, but I doubt we need an exposed API. - Sean From mshefty at ichips.intel.com Mon May 2 12:02:49 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Mon, 02 May 2005 12:02:49 -0700 Subject: [openib-general] Re: Another RMPP related question In-Reply-To: <1114958304.4477.4937.camel@localhost.localdomain> References: <1114958304.4477.4937.camel@localhost.localdomain> Message-ID: <42767959.9060506@ichips.intel.com> Hal Rosenstock wrote: > Hi Sean, > > Is a MAD agent which registers for RMPP use restricted to using > ib_create_send_mad (even when not sending a RMPP MAD) ? That may be a > current restriction despite the comments for tha routing indicating > "Users are not required to allocate outbound MADs using this call". > > If that is the case, either the comment should be changed to indicate > this or the restriction removed. There is not intended to be such a restriction. Can you point me to where the code assumes this? - Sean From mshefty at ichips.intel.com Mon May 2 12:07:21 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Mon, 02 May 2005 12:07:21 -0700 Subject: [openib-general] Re: Some Initial RMPP Comments and Questions In-Reply-To: <1115051127.4477.6035.camel@localhost.localdomain> References: <1114802602.4477.931.camel@localhost.localdomain> <4272B0AC.1050405@ichips.intel.com> <1114860788.4477.1779.camel@localhost.localdomain> <427650BF.3090004@ichips.intel.com> <1115051127.4477.6035.camel@localhost.localdomain> Message-ID: <42767A69.8050006@ichips.intel.com> Hal Rosenstock wrote: >>>That would make the receive side of user space much easier. Shall I >>>defer implementing this part until coalesce is implemented ? >> >>If you want to go ahead and do this part of the implementation, you can >>just call coalesce and assume that it'll be there shortly... > > > Sounds good. Thanks. I'm considering changing the API to this: ib_coalesce_recv_mad(mad_agent, mad_recv_wc, user_buffer, buf) The mad_agent is needed in order to know if there's an RMPP header. If there is, then there may be padding at the end of the MAD which the user didn't allocate buffer space for. The user_buffer parameter would be used to indicate which copy routine to user: memcpy or copy_to_user. - Sean From halr at voltaire.com Mon May 2 12:01:26 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 02 May 2005 15:01:26 -0400 Subject: [openib-general] Re: slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects In-Reply-To: <1115057689.4477.6122.camel@localhost.localdomain> References: <20050502084013.GR16918@mellanox.co.il> <1115036059.4477.5703.camel@localhost.localdomain> <42765027.10605@ichips.intel.com> <20050502163327.GB9939@mellanox.co.il> <42765B7C.9000903@ichips.intel.com> <20050502170852.GA10291@mellanox.co.il> <1115053681.4477.6073.camel@localhost.localdomain> <20050502172542.GA10506@mellanox.co.il> <1115055004.4477.6100.camel@localhost.localdomain> <20050502174958.GA10586@mellanox.co.il> <1115057689.4477.6122.camel@localhost.localdomain> Message-ID: <1115060486.4476.0.camel@localhost.localdomain> On Mon, 2005-05-02 at 14:14, Hal Rosenstock wrote: > I couldn't reproduce this on either x86 or x86_64. I can now reproduce it on x86. Guess I needed to try harder :-) -- Hal From halr at voltaire.com Mon May 2 12:23:14 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 02 May 2005 15:23:14 -0400 Subject: [openib-general] Re: Some Initial RMPP Comments and Questions In-Reply-To: <42767A69.8050006@ichips.intel.com> References: <1114802602.4477.931.camel@localhost.localdomain> <4272B0AC.1050405@ichips.intel.com> <1114860788.4477.1779.camel@localhost.localdomain> <427650BF.3090004@ichips.intel.com> <1115051127.4477.6035.camel@localhost.localdomain> <42767A69.8050006@ichips.intel.com> Message-ID: <1115061793.4476.44.camel@localhost.localdomain> On Mon, 2005-05-02 at 15:07, Sean Hefty wrote: > I'm considering changing the API to this: > > ib_coalesce_recv_mad(mad_agent, mad_recv_wc, user_buffer, buf) > > The mad_agent is needed in order to know if there's an RMPP header. If > there is, then there may be padding at the end of the MAD which the > user didn't allocate buffer space for. The user_buffer parameter would > be used to indicate which copy routine to user: memcpy or copy_to_user. Sounds right to me. -- Hal From halr at voltaire.com Mon May 2 12:30:19 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 02 May 2005 15:30:19 -0400 Subject: [openib-general] Re: Some more RMPP Questions and Comments (mad_rmpp.c) In-Reply-To: <42767578.10908@ichips.intel.com> References: <1115057396.4477.6120.camel@localhost.localdomain> <42767578.10908@ichips.intel.com> Message-ID: <1115062218.4476.60.camel@localhost.localdomain> On Mon, 2005-05-02 at 14:46, Sean Hefty wrote: > Hal Rosenstock wrote: > > from rmpp.c::ib_create_send_mad > > ... > > if (mad_agent->rmpp_version) { > > struct ib_rmpp_mad *rmpp_mad; > > rmpp_mad = (struct ib_rmpp_mad *)send_buf->mad; > > rmpp_mad->rmpp_hdr.paylen_newwin = cpu_to_be32(hdr_len - > > offsetof(struct ib_rmpp_mad, data) + data_len); > > } > > > > Should the above also be based on whether RMPP is active in the MAD to > > be sent ? I guess this depends on whether one MAD agent can be shared > > for RMPP and non RMPP sends. > > ib_create_send_mad doesn't have the information to know if the MAD will > active RMPP or not, just that the header is present. Right but it seems to me it needs this information. > Hmm... I was originally thinking that it wouldn't matter if this field was set or > not, but if RMPP is not active, the field is supposed to be set to 0. And also visa versa, if RMPP is active, the field is supposed to have the ACTIVE bit on. > I can think of a couple of options here. We can extend this API to > indicate if RMPP is active or not. That's what I thought at first. > Have the user clear this field if RMPP is not active. This seems fine to me and consistent with how remote_qkey is being handled. This should be documented. > Or let the RMPP code clear the field when sending > the MAD... How would it know to do this ? > It might even make sense to have calls to initialize the MAD and RMPP > header information given a list of parameters. Yes, that would save each consumer from doing this explictly. > > Also from rmpp.c::ib_create_send_mad: > > > > send_buf->send_wr.wr.ud.remote_qkey = IB_QP_SET_QKEY; > > > > Should the remote_qkey be a passed in parameter to this routine ? > > The user could override this once the MAD has been returned, but I > don't object to adding this... True. Should this be added to the comments ? > > from mad_rmpp.c: > > static int data_offset(u8 mgmt_class) > > { > > if (mgmt_class == IB_MGMT_CLASS_SUBN_ADM) > > return offsetof(struct ib_sa_mad, data); > > else if ((mgmt_class >= IB_MGMT_CLASS_VENDOR_RANGE2_START) && > > (mgmt_class <= IB_MGMT_CLASS_VENDOR_RANGE2_END)) > > return offsetof(struct ib_vendor_mad, data); > > else > > return offsetof(struct ib_rmpp_mad, data); > > } > > > > Seems like there are a number of places where vendor range 2 is checked. > > There already is a (static) function for this in mad.c. Should this be used > > (and made global) ? > > I thought about this too, but didn't think that this check was complex > or long enough to create a function for it... I doubt many clients > outside of the MAD layer itself would need such a check however, so > sharing it between mad.c and mad_rmpp.c seems fine, but I doubt we need > an exposed API. OK. -- Hal From mshefty at ichips.intel.com Mon May 2 12:58:29 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Mon, 02 May 2005 12:58:29 -0700 Subject: [openib-general] Re: Some more RMPP Questions and Comments (mad_rmpp.c) In-Reply-To: <1115062218.4476.60.camel@localhost.localdomain> References: <1115057396.4477.6120.camel@localhost.localdomain> <42767578.10908@ichips.intel.com> <1115062218.4476.60.camel@localhost.localdomain> Message-ID: <42768665.5090903@ichips.intel.com> Hal Rosenstock wrote: >>Hmm... I was originally thinking that it wouldn't matter if this field was set or >>not, but if RMPP is not active, the field is supposed to be set to 0. > > And also visa versa, if RMPP is active, the field is supposed to have > the ACTIVE bit on. Currently, users need to set the active bit, and are responsible for setting most other MAD and RMPP header fields. >>I can think of a couple of options here. We can extend this API to >>indicate if RMPP is active or not. > > That's what I thought at first. The only "issue" that I see with doing this is that I tried to avoid having create_send_mad format the MAD. I made an exception for the payload length, since the buffer size is needed to make RMPP work. Maybe the right solution here is to return the payload length rather than setting it in the MAD. >>Have the user clear this field if RMPP is not active. > > This seems fine to me and consistent with how remote_qkey is being > handled. This should be documented. > >> Or let the RMPP code clear the field when sending >>the MAD... > > How would it know to do this ? The code would know based on whether the active bit was set. If not, it can clear the payload field. I'm not as fond of this option, since the user is responsible for clearing/setting the other header fields. >>It might even make sense to have calls to initialize the MAD and RMPP >>header information given a list of parameters. > > > Yes, that would save each consumer from doing this explictly. It can better ensure that all of the necessary fields are set. > >>>Also from rmpp.c::ib_create_send_mad: >>> >>>send_buf->send_wr.wr.ud.remote_qkey = IB_QP_SET_QKEY; >>> >>>Should the remote_qkey be a passed in parameter to this routine ? >> >>The user could override this once the MAD has been returned, but I >>don't object to adding this... > > > True. Should this be added to the comments ? Sure ... I can do that this afternoon. - Sean From halr at voltaire.com Mon May 2 12:58:37 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 02 May 2005 15:58:37 -0400 Subject: [openib-general] Re: Some more RMPP Questions and Comments (mad_rmpp.c) In-Reply-To: <1115062218.4476.60.camel@localhost.localdomain> References: <1115057396.4477.6120.camel@localhost.localdomain> <42767578.10908@ichips.intel.com> <1115062218.4476.60.camel@localhost.localdomain> Message-ID: <1115063917.4476.93.camel@localhost.localdomain> On Mon, 2005-05-02 at 15:30, Hal Rosenstock wrote: > On Mon, 2005-05-02 at 14:46, Sean Hefty wrote: > > Hal Rosenstock wrote: > > > from rmpp.c::ib_create_send_mad > > > ... > > > if (mad_agent->rmpp_version) { > > > struct ib_rmpp_mad *rmpp_mad; > > > rmpp_mad = (struct ib_rmpp_mad *)send_buf->mad; > > > rmpp_mad->rmpp_hdr.paylen_newwin = cpu_to_be32(hdr_len - > > > offsetof(struct ib_rmpp_mad, data) + data_len); > > > } > > > > > > Should the above also be based on whether RMPP is active in the MAD to > > > be sent ? I guess this depends on whether one MAD agent can be shared > > > for RMPP and non RMPP sends. > > > > ib_create_send_mad doesn't have the information to know if the MAD will > > active RMPP or not, just that the header is present. > > Right but it seems to me it needs this information. > > > Hmm... I was originally thinking that it wouldn't matter if this field was set or > > not, but if RMPP is not active, the field is supposed to be set to 0. > > And also visa versa, if RMPP is active, the field is supposed to have > the ACTIVE bit on. > > > I can think of a couple of options here. We can extend this API to > > indicate if RMPP is active or not. > > That's what I thought at first. > > > Have the user clear this field if RMPP is not active. > > This seems fine to me and consistent with how remote_qkey is being > handled. This should be documented. This routine already needs hdr_len and data_len to calculate the buffer size so doing this does not remove the need to pass these to this routine. Plus it seems this would be easy to screw up if left to the user: rmpp_mad->rmpp_hdr.paylen_newwin = cpu_to_be32(hdr_len - offsetof(struct ib_rmpp_mad, data) + data_len); So I'm having second thoughts... -- Hal From tduffy at sun.com Mon May 2 13:08:37 2005 From: tduffy at sun.com (Tom Duffy) Date: Mon, 02 May 2005 13:08:37 -0700 Subject: [openib-general] [PATCH][DAPL] fix kdapltest Makefile Message-ID: <1115064517.11792.4.camel@duffman> This fixes the problem of building kdapltest on x86_64. It also makes it build outside of a kernel without dat installed. Also, if I want verbose, I will specify verbose; please don't do it for me. Signed-off-by: Tom Duffy Index: gen2/users/jlentini/linux-kernel/test/dapltest/kdapl/Makefile =================================================================== --- gen2/users/jlentini/linux-kernel/test/dapltest/kdapl/Makefile (revision 2247) +++ gen2/users/jlentini/linux-kernel/test/dapltest/kdapl/Makefile (working copy) @@ -37,7 +37,7 @@ ifeq ($(origin KERNELRELEASE), undefined) KDAPLTEST_ROOT = `pwd` # directory that contains this makefile -KERNEL_ROOT = /lib/modules/`uname -r`/build # root of kernel sources +KERNEL_ROOT ?= /lib/modules/`uname -r`/build # root of kernel sources # Tools ECHO = /bin/echo @@ -45,7 +45,7 @@ RM = /bin/rm all : @$(ECHO) "--- Compiling kdapltest module ---" - $(MAKE) -C $(KERNEL_ROOT) M=$(KDAPLTEST_ROOT) V=1 modules + $(MAKE) -C $(KERNEL_ROOT) M=$(KDAPLTEST_ROOT) modules clean : $(RM) -rf \ @@ -67,6 +67,9 @@ ifeq (${IS_i686},i686) # Override -msoft-float in arch/i386/Makefile EXTRA_CFLAGS += -mhard-float endif +ifeq (${IS_i686},x86_64) +EXTRA_CFLAGS += -msse +endif EXTRA_CFLAGS += \ -Idrivers/dat \ @@ -74,7 +77,9 @@ EXTRA_CFLAGS += \ -I$(src)/../mdep/linux \ -I$(src)/../kdapl \ -I$(src)/../common \ - -I$(src)/../test + -I$(src)/../test \ + -I$(src)/../../../dat-provider \ + -I$(src)/../../../dat EXTRA_CFLAGS += \ -D__KDAPL__ \ From halr at voltaire.com Mon May 2 13:03:27 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 02 May 2005 16:03:27 -0400 Subject: [openib-general] Re: Another RMPP related question In-Reply-To: <42767959.9060506@ichips.intel.com> References: <1114958304.4477.4937.camel@localhost.localdomain> <42767959.9060506@ichips.intel.com> Message-ID: <1115064206.4476.101.camel@localhost.localdomain> On Mon, 2005-05-02 at 15:02, Sean Hefty wrote: > Hal Rosenstock wrote: > > Is a MAD agent which registers for RMPP use restricted to using > > ib_create_send_mad (even when not sending a RMPP MAD) ? That may be a > > current restriction despite the comments for tha routing indicating > > "Users are not required to allocate outbound MADs using this call". > > > > If that is the case, either the comment should be changed to indicate > > this or the restriction removed. > > There is not intended to be such a restriction. Glad to hear this :-) I wasn't sure. > Can you point me to where the code assumes this? I'm going back through my notes and I am currently at a loss on the precisely what scenario caused me to ask this question :-( It seems that when ib_create_send_mad is called, the wr_id is the pointer to the ib_mad_send_buf structure and can no longer be used by the client. Maybe a way around this is to add another "wr_id" in that structure which the client could use in this case for any additional context. Never mind: just saw void *context[2] in that structure just for precisely what I was looking for here... -- Hal From mshefty at ichips.intel.com Mon May 2 13:13:54 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Mon, 02 May 2005 13:13:54 -0700 Subject: [openib-general] Re: Another RMPP related question In-Reply-To: <1115064206.4476.101.camel@localhost.localdomain> References: <1114958304.4477.4937.camel@localhost.localdomain> <42767959.9060506@ichips.intel.com> <1115064206.4476.101.camel@localhost.localdomain> Message-ID: <42768A02.3050709@ichips.intel.com> Hal Rosenstock wrote: > It seems that when ib_create_send_mad is called, the wr_id is the > pointer to the ib_mad_send_buf structure and can no longer be used by > the client. Maybe a way around this is to add another "wr_id" in that > structure which the client could use in this case for any additional > context. Never mind: just saw void *context[2] in that structure just > for precisely what I was looking for here... The user can set wr_id to something else, as long as they can get back to the returned ib_mad_send_buf in order to free it. But yes, the context fields are there for the user as well... - Sean From tduffy at sun.com Mon May 2 13:14:37 2005 From: tduffy at sun.com (Tom Duffy) Date: Mon, 02 May 2005 13:14:37 -0700 Subject: [openib-general] [PATCH][DAPL] fix sparse warnings on kdapltest In-Reply-To: <1115064517.11792.4.camel@duffman> References: <1115064517.11792.4.camel@duffman> Message-ID: <1115064877.11792.7.camel@duffman> Sparse fixes on kdapltest. Signed-off-by: Tom Duffy Index: gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_server.c =================================================================== --- gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_server.c (revision 2247) +++ gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_server.c (working copy) @@ -597,7 +597,7 @@ DT_cs_Server (Params_t * params_ptr) DT_Mdep_Unlock (&ps_ptr->num_clients_lock); /* we passed the pt_ptr to the thread and must now 'forget' it */ - pt_ptr = 0; + pt_ptr = NULL; ret = dat_ep_disconnect (ps_ptr->ep_handle, DAT_CLOSE_GRACEFUL_FLAG); if (ret != DAT_SUCCESS) @@ -796,7 +796,7 @@ server_exit: } /* end if ps_ptr */ /* Clean up the server list */ - pre_list = 0; + pre_list = NULL; temp_list = DT_started_server_list; while (temp_list) { Index: gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_test_data.c =================================================================== --- gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_test_data.c (revision 2247) +++ gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_test_data.c (working copy) @@ -37,7 +37,7 @@ Per_Test_Data_t * DT_Alloc_Per_Test_Data (DT_Tdep_Print_Head *phead) { Per_Test_Data_t *pt_ptr; - pt_ptr = 0; + pt_ptr = NULL; pt_ptr = DT_Mdep_Malloc (sizeof (Per_Test_Data_t)); if (!pt_ptr) Index: gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_util.c =================================================================== --- gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_util.c (revision 2247) +++ gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_util.c (working copy) @@ -74,10 +74,10 @@ int DT_ep_create (Params_t *params_ptr, { DAT_RETURN status; DT_Tdep_Print_Head *phead; - *conn_evd= 0; - *send_evd= 0; - *recv_evd= 0; - *cr_evd= 0; + *conn_evd = NULL; + *send_evd = NULL; + *recv_evd = NULL; + *cr_evd = NULL; phead = params_ptr->phead; status = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, DAT_HANDLE_NULL, @@ -125,18 +125,18 @@ int DT_ep_create (Params_t *params_ptr, /* function that initializes the connection struct */ void DT_fft_init_conn_struct (FFT_Connection_t *conn) { - conn->ia_handle = 0; - conn->pz_handle = 0; - conn->psp_handle = 0; - conn->ep_handle = 0; - conn->cr_evd = 0; - conn->send_evd = 0; - conn->conn_evd = 0; - conn->recv_evd = 0; - conn->cr_handle = 0; - conn->remote_netaddr = 0; - conn->bpool = 0; - conn->pt_ptr = 0; + conn->ia_handle = NULL; + conn->pz_handle = NULL; + conn->psp_handle = NULL; + conn->ep_handle = NULL; + conn->cr_evd = NULL; + conn->send_evd = NULL; + conn->conn_evd = NULL; + conn->recv_evd = NULL; + conn->cr_handle = NULL; + conn->remote_netaddr = NULL; + conn->bpool = NULL; + conn->pt_ptr = NULL; conn->connected = false; } @@ -216,7 +216,7 @@ int DT_fft_destroy_conn_struct (Params_t } if (conn->bpool) { - DT_Bpool_Destroy (0, phead, conn->bpool); + DT_Bpool_Destroy (NULL, phead, conn->bpool); } if (conn->psp_handle) { @@ -296,8 +296,9 @@ void DT_fft_init_server (Params_t *param DT_assert_dat (phead, rc == DAT_SUCCESS); /* allocate memory for buffers */ - conn->bpool = DT_BpoolAlloc (0, phead, conn->ia_handle, conn->pz_handle, NULL, NULL, - 8192, 2, DAT_OPTIMAL_ALIGNMENT, false, false); + conn->bpool = DT_BpoolAlloc (NULL, phead, conn->ia_handle, conn->pz_handle, + NULL, NULL, 8192, 2, DAT_OPTIMAL_ALIGNMENT, + false, false); DT_assert (phead, conn->bpool); cleanup: return; Index: gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_mem.c =================================================================== --- gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_mem.c (revision 2247) +++ gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_mem.c (working copy) @@ -50,12 +50,12 @@ int DT_mem_generic (Params_t *params_ptr rc = 0; expect = 0; res = 1; - lmr_handle = 0; + lmr_handle = NULL; lmr_context = 0; reg_addr = 0; - alloc_ptr = 0; - ia_handle = 0; - pz_handle = 0; + alloc_ptr = NULL; + ia_handle = NULL; + pz_handle = NULL; DT_fft_init_client (params_ptr, cmd, &conn); DT_assert (phead, NULL != conn.ia_handle); @@ -63,7 +63,7 @@ int DT_mem_generic (Params_t *params_ptr if (flag == 2) { buffer_size = 0; - alloc_ptr = 0; + alloc_ptr = NULL; } else { @@ -115,7 +115,7 @@ int DT_mem_generic (Params_t *params_ptr rc = dat_lmr_free (lmr_handle); DT_assert_dat (phead, rc == DAT_SUCCESS); } - lmr_handle = 0; + lmr_handle = NULL; rc = dat_lmr_kcreate (conn.ia_handle, DAT_MEM_TYPE_VIRTUAL, Index: gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c =================================================================== --- gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c (revision 2247) +++ gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c (working copy) @@ -82,7 +82,7 @@ int DT_queryinfo_basic (Params_t *params buffer_size = BUFFSIZE * sizeof (unsigned char); phead = params_ptr->phead; reg_addr = 0; - alloc_ptr = 0; + alloc_ptr = NULL; ia_handle = NULL; pz_handle = NULL; Index: gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_server_info.c =================================================================== --- gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_server_info.c (revision 2247) +++ gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_server_info.c (working copy) @@ -27,7 +27,7 @@ #include "dapl_proto.h" -Started_server_t *DT_started_server_list = 0; +Started_server_t *DT_started_server_list = NULL; void DT_Server_Info_Endian (Server_Info_t * server_info) Index: gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_transaction_test.c =================================================================== --- gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_transaction_test.c (revision 2247) +++ gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_transaction_test.c (working copy) @@ -558,7 +558,7 @@ DT_Transaction_Main (void *param) /* what, me query? just try to accept the connection */ ret = dat_cr_accept (cr_handle, - 0, /* NULL for RSP */ + NULL, /* NULL for RSP */ 0, (DAT_PVOID)0 /* no private data */ ); if (ret != DAT_SUCCESS) { Index: gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_memlist.c =================================================================== --- gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_memlist.c (revision 2247) +++ gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_memlist.c (working copy) @@ -31,7 +31,7 @@ void DT_MemListInit (Per_Test_Data_t * pt_ptr) { DT_Mdep_LockInit (&pt_ptr->MemListLock); - pt_ptr->MemListHead = 0; + pt_ptr->MemListHead = NULL; } void * @@ -42,13 +42,13 @@ DT_MemListAlloc (Per_Test_Data_t * pt_pt { void *buffptr; MemListEntry_t *entry_ptr; - buffptr = 0; - entry_ptr = 0; + buffptr = NULL; + entry_ptr = NULL; buffptr = DT_Mdep_Malloc (size); if (buffptr == 0) { - return 0; + return NULL; } if (pt_ptr == 0) /* not use mem_list */ { @@ -58,7 +58,7 @@ DT_MemListAlloc (Per_Test_Data_t * pt_pt if (entry_ptr == 0) { DT_Mdep_Free (buffptr); - return 0; + return NULL; } strcpy (entry_ptr->filename, file); entry_ptr->MemType = t; @@ -82,7 +82,7 @@ DT_MemListFree (Per_Test_Data_t * pt_ptr return; } DT_Mdep_Lock (&pt_ptr->MemListLock); - pre = 0; + pre = NULL; cur = pt_ptr->MemListHead; while (cur) { @@ -91,12 +91,12 @@ DT_MemListFree (Per_Test_Data_t * pt_ptr if (!pre) /* first entry */ { pt_ptr->MemListHead = cur->next; - cur->next = 0; + cur->next = NULL; } else { pre->next = cur->next; - cur->next = 0; + cur->next = NULL; } DT_Mdep_Free (ptr); DT_Mdep_Free (cur); Index: gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_bpool.c =================================================================== --- gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_bpool.c (revision 2247) +++ gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_bpool.c (working copy) @@ -37,7 +37,7 @@ */ -void DTs_CreateIovec ( +static void DTs_CreateIovec ( void **region_addr, u64 *buf_region, DAT_VLEN buffer_size, @@ -97,8 +97,8 @@ DT_BpoolAlloc ( DAT_BOOLEAN enable_rdma_read) { unsigned char *module = "DT_BpoolAlloc"; - unsigned char *alloc_ptr = 0; - Bpool *bpool_ptr = 0; + unsigned char *alloc_ptr = NULL; + Bpool *bpool_ptr = NULL; DAT_COUNT alloc_size, bpool_size; DAT_REGION_DESCRIPTION region; DAT_RETURN ret = DAT_SUCCESS; @@ -308,7 +308,7 @@ err: DT_MemListFree (pt_ptr, alloc_ptr); } - return ( 0 ); + return NULL; } /*****************************************************************************/ Index: gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_endpoint.c =================================================================== --- gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_endpoint.c (revision 2247) +++ gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_endpoint.c (working copy) @@ -177,14 +177,14 @@ int DT_endpoint_case2 (Params_t *params_ DT_Tdep_PT_Printf (phead, "\ Description: try to destroy ep with descriptor still in working queue\n"); res = 1; - bpool = 0; - pz_handle = 0; - ia_handle = 0; - ep_handle = 0; - send_evd = 0; - conn_evd = 0; - recv_evd = 0; - cr_evd = 0; + bpool = NULL; + pz_handle = NULL; + ia_handle = NULL; + ep_handle = NULL; + send_evd = NULL; + conn_evd = NULL; + recv_evd = NULL; + cr_evd = NULL; dev_name = cmd->device_name; rc = DT_ia_open (dev_name, &ia_handle); @@ -200,7 +200,7 @@ int DT_endpoint_case2 (Params_t *params_ &recv_evd, &ep_handle); DT_assert_dat (phead, rc == DAT_SUCCESS); - bpool = DT_BpoolAlloc (0, phead, ia_handle, pz_handle, NULL, NULL, 4096, 1, + bpool = DT_BpoolAlloc (NULL, phead, ia_handle, pz_handle, NULL, NULL, 4096, 1, DAT_OPTIMAL_ALIGNMENT, false, false); DT_assert (phead, bpool != 0); DT_assert (phead, DT_post_recv_buffer (phead, @@ -227,7 +227,7 @@ int DT_endpoint_case2 (Params_t *params_ cleanup: if (bpool) { - rc = DT_Bpool_Destroy (0, phead, bpool); + rc = DT_Bpool_Destroy (NULL, phead, bpool); DT_assert_clean (phead, rc != false); } if (pz_handle) Index: gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_pz.c =================================================================== --- gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_pz.c (revision 2247) +++ gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_pz.c (working copy) @@ -47,8 +47,8 @@ int DT_pz_case0 ( Params_t *params_ptr, Description: Test if we can normally create pz and destroy it.\n"); res=1; - ia_handle=0; - pz_handle =0; + ia_handle = NULL; + pz_handle = NULL; evd_handle = DAT_HANDLE_NULL; dev_name= cmd->device_name; @@ -88,13 +88,13 @@ int DT_pz_case1 (Params_t *params_ptr, F Description: try to destroy pz with vi still associated with it\n"); res=1; - ia_handle=0; - pz_handle =0; - ep_handle=0; - conn_evd = 0; - send_evd = 0; - recv_evd = 0; - cr_evd = 0; + ia_handle = NULL; + pz_handle = NULL; + ep_handle = NULL; + conn_evd = NULL; + send_evd = NULL; + recv_evd = NULL; + cr_evd = NULL; dev_name= cmd->device_name; rc = DT_ia_open (dev_name, &ia_handle); @@ -174,9 +174,9 @@ int DT_pz_case2 (Params_t *params_ptr, F associated with it\n"); res=1; - ia_handle=0; - pz_handle =0; - bpool=0; + ia_handle = NULL; + pz_handle = NULL; + bpool = NULL; dev_name= cmd->device_name; rc = DT_ia_open (dev_name, &ia_handle); @@ -186,7 +186,7 @@ int DT_pz_case2 (Params_t *params_ptr, F DT_assert_dat (phead, rc == DAT_SUCCESS); /* allocate and register bpool */ - bpool = DT_BpoolAlloc (0, phead, ia_handle, pz_handle, NULL, + bpool = DT_BpoolAlloc (NULL, phead, ia_handle, pz_handle, NULL, NULL, BUFFSIZE, 1, DAT_OPTIMAL_ALIGNMENT, false, false); DT_assert (phead, bpool != 0); @@ -200,7 +200,7 @@ int DT_pz_case2 (Params_t *params_ptr, F cleanup: /* deregister and free bpool */ - if (DT_Bpool_Destroy (0, phead, bpool)==false) + if (DT_Bpool_Destroy (NULL, phead, bpool)==false) { DT_Tdep_PT_Printf (phead, "Warning: Destroy bpool fails, reboot for cleanup\n"); return 0; Index: gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_hwconn.c =================================================================== --- gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_hwconn.c (revision 2247) +++ gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_hwconn.c (working copy) @@ -42,7 +42,7 @@ int DT_hwconn_case0 ( Params_t *params_p Description: Test if we can normally Open NIC and then close it\n"); dev_name= cmd->device_name; - nic_handle=0; + nic_handle = NULL; evd_handle = DAT_HANDLE_NULL; rc=dat_ia_open ((const DAT_NAME_PTR)dev_name, 10, &evd_handle, &nic_handle); @@ -128,7 +128,7 @@ int DT_hwconn_case2 (Params_t *params_pt DT_Tdep_PT_Printf (phead, "\ Description: Try to close nic with Nic handle is null (NIC not open)\n"); - nic_handle=0; + nic_handle = NULL; rc=dat_ia_close (nic_handle, DAT_CLOSE_ABRUPT_FLAG); DT_assert_dat (phead, DAT_GET_TYPE (rc) ==DAT_INVALID_HANDLE); Index: gen2/users/jlentini/linux-kernel/test/dapltest/mdep/linux/dapl_mdep_kernel.c =================================================================== --- gen2/users/jlentini/linux-kernel/test/dapltest/mdep/linux/dapl_mdep_kernel.c (revision 2247) +++ gen2/users/jlentini/linux-kernel/test/dapltest/mdep/linux/dapl_mdep_kernel.c (working copy) @@ -241,7 +241,7 @@ DT_Mdep_Thread_Start_Routine (void *thre daemonize(__func__); #endif thread_ptr->function (thread_ptr->param); - return 0; + return NULL; } /* Index: gen2/users/jlentini/linux-kernel/test/dapltest/kdapl/kdapl_module.c =================================================================== --- gen2/users/jlentini/linux-kernel/test/dapltest/kdapl/kdapl_module.c (revision 2247) +++ gen2/users/jlentini/linux-kernel/test/dapltest/kdapl/kdapl_module.c (working copy) @@ -90,7 +90,7 @@ static int kdapltest_ioctl (struct inode break; } if (copy_from_user (local_params, - (void *)param, + (void __user *)param, sizeof (Params_t))) { DT_Mdep_Free (local_params); @@ -120,14 +120,14 @@ static int kdapltest_ioctl (struct inode case KDAPL_IOCTL_GET_PRINTF: { if (copy_from_user (&print_ioctl, - (void *)param, + (void __user *)param, sizeof (DT_get_printf_ioctl))) { rval = -EFAULT; break; } rval = KDT_Get_Print_Line (print_ioctl.cookie, &buffer[0]); - if (copy_to_user (print_ioctl.buffer, + if (copy_to_user ((void __user *)print_ioctl.buffer, &buffer[0], strlen (buffer)+1)) { @@ -149,9 +149,9 @@ static int kdapltest_ioctl (struct inode /* fops */ static struct file_operations kdapltest_fops = { - open: kdapltest_open, - release: kdapltest_release, - ioctl: kdapltest_ioctl, + .open = kdapltest_open, + .release = kdapltest_release, + .ioctl = kdapltest_ioctl, }; static void @@ -169,7 +169,7 @@ DT_Tdep_Test_Thread (Params_t *user_para #endif } -__init +static __init int kdapltest_init (void) { @@ -183,7 +183,7 @@ int kdapltest_init (void) return (0); } -__exit +static __exit void kdapltest_exit (void) { unregister_chrdev (kdapltest_major, "kdapltest"); From tduffy at sun.com Mon May 2 13:28:27 2005 From: tduffy at sun.com (Tom Duffy) Date: Mon, 02 May 2005 13:28:27 -0700 Subject: [openib-general] [PATCH][DAPL] remove #define EXPORT_SYMTAB In-Reply-To: <1115064877.11792.7.camel@duffman> References: <1115064517.11792.4.camel@duffman> <1115064877.11792.7.camel@duffman> Message-ID: <1115065707.11792.10.camel@duffman> Do not need to #define EXPORT_SYMTAB. Signed-off-by: Tom Duffy Index: gen2/users/jlentini/linux-kernel/test/dapltest/kdapl/kdapl_module.c =================================================================== --- gen2/users/jlentini/linux-kernel/test/dapltest/kdapl/kdapl_module.c (revision 2247) +++ gen2/users/jlentini/linux-kernel/test/dapltest/kdapl/kdapl_module.c (working copy) @@ -39,8 +39,6 @@ MODULE_LICENSE("Dual BSD/GPL"); MODULE_DESCRIPTION("kdapltest module"); MODULE_AUTHOR("James Lentini"); -#define EXPORT_SYMTAB - /* static data */ static int kdapltest_major;/* device major number */ static DT_Mdep_LockType kdapltest_lock; /* lock global test structs */ Index: gen2/users/jlentini/linux-kernel/dat-provider/dapl_module.c =================================================================== --- gen2/users/jlentini/linux-kernel/dat-provider/dapl_module.c (revision 2247) +++ gen2/users/jlentini/linux-kernel/dat-provider/dapl_module.c (working copy) @@ -25,7 +25,6 @@ * and/or other materials provided with the distribution. */ -#define EXPORT_SYMTAB #include #include Index: gen2/users/jlentini/linux-kernel/dat/core.c =================================================================== --- gen2/users/jlentini/linux-kernel/dat/core.c (revision 2247) +++ gen2/users/jlentini/linux-kernel/dat/core.c (working copy) @@ -34,7 +34,6 @@ * $Id$ **********************************************************************/ -#define EXPORT_SYMTAB #include #include "core.h" From caitlin.bestler at gmail.com Mon May 2 13:37:00 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Mon, 2 May 2005 13:37:00 -0700 Subject: [openib-general] FMR and how they work In-Reply-To: <506C3D7B14CDD411A52C00025558DED6078F3995@mtlex01.yok.mtl.com> References: <506C3D7B14CDD411A52C00025558DED6078F3995@mtlex01.yok.mtl.com> Message-ID: <469958e005050213374423e789@mail.gmail.com> What advantages is this style of fast memory register supposed to have over the work request style found in IB 1.2 and iWARP? How is this style of fast memory register supposed to be utilized from DAPL or ITAPI when they both assume work requests? From halr at voltaire.com Mon May 2 13:45:30 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 02 May 2005 16:45:30 -0400 Subject: [openib-general] slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects In-Reply-To: <20050502163327.GB9939@mellanox.co.il> References: <20050502084013.GR16918@mellanox.co.il> <1115036059.4477.5703.camel@localhost.localdomain> <42765027.10605@ichips.intel.com> <20050502163327.GB9939@mellanox.co.il> Message-ID: <1115066730.4476.139.camel@localhost.localdomain> On Mon, 2005-05-02 at 12:33, Michael S. Tsirkin wrote: > Do you have a theory on why are MADs leaking? I don't see a MAD leak but perhaps see a couple of other possible memory leaks on shutdown. It looks to me like agent.c has the following issue on shutdown: ib_agent_port_close removes the port from the agent port list before deregistering its MAD agents. This causes any send completions to fail the lookup and hence a number of resources are not returned in this scenario. In that case, I would expect the following to be seend in dmesg: "agent_send_handler: no matching MAD agent". Since this message is not seen, this is not the scenario that is occurring but I will make up a patch for this and send it to the list. Also, does user_mad.c leak memory on the receive side if it is shutdown and the application has not drained the receive queue ? -- Hal From roland at topspin.com Mon May 2 13:59:14 2005 From: roland at topspin.com (Roland Dreier) Date: Mon, 02 May 2005 13:59:14 -0700 Subject: [openib-general] slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects In-Reply-To: <1115066730.4476.139.camel@localhost.localdomain> (Hal Rosenstock's message of "02 May 2005 16:45:30 -0400") References: <20050502084013.GR16918@mellanox.co.il> <1115036059.4477.5703.camel@localhost.localdomain> <42765027.10605@ichips.intel.com> <20050502163327.GB9939@mellanox.co.il> <1115066730.4476.139.camel@localhost.localdomain> Message-ID: <52u0llfiot.fsf@topspin.com> Hal> Also, does user_mad.c leak memory on the receive side if it Hal> is shutdown and the application has not drained the receive Hal> queue ? Yes, looks like it. Something like this (completely untested) should fix it, right? - R. --- infiniband/core/user_mad.c (revision 2211) +++ infiniband/core/user_mad.c (working copy) @@ -499,6 +499,7 @@ static int ib_umad_open(struct inode *in static int ib_umad_close(struct inode *inode, struct file *filp) { struct ib_umad_file *file = filp->private_data; + struct ib_umad_packet *packet, *tmp; int i; for (i = 0; i < IB_UMAD_MAX_AGENTS; ++i) @@ -507,6 +508,9 @@ static int ib_umad_close(struct inode *i ib_unregister_mad_agent(file->agent[i]); } + list_for_each_entry_safe(packet, tmp, &file->recv_list, list) + kfree(packet); + kfree(file); return 0; From halr at voltaire.com Mon May 2 14:00:58 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 02 May 2005 17:00:58 -0400 Subject: [openib-general] slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects In-Reply-To: <52u0llfiot.fsf@topspin.com> References: <20050502084013.GR16918@mellanox.co.il> <1115036059.4477.5703.camel@localhost.localdomain> <42765027.10605@ichips.intel.com> <20050502163327.GB9939@mellanox.co.il> <1115066730.4476.139.camel@localhost.localdomain> <52u0llfiot.fsf@topspin.com> Message-ID: <1115067658.4476.149.camel@localhost.localdomain> On Mon, 2005-05-02 at 16:59, Roland Dreier wrote: > Hal> Also, does user_mad.c leak memory on the receive side if it > Hal> is shutdown and the application has not drained the receive > Hal> queue ? > > Yes, looks like it. Something like this (completely untested) should > fix it, right? Yes, this looks right. Wish I had a test case for this to prove it :-) -- Hal > - R. > > -- infiniband/core/user_mad.c (revision 2211) > +++ infiniband/core/user_mad.c (working copy) > @@ -499,6 +499,7 @@ static int ib_umad_open(struct inode *in > static int ib_umad_close(struct inode *inode, struct file *filp) > { > struct ib_umad_file *file = filp->private_data; > + struct ib_umad_packet *packet, *tmp; > int i; > > for (i = 0; i < IB_UMAD_MAX_AGENTS; ++i) > @@ -507,6 +508,9 @@ static int ib_umad_close(struct inode *i > ib_unregister_mad_agent(file->agent[i]); > } > > + list_for_each_entry_safe(packet, tmp, &file->recv_list, list) > + kfree(packet); > + > kfree(file); > > return 0; From halr at voltaire.com Mon May 2 14:15:39 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 02 May 2005 17:15:39 -0400 Subject: [openib-general] slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects In-Reply-To: <1115066730.4476.139.camel@localhost.localdomain> References: <20050502084013.GR16918@mellanox.co.il> <1115036059.4477.5703.camel@localhost.localdomain> <42765027.10605@ichips.intel.com> <20050502163327.GB9939@mellanox.co.il> <1115066730.4476.139.camel@localhost.localdomain> Message-ID: <1115068539.4476.8.camel@localhost.localdomain> On Mon, 2005-05-02 at 16:45, Hal Rosenstock wrote: > I don't see a MAD leak but perhaps see a couple of other possible memory > leaks on shutdown. > > It looks to me like agent.c has the following issue on shutdown: > ib_agent_port_close removes the port from the agent port list before > deregistering its MAD agents. This causes any send completions to fail > the lookup and hence a number of resources are not returned in this > scenario. In that case, I would expect the following to be seend in > dmesg: "agent_send_handler: no matching MAD agent". Since this message > is not seen, this is not the scenario that is occurring but I will make > up a patch for this and send it to the list. Here's the patch for this: Index: agent.c =================================================================== -- agent.c (revision 2247) +++ agent.c (working copy) @@ -342,16 +342,19 @@ spin_lock_irqsave(&ib_agent_port_list_lock, flags); port_priv = __ib_get_agent_port(device, port_num, NULL); + spin_unlock_irqrestore(&ib_agent_port_list_lock, flags); if (port_priv == NULL) { - spin_unlock_irqrestore(&ib_agent_port_list_lock, flags); printk(KERN_ERR SPFX "Port %d not found\n", port_num); return -ENODEV; } + + ib_unregister_mad_agent(port_priv->perf_mgmt_agent); + ib_unregister_mad_agent(port_priv->smp_agent); + + spin_lock_irqsave(&ib_agent_port_list_lock, flags); list_del(&port_priv->port_list); spin_unlock_irqrestore(&ib_agent_port_list_lock, flags); - ib_unregister_mad_agent(port_priv->perf_mgmt_agent); - ib_unregister_mad_agent(port_priv->smp_agent); kfree(port_priv); return 0; From tduffy at sun.com Mon May 2 14:34:18 2005 From: tduffy at sun.com (Tom Duffy) Date: Mon, 02 May 2005 14:34:18 -0700 Subject: [openib-general] [DAPL] when to cleanup code? Message-ID: <1115069658.11792.21.camel@duffman> James, When do you plan on cleaning up the dapl code? I am referring to the following things: - remove opaque types - remove use of typedef - fix comment formatting, including function definitions If you are not planning on doing it yourself RSN, I can start sending patches to the list if you want. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From ftillier at infiniconsys.com Mon May 2 14:59:26 2005 From: ftillier at infiniconsys.com (Fab Tillier) Date: Mon, 2 May 2005 14:59:26 -0700 Subject: [openib-general] FMR and how they work In-Reply-To: <506C3D7B14CDD411A52C00025558DED6078F3995@mtlex01.yok.mtl.com> Message-ID: <000701c54f62$35b179e0$fede1142@infiniconsys.com> > From: Dror Goldenberg [mailto:gdror at mellanox.co.il] > Sent: Monday, May 02, 2005 7:18 AM > > > -----Original Message----- > > From: Fab Tillier [mailto:ftillier at infiniconsys.com] > > Sent: Monday, May 02, 2005 4:50 AM > > > > Is the HW2SW_MPT command more expensive than the SYNC_TPT > > command? > > SYNC_TPT wipes out the caches. HW2SW_MPT cleans caches for the > specific MPT. You're welcome to compare the two execution time. I believe > that the main impact on CPU is by the fact that you need to submit a > command and wait for it to complete (interrupt->EQE). The PRM doesn't clearly state that HW2SW_MPT invalidates cache entries. It doesn't say either whether just the MPT is invalidated, or also its associated MTTs. > > > Does it just flush the MPT out of the cache? What > > happens to that MPT's MTTs - do they get flushed out too > > (assuming the MPT still references them)? > > For correctness you'd have to flush MPTs and MTTs. Does the HW2SW_MPT command flush the MTTs referenced by the MPT (if any)? That is, if an MPT has mtt_seg_adr_h and mtt_seg_adr_l set during HW2SW_MPT, do the MTTs get flushed? > > > > Why aren't FMR's bindable? > > They are not. Check out the PRM. The PRM says they're not bindable but doesn't explain why. What happens if an FMR is marked as bindable? > > It seems that if the HW2SW_MPT > > flushes the MPT out of the cache, then one could use it for > > normal memory registrations and avoid the WRITE_MTT command, > > no? > > Please explain. Maybe you meant to use HW2SW_MPT for Deregistration. > If this is what you mean, then you're right. But it is against the spirit > of "lazy deregistration". This will only speed the IO operations halfway. > On the creation of mapping it'll be fast - just posted writes to the > MPT/MTT, on the destruction of the mapping, it'll be slow - because you > need HW2SW_MPT and you'd need probably to create a new blank MR > for the next reuse. > The intent was to use bulk deregistrations... No, I'm thinking about improving the "slow" registration path. If I could avoid using the WRITE_MTT command, this would save a few commands (and potential error cases) from the registration path. I'm envisioning a case where the MTTs are written via a posted write to memory, and then the SW2HW_MPT command is issued for an MPT that references the just-written MTTs. Currently (assuming 64MTTs per region), registration is a 2 command process for buffers up to 256K. It becomes 3 commands for registrations to 512K, 4 to 768K, and 5 for 1M, and so forth. Having all of these be just a single SW2HW_MPT command would be nice. Doing this would require the MTTs associated with an MPT to get flushed when the HW2SW_MPT command is invoked so that they can be reused in a subsequent registration without hitting stale cached values. This is also why I'm asking about the bindable properties. Regular memory regions need to be bindable. Can I do posted writes for the MTTs and use the SW2HW_MPT and HW2SW_MPT commands and retain the bindable property of the MPT? > > That is, during registration, do posted writes for the > > MTTs and then a SW2HW_MPT command for an MPT entry that > > references those MTTs. Would this work? > > > > How much slower are memory windows compared to FMRs (assuming > > the underlying MR is already registered)? > > You're welcomed to measure. I think that FMRs will be the fastest way > to create mapping. Faster than MWs. The reason I asked is SDP went down the FMR path, but that can only work properly if the remote peer is trusted. I don't know if this assumption should be made - the same way that we shouldn't trust user-mode processes, we shouldn't trust remote user-mode processes. It is possible for an SDP implementation to be in user-mode. Thus a window exists where a user-mode process on a remote node could corrupt memory via RDMA to stale (but still cached in the HCA) regions. The pages that used to represent those regions could well have been freed and reallocated. This design decision led me to believe that MWs were significantly slower than FMRs (hence the tradeoff with system security). Note that I'm not fully versed in the SDP implementation, so if it addresses this, great! > Also note that FMRs are not bindable. So, there are certain applications > that can benefit from FMRs (e.g. SRP), but can not leverage MWs, at least > I can't think of a model for that... Which gets back to my question of why they're not bindable. What happens if an FMR indicates that it is bindable, and a MW gets bound to it? - Fab From tduffy at sun.com Mon May 2 15:27:02 2005 From: tduffy at sun.com (Tom Duffy) Date: Mon, 02 May 2005 15:27:02 -0700 Subject: [openib-general] Re: [PATCH][DAPL] make dapl build outside of kernel tree In-Reply-To: References: <1114633684.20016.8.camel@duffman> <1114795779.24949.3.camel@duffman> Message-ID: <1115072822.11393.6.camel@duffman> On Mon, 2005-05-02 at 12:07 -0400, James Lentini wrote: > I just committed the IBAT Makefile. For now, I'd like to stick with > our current DAT and DAT Provider Makefiles. I've added your makefiles > as patches to ours. NFS needs to be built as part of the kernel, so we > need the option of building the DAT modules that way as well. I propose you put my Makefiles in the source tree as buildable outside of a kernel, then provide as a patch the Makefiles that build it *in* the kernel -- since you will have to patch it anyways to get the top level MAkefile to dive into your directories. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From mst at mellanox.co.il Tue May 3 00:27:27 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Tue, 3 May 2005 10:27:27 +0300 Subject: [openib-general] Re: slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects In-Reply-To: <52u0llfiot.fsf@topspin.com> References: <20050502084013.GR16918@mellanox.co.il> <1115036059.4477.5703.camel@localhost.localdomain> <42765027.10605@ichips.intel.com> <20050502163327.GB9939@mellanox.co.il> <1115066730.4476.139.camel@localhost.localdomain> <52u0llfiot.fsf@topspin.com> Message-ID: <20050503072727.GD9674@mellanox.co.il> Quoting r. Roland Dreier : > Subject: Re: slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects > > Hal> Also, does user_mad.c leak memory on the receive side if it > Hal> is shutdown and the application has not drained the receive > Hal> queue ? > > Yes, looks like it. Something like this (completely untested) should > fix it, right? > > - R. > > --- infiniband/core/user_mad.c (revision 2211) > +++ infiniband/core/user_mad.c (working copy) > @@ -499,6 +499,7 @@ static int ib_umad_open(struct inode *in > static int ib_umad_close(struct inode *inode, struct file *filp) > { > struct ib_umad_file *file = filp->private_data; > + struct ib_umad_packet *packet, *tmp; > int i; > > for (i = 0; i < IB_UMAD_MAX_AGENTS; ++i) > @@ -507,6 +508,9 @@ static int ib_umad_close(struct inode *i > ib_unregister_mad_agent(file->agent[i]); > } > > + list_for_each_entry_safe(packet, tmp, &file->recv_list, list) > + kfree(packet); > + > kfree(file); > > return 0; > Right, but this wont be related to the kmem_cache leak I see, will it? -- MST - Michael S. Tsirkin From addy at quadrics.com Tue May 3 00:50:12 2005 From: addy at quadrics.com (David Addison) Date: Tue, 03 May 2005 08:50:12 +0100 Subject: [openib-general] Re: RDMA memory registration In-Reply-To: <20050429201835.GA3341@greglaptop.internal.keyresearch.com> References: <52mzrlsflu.fsf@topspin.com> <20050426122850.44d06fa6.akpm@osdl.org> <5264y9s3bs.fsf@topspin.com> <426EA220.6010007@ammasso.com> <20050426133752.37d74805.akpm@osdl.org> <5ebee0d105042907265ff58a73@mail.gmail.com> <469958e005042908566f177b50@mail.gmail.com> <52d5sdjzup.fsf_-_@topspin.com> <42727B60.7010507@ens-lyon.org> <20050429193354.GJ24871@esmail.cup.hp.com> <20050429201835.GA3341@greglaptop.internal.keyresearch.com> Message-ID: <42772D34.90809@quadrics.com> Greg Lindahl wrote: > On Fri, Apr 29, 2005 at 12:33:54PM -0700, Grant Grundler wrote: > >>Being mostly clueless about Quadrics implementation, I'm probably >>missing something that makes Quadrics a MMU but not the IB variants. >>Can someone clue me in please? > > As far as I can tell it's mostly a marketing distinction. Many > Quadrics customers run with memory registration, and Mellanox could > probably alter their firmware to not require registration. Myricom > certainly can, and in fact Patrick Geoffrey claimed they were doing so > in their MX software. The only one I know of that isn't that flexible > is PathScale's InfiniPath. Ours is a pure hardware mechanism, but it > requires memory registration and is clearly not an MMU. > Greg, only a few of our evaluation customers use the patch free (and hence page-pinning) software release. Most do apply our simple IOPROC patch and run without requiring page pinning whilst still achieving the peak bandwidth and low latency of our hardware. Cheers Addy. From mst at mellanox.co.il Tue May 3 00:51:34 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Tue, 3 May 2005 10:51:34 +0300 Subject: [openib-general] Re: Advice about adapting ibv_pingpong to use UC In-Reply-To: <52d5ssccr5.fsf@topspin.com> References: <42627B5A.7010709@wooding.uklinux.net> <52d5ssccr5.fsf@topspin.com> Message-ID: <20050503075134.GG9674@mellanox.co.il> Quoting r. Roland Dreier : > Subject: Re: Advice about adapting ibv_pingpong to use UC > > Steven> Hi, I wonder if someone working on the gen2 uverbs would > Steven> be so kind as to give me some advice on adapting the > Steven> ibv_pingpong program to use a UC QP type rather than RC. I > Steven> was previously able to do this with the Mellanox stack by > Steven> changing the qp_type attribute and then not setting > Steven> variables that are only needed for RC (timeout and retry > Steven> periods etc). > > Unfortunately UC support has not been implemented in the gen2 stack. > It probably wouldn't be that hard but I can't say when I'll get a > chance to look at it. > > - R. Roland, would you like me to implement this? -- MST - Michael S. Tsirkin From mst at mellanox.co.il Tue May 3 00:51:41 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Tue, 3 May 2005 10:51:41 +0300 Subject: [openib-general] RFD: uverbs and hotplug Message-ID: <20050503075141.GH9674@mellanox.co.il> Hello, Roland, all! How should hotplug work with uverbs? Currently ib_uverbs_remove_one simply does class_device_unregister. What will if an application has the device open? Ideally, once the device is removed, applications get an error code and so can close the device freeing up any resources, and reopen another one. However, with libmthca we have a page remapped into the application memory and requests are posted there. One idea is to replace the hardware page mapped into the application with a map to zero page. This prevents the application from accessing hardware that's not there anymore, but does nothing to notify the application that it needs to take action. What do people think? Thanks, -- MST - Michael S. Tsirkin From addy at quadrics.com Tue May 3 01:05:33 2005 From: addy at quadrics.com (David Addison) Date: Tue, 03 May 2005 09:05:33 +0100 Subject: [openib-general] Re: RDMA memory registration In-Reply-To: References: <5D78D28F88822E4D8702BB9EEF1A436773DC44@mercury.infiniconsys.com> <5264y5i9jp.fsf@topspin.com> <20050429210439.GA3650@greglaptop.internal.keyresearch.com> Message-ID: <427730CD.8090408@quadrics.com> Ronald G. Minnich wrote: > > On Fri, 29 Apr 2005, Greg Lindahl wrote: > >>It doesn't imply that there's an MMU, either. I know that Myricom uses a >>little lookup routine in software on their nic, which most people >>wouldn't call an MMU. I don't know what Mellanox does for this, they >>don't talk much about what's hardware and what's software on their nic. >>I think Quadrics actually uses the TLB of their risc cpu on their nic >>for this lookup, but that's just a guess. > > but only quadrics rewrites the mm layer code .. > > Hi Ron, as our recent IOPROC patch on lkml shows, it's not that invasive. There are just 24 hooks added to the Linux VM code paths - which we have been able to maintain outside the mainline tree for many years now. As these hooks only need to synchronise the Elan's MMU state with that of the CPU, the device drivers calls don't change the Linux MM behaviour. We believe the IOPROC patch is generic and powerful and would allow other RDMA NICs to solve the page registration problems in a different manner. For NICs which require page registration, new VM hooks can be used to avoid pages being unloaded whilst DMAs are active. Our latest cut of the IOPROC patch has such a hook. Cheers Addy. From addy at quadrics.com Tue May 3 01:42:12 2005 From: addy at quadrics.com (David Addison) Date: Tue, 03 May 2005 09:42:12 +0100 Subject: [openib-general] Re: RDMA memory registration In-Reply-To: <20050429193354.GJ24871@esmail.cup.hp.com> References: <20050426084234.A10366@topspin.com> <52mzrlsflu.fsf@topspin.com> <20050426122850.44d06fa6.akpm@osdl.org> <5264y9s3bs.fsf@topspin.com> <426EA220.6010007@ammasso.com> <20050426133752.37d74805.akpm@osdl.org> <5ebee0d105042907265ff58a73@mail.gmail.com> <469958e005042908566f177b50@mail.gmail.com> <52d5sdjzup.fsf_-_@topspin.com> <42727B60.7010507@ens-lyon.org> <20050429193354.GJ24871@esmail.cup.hp.com> Message-ID: <42773964.1030101@quadrics.com> Grant Grundler wrote: > On Fri, Apr 29, 2005 at 08:22:24PM +0200, Brice Goglin wrote: >>For instance, instead of adding PROT_DONT/ALWAYSCOPY, you may use >>an ioproc hook in the fork path. This hook (a function in your driver) >>would be called for each registered page. It will decide whether >>the page should be pre-copied or not and update the registration >>table (or whatever stores address translations in the NIC). >>In addition, the driver would probably pre-copy cow pages when >>registering them. > > This doesn't scale well as more cards are added to the box. > I think I understand why it's good for single cards though. > With the IOPROC patch the device driver hooks are registered on a per process or perhaps better still, a per VMA basis. And for processes/VMAs where there are no registrations the overhead is very low. With multiple cards in a box, all using different device drivers, I guess there could end up being multiple registrations per process/VMA. But I'm not sure this will be a common case for RDMA use in real life. Cheers Addy. From halr at voltaire.com Tue May 3 03:54:29 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 03 May 2005 06:54:29 -0400 Subject: [openib-general] Re: slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects In-Reply-To: <20050503072727.GD9674@mellanox.co.il> References: <20050502084013.GR16918@mellanox.co.il> <1115036059.4477.5703.camel@localhost.localdomain> <42765027.10605@ichips.intel.com> <20050502163327.GB9939@mellanox.co.il> <1115066730.4476.139.camel@localhost.localdomain> <52u0llfiot.fsf@topspin.com> <20050503072727.GD9674@mellanox.co.il> Message-ID: <1115117669.4476.543.camel@localhost.localdomain> On Tue, 2005-05-03 at 03:27, Michael S. Tsirkin wrote: > Right, but this wont be related to the kmem_cache leak I see, will it? Right )it doesn't solve it). This is what the original mail said: "I don't see a MAD leak but perhaps see a couple of other possible memory leaks on shutdown." This was one of the two leaks found by code inspection. -- Hal From gdror at mellanox.co.il Tue May 3 05:52:18 2005 From: gdror at mellanox.co.il (Dror Goldenberg) Date: Tue, 3 May 2005 15:52:18 +0300 Subject: [openib-general] FMR and how they work Message-ID: <506C3D7B14CDD411A52C00025558DED607BE347E@mtlex01.yok.mtl.com> > -----Original Message----- > From: Fab Tillier [mailto:ftillier at infiniconsys.com] > Sent: Tuesday, May 03, 2005 12:59 AM > > Does the HW2SW_MPT command flush the MTTs referenced by the > MPT (if any)? That is, if an MPT has mtt_seg_adr_h and > mtt_seg_adr_l set during HW2SW_MPT, do the MTTs get flushed? Yes. > > The PRM says they're not bindable but doesn't explain why. > What happens if an FMR is marked as bindable? The driver is not supposed to set this bit for proper operation of the device. > > No, I'm thinking about improving the "slow" registration > path. If I could avoid using the WRITE_MTT command, this > would save a few commands (and potential error cases) from > the registration path. I'm envisioning a case where the MTTs > are written via a posted write to memory, and then the > SW2HW_MPT command is issued for an MPT that references the > just-written MTTs. You can do that, almost as you described. Think of the MR as a one shot FMRs. Just go ahead and implement according to FMR way such an MR and just use it once. > > Currently (assuming 64MTTs per region), registration is a 2 > command process for buffers up to 256K. It becomes 3 > commands for registrations to 512K, 4 to 768K, and 5 for 1M, > and so forth. Having all of these be just a single SW2HW_MPT > command would be nice. > > Doing this would require the MTTs associated with an MPT to > get flushed when the HW2SW_MPT command is invoked so that > they can be reused in a subsequent registration without > hitting stale cached values. They will get flushed. > > This is also why I'm asking about the bindable properties. > Regular memory regions need to be bindable. Can I do posted > writes for the MTTs and use the SW2HW_MPT and HW2SW_MPT > commands and retain the bindable property of the MPT? This is not true. Not all MRs should be bindable. According to IB verbs spec, at creation time, you decide if an MR is bindable or not. For sure you can implement your acceleration if the user chose not to enable bind on the MR. A lot of apps today just don't use binding, so this can be a nice assumption for performance improvement. -Dror -------------- next part -------------- An HTML attachment was scrubbed... URL: From roland at topspin.com Tue May 3 08:08:18 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 03 May 2005 08:08:18 -0700 Subject: [openib-general] Re: slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects References: <20050502084013.GR16918@mellanox.co.il> <1115036059.4477.5703.camel@localhost.localdomain> <42765027.10605@ichips.intel.com> <20050502163327.GB9939@mellanox.co.il> <1115066730.4476.139.camel@localhost.localdomain> <52u0llfiot.fsf@topspin.com> <20050503072727.GD9674@mellanox.co.il> Message-ID: <5264y0e49p.fsf@topspin.com> Michael> Right, but this wont be related to the kmem_cache leak I Michael> see, will it? I don't think so, since user_mad's recv_handler() function always calls ib_free_recv_mad(). - R. From iod00d at hp.com Tue May 3 08:36:51 2005 From: iod00d at hp.com (Grant Grundler) Date: Tue, 3 May 2005 08:36:51 -0700 Subject: [openib-general] Re: RDMA memory registration In-Reply-To: <42773964.1030101@quadrics.com> References: <20050426122850.44d06fa6.akpm@osdl.org> <5264y9s3bs.fsf@topspin.com> <426EA220.6010007@ammasso.com> <20050426133752.37d74805.akpm@osdl.org> <5ebee0d105042907265ff58a73@mail.gmail.com> <469958e005042908566f177b50@mail.gmail.com> <52d5sdjzup.fsf_-_@topspin.com> <42727B60.7010507@ens-lyon.org> <20050429193354.GJ24871@esmail.cup.hp.com> <42773964.1030101@quadrics.com> Message-ID: <20050503153651.GB11344@esmail.cup.hp.com> On Tue, May 03, 2005 at 09:42:12AM +0100, David Addison wrote: > >This doesn't scale well as more cards are added to the box. > >I think I understand why it's good for single cards though. > > With the IOPROC patch the device driver hooks are registered on a per > process or perhaps better still, a per VMA basis. I was originally thinking the registrations are global (for all memory) and not per process. Per process or per VMA seems reasonable to me. > And for processes/VMAs where there are no registrations the overhead > is very low. Yes - thanks. I'm still reading the LKML thread you started: http://lkml.org/lkml/2005/4/26/198 In particular, the comments from Brice Goglin: http://lkml.org/lkml/2005/4/26/222 openib.org folks can find the IOPROC patch for 2.6.12-rc3 archived here: http://lkml.org/lkml/diff/2005/4/26/198/1 > With multiple cards in a box, all using different device drivers, > I guess there could end up being multiple registrations per process/VMA. > But I'm not sure this will be a common case for RDMA use in real life. I agree. Gateways between fabrics is the only case I can think of. This won't be a problem until someone at a large national lab tries to connect two "legacy" fabrics together. thanks, grant From caitlin.bestler at gmail.com Tue May 3 09:42:09 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Tue, 3 May 2005 09:42:09 -0700 Subject: [openib-general] RFD: uverbs and hotplug In-Reply-To: <20050503075141.GH9674@mellanox.co.il> References: <20050503075141.GH9674@mellanox.co.il> Message-ID: <469958e005050309423adef75e@mail.gmail.com> The fundamental question here is what degree of support is desired? Is it to ensure that the OS can force a memory region to be destroyed on demand, and it is then up to the application to figure out how to respond? Or is it to allow the OS to change the meaning of a Memory Region so that it can re-arrange memory for infrequent maintenance events related to hotplug memory? When a user creates a memory region for a portion of its virtual memory space it is expecting two things: a) If it requested remote accessibility then it can advertise buffers with this Memory Region's R-Key in good faith to its peers. It is the equivalent of sending a certified check. You do not need to know the serial number of the currency that the bank will give when the check is cashed, but you are guaranteed that the cash will be there. That is the point of getting the check certified. Similarly, registering memory with remote access indicates a requirement by the user that the registered memory *will* be available for remote access on demand, and the user can relay this promise (in the form of a buffer advertisement) in good faith. b) That the RDMA device will translate Target Offsets and this R-Key or L-Key in a manner consistent with the user's virtual memory map. The RDMA Device's Memory Map may not be inconsistent with the kernel's memory map. Hotplug relates directly to point b. If the host needs to move the mapping of some virtual pages to new physical pages it must do so in a way that the RDMA Device's mapping cannot be out of sync with the true mapping. Ultimately there are only two ways to do this: a) Eliminate the Memory Region. b) Update the Memory Region. Eliminating the Memory Region requires an interface to immediately destroy a Memory Region, and then provide all of the required notifications to the user application. Such an interface is simple, but will result in most applications terminating. Most applications are not written to be able to survice a sudden loss of virtually all of its connections, and then resume all of the suspended sessions with new connections using new memory regions. Updating the Memory Region requires an interface that can suspend use of the Memory Region while the mapping is updated. After the update has completed the Memory Region can be resumed. While the Memory Region is suspended the RDMA Device will not use it to access memory, and will do what it can to keep the connections alive. InfiniBand has an advantages here, in that RNRs can do a better job of this than iWARP can do over TCP. But in either case, the expectation would be that the OS would suspend a memory region, create the new mapping, do any required copying, and then promptly resume the Memory Region. This strategy is currently being discussed in RNIC-PI. But which is used for Linux is a separate decision. The eliminate option is simpler logic within the kernel, but forces applications to create session layer persistence features. The suspend/resume option allows physical memory migration transparently to the application, but requires more co-operation between the OS and RDMA device drviers. Caitlin Bestler Director Software Architecture Siliquent Technologies caitlinb at siliquent.com On 5/3/05, Michael S. Tsirkin wrote: > Hello, Roland, all! > How should hotplug work with uverbs? > > Currently ib_uverbs_remove_one simply does class_device_unregister. > What will if an application has the device open? > > Ideally, once the device is removed, applications > get an error code and so can close the device freeing > up any resources, and reopen another one. > However, with libmthca we have a page remapped into > the application memory and requests are posted there. > > One idea is to replace the hardware page mapped into the > application with a map to zero page. This prevents > the application from accessing hardware that's not there > anymore, but does nothing to notify the application > that it needs to take action. > > What do people think? > Thanks, > > -- > MST - Michael S. Tsirkin > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From caitlin.bestler at gmail.com Tue May 3 09:56:30 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Tue, 3 May 2005 09:56:30 -0700 Subject: [openib-general] Re: RDMA memory registration In-Reply-To: <427730CD.8090408@quadrics.com> References: <5D78D28F88822E4D8702BB9EEF1A436773DC44@mercury.infiniconsys.com> <5264y5i9jp.fsf@topspin.com> <20050429210439.GA3650@greglaptop.internal.keyresearch.com> <427730CD.8090408@quadrics.com> Message-ID: <469958e005050309562513e786@mail.gmail.com> An ex post facto notification of a PTE change would enable the RDMA Device driver to know when a Memory Region had been invalidated so that it could probably declare an access violation and tear all the connections using it down. But if the intent is to allow it to migrate the memory region to the new mapping it would need a more synchronized notice. It needs to be told of a *pending* change, so that it can indicate when it has completed any data movements based on the old data. It can then use the new data. This has generally been discussed as a two part interface: suspend (to request that the old mapping no longer be used) and resume (to resume usage of the mapping with the new values), and it is generally done at a Memory Region scope rather than on a per PTE basis. RDMA has strict ordering requirements. In particular, completing a receive work request represents a guarnatee to the consumer that the prior writes have been updated in its buffer. With an unsynchronized notice that "PTE entry X has been changed" I don't see how it can fulfill those semantics. It cannot know if portions of an RDMA Write were placed to the old physical location, and therefore it cannot know that the entire RDMA Write payload will be in user memory at the anticipated locations when it generates the work completion. If it cannot make that guarantee it is obligated to terminate the connection. On 5/3/05, David Addison wrote: > Ronald G. Minnich wrote: > > > > On Fri, 29 Apr 2005, Greg Lindahl wrote: > > > >>It doesn't imply that there's an MMU, either. I know that Myricom uses a > >>little lookup routine in software on their nic, which most people > >>wouldn't call an MMU. I don't know what Mellanox does for this, they > >>don't talk much about what's hardware and what's software on their nic. > >>I think Quadrics actually uses the TLB of their risc cpu on their nic > >>for this lookup, but that's just a guess. > > > > but only quadrics rewrites the mm layer code .. > > > > > Hi Ron, > as our recent IOPROC patch on lkml shows, it's not that invasive. There > are just 24 hooks added to the Linux VM code paths - which we have been able to > maintain outside the mainline tree for many years now. > As these hooks only need to synchronise the Elan's MMU state with that of the > CPU, the device drivers calls don't change the Linux MM behaviour. > > We believe the IOPROC patch is generic and powerful and would allow other > RDMA NICs to solve the page registration problems in a different manner. > For NICs which require page registration, new VM hooks can be used to avoid > pages being unloaded whilst DMAs are active. Our latest cut of the IOPROC patch > has such a hook. > > Cheers > Addy. > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From libor at topspin.com Tue May 3 10:08:18 2005 From: libor at topspin.com (Libor Michalek) Date: Tue, 3 May 2005 10:08:18 -0700 Subject: [openib-general] FMR and how they work In-Reply-To: <469958e005050213374423e789@mail.gmail.com>; from caitlin.bestler@gmail.com on Mon, May 02, 2005 at 01:37:00PM -0700 References: <506C3D7B14CDD411A52C00025558DED6078F3995@mtlex01.yok.mtl.com> <469958e005050213374423e789@mail.gmail.com> Message-ID: <20050503100818.A21837@topspin.com> On Mon, May 02, 2005 at 01:37:00PM -0700, Caitlin Bestler wrote: > What advantages is this style of fast memory register supposed to > have over the work request style found in IB 1.2 and iWARP? It's faster since all you're doing is rewriting the device's page table mapping. It would be interesting to see benchmark results if anyone has them. > How is this style of fast memory register supposed to be utilized > from DAPL or ITAPI when they both assume work requests? It has yet to come up, so far all the consumers are written directly to IB kernel verbs. -Libor From libor at topspin.com Tue May 3 10:13:37 2005 From: libor at topspin.com (Libor Michalek) Date: Tue, 3 May 2005 10:13:37 -0700 Subject: [openib-general] FMR and how they work In-Reply-To: <000701c54f62$35b179e0$fede1142@infiniconsys.com>; from ftillier@infiniconsys.com on Mon, May 02, 2005 at 02:59:26PM -0700 References: <506C3D7B14CDD411A52C00025558DED6078F3995@mtlex01.yok.mtl.com> <000701c54f62$35b179e0$fede1142@infiniconsys.com> Message-ID: <20050503101337.B21837@topspin.com> On Mon, May 02, 2005 at 02:59:26PM -0700, Fab Tillier wrote: > > From: Dror Goldenberg [mailto:gdror at mellanox.co.il] > > > > You're welcomed to measure. I think that FMRs will be the fastest way > > to create mapping. Faster than MWs. > > The reason I asked is SDP went down the FMR path, but that can only work > properly if the remote peer is trusted. I don't know if this assumption > should be made - the same way that we shouldn't trust user-mode processes, > we shouldn't trust remote user-mode processes. It is possible for an SDP > implementation to be in user-mode. Thus a window exists where a user-mode > process on a remote node could corrupt memory via RDMA to stale (but still > cached in the HCA) regions. The pages that used to represent those regions > could well have been freed and reallocated. This design decision led me to > believe that MWs were significantly slower than FMRs (hence the tradeoff > with system security). > > Note that I'm not fully versed in the SDP implementation, so if it addresses > this, great! Nope, not addressed. A rogue SDP peer implementation could write into memory that is no longer being used as an active AIO buffer. One way to at least isolate the risk is to add a callback for when the mapping is changed/released and then unlock the buffer. At least the memory will not be used for anything else, even if it could still result in corruption of that buffer... -Libor From rminnich at lanl.gov Tue May 3 10:16:25 2005 From: rminnich at lanl.gov (Ronald G. Minnich) Date: Tue, 3 May 2005 11:16:25 -0600 (MDT) Subject: [openib-general] Re: RDMA memory registration In-Reply-To: <469958e005050309562513e786@mail.gmail.com> References: <5D78D28F88822E4D8702BB9EEF1A436773DC44@mercury.infiniconsys.com> <5264y5i9jp.fsf@topspin.com> <20050429210439.GA3650@greglaptop.internal.keyresearch.com> <427730CD.8090408@quadrics.com> <469958e005050309562513e786@mail.gmail.com> Message-ID: > On 5/3/05, David Addison wrote: > > as our recent IOPROC patch on lkml shows, it's not that invasive. There > > are just 24 hooks added to the Linux VM code paths - which we have been able to > > maintain outside the mainline tree for many years now. > > As these hooks only need to synchronise the Elan's MMU state with that of the > > CPU, the device drivers calls don't change the Linux MM behaviour. > > > > We believe the IOPROC patch is generic and powerful and would allow other > > RDMA NICs to solve the page registration problems in a different manner. > > For NICs which require page registration, new VM hooks can be used to avoid > > pages being unloaded whilst DMAs are active. Our latest cut of the IOPROC patch > > has such a hook. > > david, I just saw this. I'll need to look at that patch, it sounds pretty neat. Thanks ron From David.Brean at Sun.COM Tue May 3 10:58:58 2005 From: David.Brean at Sun.COM (David M. Brean) Date: Tue, 03 May 2005 13:58:58 -0400 Subject: [openib-general] FMR and how they work In-Reply-To: <20050503100818.A21837@topspin.com> References: <506C3D7B14CDD411A52C00025558DED6078F3995@mtlex01.yok.mtl.com> <469958e005050213374423e789@mail.gmail.com> <20050503100818.A21837@topspin.com> Message-ID: <4277BBE2.50805@sun.com> Hello, I thought that IB implementations of iSER and NFS over RDMA were using kDAPL. -David Libor Michalek wrote: >On Mon, May 02, 2005 at 01:37:00PM -0700, Caitlin Bestler wrote: > > >>What advantages is this style of fast memory register supposed to >>have over the work request style found in IB 1.2 and iWARP? >> >> > > It's faster since all you're doing is rewriting the device's >page table mapping. It would be interesting to see benchmark >results if anyone has them. > > > >>How is this style of fast memory register supposed to be utilized >>from DAPL or ITAPI when they both assume work requests? >> >> > > It has yet to come up, so far all the consumers are written >directly to IB kernel verbs. > >-Libor >_______________________________________________ >openib-general mailing list >openib-general at openib.org >http://openib.org/mailman/listinfo/openib-general > >To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > > From roland at topspin.com Tue May 3 11:01:56 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 03 May 2005 11:01:56 -0700 Subject: [openib-general] Re: Advice about adapting ibv_pingpong to use UC In-Reply-To: <20050503075134.GG9674@mellanox.co.il> (Michael S. Tsirkin's message of "Tue, 3 May 2005 10:51:34 +0300") References: <42627B5A.7010709@wooding.uklinux.net> <52d5ssccr5.fsf@topspin.com> <20050503075134.GG9674@mellanox.co.il> Message-ID: <52ll6wchnv.fsf@topspin.com> Michael> Roland, would you like me to implement this? It would be great if you wanted to implement UC support. I think the main thing that needs to be done is to sit down and go over the description of the modify QP verb in the IB spec and update all the transitions in state_table[][] in mthca_qp.c. Then the all the other modifications to the send posting functions should be pretty trivial. - R. From roland at topspin.com Tue May 3 11:07:08 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 03 May 2005 11:07:08 -0700 Subject: [openib-general] Re: RFD: uverbs and hotplug In-Reply-To: <20050503075141.GH9674@mellanox.co.il> (Michael S. Tsirkin's message of "Tue, 3 May 2005 10:51:41 +0300") References: <20050503075141.GH9674@mellanox.co.il> Message-ID: <52hdhkchf7.fsf@topspin.com> Michael> Hello, Roland, all! How should hotplug work with uverbs? It's complicated. Discussion now is useful but I would definitely prefer to hold off on any implementation until we have the basic userspace verbs support merged upstream. One possibility is that it is the administrator's responsibility to close down all applications using the device before hot-removing it. Another possibility is as you described: free all the resources allocated by userspace processes (including remapping any doorbell pages) before allowing the hot-remove to proceed. In this case, one could generate an asynchronous event to tell applications that their HCA is now gone. Do you know what happens today if one has a filesystem mounted and then hot-remove the HBA that is used to get to the underlying block device? - R. From caitlin.bestler at gmail.com Tue May 3 11:12:45 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Tue, 3 May 2005 11:12:45 -0700 Subject: [openib-general] FMR and how they work In-Reply-To: <20050503100818.A21837@topspin.com> References: <506C3D7B14CDD411A52C00025558DED6078F3995@mtlex01.yok.mtl.com> <469958e005050213374423e789@mail.gmail.com> <20050503100818.A21837@topspin.com> Message-ID: <469958e0050503111246390056@mail.gmail.com> On 5/3/05, Libor Michalek wrote: > On Mon, May 02, 2005 at 01:37:00PM -0700, Caitlin Bestler wrote: > > What advantages is this style of fast memory register supposed to > > have over the work request style found in IB 1.2 and iWARP? > > It's faster since all you're doing is rewriting the device's > page table mapping. It would be interesting to see benchmark > results if anyone has them. > Which is exactly what a Fast Memory Register work request does. I also don't see the advantage versus doing a physical rereg. There is an important difference with FMR Work Requests. They follow the normal ordering and completion rules and hence can be pipelined and interleaved with send requests, just like memory window binds. If you only do Fast Memory Registration once, doing so directly from the verbs would be ever so slightly faster because you would be copying from the source to the PBL entries directly. While with a work request you have to copy them (or a pointer to them) into the work request first. But if you are doing multiple FMRs, which would seem to be implied if there is an actual need for them to be "fast", then the ability to pipeline them is far more valuable. More importantly, the FMR work request integrates with kDAPL far better than a some special verb. > > How is this style of fast memory register supposed to be utilized > > from DAPL or ITAPI when they both assume work requests? > > It has yet to come up, so far all the consumers are written > directly to IB kernel verbs. > > -Libor > As pointed out, at least some major consumers are written to kDAPL. Additionally, is it appropriate for the Linux kernel to endorse non-standard functionality in preference to standard functionality? By default, shouldn't the IBTA's endorsement of a FMR work request be taken as an industry consensus? Especially since the other RDMA verbs (RMDAC and RNIC-PI) also take this approach? From caitlin.bestler at gmail.com Tue May 3 11:21:33 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Tue, 3 May 2005 11:21:33 -0700 Subject: [openib-general] Re: RDMA memory registration In-Reply-To: <427730CD.8090408@quadrics.com> References: <5D78D28F88822E4D8702BB9EEF1A436773DC44@mercury.infiniconsys.com> <5264y5i9jp.fsf@topspin.com> <20050429210439.GA3650@greglaptop.internal.keyresearch.com> <427730CD.8090408@quadrics.com> Message-ID: <469958e0050503112151867e50@mail.gmail.com> On 5/3/05, David Addison wrote: > We believe the IOPROC patch is generic and powerful and would allow other > RDMA NICs to solve the page registration problems in a different manner. > For NICs which require page registration, new VM hooks can be used to avoid > pages being unloaded whilst DMAs are active. Our latest cut of the IOPROC patch > has such a hook. > The key phrase here is "avoid pages being unloaded whilst DMAs are active". Correct RDMA behavior requires preventing any loss of the content of those pages in the period from the end of the DMA until the next completion is reaped. If the kernel were to start transferring the pages immediately after the DMA completed, what would prevent the associated receive completion from being generated before the migration was completed? And if a migration is in progress, how is this feedback given to RDMA device and when? Explicitly suspending a Memory Registration allows detection of the problem while the disposition of the packet is still pending. Postponing determination that the target memory is suspended until the actual DMA transfer is attempted is problematic. From mst at mellanox.co.il Tue May 3 11:27:44 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Tue, 3 May 2005 21:27:44 +0300 Subject: [openib-general] Re: RFD: uverbs and hotplug In-Reply-To: <52hdhkchf7.fsf@topspin.com> References: <20050503075141.GH9674@mellanox.co.il> <52hdhkchf7.fsf@topspin.com> Message-ID: <20050503182744.GC20528@mellanox.co.il> Quoting r. Roland Dreier : > Subject: Re: RFD: uverbs and hotplug > > Michael> Hello, Roland, all! How should hotplug work with uverbs? > > It's complicated. Discussion now is useful but I would definitely > prefer to hold off on any implementation until we have the basic > userspace verbs support merged upstream. I agree. > One possibility is that it is the administrator's responsibility to > close down all applications using the device before hot-removing it. Killing applications probably does not cut it - its the applications that are important, not networking. The administrator also may have no idea what these applications are. > Another possibility is as you described: free all the resources > allocated by userspace processes (including remapping any doorbell > pages) before allowing the hot-remove to proceed. In this case, one > could generate an asynchronous event to tell applications that their > HCA is now gone. Okay, but you still must blocks the applications from accessing the hardware - you cant trust them not to. > Do you know what happens today if one has a filesystem mounted and then > hot-remove the HBA that is used to get to the underlying block device? > > - R. No, but I know how NIC hotplug works on my laptop: push a button, wait till a little light flashes, pull it out. -- MST - Michael S. Tsirkin From roland at topspin.com Tue May 3 11:32:18 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 03 May 2005 11:32:18 -0700 Subject: [openib-general] Re: RFD: uverbs and hotplug In-Reply-To: <20050503182744.GC20528@mellanox.co.il> (Michael S. Tsirkin's message of "Tue, 3 May 2005 21:27:44 +0300") References: <20050503075141.GH9674@mellanox.co.il> <52hdhkchf7.fsf@topspin.com> <20050503182744.GC20528@mellanox.co.il> Message-ID: <52d5s8cg99.fsf@topspin.com> Michael> Okay, but you still must blocks the applications from Michael> accessing the hardware - you cant trust them not to. Right, that's why I said "including remapping any doorbell pages." Michael> No, but I know how NIC hotplug works on my laptop: push a Michael> button, wait till a little light flashes, pull it out. Sure, that's easy, because the network layer just throws packets away when a device disappears. We already have that working for IPoIB: you can remove the mthca module and any applications running on top of IPoIB just see their traffic stop. This should also work if you PCI hot-unplug an HCA, although I've never had the hardware to test. - R. From adi at hexapodia.org Tue May 3 11:43:25 2005 From: adi at hexapodia.org (Andy Isaacson) Date: Tue, 3 May 2005 11:43:25 -0700 Subject: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation In-Reply-To: <469958e00504291731eb8287c@mail.gmail.com> Message-ID: <20050503184325.GA19351@hexapodia.org> On Fri, Apr 29, 2005 at 05:31:44PM -0700, Caitlin Bestler wrote: > Attempting to provide *any* support for applications that fork children > after doing RDMA registrations is a ratshole best avoided. The general > rule that application developers should follow is to do RDMA *only* > in the child processes. I think it's unreasonable to *prohibit* fork-after-registration; for one thing, there's lots of code that forks under the covers. Setuid helpers like getpty just assume that they're going to be able to fork. Even stuff like get*by*(3) can potentially fork. And with site-configured stuff like PAM, you end up with things that work on the developer's system but break in deployment. I think it's exceedingly reasonable to say "RDMA doesn't work in children". But the child should get a sane memory image: at least zeros in fully-registered pages, and preferably copies of partially-registered pages. Differentiating between fully-registered and partially-registered pages avoids (I think) the pathological case of having to copy a GB of data just to system("/bin/ls > /tmp/tmpfile"). You can still go pathological if you've partially-registered gigabytes of address space (for example a linked list where each node is allocated with malloc and then registered) but that's a case of "Well, don't do that then". Rather than replacing the fully-registered pages with pages of zeros, you could simply unmap them. A consistent statement would be After fork(2), any regions which were registered are UNDEFINED. Region boundaries are byte-accurate; a registration can cover just part of a page, in which case the non-registered part of the page has normal fork COW semantics. Probably the most sane solution is to simply unmap the fully-registered pages at fork time, and copy any partially-registered pages. But the statement above does not require this. > Keep in mind that it is not only the memory regions that must be dealt > with, but control data invisible to the user (the QP context, etc.). This > data frequently is interlinked between kernel residente and user resident > data (such as a QP context has the PD ID somewhere on-chip or in > kernel, which the Send Queue ring needs to be in user memory). Having > two different user processes that both think they have the user half to > this type of split data structure is just asking for trouble, even if you > manage to get the copy on write bit timing problems all solved. Obviously, calling *any* RDMA-userland-stuff in the child is completely undefined [1]. One place where I can see a potential problem is in atexit()-type handlers registered by the RDMA library. Since those aren't performance-critical they can and should do sanity checks with getpid() and/or checking with the kernel driver. [1] You might want to allow the child to start a completely new RDMA context, but I don't see that as necessary. -andy From iod00d at hp.com Tue May 3 12:06:27 2005 From: iod00d at hp.com (Grant Grundler) Date: Tue, 3 May 2005 12:06:27 -0700 Subject: [openib-general] Re: RFD: uverbs and hotplug In-Reply-To: <52d5s8cg99.fsf@topspin.com> References: <20050503075141.GH9674@mellanox.co.il> <52hdhkchf7.fsf@topspin.com> <20050503182744.GC20528@mellanox.co.il> <52d5s8cg99.fsf@topspin.com> Message-ID: <20050503190627.GA12200@esmail.cup.hp.com> On Tue, May 03, 2005 at 11:32:18AM -0700, Roland Dreier wrote: > Sure, that's easy, because the network layer just throws packets away > when a device disappears. Mass storage is better example - transactions have "state". The term "nexus" is used alot in SCSI specs. > We already have that working for IPoIB: you > can remove the mthca module and any applications running on top of > IPoIB just see their traffic stop. This should also work if you PCI > hot-unplug an HCA, although I've never had the hardware to test. The rx4640 I'm using supports hotplug. What do you want tested? I think you have access to "iowa" via gsyprf10, right? grant From caitlin.bestler at gmail.com Tue May 3 12:04:26 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Tue, 3 May 2005 12:04:26 -0700 Subject: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation In-Reply-To: <20050503184325.GA19351@hexapodia.org> References: <469958e00504291731eb8287c@mail.gmail.com> <20050503184325.GA19351@hexapodia.org> Message-ID: <469958e0050503120463eaca73@mail.gmail.com> On 5/3/05, Andy Isaacson wrote: > > A consistent statement would be > > After fork(2), any regions which were registered are UNDEFINED. > Region boundaries are byte-accurate; a registration can cover just > part of a page, in which case the non-registered part of the page > has normal fork COW semantics. > That is a reasonable approach. > > Obviously, calling *any* RDMA-userland-stuff in the child is completely > undefined [1]. One place where I can see a potential problem is in > atexit()-type handlers registered by the RDMA library. Since those > aren't performance-critical they can and should do sanity checks with > getpid() and/or checking with the kernel driver. > That is also reasonable. None of the RDMA libraries I have worked on bothered to use an atexit()-type hook because the user was theoretically *required* to close the rnic, and driver code was already reuqired to clean up in case of a total process failure. Adding an intermediate safety-net for applications that exited cleanly but forget to close just didn't seem worthwhile. If the application wants the cleanup performed optimally then it can close the rnic, otherwise it can't complain about forcing the RNIC vendor to clean up in the driver code. > [1] You might want to allow the child to start a completely new RDMA > context, but I don't see that as necessary. > That should be allowed. It is actually more normal to use the parent as a dispatcher and to actually manage the connection in a child process. From lindahl at pathscale.com Tue May 3 12:05:52 2005 From: lindahl at pathscale.com (Greg Lindahl) Date: Tue, 3 May 2005 12:05:52 -0700 Subject: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation In-Reply-To: <20050503184325.GA19351@hexapodia.org> References: <469958e00504291731eb8287c@mail.gmail.com> <20050503184325.GA19351@hexapodia.org> Message-ID: <20050503190552.GA1329@greglaptop.internal.keyresearch.com> On Tue, May 03, 2005 at 11:43:25AM -0700, Andy Isaacson wrote: > [1] You might want to allow the child to start a completely new RDMA > context, but I don't see that as necessary. Some people use a hybrid OpenMP+MPI model, so some MPI implementations want to fork and have the child open a new communications context. This is fairly rare, but unfortunately is a checklist item for most MPI implementations, and this will get worse over time. -- greg From roland at topspin.com Tue May 3 12:06:43 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 03 May 2005 12:06:43 -0700 Subject: [openib-general] Re: RFD: uverbs and hotplug In-Reply-To: <20050503190627.GA12200@esmail.cup.hp.com> (Grant Grundler's message of "Tue, 3 May 2005 12:06:27 -0700") References: <20050503075141.GH9674@mellanox.co.il> <52hdhkchf7.fsf@topspin.com> <20050503182744.GC20528@mellanox.co.il> <52d5s8cg99.fsf@topspin.com> <20050503190627.GA12200@esmail.cup.hp.com> Message-ID: <52zmvcb03g.fsf@topspin.com> Grant> The rx4640 I'm using supports hotplug. What do you want Grant> tested? I think you have access to "iowa" via gsyprf10, Grant> right? Yeah, thanks. I'm not sure there's anything worth testing: mthca is a well-behaved PCI driver, so hotplug should work just like removing the module. - R. From jlentini at netapp.com Tue May 3 12:07:47 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 3 May 2005 15:07:47 -0400 (EDT) Subject: [openib-general] Re: [PATCH][DAPL] fix kdapltest Makefile In-Reply-To: <1115064517.11792.4.camel@duffman> References: <1115064517.11792.4.camel@duffman> Message-ID: Tom, Do you also want to remove the first EXTRA_CLAGS value: -Idrivers/dat \ or did you leave that there on purpose? You shouldn't need the -I$(src)/../../../dat-provider \ line. DAT consumers only need to be aware of the registry (the code in the dat directory). james On Mon, 2 May 2005, Tom Duffy wrote: > This fixes the problem of building kdapltest on x86_64. It also makes > it build outside of a kernel without dat installed. > > Also, if I want verbose, I will specify verbose; please don't do it for > me. > > Signed-off-by: Tom Duffy > > Index: gen2/users/jlentini/linux-kernel/test/dapltest/kdapl/Makefile > =================================================================== > --- gen2/users/jlentini/linux-kernel/test/dapltest/kdapl/Makefile (revision 2247) > +++ gen2/users/jlentini/linux-kernel/test/dapltest/kdapl/Makefile (working copy) > @@ -37,7 +37,7 @@ > ifeq ($(origin KERNELRELEASE), undefined) > > KDAPLTEST_ROOT = `pwd` # directory that contains this makefile > -KERNEL_ROOT = /lib/modules/`uname -r`/build # root of kernel sources > +KERNEL_ROOT ?= /lib/modules/`uname -r`/build # root of kernel sources > > # Tools > ECHO = /bin/echo > @@ -45,7 +45,7 @@ RM = /bin/rm > > all : > @$(ECHO) "--- Compiling kdapltest module ---" > - $(MAKE) -C $(KERNEL_ROOT) M=$(KDAPLTEST_ROOT) V=1 modules > + $(MAKE) -C $(KERNEL_ROOT) M=$(KDAPLTEST_ROOT) modules > > clean : > $(RM) -rf \ > @@ -67,6 +67,9 @@ ifeq (${IS_i686},i686) > # Override -msoft-float in arch/i386/Makefile > EXTRA_CFLAGS += -mhard-float > endif > +ifeq (${IS_i686},x86_64) > +EXTRA_CFLAGS += -msse > +endif > > EXTRA_CFLAGS += \ > -Idrivers/dat \ > @@ -74,7 +77,9 @@ EXTRA_CFLAGS += \ > -I$(src)/../mdep/linux \ > -I$(src)/../kdapl \ > -I$(src)/../common \ > - -I$(src)/../test > + -I$(src)/../test \ > + -I$(src)/../../../dat-provider \ > + -I$(src)/../../../dat > > EXTRA_CFLAGS += \ > -D__KDAPL__ \ > From mst at mellanox.co.il Tue May 3 12:29:24 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Tue, 3 May 2005 22:29:24 +0300 Subject: [openib-general] Re: RFD: uverbs and hotplug In-Reply-To: <52hdhkchf7.fsf@topspin.com> References: <20050503075141.GH9674@mellanox.co.il> <52hdhkchf7.fsf@topspin.com> Message-ID: <20050503192924.GD20528@mellanox.co.il> Quoting r. Roland Dreier : > Subject: Re: RFD: uverbs and hotplug > > Michael> Hello, Roland, all! How should hotplug work with uverbs? > > ... > > Another possibility is as you described: free all the resources > allocated by userspace processes (including remapping any doorbell > pages) before allowing the hot-remove to proceed. In this case, one > could generate an asynchronous event to tell applications that their > HCA is now gone. Okay, so this approach makes the most sense to me: Driver remaps the doorbell pages for all processes, signals to applications that their HCA is gone, closes the CQs/QPs/MRs, and lets the hotplug proceed. Seems to work, almost. The application will get a completion with error, and it just needs a way to know that its because of the HCA going away, so it seems to me you have to signal the event before closing the QPs, not after that. Hmm, it also seems that since posting work request on a closed qp does not produce a new work request, a post will need to check a state flag so that post on such a qp fails. Thats probably why you have /* XXX check that state is OK to post send */ at the top of mthca post send? But I dont see how to set the state field without a race? -- MST - Michael S. Tsirkin From mst at mellanox.co.il Tue May 3 12:30:34 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Tue, 3 May 2005 22:30:34 +0300 Subject: [openib-general] Re: RFD: uverbs and hotplug In-Reply-To: <52zmvcb03g.fsf@topspin.com> References: <20050503075141.GH9674@mellanox.co.il> <52hdhkchf7.fsf@topspin.com> <20050503182744.GC20528@mellanox.co.il> <52d5s8cg99.fsf@topspin.com> <20050503190627.GA12200@esmail.cup.hp.com> <52zmvcb03g.fsf@topspin.com> Message-ID: <20050503193034.GE20528@mellanox.co.il> Quoting r. Roland Dreier : > Subject: Re: [openib-general] Re: RFD: uverbs and hotplug > > Grant> The rx4640 I'm using supports hotplug. What do you want > Grant> tested? I think you have access to "iowa" via gsyprf10, > Grant> right? > > Yeah, thanks. I'm not sure there's anything worth testing: mthca is a > well-behaved PCI driver, so hotplug should work just like removing the > module. > > - R. > What about ib_umad? Can you pull the HCA while opensm is running? -- MST - Michael S. Tsirkin From mst at mellanox.co.il Tue May 3 12:32:30 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Tue, 3 May 2005 22:32:30 +0300 Subject: [openib-general] Re: RFD: uverbs and hotplug In-Reply-To: <52d5s8cg99.fsf@topspin.com> References: <20050503075141.GH9674@mellanox.co.il> <52hdhkchf7.fsf@topspin.com> <20050503182744.GC20528@mellanox.co.il> <52d5s8cg99.fsf@topspin.com> Message-ID: <20050503193230.GF20528@mellanox.co.il> Quoting r. Roland Dreier : > Subject: Re: RFD: uverbs and hotplug > Michael> No, but I know how NIC hotplug works on my laptop: push a > Michael> button, wait till a little light flashes, pull it out. > > Sure, that's easy, because the network layer just throws packets away > when a device disappears. We already have that working for IPoIB: you > can remove the mthca module and any applications running on top of > IPoIB just see their traffic stop. This should also work if you PCI > hot-unplug an HCA, although I've never had the hardware to test. > > - R. > It seems that it can be made to work for the umad device this way - MAD packets can just be thrown away since they are unreliable anyway. -- MST - Michael S. Tsirkin From mst at mellanox.co.il Tue May 3 12:33:24 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Tue, 3 May 2005 22:33:24 +0300 Subject: [openib-general] Re: Advice about adapting ibv_pingpong to use UC In-Reply-To: <52ll6wchnv.fsf@topspin.com> References: <42627B5A.7010709@wooding.uklinux.net> <52d5ssccr5.fsf@topspin.com> <20050503075134.GG9674@mellanox.co.il> <52ll6wchnv.fsf@topspin.com> Message-ID: <20050503193324.GG20528@mellanox.co.il> Quoting r. Roland Dreier : > Subject: Re: Advice about adapting ibv_pingpong to use UC > > Michael> Roland, would you like me to implement this? > > It would be great if you wanted to implement UC support. I think the > main thing that needs to be done is to sit down and go over the > description of the modify QP verb in the IB spec and update all the > transitions in state_table[][] in mthca_qp.c. Then the all the other > modifications to the send posting functions should be pretty trivial. > > - R. > Okay, I'll work on it after I close a couple of problems I have with sdp. -- MST - Michael S. Tsirkin From tduffy at sun.com Tue May 3 12:40:18 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 03 May 2005 12:40:18 -0700 Subject: [openib-general] Re: [PATCH][DAPL] fix kdapltest Makefile In-Reply-To: References: <1115064517.11792.4.camel@duffman> Message-ID: <1115149219.9495.5.camel@duffman> On Tue, 2005-05-03 at 15:07 -0400, James Lentini wrote: > Tom, > > Do you also want to remove the first EXTRA_CLAGS value: > > -Idrivers/dat \ > > or did you leave that there on purpose? > > You shouldn't need the > > -I$(src)/../../../dat-provider \ > > line. DAT consumers only need to be aware of the registry (the code in > the dat directory). OK, fair enough, although they should be harmless. Updated patch below. Signed-off-by: Tom Duffy Index: Makefile =================================================================== --- Makefile (revision 2248) +++ Makefile (working copy) @@ -37,7 +37,7 @@ ifeq ($(origin KERNELRELEASE), undefined) KDAPLTEST_ROOT = `pwd` # directory that contains this makefile -KERNEL_ROOT = /lib/modules/`uname -r`/build # root of kernel sources +KERNEL_ROOT ?= /lib/modules/`uname -r`/build # root of kernel sources # Tools ECHO = /bin/echo @@ -45,7 +45,7 @@ RM = /bin/rm all : @$(ECHO) "--- Compiling kdapltest module ---" - $(MAKE) -C $(KERNEL_ROOT) M=$(KDAPLTEST_ROOT) V=1 modules + $(MAKE) -C $(KERNEL_ROOT) M=$(KDAPLTEST_ROOT) modules clean : $(RM) -rf \ @@ -67,14 +67,17 @@ ifeq (${IS_i686},i686) # Override -msoft-float in arch/i386/Makefile EXTRA_CFLAGS += -mhard-float endif +ifeq (${IS_i686},x86_64) +EXTRA_CFLAGS += -msse +endif EXTRA_CFLAGS += \ - -Idrivers/dat \ -I$(src)/../include \ -I$(src)/../mdep/linux \ -I$(src)/../kdapl \ -I$(src)/../common \ - -I$(src)/../test + -I$(src)/../test \ + -I$(src)/../../../dat EXTRA_CFLAGS += \ -D__KDAPL__ \ From halr at voltaire.com Tue May 3 08:34:39 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 03 May 2005 11:34:39 -0400 Subject: [openib-general] Re: slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects In-Reply-To: <5264y0e49p.fsf@topspin.com> References: <20050502084013.GR16918@mellanox.co.il> <1115036059.4477.5703.camel@localhost.localdomain> <42765027.10605@ichips.intel.com> <20050502163327.GB9939@mellanox.co.il> <1115066730.4476.139.camel@localhost.localdomain> <52u0llfiot.fsf@topspin.com> <20050503072727.GD9674@mellanox.co.il> <5264y0e49p.fsf@topspin.com> Message-ID: <1115134226.4476.618.camel@localhost.localdomain> On Tue, 2005-05-03 at 11:08, Roland Dreier wrote: > I don't think so, since user_mad's recv_handler() function always > calls ib_free_recv_mad(). If this only occurs on machines which are running OpenSM, it could be a problem with the local MAD handling where ib_free_recv_mad is not called for some case where it should. I will look into this if this is the case. -- Hal From roland at topspin.com Tue May 3 12:53:09 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 03 May 2005 12:53:09 -0700 Subject: [openib-general] Re: RFD: uverbs and hotplug In-Reply-To: <20050503193034.GE20528@mellanox.co.il> (Michael S. Tsirkin's message of "Tue, 3 May 2005 22:30:34 +0300") References: <20050503075141.GH9674@mellanox.co.il> <52hdhkchf7.fsf@topspin.com> <20050503182744.GC20528@mellanox.co.il> <52d5s8cg99.fsf@topspin.com> <20050503190627.GA12200@esmail.cup.hp.com> <52zmvcb03g.fsf@topspin.com> <20050503193034.GE20528@mellanox.co.il> Message-ID: <52mzrcaxy2.fsf@topspin.com> Michael> What about ib_umad? Can you pull the HCA while opensm is Michael> running? Don't know... I never thought about it. - R. From roland at topspin.com Tue May 3 12:59:03 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 03 May 2005 12:59:03 -0700 Subject: [openib-general] Re: RFD: uverbs and hotplug In-Reply-To: <20050503192924.GD20528@mellanox.co.il> (Michael S. Tsirkin's message of "Tue, 3 May 2005 22:29:24 +0300") References: <20050503075141.GH9674@mellanox.co.il> <52hdhkchf7.fsf@topspin.com> <20050503192924.GD20528@mellanox.co.il> Message-ID: <52hdhkaxo8.fsf@topspin.com> Michael> The application will get a completion with error, and it Michael> just needs a way to know that its because of the HCA Michael> going away, so it seems to me you have to signal the Michael> event before closing the QPs, not after that. It won't get any completion at all: we just destroy the QPs (== move them to RESET) and CQs, so at a certain point the HCA stops processing work requests. The CQ buffers are in user space, so the contents stay valid and the app can continue polling until it has seen all CQEs. It's possible that a request may complete remotely but not generate a completion locally, but this is possible anyway, since the HCA might be removed before an ACK is received. And certainly the remote side of the communication will see its requests fail with a "retries exceeded" status. Michael> Hmm, it also seems that since posting work request on a Michael> closed qp does not produce a new work request, a post Michael> will need to check a state flag so that post on such a qp Michael> fails. Thats probably why you have /* XXX check that Michael> state is OK to post send */ at the top of mthca post Michael> send? But I dont see how to set the state field without Michael> a race? I don't see why this is necessary. The doorbell page no longer points at the HW, so any work requests will be silently ignored. The comment in the post send routines are just there because technically the spec requires the work request posting verbs to fail immediately if the QP state is wrong. - R. From libor at topspin.com Tue May 3 13:02:43 2005 From: libor at topspin.com (Libor Michalek) Date: Tue, 3 May 2005 13:02:43 -0700 Subject: [openib-general] [PATCH][SDP] AIO buffer corruption In-Reply-To: <200505021548.52053.steve@wooding.uklinux.net>; from steve@wooding.uklinux.net on Mon, May 02, 2005 at 03:48:51PM +0100 References: <200505021548.52053.steve@wooding.uklinux.net> Message-ID: <20050503130243.C21837@topspin.com> On Mon, May 02, 2005 at 03:48:51PM +0100, Steven Wooding wrote: > Hello Libor, > > I've tried your patch, but unfortunately it made no difference to > the -32/-104 errors I get. I have observed the following features, > which may help you diagnose my problem: > > Platform: 64 bit > OS: RHEL 4 > Kernel: 2.6.11.6 > OpenIB gen2: 2225 > SM: IO5000 switch Steve, I'm not seeing this issue on my x86_64 systems. Is there anything in the syslog on either system? If you build SDP with debug, there will be messages in the log, any errors? Which HCA are you using, and which firmware? Also, can you send the exact command line? -Libor From libor at topspin.com Tue May 3 13:06:44 2005 From: libor at topspin.com (Libor Michalek) Date: Tue, 3 May 2005 13:06:44 -0700 Subject: [openib-general] Re: SDP: unexpected conn state In-Reply-To: <20050502142148.GA9674@mellanox.co.il>; from mst@mellanox.co.il on Mon, May 02, 2005 at 05:21:48PM +0300 References: <20050502140246.GL16918@mellanox.co.il> <20050502142148.GA9674@mellanox.co.il> Message-ID: <20050503130644.D21837@topspin.com> On Mon, May 02, 2005 at 05:21:48PM +0300, Michael S. Tsirkin wrote: > Quoting r. Michael S. Tsirkin : > > Subject: SDP: unexpected conn state > > > > With ttcp, I sometimes see the following messages from sdp_rcvd_disconnect: > > > > WARN: : Unexpected conn state. conn <1> state <0404:2480> > > > > After this, I cant bind to the same port without unloading mod_sdp. > > Why could that be? > > This happened on the ttcp.aio.x server side when the client had > the ulimit set too low. > It goes away if I set the limit high enough. Looks like on the unexpected connection disconnect the port is not getting cleaned up, which means there should still be a SDP socket in the connection table. Can you send the contents of the table. (/proc/net/sdp/conn_*) Currently the connection state code is overly complicated, I'm working on simplifying it... -Libor From halr at voltaire.com Tue May 3 13:24:27 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 03 May 2005 16:24:27 -0400 Subject: [openib-general] Some Initial RMPP Wire Observations Message-ID: <1115151867.4476.216.camel@localhost.localdomain> Hi Sean, Here are some wire based observations so far on RMPP running between a toy SA client (making some GetTable requests) and OpenSM with RMPP enabled: Short RMPP messages (less than 1 MAD's worth) have first bit set but not also last in RMPPFlags. Also, if the initial RMPP send had bad status, that status is maintained in the RMPP response (ACK) by the receiver side (SA GetTable). Is this getting "reflected" ? Should such a send be ACKed ? Also, AttributeOffset is not being cleared in that ACK. In a multisegment RMPP send, the middle segments have their PayLoad lengths set to whatever the user initially set this to. While it should be ignored by the receiver, it would be better if these were sent as 0. In the last segment, it appears that the pad is not cleared. This again shouldn't matter but I didn't dig out what the spec says here. Is there a restriction on the message size ? Up to what size have you tested this ? I don't seem to get an ACK back when I send with a PayLoadLength of 0x6E0 but do get one with 0x370. Any idea if this is a problem with the sending client or a problem in the RMPP receiver code ? Thanks. -- Hal From mshefty at ichips.intel.com Tue May 3 13:55:55 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Tue, 03 May 2005 13:55:55 -0700 Subject: [openib-general] Re: Some Initial RMPP Wire Observations In-Reply-To: <1115151867.4476.216.camel@localhost.localdomain> References: <1115151867.4476.216.camel@localhost.localdomain> Message-ID: <4277E55B.4080901@ichips.intel.com> Hal Rosenstock wrote: > Short RMPP messages (less than 1 MAD's worth) have first bit set but not > also last in RMPPFlags. Can you re-check this? I'm able to successfully receive short segments, and the receive code looks for the last bit to check for completion. > Also, if the initial RMPP send had bad status, that status is maintained > in the RMPP response (ACK) by the receiver side (SA GetTable). Is this > getting "reflected" ? Should such a send be ACKed ? Also, > AttributeOffset is not being cleared in that ACK. ACKs are formed by copying the headers of the received DATA MAD. Right now there are minimal checks done to verify that the MADs are properly formatted, but adding these isn't overly difficult. Should the AttributeOffset be cleared in the ACK? I wouldn't have thought that RMPP would have changed the common MAD header information. > In a multisegment RMPP send, the middle segments have their PayLoad > lengths set to whatever the user initially set this to. While it should > be ignored by the receiver, it would be better if these were sent as 0. The spec indicates that the payload length is ignored for the middle segments, so I just left the field untouched. There doesn't appear to be a requirement to clear it. (Doing so is trivial if needed.) > In the last segment, it appears that the pad is not cleared. This again > shouldn't matter but I didn't dig out what the spec says here. Not sure which field you're referring to here. The payload length should be set to the number of valid bytes sent in the last packet. > Is there a restriction on the message size ? Up to what size have you > tested this ? I don't seem to get an ACK back when I send with a > PayLoadLength of 0x6E0 but do get one with 0x370. Any idea if this is a > problem with the sending client or a problem in the RMPP receiver code ? There's no restriction unless there's a bug. I've tested MADs up to 100,000 bytes, including sending a response. Not sure what the issue is that you're seeing. - Sean From halr at voltaire.com Tue May 3 14:17:18 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 03 May 2005 17:17:18 -0400 Subject: [openib-general] Re: Some Initial RMPP Wire Observations In-Reply-To: <4277E55B.4080901@ichips.intel.com> References: <1115151867.4476.216.camel@localhost.localdomain> <4277E55B.4080901@ichips.intel.com> Message-ID: <1115155038.4476.258.camel@localhost.localdomain> On Tue, 2005-05-03 at 16:55, Sean Hefty wrote: > Hal Rosenstock wrote: > > Short RMPP messages (less than 1 MAD's worth) have first bit set but not > > also last in RMPPFlags. > > Can you re-check this? I'm able to successfully receive short segments, > and the receive code looks for the last bit to check for completion. This was caused by a bad status in the sent segment so you can chalk this up to user error... > > Also, if the initial RMPP send had bad status, that status is maintained > > in the RMPP response (ACK) by the receiver side (SA GetTable). Is this > > getting "reflected" ? Should such a send be ACKed ? Also, > > AttributeOffset is not being cleared in that ACK. > > ACKs are formed by copying the headers of the received DATA MAD. Right > now there are minimal checks done to verify that the MADs are properly > formatted, but adding these isn't overly difficult. OK that answers the ACK formatting but not whether an ACK is correct in this case. o13-21.1.15 states that if a status code is received with an RMPPType value other than those it applies to, the Receiver shall discard the packet, terminate the protocol,and issue an RMPP ABORT packet with status code "Illegal Status". The receiver flow diagram for MAD OK indicates other checks are appropriate. Bottom line: I'm not 100% sure about this. It's probably OK to do either but I'm not sure what good or harm it does to send an ACK with a bad status. > Should the AttributeOffset be cleared in the ACK? I wouldn't have > thought that RMPP would have changed the common MAD header information. AttributeOffset is in the SA header. It's only present in a DATA packet. It says otherwise ignored on IBA 1.2 p. 884 line 31. Most of these (ignore on receive) say transmit as 0 but this one doesn't so you are right that it shouldn't need to be changed. > > In a multisegment RMPP send, the middle segments have their PayLoad > > lengths set to whatever the user initially set this to. While it should > > be ignored by the receiver, it would be better if these were sent as 0. > > The spec indicates that the payload length is ignored for the middle > segments, so I just left the field untouched. There doesn't appear to > be a requirement to clear it. (Doing so is trivial if needed.) Both true. > > In the last segment, it appears that the pad is not cleared. This again > > shouldn't matter but I didn't dig out what the spec says here. > > Not sure which field you're referring to here. I'm referring to the transferred data beyond the end of the payload length in the last segment. I'm not sure there is a requirement to do anything with this on the transmit side. > The payload length > should be set to the number of valid bytes sent in the last packet. Right so the receiver shouldn't look past this and any non 0 bytes don't matter. (It just makes it look like real data is present there). > > Is there a restriction on the message size ? Up to what size have you > > tested this ? I don't seem to get an ACK back when I send with a > > PayLoadLength of 0x6E0 but do get one with 0x370. Any idea if this is a > > problem with the sending client or a problem in the RMPP receiver code ? > > There's no restriction unless there's a bug. I've tested MADs up to > 100,000 bytes, including sending a response. Not sure what the issue is > that you're seeing. OK. That's good to know. I will look harder to try to find out what is going wrong. Probably another user error... Thanks. -- Hal From mshefty at ichips.intel.com Tue May 3 14:39:11 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Tue, 03 May 2005 14:39:11 -0700 Subject: [openib-general] Re: Some Initial RMPP Wire Observations In-Reply-To: <1115155038.4476.258.camel@localhost.localdomain> References: <1115151867.4476.216.camel@localhost.localdomain> <4277E55B.4080901@ichips.intel.com> <1115155038.4476.258.camel@localhost.localdomain> Message-ID: <4277EF7F.8040905@ichips.intel.com> Hal Rosenstock wrote: > OK that answers the ACK formatting but not whether an ACK is correct in > this case. > > o13-21.1.15 states that if a status code is received with an RMPPType > value other than those it applies to, the Receiver shall discard the > packet, terminate the protocol,and issue an RMPP ABORT packet with > status code "Illegal Status". > > The receiver flow diagram for MAD OK indicates other checks are > appropriate. I think what's necessary is for the DATA MAD to be rejected. This check is missing, but easily added. (I was deferring some of this until STOP/ABORT are implemented.) So... I think that the ACK code may be correct, but checks done earlier in the receive path should prevent this type of DATA MAD from reaching the ACK code. >>>In the last segment, it appears that the pad is not cleared. This again >>>shouldn't matter but I didn't dig out what the spec says here. >> >>Not sure which field you're referring to here. > > I'm referring to the transferred data beyond the end of the payload > length in the last segment. I'm not sure there is a requirement to do > anything with this on the transmit side. Got ya... The contents of the MAD data is the responsibility of the sender. Note that ib_create_send_mad() clears the MAD memory, so if that call is used, the pad will be cleared. - Sean From halr at voltaire.com Tue May 3 14:50:15 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 03 May 2005 17:50:15 -0400 Subject: [openib-general] Re: Some Initial RMPP Wire Observations In-Reply-To: <4277EF7F.8040905@ichips.intel.com> References: <1115151867.4476.216.camel@localhost.localdomain> <4277E55B.4080901@ichips.intel.com> <1115155038.4476.258.camel@localhost.localdomain> <4277EF7F.8040905@ichips.intel.com> Message-ID: <1115157015.4476.266.camel@localhost.localdomain> On Tue, 2005-05-03 at 17:39, Sean Hefty wrote: > I think what's necessary is for the DATA MAD to be rejected. This check > is missing, but easily added. (I was deferring some of this until > STOP/ABORT are implemented.) OK. > So... I think that the ACK code may be > correct, but checks done earlier in the receive path should prevent this > type of DATA MAD from reaching the ACK code. Right. It gets dropped because it has a status: static void process_rmpp_ack(struct ib_mad_agent_private *agent, struct ib_mad_recv_wc *mad_recv_wc) ... rmpp_mad = (struct ib_rmpp_mad *)mad_recv_wc->recv_buf.mad; if (rmpp_mad->rmpp_hdr.rmpp_status) return; > >>>In the last segment, it appears that the pad is not cleared. This again > >>>shouldn't matter but I didn't dig out what the spec says here. > >> > >>Not sure which field you're referring to here. > > > > I'm referring to the transferred data beyond the end of the payload > > length in the last segment. I'm not sure there is a requirement to do > > anything with this on the transmit side. > > Got ya... The contents of the MAD data is the responsibility of the > sender. Note that ib_create_send_mad() clears the MAD memory, so if > that call is used, the pad will be cleared. Thanks for pointing that out. I am using ib_create_mad_send for this. Not sure why they are not cleared. Maybe it's related to the other issue I am having. -- Hal From mshefty at ichips.intel.com Tue May 3 15:01:58 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Tue, 03 May 2005 15:01:58 -0700 Subject: [openib-general] Re: Some Initial RMPP Wire Observations In-Reply-To: <1115157015.4476.266.camel@localhost.localdomain> References: <1115151867.4476.216.camel@localhost.localdomain> <4277E55B.4080901@ichips.intel.com> <1115155038.4476.258.camel@localhost.localdomain> <4277EF7F.8040905@ichips.intel.com> <1115157015.4476.266.camel@localhost.localdomain> Message-ID: <4277F4D6.6040802@ichips.intel.com> Hal Rosenstock wrote: > Thanks for pointing that out. I am using ib_create_mad_send for this. > Not sure why they are not cleared. Maybe it's related to the other issue > I am having. This may or may not be relevant to what you're doing, but I should mention that there's a bug setting the mad_len on the receiving side of an RMPP MAD. I have a fix for this, along with the implementation of ib_coalesce_recv_mad that I'll post shortly. - Sean From halr at voltaire.com Tue May 3 15:25:50 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 03 May 2005 18:25:50 -0400 Subject: [openib-general] Re: Some Initial RMPP Wire Observations In-Reply-To: <4277F4D6.6040802@ichips.intel.com> References: <1115151867.4476.216.camel@localhost.localdomain> <4277E55B.4080901@ichips.intel.com> <1115155038.4476.258.camel@localhost.localdomain> <4277EF7F.8040905@ichips.intel.com> <1115157015.4476.266.camel@localhost.localdomain> <4277F4D6.6040802@ichips.intel.com> Message-ID: <1115159150.4476.268.camel@localhost.localdomain> On Tue, 2005-05-03 at 18:01, Sean Hefty wrote: > Hal Rosenstock wrote: > > Thanks for pointing that out. I am using ib_create_mad_send for this. > > Not sure why they are not cleared. Maybe it's related to the other issue > > I am having. > > This may or may not be relevant to what you're doing, but I should > mention that there's a bug setting the mad_len on the receiving side of > an RMPP MAD. I have a fix for this, along with the implementation of > ib_coalesce_recv_mad that I'll post shortly. Thanks for the heads up. Looking at the code and thinking about this, I can see a possible relation to what I am seeing. I will wait for your change and see how this changes the behavior of what I am seeing. -- Hal From mshefty at ichips.intel.com Tue May 3 16:35:00 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Tue, 03 May 2005 16:35:00 -0700 Subject: [openib-general] [PATCH] [RMPP] ib_coalesce_recv_mad Message-ID: <42780AA4.20502@ichips.intel.com> The following patch implements ib_coalesce_recv_mad and fixes an issue where the mad_len was set incorrectly on the receive side. Note, you will need to use the attached file for the patch to apply correctly, but provided inline for review. Signed-off-by: Sean Hefty Index: include/ib_mad.h =================================================================== --- include/ib_mad.h (revision 2248) +++ include/ib_mad.h (working copy) @@ -447,16 +447,20 @@ /** * ib_coalesce_recv_mad - Coalesces received MAD data into a single buffer. + * @mad_agent: Specifies the MAD agent that received the MAD. * @mad_recv_wc: Work completion information for a received MAD. * @buf: User-provided data buffer to receive the coalesced buffers. The * referenced buffer should be at least the size of the mad_len specified * by @mad_recv_wc. + * @usermode: Indicates if the specified data buffer is a userspace allocated + * data buffer. * * This call copies a chain of received RMPP MADs into a single data buffer, * removing duplicated headers. */ -void ib_coalesce_recv_mad(struct ib_mad_recv_wc *mad_recv_wc, - void *buf); +unsigned long ib_coalesce_recv_mad(struct ib_mad_agent *mad_agent, + struct ib_mad_recv_wc *mad_recv_wc, + void *buf, int usermode); /** * ib_free_recv_mad - Returns data buffers used to receive a MAD. Index: core/mad_rmpp.c =================================================================== --- core/mad_rmpp.c (revision 2248) +++ core/mad_rmpp.c (working copy) @@ -33,6 +33,7 @@ */ #include +#include #include "mad_rmpp.h" #include "mad_priv.h" @@ -396,8 +397,8 @@ hdr_size = data_offset(rmpp_mad->mad_hdr.mgmt_class); data_size = sizeof(struct ib_rmpp_mad) - hdr_size; - pad = be32_to_cpu(rmpp_mad->rmpp_hdr.paylen_newwin); - if (pad data_size) + pad = data_size - be32_to_cpu(rmpp_mad->rmpp_hdr.paylen_newwin); + if (pad data_size || pad < 0) pad = 0; return hdr_size + rmpp_recv->seg_num * data_size - pad; @@ -419,6 +420,56 @@ return rmpp_wc; } +unsigned long ib_coalesce_recv_mad(struct ib_mad_agent *mad_agent, + struct ib_mad_recv_wc *mad_recv_wc, + void *buf, int usermode) +{ + struct ib_mad_recv_buf *seg_buf; + struct ib_rmpp_mad *rmpp_mad; + void *data; + int size, len, offset, ret = 0; + u8 flags; + + len = mad_recv_wc->mad_len; + if (len <= sizeof(struct ib_mad)) { + if (usermode) + ret = copy_to_user(buf, mad_recv_wc->recv_buf.mad, len); + else + memcpy(buf, mad_recv_wc->recv_buf.mad, len); + goto out; + } + + offset = data_offset(mad_recv_wc->recv_buf.mad->mad_hdr.mgmt_class); + + list_for_each_entry(seg_buf, &mad_recv_wc->rmpp_list, list) { + rmpp_mad = (struct ib_rmpp_mad *)seg_buf->mad; + flags = ib_get_rmpp_flags(&rmpp_mad->rmpp_hdr); + + if (flags & IB_MGMT_RMPP_FLAG_FIRST) { + data = rmpp_mad; + size = sizeof(*rmpp_mad); + } else { + data = rmpp_mad + offset; + if (flags & IB_MGMT_RMPP_FLAG_LAST) + size = len; + else + size = sizeof(*rmpp_mad) - offset; + } + + if (usermode) { + ret = copy_to_user(buf, data, size); + if (ret) + goto out; + } else + memcpy(buf, data, size); + + len -= size; + buf += size; + } +out: return ret; +} +EXPORT_SYMBOL(ib_coalesce_recv_mad); + static struct ib_mad_recv_wc * continue_rmpp(struct ib_mad_agent_private *agent, struct ib_mad_recv_wc *mad_recv_wc) Index: core/mad.c =================================================================== --- core/mad.c (revision 2248) +++ core/mad.c (working copy) @@ -1027,13 +1027,6 @@ } EXPORT_SYMBOL(ib_free_recv_mad); -void ib_coalesce_recv_mad(struct ib_mad_recv_wc *mad_recv_wc, - void *buf) -{ - printk(KERN_ERR PFX "ib_coalesce_recv_mad() not implemented yet\n"); -} -EXPORT_SYMBOL(ib_coalesce_recv_mad); - struct ib_mad_agent *ib_redirect_mad_qp(struct ib_qp *qp, u8 rmpp_version, ib_mad_send_handler send_handler, -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: diffs URL: From roland at topspin.com Tue May 3 17:11:05 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 03 May 2005 17:11:05 -0700 Subject: [openib-general] [PATCH] [RMPP] ib_coalesce_recv_mad In-Reply-To: <42780AA4.20502@ichips.intel.com> (Sean Hefty's message of "Tue, 03 May 2005 16:35:00 -0700") References: <42780AA4.20502@ichips.intel.com> Message-ID: <52psw7am06.fsf@topspin.com> I'm not sure I like this "usermode" parameter that selects between memcpy() and copy_to_user(). For one thing, it means that it is impossible to fix the sparse warnings without defeating the sparse address space testing: you have to add a casts to and from __user pointers all over the place. Also, it seems that there's only one caller that's ever going to use the usermode parameter, namely user_mad.c. Unfortunately I don't see a good way to avoid duplicating the (simple) logic in ib_coalesce_recv_mad(). - R. From iod00d at hp.com Tue May 3 17:55:34 2005 From: iod00d at hp.com (Grant Grundler) Date: Tue, 3 May 2005 17:55:34 -0700 Subject: [openib-general] Re: user-mode verbs on Itanium In-Reply-To: <20050429183846.GH24871@esmail.cup.hp.com> References: <1AC79F16F5C5284499BB9591B33D6F00043B37C9@orsmsx408> <20050429051009.GT20957@esmail.cup.hp.com> <52pswdk5au.fsf@topspin.com> <20050429162127.GA24871@esmail.cup.hp.com> <52ll71k09w.fsf@topspin.com> <20050429183846.GH24871@esmail.cup.hp.com> Message-ID: <20050504005534.GA13574@esmail.cup.hp.com> On Fri, Apr 29, 2005 at 11:38:46AM -0700, Grant Grundler wrote: ... > ionize:~# ibv_pingpong -s 64 -n 100000 10.0.0.51 > local address: LID 0x000d, QPN 0x070406, PSN 0x972e9e > remote address: LID 0x000b, QPN 0x030406, PSN 0x3c7543 > 12800000 bytes in 1.65 seconds = 62.10 Mbit/sec > 100000 iters in 1.65 seconds = 16.49 usec/iter ... > ionize:~# ibv_pingpong -s 65536 -n 10000 > local address: LID 0x000d, QPN 0x0b0406, PSN 0xd79a37 > remote address: LID 0x000b, QPN 0x070406, PSN 0x940ac7 > 1310720000 bytes in 2.84 seconds = 3691.26 Mbit/sec > 10000 iters in 2.84 seconds = 284.07 usec/iter It seemed the numbers were too low and of course "ionize" had the IB card in a "1/2 bandwidth" slot (aka "single rope"). 3691 Mbit/sec is about 90% utilization (461:512 MB/s) of the rope. Re-running the extremes, I'm getting better numbers: grundler at gsyprf3:~$ ibv_pingpong -s 65536 -n 100000 10.0.0.30 local address: LID 0x000b, QPN 0x180406, PSN 0x7609fa remote address: LID 0x0010, QPN 0x010406, PSN 0xf10582 13107200000 bytes in 17.99 seconds = 5830.14 Mbit/sec 100000 iters in 17.99 seconds = 179.85 usec/iter 5830 Mbit/s still seems low for this box...only 728 MB/s. I'm expecting at least 800 MB/s and about 900 MB/s per link under "reasonable" conditions (ie > 1K size msgs). Is the issue ibv_pingpong is a single threaded test? Should I run a script to start/send several instances of ibv_pingpong on different ports? grundler at gsyprf3:~$ ibv_pingpong -s 64 -n 1000000 10.0.0.30 local address: LID 0x000b, QPN 0x190406, PSN 0x04c55b remote address: LID 0x0010, QPN 0x020406, PSN 0x350ff5 128000000 bytes in 15.96 seconds = 64.17 Mbit/sec 1000000 iters in 15.96 seconds = 15.96 usec/iter Is the 15.96 usec/iter the round trip time? thanks, grant From sean.hefty at intel.com Tue May 3 18:17:05 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Tue, 3 May 2005 18:17:05 -0700 Subject: [openib-general] [PATCH] [RMPP] ib_coalesce_recv_mad In-Reply-To: <52psw7am06.fsf@topspin.com> Message-ID: >I'm not sure I like this "usermode" parameter that selects between >memcpy() and copy_to_user(). For one thing, it means that it is >impossible to fix the sparse warnings without defeating the sparse >address space testing: you have to add a casts to and from __user >pointers all over the place. > >Also, it seems that there's only one caller that's ever going to use >the usermode parameter, namely user_mad.c. This seems true. I didn't test the usermode piece, so didn't think about the casting issues. >Unfortunately I don't see a good way to avoid duplicating the (simple) >logic in ib_coalesce_recv_mad(). I worked on trying to keep this as simple as possible. (It started out quite a bit more complex...) Looking at it, it's probably not a big deal to duplicate the logic. I'll remove the usermode parameter. - Sean From tduffy at sun.com Tue May 3 19:17:51 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 03 May 2005 19:17:51 -0700 Subject: [openib-general] RFD: uverbs and hotplug In-Reply-To: <469958e005050309423adef75e@mail.gmail.com> References: <20050503075141.GH9674@mellanox.co.il> <469958e005050309423adef75e@mail.gmail.com> Message-ID: <1115173071.29193.8.camel@duffman> On Tue, 2005-05-03 at 09:42 -0700, Caitlin Bestler wrote: > Most applications are not written to be able to > survice a sudden loss of virtually all of its connections, and then resume > all of the suspended sessions with new connections using new memory > regions. Most applications *should* be written this way. Especially since more and more people are using unreliable networks (wifi, 3G, etc.). I can see a day where we have RDMA-capable wireless adapters in which case applications will have to be written with the possibility of a sudden loss taken into account. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From caitlin.bestler at gmail.com Tue May 3 20:53:11 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Tue, 3 May 2005 20:53:11 -0700 Subject: [openib-general] RFD: uverbs and hotplug In-Reply-To: <1115173071.29193.8.camel@duffman> References: <20050503075141.GH9674@mellanox.co.il> <469958e005050309423adef75e@mail.gmail.com> <1115173071.29193.8.camel@duffman> Message-ID: <469958e00505032053552e8b3c@mail.gmail.com> On 5/3/05, Tom Duffy wrote: > On Tue, 2005-05-03 at 09:42 -0700, Caitlin Bestler wrote: > > Most applications are not written to be able to > > survice a sudden loss of virtually all of its connections, and then resume > > all of the suspended sessions with new connections using new memory > > regions. > > Most applications *should* be written this way. Especially since more > and more people are using unreliable networks (wifi, 3G, etc.). I can > see a day where we have RDMA-capable wireless adapters in which case > applications will have to be written with the possibility of a sudden > loss taken into account. > > -tduffy > And applications should be prepared for file systems to fail as well, but that doesn't mean that the file system shouldn't consider features such as journaling that minimize how often the application will be faced that sort of problem. It would certainly be valid to decide that the effort was not worthwhile given the effort required. But such a decision should be made with full awareness that the notice will enable very few applications to actually recover when an entire HCA is taken down. If the goal is to realistically allow applications to confinue functioning after the loss of multiple ports some more powerful hooks are required. From jlentini at netapp.com Tue May 3 22:54:01 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 4 May 2005 01:54:01 -0400 (EDT) Subject: [openib-general] Re: [DAPL] when to cleanup code? In-Reply-To: <1115069658.11792.21.camel@duffman> References: <1115069658.11792.21.camel@duffman> Message-ID: We are ready to start the cleanup now. I've just checked in some changes that I think simplify the interface. james On Mon, 2 May 2005, Tom Duffy wrote: > James, > > When do you plan on cleaning up the dapl code? I am referring to the > following things: > > - remove opaque types > - remove use of typedef > - fix comment formatting, including function definitions > > If you are not planning on doing it yourself RSN, I can start sending > patches to the list if you want. > > -tduffy > From jlentini at netapp.com Tue May 3 22:56:42 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 4 May 2005 01:56:42 -0400 (EDT) Subject: [openib-general] DAT API for OpenIB stack Message-ID: An open source implementation of the DAT API for the OpenIB stack (gen2) is available in the OpenIB subversion repository at https://openib.org/svn/gen2/users/jlentini/ If you are interested in learning more about the DAT API, please see http://www.datcollaborative.org/ or http://sourceforge.net/projects/dapl The DAT kernel code on OpenIB is a native Linux implementation and the provider is only intended to be used on the OpenIB stack. This is in contrast to the SourceForge code that is intended for use on multiple operating systems and verbs APIs. If you are interested in using the code, please see the README files at: https://openib.org/svn/gen2/users/jlentini/linux-kernel/README https://openib.org/svn/gen2/users/jlentini/userspace/dapl/openib/README The code is still in the early stages of development. Currently we are testing the code with the "[k]dapltest" tool (available in the source). Our goal is to be able to run all 6 of dapltest invocations in the linux-kernel/test/dapltest/scripts/kregress.sh script reliably. After obtaining that level of stability, we will begin testing the code with an NFS/RDMA client and server. A number of people have contributed to the DAT code in OpenIB. In particular I'd like to thank Arlin Davis from Intel, Tom Duffy from Sun, Itamar Rabenstein from Mellanox, and Hal Rosenstock from Voltaire. Any help would be appreciated. If anyone would like to volunteer please let me know. james James Lentini email: jlentini at netapp.com Network Appliance phone: 781-768-5359 375 Totten Pond Rd. fax: 781-895-1195 Waltham, MA 02451-2010 main: 781-768-5300 From mst at mellanox.co.il Tue May 3 23:57:59 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 4 May 2005 09:57:59 +0300 Subject: [openib-general] Re: slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects In-Reply-To: <1115134226.4476.618.camel@localhost.localdomain> References: <20050502084013.GR16918@mellanox.co.il> <1115036059.4477.5703.camel@localhost.localdomain> <42765027.10605@ichips.intel.com> <20050502163327.GB9939@mellanox.co.il> <1115066730.4476.139.camel@localhost.localdomain> <52u0llfiot.fsf@topspin.com> <20050503072727.GD9674@mellanox.co.il> <5264y0e49p.fsf@topspin.com> <1115134226.4476.618.camel@localhost.localdomain> Message-ID: <20050504065759.GA9674@mellanox.co.il> Quoting r. Hal Rosenstock : > Subject: Re: slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects > > On Tue, 2005-05-03 at 11:08, Roland Dreier wrote: > > I don't think so, since user_mad's recv_handler() function always > > calls ib_free_recv_mad(). > > If this only occurs on machines which are running OpenSM, it could be a > problem with the local MAD handling where ib_free_recv_mad is not called > for some case where it should. I will look into this if this is the > case. > > -- Hal > This is the case for me: the error only occurs on machines which are running OpenSM. -- MST - Michael S. Tsirkin From mst at mellanox.co.il Wed May 4 00:00:09 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 4 May 2005 10:00:09 +0300 Subject: [openib-general] Re: SDP: unexpected conn state In-Reply-To: <20050503130644.D21837@topspin.com> References: <20050502140246.GL16918@mellanox.co.il> <20050502142148.GA9674@mellanox.co.il> <20050503130644.D21837@topspin.com> Message-ID: <20050504070009.GB9674@mellanox.co.il> Quoting r. Libor Michalek : > Subject: Re: SDP: unexpected conn state > > On Mon, May 02, 2005 at 05:21:48PM +0300, Michael S. Tsirkin wrote: > > Quoting r. Michael S. Tsirkin : > > > Subject: SDP: unexpected conn state > > > > > > With ttcp, I sometimes see the following messages from sdp_rcvd_disconnect: > > > > > > WARN: : Unexpected conn state. conn <1> state <0404:2480> > > > > > > After this, I cant bind to the same port without unloading mod_sdp. > > > Why could that be? > > > > This happened on the ttcp.aio.x server side when the client had > > the ulimit set too low. > > It goes away if I set the limit high enough. > > Looks like on the unexpected connection disconnect the port is > not getting cleaned up, which means there should still be a SDP > socket in the connection table. Can you send the contents of the > table. (/proc/net/sdp/conn_*) Currently the connection state code > is overly complicated, I'm working on simplifying it... > > > -Libor > Couldnt reproduce this anymore. I will if I see this again. -- MST - Michael S. Tsirkin From mst at mellanox.co.il Wed May 4 00:09:10 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 4 May 2005 10:09:10 +0300 Subject: [openib-general] Re: Re: user-mode verbs on Itanium In-Reply-To: <20050504005534.GA13574@esmail.cup.hp.com> References: <1AC79F16F5C5284499BB9591B33D6F00043B37C9@orsmsx408> <20050429051009.GT20957@esmail.cup.hp.com> <52pswdk5au.fsf@topspin.com> <20050429162127.GA24871@esmail.cup.hp.com> <52ll71k09w.fsf@topspin.com> <20050429183846.GH24871@esmail.cup.hp.com> <20050504005534.GA13574@esmail.cup.hp.com> Message-ID: <20050504070910.GC9674@mellanox.co.il> Quoting r. Grant Grundler : > Subject: Re: Re: user-mode verbs on Itanium > > On Fri, Apr 29, 2005 at 11:38:46AM -0700, Grant Grundler wrote: > ... > > ionize:~# ibv_pingpong -s 64 -n 100000 10.0.0.51 > > local address: LID 0x000d, QPN 0x070406, PSN 0x972e9e > > remote address: LID 0x000b, QPN 0x030406, PSN 0x3c7543 > > 12800000 bytes in 1.65 seconds = 62.10 Mbit/sec > > 100000 iters in 1.65 seconds = 16.49 usec/iter > ... > > ionize:~# ibv_pingpong -s 65536 -n 10000 > > local address: LID 0x000d, QPN 0x0b0406, PSN 0xd79a37 > > remote address: LID 0x000b, QPN 0x070406, PSN 0x940ac7 > > 1310720000 bytes in 2.84 seconds = 3691.26 Mbit/sec > > 10000 iters in 2.84 seconds = 284.07 usec/iter > > It seemed the numbers were too low and of course "ionize" had the IB card > in a "1/2 bandwidth" slot (aka "single rope"). > 3691 Mbit/sec is about 90% utilization (461:512 MB/s) of the rope. > > Re-running the extremes, I'm getting better numbers: > grundler at gsyprf3:~$ ibv_pingpong -s 65536 -n 100000 10.0.0.30 > local address: LID 0x000b, QPN 0x180406, PSN 0x7609fa > remote address: LID 0x0010, QPN 0x010406, PSN 0xf10582 > 13107200000 bytes in 17.99 seconds = 5830.14 Mbit/sec > 100000 iters in 17.99 seconds = 179.85 usec/iter > > 5830 Mbit/s still seems low for this box...only 728 MB/s. > I'm expecting at least 800 MB/s and about 900 MB/s > per link under "reasonable" conditions (ie > 1K size msgs). > Is the issue ibv_pingpong is a single threaded test? > Should I run a script to start/send several instances > of ibv_pingpong on different ports? > > > grundler at gsyprf3:~$ ibv_pingpong -s 64 -n 1000000 10.0.0.30 > local address: LID 0x000b, QPN 0x190406, PSN 0x04c55b > remote address: LID 0x0010, QPN 0x020406, PSN 0x350ff5 > 128000000 bytes in 15.96 seconds = 64.17 Mbit/sec > 1000000 iters in 15.96 seconds = 15.96 usec/iter > > Is the 15.96 usec/iter the round trip time? > > thanks, > grant I think ibv_pingpong is not the best latency benchmark - more of a test/usage example. Since it has no pipelining, its also not the best bandwidth benchmark. Grant, if you want a latency benchmark you may want to look at contrib/mellanox/perftest. Its a variation on the pingpong test, showing how you can get latency between 3 and 4usec one-way, by using inline data on sends, and polling on data. I expect to put up a bandwidth test one of these days. -- MST - Michael S. Tsirkin From danb at voltaire.com Wed May 4 00:59:41 2005 From: danb at voltaire.com (Dan Bar Dov) Date: Wed, 4 May 2005 10:59:41 +0300 Subject: [openib-general] FMR and how they work Message-ID: Voltair's IB based ISER indeed uses kDAPL so it will be very simple to atapt it to iWARP. . Voltaire's kDAPL supports FMR using PLATFORM registration type, and it is used by the ISER implementation. Dan > -----Original Message----- > From: openib-general-bounces at openib.org > [mailto:openib-general-bounces at openib.org] On Behalf Of David M. Brean > Sent: Tuesday, May 03, 2005 8:59 PM > To: Libor Michalek > Cc: Caitlin Bestler; openib-general at openib.org > Subject: Re: [openib-general] FMR and how they work > > Hello, > > I thought that IB implementations of iSER and NFS over RDMA > were using kDAPL. > > -David > > Libor Michalek wrote: > > >On Mon, May 02, 2005 at 01:37:00PM -0700, Caitlin Bestler wrote: > > > > > >>What advantages is this style of fast memory register > supposed to have > >>over the work request style found in IB 1.2 and iWARP? > >> > >> > > > > It's faster since all you're doing is rewriting the device's page > >table mapping. It would be interesting to see benchmark results if > >anyone has them. > > > > > > > >>How is this style of fast memory register supposed to be > utilized from > >>DAPL or ITAPI when they both assume work requests? > >> > >> > > > > It has yet to come up, so far all the consumers are > written directly > >to IB kernel verbs. > > > >-Libor > >_______________________________________________ > >openib-general mailing list > >openib-general at openib.org > >http://openib.org/mailman/listinfo/openib-general > > > >To unsubscribe, please visit > >http://openib.org/mailman/listinfo/openib-general > > > > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From mst at mellanox.co.il Wed May 4 04:44:39 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 4 May 2005 14:44:39 +0300 Subject: [openib-general] Re: RFD: uverbs and hotplug In-Reply-To: <1115173071.29193.8.camel@duffman> References: <1115173071.29193.8.camel@duffman> Message-ID: <20050504114439.GK9674@mellanox.co.il> Quoting r. Tom Duffy : > Subject: Re: RFD: uverbs and hotplug > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > > > Subject: > Date: Wed, 4 May 2005 10:52:45 +0300 > > > > On Tue, 2005-05-03 at 09:42 -0700, Caitlin Bestler wrote: > > Most applications are not written to be able to > > survice a sudden loss of virtually all of its connections, and then resume > > all of the suspended sessions with new connections using new memory > > regions. > > Most applications *should* be written this way. Especially since more > and more people are using unreliable networks (wifi, 3G, etc.). I can > see a day where we have RDMA-capable wireless adapters in which case > applications will have to be written with the possibility of a sudden > loss taken into account. > > -tduffy > > > The problem is not that hard. The driver must provide sufficient information to discover that the device went down and which requests were serviced before the device went down. Libraries such as MPI will be able to use that to resend these requests on a different HCA or after an HCA is plugged back in. Applications will simply use this library and be happy. -- MST - Michael S. Tsirkin From itamar at mellanox.co.il Wed May 4 05:33:13 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Wed, 4 May 2005 15:33:13 +0300 Subject: [openib-general] [kDAPL][PATCH] delete cno user level code from kdapl Message-ID: <91DB792C7985D411BEC300B40080D29CC35A2D@mtvex01.mtv.mtl.com> clean up kdapl code : delete cno user level code Signed-off-by: Itamar Rabenstein Index: linux-kernel/dat-provider/dapl_evd_free.c =================================================================== --- linux-kernel/dat-provider/dapl_evd_free.c (revision 2251) +++ linux-kernel/dat-provider/dapl_evd_free.c (working copy) @@ -74,16 +74,8 @@ } if (cno_ptr != NULL) - { - if (atomic_read (&cno_ptr->cno_ref_count) > 0 || cno_ptr->cno_waiters > 0) - { - dat_status = DAT_ERROR (DAT_INVALID_STATE, DAT_INVALID_STATE_EVD_IN_USE); - goto bail; - } - dapl_ia_unlink_cno (cno_ptr->header.owner_ia, cno_ptr); - dapl_cno_dealloc (cno_ptr); - } - + dapl_cno_dealloc (cno_ptr); + bail: dapl_dbg_log(DAPL_DBG_TYPE_RTN, "dapl_evd_free () returns 0x%x\n", dat_status); Index: linux-kernel/dat-provider/dapl_ia_util.c =================================================================== --- linux-kernel/dat-provider/dapl_ia_util.c (revision 2251) +++ linux-kernel/dat-provider/dapl_ia_util.c (working copy) @@ -98,7 +98,6 @@ dapl_llist_init_head(&ia_ptr->rmr_list_head); dapl_llist_init_head(&ia_ptr->pz_list_head); dapl_llist_init_head(&ia_ptr->evd_list_head); - dapl_llist_init_head(&ia_ptr->cno_list_head); dapl_llist_init_head(&ia_ptr->rsp_list_head); dapl_llist_init_head(&ia_ptr->psp_list_head); @@ -491,7 +490,6 @@ dapl_os_assert(dapl_llist_is_empty(&ia_ptr->rmr_list_head)); dapl_os_assert(dapl_llist_is_empty(&ia_ptr->ep_list_head)); dapl_os_assert(dapl_llist_is_empty(&ia_ptr->evd_list_head)); - dapl_os_assert(dapl_llist_is_empty(&ia_ptr->cno_list_head)); dapl_os_assert(dapl_llist_is_empty(&ia_ptr->psp_list_head)); dapl_os_assert(dapl_llist_is_empty(&ia_ptr->rsp_list_head)); @@ -638,29 +636,6 @@ } /* - * Add an cno to the IA structure - */ -void dapl_ia_link_cno(DAPL_IA * ia_ptr, DAPL_CNO * cno_ptr) -{ - spin_lock (&ia_ptr->header.lock); - dapl_llist_add_head (&ia_ptr->cno_list_head, - &cno_ptr->header.ia_list_entry, - cno_ptr); - spin_unlock (&ia_ptr->header.lock); -} - -/* - * Remove an cno from the ia info structure - */ -void dapl_ia_unlink_cno(DAPL_IA * ia_ptr, DAPL_CNO * cno_ptr) -{ - spin_lock (&ia_ptr->header.lock); - dapl_llist_remove_entry (&ia_ptr->cno_list_head, - &cno_ptr->header.ia_list_entry); - spin_unlock (&ia_ptr->header.lock); -} - -/* * Add an psp to the IA structure */ void dapl_ia_link_psp(DAPL_IA * ia_ptr, DAPL_SP * sp_ptr) Index: linux-kernel/dat-provider/dapl_ia_util.h =================================================================== --- linux-kernel/dat-provider/dapl_ia_util.h (revision 2251) +++ linux-kernel/dat-provider/dapl_ia_util.h (working copy) @@ -71,10 +71,6 @@ void dapl_ia_unlink_evd(DAPL_IA * ia_ptr, DAPL_EVD * evd_info); -void dapl_ia_link_cno(DAPL_IA * ia_ptr, DAPL_CNO * cno_info); - -void dapl_ia_unlink_cno(DAPL_IA * ia_ptr, DAPL_CNO * cno_info); - void dapl_ia_link_psp(DAPL_IA * ia_ptr, DAPL_SP * sp_info); void dapl_ia_unlink_sp(DAPL_IA * ia_ptr, DAPL_SP * sp_info); Index: linux-kernel/dat-provider/dapl_evd_kcreate.c =================================================================== --- linux-kernel/dat-provider/dapl_evd_kcreate.c (revision 2251) +++ linux-kernel/dat-provider/dapl_evd_kcreate.c (working copy) @@ -104,10 +104,8 @@ goto bail; } - cno_ptr->cno_state = DAPL_CNO_STATE_TRIGGERED; cno_ptr->cno_upcall_policy = upcall_policy; - dapl_ia_link_cno (ia_ptr, cno_ptr); dat_status = dapl_evd_internal_create(ia_ptr, cno_ptr, @@ -125,7 +123,6 @@ bail: if (dat_status != DAT_SUCCESS && cno_ptr != NULL) { - dapl_ia_unlink_cno (ia_ptr, cno_ptr); dapl_cno_dealloc (cno_ptr); } Index: linux-kernel/dat-provider/dapl_cno_util.c =================================================================== --- linux-kernel/dat-provider/dapl_cno_util.c (revision 2251) +++ linux-kernel/dat-provider/dapl_cno_util.c (working copy) @@ -70,12 +70,7 @@ /* * Initialize the body */ - cno_ptr->cno_waiters = 0; - atomic_set (&cno_ptr->cno_ref_count, 0); - cno_ptr->cno_state = DAPL_CNO_STATE_UNTRIGGERED; - cno_ptr->cno_evd_triggered = NULL; cno_ptr->cno_upcall = *upcall; - dapl_os_wait_object_init (&cno_ptr->cno_wait_object); return cno_ptr; } @@ -88,14 +83,11 @@ DAPL_CNO *cno_ptr) { dapl_os_assert (cno_ptr->header.magic == DAPL_MAGIC_CNO); - dapl_os_assert (atomic_read (&cno_ptr->cno_ref_count) == 0); - /* * deinitialize the header */ cno_ptr->header.magic = DAPL_MAGIC_INVALID; /* reset magic to prevent reuse */ - dapl_os_wait_object_destroy (&cno_ptr->cno_wait_object); kfree (cno_ptr); } @@ -134,60 +126,16 @@ spin_lock (&cno_ptr->header.lock); - /* Maybe I should just return, but this really shouldn't happen. */ - dapl_os_assert (cno_ptr->cno_state != DAPL_CNO_STATE_DEAD); - - if (cno_ptr->cno_state == DAPL_CNO_STATE_UNTRIGGERED) - { - /* Separate assignments for windows compiler. */ - cno_ptr->cno_upcall = DAT_UPCALL_NULL; - cno_ptr->cno_evd_triggered = evd_ptr; - - /* - * Must set to triggerred and let waiter untrigger to handle - * timeout of waiter. - */ - cno_ptr->cno_state = DAPL_CNO_STATE_TRIGGERED; - if (cno_ptr->cno_waiters > 0) - { - dapl_os_wait_object_wakeup (&cno_ptr->cno_wait_object); - } - spin_unlock (&cno_ptr->header.lock); - - dat_status = dapl_evd_dequeue ( (DAT_EVD_HANDLE)evd_ptr, - &event ); - while (dat_status == DAT_SUCCESS ) - { - if (cno_ptr->cno_upcall.upcall_func != (DAT_UPCALL_FUNC) NULL) - { + dat_status = dapl_evd_dequeue ((DAT_EVD_HANDLE)evd_ptr, &event); + while (dat_status == DAT_SUCCESS ) { + if (cno_ptr->cno_upcall.upcall_func != (DAT_UPCALL_FUNC) NULL) { cno_ptr->cno_upcall.upcall_func ( cno_ptr->cno_upcall.instance_data, &event, DAT_FALSE); } - dat_status = dapl_evd_dequeue ( (DAT_EVD_HANDLE)evd_ptr, - &event ); + dat_status = dapl_evd_dequeue ( (DAT_EVD_HANDLE)evd_ptr, &event); } - } - else - { - spin_unlock (&cno_ptr->header.lock); - dat_status = dapl_evd_dequeue ( (DAT_EVD_HANDLE)evd_ptr, - &event ); - while (dat_status == DAT_SUCCESS ) - { - if (cno_ptr->cno_upcall.upcall_func != (DAT_UPCALL_FUNC) NULL) - { - cno_ptr->cno_upcall.upcall_func ( - cno_ptr->cno_upcall.instance_data, - &event, - DAT_FALSE); - } - dat_status = dapl_evd_dequeue ( (DAT_EVD_HANDLE)evd_ptr, - &event ); - } - } - return; } Index: linux-kernel/dat-provider/dapl_evd_util.c =================================================================== --- linux-kernel/dat-provider/dapl_evd_util.c (revision 2251) +++ linux-kernel/dat-provider/dapl_evd_util.c (working copy) @@ -174,11 +174,6 @@ dapl_os_wait_object_init(&evd_ptr->wait_object); evd_ptr->cno_active_count = 0; - if ( cno_ptr != NULL ) - { - /* Take a reference count on the CNO */ - atomic_inc (&cno_ptr->cno_ref_count); - } evd_ptr->cno_ptr = cno_ptr; bail: @@ -323,7 +318,6 @@ /* Release reference on the CNO if it exists */ if ( evd_ptr->cno_ptr != NULL ) { - atomic_dec ( &evd_ptr->cno_ptr->cno_ref_count ); evd_ptr->cno_ptr = NULL; } Index: linux-kernel/dat-provider/dapl.h =================================================================== --- linux-kernel/dat-provider/dapl.h (revision 2251) +++ linux-kernel/dat-provider/dapl.h (working copy) @@ -78,12 +78,6 @@ DAPL_EVD_STATE_UNSIGNALLED } DAPL_EVD_COMPLETION; -typedef enum dapl_cno_state { - DAPL_CNO_STATE_UNTRIGGERED, - DAPL_CNO_STATE_TRIGGERED, - DAPL_CNO_STATE_DEAD = 0xDeadFeed, -} DAPL_CNO_STATE; - typedef enum dapl_qp_state { DAPL_QP_STATE_UNCONNECTED, DAPL_QP_STATE_RESERVED, @@ -256,7 +250,6 @@ DAPL_LLIST_HEAD rmr_list_head; /* RMR queue */ DAPL_LLIST_HEAD pz_list_head; /* PZ queue */ DAPL_LLIST_HEAD evd_list_head; /* EVD queue */ - DAPL_LLIST_HEAD cno_list_head; /* CNO queue */ DAPL_LLIST_HEAD psp_list_head; /* PSP queue */ DAPL_LLIST_HEAD rsp_list_head; /* RSP queue */ DAPL_LLIST_HEAD srq_list_head; /* SRQ queue */ @@ -266,16 +259,8 @@ struct dapl_cno { DAPL_HEADER header; - - /* A CNO cannot be freed while it is referenced elsewhere. */ - atomic_t cno_ref_count; - DAPL_CNO_STATE cno_state; - - DAT_COUNT cno_waiters; - DAPL_EVD *cno_evd_triggered; DAT_UPCALL_OBJECT cno_upcall; DAT_UPCALL_POLICY cno_upcall_policy; - DAPL_OS_WAIT_OBJECT cno_wait_object; }; /* DAPL_EVD maps to DAT_EVD_HANDLE */ From iod00d at hp.com Wed May 4 07:36:20 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 4 May 2005 07:36:20 -0700 Subject: [openib-general] Re: Re: user-mode verbs on Itanium In-Reply-To: <20050504070910.GC9674@mellanox.co.il> References: <1AC79F16F5C5284499BB9591B33D6F00043B37C9@orsmsx408> <20050429051009.GT20957@esmail.cup.hp.com> <52pswdk5au.fsf@topspin.com> <20050429162127.GA24871@esmail.cup.hp.com> <52ll71k09w.fsf@topspin.com> <20050429183846.GH24871@esmail.cup.hp.com> <20050504005534.GA13574@esmail.cup.hp.com> <20050504070910.GC9674@mellanox.co.il> Message-ID: <20050504143620.GA15486@esmail.cup.hp.com> On Wed, May 04, 2005 at 10:09:10AM +0300, Michael S. Tsirkin wrote: > I think ibv_pingpong is not the best latency benchmark - more of > a test/usage example. Well, this is the example newbies will follow when they want to write code using the same interfaces. Given the lack of documentation, we really depend on good examples for people (like me) who have never written code for any infiniband implementation. > Since it has no pipelining, its also not the best bandwidth benchmark. Yeah, I suspected that. > Grant, if you want a latency benchmark you may want to look at > contrib/mellanox/perftest. > Its a variation on the pingpong test, showing how you can get latency > between 3 and 4usec one-way, by using inline data on sends, and polling on data. Ok - will do - thanks! > I expect to put up a bandwidth test one of these days. That would be good. Harping on the same theme as above, that might be an opportunity to hack something into an existing benchmark like netpipe or netperf. Ie provide an example of how to adapt an existing application to optimally use openib interconnect. thanks, grant From iod00d at hp.com Wed May 4 07:47:23 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 4 May 2005 07:47:23 -0700 Subject: [openib-general] RFD: uverbs and hotplug In-Reply-To: <469958e00505032053552e8b3c@mail.gmail.com> References: <20050503075141.GH9674@mellanox.co.il> <469958e005050309423adef75e@mail.gmail.com> <1115173071.29193.8.camel@duffman> <469958e00505032053552e8b3c@mail.gmail.com> Message-ID: <20050504144723.GB15486@esmail.cup.hp.com> On Tue, May 03, 2005 at 08:53:11PM -0700, Caitlin Bestler wrote: > And applications should be prepared for file systems to fail as well, > but that doesn't mean that the file system shouldn't consider features > such as journaling that minimize how often the application will be > faced that sort of problem. Sorry - journaling seems like a bad example. Journalling is how the OS recovers/restores sane state to the filesystem and is not a mechanism that helps the application recover. Applications normally depend on another layer to "tolerate" storage link failures - aka multipath or mirroring. But it's not any form of recovery either. There's alot more history behind the assumption that storage has a reliable transport. Networking never had that assumption though TCP faked it for the most part. > It would certainly be valid to decide that the effort was not worthwhile > given the effort required. But such a decision should be made with > full awareness that the notice will enable very few applications to > actually recover when an entire HCA is taken down. Agreed. thanks, grant > If the goal > is to realistically allow applications to confinue functioning after > the loss of multiple ports some more powerful hooks are required. > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From tduffy at sun.com Wed May 4 09:14:02 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 04 May 2005 09:14:02 -0700 Subject: [openib-general] [kDAPL][PATCH] delete cno user level code from kdapl In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35A2D@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35A2D@mtvex01.mtv.mtl.com> Message-ID: <1115223242.14124.1.camel@duffman> On Wed, 2005-05-04 at 15:33 +0300, Itamar Rabenstein wrote: > clean up kdapl code : delete cno user level code > > Signed-off-by: Itamar Rabenstein > > Index: linux-kernel/dat-provider/dapl_evd_free.c > =================================================================== > --- linux-kernel/dat-provider/dapl_evd_free.c (revision 2251) > +++ linux-kernel/dat-provider/dapl_evd_free.c (working copy) > @@ -74,16 +74,8 @@ > } > > if (cno_ptr != NULL) > - { > - if (atomic_read (&cno_ptr->cno_ref_count) > 0 || > cno_ptr->cno_waiters > 0) > - { > - dat_status = DAT_ERROR (DAT_INVALID_STATE, > DAT_INVALID_STATE_EVD_IN_USE); > - goto bail; Your patch was wrapped by your MUA. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From jlentini at netapp.com Wed May 4 09:27:12 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 4 May 2005 12:27:12 -0400 (EDT) Subject: [openib-general] Re: [PATCH][DAPL] fix sparse warnings on kdapltest In-Reply-To: <1115064877.11792.7.camel@duffman> References: <1115064517.11792.4.camel@duffman> <1115064877.11792.7.camel@duffman> Message-ID: Thanks Tom. Committed in revision 2253. An explanation of kdapltest is probably in order here. It is a test tool that has grown organically over time. The entire test could use a review for consistency and correctness. Here's a quick list of things that could be done: - review flow control, especially around disconnecting. There is a known race condition at disconnect w/ certain non-standard command lines (if the last operation is an RDMA, the target may exit before the operation is launched). - review naming conventions (e.g. use of "dapl_" prefix for files is confusing since the test is not related to the dat-provider code - separate user and kernel components more clearly, remove kdapltest reliance on the user level DAT headers james On Mon, 2 May 2005, Tom Duffy wrote: > Sparse fixes on kdapltest. > > Signed-off-by: Tom Duffy > > Index: gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_server.c > =================================================================== > --- gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_server.c (revision 2247) > +++ gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_server.c (working copy) > @@ -597,7 +597,7 @@ DT_cs_Server (Params_t * params_ptr) > DT_Mdep_Unlock (&ps_ptr->num_clients_lock); > > /* we passed the pt_ptr to the thread and must now 'forget' it */ > - pt_ptr = 0; > + pt_ptr = NULL; > > ret = dat_ep_disconnect (ps_ptr->ep_handle, DAT_CLOSE_GRACEFUL_FLAG); > if (ret != DAT_SUCCESS) > @@ -796,7 +796,7 @@ server_exit: > } /* end if ps_ptr */ > > /* Clean up the server list */ > - pre_list = 0; > + pre_list = NULL; > temp_list = DT_started_server_list; > while (temp_list) > { > Index: gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_test_data.c > =================================================================== > --- gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_test_data.c (revision 2247) > +++ gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_test_data.c (working copy) > @@ -37,7 +37,7 @@ Per_Test_Data_t * > DT_Alloc_Per_Test_Data (DT_Tdep_Print_Head *phead) > { > Per_Test_Data_t *pt_ptr; > - pt_ptr = 0; > + pt_ptr = NULL; > > pt_ptr = DT_Mdep_Malloc (sizeof (Per_Test_Data_t)); > if (!pt_ptr) > Index: gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_util.c > =================================================================== > --- gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_util.c (revision 2247) > +++ gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_util.c (working copy) > @@ -74,10 +74,10 @@ int DT_ep_create (Params_t *params_ptr, > { > DAT_RETURN status; > DT_Tdep_Print_Head *phead; > - *conn_evd= 0; > - *send_evd= 0; > - *recv_evd= 0; > - *cr_evd= 0; > + *conn_evd = NULL; > + *send_evd = NULL; > + *recv_evd = NULL; > + *cr_evd = NULL; > phead = params_ptr->phead; > > status = DT_Tdep_evd_create (ia_handle, DEFAULT_QUEUE_LEN, DAT_HANDLE_NULL, > @@ -125,18 +125,18 @@ int DT_ep_create (Params_t *params_ptr, > /* function that initializes the connection struct */ > void DT_fft_init_conn_struct (FFT_Connection_t *conn) > { > - conn->ia_handle = 0; > - conn->pz_handle = 0; > - conn->psp_handle = 0; > - conn->ep_handle = 0; > - conn->cr_evd = 0; > - conn->send_evd = 0; > - conn->conn_evd = 0; > - conn->recv_evd = 0; > - conn->cr_handle = 0; > - conn->remote_netaddr = 0; > - conn->bpool = 0; > - conn->pt_ptr = 0; > + conn->ia_handle = NULL; > + conn->pz_handle = NULL; > + conn->psp_handle = NULL; > + conn->ep_handle = NULL; > + conn->cr_evd = NULL; > + conn->send_evd = NULL; > + conn->conn_evd = NULL; > + conn->recv_evd = NULL; > + conn->cr_handle = NULL; > + conn->remote_netaddr = NULL; > + conn->bpool = NULL; > + conn->pt_ptr = NULL; > conn->connected = false; > } > > @@ -216,7 +216,7 @@ int DT_fft_destroy_conn_struct (Params_t > } > if (conn->bpool) > { > - DT_Bpool_Destroy (0, phead, conn->bpool); > + DT_Bpool_Destroy (NULL, phead, conn->bpool); > } > if (conn->psp_handle) > { > @@ -296,8 +296,9 @@ void DT_fft_init_server (Params_t *param > DT_assert_dat (phead, rc == DAT_SUCCESS); > > /* allocate memory for buffers */ > - conn->bpool = DT_BpoolAlloc (0, phead, conn->ia_handle, conn->pz_handle, NULL, NULL, > - 8192, 2, DAT_OPTIMAL_ALIGNMENT, false, false); > + conn->bpool = DT_BpoolAlloc (NULL, phead, conn->ia_handle, conn->pz_handle, > + NULL, NULL, 8192, 2, DAT_OPTIMAL_ALIGNMENT, > + false, false); > DT_assert (phead, conn->bpool); > cleanup: > return; > Index: gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_mem.c > =================================================================== > --- gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_mem.c (revision 2247) > +++ gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_mem.c (working copy) > @@ -50,12 +50,12 @@ int DT_mem_generic (Params_t *params_ptr > rc = 0; > expect = 0; > res = 1; > - lmr_handle = 0; > + lmr_handle = NULL; > lmr_context = 0; > reg_addr = 0; > - alloc_ptr = 0; > - ia_handle = 0; > - pz_handle = 0; > + alloc_ptr = NULL; > + ia_handle = NULL; > + pz_handle = NULL; > > DT_fft_init_client (params_ptr, cmd, &conn); > DT_assert (phead, NULL != conn.ia_handle); > @@ -63,7 +63,7 @@ int DT_mem_generic (Params_t *params_ptr > if (flag == 2) > { > buffer_size = 0; > - alloc_ptr = 0; > + alloc_ptr = NULL; > } > else > { > @@ -115,7 +115,7 @@ int DT_mem_generic (Params_t *params_ptr > rc = dat_lmr_free (lmr_handle); > DT_assert_dat (phead, rc == DAT_SUCCESS); > } > - lmr_handle = 0; > + lmr_handle = NULL; > > rc = dat_lmr_kcreate (conn.ia_handle, > DAT_MEM_TYPE_VIRTUAL, > Index: gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c > =================================================================== > --- gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c (revision 2247) > +++ gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c (working copy) > @@ -82,7 +82,7 @@ int DT_queryinfo_basic (Params_t *params > buffer_size = BUFFSIZE * sizeof (unsigned char); > phead = params_ptr->phead; > reg_addr = 0; > - alloc_ptr = 0; > + alloc_ptr = NULL; > > ia_handle = NULL; > pz_handle = NULL; > Index: gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_server_info.c > =================================================================== > --- gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_server_info.c (revision 2247) > +++ gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_server_info.c (working copy) > @@ -27,7 +27,7 @@ > > #include "dapl_proto.h" > > -Started_server_t *DT_started_server_list = 0; > +Started_server_t *DT_started_server_list = NULL; > > void > DT_Server_Info_Endian (Server_Info_t * server_info) > Index: gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_transaction_test.c > =================================================================== > --- gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_transaction_test.c (revision 2247) > +++ gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_transaction_test.c (working copy) > @@ -558,7 +558,7 @@ DT_Transaction_Main (void *param) > > /* what, me query? just try to accept the connection */ > ret = dat_cr_accept (cr_handle, > - 0, /* NULL for RSP */ > + NULL, /* NULL for RSP */ > 0, (DAT_PVOID)0 /* no private data */ ); > if (ret != DAT_SUCCESS) > { > Index: gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_memlist.c > =================================================================== > --- gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_memlist.c (revision 2247) > +++ gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_memlist.c (working copy) > @@ -31,7 +31,7 @@ void > DT_MemListInit (Per_Test_Data_t * pt_ptr) > { > DT_Mdep_LockInit (&pt_ptr->MemListLock); > - pt_ptr->MemListHead = 0; > + pt_ptr->MemListHead = NULL; > } > > void * > @@ -42,13 +42,13 @@ DT_MemListAlloc (Per_Test_Data_t * pt_pt > { > void *buffptr; > MemListEntry_t *entry_ptr; > - buffptr = 0; > - entry_ptr = 0; > + buffptr = NULL; > + entry_ptr = NULL; > > buffptr = DT_Mdep_Malloc (size); > if (buffptr == 0) > { > - return 0; > + return NULL; > } > if (pt_ptr == 0) /* not use mem_list */ > { > @@ -58,7 +58,7 @@ DT_MemListAlloc (Per_Test_Data_t * pt_pt > if (entry_ptr == 0) > { > DT_Mdep_Free (buffptr); > - return 0; > + return NULL; > } > strcpy (entry_ptr->filename, file); > entry_ptr->MemType = t; > @@ -82,7 +82,7 @@ DT_MemListFree (Per_Test_Data_t * pt_ptr > return; > } > DT_Mdep_Lock (&pt_ptr->MemListLock); > - pre = 0; > + pre = NULL; > cur = pt_ptr->MemListHead; > while (cur) > { > @@ -91,12 +91,12 @@ DT_MemListFree (Per_Test_Data_t * pt_ptr > if (!pre) /* first entry */ > { > pt_ptr->MemListHead = cur->next; > - cur->next = 0; > + cur->next = NULL; > } > else > { > pre->next = cur->next; > - cur->next = 0; > + cur->next = NULL; > } > DT_Mdep_Free (ptr); > DT_Mdep_Free (cur); > Index: gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_bpool.c > =================================================================== > --- gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_bpool.c (revision 2247) > +++ gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_bpool.c (working copy) > @@ -37,7 +37,7 @@ > */ > > > -void DTs_CreateIovec ( > +static void DTs_CreateIovec ( > void **region_addr, > u64 *buf_region, > DAT_VLEN buffer_size, > @@ -97,8 +97,8 @@ DT_BpoolAlloc ( > DAT_BOOLEAN enable_rdma_read) > { > unsigned char *module = "DT_BpoolAlloc"; > - unsigned char *alloc_ptr = 0; > - Bpool *bpool_ptr = 0; > + unsigned char *alloc_ptr = NULL; > + Bpool *bpool_ptr = NULL; > DAT_COUNT alloc_size, bpool_size; > DAT_REGION_DESCRIPTION region; > DAT_RETURN ret = DAT_SUCCESS; > @@ -308,7 +308,7 @@ err: > DT_MemListFree (pt_ptr, alloc_ptr); > } > > - return ( 0 ); > + return NULL; > } > > /*****************************************************************************/ > Index: gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_endpoint.c > =================================================================== > --- gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_endpoint.c (revision 2247) > +++ gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_endpoint.c (working copy) > @@ -177,14 +177,14 @@ int DT_endpoint_case2 (Params_t *params_ > DT_Tdep_PT_Printf (phead, "\ > Description: try to destroy ep with descriptor still in working queue\n"); > res = 1; > - bpool = 0; > - pz_handle = 0; > - ia_handle = 0; > - ep_handle = 0; > - send_evd = 0; > - conn_evd = 0; > - recv_evd = 0; > - cr_evd = 0; > + bpool = NULL; > + pz_handle = NULL; > + ia_handle = NULL; > + ep_handle = NULL; > + send_evd = NULL; > + conn_evd = NULL; > + recv_evd = NULL; > + cr_evd = NULL; > dev_name = cmd->device_name; > > rc = DT_ia_open (dev_name, &ia_handle); > @@ -200,7 +200,7 @@ int DT_endpoint_case2 (Params_t *params_ > &recv_evd, > &ep_handle); > DT_assert_dat (phead, rc == DAT_SUCCESS); > - bpool = DT_BpoolAlloc (0, phead, ia_handle, pz_handle, NULL, NULL, 4096, 1, > + bpool = DT_BpoolAlloc (NULL, phead, ia_handle, pz_handle, NULL, NULL, 4096, 1, > DAT_OPTIMAL_ALIGNMENT, false, false); > DT_assert (phead, bpool != 0); > DT_assert (phead, DT_post_recv_buffer (phead, > @@ -227,7 +227,7 @@ int DT_endpoint_case2 (Params_t *params_ > cleanup: > if (bpool) > { > - rc = DT_Bpool_Destroy (0, phead, bpool); > + rc = DT_Bpool_Destroy (NULL, phead, bpool); > DT_assert_clean (phead, rc != false); > } > if (pz_handle) > Index: gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_pz.c > =================================================================== > --- gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_pz.c (revision 2247) > +++ gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_pz.c (working copy) > @@ -47,8 +47,8 @@ int DT_pz_case0 ( Params_t *params_ptr, > Description: Test if we can normally create pz and destroy it.\n"); > > res=1; > - ia_handle=0; > - pz_handle =0; > + ia_handle = NULL; > + pz_handle = NULL; > evd_handle = DAT_HANDLE_NULL; > dev_name= cmd->device_name; > > @@ -88,13 +88,13 @@ int DT_pz_case1 (Params_t *params_ptr, F > Description: try to destroy pz with vi still associated with it\n"); > > res=1; > - ia_handle=0; > - pz_handle =0; > - ep_handle=0; > - conn_evd = 0; > - send_evd = 0; > - recv_evd = 0; > - cr_evd = 0; > + ia_handle = NULL; > + pz_handle = NULL; > + ep_handle = NULL; > + conn_evd = NULL; > + send_evd = NULL; > + recv_evd = NULL; > + cr_evd = NULL; > dev_name= cmd->device_name; > > rc = DT_ia_open (dev_name, &ia_handle); > @@ -174,9 +174,9 @@ int DT_pz_case2 (Params_t *params_ptr, F > associated with it\n"); > > res=1; > - ia_handle=0; > - pz_handle =0; > - bpool=0; > + ia_handle = NULL; > + pz_handle = NULL; > + bpool = NULL; > dev_name= cmd->device_name; > > rc = DT_ia_open (dev_name, &ia_handle); > @@ -186,7 +186,7 @@ int DT_pz_case2 (Params_t *params_ptr, F > DT_assert_dat (phead, rc == DAT_SUCCESS); > > /* allocate and register bpool */ > - bpool = DT_BpoolAlloc (0, phead, ia_handle, pz_handle, NULL, > + bpool = DT_BpoolAlloc (NULL, phead, ia_handle, pz_handle, NULL, > NULL, BUFFSIZE, 1, DAT_OPTIMAL_ALIGNMENT, > false, false); > DT_assert (phead, bpool != 0); > @@ -200,7 +200,7 @@ int DT_pz_case2 (Params_t *params_ptr, F > cleanup: > > /* deregister and free bpool */ > - if (DT_Bpool_Destroy (0, phead, bpool)==false) > + if (DT_Bpool_Destroy (NULL, phead, bpool)==false) > { > DT_Tdep_PT_Printf (phead, "Warning: Destroy bpool fails, reboot for cleanup\n"); > return 0; > Index: gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_hwconn.c > =================================================================== > --- gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_hwconn.c (revision 2247) > +++ gen2/users/jlentini/linux-kernel/test/dapltest/test/dapl_fft_hwconn.c (working copy) > @@ -42,7 +42,7 @@ int DT_hwconn_case0 ( Params_t *params_p > Description: Test if we can normally Open NIC and then close it\n"); > > dev_name= cmd->device_name; > - nic_handle=0; > + nic_handle = NULL; > evd_handle = DAT_HANDLE_NULL; > > rc=dat_ia_open ((const DAT_NAME_PTR)dev_name, 10, &evd_handle, &nic_handle); > @@ -128,7 +128,7 @@ int DT_hwconn_case2 (Params_t *params_pt > > DT_Tdep_PT_Printf (phead, "\ > Description: Try to close nic with Nic handle is null (NIC not open)\n"); > - nic_handle=0; > + nic_handle = NULL; > rc=dat_ia_close (nic_handle, DAT_CLOSE_ABRUPT_FLAG); > DT_assert_dat (phead, DAT_GET_TYPE (rc) ==DAT_INVALID_HANDLE); > > Index: gen2/users/jlentini/linux-kernel/test/dapltest/mdep/linux/dapl_mdep_kernel.c > =================================================================== > --- gen2/users/jlentini/linux-kernel/test/dapltest/mdep/linux/dapl_mdep_kernel.c (revision 2247) > +++ gen2/users/jlentini/linux-kernel/test/dapltest/mdep/linux/dapl_mdep_kernel.c (working copy) > @@ -241,7 +241,7 @@ DT_Mdep_Thread_Start_Routine (void *thre > daemonize(__func__); > #endif > thread_ptr->function (thread_ptr->param); > - return 0; > + return NULL; > } > > /* > Index: gen2/users/jlentini/linux-kernel/test/dapltest/kdapl/kdapl_module.c > =================================================================== > --- gen2/users/jlentini/linux-kernel/test/dapltest/kdapl/kdapl_module.c (revision 2247) > +++ gen2/users/jlentini/linux-kernel/test/dapltest/kdapl/kdapl_module.c (working copy) > @@ -90,7 +90,7 @@ static int kdapltest_ioctl (struct inode > break; > } > if (copy_from_user (local_params, > - (void *)param, > + (void __user *)param, > sizeof (Params_t))) > { > DT_Mdep_Free (local_params); > @@ -120,14 +120,14 @@ static int kdapltest_ioctl (struct inode > case KDAPL_IOCTL_GET_PRINTF: > { > if (copy_from_user (&print_ioctl, > - (void *)param, > + (void __user *)param, > sizeof (DT_get_printf_ioctl))) > { > rval = -EFAULT; > break; > } > rval = KDT_Get_Print_Line (print_ioctl.cookie, &buffer[0]); > - if (copy_to_user (print_ioctl.buffer, > + if (copy_to_user ((void __user *)print_ioctl.buffer, > &buffer[0], > strlen (buffer)+1)) > { > @@ -149,9 +149,9 @@ static int kdapltest_ioctl (struct inode > /* fops */ > static struct file_operations kdapltest_fops = > { > - open: kdapltest_open, > - release: kdapltest_release, > - ioctl: kdapltest_ioctl, > + .open = kdapltest_open, > + .release = kdapltest_release, > + .ioctl = kdapltest_ioctl, > }; > > static void > @@ -169,7 +169,7 @@ DT_Tdep_Test_Thread (Params_t *user_para > #endif > } > > -__init > +static __init > int kdapltest_init (void) > { > > @@ -183,7 +183,7 @@ int kdapltest_init (void) > return (0); > } > > -__exit > +static __exit > void kdapltest_exit (void) > { > unregister_chrdev (kdapltest_major, "kdapltest"); > From halr at voltaire.com Wed May 4 09:32:09 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 04 May 2005 12:32:09 -0400 Subject: [openib-general] RMPP Message-ID: <1115224329.4476.56.camel@localhost.localdomain> Hi Sean, In addition to passing the hdr_len and data_len to ib_create_send_mad: rmpp_mad->rmpp_hdr.paylen_newwin = cpu_to_be32(hdr_len - offsetof(struct ib_rmpp_mad, data) + data_len); That's not the "real" payload length that would go into the packet; just one segment's worth of class header. Correct ? If the above is correct, I'm not sure the actual payload lengths in the DATA packets are correct. I also see the padding on the last segment of a multipacket send not cleared (I integrated the part of your patch relating to the pad calculation). In one test case, I see the first RMPP DATA segment sent and there is no response (ACK) from the receiver (this was due to a (receiver) test program issue). The transmitter retry depends on retries in the send_wr ud structure. Does it need to known/Is there a way to know that this failed (no ACK, etc.) when retries are exhausted or is this reliant on the receiver rerequesting or is the entire RMPP transaction treated like a UD send (e.g. unreliable) ? Here's a summary of changes so far: 1. ib_create_send_mad either needs an additional parameter (RMPP active in current send packet) or the paylen_newwin needs to be set by the user outside of this routine. 2. Some minor ib_mad.h commentary changes for more clarity on the assumptions of the RMPP API. -- Hal From jlentini at netapp.com Wed May 4 09:42:31 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 4 May 2005 12:42:31 -0400 (EDT) Subject: [openib-general] [kDAPL][PATCH] delete cno user level code from kdapl In-Reply-To: <1115223242.14124.1.camel@duffman> References: <91DB792C7985D411BEC300B40080D29CC35A2D@mtvex01.mtv.mtl.com> <1115223242.14124.1.camel@duffman> Message-ID: Itamar, Can you resend this as an attachment? This has needed to be cleaned up for a long time. Thanks for working on it. james On Wed, 4 May 2005, Tom Duffy wrote: > On Wed, 2005-05-04 at 15:33 +0300, Itamar Rabenstein wrote: >> clean up kdapl code : delete cno user level code >> >> Signed-off-by: Itamar Rabenstein >> >> Index: linux-kernel/dat-provider/dapl_evd_free.c >> =================================================================== >> --- linux-kernel/dat-provider/dapl_evd_free.c (revision 2251) >> +++ linux-kernel/dat-provider/dapl_evd_free.c (working copy) >> @@ -74,16 +74,8 @@ >> } >> >> if (cno_ptr != NULL) >> - { >> - if (atomic_read (&cno_ptr->cno_ref_count) > 0 || >> cno_ptr->cno_waiters > 0) >> - { >> - dat_status = DAT_ERROR (DAT_INVALID_STATE, >> DAT_INVALID_STATE_EVD_IN_USE); >> - goto bail; > > Your patch was wrapped by your MUA. > > -tduffy > From itamar at mellanox.co.il Wed May 4 10:42:25 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Wed, 4 May 2005 20:42:25 +0300 Subject: [openib-general] [kDAPL][PATCH] delete cno user level code fr om kdapl Message-ID: <91DB792C7985D411BEC300B40080D29CC35A3A@mtvex01.mtv.mtl.com> resend with an a attachment. -Itamar > -----Original Message----- > From: James Lentini [mailto:jlentini at netapp.com] > Sent: Wednesday, May 04, 2005 7:43 PM > To: Itamar Rabenstein > Cc: openib-general at openib.org > Subject: Re: [openib-general] [kDAPL][PATCH] delete cno user > level code > from kdapl > > > > Itamar, > > Can you resend this as an attachment? > > This has needed to be cleaned up for a long time. Thanks for working > on it. > > james > > On Wed, 4 May 2005, Tom Duffy wrote: > > > On Wed, 2005-05-04 at 15:33 +0300, Itamar Rabenstein wrote: > >> clean up kdapl code : delete cno user level code > >> > >> Signed-off-by: Itamar Rabenstein > >> > >> Index: linux-kernel/dat-provider/dapl_evd_free.c > >> =================================================================== > >> --- linux-kernel/dat-provider/dapl_evd_free.c (revision 2251) > >> +++ linux-kernel/dat-provider/dapl_evd_free.c (working copy) > >> @@ -74,16 +74,8 @@ > >> } > >> > >> if (cno_ptr != NULL) > >> - { > >> - if (atomic_read (&cno_ptr->cno_ref_count) > 0 || > >> cno_ptr->cno_waiters > 0) > >> - { > >> - dat_status = DAT_ERROR (DAT_INVALID_STATE, > >> DAT_INVALID_STATE_EVD_IN_USE); > >> - goto bail; > > > > Your patch was wrapped by your MUA. > > > > -tduffy > > > -------------- next part -------------- A non-text attachment was scrubbed... Name: cno_clean.patch Type: application/octet-stream Size: 8767 bytes Desc: not available URL: From halr at voltaire.com Wed May 4 10:24:37 2005 From: halr at voltaire.com (Hal Rosenstock) Date: Wed, 4 May 2005 13:24:37 -0400 Subject: [openib-general] PF 5/4 Telecon Minutes Message-ID: <001101c550ce$29ad8590$6801a8c0@ownerdnt868gwf> Hi, Attached are the minutes from today's meeting. Let me know if there are any comments, questions, or issues. Thanks. Next meeting will be 5/11. -- Hal -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: PathForward Project Telecon 050504.txt URL: From ftillier at infiniconsys.com Wed May 4 10:32:27 2005 From: ftillier at infiniconsys.com (Fab Tillier) Date: Wed, 4 May 2005 10:32:27 -0700 Subject: [openib-general] FMR and how they work In-Reply-To: Message-ID: <000101c550cf$42608000$8d5aa8c0@infiniconsys.com> > From: Dan Bar Dov [mailto:danb at voltaire.com] > Sent: Wednesday, May 04, 2005 1:00 AM > > Voltair's IB based ISER indeed uses kDAPL so it will be very simple to > atapt it to iWARP. > . > Voltaire's kDAPL supports FMR using PLATFORM registration type, and it is > used by the ISER implementation. > Why doesn't iSER just use a single un-translated MR for all of physical memory (like the other kernel ULPs)? It would then just call the DMA mapping functions and be done with it - no registration at all is going to be faster than FMRs. - Fab From krause at cup.hp.com Wed May 4 08:42:45 2005 From: krause at cup.hp.com (Michael Krause) Date: Wed, 04 May 2005 08:42:45 -0700 Subject: [openib-general] RFD: uverbs and hotplug In-Reply-To: <469958e00505032053552e8b3c@mail.gmail.com> References: <20050503075141.GH9674@mellanox.co.il> <469958e005050309423adef75e@mail.gmail.com> <1115173071.29193.8.camel@duffman> <469958e00505032053552e8b3c@mail.gmail.com> Message-ID: <6.2.0.14.2.20050504083749.02a6f908@esmail.cup.hp.com> This string is somewhat missing the point: If a HCA is being used for a vital application service, then one needs an infrastructure that: (1) Enables the admin to identify whether a service will be impacted and to what extent - This is a very hard problem if the device acts as a consolidated service point (2) Enables the admin to migrate services to alternative devices if available (3) Enables the admin to prevent migration if no alternative is present (4) Translates attention button to notification (5) Automates much of the above since actual people are expensive (6) Enables remote service interaction points to recognize the migration and either adapt or clean up BTW, whether an application is written to deal with surprise remove isn't really the issue. All applications and kernel must deal with failures. How graceful and how transparent to the end user is what matters. Hence, The above is not unique to IB or really any device. It is an OS problem and what is required is an OS driven strategy that can support multiple device types and usage paradigms. Mike At 08:53 PM 5/3/2005, Caitlin Bestler wrote: >On 5/3/05, Tom Duffy wrote: > > On Tue, 2005-05-03 at 09:42 -0700, Caitlin Bestler wrote: > > > Most applications are not written to be able to > > > survice a sudden loss of virtually all of its connections, and then > resume > > > all of the suspended sessions with new connections using new memory > > > regions. > > > > Most applications *should* be written this way. Especially since more > > and more people are using unreliable networks (wifi, 3G, etc.). I can > > see a day where we have RDMA-capable wireless adapters in which case > > applications will have to be written with the possibility of a sudden > > loss taken into account. > > > > -tduffy > > >And applications should be prepared for file systems to fail as well, >but that doesn't mean that the file system shouldn't consider features >such as journaling that minimize how often the application will be >faced that sort of problem. > >It would certainly be valid to decide that the effort was not worthwhile >given the effort required. But such a decision should be made with >full awareness that the notice will enable very few applications to >actually recover when an entire HCA is taken down. If the goal >is to realistically allow applications to confinue functioning after >the loss of multiple ports some more powerful hooks are required. >_______________________________________________ >openib-general mailing list >openib-general at openib.org >http://openib.org/mailman/listinfo/openib-general > >To unsubscribe, please visit >http://openib.org/mailman/listinfo/openib-general -------------- next part -------------- An HTML attachment was scrubbed... URL: From krause at cup.hp.com Wed May 4 08:46:07 2005 From: krause at cup.hp.com (Michael Krause) Date: Wed, 04 May 2005 08:46:07 -0700 Subject: [openib-general] Re: RFD: uverbs and hotplug In-Reply-To: <20050504114439.GK9674@mellanox.co.il> References: <1115173071.29193.8.camel@duffman> <20050504114439.GK9674@mellanox.co.il> Message-ID: <6.2.0.14.2.20050504084312.02a6ee50@esmail.cup.hp.com> At 04:44 AM 5/4/2005, Michael S. Tsirkin wrote: >The problem is not that hard. The driver must provide sufficient >information to discover that the device went down and which requests >were serviced before the device went down. > >Libraries such as MPI will be able to use that to resend these requests on >a different HCA or after an HCA is plugged back in. > >Applications will simply use this library and be happy. The operating assumption here is that applications or middleware can simply restart. That is a nice theory but not always practical especially when taking the service usage model. Some applications seeing failure may deem a cluster partitioning event has occurred and initiate membership quorum, etc. algorithms. Others might invoke DR algorithms and fail over to alternate systems or sites. This problem has been worked for many years by various vendors and it still isn't fully solved. Do not think that simple resource recovery and restart is an acceptable answer to customers. It is the minimum any subsystem or device should support but that does not translate automatically into a viable solution. Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From mshefty at ichips.intel.com Wed May 4 10:43:36 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Wed, 04 May 2005 10:43:36 -0700 Subject: [openib-general] Re: RMPP In-Reply-To: <1115224329.4476.56.camel@localhost.localdomain> References: <1115224329.4476.56.camel@localhost.localdomain> Message-ID: <427909C8.4030307@ichips.intel.com> Hal Rosenstock wrote: > Hi Sean, > > In addition to passing the hdr_len and data_len to ib_create_send_mad: > rmpp_mad->rmpp_hdr.paylen_newwin = cpu_to_be32(hdr_len - > offsetof(struct ib_rmpp_mad, data) + data_len); > That's not the "real" payload length that would go into the packet; just > one segment's worth of class header. Correct ? There are two sizes needed by the RMPP code. First, it needs the total length of the data buffer, which includes any necessary padding. This is set in the scatter-gather entry for the work request. Second, it needs to know how much of the data buffer contains valid user data. ib_create_send_mad returns the length of the of the valid user data in a slightly encoded format. It subtracts the size of the RMPP and common MAD headers from the size (see below). The intent is that the payload length value is set to the correct value for MADs that are a single segment in length. ------------- MAD header ------------- RMPP header ------------- SA/Vendor hdr ------------- User data ------------- Pad ------------- sge length = size from MAD header to end of pad. payload = size from SA/Vendor hdr to end of user data. > If the above is correct, I'm not sure the actual payload lengths in the > DATA packets are correct. I also see the padding on the last segment of > a multipacket send not cleared (I integrated the part of your patch > relating to the pad calculation). I will run some tests and verify the payload values using madeye. I'm not sure why the padding isn't cleared. It may be an indication that create_send_mad isn't allocating the correct pad size. > In one test case, I see the first RMPP DATA segment sent and there is no > response (ACK) from the receiver (this was due to a (receiver) test > program issue). The transmitter retry depends on retries in the send_wr > ud structure. Does it need to known/Is there a way to know that this > failed (no ACK, etc.) when retries are exhausted or is this reliant on > the receiver rerequesting or is the entire RMPP transaction treated like > a UD send (e.g. unreliable) ? The RMPP send is treated as reliable, even if no response is expected. If the send is not ACKed completely, the request will timeout, and a send failure is reported to the user. If the send completes successfully, then the user knows that it was received by the remote side; although, there's no guarantee that it was processed by anyone. The code will retry a given segment the number of times specified by the user. If forward progress is made on the send, the retry count is reset. > Here's a summary of changes so far: > 1. ib_create_send_mad either needs an additional parameter (RMPP active > in current send packet) or the paylen_newwin needs to be set by the user > outside of this routine. I have this on my short term to-do list. I'm just not sure on the best approach yet... > 2. Some minor ib_mad.h commentary changes for more clarity on the > assumptions of the RMPP API. From eitan at mellanox.co.il Wed May 4 10:51:30 2005 From: eitan at mellanox.co.il (Eitan Zahavi) Date: Wed, 4 May 2005 20:51:30 +0300 Subject: [openib-general] OpenSM work Q1 + Apr. Message-ID: <506C3D7B14CDD411A52C00025558DED6047EF1F8@mtlex01.yok.mtl.com> Hi, I have just uploaded to OpenIB https://openib.org/svn/gen1/trunk/src/userspace/osm a new change set (2255) with OpenSM improvements. The work was done according to the plan outlined in the attached mail. The main changes are: 1. Semi-Static LID assignment - preventing LID changes due to SM restart or nodes rebooting. Important due to the bad affect LID changes have on IPoIB (and other ULPs) path record caching. 2. Irresponsive port scanning during light sweep - Allow the SM to recognize ports that were not responding in the first sweep (but had a link state that is not down). 3. Switch Ports that connect to HCA ports now have a lower HOQ value. This provides faster "drain" for packets waiting for bad HCA such that the entire fabric is not affected by a single HCA. I will outline our next months work plan in my next mail for your comments. Eitan Zahavi Design Technology Director Mellanox Technologies LTD Tel:+972-4-9097208 Fax:+972-4-9593245 P.O. Box 586 Yokneam 20692 ISRAEL -----Original Message----- From: Eitan Zahavi [mailto:eitan at mellanox.co.il] Sent: Friday, April 08, 2005 7:40 PM To: 'openib-general at openib.org' Subject: [openib-general] OpenSM work Hi All, FYI: Mellanox is focusing on the following items on OpenSM development for the last few weeks: 1. Stability testing over the IB management simulator: a. Randomly pick bad links with high packet drop statistics - success is SUBNET UP b. Route using up/down algorithm - success is no credit loops 2. Semi-static LID assignment: a. Developed an interface for persistent storage of arbitrary data. The goal is to enable further development of LDAP (ala Troy's request) or SQL module. Please see osm_db.h attached <> b. Developed file based implementation for osm_db.h c. Modify osm_lid_mgr (lid assignment algorithm) to use the LIDs stored in the persistent storage. Handle all cases of bad file and new LIDs on the fabric. The -r flag now lets OpenSM overwrite the known data. Persistent Guid to LIDs data is kept even if the GUID disappears for a while. The code also handles LID assignment for LMC > 0 in a way better then the previous algorithm: It used to assign 2^LMC LIDs for every port - even for switches port 0. Now it will only preserve 1 LID for switch port 0. 3. Irresponsive port: a. The phenomenon is: A port does not respond to the SM during the discovery stage. OpenSM can not obtain enough data about the port and thus it does not appear in the final database. Since OpenSM uses light sweeps when there is no "change detected" it will not query the port until either a switch sets its "change bit" or send a trap. So that irresponsive port will never be polled again if there is no heavy sweep. b. The solution: i. During discovery track ports (physical ports) that have their logical link state != DOWN but the port on the other side of the link is not known to the SM. ii. During light sweep: not only scan the switches "change bit" but also test to see if the port on the other side on these ports (from i) is responding. If it does - issue a heavy sweep. 4. Head of Queue Life: a. Problem: In cases of PCI hardware failure HCAs can not complete RDMA requests and loose all credits from their input ports (in other words: their input buffers are filled). So they create back pressure on the fabric. b. Solution: use a fast head of queue time limit on every switch port that drives an HCA. 5. SA queries stress testing: a. We are exploring max performance of the SA and ways to improve it. Eitan Eitan Zahavi Design Technology Director Mellanox Technologies LTD Tel:+972-4-9097208 Fax:+972-4-9593245 P.O. Box 586 Yokneam 20692 ISRAEL -------------- next part -------------- An HTML attachment was scrubbed... URL: From eitan at mellanox.co.il Wed May 4 10:59:23 2005 From: eitan at mellanox.co.il (Eitan Zahavi) Date: Wed, 4 May 2005 20:59:23 +0300 Subject: [openib-general] OpenSM Work Q2 2005 Message-ID: <506C3D7B14CDD411A52C00025558DED6047EF1F9@mtlex01.yok.mtl.com> Hi All, For your comments and information Mellanox plans to work on the following new OpenSM feature and enhancements this quarter (and maybe next one): 1. PKey re-ordering - current implementation of PKey tables in OpenSM "re-orders" the PKeys. This breaks any installation that uses PKeys. 2. Routing algorithm time - 2minutes for 1K nodes. The solution is to implement a new "tree based" router that scales to 50K nodes fat tree and also implement "loading" of pre-routes. NOTE: I will send a mail describing the proposed algorithms and will highly appreciate any input regarding other alternatives and review of the provided Order of complexity. 3. SL2VL support by Path Record - current implementation ignores any SL2VL mapping. If one will configure those somehow, wrong path record results will break connectivity. 4. Default PKey values are not set - New hardware that does not have default known PKey (0xffff) will fail any communication as the SM is supposed to set that value for it. 5. QoS - OpenSM does not implement any QoS policy. Seems like people are more interested in that too. The solution is to implement a simple uniform QoS based on SL2VL and VLArb. Your comments and ideas are welcome. Thanks Eitan Zahavi Design Technology Director Mellanox Technologies LTD Tel:+972-4-9097208 Fax:+972-4-9593245 P.O. Box 586 Yokneam 20692 ISRAEL -------------- next part -------------- An HTML attachment was scrubbed... URL: From halr at voltaire.com Wed May 4 11:00:08 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 04 May 2005 14:00:08 -0400 Subject: [openib-general] OpenSM work Q1 + Apr. In-Reply-To: <506C3D7B14CDD411A52C00025558DED6047EF1F8@mtlex01.yok.mtl.com> References: <506C3D7B14CDD411A52C00025558DED6047EF1F8@mtlex01.yok.mtl.com> Message-ID: <1115229607.4476.146.camel@localhost.localdomain> Hi Eitan, On Wed, 2005-05-04 at 13:51, Eitan Zahavi wrote: > 1. Irresponsive port scanning during light sweep - Allow the SM > to recognize ports that were not responding in the first sweep > (but had a link state that is not down). This sounds like the bad port handling algorithm we bashed on this list. Is it ? -- Hal From caitlin.bestler at gmail.com Wed May 4 11:17:37 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Wed, 4 May 2005 11:17:37 -0700 Subject: [openib-general] FMR and how they work In-Reply-To: <000101c550cf$42608000$8d5aa8c0@infiniconsys.com> References: <000101c550cf$42608000$8d5aa8c0@infiniconsys.com> Message-ID: <469958e00505041117429ac4c2@mail.gmail.com> On 5/4/05, Fab Tillier wrote: > > From: Dan Bar Dov [mailto:danb at voltaire.com] > > Sent: Wednesday, May 04, 2005 1:00 AM > > > > Voltair's IB based ISER indeed uses kDAPL so it will be very simple to > > atapt it to iWARP. > > . > > Voltaire's kDAPL supports FMR using PLATFORM registration type, and it is > > used by the ISER implementation. > > > > Why doesn't iSER just use a single un-translated MR for all of physical > memory (like the other kernel ULPs)? It would then just call the DMA > mapping functions and be done with it - no registration at all is going to > be faster than FMRs. > An FMR can be used to advertise a target buffer that is being read into as a single logical buffer. Using physical memory would require a) exporting the physical page list of where your buffers were (making the buffer advertisement larger and more complex) b) Trusting whoever is on the other end of the connection with access to your entire physical memory. From caitlin.bestler at gmail.com Wed May 4 11:14:25 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Wed, 4 May 2005 11:14:25 -0700 Subject: [openib-general] Re: RFD: uverbs and hotplug In-Reply-To: <6.2.0.14.2.20050504084312.02a6ee50@esmail.cup.hp.com> References: <1115173071.29193.8.camel@duffman> <20050504114439.GK9674@mellanox.co.il> <6.2.0.14.2.20050504084312.02a6ee50@esmail.cup.hp.com> Message-ID: <469958e005050411144a4e3eaa@mail.gmail.com> On 5/4/05, Michael Krause wrote: > At 04:44 AM 5/4/2005, Michael S. Tsirkin wrote: > > > > The problem is not that hard. The driver must provide sufficient > information to discover that the device went down and which requests > were serviced before the device went down. > > Libraries such as MPI will be able to use that to resend these requests on > a different HCA or after an HCA is plugged back in. > > Applications will simply use this library and be happy. > The operating assumption here is that applications or middleware can simply > restart. That is a nice theory but not always practical especially when > taking the service usage model. Some applications seeing failure may deem a > cluster partitioning event has occurred and initiate membership quorum, etc. > algorithms. Others might invoke DR algorithms and fail over to alternate > systems or sites. This problem has been worked for many years by various > vendors and it still isn't fully solved. Do not think that simple resource > recovery and restart is an acceptable answer to customers. It is the > minimum any subsystem or device should support but that does not translate > automatically into a viable solution. > > Mike And even if the application properly diagnoses the problems and does not falsely take any of the more draconian solutions outlined, we still need to recognize that "just restarting" is a much bigger problem for an RDMA application than it is for a socket application. If the HCA/RNIC closes, then the application has to do more than "just" reopen it and re-establish the connection. It has to recreate the PDs, CQs, SRQs, MRs, MWs and QPs. And *after* the connections are re-established any persistent buffer adverisements must be re-exchanged. Avoiding imposing that heavy of a burden whenever possible strikes me as a worthwhile objective. At the minimum it needs to be clear on the burden that is being created if suspend/resume support is not included. It is not "just" a matter of reopeing and reconnecting. From timur.tabi at ammasso.com Wed May 4 11:27:54 2005 From: timur.tabi at ammasso.com (Timur Tabi) Date: Wed, 04 May 2005 13:27:54 -0500 Subject: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation In-Reply-To: <20050425203110.A9729@topspin.com> References: <200544159.Ahk9l0puXy39U6u6@topspin.com> <20050411142213.GC26127@kalmia.hozed.org> <52mzs51g5g.fsf@topspin.com> <20050411163342.GE26127@kalmia.hozed.org> <5264yt1cbu.fsf@topspin.com> <20050411180107.GF26127@kalmia.hozed.org> <52oeclyyw3.fsf@topspin.com> <20050411171347.7e05859f.akpm@osdl.org> <20050412180447.E6958@topspin.com> <20050425203110.A9729@topspin.com> Message-ID: <4279142A.8050501@ammasso.com> Libor Michalek wrote: > The program opens the charcter device file descriptor, pins the pages > and waits for a signal, before checking the pages, which is sent to the > process after running some other program which exercises the VM. On older > kernels the check fails, on my 2.6.11 kernel the check succeeds. So > mlock is not needed on top of get_user_pages() as it was before. Libor, When you say "older", what exactly do you mean? I have different test that normally fails with just get_user_pages(), but it works with 2.6.9 and above. I haven't been able to get any kernel earlier than 2.6.9 to compile or boot properly, so I'm having a hard time narrowing down the actual point when get_user_pages() started working. -- Timur Tabi Staff Software Engineer timur.tabi at ammasso.com One thing a Southern boy will never say is, "I don't think duct tape will fix it." -- Ed Smylie, NASA engineer for Apollo 13 From bjordan.ics at gmail.com Wed May 4 11:22:00 2005 From: bjordan.ics at gmail.com (William Jordan) Date: Wed, 4 May 2005 14:22:00 -0400 Subject: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation In-Reply-To: <20050503184325.GA19351@hexapodia.org> References: <469958e00504291731eb8287c@mail.gmail.com> <20050503184325.GA19351@hexapodia.org> Message-ID: <78d18e2050504112240e43a08@mail.gmail.com> On 5/3/05, Andy Isaacson wrote: > Rather than replacing the fully-registered pages with pages of zeros, > you could simply unmap them. I don't like this option. It is nearly free to map all of the pages to the zero-page. You never have to allocate a page if the user never writes to it. Buf if you unmap the page, there could be issues. The memory region could be on the stack, or malloc'ed. In these cases, the child should be able to return from the function, or free the memory without setting a timebomb. -- Bill Jordan InfiniCon Systems From ftillier at infiniconsys.com Wed May 4 11:31:00 2005 From: ftillier at infiniconsys.com (Fab Tillier) Date: Wed, 4 May 2005 11:31:00 -0700 Subject: [openib-general] FMR and how they work In-Reply-To: <469958e00505041117429ac4c2@mail.gmail.com> Message-ID: <000201c550d7$6c7b0740$8d5aa8c0@infiniconsys.com> > From: Caitlin Bestler [mailto:caitlin.bestler at gmail.com] > Sent: Wednesday, May 04, 2005 11:18 AM > > On 5/4/05, Fab Tillier wrote: > > > From: Dan Bar Dov [mailto:danb at voltaire.com] > > > Sent: Wednesday, May 04, 2005 1:00 AM > > > > > > Voltair's IB based ISER indeed uses kDAPL so it will be very simple to > > > atapt it to iWARP. > > > > > > Voltaire's kDAPL supports FMR using PLATFORM registration type, and it > > > is used by the ISER implementation. > > > > > > > Why doesn't iSER just use a single un-translated MR for all of physical > > memory (like the other kernel ULPs)? It would then just call the DMA > > mapping functions and be done with it - no registration at all is going > > to be faster than FMRs. > > > > > An FMR can be used to advertise a target buffer that is being read into > as a single logical buffer. Using physical memory would require > > a) exporting the physical page list of where your buffers were (making > the buffer advertisement larger and more complex) > > b) Trusting whoever is on the other end of the connection with access > to your entire physical memory. I assume that iSER (like SRP) supports scatter gather in I/O requests, so a) shouldn't matter - that is, I expect that the storage subsystem hands the iSER driver a list of physical addresses and a SCSI-like command. Copying the page list is far simpler than getting a virtual address. I see no issue with trusting I/O controllers on the fabric. They are no more a threat than local I/O controllers. This addresses issue b). - Fab From halr at voltaire.com Wed May 4 11:25:57 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 04 May 2005 14:25:57 -0400 Subject: [openib-general] Re: [openib-commits] r2256 - in gen2/trunk/src/linux-kernel/infiniband: core include In-Reply-To: <20050504181656.AED7022834D@openib.ca.sandia.gov> References: <20050504181656.AED7022834D@openib.ca.sandia.gov> Message-ID: <1115230835.4476.207.camel@localhost.localdomain> On Wed, 2005-05-04 at 14:16, sean.hefty at openib.org wrote: > +void ib_coalesce_recv_mad(struct ib_mad_recv_wc *mad_recv_wc, void *buf) > +{ > + struct ib_mad_recv_buf *seg_buf; > + struct ib_rmpp_mad *rmpp_mad; > + void *data; > + int size, len, offset; > + u8 flags; > + > + len = mad_recv_wc->mad_len; > + if (len <= sizeof(struct ib_mad)) { > + memcpy(buf, mad_recv_wc->recv_buf.mad, len); > + return; > + } > + > + offset = data_offset(mad_recv_wc->recv_buf.mad->mad_hdr.mgmt_class); > + > + list_for_each_entry(seg_buf, &mad_recv_wc->rmpp_list, list) { > + rmpp_mad = (struct ib_rmpp_mad *)seg_buf->mad; > + flags = ib_get_rmpp_flags(&rmpp_mad->rmpp_hdr); > + > + if (flags & IB_MGMT_RMPP_FLAG_FIRST) { > + data = rmpp_mad; > + size = sizeof(*rmpp_mad); > + } else { > + data = rmpp_mad + offset; > + if (flags & IB_MGMT_RMPP_FLAG_LAST) > + size = len; > + else > + size = sizeof(*rmpp_mad) - offset; > + } > + > + memcpy(buf, data, size); > + len -= size; > + buf += size; > + } > +} So user_mad should have similar code on the receive side with copy_to_user rather than memcpy, as it should be the only consumer which needs this, right ? -- Hal From mshefty at ichips.intel.com Wed May 4 11:35:05 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Wed, 04 May 2005 11:35:05 -0700 Subject: [openib-general] Re: [openib-commits] r2256 - in gen2/trunk/src/linux-kernel/infiniband: core include In-Reply-To: <1115230835.4476.207.camel@localhost.localdomain> References: <20050504181656.AED7022834D@openib.ca.sandia.gov> <1115230835.4476.207.camel@localhost.localdomain> Message-ID: <427915D9.9060007@ichips.intel.com> Hal Rosenstock wrote: > So user_mad should have similar code on the receive side with > copy_to_user rather than memcpy, as it should be the only consumer which > needs this, right ? Correct. From halr at voltaire.com Wed May 4 11:29:19 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 04 May 2005 14:29:19 -0400 Subject: [openib-general] Re: RMPP In-Reply-To: <427909C8.4030307@ichips.intel.com> References: <1115224329.4476.56.camel@localhost.localdomain> <427909C8.4030307@ichips.intel.com> Message-ID: <1115231304.4476.214.camel@localhost.localdomain> On Wed, 2005-05-04 at 13:43, Sean Hefty wrote: > Hal Rosenstock wrote: > > Hi Sean, > > > > In addition to passing the hdr_len and data_len to ib_create_send_mad: > > rmpp_mad->rmpp_hdr.paylen_newwin = cpu_to_be32(hdr_len - > > offsetof(struct ib_rmpp_mad, data) + data_len); > > That's not the "real" payload length that would go into the packet; just > > one segment's worth of class header. Correct ? > > There are two sizes needed by the RMPP code. First, it needs the total > length of the data buffer, which includes any necessary padding. This is > set in the scatter-gather entry for the work request. Second, it needs to > know how much of the data buffer contains valid user data. Got it. > ib_create_send_mad returns the length of the of the valid user data in a > slightly encoded format. It subtracts the size of the RMPP and common MAD > headers from the size (see below). The intent is that the payload length > value is set to the correct value for MADs that are a single segment in length. Right. That might be a better way of expressing it than how I did :-) > ------------- > MAD header > ------------- > RMPP header > ------------- > SA/Vendor hdr > ------------- > User data > ------------- > Pad > ------------- > > sge length = size from MAD header to end of pad. > payload = size from SA/Vendor hdr to end of user data. > > > If the above is correct, I'm not sure the actual payload lengths in the > > DATA packets are correct. I also see the padding on the last segment of > > a multipacket send not cleared (I integrated the part of your patch > > relating to the pad calculation). > > I will run some tests and verify the payload values using madeye. I'm not > sure why the padding isn't cleared. It may be an indication that > create_send_mad isn't allocating the correct pad size. Thanks. > > In one test case, I see the first RMPP DATA segment sent and there is no > > response (ACK) from the receiver (this was due to a (receiver) test > > program issue). The transmitter retry depends on retries in the send_wr > > ud structure. Does it need to known/Is there a way to know that this > > failed (no ACK, etc.) when retries are exhausted or is this reliant on > > the receiver rerequesting or is the entire RMPP transaction treated like > > a UD send (e.g. unreliable) ? > > The RMPP send is treated as reliable, even if no response is expected. If > the send is not ACKed completely, the request will timeout, and a send > failure is reported to the user. If the send completes successfully, then > the user knows that it was received by the remote side; although, there's no > guarantee that it was processed by anyone. Is this on any (RMPP) transmits or just requests ? I will check to see if I can see this. > The code will retry a given segment the number of times specified by the > user. If forward progress is made on the send, the retry count is reset. for each subsequent segment. Makes sense. > > Here's a summary of changes so far: > > 1. ib_create_send_mad either needs an additional parameter (RMPP active > > in current send packet) or the paylen_newwin needs to be set by the user > > outside of this routine. > > I have this on my short term to-do list. I'm just not sure on the best > approach yet... OK. I'll wait. I have my workaround right now. > > 2. Some minor ib_mad.h commentary changes for more clarity on the > > assumptions of the RMPP API. Thanks. -- Hal From halr at voltaire.com Wed May 4 11:32:52 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 04 May 2005 14:32:52 -0400 Subject: [openib-general] OpenSM work Q1 + Apr. In-Reply-To: <506C3D7B14CDD411A52C00025558DED6047EF1F8@mtlex01.yok.mtl.com> References: <506C3D7B14CDD411A52C00025558DED6047EF1F8@mtlex01.yok.mtl.com> Message-ID: <1115231473.4476.216.camel@localhost.localdomain> On Wed, 2005-05-04 at 13:51, Eitan Zahavi wrote: > I have just uploaded to OpenIB > https://openib.org/svn/gen1/trunk/src/userspace/osm > > a new change set (2255) with OpenSM improvements. Also, does this translate to an IBGD release (number) ? Did this release pick up any of the fixes from OpenIB (gen2) ? -- Hal From tduffy at sun.com Wed May 4 11:41:47 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 04 May 2005 11:41:47 -0700 Subject: [openib-general] OpenSM work Q1 + Apr. In-Reply-To: <506C3D7B14CDD411A52C00025558DED6047EF1F8@mtlex01.yok.mtl.com> References: <506C3D7B14CDD411A52C00025558DED6047EF1F8@mtlex01.yok.mtl.com> Message-ID: <1115232107.1657.3.camel@duffman> On Wed, 2005-05-04 at 20:51 +0300, Eitan Zahavi wrote: > I have just uploaded to OpenIB > https://openib.org/svn/gen1/trunk/src/userspace/osm > > a new change set (2255) with OpenSM improvements. I am confused. Why did you check into gen1? Weren't we going to move gen1 to an archive section and make it read only? -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From mshefty at ichips.intel.com Wed May 4 11:49:08 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Wed, 04 May 2005 11:49:08 -0700 Subject: [openib-general] Re: RMPP In-Reply-To: <1115231304.4476.214.camel@localhost.localdomain> References: <1115224329.4476.56.camel@localhost.localdomain> <427909C8.4030307@ichips.intel.com> <1115231304.4476.214.camel@localhost.localdomain> Message-ID: <42791924.4070108@ichips.intel.com> Hal Rosenstock wrote: >>>In one test case, I see the first RMPP DATA segment sent and there is no >>>response (ACK) from the receiver (this was due to a (receiver) test >>>program issue). The transmitter retry depends on retries in the send_wr >>>ud structure. Does it need to known/Is there a way to know that this >>>failed (no ACK, etc.) when retries are exhausted or is this reliant on >>>the receiver rerequesting or is the entire RMPP transaction treated like >>>a UD send (e.g. unreliable) ? >> >>The RMPP send is treated as reliable, even if no response is expected. If >>the send is not ACKed completely, the request will timeout, and a send >>failure is reported to the user. If the send completes successfully, then >>the user knows that it was received by the remote side; although, there's no >>guarantee that it was processed by anyone. > > > Is this on any (RMPP) transmits or just requests ? I will check to see > if I can see this. This is true for any RMPP send operation. You can test for this by running a grmpp client, without having the grmpp server loaded on the destination node. You should see a timeout error sending the MAD in the messages log file. You can also see timeouts on both requests and replies if you run grmpp and increase the number of messages to a few thousand. (e.g. insmod ib_grmpp.ko "slid=1" "dlid=2" "message_count=15000" "message_size=100" "responses=1") - Sean From tduffy at sun.com Wed May 4 11:48:15 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 04 May 2005 11:48:15 -0700 Subject: [openib-general] [kDAPL][PATCH] delete cno user level code fr om kdapl In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35A3A@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35A3A@mtvex01.mtv.mtl.com> Message-ID: <1115232495.1657.6.camel@duffman> On Wed, 2005-05-04 at 20:42 +0300, Itamar Rabenstein wrote: > resend with an a attachment. > Content-Type: application/octet-stream; name="cno_clean.patch" Please, if you are going to send as an attachment, send as text/plain. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From mst at mellanox.co.il Wed May 4 12:00:35 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 4 May 2005 22:00:35 +0300 Subject: [openib-general] Re: [PATCH][SDP] AIO buffer corruption In-Reply-To: <20050427182254.A11429@topspin.com> References: <20050427182254.A11429@topspin.com> Message-ID: <20050504190035.GA32291@mellanox.co.il> Quoting r. Libor Michalek : > Subject: [PATCH][SDP] AIO buffer corruption > > > Patch to fix the problem a few people reported as ttcp.aio.c > aborting with an error (-104) on longer AIO runs. > > The bug is in the calculation of an AIO buffers starting address. > It would cause data to potentially be written past the end of the > AIO buffer corrupting whatever happen to be there. In the case of > ttcp.aio.c this happen to be the iocb array, which once corrupted > would generate this error when passed to io_submit. > > -Libor > > Signed-off-by: Libor Michalek > > Index: sdp_recv.c > =================================================================== > --- sdp_recv.c (revision 2220) > +++ sdp_recv.c (working copy) > @@ -674,14 +674,16 @@ > #ifndef _SDP_DATA_PATH_NULL > memcpy((addr + offset), buff->data, copy); > #endif > - > + > buff->data += copy; > iocb->post += copy; > iocb->len -= copy; > > offset += copy; > offset &= (~PAGE_MASK); > - > + > + iocb->io_addr += copy; > + > sdp_kunmap(iocb->page_array[counter++]); > } > /* > @@ -1443,7 +1445,8 @@ > iocb->size = size; > iocb->req = req; > iocb->key = req->ki_key; > - iocb->addr = (unsigned long)msg->msg_iov->iov_base; > + iocb->addr = ((unsigned long)msg->msg_iov->iov_base - > + copied); > > req->ki_cancel = sdp_inet_read_cancel; > > Index: sdp_send.c > =================================================================== > --- sdp_send.c (revision 2220) > +++ sdp_send.c (working copy) > @@ -751,6 +751,7 @@ > buff->tail += copy; > iocb->post += copy; > iocb->len -= copy; > + iocb->io_addr += copy; > > offset += copy; > offset &= (~PAGE_MASK); > @@ -2195,7 +2196,7 @@ > iocb->size = size; > iocb->req = req; > iocb->key = req->ki_key; > - iocb->addr = (unsigned long)msg->msg_iov->iov_base; > + iocb->addr = (unsigned long)msg->msg_iov->iov_base - copied; > > req->ki_cancel = sdp_inet_write_cancel; > Unfortunately I still see data corruptions sometimes with this patch applied. The result for me is the server reporting verification error, closing the socket, and client printing the 104 event. I'm still debugging, but wanted to ask if someone else is seeing this too. Libor, on an unrelated note, could you please generate diffs with -p flag to make it easier to see which function got changed? Thanks, -- MST - Michael S. Tsirkin From caitlin.bestler at gmail.com Wed May 4 12:01:53 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Wed, 4 May 2005 12:01:53 -0700 Subject: [openib-general] FMR and how they work In-Reply-To: <000201c550d7$6c7b0740$8d5aa8c0@infiniconsys.com> References: <469958e00505041117429ac4c2@mail.gmail.com> <000201c550d7$6c7b0740$8d5aa8c0@infiniconsys.com> Message-ID: <469958e005050412013a51f5bc@mail.gmail.com> > > > > > > An FMR can be used to advertise a target buffer that is being read into > > as a single logical buffer. Using physical memory would require > > > > a) exporting the physical page list of where your buffers were (making > > the buffer advertisement larger and more complex) > > > > b) Trusting whoever is on the other end of the connection with access > > to your entire physical memory. > > I assume that iSER (like SRP) supports scatter gather in I/O requests, so a) > shouldn't matter - that is, I expect that the storage subsystem hands the > iSER driver a list of physical addresses and a SCSI-like command. Copying > the page list is far simpler than getting a virtual address. > > I see no issue with trusting I/O controllers on the fabric. They are no > more a threat than local I/O controllers. This addresses issue b). > I must have missed something in the spec. Where is the cryptographic header that guarantees that you are indeed connected to another I/O controller and not somebody just claiming to be an iSER device? Isn't that akin to assuming that the 'root' user on that other system is another sysadmin so I can trust them? Generally I assume the entity on the other end of the wire is under the complete control of an attacker. So if I only want to expose a single 1MB buffer consisting of 256 scattered pages I should be able to do so, because I have designed my daemon so that it will not be damaged no matter what the remote peer puts in that 1MB. They can damage their content, but that's all. From ftillier at infiniconsys.com Wed May 4 12:17:17 2005 From: ftillier at infiniconsys.com (Fab Tillier) Date: Wed, 4 May 2005 12:17:17 -0700 Subject: [openib-general] FMR and how they work In-Reply-To: <469958e005050412013a51f5bc@mail.gmail.com> Message-ID: <000401c550dd$e40cb640$8d5aa8c0@infiniconsys.com> > From: Caitlin Bestler [mailto:caitlin.bestler at gmail.com] > Sent: Wednesday, May 04, 2005 12:02 PM > > Generally I assume the entity on the other end of the wire is > under the complete control of an attacker. So if I only want > to expose a single 1MB buffer consisting of 256 scattered > pages I should be able to do so, because I have designed > my daemon so that it will not be damaged no matter what > the remote peer puts in that 1MB. They can damage their > content, but that's all. In this case Mellanox FMRs don't help you since the mapping is still potentially valid until the local system invalidates the whole TPT cache in the HCA. Until the TPT cache is flushed, the remote entity can continue to RDMA into a previous region causing who knows what damage. The memory from that previous I/O might have been re-assigned for some other purpose. Invalidating this cache after every I/O defeats the advantage of FMRs. Mellanox FMRs require a level of trust for the performance advantage to be achieved. If trust is out of the question, FMRs should be too and we're back to using regular MRs. - Fab From mshefty at ichips.intel.com Wed May 4 12:33:50 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Wed, 04 May 2005 12:33:50 -0700 Subject: [openib-general] Re: RMPP In-Reply-To: <1115224329.4476.56.camel@localhost.localdomain> References: <1115224329.4476.56.camel@localhost.localdomain> Message-ID: <4279239E.60401@ichips.intel.com> Hal Rosenstock wrote: > In addition to passing the hdr_len and data_len to ib_create_send_mad: > rmpp_mad->rmpp_hdr.paylen_newwin = cpu_to_be32(hdr_len - > offsetof(struct ib_rmpp_mad, data) + data_len); > That's not the "real" payload length that would go into the packet; just > one segment's worth of class header. Correct ? > > If the above is correct, I'm not sure the actual payload lengths in the > DATA packets are correct. I also see the padding on the last segment of > a multipacket send not cleared (I integrated the part of your patch > relating to the pad calculation). Running grmpp and using madeye to check the packets, I ran 3 tests: user-data size payload 1st in segment payload in last segment 1200 1320 (220x6) 124 50 54 54 580 660 (220x3) 152 For multi-packet segments, I think that the payload in the 1st segment will always be a multiple of 220 bytes (256 - sizeof common MAD header - sizeof RMPP header). For the tests that I ran, these appear to be correct. Since grmpp uses vendor MADs, it can transfer 216 bytes of user-data per segment. Using the 580 example, the data is segmented into 216 + 216 + 148 bytes. The payload in the last segment includes the extra 4 byte vendor specific header. Likewise, 1200 = 216 x 5 + 120, with an extra 4 bytes added for the header. I haven't looked at why the data isn't cleared yet, but will do so next. The memory allocations in create_send_mad look right to me... - Sean From steve at wooding.uklinux.net Wed May 4 13:10:59 2005 From: steve at wooding.uklinux.net (Steven Wooding) Date: Wed, 04 May 2005 21:10:59 +0100 Subject: [openib-general] [PATCH][SDP] AIO buffer corruption In-Reply-To: <20050503130243.C21837@topspin.com> References: <200505021548.52053.steve@wooding.uklinux.net> <20050503130243.C21837@topspin.com> Message-ID: <42792C53.1020304@wooding.uklinux.net> Libor Michalek wrote: >On Mon, May 02, 2005 at 03:48:51PM +0100, Steven Wooding wrote: > > >>Hello Libor, >> >>I've tried your patch, but unfortunately it made no difference to >>the -32/-104 errors I get. I have observed the following features, >>which may help you diagnose my problem: >> >>Platform: 64 bit >>OS: RHEL 4 >>Kernel: 2.6.11.6 >>OpenIB gen2: 2225 >>SM: IO5000 switch >> >> > >Steve, > > I'm not seeing this issue on my x86_64 systems. Is there anything >in the syslog on either system? If you build SDP with debug, there >will be messages in the log, any errors? Which HCA are you using, and >which firmware? Also, can you send the exact command line? > >-Libor > > > > Hi Libor, Actually your patch did make a difference (thanks). After playing around with ttcp.aio a bit more I now find that long runs now work fine if the message size is larger than the zero copy threshold. I'll get back to you tomorrow about the details of my setup. I did turn debugging on, but did not see any different messages when the -32/-104 error occurred, now for message sizes smaller than the zero threshold. Cheers, Steve. From eitan at mellanox.co.il Wed May 4 13:29:30 2005 From: eitan at mellanox.co.il (Eitan Zahavi) Date: Wed, 4 May 2005 23:29:30 +0300 Subject: [openib-general] OpenSM work Q1 + Apr. Message-ID: <506C3D7B14CDD411A52C00025558DED6047EF1FA@mtlex01.yok.mtl.com> As was discussed more then once in several forums, my work on gen1 is ported by Hal and Shahar to gen2. Actually, the code can be directly copied over except for the "vendor layer" which are the osm_vendoc_mlx_* files. EZ Eitan Zahavi Design Technology Director Mellanox Technologies LTD Tel:+972-4-9097208 Fax:+972-4-9593245 P.O. Box 586 Yokneam 20692 ISRAEL > -----Original Message----- > From: Tom Duffy [mailto:tduffy at sun.com] > Sent: Wednesday, May 04, 2005 9:42 PM > To: Eitan Zahavi > Cc: 'openib-general at openib.org' > Subject: RE: [openib-general] OpenSM work Q1 + Apr. > > On Wed, 2005-05-04 at 20:51 +0300, Eitan Zahavi wrote: > > > I have just uploaded to OpenIB > > https://openib.org/svn/gen1/trunk/src/userspace/osm > > > > a new change set (2255) with OpenSM improvements. > > I am confused. Why did you check into gen1? > > Weren't we going to move gen1 to an archive section and make it read > only? > > -tduffy -------------- next part -------------- An HTML attachment was scrubbed... URL: From eitan at mellanox.co.il Wed May 4 13:34:30 2005 From: eitan at mellanox.co.il (Eitan Zahavi) Date: Wed, 4 May 2005 23:34:30 +0300 Subject: [openib-general] OpenSM work Q1 + Apr. Message-ID: <506C3D7B14CDD411A52C00025558DED6047EF1FC@mtlex01.yok.mtl.com> > > Also, does this translate to an IBGD release (number) ? [EZ] It will - next release. > > Did this release pick up any of the fixes from OpenIB (gen2) ? > [EZ] Some of them. Not all. > -- Hal -------------- next part -------------- An HTML attachment was scrubbed... URL: From tduffy at sun.com Wed May 4 13:37:49 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 04 May 2005 13:37:49 -0700 Subject: [openib-general] OpenSM work Q1 + Apr. In-Reply-To: <506C3D7B14CDD411A52C00025558DED6047EF1FA@mtlex01.yok.mtl.com> References: <506C3D7B14CDD411A52C00025558DED6047EF1FA@mtlex01.yok.mtl.com> Message-ID: <1115239069.8231.2.camel@duffman> On Wed, 2005-05-04 at 23:29 +0300, Eitan Zahavi wrote: > As was discussed more then once in several forums, my work on gen1 is > ported by Hal and Shahar to gen2. Actually, the code can be directly > copied over except for the "vendor layer" which are the > osm_vendoc_mlx_* files. Sorry if you had already answered. In any event, I thought we were locking the gen1 tree to no further development and sticking it in an archive section. There is some confusion out there with the state of the tree -- which part should people use, what version of the kernel do we support, etc. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From eitan at mellanox.co.il Wed May 4 13:38:24 2005 From: eitan at mellanox.co.il (Eitan Zahavi) Date: Wed, 4 May 2005 23:38:24 +0300 Subject: [openib-general] OpenSM work Q1 + Apr. Message-ID: <506C3D7B14CDD411A52C00025558DED6047EF1FD@mtlex01.yok.mtl.com> > On Wed, 2005-05-04 at 13:51, Eitan Zahavi wrote: > > 1. Irresponsive port scanning during light sweep - Allow the SM > > to recognize ports that were not responding in the first sweep > > (but had a link state that is not down). > > This sounds like the bad port handling algorithm we bashed on this list. > Is it ? [EZ] It is solving exactly the opposite phenomenon: Your proposal was to solve a case (I still did not see a log showing it is really happening) where OpenSM is continuously polling a non responsive port. While this solution is for the reverse problem where OpenSM totally ignores an Irresponsive Port after the first sweep is done - not giving it a chance to join the fabric even if it started to respond later. > > -- Hal -------------- next part -------------- An HTML attachment was scrubbed... URL: From halr at voltaire.com Wed May 4 13:32:08 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 04 May 2005 16:32:08 -0400 Subject: [openib-general] OpenSM work Q1 + Apr. In-Reply-To: <506C3D7B14CDD411A52C00025558DED6047EF1FA@mtlex01.yok.mtl.com> References: <506C3D7B14CDD411A52C00025558DED6047EF1FA@mtlex01.yok.mtl.com> Message-ID: <1115238708.4476.30.camel@localhost.localdomain> On Wed, 2005-05-04 at 16:29, Eitan Zahavi wrote: > As was discussed more then once in several forums, my work on gen1 is > ported by Hal and Shahar to gen2. Actually, the code can be directly > copied over except for the "vendor layer" which are the > osm_vendoc_mlx_* files. Actually I don't think so for a number of reasons: 1. copyright is different 2. I am not sure you have merged changes from gen2 OpenSM back into the gen1 OpenSM. -- Hal From eitan at mellanox.co.il Wed May 4 13:40:55 2005 From: eitan at mellanox.co.il (Eitan Zahavi) Date: Wed, 4 May 2005 23:40:55 +0300 Subject: [openib-general] OpenSM work Q1 + Apr. Message-ID: <506C3D7B14CDD411A52C00025558DED607C3010B@mtlex01.yok.mtl.com> I was asked to push in OpenSM changes as soon they become stable enough to be used. This is what I'm doing. Once the gen2 stabilize and support RMPP etc, I will probably start working on it. But until then, gen1 is my driver. Where should I post the code if gen1 branch will close? Eitan Zahavi Design Technology Director Mellanox Technologies LTD Tel:+972-4-9097208 Fax:+972-4-9593245 P.O. Box 586 Yokneam 20692 ISRAEL > -----Original Message----- > From: Tom Duffy [mailto:tduffy at sun.com] > Sent: Wednesday, May 04, 2005 11:38 PM > To: Eitan Zahavi > Cc: 'openib-general at openib.org' > Subject: RE: [openib-general] OpenSM work Q1 + Apr. > > On Wed, 2005-05-04 at 23:29 +0300, Eitan Zahavi wrote: > > As was discussed more then once in several forums, my work on gen1 is > > ported by Hal and Shahar to gen2. Actually, the code can be directly > > copied over except for the "vendor layer" which are the > > osm_vendoc_mlx_* files. > > Sorry if you had already answered. > > In any event, I thought we were locking the gen1 tree to no further > development and sticking it in an archive section. There is some > confusion out there with the state of the tree -- which part should > people use, what version of the kernel do we support, etc. > > -tduffy -------------- next part -------------- An HTML attachment was scrubbed... URL: From eitan at mellanox.co.il Wed May 4 13:42:48 2005 From: eitan at mellanox.co.il (Eitan Zahavi) Date: Wed, 4 May 2005 23:42:48 +0300 Subject: [openib-general] OpenSM work Q1 + Apr. Message-ID: <506C3D7B14CDD411A52C00025558DED607C3010C@mtlex01.yok.mtl.com> This is true - a simple copy will not do. But maybe a merge. > > 1. copyright is different > > 2. I am not sure you have merged changes from gen2 OpenSM back into the > gen1 OpenSM. > > -- Hal -------------- next part -------------- An HTML attachment was scrubbed... URL: From mshefty at ichips.intel.com Wed May 4 13:45:42 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Wed, 04 May 2005 13:45:42 -0700 Subject: [openib-general] Re: RMPP In-Reply-To: <1115224329.4476.56.camel@localhost.localdomain> References: <1115224329.4476.56.camel@localhost.localdomain> Message-ID: <42793476.7060704@ichips.intel.com> Hal Rosenstock wrote: > I also see the padding on the last segment of > a multipacket send not cleared (I integrated the part of your patch > relating to the pad calculation). I ran some tests, and didn't see any cases where the padding wasn't zero. The RMPP code doesn't touch the padding itself, and create_send should allocate it zeroed. Are you using an analyzer and seeing that it's not zeroed? - Sean From tduffy at sun.com Wed May 4 13:46:50 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 04 May 2005 13:46:50 -0700 Subject: [openib-general] OpenSM work Q1 + Apr. In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C3010B@mtlex01.yok.mtl.com> References: <506C3D7B14CDD411A52C00025558DED607C3010B@mtlex01.yok.mtl.com> Message-ID: <1115239610.8231.10.camel@duffman> On Wed, 2005-05-04 at 23:40 +0300, Eitan Zahavi wrote: > I was asked to push in OpenSM changes as soon they become stable > enough to be used. > This is what I'm doing. And we thank you very much for this -- please don't take my questions as criticizing your effort. > Once the gen2 stabilize and support RMPP etc, I will probably start > working on it. Good. I am glad to hear that. > But until then, gen1 is my driver. Fair enough. > Where should I post the code if gen1 branch will close? Two options: 1) in a users/eitan directory 2) somewhere in gen2 with the explicit warning that it won't currently compile under gen2 -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From tduffy at sun.com Wed May 4 13:48:17 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 04 May 2005 13:48:17 -0700 Subject: [openib-general] OpenSM work Q1 + Apr. In-Reply-To: <1115238708.4476.30.camel@localhost.localdomain> References: <506C3D7B14CDD411A52C00025558DED6047EF1FA@mtlex01.yok.mtl.com> <1115238708.4476.30.camel@localhost.localdomain> Message-ID: <1115239697.8231.13.camel@duffman> On Wed, 2005-05-04 at 16:32 -0400, Hal Rosenstock wrote: > 2. I am not sure you have merged changes from gen2 OpenSM back into the > gen1 OpenSM. So, the gen2 sm *was* a branch of the gen1 sm, no? If so, you should let the tools do the work for you and merge the changes over... -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From halr at voltaire.com Wed May 4 13:43:18 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 04 May 2005 16:43:18 -0400 Subject: [openib-general] Re: RMPP In-Reply-To: <42793476.7060704@ichips.intel.com> References: <1115224329.4476.56.camel@localhost.localdomain> <42793476.7060704@ichips.intel.com> Message-ID: <1115239398.4476.45.camel@localhost.localdomain> On Wed, 2005-05-04 at 16:45, Sean Hefty wrote: > Hal Rosenstock wrote: > > I also see the padding on the last segment of > > a multipacket send not cleared (I integrated the part of your patch > > relating to the pad calculation). > > I ran some tests, and didn't see any cases where the padding wasn't zero. > The RMPP code doesn't touch the padding itself, and create_send should > allocate it zeroed. Are you using an analyzer and seeing that it's not zeroed? Yes. I stating this from what I see on the IB "wire". -- Hal From halr at voltaire.com Wed May 4 13:46:34 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 04 May 2005 16:46:34 -0400 Subject: [openib-general] OpenSM work Q1 + Apr. In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C3010B@mtlex01.yok.mtl.com> References: <506C3D7B14CDD411A52C00025558DED607C3010B@mtlex01.yok.mtl.com> Message-ID: <1115239589.4476.52.camel@localhost.localdomain> On Wed, 2005-05-04 at 16:40, Eitan Zahavi wrote: > I was asked to push in OpenSM changes as soon they become stable > enough to be used. > This is what I'm doing. > Once the gen2 stabilize Not sure what your criteria to determine stability are > and support RMPP etc, RMPP is getting close now > I will probably start working on it. > But until then, gen1 is my driver. > Where should I post the code if gen1 branch will close? Something like https://openib.org/svn/gen2/branches/eitan-opensm/ -- Hal > > Eitan Zahavi > Design Technology Director > Mellanox Technologies LTD > Tel:+972-4-9097208 > Fax:+972-4-9593245 > P.O. Box 586 Yokneam 20692 ISRAEL > > > > -----Original Message----- > > From: Tom Duffy [mailto:tduffy at sun.com] > > Sent: Wednesday, May 04, 2005 11:38 PM > > To: Eitan Zahavi > > Cc: 'openib-general at openib.org' > > Subject: RE: [openib-general] OpenSM work Q1 + Apr. > > > > On Wed, 2005-05-04 at 23:29 +0300, Eitan Zahavi wrote: > > > As was discussed more then once in several forums, my work on gen1 > is > > > ported by Hal and Shahar to gen2. Actually, the code can be > directly > > > copied over except for the "vendor layer" which are the > > > osm_vendoc_mlx_* files. > > > > Sorry if you had already answered. > > > > In any event, I thought we were locking the gen1 tree to no further > > development and sticking it in an archive section. There is some > > confusion out there with the state of the tree -- which part should > > people use, what version of the kernel do we support, etc. > > > > -tduffy > > > > ______________________________________________________________________ > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From mshefty at ichips.intel.com Wed May 4 14:02:16 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Wed, 04 May 2005 14:02:16 -0700 Subject: [openib-general] Re: RMPP In-Reply-To: <1115239398.4476.45.camel@localhost.localdomain> References: <1115224329.4476.56.camel@localhost.localdomain> <42793476.7060704@ichips.intel.com> <1115239398.4476.45.camel@localhost.localdomain> Message-ID: <42793858.7080105@ichips.intel.com> Hal Rosenstock wrote: > On Wed, 2005-05-04 at 16:45, Sean Hefty wrote: > >>Hal Rosenstock wrote: >> >>>I also see the padding on the last segment of >>>a multipacket send not cleared (I integrated the part of your patch >>>relating to the pad calculation). >> >>I ran some tests, and didn't see any cases where the padding wasn't zero. >>The RMPP code doesn't touch the padding itself, and create_send should >>allocate it zeroed. Are you using an analyzer and seeing that it's not zeroed? > > > Yes. I stating this from what I see on the IB "wire". I think my tests were just lucky... It looks like create_send_mad zeros only the top portion of the data buffer. - Sean From krause at cup.hp.com Wed May 4 13:59:55 2005 From: krause at cup.hp.com (Michael Krause) Date: Wed, 04 May 2005 13:59:55 -0700 Subject: [openib-general] FMR and how they work In-Reply-To: <000201c550d7$6c7b0740$8d5aa8c0@infiniconsys.com> References: <469958e00505041117429ac4c2@mail.gmail.com> <000201c550d7$6c7b0740$8d5aa8c0@infiniconsys.com> Message-ID: <6.2.0.14.2.20050504135744.02782c80@esmail.cup.hp.com> At 11:31 AM 5/4/2005, Fab Tillier wrote: > > From: Caitlin Bestler [mailto:caitlin.bestler at gmail.com] > > Sent: Wednesday, May 04, 2005 11:18 AM > > > > On 5/4/05, Fab Tillier wrote: > > > > From: Dan Bar Dov [mailto:danb at voltaire.com] > > > > Sent: Wednesday, May 04, 2005 1:00 AM > > > > > > > > Voltair's IB based ISER indeed uses kDAPL so it will be very simple to > > > > atapt it to iWARP. > > > > > > > > Voltaire's kDAPL supports FMR using PLATFORM registration type, and it > > > > is used by the ISER implementation. > > > > > > > > > > Why doesn't iSER just use a single un-translated MR for all of physical > > > memory (like the other kernel ULPs)? It would then just call the DMA > > > mapping functions and be done with it - no registration at all is going > > > to be faster than FMRs. > > > > > > > > > An FMR can be used to advertise a target buffer that is being read into > > as a single logical buffer. Using physical memory would require > > > > a) exporting the physical page list of where your buffers were (making > > the buffer advertisement larger and more complex) > > > > b) Trusting whoever is on the other end of the connection with access > > to your entire physical memory. > >I assume that iSER (like SRP) supports scatter gather in I/O requests, so a) >shouldn't matter - that is, I expect that the storage subsystem hands the >iSER driver a list of physical addresses and a SCSI-like command. Copying >the page list is far simpler than getting a virtual address. > >I see no issue with trusting I/O controllers on the fabric. They are no >more a threat than local I/O controllers. This addresses issue b). In the future world of virtualized environments, I can assure that we trust no device or fabric even for local I/O. Fabrics like IB are no different in this regard and are subject to more types of errors / attacks. As such, people need to agree on what is the scope of a given trust domain and determine whether the proposed solution is sufficient or not. What has been discussed here may be fine for some environments but I can think of enterprise environments where it would not be trusted to the same extent. Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From halr at voltaire.com Wed May 4 14:01:50 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 04 May 2005 17:01:50 -0400 Subject: [openib-general] OpenSM work Q1 + Apr. In-Reply-To: <1115239697.8231.13.camel@duffman> References: <506C3D7B14CDD411A52C00025558DED6047EF1FA@mtlex01.yok.mtl.com> <1115238708.4476.30.camel@localhost.localdomain> <1115239697.8231.13.camel@duffman> Message-ID: <1115240509.4476.84.camel@localhost.localdomain> On Wed, 2005-05-04 at 16:48, Tom Duffy wrote: > On Wed, 2005-05-04 at 16:32 -0400, Hal Rosenstock wrote: > > 2. I am not sure you have merged changes from gen2 OpenSM back into the > > gen1 OpenSM. > > So, the gen2 sm *was* a branch of the gen1 sm, no? If so, you should > let the tools do the work for you and merge the changes over... I do not believe it was started that way (or at one point it may have but was then based on an IBGD release after that) but the tools can still help with this. The main issue is the testing. -- Hal From sean.hefty at intel.com Wed May 4 14:51:41 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Wed, 4 May 2005 14:51:41 -0700 Subject: [openib-general] Re: RMPP In-Reply-To: <1115239398.4476.45.camel@localhost.localdomain> Message-ID: >> > I also see the padding on the last segment of >> > a multipacket send not cleared (I integrated the part of your patch >> > relating to the pad calculation). >> >> I ran some tests, and didn't see any cases where the padding wasn't zero. >> The RMPP code doesn't touch the padding itself, and create_send should >> allocate it zeroed. Are you using an analyzer and seeing that it's not zeroed? > >Yes. I stating this from what I see on the IB "wire". Can you try with this patch? Signed-off-by: Sean Hefty Index: core/mad.c =================================================================== --- core/mad.c (revision 2256) +++ core/mad.c (working copy) @@ -796,9 +796,9 @@ buf = kmalloc(sizeof *send_buf + buf_size, gfp_mask); if (!buf) return ERR_PTR(-ENOMEM); + memset(buf, 0, sizeof *send_buf + buf_size); send_buf = buf + buf_size; - memset(send_buf, 0, sizeof *send_buf); send_buf->mad = buf; send_buf->sge.addr = dma_map_single(mad_agent->device->dma_device, From tduffy at sun.com Wed May 4 14:59:27 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 04 May 2005 14:59:27 -0700 Subject: [openib-general] [kDAPL][PATCH] fix panic in server side Message-ID: <1115243969.14238.6.camel@duffman> This occurred when I did a connect without first setting up a listen using kdapltest. Unable to handle kernel NULL pointer dereference at 0000000000000020 RIP: {:ib_dat_provider:dapl_cm_passive_cb_handler+27} PGD 371ab067 PUD 36822067 PMD 0 Oops: 0000 [1] SMP CPU 2 Modules linked in: kdapltest ib_dat_provider dat ib_at ib_cm ib_ipoib ib_sa ib_umad md5 ipv6 parport_pc lp parport autofs4 nfs lockd sunrpc pcmcia yenta_socket rsrc_nonstatic pcmcia_core ext3 jbd video container button battery ac uhci_hcd ehci_hcd hw_random i2c_i801 i2c_core ib_mthca ib_mad ib_core e1000 floppy dm_snapshot dm_zero dm_mirror xfs exportfs dm_mod mptscsih mptbase sd_mod scsi_mod Pid: 11326, comm: ib_cm/2 Not tainted 2.6.12-rc3openib RIP: 0010:[] {:ib_dat_provider:dapl_cm_passive_cb_handler+27} RSP: 0018:ffff810038f65d90 EFLAGS: 00010202 RAX: 0000000000000000 RBX: ffff81000b3764b0 RCX: ffff81000b376550 RDX: 0000000000000056 RSI: ffff81000b376548 RDI: ffff81003f0cd8a0 RBP: ffff81003f0cd8a0 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000020 R11: ffff81003925ae10 R12: ffff81003f0cd8a0 R13: ffff81003dcb8a88 R14: 0000000000000000 R15: ffff81000b3765a0 FS: 0000000000000000(0000) GS:ffffffff80487a00(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 0000000000000020 CR3: 0000000038bbd000 CR4: 00000000000006e0 Process ib_cm/2 (pid: 11326, threadinfo ffff810038f64000, task ffff81003e75af30)Stack: ffffffff882aae32 0100000000000293 0000000000000000 ffff81000b3764b0 000000003f0cd8a0 ffff81003dcb8a88 0000000000000000 ffff81000b3765a0 ffffffff882abd1e ffff810001e1b680 Call Trace:{:ib_cm:cm_process_work+50} {:ib_cm:cm_work_handler+1438} {do_sigaction+520} {:ib_cm:cm_work_handler+0} {worker_thread+476} {default_wake_function+0} {__wake_up_common+67} {worker_thread+0} {keventd_create_kthread+0} {kthread+217} {schedule_tail+64} {child_rip+8} {keventd_create_kthread+0} {kthread+0} {child_rip+0} Code: 8b 40 20 89 44 24 04 83 7c 24 04 03 74 30 83 7c 24 04 03 77 RIP {:ib_dat_provider:dapl_cm_passive_cb_handler+27} RSP CR2: 0000000000000020 This cuteness happened in the kdapltest userland program: Warning: conn_event_wait DAT_CONNECTION_EVENT_DISCONNECTED DT_cs_Client: bad connection event DT_cs_Client: Cleaning Up ... DT_cs_Client: IA mthca0a closed DT_cs_Client: ========== End of Work -- Client Exiting TEST INSTANCE 1 *** glibc detected *** free(): invalid next size (fast): 0x000000000050ca50 *** Aborted (core dumped) This patches fixes two things. It makes it so that dapl_cm_passive_cb_handler won't dereference a potentially NULL ib_cm_id and also that it returns ret instead of 0 all the time. Signed-off-by: Tom Duffy Index: dapl_openib_cm.c =================================================================== --- linux-kernel/dat-provider/dapl_openib_cm.c (revision 2257) +++ linux-kernel/dat-provider/dapl_openib_cm.c (working copy) @@ -227,6 +227,9 @@ int dapl_cm_passive_cb_handler(struct ib { int ret = 0; + if (!comm_id) + return -1; + switch (comm_id->state) { case IB_CM_IDLE: ret = do_passive_idle(comm_id); @@ -247,7 +250,7 @@ int dapl_cm_passive_cb_handler(struct ib break; } - return 0; + return ret; } static void dapl_ib_destroy_cm_id_work(void *data) From halr at voltaire.com Wed May 4 15:01:07 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 04 May 2005 18:01:07 -0400 Subject: [openib-general] [kDAPL][PATCH] fix panic in server side In-Reply-To: <1115243969.14238.6.camel@duffman> References: <1115243969.14238.6.camel@duffman> Message-ID: <1115244066.4633.2.camel@localhost.localdomain> On Wed, 2005-05-04 at 17:59, Tom Duffy wrote: > This patches fixes two things. It makes it so that > dapl_cm_passive_cb_handler won't dereference a potentially NULL ib_cm_id > and also that it returns ret instead of 0 all the time. > > Signed-off-by: Tom Duffy The second part is not what is intended. A non 0 return in a CM callback means to destroy the CM ID. That would not be a good thing... -- Hal From halr at voltaire.com Wed May 4 15:14:26 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 04 May 2005 18:14:26 -0400 Subject: [openib-general] [kDAPL][PATCH] fix panic in server side In-Reply-To: <1115243969.14238.6.camel@duffman> References: <1115243969.14238.6.camel@duffman> Message-ID: <1115244747.4633.7.camel@localhost.localdomain> On Wed, 2005-05-04 at 17:59, Tom Duffy wrote: > Signed-off-by: Tom Duffy > > Index: dapl_openib_cm.c > =================================================================== > --- linux-kernel/dat-provider/dapl_openib_cm.c (revision 2257) > +++ linux-kernel/dat-provider/dapl_openib_cm.c (working copy) > @@ -227,6 +227,9 @@ int dapl_cm_passive_cb_handler(struct ib > { > int ret = 0; > > + if (!comm_id) > + return -1; > + I also think this should be: if (!comm_id) return 0; but I'm not sure why the CM would callback with a NULL comm_id. Also, I wonder if the same thing could occur on the active side too. -- Hal From mshefty at ichips.intel.com Wed May 4 15:27:56 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Wed, 04 May 2005 15:27:56 -0700 Subject: [openib-general] [kDAPL][PATCH] fix panic in server side In-Reply-To: <1115244747.4633.7.camel@localhost.localdomain> References: <1115243969.14238.6.camel@duffman> <1115244747.4633.7.camel@localhost.localdomain> Message-ID: <42794C6C.9000103@ichips.intel.com> Hal Rosenstock wrote: > On Wed, 2005-05-04 at 17:59, Tom Duffy wrote: > >>Signed-off-by: Tom Duffy >> >>Index: dapl_openib_cm.c >>=================================================================== >>--- linux-kernel/dat-provider/dapl_openib_cm.c (revision 2257) >>+++ linux-kernel/dat-provider/dapl_openib_cm.c (working copy) >>@@ -227,6 +227,9 @@ int dapl_cm_passive_cb_handler(struct ib >> { >> int ret = 0; >> >>+ if (!comm_id) >>+ return -1; >>+ > > > I also think this should be: > if (!comm_id) > return 0; > > but I'm not sure why the CM would callback with a NULL comm_id. > > Also, I wonder if the same thing could occur on the active side too. The CM shouldn't invoke a callback with a NULL cm_id. The actual calls look like this: ret = cm_id_priv->id.cm_handler(&cm_id_priv->id, &work->cm_event); - Sean From mshefty at ichips.intel.com Wed May 4 15:35:18 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Wed, 04 May 2005 15:35:18 -0700 Subject: [openib-general] [kDAPL][PATCH] fix panic in server side In-Reply-To: <1115243969.14238.6.camel@duffman> References: <1115243969.14238.6.camel@duffman> Message-ID: <42794E26.6080205@ichips.intel.com> Tom Duffy wrote: > Index: dapl_openib_cm.c > =================================================================== > --- linux-kernel/dat-provider/dapl_openib_cm.c (revision 2257) > +++ linux-kernel/dat-provider/dapl_openib_cm.c (working copy) > @@ -227,6 +227,9 @@ int dapl_cm_passive_cb_handler(struct ib > { > int ret = 0; > > + if (!comm_id) > + return -1; > + > switch (comm_id->state) { > case IB_CM_IDLE: Separate issue here... The state of the cm_id was exposed more for debugging purposes than for use by the consumer. The state is protected by locks within the CM and can change at anytime, depending on which messages are being processed. What should be checked here is the event that's being reported. There could be multiple callbacks invoked in succession, all with the cm_id state set to idle. - Sean From robert.j.woodruff at intel.com Wed May 4 16:11:09 2005 From: robert.j.woodruff at intel.com (Woodruff, Robert J) Date: Wed, 4 May 2005 16:11:09 -0700 Subject: [openib-general] [OOPS] user-mode verbs Message-ID: <1AC79F16F5C5284499BB9591B33D6F00044D66C7@orsmsx408> Hi Roland, I have started to stress test the user-mode verbs code using uDAPL and Intel MPI running multiple copies of the Pallas MPI benchmark simultaniously. This is on a RedHat EL4.0 2.6.9ELsmp kernel, SVN 2245 backported to the RedHat kernel, EM64T 2.4 Ghz Xeon boxes PCI-E HCAs Note that last night I ran the code on a straight 2.6.9 kernel from kernel.org with the infiniband backport, but in UP mode and it ran all night without any issues. woody Here are two separate oops traces. general protection fault: 0000 [1] SMP CPU 0 Modules linked in: nfsd(U) exportfs(U) lockd(U) det(U) ib_uverbs(U) ib_sdp(U) ib_cm(U) ib_ipoib(U) ib_sa(U) md5(U) ipv6(U) parport_pc(U) lp(U) parport(U) autofs4(U) i2c_dev(U) i2c_core(U) sunrpc(U) dm_mod(U) button(U) battery(U) ac(U) uhci_hcd(U) ehci_hcd(U) hw_random(U) ib_mthca(U) ib_mad(U) ib_core(U) e1000(U) floppy(U) ext3(U) jbd(U) Pid: 4942, comm: PMB-MPI1 Tainted: P 2.6.9-prep RIP: 0010:[] {:ib_uverbs:ib_uverbs_destroy_qp+187} RSP: 0018:000001012ea45ea8 EFLAGS: 00010046 RAX: 0000000000000010 RBX: 0000010131678400 RCX: 000001000000e000 RDX: 37de000000000000 RSI: 37de000000000010 RDI: 00000101382ddf38 RBP: 000001013218ca80 R08: 000001012eca5668 R09: 0000000000000246 R10: 000000000000ea60 R11: 0000000000000246 R12: ffffffffa01af100 R13: 0000000000000000 R14: 000000000000000c R15: 0000000000000000 FS: 0000002a959015a0(0000) GS:ffffffff804bf580(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000002a95b4f002 CR3: 0000000000101000 CR4: 00000000000006e0 Process PMB-MPI1 (pid: 4942, threadinfo 000001012ea44000, task 00000101303a37f0) Stack: 000000030000001d 000000000000001d 000000000000000c 0000007fbffff190 000001013218ca80 000000000000000c 0000007fbffff190 ffffffffa01a957d 0000000000000000 000000030000001d Call Trace:{:ib_uverbs:ib_uverbs_write+137} {vfs_write+207} {sys_write+69} {system_call+126} Code: 48 8b 42 10 48 8b 4e 08 48 89 48 08 48 89 01 48 c7 46 08 00 RIP {:ib_uverbs:ib_uverbs_destroy_qp+187} RSP <000001012ea45ea8> And the second one, errfatal[7715]: segfault at 0000000000000008 rip 0000002a955c9e30 rsp 0000007fbffff530 error 4 Unable to handle kernel NULL pointer dereference at 0000000000000028 RIP: {:ib_uverbs:__ib_umem_unmark+5} PML4 12d90f067 PGD 12f247067 PMD 0 Oops: 0000 [1] SMP CPU 0 Modules linked in: nfsd(U) exportfs(U) lockd(U) det(U) ib_uverbs(U) ib_sdp(U) ib_cm(U) ib_ipoib(U) ib_sa(U) md5(U) ipv6(U) parport_pc(U) lp(U) parport(U) autofs4(U) i2c_dev(U) i2c_core(U) sunrpc(U) dm_mod(U) button(U) battery(U) ac(U) uhci_hcd(U) ehci_hcd(U) hw_random(U) ib_mthca(U) ib_mad(U) ib_core(U) e1000(U) floppy(U) ext3(U) jbd(U) Pid: 11522, comm: PMB-MPI1 Tainted: P 2.6.9-prep RIP: 0010:[] {:ib_uverbs:__ib_umem_unmark+5} RSP: 0018:000001012397be68 EFLAGS: 00010216 RAX: 000001013db82940 RBX: 000001013db82998 RCX: 000001012b3abd10 RDX: 0000000000000000 RSI: 000001013db82940 RDI: 0000000000000028 RBP: 000001013db82940 R08: 000001012bbda2c8 R09: 000000000000000a R10: 000000000000ea60 R11: 000001012397be6f R12: 0000000000000028 R13: 000001013d9a2000 R14: 000000000000000c R15: 0000000000000018 FS: 0000002a959015a0(0000) GS:ffffffff804bf580(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000000000028 CR3: 0000000000101000 CR4: 00000000000006e0 Process PMB-MPI1 (pid: 11522, threadinfo 000001012397a000, task 00000101212df7f0) Stack: 000001012bbda2c8 ffffffffa01ab6a1 000001013fca98c0 0000000000000000 000001013fca98c0 ffffffffa01af100 0000000000000000 ffffffffa01aa747 000001013fca98c0 000000d03fca98c0 Call Trace:{:ib_uverbs:ib_umem_release+77} {:ib_uverbs:ib_uverbs_dereg_mr+263} {:ib_uverbs:ib_uverbs_write+137} {vfs_write+207} {sys_write+69} {system_call+126} Code: 48 8b 37 48 89 c7 e8 6b 8b fb df 5e c3 41 57 49 89 d7 41 56 RIP {:ib_uverbs:__ib_umem_unmark+5} RSP <000001012397be68> CR2: 0000000000000028 From halr at voltaire.com Wed May 4 16:12:49 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 04 May 2005 19:12:49 -0400 Subject: [openib-general] Re: RMPP In-Reply-To: References: Message-ID: <1115248106.4476.6.camel@localhost.localdomain> On Wed, 2005-05-04 at 17:51, Sean Hefty wrote: > Can you try with this patch? > > Signed-off-by: Sean Hefty > > Index: core/mad.c > =================================================================== > -- core/mad.c (revision 2256) > +++ core/mad.c (working copy) > @@ -796,9 +796,9 @@ > buf = kmalloc(sizeof *send_buf + buf_size, gfp_mask); > if (!buf) > return ERR_PTR(-ENOMEM); > + memset(buf, 0, sizeof *send_buf + buf_size); > > send_buf = buf + buf_size; > - memset(send_buf, 0, sizeof *send_buf); > send_buf->mad = buf; > > send_buf->sge.addr = dma_map_single(mad_agent->device->dma_device, Yes, that looks better in terms of clearing the padding. I still need to double check my math on the PayloadLengths. Thanks. -- Hal From mshefty at ichips.intel.com Wed May 4 16:26:32 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Wed, 04 May 2005 16:26:32 -0700 Subject: [openib-general] Re: RMPP In-Reply-To: <1115248106.4476.6.camel@localhost.localdomain> References: <1115248106.4476.6.camel@localhost.localdomain> Message-ID: <42795A28.1020600@ichips.intel.com> Hal Rosenstock wrote: > Yes, that looks better in terms of clearing the padding. I still need to > double check my math on the PayloadLengths. It turned out that grmpp was clearing the MAD, which was why I wasn't detecting the issue. Also, I've added some documentation on using RMPP to ib_mad.h and added an rmpp_active flag to ib_create_send_mad(). My current solution is to have ib_create_send_mad() format the RMPP header for the user. - Sean From sean.hefty at intel.com Wed May 4 17:05:06 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Wed, 4 May 2005 17:05:06 -0700 Subject: [openib-general] [PATCH] [MAD] changes to ib_create_send_mad Message-ID: The following patch adds an rmpp_active parameter to ib_create_send_mad, so it can properly format the RMPP header. This patch also contains the fix to clear the MAD data buffer. Additional comments were added regarding RMPP use. Signed-off-by: Sean Hefty Index: include/ib_mad.h =================================================================== --- include/ib_mad.h (revision 2256) +++ include/ib_mad.h (working copy) @@ -440,6 +440,16 @@ int ib_unregister_mad_agent(struct ib_ma * @bad_send_wr: Specifies the MAD on which an error was encountered. * * Sent MADs are not guaranteed to complete in the order that they were posted. + * + * If the MAD requires RMPP, the data buffer should contain a single copy + * of the common MAD, RMPP, and class specific headers, followed by the class + * defined data. If the class defined data would not divide evenly into + * RMPP segments, then space must be allocated at the end of the referenced + * buffer for any required padding. To indicate the amount of class defined + * data being transferred, the paylen_newwin field in the RMPP header should + * be set to the size of the class specific header plus the amount of class + * defined data being transferred. The paylen_newwin field should be + * specified in network-byte order. */ int ib_post_send_mad(struct ib_mad_agent *mad_agent, struct ib_send_wr *send_wr, @@ -452,7 +462,7 @@ int ib_post_send_mad(struct ib_mad_agent * referenced buffer should be at least the size of the mad_len specified * by @mad_recv_wc. * - * This call copies a chain of received RMPP MADs into a single data buffer, + * This call copies a chain of received MAD segments into a single data buffer, * removing duplicated headers. */ void ib_coalesce_recv_mad(struct ib_mad_recv_wc *mad_recv_wc, void *buf); @@ -525,6 +535,7 @@ int ib_process_mad_wc(struct ib_mad_agen * @pkey_index: Specifies which PKey the MAD will be sent using. This field * is valid only if the remote_qpn is QP 1. * @ah: References the address handle used to transfer to the remote node. + * @rmpp_active: Indicates if the send will enable RMPP. * @hdr_len: Indicates the size of the data header of the MAD. This length * should include the common MAD header, RMPP header, plus any class * specific header. @@ -536,11 +547,16 @@ int ib_process_mad_wc(struct ib_mad_agen * This is a helper routine that may be used to allocate a MAD. Users are * not required to allocate outbound MADs using this call. The returned * MAD send buffer will reference a data buffer usable for sending a MAD, along - * with an initialized work request structure. + * with an initialized work request structure. Users may modify the returned + * MAD data buffer or work request before posting the send. + * + * The returned data buffer will be cleared. Users are responsible for + * initializing the common MAD and any class specific headers. If @rmpp_active + * is set, the RMPP header will be initialized for sending. */ struct ib_mad_send_buf * ib_create_send_mad(struct ib_mad_agent *mad_agent, u32 remote_qpn, u16 pkey_index, - struct ib_ah *ah, + struct ib_ah *ah, int rmpp_active, int hdr_len, int data_len, int gfp_mask); Index: core/mad.c =================================================================== --- core/mad.c (revision 2256) +++ core/mad.c (working copy) @@ -780,7 +780,7 @@ static int get_buf_length(int hdr_len, i struct ib_mad_send_buf * ib_create_send_mad(struct ib_mad_agent *mad_agent, u32 remote_qpn, u16 pkey_index, - struct ib_ah *ah, + struct ib_ah *ah, int rmpp_active, int hdr_len, int data_len, int gfp_mask) { @@ -793,12 +793,17 @@ struct ib_mad_send_buf * ib_create_send_ struct ib_mad_agent_private, agent); buf_size = get_buf_length(hdr_len, data_len); + if ((!mad_agent->rmpp_version && + (rmpp_active || buf_size > sizeof(struct ib_mad))) || + (!rmpp_active && buf_size > sizeof(struct ib_mad))) + return ERR_PTR(-EINVAL); + buf = kmalloc(sizeof *send_buf + buf_size, gfp_mask); if (!buf) return ERR_PTR(-ENOMEM); + memset(buf, 0, sizeof *send_buf + buf_size); send_buf = buf + buf_size; - memset(send_buf, 0, sizeof *send_buf); send_buf->mad = buf; send_buf->sge.addr = dma_map_single(mad_agent->device->dma_device, @@ -818,11 +823,15 @@ struct ib_mad_send_buf * ib_create_send_ send_buf->send_wr.wr.ud.remote_qkey = IB_QP_SET_QKEY; send_buf->send_wr.wr.ud.pkey_index = pkey_index; - if (mad_agent->rmpp_version) { + if (rmpp_active) { struct ib_rmpp_mad *rmpp_mad; rmpp_mad = (struct ib_rmpp_mad *)send_buf->mad; rmpp_mad->rmpp_hdr.paylen_newwin = cpu_to_be32(hdr_len - offsetof(struct ib_rmpp_mad, data) + data_len); + rmpp_mad->rmpp_hdr.rmpp_version = mad_agent->rmpp_version; + rmpp_mad->rmpp_hdr.rmpp_type = IB_MGMT_RMPP_TYPE_DATA; + ib_set_rmpp_flags(&rmpp_mad->rmpp_hdr, + IB_MGMT_RMPP_FLAG_ACTIVE); } send_buf->mad_agent = mad_agent; From iod00d at hp.com Wed May 4 17:17:34 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 4 May 2005 17:17:34 -0700 Subject: [openib-general] PF 5/4 Telecon Minutes In-Reply-To: <001101c550ce$29ad8590$6801a8c0@ownerdnt868gwf> References: <001101c550ce$29ad8590$6801a8c0@ownerdnt868gwf> Message-ID: <20050505001734.GD16887@esmail.cup.hp.com> On Wed, May 04, 2005 at 01:24:37PM -0400, Hal Rosenstock wrote: > Administrivia > OLS paper almost complete Hal, Which paper are you referring to? I just checked: http://www.linuxsymposium.org/2005/speakers.php?types=TALK and didn't see about infiniband. Ah, found it under BOFs: http://www.linuxsymposium.org/2005/view_abstract.php?content_key=87 Plan on handing out printed copies or post a URL where it's to find (e.g. openib.org/talks/ols2005). thanks, grant From halr at voltaire.com Wed May 4 17:15:58 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 04 May 2005 20:15:58 -0400 Subject: [openib-general] PF 5/4 Telecon Minutes In-Reply-To: <20050505001734.GD16887@esmail.cup.hp.com> References: <001101c550ce$29ad8590$6801a8c0@ownerdnt868gwf> <20050505001734.GD16887@esmail.cup.hp.com> Message-ID: <1115252158.4476.120.camel@localhost.localdomain> On Wed, 2005-05-04 at 20:17, Grant Grundler wrote: > Which paper are you referring to? > I just checked: > http://www.linuxsymposium.org/2005/speakers.php?types=TALK > > and didn't see about infiniband. > Ah, found it under BOFs: > http://www.linuxsymposium.org/2005/view_abstract.php?content_key=87 > > Plan on handing out printed copies or post a URL where it's > to find (e.g. openib.org/talks/ols2005). It's Woody's paper which Sean, Roland, and I contributed to. He should speak to this. -- Hal From halr at voltaire.com Wed May 4 17:33:48 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 04 May 2005 20:33:48 -0400 Subject: [openib-general] [PATCH] [MAD] changes to ib_create_send_mad In-Reply-To: References: Message-ID: <1115253041.4476.147.camel@localhost.localdomain> On Wed, 2005-05-04 at 20:05, Sean Hefty wrote: > If the class defined data would not divide evenly into > + * RMPP segments, then space must be allocated at the end of the referenced > + * buffer for any required padding. This could be the source of the problem with PayloadLength. I need to see if the buffer meets the criteria and if not, how to make it do this. I will try this in the AM as well as your two related MAD patches. -- Hal From sean.hefty at intel.com Wed May 4 18:26:45 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Wed, 4 May 2005 18:26:45 -0700 Subject: [openib-general] [PATCH] [MAD] changes to ib_create_send_mad In-Reply-To: <1115253041.4476.147.camel@localhost.localdomain> Message-ID: >On Wed, 2005-05-04 at 20:05, Sean Hefty wrote: >> If the class defined data would not divide evenly into >> + * RMPP segments, then space must be allocated at the end of the referenced >> + * buffer for any required padding. > >This could be the source of the problem with PayloadLength. I need to >see if the buffer meets the criteria and if not, how to make it do this. >I will try this in the AM as well as your two related MAD patches. I calculate the padding in get_buf_length() in mad.c for ib_create_send_mad(). Basically: seg_size = sizeof(struct ib_mad) - hdr_len; pad = seg_size - data_len % seg_size; if (pad == seg_size) pad = 0; I then allocate a buffer sized to hdr_len + data_len + pad. - Sean From riel at redhat.com Wed May 4 18:27:21 2005 From: riel at redhat.com (Rik van Riel) Date: Wed, 4 May 2005 21:27:21 -0400 (EDT) Subject: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation In-Reply-To: <78d18e2050504112240e43a08@mail.gmail.com> References: <469958e00504291731eb8287c@mail.gmail.com> <20050503184325.GA19351@hexapodia.org> <78d18e2050504112240e43a08@mail.gmail.com> Message-ID: On Wed, 4 May 2005, William Jordan wrote: > On 5/3/05, Andy Isaacson wrote: > > Rather than replacing the fully-registered pages with pages of zeros, > > you could simply unmap them. > > I don't like this option. It is nearly free to map all of the pages to > the zero-page. You never have to allocate a page if the user never > writes to it. Unmapping should work fine, as long as the VMA flags are set appropriately. The page fault handler can take care of the rest... -- "Debugging is twice as hard as writing the code in the first place. Therefore, if you write the code as cleverly as possible, you are, by definition, not smart enough to debug it." - Brian W. Kernighan From adi at hexapodia.org Wed May 4 18:57:00 2005 From: adi at hexapodia.org (Andy Isaacson) Date: Wed, 4 May 2005 18:57:00 -0700 Subject: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation In-Reply-To: References: <469958e00504291731eb8287c@mail.gmail.com> <20050503184325.GA19351@hexapodia.org> <78d18e2050504112240e43a08@mail.gmail.com> Message-ID: <20050505015700.GA5205@hexapodia.org> On Wed, May 04, 2005 at 09:27:21PM -0400, Rik van Riel wrote: > On Wed, 4 May 2005, William Jordan wrote: > > On 5/3/05, Andy Isaacson wrote: > > > Rather than replacing the fully-registered pages with pages of zeros, > > > you could simply unmap them. > > > > I don't like this option. It is nearly free to map all of the pages to > > the zero-page. You never have to allocate a page if the user never > > writes to it. > > Unmapping should work fine, as long as the VMA flags are > set appropriately. The page fault handler can take care > of the rest... I think there may be a difference in terminology here. What I originally proposed (and what I think Bill was reacting to) is the equivalent of sys_munmap() on the range of registered pages. That has the downsides that he mentioned; an address that was valid in the parent will now result in SIGSEGV or SIGBUS in the child, and it's explicitly endorsed by the userland APIs (such as MPI2) that it's valid to register stack addresses (for example). What I think you're proposing, Rik, is that VMA get destroyed (or split, if only part of it had been registered) and replaced with an anonymous one. That's a very low-overhead way of going about it, I think. Then as you say, the page fault handler will automatically give a zero page to the process when it faults on those addresses. Did I understand your suggestion correctly? I think I agree with Bill that having the child fault on pages which happened to have been registered by the parent would be a bad thing. This would, if I understand correctly, be visible in /proc/$$/maps. Which is OK, if a little bit suprising; but the alternatives are worse. -andy From roland at topspin.com Wed May 4 20:44:50 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 04 May 2005 20:44:50 -0700 Subject: [openib-general] Re: [OOPS] user-mode verbs In-Reply-To: <1AC79F16F5C5284499BB9591B33D6F00044D66C7@orsmsx408> (Robert J. Woodruff's message of "Wed, 4 May 2005 16:11:09 -0700") References: <1AC79F16F5C5284499BB9591B33D6F00044D66C7@orsmsx408> Message-ID: <527jie72vh.fsf@topspin.com> Thanks for the stress testing, it's definitely very useful. I think I now understand what's causing the issue you're seeing -- I wrote some pretty dumb code. For example, ib_uverbs_destroy_qp() does: ret = ib_destroy_qp(qp); and then a few lines later: list_del(&qp->uobject->list); Of course "qp" has already been freed so if we're unlucky the contents may have already changed by the time we get to read the qp->uobject value. It turns out I made this mistake a bunch of places in uverbs_cmd.c, so I won't have a fix until tomorrow. I think the other oops is probably caused by: ret = ib_dereg_mr(mr); /* la la la ... */ memobj = container_of(mr->uobject, struct ib_umem_object, uobject); ib_umem_release(file->device->ib_dev, &memobj->umem); I promise to start testing with CONFIG_DEBUG_SLAB=y. - R. From eitan at mellanox.co.il Wed May 4 22:24:53 2005 From: eitan at mellanox.co.il (Eitan Zahavi) Date: Thu, 5 May 2005 08:24:53 +0300 Subject: [openib-general] OpenSM work Q1 + Apr. Message-ID: <506C3D7B14CDD411A52C00025558DED607C3010E@mtlex01.yok.mtl.com> Ok - my next release will go to: https://openib.org/svn/gen2/branches/eitan-opensm/ Eitan Zahavi Design Technology Director Mellanox Technologies LTD Tel:+972-4-9097208 Fax:+972-4-9593245 P.O. Box 586 Yokneam 20692 ISRAEL > -----Original Message----- > From: Hal Rosenstock [mailto:halr at voltaire.com] > Sent: Wednesday, May 04, 2005 11:47 PM > To: Eitan Zahavi > Cc: 'Tom Duffy'; 'openib-general at openib.org' > Subject: RE: [openib-general] OpenSM work Q1 + Apr. > > On Wed, 2005-05-04 at 16:40, Eitan Zahavi wrote: > > I was asked to push in OpenSM changes as soon they become stable > > enough to be used. > > This is what I'm doing. > > Once the gen2 stabilize > > Not sure what your criteria to determine stability are > > > and support RMPP etc, > > RMPP is getting close now > > > I will probably start working on it. > > But until then, gen1 is my driver. > > Where should I post the code if gen1 branch will close? > > Something like > https://openib.org/svn/gen2/branches/eitan-opensm/ > > -- Hal > > > > > Eitan Zahavi > > Design Technology Director > > Mellanox Technologies LTD > > Tel:+972-4-9097208 > > Fax:+972-4-9593245 > > P.O. Box 586 Yokneam 20692 ISRAEL > > > > > > > -----Original Message----- > > > From: Tom Duffy [mailto:tduffy at sun.com] > > > Sent: Wednesday, May 04, 2005 11:38 PM > > > To: Eitan Zahavi > > > Cc: 'openib-general at openib.org' > > > Subject: RE: [openib-general] OpenSM work Q1 + Apr. > > > > > > On Wed, 2005-05-04 at 23:29 +0300, Eitan Zahavi wrote: > > > > As was discussed more then once in several forums, my work on gen1 > > is > > > > ported by Hal and Shahar to gen2. Actually, the code can be > > directly > > > > copied over except for the "vendor layer" which are the > > > > osm_vendoc_mlx_* files. > > > > > > Sorry if you had already answered. > > > > > > In any event, I thought we were locking the gen1 tree to no further > > > development and sticking it in an archive section. There is some > > > confusion out there with the state of the tree -- which part should > > > people use, what version of the kernel do we support, etc. > > > > > > -tduffy > > > > > > > > ______________________________________________________________________ > > > > _______________________________________________ > > openib-general mailing list > > openib-general at openib.org > > http://openib.org/mailman/listinfo/openib-general > > > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general -------------- next part -------------- An HTML attachment was scrubbed... URL: From ogerlitz at voltaire.com Wed May 4 22:44:35 2005 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Thu, 5 May 2005 08:44:35 +0300 Subject: [openib-general] [kDAPL][PATCH] fix panic in server side Message-ID: Tom, >This occurred when I did a connect without first setting up a listen using kdapltest. >RIP: 0010:[] {:ib_dat_provider:dapl_cm_passive_cb_handler+27} >Call Trace:{:ib_cm:cm_process_work+50} {:ib_cm:cm_work_handler+1438} If you you haven't set up a (kdapltest server) listener - why did the CM invoked a (kdapl provider) consumer callback for this SID? Or. -----Original Message----- From: openib-general-bounces at openib.org [mailto:openib-general-bounces at openib.org] On Behalf Of Tom Duffy Sent: Thursday, May 05, 2005 12:59 AM To: 'openib-general at openib.org' Subject: [openib-general] [kDAPL][PATCH] fix panic in server side This occurred when I did a connect without first setting up a listen using kdapltest. Unable to handle kernel NULL pointer dereference at 0000000000000020 RIP: {:ib_dat_provider:dapl_cm_passive_cb_handler+27} PGD 371ab067 PUD 36822067 PMD 0 Oops: 0000 [1] SMP CPU 2 Modules linked in: kdapltest ib_dat_provider dat ib_at ib_cm ib_ipoib ib_sa ib_umad md5 ipv6 parport_pc lp parport autofs4 nfs lockd sunrpc pcmcia yenta_socket rsrc_nonstatic pcmcia_core ext3 jbd video container button battery ac uhci_hcd ehci_hcd hw_random i2c_i801 i2c_core ib_mthca ib_mad ib_core e1000 floppy dm_snapshot dm_zero dm_mirror xfs exportfs dm_mod mptscsih mptbase sd_mod scsi_mod Pid: 11326, comm: ib_cm/2 Not tainted 2.6.12-rc3openib RIP: 0010:[] {:ib_dat_provider:dapl_cm_passive_cb_handler+27} RSP: 0018:ffff810038f65d90 EFLAGS: 00010202 RAX: 0000000000000000 RBX: ffff81000b3764b0 RCX: ffff81000b376550 RDX: 0000000000000056 RSI: ffff81000b376548 RDI: ffff81003f0cd8a0 RBP: ffff81003f0cd8a0 R08: 0000000000000000 R09: 0000000000000000 R10: 0000000000000020 R11: ffff81003925ae10 R12: ffff81003f0cd8a0 R13: ffff81003dcb8a88 R14: 0000000000000000 R15: ffff81000b3765a0 FS: 0000000000000000(0000) GS:ffffffff80487a00(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 0000000000000020 CR3: 0000000038bbd000 CR4: 00000000000006e0 Process ib_cm/2 (pid: 11326, threadinfo ffff810038f64000, task ffff81003e75af30)Stack: ffffffff882aae32 0100000000000293 0000000000000000 ffff81000b3764b0 000000003f0cd8a0 ffff81003dcb8a88 0000000000000000 ffff81000b3765a0 ffffffff882abd1e ffff810001e1b680 Call Trace:{:ib_cm:cm_process_work+50} {:ib_cm:cm_work_handler+1438} {do_sigaction+520} {:ib_cm:cm_work_handler+0} {worker_thread+476} {default_wake_function+0} {__wake_up_common+67} {worker_thread+0} {keventd_create_kthread+0} {kthread+217} {schedule_tail+64} {child_rip+8} {keventd_create_kthread+0} {kthread+0} {child_rip+0} Code: 8b 40 20 89 44 24 04 83 7c 24 04 03 74 30 83 7c 24 04 03 77 RIP {:ib_dat_provider:dapl_cm_passive_cb_handler+27} RSP CR2: 0000000000000020 This cuteness happened in the kdapltest userland program: Warning: conn_event_wait DAT_CONNECTION_EVENT_DISCONNECTED DT_cs_Client: bad connection event DT_cs_Client: Cleaning Up ... DT_cs_Client: IA mthca0a closed DT_cs_Client: ========== End of Work -- Client Exiting TEST INSTANCE 1 *** glibc detected *** free(): invalid next size (fast): 0x000000000050ca50 *** Aborted (core dumped) This patches fixes two things. It makes it so that dapl_cm_passive_cb_handler won't dereference a potentially NULL ib_cm_id and also that it returns ret instead of 0 all the time. Signed-off-by: Tom Duffy Index: dapl_openib_cm.c =================================================================== --- linux-kernel/dat-provider/dapl_openib_cm.c (revision 2257) +++ linux-kernel/dat-provider/dapl_openib_cm.c (working copy) @@ -227,6 +227,9 @@ int dapl_cm_passive_cb_handler(struct ib { int ret = 0; + if (!comm_id) + return -1; + switch (comm_id->state) { case IB_CM_IDLE: ret = do_passive_idle(comm_id); @@ -247,7 +250,7 @@ int dapl_cm_passive_cb_handler(struct ib break; } - return 0; + return ret; } static void dapl_ib_destroy_cm_id_work(void *data) _______________________________________________ openib-general mailing list openib-general at openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From halr at voltaire.com Wed May 4 22:38:35 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 05 May 2005 01:38:35 -0400 Subject: [openib-general] [PATCH] [kDAPL] Remove unused variable in CM callbacks Message-ID: <1115271268.4476.32.camel@localhost.localdomain> Remove unused variable in CM callbacks Also, ensure CM ID of connection listener set to invalid when removed Signed-off-by: Hal Rosenstock Index: dapl_openib_cm.c =================================================================== -- dapl_openib_cm.c (revision 2254) +++ dapl_openib_cm.c (working copy) @@ -47,12 +47,12 @@ #define IB_TARGET_MAX 4 /* responder resources (max_qp_ous_rd_atom) */ #define IB_INITIATOR_DEPTH 4 /* initiator depth (max_qp_init_rd_atom) */ -static int do_active_idle(DAPL_EP * ep_ptr) +static void do_active_idle(DAPL_EP * ep_ptr) { ib_cm_events_t cm_event; if (ep_ptr->cm_handle == IB_INVALID_HANDLE) - return 0; + return; if (ep_ptr->param.connect_evd_handle) { cm_event = IB_CME_DISCONNECTED; @@ -60,21 +60,20 @@ cm_event, NULL, (void *)ep_ptr); } - return 0; } -static int do_active_rep_recv(DAPL_EP * ep_ptr, +static void do_active_rep_recv(DAPL_EP * ep_ptr, struct ib_cm_id *comm_id, struct ib_cm_event *rep_recv_param) { int status; if (ep_ptr->cm_handle == IB_INVALID_HANDLE) - return 0; + return; if (ep_ptr->qp_handle == IB_INVALID_HANDLE) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, " do_active_rep_recv: invalid qp handle\n"); - return 0; + return; } /* First, transition QP to RTR */ @@ -107,21 +106,18 @@ rep_recv_param->private_data, (void *)ep_ptr); - return 0; } -static int do_active_disc(DAPL_EP * ep_ptr, struct ib_cm_id *comm_id) +static void do_active_disc(DAPL_EP * ep_ptr, struct ib_cm_id *comm_id) { dapl_evd_connection_callback(comm_id, IB_CME_DISCONNECTED, NULL, (void *)ep_ptr); - return 0; } int dapl_cm_active_cb_handler(struct ib_cm_id *comm_id, struct ib_cm_event *event) { DAPL_EP *ep_ptr; - int ret = 0; ep_ptr = (DAPL_EP *) comm_id->context; /* Is this check needed ? */ @@ -133,16 +129,16 @@ switch (comm_id->state) { case IB_CM_IDLE: - ret = do_active_idle(ep_ptr); + do_active_idle(ep_ptr); break; case IB_CM_REP_RCVD: - ret = do_active_rep_recv(ep_ptr, comm_id, event); + do_active_rep_recv(ep_ptr, comm_id, event); break; case IB_CM_ESTABLISHED: /* Nothing to do */ break; case IB_CM_DREQ_RCVD: - ret = do_active_disc(ep_ptr, comm_id); + do_active_disc(ep_ptr, comm_id); break; default: dapl_dbg_log(DAPL_DBG_TYPE_ERR, @@ -154,7 +150,7 @@ return 0; } -static int do_passive_idle(struct ib_cm_id *comm_id) +static void do_passive_idle(struct ib_cm_id *comm_id) { ib_cm_events_t cm_event; @@ -162,20 +158,18 @@ dapl_cr_callback(comm_id, cm_event, NULL, comm_id->context); - return 0; } -static int do_passive_req_recv(struct ib_cm_id *comm_id, +static void do_passive_req_recv(struct ib_cm_id *comm_id, struct ib_cm_event *req_recv_param) { dapl_cr_callback(comm_id, IB_CME_CONNECTION_REQUEST_PENDING, req_recv_param->private_data, comm_id->context); - return 0; } -static int do_passive_established(struct ib_cm_id *comm_id) +static void do_passive_established(struct ib_cm_id *comm_id) { DAPL_SP *sp_ptr; DAPL_CR *cr_ptr; @@ -186,7 +180,7 @@ cr_ptr = dapl_sp_search_cr(sp_ptr, comm_id); if (cr_ptr == NULL) { printk(KERN_ERR "do_passive_established: No CR found for CM ID 0x%p\n", comm_id); - return 0; + return; } ep_ptr = (DAPL_EP *)cr_ptr->param.local_ep_handle; /* Quick check to ensure our EP is still valid */ @@ -211,34 +205,30 @@ IB_CME_CONNECTED, NULL, (void *)comm_id->context); - return 0; } -static int do_passive_disc(struct ib_cm_id *comm_id) +static void do_passive_disc(struct ib_cm_id *comm_id) { dapl_cr_callback(comm_id, IB_CME_DISCONNECTED, NULL, (void *)comm_id->context); - return 0; } int dapl_cm_passive_cb_handler(struct ib_cm_id *comm_id, struct ib_cm_event *event) { - int ret = 0; - switch (comm_id->state) { case IB_CM_IDLE: - ret = do_passive_idle(comm_id); + do_passive_idle(comm_id); break; case IB_CM_REQ_RCVD: - ret = do_passive_req_recv(comm_id, event); + do_passive_req_recv(comm_id, event); break; case IB_CM_ESTABLISHED: - ret = do_passive_established(comm_id); + do_passive_established(comm_id); break; case IB_CM_DREQ_RCVD: - ret = do_passive_disc(comm_id); + do_passive_disc(comm_id); break; default: dapl_dbg_log(DAPL_DBG_TYPE_ERR, @@ -510,6 +500,7 @@ * Input: * ep_ptr DAPL_EP * active Indicates active side of connection + * ib_cm_event * * Output: * none @@ -621,10 +612,9 @@ status = ib_cm_listen(sp_ptr->cm_srvc_handle, be64_to_cpu(ServiceID), 0); if (status) { - /* need to destroy CM ID ??? */ + /* Need to also destroy CM ID ? */ + sp_ptr->cm_srvc_handle = IB_INVALID_HANDLE; - sp_ptr->cm_srvc_handle = NULL; - if (status == -EBUSY) return DAT_CONN_QUAL_IN_USE; else @@ -659,6 +649,7 @@ return DAT_SUCCESS; status = dapl_ib_queue_destroy_cm_id(sp_ptr->cm_srvc_handle); + sp_ptr->cm_srvc_handle = IB_INVALID_HANDLE; if (status) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, " dapl_ib_remove_conn_listener: ib_destroy_cm_id failed: %d\n", status); From halr at voltaire.com Wed May 4 23:00:56 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 05 May 2005 02:00:56 -0400 Subject: [openib-general] [kDAPL][PATCH] fix panic in server side In-Reply-To: <1115243969.14238.6.camel@duffman> References: <1115243969.14238.6.camel@duffman> Message-ID: <1115272856.4476.52.camel@localhost.localdomain> On Wed, 2005-05-04 at 17:59, Tom Duffy wrote: > This occurred when I did a connect without first setting up a listen > using kdapltest. > > Unable to handle kernel NULL pointer dereference at 0000000000000020 RIP: > {:ib_dat_provider:dapl_cm_passive_cb_handler+27} > PGD 371ab067 PUD 36822067 PMD 0 > Oops: 0000 [1] SMP > CPU 2 > Modules linked in: kdapltest ib_dat_provider dat ib_at ib_cm ib_ipoib ib_sa ib_umad md5 ipv6 parport_pc lp parport autofs4 nfs lockd sunrpc pcmcia yenta_socket rsrc_nonstatic pcmcia_core ext3 jbd video container button battery ac uhci_hcd ehci_hcd hw_random i2c_i801 i2c_core ib_mthca ib_mad ib_core e1000 floppy dm_snapshot dm_zero dm_mirror xfs exportfs dm_mod mptscsih mptbase sd_mod scsi_mod > Pid: 11326, comm: ib_cm/2 Not tainted 2.6.12-rc3openib > RIP: 0010:[] {:ib_dat_provider:dapl_cm_passive_cb_handler+27} > RSP: 0018:ffff810038f65d90 EFLAGS: 00010202 > RAX: 0000000000000000 RBX: ffff81000b3764b0 RCX: ffff81000b376550 > RDX: 0000000000000056 RSI: ffff81000b376548 RDI: ffff81003f0cd8a0 > RBP: ffff81003f0cd8a0 R08: 0000000000000000 R09: 0000000000000000 > R10: 0000000000000020 R11: ffff81003925ae10 R12: ffff81003f0cd8a0 > R13: ffff81003dcb8a88 R14: 0000000000000000 R15: ffff81000b3765a0 > FS: 0000000000000000(0000) GS:ffffffff80487a00(0000) knlGS:0000000000000000 > CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > CR2: 0000000000000020 CR3: 0000000038bbd000 CR4: 00000000000006e0 > Process ib_cm/2 (pid: 11326, threadinfo ffff810038f64000, task ffff81003e75af30)Stack: ffffffff882aae32 0100000000000293 0000000000000000 ffff81000b3764b0 > 000000003f0cd8a0 ffff81003dcb8a88 0000000000000000 ffff81000b3765a0 > ffffffff882abd1e ffff810001e1b680 > Call Trace:{:ib_cm:cm_process_work+50} {:ib_cm:cm_work_handler+1438} > {do_sigaction+520} {:ib_cm:cm_work_handler+0} > {worker_thread+476} {default_wake_function+0} > {__wake_up_common+67} {worker_thread+0} > {keventd_create_kthread+0} {kthread+217} > {schedule_tail+64} {child_rip+8} > {keventd_create_kthread+0} {kthread+0} > {child_rip+0} > > Code: 8b 40 20 89 44 24 04 83 7c 24 04 03 74 30 83 7c 24 04 03 77 > RIP {:ib_dat_provider:dapl_cm_passive_cb_handler+27} RSP > CR2: 0000000000000020 > > This cuteness happened in the kdapltest userland program: > > Warning: conn_event_wait DAT_CONNECTION_EVENT_DISCONNECTED > DT_cs_Client: bad connection event > DT_cs_Client: Cleaning Up ... > DT_cs_Client: IA mthca0a closed > DT_cs_Client: ========== End of Work -- Client Exiting > TEST INSTANCE 1 > *** glibc detected *** free(): invalid next size (fast): 0x000000000050ca50 *** > Aborted (core dumped) Hmm, when I do this (on x86 and ), I get the following: DT_cs_Client: Connect Endpoint DT_cs_Client: Await connection ... Warning: conn_event_wait DAT_CONNECTION_EVENT_DISCONNECTED DT_cs_Client: bad connection event DT_cs_Client: Cleaning Up ... DT_cs_Client: IA mthca0a closed DT_cs_Client: ========== End of Work -- Client Exiting This takes a while until REQ timeout/retries are exhausted. On x86_64, I do get the following (same as you in userland): DT_cs_Client: Connect Endpoint DT_cs_Client: Await connection ... Warning: conn_event_wait DAT_CONNECTION_EVENT_DISCONNECTED DT_cs_Client: bad connection event DT_cs_Client: Cleaning Up ... DT_cs_Client: IA mthca0a closed DT_cs_Client: ========== End of Work -- Client Exiting *** glibc detected *** double free or corruption: 0x000000000050c010 *** Aborted but no oops. That is a kdapltest test program issue as you noted. How long did you wait ? Did you ctl-C out ? I'm presuming you loaded IPoIB and are pointing the client at an IPoIB which is pingable but does not have a kdapl server (connection listener) started up. I don't understand why the passive CM handler would be invoked without a server started. Would you describe your procedure some more so I can see if I can recreate this ? -- Hal From mst at mellanox.co.il Thu May 5 04:01:58 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Thu, 5 May 2005 14:01:58 +0300 Subject: [openib-general] [PATCH] sdp: replace mlock with get_user_pages Message-ID: <20050505110158.GH30653@mellanox.co.il> Hello, Libor! The following patch replaces the mlock hack with call to get_user_pages. Since the application could have forked while an iocb is outstanding, when an iocb is done I do get_user_pages for a second time and copy data if the physical address has changed. Thus, changing ulimit is no longer required to get aio working, processes are also allowed to fork and to call mlock/munlock on the buffer. Tested by the ttcp.aio benchmark, and works fine for me. Signed-off-by: Michael S. Tsirkin Index: ulp/sdp/sdp_send.c =================================================================== --- ulp/sdp/sdp_send.c (revision 2235) +++ ulp/sdp/sdp_send.c (working copy) @@ -2197,6 +2202,7 @@ skip: /* entry point for IOCB based tran iocb->req = req; iocb->key = req->ki_key; iocb->addr = (unsigned long)msg->msg_iov->iov_base - copied; + iocb->is_receive = 0; req->ki_cancel = sdp_inet_write_cancel; Index: ulp/sdp/sdp_recv.c =================================================================== --- ulp/sdp/sdp_recv.c (revision 2235) +++ ulp/sdp/sdp_recv.c (working copy) @@ -1447,6 +1448,7 @@ int sdp_inet_recv(struct kiocb *req, st iocb->key = req->ki_key; iocb->addr = ((unsigned long)msg->msg_iov->iov_base - copied); + iocb->is_receive = 1; req->ki_cancel = sdp_inet_read_cancel; Index: ulp/sdp/sdp_iocb.c =================================================================== --- ulp/sdp/sdp_iocb.c (revision 2235) +++ ulp/sdp/sdp_iocb.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies Ltd. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -31,389 +32,197 @@ * * $Id$ */ - +#include #include "sdp_main.h" static kmem_cache_t *sdp_iocb_cache = NULL; -/* - * memory locking functions - */ -#include - -typedef int (*do_mlock_ptr_t)(unsigned long, size_t, int); -static do_mlock_ptr_t mlock_ptr = NULL; - -/* - * do_iocb_unlock - unlock the memory for an IOCB - */ -static int do_iocb_unlock(struct sdpc_iocb *iocb) -{ - struct vm_area_struct *vma; - - vma = find_vma(iocb->mm, (iocb->addr & PAGE_MASK)); - if (!vma) - sdp_warn("No VMA for IOCB <%lx:%Zu> unlock", - iocb->addr, iocb->size); - - while (vma) { - sdp_dbg_data(NULL, - "unmark <%lx> <%p> <%08lx:%08lx> <%08lx> <%ld>", - iocb->addr, vma, vma->vm_start, vma->vm_end, - vma->vm_flags, (long)vma->vm_private_data); - - spin_lock(&iocb->mm->page_table_lock); - /* - * if there are no more references to the vma - */ - vma->vm_private_data--; - - if (!vma->vm_private_data) { - /* - * modify VM flags. - */ - vma->vm_flags &= ~(VM_DONTCOPY|VM_LOCKED); - /* - * adjust locked page count - */ - vma->vm_mm->locked_vm -= ((vma->vm_end - - vma->vm_start) >> - PAGE_SHIFT); - } - - spin_unlock(&iocb->mm->page_table_lock); - /* - * continue if the buffer continues onto the next vma - */ - if ((iocb->addr + iocb->size) > vma->vm_end) - vma = vma->vm_next; - else - vma = NULL; - } - - return 0; +static void sdp_copy_one_page(struct page *from, struct page* to, + unsigned long iocb_addr, size_t iocb_size, + unsigned long uaddr) +{ + size_t size_left = iocb_addr + iocb_size - uaddr; + size_t size = min(size_left,PAGE_SIZE); + unsigned long offset = uaddr % PAGE_SIZE; + unsigned long flags; + + void* fptr; + void* tptr; + + local_irq_save(flags); + fptr = kmap_atomic(from, KM_IRQ0); + tptr = kmap_atomic(to, KM_IRQ1); + + memcpy(tptr + offset, fptr + offset, size); + + kunmap_atomic(tptr, KM_IRQ1); + kunmap_atomic(fptr, KM_IRQ0); + local_irq_restore(flags); + set_page_dirty_lock(to); } /* * sdp_iocb_unlock - unlock the memory for an IOCB + * Copy if pages moved since. + * TODO: is this needed? */ int sdp_iocb_unlock(struct sdpc_iocb *iocb) { int result; + struct page ** pages = NULL; + unsigned long uaddr; + int i; + - /* - * check if IOCB is locked. - */ if (!(iocb->flags & SDP_IOCB_F_LOCKED)) return 0; - /* - * spin lock since this could be from interrupt context. - */ - down_write(&iocb->mm->mmap_sem); - - result = do_iocb_unlock(iocb); - up_write(&iocb->mm->mmap_sem); + /* For read, unlock and we are done */ + if (!iocb->is_receive) { + for (i = 0;i < iocb->page_count; ++i) + put_page(iocb->page_array[i]); + goto done; + } - kfree(iocb->page_array); - kfree(iocb->addr_array); + /* For write, we must check the virtual pages did not get remapped */ - iocb->page_array = NULL; - iocb->addr_array = NULL; - iocb->mm = NULL; - /* - * mark IOCB unlocked. - */ - iocb->flags &= ~SDP_IOCB_F_LOCKED; + /* As an optimisation (to avoid scanning the vma tree each time), + * try to get all pages in one go. */ + /* TODO: use cache for allocations? Allocate by chunks? */ - return result; -} + pages = kmalloc((sizeof(struct page *) * iocb->page_count), GFP_KERNEL); -/* - * sdp_iocb_page_save - save page information for an IOCB - */ -static int sdp_iocb_page_save(struct sdpc_iocb *iocb) -{ - unsigned int counter; - unsigned long addr; - size_t size; - int result = -ENOMEM; - struct page *page; - unsigned long pfn; - pgd_t *pgd; - pud_t *pud; - pmd_t *pmd; - pte_t *ptep; - pte_t pte; + down_read(&iocb->mm->mmap_sem); - if (iocb->page_count <= 0 || iocb->size <= 0 || !iocb->addr) - return -EINVAL; - /* - * create array to hold page value which are later needed to register - * the buffer with the HCA - */ - iocb->addr_array = kmalloc((sizeof(u64) * iocb->page_count), - GFP_KERNEL); - if (!iocb->addr_array) - goto err_addr; + if (pages) { + result=get_user_pages(iocb->tsk, iocb->mm, + iocb->addr, + iocb->page_count , iocb->is_receive, 0, + pages, NULL); - iocb->page_array = kmalloc((sizeof(struct page *) * iocb->page_count), - GFP_KERNEL); - if (!iocb->page_array) - goto err_page; - /* - * iocb->addr - buffer start address - * iocb->size - buffer length - * addr - page aligned - * size - page multiple - */ - addr = iocb->addr & PAGE_MASK; - size = PAGE_ALIGN(iocb->size + (iocb->addr & ~PAGE_MASK)); + if (result != iocb->page_count) { + kfree(pages); + pages = NULL; + } + } - iocb->page_offset = iocb->addr - addr; - /* - * Find pages used within the buffer which will then be registered - * for RDMA - */ - spin_lock(&iocb->mm->page_table_lock); + for (i = 0, uaddr = iocb->addr; i < iocb->page_count; + ++i, uaddr = (uaddr & PAGE_MASK) + PAGE_SIZE) + { + struct page* page; + set_page_dirty_lock(iocb->page_array[i]); + + if (pages) + page = pages[i]; + else { + result=get_user_pages(iocb->tsk, iocb->mm, + uaddr & PAGE_MASK, + 1 , 1, 0, &page, NULL); + if (result != 1) { + page = NULL; + } + } - for (counter = 0; - size > 0; - counter++, addr += PAGE_SIZE, size -= PAGE_SIZE) { - pgd = pgd_offset_gate(iocb->mm, addr); - if (!pgd || pgd_none(*pgd)) - break; - - pud = pud_offset(pgd, addr); - if (!pud || pud_none(*pud)) - break; - - pmd = pmd_offset(pud, addr); - if (!pmd || pmd_none(*pmd)) - break; - - ptep = pte_offset_map(pmd, addr); - if (!ptep) - break; - - pte = *ptep; - pte_unmap(ptep); - - if (!pte_present(pte)) - break; - - pfn = pte_pfn(pte); - if (!pfn_valid(pfn)) - break; - - page = pfn_to_page(pfn); + if (page && iocb->page_array[i] != page) + sdp_copy_one_page(iocb->page_array[i], page, + iocb->addr, iocb->size, uaddr); - iocb->page_array[counter] = page; - iocb->addr_array[counter] = page_to_phys(page); + if (page) + put_page(page); + put_page(iocb->page_array[i]); } - spin_unlock(&iocb->mm->page_table_lock); - - if (size > 0) { - result = -EFAULT; - goto err_find; - } + up_read(&iocb->mm->mmap_sem); - return 0; -err_find: - - kfree(iocb->page_array); - iocb->page_array = NULL; -err_page: + if (pages) + kfree(pages); +done: + kfree(iocb->page_array); kfree(iocb->addr_array); + + iocb->page_array = NULL; iocb->addr_array = NULL; -err_addr: + iocb->mm = NULL; + iocb->tsk = NULL; - return result; + iocb->flags &= ~SDP_IOCB_F_LOCKED; + + return 0; } /* * sdp_iocb_lock - lock the memory for an IOCB + * We do not take a reference on the mm, AIO handles this for us. */ int sdp_iocb_lock(struct sdpc_iocb *iocb) { - struct vm_area_struct *vma; - kernel_cap_t real_cap; - unsigned long limit; int result = -ENOMEM; unsigned long addr; - size_t size; - - /* - * mark IOCB as locked. We do not take a reference on the mm, AIO - * handles this for us. - */ - iocb->flags |= SDP_IOCB_F_LOCKED; - iocb->mm = current->mm; + size_t size; + int i; /* - * save and raise capabilities + * iocb->addr - buffer start address + * iocb->size - buffer length + * addr - page aligned + * size - page multiple */ - real_cap = cap_t(current->cap_effective); - cap_raise(current->cap_effective, CAP_IPC_LOCK); - - size = PAGE_ALIGN(iocb->size + (iocb->addr & ~PAGE_MASK)); addr = iocb->addr & PAGE_MASK; + size = PAGE_ALIGN(iocb->size + (iocb->addr & ~PAGE_MASK)); + iocb->page_offset = iocb->addr - addr; + iocb->page_count = size >> PAGE_SHIFT; - - limit = current->signal->rlim[RLIMIT_MEMLOCK].rlim_cur; - limit >>= PAGE_SHIFT; - /* - * lock the mm, if within the limit lock the address range. - */ - down_write(&iocb->mm->mmap_sem); - - if (!((iocb->page_count + current->mm->locked_vm) > limit)) - result = (*mlock_ptr)(addr, size, 1); /* - * process result - */ - if (result) { - sdp_dbg_err("VMA lock <%lx:%Zu> error <%d> <%d:%lu:%lu>", - iocb->addr, iocb->size, result, - iocb->page_count, iocb->mm->locked_vm, limit); - goto err_lock; - } - /* - * look up the head of the vma queue, loop through the vmas, marking - * them do not copy, reference counting, and saving them. + * create array to hold page value which are later needed to register + * the buffer with the HCA */ - vma = find_vma(iocb->mm, addr); - if (!vma) - /* - * sanity check. - */ - sdp_warn("No VMA for IOCB! <%lx:%Zu> lock", - iocb->addr, iocb->size); - while (vma) { - spin_lock(&iocb->mm->page_table_lock); + /* TODO: use cache for allocations? Allocate by chunks? */ + iocb->addr_array = kmalloc((sizeof(u64) * iocb->page_count), + GFP_KERNEL); + if (!iocb->addr_array) + goto err_addr; - if (!(VM_LOCKED & vma->vm_flags)) - sdp_warn("Unlocked vma! <%08lx>", vma->vm_flags); + iocb->page_array = kmalloc((sizeof(struct page *) * iocb->page_count), + GFP_KERNEL); + if (!iocb->page_array) + goto err_page; - if (PAGE_SIZE < (unsigned long)vma->vm_private_data) - sdp_dbg_err("VMA: private daya in use! <%08lx>", - (unsigned long)vma->vm_private_data); - - vma->vm_flags |= VM_DONTCOPY; - vma->vm_private_data++; + down_read(¤t->mm->mmap_sem); - spin_unlock(&iocb->mm->page_table_lock); + result=get_user_pages(current, current->mm, iocb->addr, + iocb->page_count , iocb->is_receive, 0, + iocb->page_array, NULL); + + up_read(¤t->mm->mmap_sem); + + if (result != iocb->page_count) { + sdp_dbg_err("unable to lock <%lx:%Zu> error <%d> <%d>", + iocb->addr, iocb->size, result, iocb->page_count); + goto err_get; + } - sdp_dbg_data(NULL, - "mark <%lx> <0x%p> <%08lx:%08lx> <%08lx> <%ld>", - iocb->addr, vma, vma->vm_start, vma->vm_end, - vma->vm_flags, (long)vma->vm_private_data); + iocb->flags |= SDP_IOCB_F_LOCKED; + iocb->mm = current->mm; + iocb->tsk = current; - if ((addr + size) > vma->vm_end) - vma = vma->vm_next; - else - vma = NULL; - } - result = sdp_iocb_page_save(iocb); - if (result) { - sdp_dbg_err("Error <%d> saving pages for IOCB <%lx:%Zu>", - result, iocb->addr, iocb->size); - goto err_save; + for (i = 0; i< iocb->page_count; ++i) { + iocb->addr_array[i] = page_to_phys(iocb->page_array[i]); } - up_write(&iocb->mm->mmap_sem); - cap_t(current->cap_effective) = real_cap; - return 0; -err_save: - - (void)do_iocb_unlock(iocb); -err_lock: - /* - * unlock the mm and restore capabilities. - */ - up_write(&iocb->mm->mmap_sem); - cap_t(current->cap_effective) = real_cap; - - iocb->flags &= ~SDP_IOCB_F_LOCKED; - iocb->mm = NULL; +err_get: + kfree(iocb->page_array); +err_page: + kfree(iocb->addr_array); +err_addr: return result; } /* - * IOCB memory locking init functions - */ -struct kallsym_iter { - loff_t pos; - struct module *owner; - unsigned long value; - unsigned int nameoff; /* If iterating in core kernel symbols */ - char type; - char name[128]; -}; - -/* - * sdp_mem_lock_init - initialize the userspace memory locking - */ -static int sdp_mem_lock_init(void) -{ - struct file *kallsyms; - struct seq_file *seq; - struct kallsym_iter *iter; - loff_t pos = 0; - int ret = -EINVAL; - - sdp_dbg_init("Memory Locking initialization."); - - kallsyms = filp_open("/proc/kallsyms", O_RDONLY, 0); - if (!kallsyms) { - sdp_warn("Failed to open /proc/kallsyms"); - goto done; - } - - seq = (struct seq_file *)kallsyms->private_data; - if (!seq) { - sdp_warn("Failed to fetch sequential file."); - goto err_close; - } - - for (iter = seq->op->start(seq, &pos); - iter != NULL; - iter = seq->op->next(seq, iter, &pos)) - if (!strcmp(iter->name, "do_mlock")) - mlock_ptr = (do_mlock_ptr_t)iter->value; - - if (!mlock_ptr) - sdp_warn("Failed to find lock pointer."); - else - ret = 0; - -err_close: - filp_close(kallsyms, NULL); -done: - return ret; -} - -/* - * sdp_mem_lock_cleanup - cleanup the memory locking tables - */ -static int sdp_mem_lock_cleanup(void) -{ - sdp_dbg_init("Memory Locking cleanup."); - /* - * null out entries. - */ - mlock_ptr = NULL; - - return 0; -} - -/* * IOCB memory registration functions */ @@ -831,28 +640,12 @@ void sdp_iocb_q_clear(struct sdpc_iocb_q } /* - * primary initialization/cleanup functions - */ - -/* * sdp_main_iocb_init - initialize the advertisment caches */ int sdp_main_iocb_init(void) { - int result; - sdp_dbg_init("IOCB cache initialization."); - /* - * initialize locking code. - */ - result = sdp_mem_lock_init(); - if (result < 0) { - sdp_warn("Error <%d> initializing memory locking.", result); - return result; - } - /* - * initialize the caches only once. - */ + if (sdp_iocb_cache) { sdp_warn("IOCB caches already initialized."); return -EINVAL; @@ -862,15 +655,10 @@ int sdp_main_iocb_init(void) sizeof(struct sdpc_iocb), 0, SLAB_HWCACHE_ALIGN, NULL, NULL); - if (!sdp_iocb_cache) { - result = -ENOMEM; - goto error_iocb_c; - } + if (!sdp_iocb_cache) + return -ENOMEM; return 0; -error_iocb_c: - (void)sdp_mem_lock_cleanup(); - return result; } /* @@ -879,16 +667,6 @@ error_iocb_c: void sdp_main_iocb_cleanup(void) { sdp_dbg_init("IOCB cache cleanup."); - /* - * cleanup the caches - */ kmem_cache_destroy(sdp_iocb_cache); - /* - * null out entries. - */ sdp_iocb_cache = NULL; - /* - * cleanup memory locking - */ - (void)sdp_mem_lock_cleanup(); } Index: ulp/sdp/sdp_iocb.h =================================================================== --- ulp/sdp/sdp_iocb.h (revision 2235) +++ ulp/sdp/sdp_iocb.h (working copy) @@ -99,9 +99,11 @@ struct sdpc_iocb { /* * page list. data for locking/registering userspace */ - struct mm_struct *mm; /* user mm struct */ - unsigned long addr; /* user space address */ - size_t size; /* total size of the user buffer */ + struct mm_struct *mm; /* user mm struct */ + struct task_struct *tsk; + unsigned long addr; /* user space address */ + size_t size; /* total size of the user buffer */ + int is_receive; struct page **page_array; /* list of page structure pointers. */ u64 *addr_array; /* list of physical page addresses. */ -- MST - Michael S. Tsirkin From mst at mellanox.co.il Thu May 5 05:04:55 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Thu, 5 May 2005 15:04:55 +0300 Subject: [openib-general] [PATCH] sdp: get rid of in_atomic Message-ID: <20050505120455.GJ30653@mellanox.co.il> Hello, Libor! Here's a rediff against the latest bits. Could you please comment on this patch? sdp_kmap and sdp_kunmap use in_atomic and kmap_atomic in an unsafe manner. The following patch replaces all uses of these macros with kmap_atomic on IRQ0/IRQ1 slot, while disabling local interrupts. Works fine for me. Signed-off-by: Michael S. Tsirkin Index: ulp/sdp/sdp_send.c =================================================================== --- ulp/sdp/sdp_send.c (revision 2235) +++ ulp/sdp/sdp_send.c (working copy) @@ -641,6 +641,7 @@ static int sdp_send_data_iocb_src(struct SDP_BUFF_F_CLR_UNSIG(buff); if (conn->send_mode == SDP_MODE_COMB) { + unsigned long flags; void *addr; int pos; int off; @@ -662,21 +663,22 @@ static int sdp_send_data_iocb_src(struct result = -EFAULT; goto error; } - /* - * map, copy, unmap. - */ - addr = sdp_kmap(iocb->page_array[pos]); + + local_irq_save(flags); + + addr = kmap_atomic(iocb->page_array[pos], KM_IRQ0); if (!addr) { result = -ENOMEM; + local_irq_restore(flags); goto error; } memcpy(buff->tail, (addr + off), len); - sdp_kunmap(iocb->page_array[pos]); - /* - * update pointers - */ + kunmap_atomic(iocb->page_array[pos], KM_IRQ0); + + local_irq_restore(flags); + buff->data_size = len; buff->tail += len; @@ -731,14 +733,16 @@ static int sdp_send_iocb_buff_write(stru counter = (iocb->post + iocb->page_offset) >> PAGE_SHIFT; offset = (iocb->post + iocb->page_offset) & (~PAGE_MASK); - while (buff->tail < buff->end && - iocb->len > 0) { - /* - * map correct page of iocb - */ - addr = sdp_kmap(iocb->page_array[counter]); - if (!addr) + + while (buff->tail < buff->end && iocb->len > 0) { + unsigned long flags; + local_irq_save(flags); + + addr = kmap_atomic(iocb->page_array[counter], KM_IRQ0); + if (!addr) { + local_irq_restore(flags); break; + } copy = min((PAGE_SIZE - offset), (unsigned long)(buff->end - buff->tail)); @@ -756,7 +760,8 @@ static int sdp_send_iocb_buff_write(stru offset += copy; offset &= (~PAGE_MASK); - sdp_kunmap(iocb->page_array[counter++]); + kunmap_atomic(iocb->page_array[counter++], KM_IRQ0); + local_irq_restore(flags); } return 0; Index: ulp/sdp/sdp_recv.c =================================================================== --- ulp/sdp/sdp_recv.c (revision 2235) +++ ulp/sdp/sdp_recv.c (working copy) @@ -659,12 +659,11 @@ static int sdp_read_buff_iocb(struct sdp counter = (iocb->post + iocb->page_offset) >> PAGE_SHIFT; offset = (iocb->post + iocb->page_offset) & (~PAGE_MASK); - while (buff->data < buff->tail && - iocb->len > 0) { - /* - * map correct page of iocb - */ - addr = sdp_kmap(iocb->page_array[counter]); + while (buff->data < buff->tail && iocb->len > 0) { + unsigned long flags; + local_irq_save(flags); + + addr = kmap_atomic(iocb->page_array[counter], KM_IRQ0); if (!addr) break; @@ -684,7 +683,9 @@ static int sdp_read_buff_iocb(struct sdp iocb->io_addr += copy; - sdp_kunmap(iocb->page_array[counter++]); + kunmap_atomic(iocb->page_array[counter++], KM_IRQ0); + + local_irq_restore(flags); } /* * restore tail from OOB offset. Index: ulp/sdp/sdp_iocb.h =================================================================== --- ulp/sdp/sdp_iocb.h (revision 2235) +++ ulp/sdp/sdp_iocb.h (working copy) @@ -124,30 +126,4 @@ struct sdpc_iocb_q { int size; /* current number of IOCBs in table */ }; -/* - * Address translations - */ - -/* - * sdp_kmap - map a page into kernel space - */ -static inline void *sdp_kmap(struct page *page) -{ - if (in_atomic() || irqs_disabled()) - return kmap_atomic(page, KM_IRQ0); - else - return kmap(page); -} - -/* - * sdp_kunmap - unmap a page into kernel space - */ -static inline void sdp_kunmap(struct page *page) -{ - if (in_atomic() || irqs_disabled()) - kunmap_atomic(page, KM_IRQ0); - else - kunmap(page); -} - #endif /* _SDP_IOCB_H */ -- MST - Michael S. Tsirkin From eaff at fadmail.com Thu May 5 07:31:54 2005 From: eaff at fadmail.com (Nick Gee) Date: Thu, 05 May 2005 19:31:54 +0500 Subject: [openib-general] Approved mortage rate Message-ID: <118241.3024.eaff@fadmail.com> Hello, We tried contacting you awhile ago about your low interest morta(ge rate. You have qualified for the lowest rate in years... You could get over $380,000 for as little as $500 a month! Ba(d credit? Doesn't matter, low rates are fixed no matter what! To get a free, no obli,gation consultation click below: http://www.h0us1ng.net/sign.asp Best Regards, Shane Brantley to be remov(ed: http://www.h0us1ng.net/gone.asp this process takes one week, so please be patient. we do our best to take your email/s off but you have to fill out a rem/ove or else you will continue to recieve email/s. From halr at voltaire.com Thu May 5 06:49:20 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 05 May 2005 09:49:20 -0400 Subject: [openib-general] [PATCH] [MAD] changes to ib_create_send_mad In-Reply-To: <1115253041.4476.147.camel@localhost.localdomain> References: <1115253041.4476.147.camel@localhost.localdomain> Message-ID: <1115300959.4476.189.camel@localhost.localdomain> Hi Sean, On Wed, 2005-05-04 at 20:33, Hal Rosenstock wrote: > On Wed, 2005-05-04 at 20:05, Sean Hefty wrote: > > If the class defined data would not divide evenly into > > + * RMPP segments, then space must be allocated at the end of the referenced > > + * buffer for any required padding. > > This could be the source of the problem with PayloadLength. I need to > see if the buffer meets the criteria and if not, how to make it do this. > I will try this in the AM as well as your two related MAD patches. On PayloadLength, single segment sends are fine. It is multisegment sends which seem wrong to me. Case 1: ib_create_send_mad with hdr_len 0x38 data_len 0x278 paylen_newwin stored in header is 0x28C which seems correct this creates 4 segments 1-3 segments paylen_newwin is 0x6E0, 4th segment is 0x34 Case 2: ib_create_send_mad with hdr_len 0x38 data_len 0x620 paylen_newwin stored in header is 0x634 which seems correct this creates 8 segments 1-7 segments paylen_newwin is 0x6E0, 4th segment is 0xBC Last paylen_newwin in both cases appears to me to be correct but the paylen_newwin in the 1-n segments (0x6E0) seems wrong to me. Also, I did more investigation of send failures. In terms of send failures upon not receiving ACKs, SA is different from vendor class 2. I think the problem starts (and hopefully ends) with response_mad(). In the case of SA GetTableResp being sent, the non data packets (ACK, etc.) come back as SA GetTable so this is not currently considered a response but I think it needs to be. I'm not sure if there are other issues behind this as I didn't chase it further. Let me know if you want me to do this. -- Hal From itamar at mellanox.co.il Thu May 5 07:37:53 2005 From: itamar at mellanox.co.il (Itamar) Date: Thu, 5 May 2005 17:37:53 +0300 Subject: [openib-general] [PATCH][KDAPL] clean up code : evd flow Message-ID: <20050505143753.GA1517@mellanox.co.il> Clean up kdapl : evd flow Signed-off-by: Itamar Rabenstein Index: dapl_evd_dto_callb.c =================================================================== --- dapl_evd_dto_callb.c (revision 2257) +++ dapl_evd_dto_callb.c (working copy) @@ -98,46 +98,26 @@ void dapl_evd_dto_callback(ib_cq_handle_ * EVD is enabled. */ - if (state == DAPL_EVD_STATE_WAITED) { - /* - * If we could, it would be best to avoid this wakeup - * (and the context switch) unless the number of events/CQs - * waiting for the waiter was its threshold. We don't - * currently have the ability to determine that without - * dequeueing the events, and we can't do that for - * synchronization reasons (racing with the waiter waking - * up and dequeuing, sparked by other callbacks). - */ + if (state == DAPL_EVD_STATE_OPEN && (evd_ptr->cno_ptr != NULL)) { + /* + * Re-enable callback, *then* trigger. + * This guarantees we won't miss any events. + */ + dat_status = dapl_ib_completion_notify(evd_ptr, + IB_NOTIFY_ON_NEXT_COMP); - /* - * We don't need to worry about taking the lock for the - * wakeup because wakeups are sticky. - */ - dapl_os_wait_object_wakeup(&evd_ptr->wait_object); - } else if (state == DAPL_EVD_STATE_OPEN) { - DAPL_CNO *cno = evd_ptr->cno_ptr; - if (evd_ptr->evd_enabled && (evd_ptr->cno_ptr != NULL)) { - /* - * Re-enable callback, *then* trigger. - * This guarantees we won't miss any events. - */ - dat_status = dapl_ib_completion_notify(evd_ptr, - IB_NOTIFY_ON_NEXT_COMP); - - if (DAT_SUCCESS != dat_status) { - (void)dapl_evd_post_async_error_event(evd_ptr-> - header. - owner_ia-> + if (DAT_SUCCESS != dat_status) { + (void)dapl_evd_post_async_error_event(evd_ptr-> + header. + owner_ia-> async_error_evd, DAT_ASYNC_ERROR_PROVIDER_INTERNAL_ERROR, (DAT_IA_HANDLE) evd_ptr-> header. owner_ia); - } - - dapl_cno_trigger(cno, evd_ptr); - } - } + } + dapl_cno_trigger(evd_ptr->cno_ptr, evd_ptr); + } dapl_dbg_log(DAPL_DBG_TYPE_RTN, "dapl_evd_dto_callback () returns\n"); } Index: dapl_ep_create.c =================================================================== --- dapl_ep_create.c (revision 2257) +++ dapl_ep_create.c (working copy) @@ -74,7 +74,6 @@ dapl_ep_create(DAT_IA_HANDLE ia_handle, DAPL_IA *ia_ptr; DAPL_EP *ep_ptr; DAT_EP_ATTR ep_attr_limit; - DAPL_EVD *evd_ptr; DAT_RETURN dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, @@ -194,39 +193,6 @@ dapl_ep_create(DAT_IA_HANDLE ia_handle, } } - /* - * Verify the completion flags for the EVD and the EP - */ - /* - * XXX FIXME - * XXX Need to make assign the EVD to the right completion type - * XXX depending on the EP attributes. Fail if the types don't - * XXX match, they are mutually exclusive. - */ - evd_ptr = (DAPL_EVD *) recv_evd_handle; - if (evd_ptr != NULL && evd_ptr->completion_type == DAPL_EVD_STATE_INIT) { - if (ep_attr != NULL && - ep_attr->recv_completion_flags == - DAT_COMPLETION_DEFAULT_FLAG) { - evd_ptr->completion_type = DAPL_EVD_STATE_THRESHOLD; - } else { - evd_ptr->completion_type = - ep_attr->recv_completion_flags; - } - } - - evd_ptr = (DAPL_EVD *) request_evd_handle; - if (evd_ptr != NULL && evd_ptr->completion_type == DAPL_EVD_STATE_INIT) { - if (ep_attr != NULL && - ep_attr->recv_completion_flags == - DAT_COMPLETION_DEFAULT_FLAG) { - evd_ptr->completion_type = DAPL_EVD_STATE_THRESHOLD; - } else { - evd_ptr->completion_type = - ep_attr->recv_completion_flags; - } - } - /* Allocate EP */ ep_ptr = dapl_ep_alloc(ia_ptr, ep_attr); if (ep_ptr == NULL) { Index: dapl_openib_dto.h =================================================================== --- dapl_openib_dto.h (revision 2257) +++ dapl_openib_dto.h (working copy) @@ -167,5 +167,6 @@ dapl_ib_post_send(DAPL_EP * ep_ptr, #define DAPL_GET_CQE_OPTYPE(cqe_p) ((ib_work_completion_t *)cqe_p)->opcode #define DAPL_GET_CQE_BYTESNUM(cqe_p) ((ib_work_completion_t *)cqe_p)->byte_len #define DAPL_GET_CQE_STATUS(cqe_p) ((ib_work_completion_t *)cqe_p)->status +#define DAPL_GET_CQE_VND_SYND(cqe_p) ((ib_work_completion_t *)cqe_ptr)->vendor_err #endif /* DAPL_OPENIB_DTO_H */ Index: dapl_evd_resize.c =================================================================== --- dapl_evd_resize.c (revision 2257) +++ dapl_evd_resize.c (working copy) @@ -93,13 +93,6 @@ DAT_RETURN dapl_evd_resize(DAT_EVD_HANDL spin_lock(&evd_ptr->header.lock); - /* Don't try to resize if we are actively waiting */ - if (evd_ptr->evd_state == DAPL_EVD_STATE_WAITED) { - spin_unlock(&evd_ptr->header.lock); - dat_status = DAT_ERROR(DAT_INVALID_STATE, 0); - goto bail; - } - pend_cnt = dapl_rbuf_count(&evd_ptr->pending_event_queue); if (pend_cnt > evd_qlen) { spin_unlock(&evd_ptr->header.lock); Index: dapl_evd_kcreate.c =================================================================== --- dapl_evd_kcreate.c (revision 2257) +++ dapl_evd_kcreate.c (working copy) @@ -96,16 +96,13 @@ DAT_RETURN dapl_evd_kcreate(DAT_IA_HANDL goto bail; } - cno_ptr = dapl_cno_alloc (ia_ptr, upcall); - - if (!cno_ptr) - { - dat_status = DAT_INSUFFICIENT_RESOURCES; - goto bail; - } - - cno_ptr->cno_upcall_policy = upcall_policy; + cno_ptr = dapl_cno_alloc (ia_ptr, upcall); + if (!cno_ptr) { + dat_status = DAT_INSUFFICIENT_RESOURCES; + goto bail; + } + cno_ptr->cno_upcall_policy = upcall_policy; dat_status = dapl_evd_internal_create(ia_ptr, cno_ptr, @@ -115,9 +112,6 @@ DAT_RETURN dapl_evd_kcreate(DAT_IA_HANDL goto bail; } - evd_ptr->evd_state = DAPL_EVD_STATE_OPEN; - evd_ptr->evd_enabled = DAT_TRUE; - evdhandle = *evd_handle = (DAT_EVD_HANDLE) evd_ptr; bail: Index: dapl_openib_util.h =================================================================== --- dapl_openib_util.h (revision 2257) +++ dapl_openib_util.h (working copy) @@ -89,22 +89,6 @@ typedef struct ib_hca_transport { u16 lid; } ib_hca_transport_t; -/* - * ib_shm_transport_t structure. Define fields specific to this - * provider implementation necessary to maintain shared memory state - * - * OpenIB does not have an API to directly register memory as shared - * memory. Instead, we must register memory, then register that memory - * as shared. This gives us a new mr_handle, as well as a new l_key and - * r_key, so we need to save the original handle until it can be - * disposed of when the user does an lmr_free. The l_key and r_key are - * not available and will disappear when we deallocate the mr_handle, so - * we don't save them. - */ -typedef struct ib_shm_transport { - ib_mr_handle_t mr_handle; -} ib_shm_transport_t; - #define IB_INVALID_HANDLE NULL #define IB_MAX_REQ_PDATA_SIZE 92 Index: dapl_ep_create_with_srq.c =================================================================== --- dapl_ep_create_with_srq.c (revision 2257) +++ dapl_ep_create_with_srq.c (working copy) @@ -81,7 +81,6 @@ dapl_ep_create_with_srq(DAT_IA_HANDLE ia DAPL_IA *ia_ptr; DAPL_EP *ep_ptr; DAT_EP_ATTR ep_attr_limit; - DAPL_EVD *evd_ptr; DAT_RETURN dat_status; dat_status = DAT_SUCCESS; @@ -209,39 +208,6 @@ dapl_ep_create_with_srq(DAT_IA_HANDLE ia } } - /* - * Verify the completion flags for the EVD and the EP - */ - /* - * XXX FIXME - * XXX Need to make assign the EVD to the right completion type - * XXX depending on the EP attributes. Fail if the types don't - * XXX match, they are mutually exclusive. - */ - evd_ptr = (DAPL_EVD *) recv_evd_handle; - if (evd_ptr != NULL && evd_ptr->completion_type == DAPL_EVD_STATE_INIT) { - if (ep_attr != NULL && - ep_attr->recv_completion_flags == - DAT_COMPLETION_DEFAULT_FLAG) { - evd_ptr->completion_type = DAPL_EVD_STATE_THRESHOLD; - } else { - evd_ptr->completion_type = - ep_attr->recv_completion_flags; - } - } - - evd_ptr = (DAPL_EVD *) request_evd_handle; - if (evd_ptr != NULL && evd_ptr->completion_type == DAPL_EVD_STATE_INIT) { - if (ep_attr != NULL && - ep_attr->recv_completion_flags == - DAT_COMPLETION_DEFAULT_FLAG) { - evd_ptr->completion_type = DAPL_EVD_STATE_THRESHOLD; - } else { - evd_ptr->completion_type = - ep_attr->recv_completion_flags; - } - } - dat_status = DAT_NOT_IMPLEMENTED; /* Index: dapl_evd_util.c =================================================================== --- dapl_evd_util.c (revision 2257) +++ dapl_evd_util.c (working copy) @@ -77,9 +77,6 @@ dapl_evd_internal_create(DAPL_IA * ia_pt evd_ptr->evd_producer_locking_needed = ((evd_flags & ~(DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG)) != 0); - /* Before we setup any callbacks, transition state to OPEN. */ - evd_ptr->evd_state = DAPL_EVD_STATE_OPEN; - if (evd_flags & DAT_EVD_ASYNC_FLAG) { /* * There is no cq associate with async evd. Set it to invalid @@ -161,19 +158,13 @@ DAPL_EVD *dapl_evd_alloc(DAPL_IA * ia_pt /* * Initialize the body */ - evd_ptr->evd_state = DAPL_EVD_STATE_INITIAL; + evd_ptr->evd_state = DAPL_EVD_STATE_OPEN; evd_ptr->evd_flags = evd_flags; - evd_ptr->evd_enabled = DAT_TRUE; - evd_ptr->evd_waitable = DAT_TRUE; evd_ptr->evd_producer_locking_needed = 1; /* Conservative value. */ evd_ptr->ib_cq_handle = IB_INVALID_HANDLE; atomic_set(&evd_ptr->evd_ref_count, 0); evd_ptr->catastrophic_overflow = DAT_FALSE; evd_ptr->qlen = qlen; - evd_ptr->completion_type = DAPL_EVD_STATE_THRESHOLD; /* FIXME: should be DAPL_EVD_STATE_INIT */ - dapl_os_wait_object_init(&evd_ptr->wait_object); - - evd_ptr->cno_active_count = 0; evd_ptr->cno_ptr = cno_ptr; bail: @@ -219,10 +210,6 @@ DAT_RETURN dapl_evd_event_alloc(DAPL_EVD event_ptr++; } - evd_ptr->cq_notified = DAT_FALSE; - evd_ptr->cq_notified_when = 0; - evd_ptr->threshold = 0; - bail: return dat_status; } @@ -331,7 +318,6 @@ DAT_RETURN dapl_evd_dealloc(DAPL_EVD * e kfree(evd_ptr->events); } - dapl_os_wait_object_destroy(&evd_ptr->wait_object); kfree(evd_ptr); bail: @@ -445,73 +431,20 @@ static void dapl_evd_post_event(DAPL_EVD (void *)event_ptr); dapl_os_assert(dat_status == DAT_SUCCESS); - dapl_os_assert(evd_ptr->evd_state == DAPL_EVD_STATE_WAITED - || evd_ptr->evd_state == DAPL_EVD_STATE_OPEN); + dapl_os_assert(evd_ptr->evd_state == DAPL_EVD_STATE_OPEN); if (evd_ptr->evd_state == DAPL_EVD_STATE_OPEN) { /* No waiter. Arrange to trigger a CNO if it exists. */ - - if (evd_ptr->evd_enabled) { - cno_to_trigger = evd_ptr->cno_ptr; - } - - if (evd_ptr->evd_producer_locking_needed) { - spin_unlock(&evd_ptr->header.lock); - } - } else { - /* - * We're in DAPL_EVD_STATE_WAITED. Take the lock if - * we don't have it, recheck, and signal. - */ - if (!evd_ptr->evd_producer_locking_needed) { - spin_lock(&evd_ptr->header.lock); - } - - if (evd_ptr->evd_state == DAPL_EVD_STATE_WAITED - && (dapl_rbuf_count(&evd_ptr->pending_event_queue) - >= evd_ptr->threshold)) { - spin_unlock(&evd_ptr->header.lock); - dapl_os_wait_object_wakeup(&evd_ptr->wait_object); - } else { - spin_unlock(&evd_ptr->header.lock); - } + cno_to_trigger = evd_ptr->cno_ptr; + if (evd_ptr->evd_producer_locking_needed) { + spin_unlock(&evd_ptr->header.lock); + } } - if (cno_to_trigger != NULL) { dapl_cno_trigger (cno_to_trigger, evd_ptr); } } -/* - * Post the to the evd. Do not do any wakeup processing. - * This function should only be called if it is known that there are - * no waiters that it is appropriate to wakeup on this EVD. An example - * of such a situation is during internal dat_evd_wait() processing. - * - * If producer side locking is required, the EVD lock must be held upon - * entry to this function. - */ - -static void -dapl_evd_post_event_nosignal(DAPL_EVD * evd_ptr, const DAT_EVENT * event_ptr) -{ - DAT_RETURN dat_status; - - dapl_dbg_log(DAPL_DBG_TYPE_EVD, - "dapl_evd_post_event_nonsignal: Called with event # %x\n", - event_ptr->event_number); - - dat_status = dapl_rbuf_add(&evd_ptr->pending_event_queue, - (void *)event_ptr); - dapl_os_assert(dat_status == DAT_SUCCESS); - - dapl_os_assert(evd_ptr->evd_state == DAPL_EVD_STATE_WAITED - || evd_ptr->evd_state == DAPL_EVD_STATE_OPEN); - - if (evd_ptr->evd_producer_locking_needed) { - spin_unlock(&evd_ptr->header.lock); - } -} /* * format an overflow event for posting @@ -689,49 +622,6 @@ dapl_evd_post_software_event(DAPL_EVD * } /* - * dapl_evd_post_generic_event - * - * Post a generic event type. Not used by all providers - * - * Input: - * evd_ptr - * event_number - * data - * - * Output: - * none - * - * Returns: - * DAT_SUCCESS - * - */ -DAT_RETURN -dapl_evd_post_generic_event(DAPL_EVD * evd_ptr, - DAT_EVENT_NUMBER event_number, - DAT_EVENT_DATA * data) -{ - DAT_EVENT *event_ptr; - - event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number); - /* - * Note event lock may be held on successful return - * to be released by dapl_evd_post_event(), if provider side locking - * is needed. - */ - - if (event_ptr == NULL) { - return DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, - DAT_RESOURCE_MEMORY); - } - - event_ptr->event_data = *data; - - dapl_evd_post_event(evd_ptr, event_ptr); - - return DAT_SUCCESS; -} - -/* * dapl_evd_cqe_to_event * * Convert a CQE into an event structure. @@ -866,128 +756,13 @@ dapl_evd_cqe_to_event(DAPL_EVD * evd_ptr } } /* end switch */ - /* - * Most error DTO ops result in disconnecting the EP. See - * IBTA Vol 1.1, Chapter 10,Table 68, for expected effect on - * state. - */ - if ((dto_status != DAT_DTO_SUCCESS) && - (dto_status != DAT_DTO_ERR_FLUSHED)) { -#ifdef CONN_EV_BROKEN_SUPPORTED - DAPL_EVD *evd_ptr; - - /* - * If we are connected, generate disconnect and generate an - * event. We may be racing with other disconnect ops, so we - * need to check. We may also be racing CM connection events, - * requiring us to check for connection pending states too. - */ - spin_lock(&ep_ptr->header.lock); - if (ep_ptr->param.ep_state == DAT_EP_STATE_CONNECTED || - ep_ptr->param.ep_state == - DAT_EP_STATE_ACTIVE_CONNECTION_PENDING - || ep_ptr->param.ep_state == - DAT_EP_STATE_PASSIVE_CONNECTION_PENDING - || ep_ptr->param.ep_state == - DAT_EP_STATE_COMPLETION_PENDING) { - ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; - spin_unlock(&ep_ptr->header.lock); - - /* Let the other side know we have disconnected */ - // cant call to dapl_ib_disconnect besause it is in interrupt context - // and this call is blocking need to fix ??? - // (void) dapl_ib_disconnect (ep_ptr, DAT_CLOSE_ABRUPT_FLAG); - printk("FIXME DAPL BUG %s %s %d \n", __FILE__, - __FUNCTION__, __LINE__); - /* ... and clean up the local side */ - evd_ptr = (DAPL_EVD *) ep_ptr->param.connect_evd_handle; - if (evd_ptr != NULL) { - dapl_evd_post_connection_event(evd_ptr, - DAT_CONNECTION_EVENT_BROKEN, - (DAT_HANDLE) - ep_ptr, 0, 0); - } - } else { - spin_unlock(&ep_ptr->header.lock); - } - - dapl_dbg_log(DAPL_DBG_TYPE_DTO_COMP_ERR, - " DTO completion ERROR: %d: op %#x (ep disconnected)\n", - DAPL_GET_CQE_STATUS(cqe_ptr), - DAPL_GET_CQE_OPTYPE(cqe_ptr)); -#else - dapl_dbg_log(DAPL_DBG_TYPE_DTO_COMP_ERR, - " DTO completion ERROR: %d: op %#x\n", - DAPL_GET_CQE_STATUS(cqe_ptr), - DAPL_GET_CQE_OPTYPE(cqe_ptr)); -#endif - } -} - -/* - * dapl_evd_copy_cq - * - * Copy all entries on a CQ associated with the EVD onto that EVD - * Up to caller to handle races, if any. Note that no EVD waiters will - * be awoken by this copy. - * - * Input: - * evd_ptr - * - * Output: - * None - * - * Returns: - * none - * - */ -void dapl_evd_copy_cq(DAPL_EVD * evd_ptr) -{ - ib_work_completion_t cur_cqe; - DAT_RETURN dat_status; - DAT_EVENT *event; - - if (evd_ptr->ib_cq_handle == IB_INVALID_HANDLE) { - /* Nothing to do if no CQ. */ - return; - } - - while (1) { - dat_status = - dapl_ib_completion_poll(evd_ptr->header.owner_ia->hca_ptr, - evd_ptr, &cur_cqe); - - if (dat_status != DAT_SUCCESS) { - break; - } - - /* For debugging. */ - dapl_evd_eh_print_cqe(&cur_cqe); - - /* - * Can use DAT_DTO_COMPLETION_EVENT because dapl_evd_cqe_to_event - * will overwrite. - */ - - event = - dapl_evd_get_and_init_event(evd_ptr, - DAT_DTO_COMPLETION_EVENT); - if (event == NULL) { - /* We've already attempted the overflow post; return. */ - return; - } - - dapl_evd_cqe_to_event(evd_ptr, &cur_cqe, event); - - dapl_evd_post_event_nosignal(evd_ptr, event); - } - - if (DAT_GET_TYPE(dat_status) != DAT_QUEUE_EMPTY) { - dapl_dbg_log(DAPL_DBG_TYPE_EVD, - "dapl_evd_copy_cq: dapl_ib_completion_poll returned 0x%x\n", - dat_status); - dapl_os_assert(!"Bad return from dapl_ib_completion_poll"); - } + if (dto_status != DAT_DTO_SUCCESS) { + dapl_dbg_log(DAPL_DBG_TYPE_DTO_COMP_ERR, + " DTO completion ERROR:status %d: op %#x vendore synd: 0x%x\n", + DAPL_GET_CQE_STATUS (cqe_ptr), + DAPL_GET_CQE_OPTYPE (cqe_ptr), + DAPL_GET_CQE_VND_SYND(cqe_ptr)); + } } /* Index: dapl_evd_util.h =================================================================== --- dapl_evd_util.h (revision 2257) +++ dapl_evd_util.h (working copy) @@ -87,10 +87,6 @@ DAT_RETURN dapl_evd_post_software_event(DAPL_EVD * evd_ptr, DAT_EVENT_NUMBER event_number, DAT_PVOID pointer); -DAT_RETURN -dapl_evd_post_generic_event(DAPL_EVD * evd_ptr, - DAT_EVENT_NUMBER event_number, - DAT_EVENT_DATA * data); /************************************* * dapl internal callbacks functions * @@ -115,8 +111,6 @@ extern void dapl_evd_cq_async_error_call extern void dapl_evd_qp_async_error_callback(ib_error_record_t * cause_ptr, void *context); -extern void dapl_evd_copy_cq(DAPL_EVD * evd_ptr); - extern DAT_RETURN dapl_evd_cq_poll_to_event(DAPL_EVD * evd_ptr, DAT_EVENT * event); Index: dapl.h =================================================================== --- dapl.h (revision 2257) +++ dapl.h (working copy) @@ -64,10 +64,7 @@ typedef enum dapl_magic { } DAPL_MAGIC; typedef enum dapl_evd_state { - DAPL_EVD_STATE_TERMINAL, - DAPL_EVD_STATE_INITIAL, DAPL_EVD_STATE_OPEN, - DAPL_EVD_STATE_WAITED, DAPL_EVD_STATE_DEAD = 0xDEAD } DAPL_EVD_STATE; @@ -270,8 +267,6 @@ struct dapl_evd DAPL_EVD_STATE evd_state; DAT_EVD_FLAGS evd_flags; - DAT_BOOLEAN evd_enabled; /* For attached CNO. */ - DAT_BOOLEAN evd_waitable; /* EVD state. */ /* Derived from evd_flags; see dapls_evd_internal_create. */ DAT_BOOLEAN evd_producer_locking_needed; @@ -279,9 +274,6 @@ struct dapl_evd /* Every EVD has a CQ unless it is a SOFTWARE_EVENT only EVD */ ib_cq_handle_t ib_cq_handle; - /* Mellanox Specific completion handle for registration/de-registration */ - ib_comp_handle_t ib_comp_handle; - /* An Event Dispatcher cannot be freed while * it is referenced elsewhere. */ @@ -295,20 +287,7 @@ struct dapl_evd DAT_EVENT *events; DAPL_RING_BUFFER free_event_queue; DAPL_RING_BUFFER pending_event_queue; - - /* CQ Completions are not placed into 'deferred_events' - ** rather they are simply left on the Completion Queue - ** and the fact that there was a notification is flagged. - */ - DAT_BOOLEAN cq_notified; - DAPL_OS_TICKS cq_notified_when; - - DAT_COUNT cno_active_count; DAPL_CNO *cno_ptr; - - DAPL_OS_WAIT_OBJECT wait_object; - DAT_COUNT threshold; - DAPL_EVD_COMPLETION completion_type; }; /* uDAPL timer entry, used to queue timeouts */ -- Itamar From robert.j.woodruff at intel.com Thu May 5 08:30:05 2005 From: robert.j.woodruff at intel.com (Bob Woodruff) Date: Thu, 5 May 2005 08:30:05 -0700 Subject: [openib-general] PF 5/4 Telecon Minutes In-Reply-To: <20050505001734.GD16887@esmail.cup.hp.com> Message-ID: Hal wrote, >Hal, >Which paper are you referring to? >I just checked: > http://www.linuxsymposium.org/2005/speakers.php?types=TALK >and didn't see about infiniband. >Ah, found it under BOFs: > http://www.linuxsymposium.org/2005/view_abstract.php?content_key=87 >Plan on handing out printed copies or post a URL where it's >to find (e.g. openib.org/talks/ols2005). >thanks, >grant I just noticed this too and sent an email to AJ. Apparently, there was a mix-up and even though I received a confirmation that the paper had been accepted, it was suppose to be an acceptance of a BOF instead, so they gave us a BOF time slot rather than a presentation time slot. He did say that we can submit the paper for printing in the proceedings, so it really does not matter to much if we present it in a presentation slot or a BOF time slot. woody From robert.j.woodruff at intel.com Thu May 5 08:38:32 2005 From: robert.j.woodruff at intel.com (Bob Woodruff) Date: Thu, 5 May 2005 08:38:32 -0700 Subject: [openib-general] RE: [OOPS] user-mode verbs In-Reply-To: <527jie72vh.fsf@topspin.com> Message-ID: Roland wrote, >It turns out I made this mistake a bunch of places in uverbs_cmd.c, so >I won't have a fix until tomorrow. Ok, let me know when you have the fixes checked in and I will do a fresh pull from SVN. woody. From eitan at mellanox.co.il Thu May 5 08:45:49 2005 From: eitan at mellanox.co.il (Eitan Zahavi) Date: Thu, 5 May 2005 18:45:49 +0300 Subject: [openib-general] OpenSM Routing Scalability Proposal Message-ID: <506C3D7B14CDD411A52C00025558DED607C30115@mtlex01.yok.mtl.com> Hi All, Here are some of my thoughts/proposals regarding faster and low memory profile OpenSM routing solution. Any feedback is welcome. I am not comfortable with the complexity order numbers and will highly appreciate your double check. The "persistency" area is also in its initial stages. <> Thanks Eitan Zahavi Design Technology Director Mellanox Technologies LTD Tel:+972-4-9097208 Fax:+972-4-9593245 P.O. Box 586 Yokneam 20692 ISRAEL -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OsmRoutingScalability.pdf Type: application/octet-stream Size: 43508 bytes Desc: not available URL: From blist at aon.at Thu May 5 08:58:03 2005 From: blist at aon.at (Bernhard Fischer) Date: Thu, 5 May 2005 17:58:03 +0200 Subject: [openib-general] 0 op factor Message-ID: <20050505155803.GS15688@aon.at> Hi, I forgot, but what's the purpose of statements like: $ egrep -ri "[^[:digit:]|\.]0[[:space:]]*[-+*/]" *|egrep -v "(\.(svn-base|txt):|/\*)" infiniband/ulp/sdp/sdp_inet.c: * flag: 0 - recv shutdown infiniband/ulp/sdp/sdp_actv.c: sdp_iocb_q_cancel_all(conn, (0 - error)); infiniband/ulp/sdp/sdp_actv.c: sdp_cm_actv_error(conn, (0 - result)); infiniband/ulp/sdp/sdp_actv.c: sdp_cm_actv_error(conn, (0 - status)); infiniband/ulp/sdp/sdp_pass.c: sdp_iocb_q_cancel_all(conn, (0 - error)); infiniband/ulp/sdp/sdp_pass.c: sdp_cm_pass_error(conn, (0 - result)); infiniband/ulp/sdp/sdp_rcvd.c: 0 - ((SDP_SRC_AVAIL_RECV > size) ? size : 0); infiniband/hw/mthca/mthca_cmd.c: __raw_writel(cpu_to_be32(in_param >> 32), dev->hcr + 0 * 4); Could somebody please elaborate? thanks From tduffy at sun.com Thu May 5 09:42:15 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 05 May 2005 09:42:15 -0700 Subject: [openib-general] [kDAPL][PATCH] fix panic in server side In-Reply-To: References: Message-ID: <1115311335.25119.10.camel@duffman> On Thu, 2005-05-05 at 08:44 +0300, Or Gerlitz wrote: > Tom, > > >This occurred when I did a connect without first setting up a listen > using kdapltest. > >RIP: 0010:[] > {:ib_dat_provider:dapl_cm_passive_cb_handler+27} > >Call Trace:{:ib_cm:cm_process_work+50} > {:ib_cm:cm_work_handler+1438} > > If you you haven't set up a (kdapltest server) listener - why did the CM > invoked a (kdapl provider) consumer callback for this SID? I have no idea. I loaded the kdapltest.ko kernel modules, but had not started the kdapltest userland program when this occured. I am pretty sure I had rebooted right before trying this, but it could have been that I had just unloaded and reloaded all the (dapl) modules (so maybe something was left from a previous incarnation). I will reboot both machines and try to reproduce. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From roland at topspin.com Thu May 5 09:59:53 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 05 May 2005 09:59:53 -0700 Subject: [openib-general] Re: [OOPS] user-mode verbs In-Reply-To: (Bob Woodruff's message of "Thu, 5 May 2005 08:38:32 -0700") References: Message-ID: <52mzr9622e.fsf@topspin.com> Bob> Ok, let me know when you have the fixes checked in and I will Bob> do a fresh pull from SVN. I just checked in a new version of uverbs_cmd.c that should fix the use-after-free bugs. - R. From blist at aon.at Thu May 5 10:09:56 2005 From: blist at aon.at (Bernhard Fischer) Date: Thu, 5 May 2005 19:09:56 +0200 Subject: [openib-general] [patch] misc trivia nitpicks 01 Message-ID: <20050505170956.GU15688@aon.at> Hi, please consider attached. diffstat ./gen2.2257.cow-20050505-1902.diff linux-kernel/infiniband/ulp/sdp/sdp_actv.c | 27 +-- userspace/libmthca/src/mthca.c | 4 userspace/management/osm/include/opensm/osm_subnet.h | 79 +++++----- userspace/management/osm/include/vendor/osm_vendor_mtl_transaction_mgr.h | 43 ++--- userspace/management/osm/opensm/osm_sa_service_record.c | 8 - userspace/management/osm/opensm/osm_sm_state_mgr.c | 38 ++-- userspace/management/osm/opensm/osm_sw_info_rcv.c | 9 - userspace/management/osm/opensm/osm_trap_rcv.c | 62 +++---- 8 files changed, 134 insertions(+), 136 deletions(-) Thank you, -------------- next part -------------- - s/reciev/receiv/g; trim superfluous whitespace while touching those. - src/linux-kernel/infiniband/ulp/sdp/sdp_actv.c (sdp_cm_actv_error): trim return off void(). - src/userspace/libmthca/src/mthca.c (openib_driver_init): vendor is unsigned. -------------- next part -------------- diff -X excl -rduNp gen2.2257.oorig/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_actv.c gen2.2257/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_actv.c --- gen2.2257.oorig/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_actv.c 2005-04-26 20:58:46.000000000 +0200 +++ gen2.2257/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_actv.c 2005-05-05 17:20:25.968369544 +0200 @@ -43,7 +43,7 @@ void sdp_cm_actv_error(struct sdp_opt *c /* * error value is positive error. * - * Handle errors within active connections stream. + * Handle errors within active connections stream. * First generate appropriate response, REJ, DREQ or nothing. * Second the socket must be notified of the error. */ @@ -59,14 +59,14 @@ void sdp_cm_actv_error(struct sdp_opt *c break; case SDP_CONN_ST_ERROR_STRM: /* socket has been destroyed. */ error = ECONNRESET; - case SDP_CONN_ST_REQ_SENT: + case SDP_CONN_ST_REQ_SENT: case SDP_CONN_ST_REP_RECV: case SDP_CONN_ST_RTU_SENT: /* * All four states we have gotten a REP and are now in * one of these states. */ - result = ib_send_cm_rej(conn->cm_id, + result = ib_send_cm_rej(conn->cm_id, IB_CM_REJ_CONSUMER_DEFINED, NULL, 0, NULL, 0); @@ -78,14 +78,14 @@ void sdp_cm_actv_error(struct sdp_opt *c break; case SDP_CONN_ST_ESTABLISHED: /* - * Made it all the way to esablished, need to initiate a + * Made it all the way to established, need to initiate a * full disconnect. */ result = ib_send_cm_dreq(conn->cm_id, NULL, 0); if (result < 0) - sdp_dbg_warn(NULL, "Error <%d> sending CM DREQ", + sdp_dbg_warn(NULL, "Error <%d> sending CM DREQ", result); - + conn->state = SDP_CONN_ST_TIME_WAIT_1; break; } @@ -100,7 +100,6 @@ void sdp_cm_actv_error(struct sdp_opt *c sdp_iocb_q_cancel_all(conn, (0 - error)); sdp_inet_wake_error(conn->sk); - return; } /* @@ -113,8 +112,8 @@ static int sdp_cm_actv_establish(struct struct sock *sk; int result; - sdp_dbg_ctrl(conn, "active etablish. src <%08x:%04x> dst <%08x:%04x>", - conn->src_addr, conn->src_port, + sdp_dbg_ctrl(conn, "active establish. src <%08x:%04x> dst <%08x:%04x>", + conn->src_addr, conn->src_port, conn->dst_addr, conn->dst_port); sk = conn->sk; @@ -134,7 +133,7 @@ static int sdp_cm_actv_establish(struct } qp_attr->rq_psn = conn->rq_psn; - + attr_mask |= IB_QP_RQ_PSN; result = ib_modify_qp(conn->qp, qp_attr, attr_mask); @@ -190,7 +189,7 @@ static int sdp_cm_actv_establish(struct * release disconnects. */ conn->flags &= ~SDP_CONN_F_DIS_HOLD; - + inet_sk(sk)->saddr = htonl(conn->src_addr); inet_sk(sk)->rcv_saddr = htonl(conn->src_addr); @@ -204,7 +203,7 @@ static int sdp_cm_actv_establish(struct */ sdp_inet_wake_send(sk); sdp_inet_wake_recv(sk, 0); - + result = 0; done: kfree(qp_attr); @@ -273,7 +272,7 @@ int sdp_cm_rep_handler(struct ib_cm_id * { struct sdp_msg_hello_ack *hello_ack; int result = -EPROTO; - + if (cm_id != conn->cm_id) { sdp_dbg_warn(conn, "REP comm ID mismatch. <%08x:%08x>", conn->cm_id->local_id, cm_id->local_id); @@ -310,7 +309,7 @@ int sdp_cm_rep_handler(struct ib_cm_id * conn->d_qpn = event->param.rep_rcvd.remote_qpn; /* * The maximum amount of data that can be sent to the remote - * peer is the smaller of the local and remote buffer sizes, + * peer is the smaller of the local and remote buffer sizes, * minus the size of the message header. */ conn->send_size = min((u16)sdp_buff_pool_buff_size(), diff -X excl -rduNp gen2.2257.oorig/trunk/src/userspace/libmthca/src/mthca.c gen2.2257/trunk/src/userspace/libmthca/src/mthca.c --- gen2.2257.oorig/trunk/src/userspace/libmthca/src/mthca.c 2005-04-26 20:58:01.000000000 +0200 +++ gen2.2257/trunk/src/userspace/libmthca/src/mthca.c 2005-05-05 14:00:11.916783000 +0200 @@ -220,13 +220,13 @@ struct ibv_device *openib_driver_init(st attr = sysfs_get_device_attr(pcidev, "vendor"); if (!attr) return NULL; - sscanf(attr->value, "%i", &vendor); + sscanf(attr->value, "%u", &vendor); sysfs_close_attribute(attr); attr = sysfs_get_device_attr(pcidev, "device"); if (!attr) return NULL; - sscanf(attr->value, "%i", &device); + sscanf(attr->value, "%u", &device); sysfs_close_attribute(attr); for (i = 0; i < sizeof hca_table / sizeof hca_table[0]; ++i) diff -X excl -rduNp gen2.2257.oorig/trunk/src/userspace/management/osm/include/opensm/osm_subnet.h gen2.2257/trunk/src/userspace/management/osm/include/opensm/osm_subnet.h --- gen2.2257.oorig/trunk/src/userspace/management/osm/include/opensm/osm_subnet.h 2005-04-26 20:58:23.000000000 +0200 +++ gen2.2257/trunk/src/userspace/management/osm/include/opensm/osm_subnet.h 2005-05-05 18:59:01.310100912 +0200 @@ -37,12 +37,12 @@ /* * Abstract: - * Declaration of osm_subn_t. + * Declaration of osm_subn_t. * This object represents an IBA subnet. * This object is part of the OpenSM family of objects. * * Environment: - * Linux User Mode + * Linux User Mode * * $Revision: 1.12 $ */ @@ -112,7 +112,7 @@ typedef void * context * [in] Client specific context specified in the subnet opt * Same prefix as the UI funciton (suffixed by ctx) -* +* * RETURN VALUE * This function does not return a value. * @@ -259,13 +259,14 @@ typedef struct _osm_subn_opt * Limit the maximal operational VLs. default is 1. * * reassign_lids -* If TRUE cause all lids to be re-assigend. Otherwise (teh default) +* If TRUE cause all lids to be re-assigend. +* Otherwise (the default) * OpenSM always try to preserve as much LIDs as posible. * * reassign_lfts * If TRUE ignore existing LFT entries on first sweep (default). * Otherwise only non minimal hop cases are modified. -* NOTE: A standby SM clears its first sweep flag - since the +* NOTE: A standby SM clears its first sweep flag - since the * master SM already sweeps... * * ignore_other_sm_option @@ -273,10 +274,10 @@ typedef struct _osm_subn_opt * * no_multicast_option * This flag is TRUE if OpenSM should disable multicast support. -* +* * subnet_timeout * The subnet_timeout that will be set for all the ports in the -* design SubnMgt.Set(PortInfo.vl_stall_life)) +* design SubnMgt.Set(PortInfo.vl_stall_life)) * * head_of_queue_lifetime * The max imal time a packet can live at the head of a VL queue @@ -284,7 +285,7 @@ typedef struct _osm_subn_opt * local_phy_errors_threshold * Threshold of local phy errors for sending Trap 129 * -* overrun_errors_threshold +* overrun_errors_threshold * Threshold of credits overr-run errors for sending Trap 129 * * packet_life_time @@ -302,9 +303,9 @@ typedef struct _osm_subn_opt * the link. If FALSE - only CA/RT nodes are counted. * * max_port_profile -* Prevent routing through a port subscribed with more then this +* Prevent routing through a port subscribed with more then this * number of routes. -* +* * pfn_ui_pre_lid_assign * A UI function to be invoked prior to lid assigment. * @@ -312,16 +313,16 @@ typedef struct _osm_subn_opt * A UI context (void *) to be provided to the pfn_ui_pre_lid_assign * * pfn_ui_ucast_fdb_assign -* A UI function to be called instead of the ucast manager FDB -* configuration. +* A UI function to be called instead of the ucast manager FDB +* configuration. * * ui_ucast_fdb_assign_ctx * A UI context (void *) to be provided to the pfn_ui_ucast_fdb_assign * * pfn_ui_mcast_fdb_assign -* A UI function to be called inside the mcast manager instead of the -* call for the build spanning tree. This will be called on every -* multicast call for create, join and leave, and is responsible for +* A UI function to be called inside the mcast manager instead of the +* call for the build spanning tree. This will be called on every +* multicast call for create, join and leave, and is responsible for * the mcast FDB configuration. * * ui_mcast_fdb_assign_ctx @@ -427,8 +428,8 @@ typedef struct _osm_subn * master_sm_base_lid * The base LID owned by the subnet's master SM. * -* sm_base_lid -* The base LID of the local port where the SM is. +* sm_base_lid +* The base LID of the local port where the SM is. * * sm_port_guid * This SM's own port GUID. @@ -456,17 +457,17 @@ typedef struct _osm_subn * - Set to FALSE on end of all lft assignments. * * subnet_initalization_error -* Similar to the force_immediate_heavy_sweep flag. If TRUE - +* Similar to the force_immediate_heavy_sweep flag. If TRUE - * means that we had errors during initialization (due to SubnSet requests -* that failed). We want to declare the subnet as un-healthy, and force -* another heavy sweep. +* that failed). We want to declare the subnet as un-healthy, and force +* another heavy sweep. * force_immediate_heavy_sweep * If TRUE - we want to force a heavy sweep. This can be done either -* due to recieving of trap - meaning there is some change on the subnet, -* or we recieved a handover from a remote sm. +* due to receiving of trap - meaning there is some change on the subnet, +* or we received a handover from a remote sm. * In this case we want to sweep and reconfigure the entire subnet. -* This will cause another heavy sweep to occure when the current sweep +* This will cause another heavy sweep to occure when the current sweep * is done. * * force_delayed_heavy_sweep @@ -486,7 +487,7 @@ typedef struct _osm_subn * in sweep_hop_0 - meaning we do not want to continue beyond * the current node. * This is relevant for the case of SM on switch, since in the -* switch info we need to signal somehow not to continue +* switch info we need to signal somehow not to continue * the sweeping. * * moved_to_master_state @@ -498,7 +499,7 @@ typedef struct _osm_subn * This flag is used for the PortInfo setting. On the first sweep as master * (meaning after moving from Standby|Discovering state), the SM must send * a PortInfoSet to all ports. After that - we want to minimize the number of -* PortInfoSet requests sent, and to send only requests that change the value +* PortInfoSet requests sent, and to send only requests that change the value * from what is updated in the port (or send a first request if this is a new port). * We will set this flag to TRUE when entering the master state, and set it back * to FALSE at the end of the drop manager. This is done since at the end of the @@ -618,8 +619,8 @@ struct _osm_port; * * DESCRIPTION * Looks for the requestor gid in the mad address. -* -* Note: This code is not thread safe. Need to grab the lock before +* +* Note: This code is not thread safe. Need to grab the lock before * calling it. * * SYNOPSIS @@ -654,8 +655,8 @@ osm_get_gid_by_mad_addr( * * DESCRIPTION * Looks for the requestor physical port in the mad address. -* -* Note: This code is not thread safe. Need to grab the lock before +* +* Note: This code is not thread safe. Need to grab the lock before * calling it. * * SYNOPSIS @@ -690,8 +691,8 @@ osm_get_physp_by_mad_addr( * * DESCRIPTION * Looks for the requestor port in the mad address. -* -* Note: This code is not thread safe. Need to grab the lock before +* +* Note: This code is not thread safe. Need to grab the lock before * calling it. * * SYNOPSIS @@ -726,7 +727,7 @@ osm_get_port_by_mad_addr( * * DESCRIPTION * The looks for the given switch guid in the subnet table of switches by guid. -* NOTE: this code is not thread safe. Need to grab the lock before +* NOTE: this code is not thread safe. Need to grab the lock before * calling it. * * SYNOPSIS @@ -738,7 +739,7 @@ osm_get_switch_by_guid( /* * PARAMETERS * p_subn -* [in] Pointer to an osm_subn_t object +* [in] Pointer to an osm_subn_t object * * guid * [in] The node guid in host order @@ -758,7 +759,7 @@ osm_get_switch_by_guid( * * DESCRIPTION * The looks for the given node giud in the subnet table of nodes by guid. -* NOTE: this code is not thread safe. Need to grab the lock before +* NOTE: this code is not thread safe. Need to grab the lock before * calling it. * * SYNOPSIS @@ -770,7 +771,7 @@ osm_get_node_by_guid( /* * PARAMETERS * p_subn -* [in] Pointer to an osm_subn_t object +* [in] Pointer to an osm_subn_t object * * guid * [in] The node guid in host order @@ -790,7 +791,7 @@ osm_get_node_by_guid( * * DESCRIPTION * The looks for the given port guid in the subnet table of ports by guid. -* NOTE: this code is not thread safe. Need to grab the lock before +* NOTE: this code is not thread safe. Need to grab the lock before * calling it. * * SYNOPSIS @@ -802,7 +803,7 @@ osm_get_port_by_guid( /* * PARAMETERS * p_subn -* [in] Pointer to an osm_subn_t object +* [in] Pointer to an osm_subn_t object * * guid * [in] The port guid in host order @@ -822,8 +823,8 @@ osm_get_port_by_guid( * * DESCRIPTION * Looks for the requestor physical port in the mad address. -* -* Note: This code is not thread safe. Need to grab the lock before +* +* Note: This code is not thread safe. Need to grab the lock before * calling it. * * SYNOPSIS diff -X excl -rduNp gen2.2257.oorig/trunk/src/userspace/management/osm/include/vendor/osm_vendor_mtl_transaction_mgr.h gen2.2257/trunk/src/userspace/management/osm/include/vendor/osm_vendor_mtl_transaction_mgr.h --- gen2.2257.oorig/trunk/src/userspace/management/osm/include/vendor/osm_vendor_mtl_transaction_mgr.h 2005-04-26 20:58:29.000000000 +0200 +++ gen2.2257/trunk/src/userspace/management/osm/include/vendor/osm_vendor_mtl_transaction_mgr.h 2005-05-05 18:55:55.240387808 +0200 @@ -38,11 +38,11 @@ /* * Abstract: - * Definition of interface for the MTL Vendor + * Definition of interface for the MTL Vendor * This object is part of the OpenSM family of objects. * * Environment: - * Linux User Mode + * Linux User Mode * * $Revision: 1.4 $ */ @@ -108,13 +108,14 @@ typedef struct _osm_madw_req * List item for qlist linkage. Must be first element!! * * map_item -* Map item for qmap linkage. +* Map item for qmap linkage. * -* p_madw +* p_madw * pointer to mad wrapper that is expecting to get a response. * * waking_time -* Time stamp (in microseconds) when the p_madw needs to wake up. This value is +* Time stamp (in microseconds) when the p_madw needs to wake up. +* This value is * cl_get_time_stamp() + timeout during the sending of the mad. * when timeout should be given in microseconds. * @@ -128,7 +129,7 @@ typedef struct _osm_madw_req * * DESCRIPTION * This structure defines the transaction manager. -* It holds a qlist and a qmap, a lock on the transaction manager, and +* It holds a qlist and a qmap, a lock on the transaction manager, and * a timer used for the list. * The manager is responsible for keeping track of every request mad that was * sent. It is used for finding mads according to their transaction id, and for @@ -136,10 +137,10 @@ typedef struct _osm_madw_req * a response and didn't get one by the timeout time expected. * * Both the list and the map hold the osm_madw_req_t objects - one for every madw. -* +* * Managing of the list: * The timer wakes on the timeout of the first madw. If the waking_time is greater than -* the current time - then the mad recieved a response. If not - the mad didn't get +* the current time - then the mad received a response. If not - the mad didn't get * its response. * * SYNOPSIS @@ -155,12 +156,12 @@ osm_transaction_mgr_t; /* * FIELDS -* madw_by_tid_map_p +* madw_by_tid_map_p * A qmap with key = transaction id. and value of osm_madw_req_t. * -* madw_reqs_list_p +* madw_reqs_list_p * A qlist of all the madw with their waking time. -* +* * transaction_mgr_lock * Lock used on the transaction manager - make sure changes on it are serial. * @@ -173,13 +174,13 @@ osm_transaction_mgr_t; * osm_transaction_mgr_init * * DESCRIPTION -* Initialize the transaction manager. +* Initialize the transaction manager. * Will update the p_transaction_mgr in the vendor object with * the new Transaction Manager created.* * * SYNOPSIS */ -void +void osm_transaction_mgr_init( IN osm_vendor_t * const p_vend ); /* @@ -195,13 +196,13 @@ osm_transaction_mgr_init( IN osm_vendor_ * osm_transaction_mgr_destroy * * DESCRIPTION -* Destroy the transaction manager. -* Will de-allocate all memory allocated by the Transaction +* Destroy the transaction manager. +* Will de-allocate all memory allocated by the Transaction * Manager up to now. * * SYNOPSIS */ -void +void osm_transaction_mgr_destroy ( IN osm_vendor_t * const p_vend ); /* @@ -243,9 +244,9 @@ osm_transaction_mgr_insert_madw( IN osm_ * osm_transaction_mgr_erase_madw * * DESCRIPTION -* Erase a madw object from the manager. +* Erase a madw object from the manager. * The removal is done using the transaction id of the mad - using -* it the madw_p is allocated (in the qmap) and removed from the +* it the madw_p is allocated (in the qmap) and removed from the * qmap and qlist. * * SYNOPSIS @@ -259,7 +260,7 @@ osm_transaction_mgr_erase_madw( IN osm_v * [in] Pointer to a Osm Vendor object. * * p_mad -* [in] Pointer to the Mad to be removed. +* [in] Pointer to the Mad to be removed. * *********/ @@ -285,7 +286,7 @@ osm_transaction_mgr_get_madw_for_tid( IN * [in] Pointer to the Mad to be located. * * req_madw_p -* [out] Pointer to the mad Wrapper to be found. +* [out] Pointer to the mad Wrapper to be found. * *********/ @@ -298,7 +299,7 @@ osm_transaction_mgr_get_madw_for_tid( IN * This callback is called on timeout of the timer. * It checks the time of the head madw in the qlist, and compares it to * the current time. -* Will send an error callback if the time of the madw is less than the +* Will send an error callback if the time of the madw is less than the * current time - this means that the madw wasn't removed in the timeout * it was supposed to be handled. * diff -X excl -rduNp gen2.2257.oorig/trunk/src/userspace/management/osm/opensm/osm_sa_service_record.c gen2.2257/trunk/src/userspace/management/osm/opensm/osm_sa_service_record.c --- gen2.2257.oorig/trunk/src/userspace/management/osm/opensm/osm_sa_service_record.c 2005-04-26 20:58:16.000000000 +0200 +++ gen2.2257/trunk/src/userspace/management/osm/opensm/osm_sa_service_record.c 2005-05-05 18:50:17.450739640 +0200 @@ -732,7 +732,7 @@ __get_matching_sr( } /* Check that the requestor port has the pkey which is the service_pkey. - If not - then it cannot recieve this serviceRecord. */ + If not - then it cannot receive this serviceRecord. */ /* The check is relevant only if the service_pkey is valid */ if (!ib_pkey_is_invalid(p_svcr->service_record.service_pkey)) { @@ -743,7 +743,7 @@ __get_matching_sr( osm_log( p_sr_item->p_rcv->p_log, OSM_LOG_VERBOSE, "__get_matching_sr: " "requestor port doesn't have the service_pkey: 0x%X\n", - cl_ntoh16(p_svcr->service_record.service_pkey) ); + cl_ntoh16(p_svcr->service_record.service_pkey) ); return; } } @@ -1154,8 +1154,8 @@ osm_sr_rcv_lease_cb( if(elapsed_time < p_svcr->lease_period) { - /* - Just update the service lease period + /* + Just update the service lease period note: for simplicity we work with a uint32_t field external to the network order lease_period of the MAD */ diff -X excl -rduNp gen2.2257.oorig/trunk/src/userspace/management/osm/opensm/osm_sm_state_mgr.c gen2.2257/trunk/src/userspace/management/osm/opensm/osm_sm_state_mgr.c --- gen2.2257.oorig/trunk/src/userspace/management/osm/opensm/osm_sm_state_mgr.c 2005-04-26 20:58:16.000000000 +0200 +++ gen2.2257/trunk/src/userspace/management/osm/opensm/osm_sm_state_mgr.c 2005-05-05 18:52:54.883806176 +0200 @@ -75,7 +75,7 @@ static void __osm_sm_state_mgr_standby_msg( IN const osm_sm_state_mgr_t* p_sm_mgr ) { - osm_log( p_sm_mgr->p_log, OSM_LOG_SYS, + osm_log( p_sm_mgr->p_log, OSM_LOG_SYS, "Entering STANDBY state"); if( osm_log_is_active( p_sm_mgr->p_log, OSM_LOG_VERBOSE ) ) @@ -221,18 +221,18 @@ __osm_sm_state_mgr_send_master_sm_info_r { /* We are in STANDBY state - this means we need to poll on the master - SM (according to master_guid) + SM (according to master_guid) Send a query of SubnGet(SMInfo) to the subn master_sm_base_lid object. */ p_port = (osm_port_t*)cl_qmap_get( &p_sm_mgr->p_subn->port_guid_tbl, p_sm_mgr->master_guid ); } - else + else { - /* + /* We are not in STANDBY - this means we are in MASTER state - so we need - to poll on the SM that is saved in p_polling_sm under p_sm_mgr. - Send a query of SubnGet(SMInfo) to that SM. + to poll on the SM that is saved in p_polling_sm under p_sm_mgr. + Send a query of SubnGet(SMInfo) to that SM. */ p_port = p_sm_mgr->p_polling_sm->p_port; } @@ -324,13 +324,13 @@ __osm_sm_state_mgr_polling_callback( IN If we are not in one of these cases - don't need to restart the poller. */ if (!( (p_sm_mgr->p_subn->sm_state == IB_SMINFO_STATE_MASTER && - p_sm_mgr->p_polling_sm != NULL ) || + p_sm_mgr->p_polling_sm != NULL ) || (p_sm_mgr->p_subn->sm_state == IB_SMINFO_STATE_STANDBY ) ) ) { goto Exit; } - /* + /* If we are a STANDBY sm, and the osm_exit_flag is 1, then let's signal the subnet_up. This is relevant for the case of running only once. It that case - the program is stuck until this signal is received. In other cases - @@ -346,7 +346,7 @@ __osm_sm_state_mgr_polling_callback( IN cl_event_signal( p_sm_mgr->p_state_mgr->p_subnet_up_event ); goto Exit; } - + /* Incr the retry number. If it reached the max_retry_number in the subnet opt - call @@ -577,7 +577,7 @@ osm_sm_state_mgr_process( */ __osm_sm_state_mgr_standby_msg(p_sm_mgr); p_sm_mgr->p_subn->sm_state = IB_SMINFO_STATE_STANDBY; - /* + /* Since another SM is doing the LFT config - we should not ignore the results of it */ @@ -674,23 +674,23 @@ osm_sm_state_mgr_process( { case OSM_SM_SIGNAL_POLLING_TIMEOUT: /* - we recieved a polling timeout - this means that we waited for + we received a polling timeout - this means that we waited for a remote master sm to send us a handover, but didn't get it, and didn't get a response from that remote sm. We want to force a heavy sweep - hopefully this occurred because the remote sm died, and we'll find this out and configure the - subnet after a heavy sweep. - We also want to clear the p_polling_sm object - since we are + subnet after a heavy sweep. + We also want to clear the p_polling_sm object - since we are done polling on that remote sm - we are sweeping again. */ case OSM_SM_SIGNAL_HANDOVER: /* - If we recieved a handover in a master state - then we want to + If we received a handover in a master state - then we want to force a heavy sweep. This means that either we are in a sweep - currently - in this case - no change, or we are in idle state - + currently - in this case - no change, or we are in idle state - since we recognized a master SM before - so we want to make a heavy sweep and reconfigure the new subnet. - We also want to clear the p_polling_sm object - since we are + We also want to clear the p_polling_sm object - since we are done polling on that remote sm - we got a handover from it. */ osm_log( p_sm_mgr->p_log, OSM_LOG_VERBOSE, @@ -712,12 +712,12 @@ osm_sm_state_mgr_process( __osm_sm_state_mgr_start_polling(p_sm_mgr); break; case OSM_SM_SIGNAL_WAIT_FOR_HANDOVER: - /* + /* We found a remote master SM, and we are waiting for it - to handover the mastership to us. Need to start polling + to handover the mastership to us. Need to start polling on that SM, to make sure it is alive, if it isn't - then we should move back to discovering, since something must - have happend to it. + have happend to it. */ __osm_sm_state_mgr_start_polling(p_sm_mgr); break; diff -X excl -rduNp gen2.2257.oorig/trunk/src/userspace/management/osm/opensm/osm_sw_info_rcv.c gen2.2257/trunk/src/userspace/management/osm/opensm/osm_sw_info_rcv.c --- gen2.2257.oorig/trunk/src/userspace/management/osm/opensm/osm_sw_info_rcv.c 2005-04-26 20:58:16.000000000 +0200 +++ gen2.2257/trunk/src/userspace/management/osm/opensm/osm_sw_info_rcv.c 2005-05-05 18:54:21.451645864 +0200 @@ -420,10 +420,10 @@ __osm_si_rcv_process_new( osm_switch_delete( &p_sw ); goto Exit; } - + /* - Update the switch info according to the - info we just recieved. + Update the switch info according to the + info we just received. */ osm_switch_set_switch_info( p_sw, p_si ); osm_switch_discovery_count_inc( p_sw ); @@ -557,7 +557,6 @@ __osm_si_rcv_process_existing( "Not discovering again through switch:0x%" PRIx64 ".\n", osm_node_get_node_guid( p_sw->p_node) ); - } } } @@ -699,7 +698,7 @@ osm_si_rcv_process( if( p_sw == (osm_switch_t*)cl_qmap_end( p_sw_guid_tbl ) ) { __osm_si_rcv_process_new( p_rcv, p_node, p_madw ); - /* + /* A new switch was found during the sweep so we need to ignore the current LFT settings. */ diff -X excl -rduNp gen2.2257.oorig/trunk/src/userspace/management/osm/opensm/osm_trap_rcv.c gen2.2257/trunk/src/userspace/management/osm/opensm/osm_trap_rcv.c --- gen2.2257.oorig/trunk/src/userspace/management/osm/opensm/osm_trap_rcv.c 2005-04-26 20:58:16.000000000 +0200 +++ gen2.2257/trunk/src/userspace/management/osm/opensm/osm_trap_rcv.c 2005-05-05 19:01:10.263497016 +0200 @@ -136,7 +136,7 @@ osm_trap_rcv_aging_tracker_callback( if (osm_exit_flag) /* We got an exit flag - do nothing */ return 0; - + lid = cl_ntoh16(( key & 0x0000FFFF00000000ULL) >> 32); port_num = ( key & 0x00FF000000000000ULL) >> 48; @@ -161,7 +161,7 @@ osm_trap_rcv_aging_tracker_callback( "osm_trap_rcv_aging_tracker_callback: " "Clearing health bit of port num:%u with lid:%u\n", port_num, lid ); - + /* Clear its health bit */ osm_physp_set_health(p_physp, TRUE); } @@ -169,8 +169,8 @@ osm_trap_rcv_aging_tracker_callback( } OSM_LOG_EXIT (p_rcv->p_log ); - - /* We want to remove the event from the tracker - so + + /* We want to remove the event from the tracker - so need to return zero. */ return 0; } @@ -264,7 +264,7 @@ __osm_trap_calc_crc32(void *buffer, uint } first = FALSE; } - + crc = -1L; /* do the calculation */ while (count-- != 0) @@ -331,13 +331,13 @@ __osm_trap_rcv_process_request( if (osm_exit_flag) { - /* - We got an exit flag - do nothing + /* + We got an exit flag - do nothing Otherwise we start a sweep on the trap 144 caused by cleaning up - SM Cap bit ... + SM Cap bit ... */ goto Exit; - } + } /* update the is_gsi flag according to the mgmt_class field */ if (p_madw->p_mad->mgmt_class == IB_MCLASS_SUBN_LID || @@ -371,13 +371,13 @@ __osm_trap_rcv_process_request( cl_memcpy(&tmp_madw, p_madw, sizeof( tmp_madw )); if (is_gsi == FALSE) - { + { /* We are in smi flow */ /* * When we received a TRAP with dlid = 0 - it means it * came from our own node. So we need to fix it. */ - + if (p_madw->mad_addr.addr_type.smi.source_lid == 0) { osm_log( p_rcv->p_log, OSM_LOG_DEBUG, @@ -462,7 +462,7 @@ __osm_trap_rcv_process_request( p_ntci, &trap_key); } - else + else __osm_trap_get_key(source_lid, 0, p_ntci, @@ -471,7 +471,7 @@ __osm_trap_rcv_process_request( /* try to find it in the aging tracker */ num_received = cl_event_wheel_num_regs(&p_rcv->trap_aging_tracker, trap_key); - + /* Now we know how many times it provided this trap */ if (num_received > 10) { @@ -492,7 +492,7 @@ __osm_trap_rcv_process_request( cl_ntoh16(p_ntci->data_details.ntc_129_131.lid), port_num ); - + if (! p_physp) { osm_log( p_rcv->p_log, OSM_LOG_ERROR, @@ -522,17 +522,16 @@ __osm_trap_rcv_process_request( run_heavy_sweep = TRUE; } /* If we are marking the port as unhealthy - we want to - keep this for a longer period of time then the - OSM_DEFAULT_TRAP_SUPRESSION_TIMEOUT. Use the + keep this for a longer period of time then the + OSM_DEFAULT_TRAP_SUPRESSION_TIMEOUT. Use the OSM_DEFAULT_UNHEALTHY_TIMEOUT */ event_wheel_timeout = OSM_DEFAULT_UNHEALTHY_TIMEOUT; } - } } /* restart the aging anyway */ - /* If physp_change_trap is TRUE - then use a callback to unset the + /* If physp_change_trap is TRUE - then use a callback to unset the healthy bit. If not - no need to use a callback. */ if (physp_change_trap == TRUE ) cl_event_wheel_reg(&p_rcv->trap_aging_tracker, @@ -549,7 +548,6 @@ __osm_trap_rcv_process_request( NULL /* no conetxt */ ); - /* If was already registered do nothing more */ if ( num_received > 10 && run_heavy_sweep == FALSE ) { @@ -561,41 +559,41 @@ __osm_trap_rcv_process_request( } } - /* do a sweep if we recieved a trap */ + /* do a sweep if we received a trap */ if (p_rcv->p_subn->opt.sweep_on_trap) { - /* if this is trap number 128 or run_heavy_sweep is TRUE - update the - force_single_heavy_sweep flag of the subnet. + /* if this is trap number 128 or run_heavy_sweep is TRUE - update the + force_single_heavy_sweep flag of the subnet. Sweep also on traps 144/145 - these traps signal a change on a certain - port capability/system image guid. + port capability/system image guid. TODO: In the future we can change this to just getting PortInfo on this port instead of sweeping the entire subnet. */ if (ib_notice_is_generic(p_ntci) && ( (cl_ntoh16(p_ntci->g_or_v.generic.trap_num) == 128) || (cl_ntoh16(p_ntci->g_or_v.generic.trap_num) == 144) || (cl_ntoh16(p_ntci->g_or_v.generic.trap_num) == 145) || - run_heavy_sweep )) + run_heavy_sweep )) { osm_log( p_rcv->p_log, OSM_LOG_VERBOSE, "__osm_trap_rcv_process_request: " "Forcing immediate heavy sweep. " "Received trap:%u \n", cl_ntoh16(p_ntci->g_or_v.generic.trap_num) ); - + p_rcv->p_subn->force_immediate_heavy_sweep = TRUE; } osm_state_mgr_process( p_rcv->p_state_mgr, OSM_SIGNAL_SWEEP ); } - /* If we reached here due to trap 129/130/131 - do not need to do + /* If we reached here due to trap 129/130/131 - do not need to do the notice report. Just goto exit. We know this is the case if physp_change_trap is TRUE. */ if ( physp_change_trap == TRUE ) goto Exit; /* Add a call to osm_report_notice */ - /* We are going to report the notice - so need to fix the IssuerGID + /* We are going to report the notice - so need to fix the IssuerGID accordingly. See IBA 1.1 P.653 or IBA 1.2 P.739 for details. */ if (is_gsi) { @@ -605,7 +603,7 @@ __osm_trap_rcv_process_request( &(tmp_madw.mad_addr.addr_type.gsi.grh_info.src_gid), sizeof(ib_gid_t)); } - else + else { osm_log( p_rcv->p_log, OSM_LOG_ERROR, "__osm_trap_rcv_process_request: ERR 3806: " @@ -618,7 +616,7 @@ __osm_trap_rcv_process_request( { /* Need to use the IssuerLID. */ p_tbl = &p_rcv->p_subn->port_lid_tbl; - + if (cl_ptr_vector_get_size(p_tbl) <= cl_ntoh16(source_lid) ) { /* the source lid is out of range. */ @@ -626,7 +624,7 @@ __osm_trap_rcv_process_request( "__osm_trap_rcv_process_request: " "source lid is out of range:0x%X \n", cl_ntoh16(source_lid) ); - + goto Exit; } p_port = cl_ptr_vector_get( p_tbl, cl_ntoh16(source_lid) ); @@ -637,10 +635,10 @@ __osm_trap_rcv_process_request( "__osm_trap_rcv_process_request: " "Cannot find port according to lid:0x%X \n", cl_ntoh16(source_lid) ); - + goto Exit; } - + p_ntci->issuer_gid.unicast.prefix = p_rcv->p_subn->opt.subnet_prefix; p_ntci->issuer_gid.unicast.interface_id = p_port->guid; } From mshefty at ichips.intel.com Thu May 5 10:10:17 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Thu, 05 May 2005 10:10:17 -0700 Subject: [openib-general] [PATCH] [MAD] changes to ib_create_send_mad In-Reply-To: <1115300959.4476.189.camel@localhost.localdomain> References: <1115253041.4476.147.camel@localhost.localdomain> <1115300959.4476.189.camel@localhost.localdomain> Message-ID: <427A5379.6060503@ichips.intel.com> Hal Rosenstock wrote: > On PayloadLength, single segment sends are fine. It is multisegment > sends which seem wrong to me. > > Case 1: ib_create_send_mad with hdr_len 0x38 data_len 0x278 > paylen_newwin stored in header is 0x28C which seems correct > this creates 4 segments > 1-3 segments paylen_newwin is 0x6E0, 4th segment is 0x34 Converting to decimal so I can use my fingers... hdr_len is 56 and data_len is 632. There are 256-56 = 200 bytes available per segment for the data, giving 4 segments. The paylen_newwin should be set to x28C when sending, so that seems correct. The first segment should have a paylen_newwin value of 220 x 4 = 880 (x370) on the wire. The last segment should have a value of 52 (x34). If you're seeing x6E0 (1760), that's exactly twice the expected payload length value. I will see if I can reproduce this using the numbers that you gave. What size did you specify for the sge? The total number of segments is calculated using the sge sizes. > Case 2: ib_create_send_mad with hdr_len 0x38 data_len 0x620 > paylen_newwin stored in header is 0x634 which seems correct > this creates 8 segments > 1-7 segments paylen_newwin is 0x6E0, 4th segment is 0xBC > > Last paylen_newwin in both cases appears to me to be correct but the > paylen_newwin in the 1-n segments (0x6E0) seems wrong to me. The paylen_newwin is only set by the code when sending the first and last segments. I would have expected the middle segments to carry the same value as the first, and I see this behavior on my tests. The fact that you see the same payload length for both of these makes me think that either the sge size is off, or there's an issue in the RMPP code using it. > Also, I did more investigation of send failures. In terms of send > failures upon not receiving ACKs, SA is different from vendor class 2. I > think the problem starts (and hopefully ends) with response_mad(). In > the case of SA GetTableResp being sent, the non data packets (ACK, etc.) > come back as SA GetTable so this is not currently considered a response > but I think it needs to be. I'm not sure if there are other issues > behind this as I didn't chase it further. Let me know if you want me to > do this. The RMPP code formats ACKs by inverting the response bit. This is necessary in order to route the ACK to the correct agent, and meets the SA requirements. Response MADs are routed to the correct agent using the TID. Non-response MADs are routed by using the lookup tables. E.g. if you look at the SA, the ACK (GetTable) needs to go to the agent using the lookup tables. The TID carried in the ACK was actually set by the sender of the ACK, and cannot be used to route to the SA's agent. Can you describe in more detail what the exact failure that you're seeing is and what you were expecting? Thanks for looking into this more. - Sean From mshefty at ichips.intel.com Thu May 5 10:20:17 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Thu, 05 May 2005 10:20:17 -0700 Subject: [openib-general] [kDAPL][PATCH] fix panic in server side In-Reply-To: <1115311335.25119.10.camel@duffman> References: <1115311335.25119.10.camel@duffman> Message-ID: <427A55D1.4070105@ichips.intel.com> Tom Duffy wrote: > On Thu, 2005-05-05 at 08:44 +0300, Or Gerlitz wrote: > >> Tom, >> >> >>>This occurred when I did a connect without first setting up a listen >> >>using kdapltest. >> >>>RIP: 0010:[] >> >>{:ib_dat_provider:dapl_cm_passive_cb_handler+27} >> >>>Call Trace:{:ib_cm:cm_process_work+50} >> >>{:ib_cm:cm_work_handler+1438} >> >>If you you haven't set up a (kdapltest server) listener - why did the CM >>invoked a (kdapl provider) consumer callback for this SID? > > > I have no idea. I loaded the kdapltest.ko kernel modules, but had not > started the kdapltest userland program when this occured. > > I am pretty sure I had rebooted right before trying this, but it could > have been that I had just unloaded and reloaded all the (dapl) modules > (so maybe something was left from a previous incarnation). > > I will reboot both machines and try to reproduce. If you had just unloaded/reloaded the modules, we should probably look at kdapl to see that it destroys all cm_id's. - Sean From halr at voltaire.com Thu May 5 10:16:17 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 05 May 2005 13:16:17 -0400 Subject: [openib-general] [PATCH] [MAD] changes to ib_create_send_mad In-Reply-To: <427A5379.6060503@ichips.intel.com> References: <1115253041.4476.147.camel@localhost.localdomain> <1115300959.4476.189.camel@localhost.localdomain> <427A5379.6060503@ichips.intel.com> Message-ID: <1115313377.4476.278.camel@localhost.localdomain> On Thu, 2005-05-05 at 13:10, Sean Hefty wrote: > I will see if I can reproduce this using the numbers that you gave. What > size did you specify for the sge? The total number of segments is > calculated using the sge sizes. Isn't that calculated and filled in by ib_create_send_mad ? buf_size = get_buf_length(hdr_len, data_len) ... send_buf->sge.length = buf_size; I pretty sure I leave this along after calling ib_create_send_mad. Is there something that needs to be done here ? > The fact that you see the same payload length for both of these makes me > think that either the sge size is off, or there's an issue in the RMPP code > using it. Yes, the fact that it was the same seemed weird to me too but wasn't sure of the origin. > > Also, I did more investigation of send failures. In terms of send > > failures upon not receiving ACKs, SA is different from vendor class 2. I > > think the problem starts (and hopefully ends) with response_mad(). In > > the case of SA GetTableResp being sent, the non data packets (ACK, etc.) > > come back as SA GetTable so this is not currently considered a response > > but I think it needs to be. I'm not sure if there are other issues > > behind this as I didn't chase it further. Let me know if you want me to > > do this. > > The RMPP code formats ACKs by inverting the response bit. This is necessary > in order to route the ACK to the correct agent, and meets the SA > requirements. Response MADs are routed to the correct agent using the TID. > Non-response MADs are routed by using the lookup tables. > > E.g. if you look at the SA, the ACK (GetTable) needs to go to the agent > using the lookup tables. The TID carried in the ACK was actually set by the > sender of the ACK, and cannot be used to route to the SA's agent. > > Can you describe in more detail what the exact failure that you're seeing is > and what you were expecting? I will answer this separately as this will take more time. -- Hal From jlentini at netapp.com Thu May 5 10:25:38 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 5 May 2005 13:25:38 -0400 (EDT) Subject: [openib-general] [kDAPL] fix kdapltest to support ia64 and add return value Message-ID: Revision 2258 contains fixes to kdapltest for the IA64 platform from Itamar. Thanks Itamar! If anyone continues to experience problems with DAPL on IA64 systems, please let me know. james ---------- Forwarded message ---------- Date: Thu, 5 May 2005 16:08:47 +0300 From: Itamar To: James.Lentini at netapp.com Subject: fix kdapltest to support ia64 and add return value This patch is based on gen2 svn rev 2257 new feature in kdapltest 1) add return value to tests (-T T,-T P,-T S) 2) move double code from kernel to user Signed-off-by: Itamar Rabenstein Index: dapltest/test/dapl_performance_util.c =================================================================== --- dapltest/test/dapl_performance_util.c (revision 2257) +++ dapltest/test/dapl_performance_util.c (working copy) @@ -254,6 +254,8 @@ DT_Performance_Test_Create ( test_ptr->ep_context.op.Rdma_Address = 0; test_ptr->ep_context.port = test_ptr->base_port; test_ptr->ep_context.pipeline_len = pipeline_len; + // update pipeline_len + pt_ptr->Params.Client_Stats_P.pipeline_len = pipeline_len; return true; } @@ -284,6 +286,7 @@ DT_Performance_Test_Destroy ( DT_Tdep_PT_Printf (phead, "Test[" F64x "]: Warning: Bpool destroy fails\n", test_ptr->base_port); /* carry on trying, regardless */ + g_status = 1; } } @@ -297,6 +300,7 @@ DT_Performance_Test_Destroy ( DT_Tdep_PT_Printf (phead, "Test[" F64x "]: Warning: Bpool destroy fails\n", test_ptr->base_port); /* carry on trying, regardless */ + g_status = 1; } } @@ -315,12 +319,14 @@ DT_Performance_Test_Destroy ( test_ptr->base_port, DT_RetToString (ret)); /* carry on trying, regardless */ + g_status = 1; } else if (!DT_disco_event_wait ( phead, test_ptr->conn_evd_hdl, &ep_handle)) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: bad disconnect event\n", test_ptr->base_port); + g_status = 1; } } @@ -333,6 +339,7 @@ DT_Performance_Test_Destroy ( DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ep_free error: %s\n", test_ptr->base_port, DT_RetToString (ret)); /* carry on trying, regardless */ + g_status = 1; } } @@ -345,6 +352,7 @@ DT_Performance_Test_Destroy ( DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_free (conn) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); /* fall through, keep trying */ + g_status = 1; } } if (is_server) @@ -357,6 +365,7 @@ DT_Performance_Test_Destroy ( DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_free (creq) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); /* fall through, keep trying */ + g_status = 1; } } } @@ -368,6 +377,7 @@ DT_Performance_Test_Destroy ( DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_free (reqt) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); /* fall through, keep trying */ + g_status = 1; } } if (test_ptr->recv_evd_hdl) @@ -378,6 +388,7 @@ DT_Performance_Test_Destroy ( DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_free (recv) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); /* fall through, keep trying */ + g_status = 1; } } @@ -390,6 +401,7 @@ DT_Performance_Test_Destroy ( DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_pz_free error: %s\n", test_ptr->base_port, DT_RetToString (ret)); /* fall through, keep trying */ + g_status = 1; } } Index: dapltest/test/dapl_performance_client.c =================================================================== --- dapltest/test/dapl_performance_client.c (revision 2257) +++ dapltest/test/dapl_performance_client.c (working copy) @@ -53,18 +53,21 @@ DT_Performance_Test_Client ( &test_ptr) ) { DT_Tdep_PT_Debug (1,(phead,"Client: Resource Creation Failed\n")); - connected = 0; + g_status = 1; + connected = 0; } else if ( !DT_Performance_Test_Client_Connect (phead, test_ptr) ) { DT_Tdep_PT_Debug (1,(phead,"Client: Connection Failed\n")); - connected = 0; + g_status = 1; + connected = 0; } - + params_ptr->Client_Stats_P.pipeline_len = pt_ptr->Params.Client_Stats_P.pipeline_len; if ( connected ) { if ( !DT_Performance_Test_Client_Exchange (params_ptr,phead, test_ptr) ) { + g_status = 1; DT_Tdep_PT_Debug (1,(phead,"Client: Test Failed\n")); } } @@ -233,31 +236,8 @@ DT_Performance_Test_Client_Phase1 ( { return false; } - - /* calculate CPU utilization */ - { - unsigned long int system; - unsigned long int user; - unsigned long int idle; - unsigned long int total; - - system = post_cpu_stat.system - pre_cpu_stat.system; - user = post_cpu_stat.user - pre_cpu_stat.user; - idle = post_cpu_stat.idle - pre_cpu_stat.idle; - - total = system + user + idle; - - if ( 0 == total ) - { - stats->cpu_utilization = 0.0; - } - else - { - stats->cpu_utilization = (double)1.0 - ((double) idle / (double) total ); - stats->cpu_utilization *= 100.0; - } - } - + stats->post_cpu_stat = post_cpu_stat; + stats->pre_cpu_stat = pre_cpu_stat; return true; } @@ -502,7 +482,7 @@ DT_Performance_Test_Client_Exchange ( { return false; } - DT_performance_stats_print (params_ptr,phead, &stats, test_ptr->cmd, test_ptr); - + stats.pipeline_len = params_ptr->Client_Stats_P.pipeline_len; + params_ptr->Client_Stats_P = stats; return true; } Index: dapltest/test/dapl_execute.c =================================================================== --- dapltest/test/dapl_execute.c (revision 2257) +++ dapltest/test/dapl_execute.c (working copy) @@ -32,6 +32,8 @@ #include "dapl_quit_cmd.h" #include "dapl_limit_cmd.h" +int g_status; + void DT_Execute_Test (Params_t *params_ptr) { Index: dapltest/test/dapl_transaction_stats.c =================================================================== --- dapltest/test/dapl_transaction_stats.c (revision 2257) +++ dapltest/test/dapl_transaction_stats.c (working copy) @@ -101,73 +101,3 @@ DT_update_transaction_stats (Transaction transaction_stats->bytes_rdma_write += bytes_rdma_write; DT_Mdep_Unlock (&transaction_stats->lock); } - -/* - * - */ -void -DT_print_transaction_stats (DT_Tdep_Print_Head *phead, - Transaction_Stats_t * transaction_stats, - unsigned int num_threads, - unsigned int num_EPs) -{ - double time_s; - double mbytes_send; - double mbytes_recv; - double mbytes_rdma_read; - double mbytes_rdma_write; - int total_ops; - DT_Mdep_Lock (&transaction_stats->lock); - time_s = (double) (transaction_stats->time_ms) / 1000; - if (time_s == 0.0) - { - DT_Tdep_PT_Printf (phead, - "----- Test completed successfully, but cannot calculate stats as not\n" - "----- enough time has lapsed.\n" - "----- Try running the test with more iterations.\n"); - goto unlock_and_return; - } - mbytes_send = (double) transaction_stats->bytes_send / 1000 / 1000; - mbytes_recv = (double) transaction_stats->bytes_recv / 1000 / 1000; - mbytes_rdma_read = (double) transaction_stats->bytes_rdma_read / 1000 / 1000; - mbytes_rdma_write = (double) transaction_stats->bytes_rdma_write / 1000 / 1000; - total_ops = transaction_stats->num_ops; - - if ( 0 == total_ops ) - { - DT_Tdep_PT_Printf (phead, "----- Test completed successfully, but no operations!\n"); - goto unlock_and_return; - } - - DT_Tdep_PT_Printf (phead, "----- Stats ---- : %u threads, %u EPs\n", - num_threads, num_EPs); - DT_Tdep_PT_Printf (phead, "Total WQE : %7d.%02d WQE/Sec\n", - whole (total_ops / time_s), - hundredths (total_ops / time_s)); - DT_Tdep_PT_Printf (phead, "Total Time : %7d.%02d sec\n", - whole (time_s), - hundredths (time_s)); - DT_Tdep_PT_Printf (phead, "Total Send : %7d.%02d MB - %7d.%02d MB/Sec\n", - whole (mbytes_send), - hundredths (mbytes_send), - whole (mbytes_send / time_s), - hundredths (mbytes_send / time_s)); - DT_Tdep_PT_Printf (phead, "Total Recv : %7d.%02d MB - %7d.%02d MB/Sec\n", - whole (mbytes_recv), - hundredths (mbytes_recv), - whole (mbytes_recv / time_s), - hundredths (mbytes_recv / time_s)); - DT_Tdep_PT_Printf (phead, "Total RDMA Read : %7d.%02d MB - %7d.%02d MB/Sec\n", - whole (mbytes_rdma_read), - hundredths (mbytes_rdma_read), - whole (mbytes_rdma_read / time_s), - hundredths (mbytes_rdma_read / time_s)); - DT_Tdep_PT_Printf (phead, "Total RDMA Write : %7d.%02d MB - %7d.%02d MB/Sec\n", - whole (mbytes_rdma_write), - hundredths (mbytes_rdma_write), - whole (mbytes_rdma_write / time_s), - hundredths (mbytes_rdma_write / time_s)); - -unlock_and_return: - DT_Mdep_Unlock (&transaction_stats->lock); -} Index: dapltest/test/dapl_server.c =================================================================== --- dapltest/test/dapl_server.c (revision 2257) +++ dapltest/test/dapl_server.c (working copy) @@ -65,6 +65,7 @@ DT_cs_Server (Params_t * params_ptr) DT_Tdep_PT_Printf (phead, "NOTICE: server already started for this NIC: %s\n", Server_Cmd->dapl_name); + g_status = 1; return; } temp_list = temp_list->next; @@ -75,6 +76,7 @@ DT_cs_Server (Params_t * params_ptr) if (temp_list == 0) { DT_Tdep_PT_Printf (phead, "no memory for server_list\n"); + g_status = 1; return; } strcpy (temp_list->devicename, Server_Cmd->dapl_name); @@ -111,6 +113,7 @@ DT_cs_Server (Params_t * params_ptr) Server_Cmd->dapl_name, DT_RetToString (ret)); ps_ptr->ia_handle = DAT_HANDLE_NULL; + g_status = 1; goto server_exit; } DT_Tdep_PT_Debug (1,(phead,"%s: IA %s opened\n", module, Server_Cmd->dapl_name)); @@ -124,6 +127,7 @@ DT_cs_Server (Params_t * params_ptr) module, DT_RetToString (ret)); ps_ptr->pz_handle = DAT_HANDLE_NULL; + g_status = 1; goto server_exit; } DT_Tdep_PT_Debug (1,(phead,"%s: PZ created\n", module)); @@ -140,6 +144,7 @@ DT_cs_Server (Params_t * params_ptr) "%s: dat_evd_create (recv) failed %s\n", module, DT_RetToString (ret)); ps_ptr->recv_evd_hdl = DAT_HANDLE_NULL; + g_status = 1; goto server_exit; } ret = DT_Tdep_evd_create (ps_ptr->ia_handle, @@ -153,6 +158,7 @@ DT_cs_Server (Params_t * params_ptr) "%s: dat_evd_create (send) failed %s\n", module, DT_RetToString (ret)); ps_ptr->reqt_evd_hdl = DAT_HANDLE_NULL; + g_status = 1; goto server_exit; } ret = DT_Tdep_evd_create (ps_ptr->ia_handle, @@ -166,6 +172,7 @@ DT_cs_Server (Params_t * params_ptr) "%s: dat_evd_create (cr) failed %s\n", module, DT_RetToString (ret)); ps_ptr->creq_evd_hdl = DAT_HANDLE_NULL; + g_status = 1; goto server_exit; } ret = DT_Tdep_evd_create (ps_ptr->ia_handle, @@ -179,6 +186,7 @@ DT_cs_Server (Params_t * params_ptr) "%s: dat_evd_create (conn) failed %s\n", module, DT_RetToString (ret)); ps_ptr->conn_evd_hdl = DAT_HANDLE_NULL; + g_status = 1; goto server_exit; } @@ -197,6 +205,7 @@ DT_cs_Server (Params_t * params_ptr) module, DT_RetToString (ret)); ps_ptr->ep_handle = DAT_HANDLE_NULL; + g_status = 1; goto server_exit; } DT_Tdep_PT_Debug (1,(phead,"%s: EP created\n", module)); @@ -214,6 +223,7 @@ DT_cs_Server (Params_t * params_ptr) module, DT_RetToString (ret)); ps_ptr->psp_handle = DAT_HANDLE_NULL; + g_status = 1; goto server_exit; } DT_Tdep_PT_Debug (1,(phead,"%s: PSP created\n", module)); @@ -238,6 +248,7 @@ DT_cs_Server (Params_t * params_ptr) DT_Tdep_PT_Printf (phead, "%s: no memory for command buffer pool.\n", module); + g_status = 1; goto server_exit; } @@ -291,6 +302,7 @@ DT_cs_Server (Params_t * params_ptr) /* Gather whatever info we want about defaults */ if (!DT_query (pt_ptr, ps_ptr->ia_handle, ps_ptr->ep_handle)) { + g_status = 1; goto server_exit; } @@ -305,6 +317,7 @@ DT_cs_Server (Params_t * params_ptr) DT_Tdep_PT_Printf (phead, "%s: cannot post ClientInfo recv buffer\n", module); + g_status = 1; goto server_exit; } if (!DT_post_recv_buffer (phead, @@ -316,6 +329,7 @@ DT_cs_Server (Params_t * params_ptr) DT_Tdep_PT_Printf (phead, "%s: cannot post Transaction_Cmd_t recv buffer\n", module); + g_status = 1; goto server_exit; } @@ -339,6 +353,7 @@ DT_cs_Server (Params_t * params_ptr) DT_Tdep_PT_Printf (phead, "CR Check failed, file %s line %d\n", __FILE__, __LINE__); + g_status = 1; goto server_exit; } @@ -350,6 +365,7 @@ DT_cs_Server (Params_t * params_ptr) "%s: dat_cr_accept error: %s\n", module, DT_RetToString (ret)); + g_status = 1; goto server_exit; } @@ -362,6 +378,7 @@ DT_cs_Server (Params_t * params_ptr) DT_Tdep_PT_Printf (phead, "%s: error awaiting conn-established event\n", module); + g_status = 1; goto server_exit; } @@ -384,6 +401,7 @@ DT_cs_Server (Params_t * params_ptr) dto_cookie, "Client_Info_Recv")) { + g_status = 1; goto server_exit; } DT_Tdep_PT_Debug (1,(phead,"%s: Got Client_Info\n", module)); @@ -408,6 +426,7 @@ DT_cs_Server (Params_t * params_ptr) dto_cookie, "Client_Cmd_Recv")) { + g_status = 1; goto server_exit; } @@ -481,6 +500,7 @@ DT_cs_Server (Params_t * params_ptr) if (pt_ptr->thread == 0) { DT_Tdep_PT_Printf (phead, "no memory to create thread\n"); + g_status = 1; goto server_exit; } break; @@ -513,6 +533,7 @@ DT_cs_Server (Params_t * params_ptr) if (pt_ptr->thread == 0) { DT_Tdep_PT_Printf (phead, "no memory to create thread\n"); + g_status = 1; goto server_exit; } /* take the performance test lock to serialize */ @@ -538,6 +559,7 @@ DT_cs_Server (Params_t * params_ptr) if (DT_Thread_Start (pt_ptr->thread) == false) { DT_Tdep_PT_Debug (1,(phead,"failed to start test thread\n")); + g_status = 1; goto server_exit; } @@ -556,6 +578,7 @@ DT_cs_Server (Params_t * params_ptr) "DAPLTEST VERSION MISMATCH", DAPLTEST_VERSION, pt_ptr->Client_Info.dapltest_version); + g_status = 1; goto server_exit; } DT_Tdep_PT_Debug (1,(phead,"%s: Version OK!\n", module)); @@ -573,6 +596,7 @@ DT_cs_Server (Params_t * params_ptr) DT_Bpool_GetBuffSize (ps_ptr->bpool, 2))) { DT_Tdep_PT_Printf (phead, "%s: cannot send Server_Info\n", module); + g_status = 1; goto server_exit; } /* reap the send and verify it */ @@ -587,6 +611,7 @@ DT_cs_Server (Params_t * params_ptr) dto_cookie, "Server_Info_Send")) { + g_status = 1; goto server_exit; } @@ -604,11 +629,13 @@ DT_cs_Server (Params_t * params_ptr) { DT_Tdep_PT_Printf (phead, "%s: dat_ep_disconnect fails: %s\n", module, DT_RetToString (ret)); + g_status = 1; goto server_exit; } if (!DT_disco_event_wait ( phead, ps_ptr->conn_evd_hdl, NULL)) { DT_Tdep_PT_Printf (phead, "%s: bad disconnect event\n", module); + g_status = 1; goto server_exit; } @@ -658,6 +685,7 @@ server_exit: DT_Tdep_PT_Printf (phead, "%s: dat_ep_disconnect fails: %s\n", module, DT_RetToString (ret)); /* keep trying */ + g_status = 1; } else if (!DT_disco_event_wait ( phead, ps_ptr->conn_evd_hdl, @@ -666,6 +694,7 @@ server_exit: DT_Tdep_PT_Printf (phead, "%s: bad disconnect event\n", module); + g_status = 1; } } @@ -677,6 +706,7 @@ server_exit: DT_Tdep_PT_Printf (phead, "%s: error destroying buffer pool\n", module); /* keep trying */ + g_status = 1; } } @@ -690,6 +720,7 @@ server_exit: module, DT_RetToString (ret)); /* keep trying */ + g_status = 1; } } @@ -702,6 +733,7 @@ server_exit: DT_Tdep_PT_Printf (phead, "%s: dat_ep_free error: %s\n", module, DT_RetToString (ret)); /* keep trying */ + g_status = 1; } } @@ -715,6 +747,7 @@ server_exit: "%s: dat_evd_free (conn) error: %s\n", module, DT_RetToString (ret)); /* keep trying */ + g_status = 1; } } if (ps_ptr->creq_evd_hdl) @@ -726,6 +759,7 @@ server_exit: "%s: dat_evd_free (creq) error: %s\n", module, DT_RetToString (ret)); /* keep trying */ + g_status = 1; } } if (ps_ptr->reqt_evd_hdl) @@ -737,6 +771,7 @@ server_exit: "%s: dat_evd_free (reqt) error: %s\n", module, DT_RetToString (ret)); /* keep trying */ + g_status = 1; } } if (ps_ptr->recv_evd_hdl) @@ -748,6 +783,7 @@ server_exit: "%s: dat_evd_free (recv) error: %s\n", module, DT_RetToString (ret)); /* keep trying */ + g_status = 1; } } @@ -760,6 +796,7 @@ server_exit: DT_Tdep_PT_Printf (phead, "%s: dat_pz_free error: %s\n", module, DT_RetToString (ret)); /* keep trying */ + g_status = 1; } } @@ -773,12 +810,14 @@ server_exit: DT_Tdep_PT_Printf (phead, "%s: dat_ia_close (graceful) error: %s\n", module, DT_RetToString (ret)); + g_status = 1; ret = dat_ia_close (ps_ptr->ia_handle, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_ia_close (abrupt) error: %s\n", module, DT_RetToString (ret)); + g_status = 1; } /* keep trying */ } Index: dapltest/test/dapl_performance_stats.c =================================================================== --- dapltest/test/dapl_performance_stats.c (revision 2257) +++ dapltest/test/dapl_performance_stats.c (working copy) @@ -36,7 +36,6 @@ DT_performance_stats_init ( stats->post_ctxt_switch_num = 0; stats->reap_ctxt_switch_num = 0; - stats->cpu_utilization = 0.0; stats->time_ts = 0; stats->posts_sans_ctxt.num = 0; @@ -168,8 +167,6 @@ DT_performance_stats_combine ( dest->reap_ctxt_switch_num = src_b->reap_ctxt_switch_num + src_b->reap_ctxt_switch_num; - dest->cpu_utilization = DT_max (src_a->cpu_utilization, - src_b->cpu_utilization); dest->time_ts = DT_max (src_a->time_ts, src_b->time_ts); DT_performance_stats_data_combine (&dest->posts_sans_ctxt, @@ -194,203 +191,3 @@ DT_performance_stats_combine ( } -double -DT_performance_stats_data_print ( - DT_Tdep_Print_Head *phead, - Performance_Stats_Data_t *data, - double cpu_mhz) -{ - double average; - - average = (int64_t)data->total_ts / (data->num * cpu_mhz); - - DT_Tdep_PT_Printf (phead, - " Arithmetic mean : %d.%d us\n" - " maximum : %d.%d us\n" - " minimum : %d.%d us\n", - DT_whole(average),DT_hundredths(average), - DT_whole((int64_t)data->max_ts / cpu_mhz),DT_hundredths((int64_t)data->max_ts / cpu_mhz), - DT_whole((int64_t)data->min_ts / cpu_mhz),DT_hundredths((int64_t)data->min_ts / cpu_mhz)); - return average; -} - - -void -DT_performance_stats_print ( - Params_t *params_ptr, - DT_Tdep_Print_Head *phead, - Performance_Stats_t *stats, - Performance_Cmd_t *cmd, - Performance_Test_t *test) -{ - double cpu_mhz; - double time_s; - double mbytes; - double ops_per_sec; - double bandwidth; - double latency; - double time_per_post; - double time_per_reap; - - cpu_mhz = params_ptr->cpu_mhz; - latency = 0; -#if defined(WIN32) - /* - * The Microsoft compiler is unable to do a 64 bit conversion when - * working with double. time_ts is a 64 bit value, so we - * potentially lose precision, so limit it to the Windows - * platform. Trying to do the operation below without casting - * a 64 bit value to an unsigned int results in the error when - * using Visual C 6.0: - * - * Compiler Error C2520: conversion from unsigned __int64 to - * double not implemented, use signed __int64. - * - * Note that signed __int64 doesn't work either! - */ - time_s = (double) ((unsigned int)stats->time_ts / (1000000.0 * cpu_mhz)); -#else - time_s = (double) (stats->time_ts / (1000000.0 * cpu_mhz)); -#endif - mbytes = (double) (1.0 * stats->bytes) / 1024 / 1024; - - if ( 0.0 == time_s ) - { - DT_Tdep_PT_Printf (phead, "Error determining time\n"); - return; - } - else if ( 0 == stats->num_ops ) - { - DT_Tdep_PT_Printf (phead, "Error determining number of operations\n"); - return; - } - else if ( 0.0 == cpu_mhz ) - { - DT_Tdep_PT_Printf (phead, "Error determining CPU speed\n"); - return; - } - - ops_per_sec = stats->num_ops / time_s; - bandwidth = mbytes / time_s; - DT_Tdep_PT_Printf (phead, "------------------------- Statistics -------------------------\n"); - DT_Tdep_PT_Printf (phead, - " Mode : %s\n" - " Operation Type : %s\n" - " Number of Operations : %u\n" - " Segment Size : %u\n" - " Number of Segments : %u bytes\n" - " Pipeline Length : %u\n\n", - DT_PerformanceModeToString (cmd->mode), - DT_TransferTypeToString (cmd->op.transfer_type), - cmd->num_iterations, - cmd->op.seg_size, - cmd->op.num_segs, - test->ep_context.pipeline_len); - - DT_Tdep_PT_Printf (phead, - " Total Time : %d.%d sec\n" - " Total Data Exchanged : %d.%d MB\n" - " CPU Utilization : %d.%d\n" - " Operation Throughput : %d.%d ops/sec\n" - " Bandwidth : %d.%d MB/sec\n", - DT_whole(time_s),DT_hundredths(time_s), - DT_whole(mbytes),DT_hundredths(mbytes), - DT_whole(stats->cpu_utilization),DT_hundredths(stats->cpu_utilization), - DT_whole(ops_per_sec),DT_hundredths(ops_per_sec), - DT_whole(bandwidth),DT_hundredths(bandwidth)); - - DT_Tdep_PT_Printf (phead, "\nLatency\n"); - - if ( stats->latency.num ) - { - latency = DT_performance_stats_data_print (phead, - &stats->latency, - cpu_mhz); - } - DT_Tdep_PT_Printf (phead, "\n" - "Time Per Post\n" - " %u posts without context switches\n", - stats->posts_sans_ctxt.num); - - if ( stats->posts_sans_ctxt.num ) - { - DT_performance_stats_data_print (phead, - &stats->posts_sans_ctxt, - cpu_mhz); - } - - DT_Tdep_PT_Printf (phead, "\n" - " %u posts with context switches\n", - stats->posts_with_ctxt.num); - - if ( stats->posts_with_ctxt.num ) - { - DT_Tdep_PT_Printf (phead, " %u number of context switches\n", - stats->post_ctxt_switch_num); - DT_performance_stats_data_print (phead, - &stats->posts_with_ctxt, - cpu_mhz); - } - - DT_Tdep_PT_Printf (phead, "\n" - "Time Per Reap\n" - " %u reaps without context switches\n", - stats->reaps_sans_ctxt.num); - - if ( stats->reaps_sans_ctxt.num ) - { - DT_performance_stats_data_print (phead, - &stats->reaps_sans_ctxt, - cpu_mhz); - } - - - DT_Tdep_PT_Printf (phead, "\n" - " %u reaps with context switches\n", - stats->reaps_with_ctxt.num); - - if ( stats->reaps_with_ctxt.num ) - { - DT_Tdep_PT_Printf (phead, "\n" - " %u number of context switches\n", - stats->reap_ctxt_switch_num); - - DT_performance_stats_data_print (phead, - &stats->reaps_with_ctxt, - cpu_mhz); - } - - time_per_post = - (int64_t) (stats->posts_sans_ctxt.total_ts + stats->posts_with_ctxt.total_ts) / - (cpu_mhz * (stats->posts_sans_ctxt.num + stats->posts_with_ctxt.num)); - - time_per_reap = - (int64_t) (stats->reaps_sans_ctxt.total_ts + stats->reaps_with_ctxt.total_ts) / - (cpu_mhz * (stats->reaps_sans_ctxt.num + stats->reaps_with_ctxt.num)); - - DT_Tdep_PT_Printf (phead, - "\nNOTE: 1 MB = 1024 KB = 1048576 B \n"); - DT_Tdep_PT_Printf (phead, - "---------------------------------------------------------------------\n"); - DT_Tdep_PT_Printf (phead, - "raw: %s, %u, %u, %u, %u, %d.%d, %d.%d, %d.%d, %d.%d, %d.%d, %d.%d \n", - DT_TransferTypeToString (cmd->op.transfer_type), - cmd->num_iterations, - cmd->op.seg_size, - cmd->op.num_segs, - test->ep_context.pipeline_len, - DT_whole (stats->cpu_utilization), - DT_hundredths (stats->cpu_utilization), - DT_whole (ops_per_sec), - DT_hundredths (ops_per_sec), - DT_whole (bandwidth), - DT_hundredths (bandwidth), - DT_whole (latency), - DT_hundredths (latency), - DT_whole (time_per_post), - DT_hundredths (time_per_post), - DT_whole (time_per_reap), - DT_hundredths (time_per_reap)); - DT_Tdep_PT_Printf (phead, - "---------------------------------------------------------------------\n"); -} Index: dapltest/test/dapl_transaction_test.c =================================================================== --- dapltest/test/dapl_transaction_test.c (revision 2257) +++ dapltest/test/dapl_transaction_test.c (working copy) @@ -74,6 +74,7 @@ DT_Transaction_Test_Client (Per_Test_Dat remote_ia_addr)) { DT_Tdep_PT_Printf (phead, "Client: Cannot Create Test!\n"); + g_status = 1; break; } } @@ -83,11 +84,6 @@ DT_Transaction_Test_Client (Per_Test_Dat { DT_Mdep_Sleep (100); } - - DT_print_transaction_stats (phead, - &pt_ptr->Client_Stats, - cmd->num_threads, - cmd->eps_per_thread); } @@ -117,6 +113,7 @@ DT_Transaction_Test_Server (void *params (DAT_IA_ADDRESS_PTR) 0)) { DT_Tdep_PT_Printf (phead, "Server: Cannot Create Test!\n"); + g_status = 1; break; } } @@ -248,6 +245,7 @@ DT_Transaction_Main (void *param) DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_pz_create error: %s\n", test_ptr->base_port, DT_RetToString (ret)); test_ptr->pz_handle = DAT_HANDLE_NULL; + g_status = 1; goto test_failure; } @@ -262,6 +260,7 @@ DT_Transaction_Main (void *param) DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (recv) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); test_ptr->recv_evd_hdl = DAT_HANDLE_NULL; + g_status = 1; goto test_failure; } @@ -275,6 +274,7 @@ DT_Transaction_Main (void *param) DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (request) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); test_ptr->reqt_evd_hdl = DAT_HANDLE_NULL; + g_status = 1; goto test_failure; } @@ -291,6 +291,7 @@ DT_Transaction_Main (void *param) DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (cr) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); test_ptr->creq_evd_hdl = DAT_HANDLE_NULL; + g_status = 1; goto test_failure; } } @@ -305,6 +306,7 @@ DT_Transaction_Main (void *param) DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (conn) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); test_ptr->conn_evd_hdl = DAT_HANDLE_NULL; + g_status = 1; goto test_failure; } @@ -319,6 +321,7 @@ DT_Transaction_Main (void *param) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: no memory for EP context\n", test_ptr->base_port); + g_status = 1; goto test_failure; } @@ -367,6 +370,7 @@ DT_Transaction_Main (void *param) DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ep_create #%d error: %s\n", test_ptr->base_port, i, DT_RetToString (ret)); test_ptr->ep_context[i].ep_handle = DAT_HANDLE_NULL; + g_status = 1; goto test_failure; } @@ -389,6 +393,7 @@ DT_Transaction_Main (void *param) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: no memory for remote memory buffers\n", test_ptr->base_port); + g_status = 1; goto test_failure; } @@ -419,6 +424,7 @@ DT_Transaction_Main (void *param) buff_size)) { /* error message printed by DT_post_recv_buffer */ + g_status = 1; goto test_failure; } if (!DT_post_recv_buffer ( phead, @@ -428,6 +434,7 @@ DT_Transaction_Main (void *param) SYNC_BUFF_SIZE)) { /* error message printed by DT_post_recv_buffer */ + g_status = 1; goto test_failure; } @@ -454,6 +461,7 @@ DT_Transaction_Main (void *param) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_rsp_create #%d error: %s\n", test_ptr->base_port, i, DT_RetToString (ret)); + g_status = 1; goto test_failure; } } @@ -468,6 +476,7 @@ DT_Transaction_Main (void *param) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_psp_create #%d error: %s\n", test_ptr->base_port, i, DT_RetToString (ret)); + g_status = 1; goto test_failure; } @@ -529,6 +538,7 @@ DT_Transaction_Main (void *param) &cr_handle, "Server") ) { + g_status = 1; goto test_failure; } @@ -537,6 +547,7 @@ DT_Transaction_Main (void *param) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_query #%d error:(%x) %s\n", test_ptr->base_port, i, ret, DT_RetToString (ret)); + g_status = 1; } else { @@ -563,6 +574,7 @@ DT_Transaction_Main (void *param) DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_accept #%d error: %s\n", test_ptr->base_port, i, DT_RetToString (ret)); /* cr_handle consumed on failure */ + g_status = 1; goto test_failure; } @@ -573,6 +585,7 @@ DT_Transaction_Main (void *param) &event_num)) { /* error message printed by DT_conn_event_wait */ + g_status = 1; goto test_failure; } /* throw away single-use PSP */ @@ -581,6 +594,7 @@ DT_Transaction_Main (void *param) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_rsp_free #%d error: %s\n", test_ptr->base_port, i, DT_RetToString (ret)); + g_status = 1; goto test_failure; } @@ -597,6 +611,7 @@ DT_Transaction_Main (void *param) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_psp_create #%d error: %s\n", test_ptr->base_port, i, DT_RetToString (ret)); + g_status = 1; goto test_failure; } @@ -607,6 +622,7 @@ DT_Transaction_Main (void *param) &cr_handle, "Server") ) { + g_status = 1; goto test_failure; } @@ -615,6 +631,7 @@ DT_Transaction_Main (void *param) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_query #%d error: %s\n", test_ptr->base_port, i, DT_RetToString (ret)); + g_status = 1; } else { @@ -643,6 +660,7 @@ DT_Transaction_Main (void *param) test_ptr->base_port, i, DT_RetToString (ret)); /* cr_handle consumed on failure */ (void) dat_psp_free (test_ptr->ep_context[i].psp_handle); + g_status = 1; goto test_failure; } @@ -654,6 +672,7 @@ DT_Transaction_Main (void *param) { /* error message printed by DT_cr_event_wait */ (void) dat_psp_free (&test_ptr->ep_context[i].psp_handle); + g_status = 1; goto test_failure; } @@ -663,6 +682,7 @@ DT_Transaction_Main (void *param) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_psp_free #%d error: %s\n", test_ptr->base_port, i, DT_RetToString (ret)); + g_status = 1; goto test_failure; } } /* end short-lived PSP */ @@ -695,6 +715,7 @@ retry: { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ep_connect #%d error: %s (0x%x)\n", test_ptr->base_port, i, DT_RetToString (ret), ret); + g_status = 1; goto test_failure; } @@ -736,6 +757,7 @@ retry: buff_size)) { /* error message printed by DT_post_recv_buffer */ + g_status = 1; goto test_failure; } if (!DT_post_recv_buffer ( phead, @@ -745,6 +767,7 @@ retry: SYNC_BUFF_SIZE)) { /* error message printed by DT_post_recv_buffer */ + g_status = 1; goto test_failure; } } @@ -758,6 +781,7 @@ retry: } } /* error message printed by DT_cr_event_wait */ + g_status = 1; goto test_failure; } @@ -819,6 +843,7 @@ retry: { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: no memory for buffers (RDMA/RD)\n", test_ptr->base_port); + g_status = 1; goto test_failure; } if (!us) @@ -854,6 +879,7 @@ retry: { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: no memory for buffers (RDMA/WR)\n", test_ptr->base_port); + g_status = 1; goto test_failure; } if (!us) @@ -888,6 +914,7 @@ retry: { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: no memory for buffers (S/R)\n", test_ptr->base_port); + g_status = 1; goto test_failure; } @@ -967,6 +994,7 @@ retry: test_ptr->is_server ? "Client_Mem_Info_Send" : "Server_Mem_Info_Send")) { + g_status = 1; goto test_failure; } @@ -990,6 +1018,7 @@ retry: test_ptr->is_server ? "Client_Mem_Info_Recv" : "Server_Mem_Info_Recv")) { + g_status = 1; goto test_failure; } @@ -1037,6 +1066,8 @@ retry: * Finally! Run the test. */ success = DT_Transaction_Run (phead, test_ptr); + g_status = !success; + //leave the qp open for debug and dont close the test // while (1) { // DT_Mdep_Sleep(1000); // } @@ -1065,6 +1096,7 @@ test_failure: { DT_Tdep_PT_Printf (phead, "test[" F64x "]: Warning: Bpool destroy fails\n", test_ptr->base_port); + g_status = 1; /* carry on trying, regardless */ } } @@ -1079,6 +1111,7 @@ test_failure: { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: Warning: Bpool destroy fails\n", test_ptr->base_port); + g_status = 1; /* carry on trying, regardless */ } } @@ -1100,6 +1133,7 @@ test_failure: success ? "graceful" : "abrupt", i, DT_RetToString (ret)); /* carry on trying, regardless */ + g_status = 1; } } @@ -1117,6 +1151,7 @@ test_failure: { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: bad disconnect event\n", test_ptr->base_port); + g_status = 1; } else { @@ -1165,6 +1200,7 @@ test_failure: DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ep_free #%d error: %s\n", test_ptr->base_port, i, DT_RetToString (ret)); /* carry on trying, regardless */ + g_status = 1; } } } /* end foreach per-EP context */ @@ -1181,6 +1217,7 @@ test_failure: DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_free (conn) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); /* fall through, keep trying */ + g_status = 1; } } if (pt_ptr->local_is_server) @@ -1193,6 +1230,7 @@ test_failure: DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_free (creq) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); /* fall through, keep trying */ + g_status = 1; } } } @@ -1204,6 +1242,7 @@ test_failure: DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_free (reqt) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); /* fall through, keep trying */ + g_status = 1; } } if (test_ptr->recv_evd_hdl) @@ -1214,6 +1253,7 @@ test_failure: DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_free (recv) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); /* fall through, keep trying */ + g_status = 1; } } @@ -1226,6 +1266,7 @@ test_failure: DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_pz_free error: %s\n", test_ptr->base_port, DT_RetToString (ret)); /* fall through, keep trying */ + g_status = 1; } } @@ -1892,51 +1933,3 @@ DT_Print_Transaction_Test (DT_Tdep_Print test_ptr->stats.stat_bytes_rdma_write); } - -/*------------------------------------------------------------------------------ */ -void -DT_Print_Transaction_Stats (DT_Tdep_Print_Head *phead, Transaction_Test_t * test_ptr) -{ - double time; - double mbytes_send; - double mbytes_recv; - double mbytes_rdma_read; - double mbytes_rdma_write; - int total_ops; - time = (double) (test_ptr->stats.end_time - test_ptr->stats.start_time) / 1000; - mbytes_send = (double) test_ptr->stats.stat_bytes_send / 1024 / 1024; - mbytes_recv = (double) test_ptr->stats.stat_bytes_recv / 1024 / 1024; - mbytes_rdma_read = (double) test_ptr->stats.stat_bytes_rdma_read / 1024 / 1024; - mbytes_rdma_write = (double) test_ptr->stats.stat_bytes_rdma_write / 1024 / 1024; - total_ops = test_ptr->cmd->num_ops * test_ptr->cmd->num_iterations; - - DT_Tdep_PT_Printf (phead, "Test[: " F64x "] ---- Stats ----\n", test_ptr->base_port); - DT_Tdep_PT_Printf (phead, "Iterations : %u\n", test_ptr->cmd->num_iterations); - DT_Tdep_PT_Printf (phead, "Ops : %7d.%02d Ops/Sec\n", - whole (total_ops / time), - hundredths (total_ops / time)); - DT_Tdep_PT_Printf (phead, "Time : %7d.%02d sec\n", - whole (time), - hundredths (time)); - DT_Tdep_PT_Printf (phead, "Sent : %7d.%02d MB - %7d.%02d MB/Sec\n", - whole (mbytes_send), - hundredths (mbytes_send), - whole (mbytes_send / time), - hundredths (mbytes_send / time)); - DT_Tdep_PT_Printf (phead, "Recv : %7d.%02d MB - %7d.%02d MB/Sec\n", - whole (mbytes_recv), - hundredths (mbytes_recv), - whole (mbytes_recv / time), - hundredths (mbytes_recv / time)); - DT_Tdep_PT_Printf (phead, "RDMA Read : %7d.%02d MB - %7d.%02d MB/Sec\n", - whole (mbytes_rdma_read), - hundredths (mbytes_rdma_read), - whole (mbytes_rdma_read / time), - hundredths (mbytes_rdma_read / time)); - DT_Tdep_PT_Printf (phead, "RDMA Write : %7d.%02d MB - %7d.%02d MB/Sec\n", - whole (mbytes_rdma_write), - hundredths (mbytes_rdma_write), - whole (mbytes_rdma_write / time), - hundredths (mbytes_rdma_write / time)); -} - Index: dapltest/test/dapl_performance_server.c =================================================================== --- dapltest/test/dapl_performance_server.c (revision 2257) +++ dapltest/test/dapl_performance_server.c (working copy) @@ -49,6 +49,7 @@ DT_Performance_Test_Server ( { DT_Tdep_PT_Printf (phead, "Server: Resource Creation Failed\n"); success = 0; + g_status = 1; } if ( 1 == success ) { @@ -56,6 +57,7 @@ DT_Performance_Test_Server ( { success = 0; DT_Tdep_PT_Printf (phead, "Server: Connection Failed\n"); + g_status = 1; } } @@ -64,6 +66,7 @@ DT_Performance_Test_Server ( if ( ! DT_Performance_Test_Server_Exchange (phead, test_ptr) ) { success = 0; + g_status = 1; DT_Tdep_PT_Printf (phead, "Server: Test Failed\n"); } } @@ -196,6 +199,7 @@ DT_Performance_Test_Server_Connect ( * this thread is ready to wait for a connection request * from the remote end. */ + DT_Mdep_wait_object_wakeup (&test_ptr->pt_ptr->synch_wait_object); DT_Tdep_PT_Debug (1,(phead,"Server[" F64x "]: Listen on PSP port 0x" F64x "\n", Index: dapltest/test/dapl_util.c =================================================================== --- dapltest/test/dapl_util.c (revision 2257) +++ dapltest/test/dapl_util.c (working copy) @@ -49,29 +49,6 @@ DT_RetToString (DAT_RETURN ret_value) return errmsg; } -/* - * Map DAT_RETURN values to readable strings, - * but don't assume the values are zero-based or contiguous. - */ -const char * -DT_TransferTypeToString (DT_Transfer_Type type) -{ - static char *DT_Type[] = - { - "RR", - "RW", - "SR" - }; - - if ( (0 <= type) && (type <= 2) ) - { - return DT_Type[type]; - } - else - { - return "Error: Unkown Transfer Type"; - } -} /* Index: dapltest/test/dapl_client.c =================================================================== --- dapltest/test/dapl_client.c (revision 2257) +++ dapltest/test/dapl_client.c (working copy) @@ -77,6 +77,7 @@ DT_cs_Client (Params_t * params_ptr, if (!pt_ptr) { DT_Tdep_PT_Printf (phead, "%s: no memory for Per_Test_Data\n", module); + g_status = 1; return; } DT_MemListInit (pt_ptr); /* init MemlistLock and memListHead */ @@ -106,7 +107,8 @@ DT_cs_Client (Params_t * params_ptr, dapl_name, DT_RetToString (ret)); ia_handle = DAT_HANDLE_NULL; - goto client_exit; + g_status = 1; + goto client_exit; } DT_Tdep_PT_Debug (1,(phead, "%s: IA %s opened\n", module, dapl_name)); @@ -119,6 +121,7 @@ DT_cs_Client (Params_t * params_ptr, module, DT_RetToString (ret)); pz_handle = DAT_HANDLE_NULL; + g_status = 1; goto client_exit; } @@ -136,6 +139,7 @@ DT_cs_Client (Params_t * params_ptr, module, DT_RetToString (ret)); recv_evd_hdl = DAT_HANDLE_NULL; + g_status = 1; goto client_exit; } ret = DT_Tdep_evd_create (ia_handle, @@ -150,6 +154,7 @@ DT_cs_Client (Params_t * params_ptr, module, DT_RetToString (ret)); reqt_evd_hdl = DAT_HANDLE_NULL; + g_status = 1; goto client_exit; } ret = DT_Tdep_evd_create (ia_handle, @@ -164,6 +169,7 @@ DT_cs_Client (Params_t * params_ptr, module, DT_RetToString (ret)); conn_evd_hdl = DAT_HANDLE_NULL; + g_status = 1; goto client_exit; } @@ -182,6 +188,7 @@ DT_cs_Client (Params_t * params_ptr, module, DT_RetToString (ret)); ep_handle = DAT_HANDLE_NULL; + g_status = 1; goto client_exit; } DT_Tdep_PT_Debug (1, (phead, "%s: EP created\n", module)); @@ -193,6 +200,7 @@ DT_cs_Client (Params_t * params_ptr, if (!DT_query (pt_ptr, ia_handle, ep_handle) || !DT_check_params (pt_ptr, module)) { + g_status = 1; goto client_exit; } @@ -212,6 +220,7 @@ DT_cs_Client (Params_t * params_ptr, DT_Tdep_PT_Printf (phead, "%s: no memory for command buffer pool.\n", module); + g_status = 1; goto client_exit; } @@ -237,6 +246,7 @@ retry_repost: DT_Tdep_PT_Printf (phead, "%s: cannot post Server_Info recv buffer.\n", module); + g_status = 1; goto client_exit; } @@ -256,6 +266,7 @@ retry: "%s: Cannot connect Endpoint %s\n", module, DT_RetToString (ret)); + g_status = 1; goto client_exit; } @@ -296,6 +307,7 @@ retry: } } DT_Tdep_PT_Printf (phead, "%s: bad connection event\n", module); + g_status = 1; goto client_exit; } @@ -320,6 +332,7 @@ retry: DT_Bpool_GetBuffSize (bpool, 1))) { DT_Tdep_PT_Printf (phead, "%s: cannot send Client_Info\n", module); + g_status = 1; goto client_exit; } /* reap the send and verify it */ @@ -335,6 +348,7 @@ retry: dto_cookie, "Client_Info_Send")) { + g_status = 1; goto client_exit; } @@ -375,6 +389,7 @@ retry: default: { DT_Tdep_PT_Printf (phead, "Unknown Test Type\n"); + g_status = 1; goto client_exit; } } @@ -387,6 +402,7 @@ retry: DT_Bpool_GetBuffSize (bpool, 2))) { DT_Tdep_PT_Printf (phead, "%s: cannot send Command\n", module); + g_status = 1; goto client_exit; } /* reap the send and verify it */ @@ -403,6 +419,7 @@ retry: dto_cookie, "Client_Cmd_Send")) { + g_status = 1; goto client_exit; } @@ -418,6 +435,7 @@ retry: dto_cookie, "Server_Info_Recv")) { + g_status = 1; goto client_exit; } @@ -436,6 +454,7 @@ retry: module, pt_ptr->Server_Info.dapltest_version, DAPLTEST_VERSION); + g_status = 1; goto client_exit; } DT_Tdep_PT_Debug (1,(phead, "%s: Version OK!\n", module)); @@ -459,6 +478,12 @@ retry: DT_Transaction_Test_Client (pt_ptr, ia_handle, server_netaddr); + params_ptr->Client_Stats_T.bytes_rdma_read = pt_ptr->Client_Stats.bytes_rdma_read; + params_ptr->Client_Stats_T.bytes_rdma_write = pt_ptr->Client_Stats.bytes_rdma_write; + params_ptr->Client_Stats_T.bytes_recv = pt_ptr->Client_Stats.bytes_recv; + params_ptr->Client_Stats_T.bytes_send = pt_ptr->Client_Stats.bytes_send; + params_ptr->Client_Stats_T.num_ops = pt_ptr->Client_Stats.num_ops; + params_ptr->Client_Stats_T.time_ms = pt_ptr->Client_Stats.time_ms; break; } @@ -503,11 +528,13 @@ client_exit: "%s: dat_ep_disconnect (abrupt) error: %s\n", module, DT_RetToString (ret)); + g_status = 1; } else if (did_connect && !DT_disco_event_wait (phead, conn_evd_hdl, NULL)) { DT_Tdep_PT_Printf (phead, "%s: bad disconnect event\n", module); + g_status = 1; } } @@ -536,6 +563,7 @@ client_exit: module, DT_RetToString (ret)); /* keep going */ + g_status = 1; } } @@ -550,6 +578,7 @@ client_exit: module, DT_RetToString (ret)); /* keep going */ + g_status = 1; } } if (reqt_evd_hdl) @@ -562,6 +591,7 @@ client_exit: module, DT_RetToString (ret)); /* keep going */ + g_status = 1; } } if (recv_evd_hdl) @@ -574,6 +604,7 @@ client_exit: module, DT_RetToString (ret)); /* keep going */ + g_status = 1; } } @@ -588,6 +619,7 @@ client_exit: module, DT_RetToString (ret)); /* keep going */ + g_status = 1; } } @@ -602,6 +634,7 @@ client_exit: "%s: dat_ia_close (graceful) error: %s\n", module, DT_RetToString (ret)); + g_status = 1; ret = dat_ia_close (ia_handle, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) { @@ -609,6 +642,7 @@ client_exit: "%s: dat_ia_close (abrupt) error: %s\n", module, DT_RetToString (ret)); + g_status = 1; } /* keep going */ } @@ -623,8 +657,4 @@ client_exit: DT_PrintMemList (pt_ptr); /* check if we return all space allocated */ DT_Mdep_LockDestroy (&pt_ptr->MemListLock); DT_Free_Per_Test_Data (pt_ptr); - - DT_Tdep_PT_Printf (phead, - "%s: ========== End of Work -- Client Exiting\n", - module); } Index: dapltest/include/dapl_tdep.h =================================================================== --- dapltest/include/dapl_tdep.h (revision 2257) +++ dapltest/include/dapl_tdep.h (working copy) @@ -41,7 +41,7 @@ DT_Tdep_Init ( void ) ; void DT_Tdep_End ( void ) ; -void +int DT_Tdep_Execute_Test ( Params_t *params_ptr ) ; DAT_RETURN Index: dapltest/include/dapl_proto.h =================================================================== --- dapltest/include/dapl_proto.h (revision 2257) +++ dapltest/include/dapl_proto.h (working copy) @@ -49,6 +49,8 @@ #endif #include "dapl_mdep.h" +#include "dapl_transaction_stats.h" +#include "dapl_performance_stats.h" #include "dapl_tdep_print.h" #include "dapl_bpool.h" #include "dapl_client_info.h" @@ -62,7 +64,6 @@ #include "dapl_limit_cmd.h" #include "dapl_memlist.h" #include "dapl_params.h" -#include "dapl_performance_stats.h" #include "dapl_performance_test.h" #include "dapl_quit_cmd.h" #include "dapl_server_info.h" @@ -70,9 +71,9 @@ #include "dapl_test_data.h" #include "dapl_transaction_cmd.h" #include "dapl_transaction_test.h" -#include "dapl_transaction_stats.h" #include "dapl_version.h" +extern int g_status; /* * Prototypes */ @@ -125,15 +126,9 @@ void DT_init_transaction_stat void DT_transaction_stats_set_ready (DT_Tdep_Print_Head* phead, Transaction_Stats_t* transaction_stats); -void DT_transaction_stats2_set_ready (DT_Tdep_Print_Head* phead, - Transaction_Stats_t* transaction_stats); - bool DT_transaction_stats_wait_for_all (DT_Tdep_Print_Head* phead, Transaction_Stats_t* transaction_stats); -bool DT_transaction_stats2_wait_for_all (DT_Tdep_Print_Head* phead, - Transaction_Stats_t* transaction_stats); - void DT_update_transaction_stats (Transaction_Stats_t * transaction_stats, unsigned int num_ops, unsigned int time_ms, @@ -142,8 +137,7 @@ void DT_update_transaction_st unsigned int bytes_rdma_read, unsigned int bytes_rdma_write); -void DT_print_transaction_stats (DT_Tdep_Print_Head* phead, - Transaction_Stats_t* transaction_stats, +void DT_print_transaction_stats (Transaction_Stats2_t* transaction_stats, unsigned int num_threads, unsigned int num_EPs); @@ -320,16 +314,11 @@ void DT_performance_stats_com Performance_Stats_t * src_a, Performance_Stats_t * src_b); -double DT_performance_stats_data_print (DT_Tdep_Print_Head* phead, - Performance_Stats_Data_t* data, - double cpu_mhz); - -void DT_performance_stats_print (Params_t * params_ptr, - DT_Tdep_Print_Head* phead, - Performance_Stats_t * stats, - Performance_Cmd_t * cmd, - Performance_Test_t * test); +double DT_performance_stats_data_print (Performance_Stats_Data_t *data, + double cpu_mhz); +void DT_performance_stats_print (Performance_Stats_t *stats, + Performance_Cmd_t *cmd); /* dapl_server.c */ void DT_cs_Server (Params_t * params_ptr); @@ -498,9 +487,6 @@ bool DT_Transaction_Validatio int iteration); void DT_Print_Transaction_Test (DT_Tdep_Print_Head* phead, Transaction_Test_t* test_ptr); -void DT_Print_Transaction_Stats (DT_Tdep_Print_Head* phead, - Transaction_Test_t* test_ptr); - /* dapl_transaction_util.c */ bool DT_handle_post_recv_buf (DT_Tdep_Print_Head* phead, Ep_Context_t * ep_context, Index: dapltest/include/dapl_transaction_stats.h =================================================================== --- dapltest/include/dapl_transaction_stats.h (revision 2257) +++ dapltest/include/dapl_transaction_stats.h (working copy) @@ -44,4 +44,16 @@ typedef struct unsigned int bytes_rdma_read; unsigned int bytes_rdma_write; } Transaction_Stats_t; + +typedef struct +{ + unsigned int wait_count; + unsigned int num_ops; + unsigned int time_ms; + unsigned int bytes_send; + unsigned int bytes_recv; + unsigned int bytes_rdma_read; + unsigned int bytes_rdma_write; +} Transaction_Stats2_t; + #endif Index: dapltest/include/dapl_performance_stats.h =================================================================== --- dapltest/include/dapl_performance_stats.h (revision 2257) +++ dapltest/include/dapl_performance_stats.h (working copy) @@ -50,13 +50,15 @@ typedef struct int64_t bytes; unsigned int post_ctxt_switch_num; unsigned int reap_ctxt_switch_num; - double cpu_utilization; DT_Mdep_TimeStamp time_ts; Performance_Stats_Data_t posts_sans_ctxt; Performance_Stats_Data_t posts_with_ctxt; Performance_Stats_Data_t reaps_sans_ctxt; Performance_Stats_Data_t reaps_with_ctxt; Performance_Stats_Data_t latency; + DT_CpuStat pre_cpu_stat; + DT_CpuStat post_cpu_stat; + DAT_COUNT pipeline_len; } Performance_Stats_t; #endif Index: dapltest/include/dapl_params.h =================================================================== --- dapltest/include/dapl_params.h (revision 2257) +++ dapltest/include/dapl_params.h (working copy) @@ -66,8 +66,9 @@ typedef struct void * phead; bool local_is_little_endian; bool debug; - double cpu_mhz; DAT_MEM_TYPE mem_type; + Transaction_Stats2_t Client_Stats_T; + Performance_Stats_t Client_Stats_P; } Params_t; #endif Index: dapltest/cmd/dapl_main.c =================================================================== --- dapltest/cmd/dapl_main.c (revision 2257) +++ dapltest/cmd/dapl_main.c (working copy) @@ -42,6 +42,7 @@ int dapltest (int argc, char *argv[]) { Params_t *params_ptr; + int status; /* check memory leaking */ /* @@ -72,7 +73,7 @@ dapltest (int argc, char *argv[]) DT_Mdep_printf ("Cannot allocate memory for Params structure\n"); return ( 1 ); } - + memset((void *)params_ptr, 0, sizeof (Params_t)); DT_Tdep_Init (); /* init (kdapl/udapl)test-dependent code */ DT_Endian_Init (); /* init endian of local machine */ DT_Mdep_Init (); /* init OS, libraries, etc. */ @@ -87,10 +88,10 @@ dapltest (int argc, char *argv[]) DT_Mdep_printf ("Command line syntax error\n"); return 1; } - params_ptr->cpu_mhz = DT_Mdep_GetCpuMhz (); /* call the test-dependent code for invoking the actual test */ - DT_Tdep_Execute_Test (params_ptr); + status = DT_Tdep_Execute_Test (params_ptr); + printf("TEST return code = %d \n",status); /* cleanup */ DT_Mdep_End (); @@ -108,7 +109,7 @@ dapltest (int argc, char *argv[]) * alloc_count); DT_Mdep_LockDestroy(&Alloc_Count_Lock); */ - return ( 0 ); + return status; } Index: dapltest/cmd/dapl_performance_cmd.c =================================================================== --- dapltest/cmd/dapl_performance_cmd.c (revision 2257) +++ dapltest/cmd/dapl_performance_cmd.c (working copy) @@ -327,3 +327,235 @@ DT_Performance_Cmd_Print ( cmd->op.seg_size); } +const char * +DT_TransferTypeToString (DT_Transfer_Type type) +{ + static char *DT_Type[] = + { + "RR", + "RW", + "SR" + }; + + if ( (0 <= type) && (type <= 2) ) + { + return DT_Type[type]; + } + else + { + return "Error: Unkown Transfer Type"; + } +} + +double +DT_performance_stats_data_print (Performance_Stats_Data_t *data, + double cpu_mhz) +{ + double average; + + average = (int64_t)data->total_ts / (data->num * cpu_mhz); + + printf(" Arithmetic mean : %d.%d us\n" + " maximum : %d.%d us\n" + " minimum : %d.%d us\n", + DT_whole(average),DT_hundredths(average), + DT_whole((int64_t)data->max_ts / cpu_mhz),DT_hundredths((int64_t)data->max_ts / cpu_mhz), + DT_whole((int64_t)data->min_ts / cpu_mhz),DT_hundredths((int64_t)data->min_ts / cpu_mhz)); + return average; +} + + +void +DT_performance_stats_print ( + Performance_Stats_t *stats, + Performance_Cmd_t *cmd) +{ + double cpu_mhz; + double time_s; + double mbytes; + double ops_per_sec; + double bandwidth; + double latency; + double time_per_post; + double time_per_reap; + + /* calculate CPU utilization */ + unsigned long int system; + unsigned long int user; + unsigned long int idle; + unsigned long int total; + double cpu_utilization; + system = stats->post_cpu_stat.system - stats->pre_cpu_stat.system; + user = stats->post_cpu_stat.user - stats->pre_cpu_stat.user; + idle = stats->post_cpu_stat.idle - stats->pre_cpu_stat.idle; + + total = system + user + idle; + + if ( 0 == total ) + { + cpu_utilization = 0.0; + } + else + { + cpu_utilization = (double)1.0 - ((double) idle / (double) total ); + cpu_utilization *= 100.0; + } + + cpu_mhz = DT_Mdep_GetCpuMhz (); + latency = 0; +#if defined(WIN32) + /* + * The Microsoft compiler is unable to do a 64 bit conversion when + * working with double. time_ts is a 64 bit value, so we + * potentially lose precision, so limit it to the Windows + * platform. Trying to do the operation below without casting + * a 64 bit value to an unsigned int results in the error when + * using Visual C 6.0: + * + * Compiler Error C2520: conversion from unsigned __int64 to + * double not implemented, use signed __int64. + * + * Note that signed __int64 doesn't work either! + */ + time_s = (double) ((unsigned int)stats->time_ts / (1000000.0 * cpu_mhz)); +#else + time_s = (double) (stats->time_ts / (1000000.0 * cpu_mhz)); +#endif + mbytes = (double) (1.0 * stats->bytes) / 1024 / 1024; + + if ( 0.0 == time_s ) + { + printf( "Error determining time\n"); + return; + } + else if ( 0 == stats->num_ops ) + { + printf( "Error determining number of operations\n"); + return; + } + else if ( 0.0 == cpu_mhz ) + { + printf( "Error determining CPU speed\n"); + return; + } + + ops_per_sec = stats->num_ops / time_s; + bandwidth = mbytes / time_s; + printf( "------------------------- Statistics -------------------------\n"); + printf( + " Mode : %s\n" + " Operation Type : %s\n" + " Number of Operations : %u\n" + " Segment Size : %u\n" + " Number of Segments : %u bytes\n" + " Pipeline Length : %u\n\n", + DT_PerformanceModeToString (cmd->mode), + DT_TransferTypeToString (cmd->op.transfer_type), + cmd->num_iterations, + cmd->op.seg_size, + cmd->op.num_segs, + stats->pipeline_len); + + printf( + " Total Time : %d.%d sec\n" + " Total Data Exchanged : %d.%d MB\n" + " CPU Utilization : %d.%d\n" + " Operation Throughput : %d.%d ops/sec\n" + " Bandwidth : %d.%d MB/sec\n", + DT_whole(time_s),DT_hundredths(time_s), + DT_whole(mbytes),DT_hundredths(mbytes), + DT_whole(cpu_utilization),DT_hundredths(cpu_utilization), + DT_whole(ops_per_sec),DT_hundredths(ops_per_sec), + DT_whole(bandwidth),DT_hundredths(bandwidth)); + + printf( "\nLatency\n"); + + if ( stats->latency.num ) + { + latency = DT_performance_stats_data_print (&stats->latency, + cpu_mhz); + } + printf( "\n" + "Time Per Post\n" + " %u posts without context switches\n", + stats->posts_sans_ctxt.num); + + if ( stats->posts_sans_ctxt.num ) + { + DT_performance_stats_data_print (&stats->posts_sans_ctxt, + cpu_mhz); + } + + printf( "\n" + " %u posts with context switches\n", + stats->posts_with_ctxt.num); + + if ( stats->posts_with_ctxt.num ) + { + printf( " %u number of context switches\n", + stats->post_ctxt_switch_num); + DT_performance_stats_data_print (&stats->posts_with_ctxt, + cpu_mhz); + } + + printf( "\n" + "Time Per Reap\n" + " %u reaps without context switches\n", + stats->reaps_sans_ctxt.num); + + if ( stats->reaps_sans_ctxt.num ) + { + DT_performance_stats_data_print (&stats->reaps_sans_ctxt, + cpu_mhz); + } + + + printf( "\n" + " %u reaps with context switches\n", + stats->reaps_with_ctxt.num); + + if ( stats->reaps_with_ctxt.num ) + { + printf( "\n" + " %u number of context switches\n", + stats->reap_ctxt_switch_num); + + DT_performance_stats_data_print (&stats->reaps_with_ctxt, + cpu_mhz); + } + + time_per_post = + (int64_t) (stats->posts_sans_ctxt.total_ts + stats->posts_with_ctxt.total_ts) / + (cpu_mhz * (stats->posts_sans_ctxt.num + stats->posts_with_ctxt.num)); + + time_per_reap = + (int64_t) (stats->reaps_sans_ctxt.total_ts + stats->reaps_with_ctxt.total_ts) / + (cpu_mhz * (stats->reaps_sans_ctxt.num + stats->reaps_with_ctxt.num)); + + printf( + "\nNOTE: 1 MB = 1024 KB = 1048576 B \n"); + printf( + "---------------------------------------------------------------------\n"); + printf( + "raw: %s, %u, %u, %u, %u, %d.%d, %d.%d, %d.%d, %d.%d, %d.%d, %d.%d \n", + DT_TransferTypeToString (cmd->op.transfer_type), + cmd->num_iterations, + cmd->op.seg_size, + cmd->op.num_segs, + stats->pipeline_len, + DT_whole (cpu_utilization), + DT_hundredths (cpu_utilization), + DT_whole (ops_per_sec), + DT_hundredths (ops_per_sec), + DT_whole (bandwidth), + DT_hundredths (bandwidth), + DT_whole (latency), + DT_hundredths (latency), + DT_whole (time_per_post), + DT_hundredths (time_per_post), + DT_whole (time_per_reap), + DT_hundredths (time_per_reap)); + printf( + "---------------------------------------------------------------------\n"); +} + Index: dapltest/cmd/dapl_transaction_cmd.c =================================================================== --- dapltest/cmd/dapl_transaction_cmd.c (revision 2257) +++ dapltest/cmd/dapl_transaction_cmd.c (working copy) @@ -507,4 +507,72 @@ DT_Transaction_Cmd_Print (Transaction_Cm cmd->op[i].reap_send_on_recv); } } +/* + * + */ +void +DT_print_transaction_stats (Transaction_Stats2_t * transaction_stats, + unsigned int num_threads, + unsigned int num_EPs) +{ + double time_s; + double mbytes_send; + double mbytes_recv; + double mbytes_rdma_read; + double mbytes_rdma_write; + int total_ops; + // DT_Mdep_Lock (&transaction_stats->lock); + time_s = (double) (transaction_stats->time_ms) / 1000; + if (time_s == 0.0) + { + printf("----- Test completed successfully, but cannot calculate stats as not\n" + "----- enough time has lapsed.\n" + "----- Try running the test with more iterations.\n"); + goto unlock_and_return; + } + mbytes_send = (double) transaction_stats->bytes_send / 1000 / 1000; + mbytes_recv = (double) transaction_stats->bytes_recv / 1000 / 1000; + mbytes_rdma_read = (double) transaction_stats->bytes_rdma_read / 1000 / 1000; + mbytes_rdma_write = (double) transaction_stats->bytes_rdma_write / 1000 / 1000; + total_ops = transaction_stats->num_ops; + + if ( 0 == total_ops ) + { + printf("----- Test completed successfully, but no operations!\n"); + goto unlock_and_return; + } + + printf("----- Stats ---- : %u threads, %u EPs\n", + num_threads, num_EPs); + printf( "Total WQE : %7d.%02d WQE/Sec\n", + whole (total_ops / time_s), + hundredths (total_ops / time_s)); + printf( "Total Time : %7d.%02d sec\n", + whole (time_s), + hundredths (time_s)); + printf( "Total Send : %7d.%02d MB - %7d.%02d MB/Sec\n", + whole (mbytes_send), + hundredths (mbytes_send), + whole (mbytes_send / time_s), + hundredths (mbytes_send / time_s)); + printf( "Total Recv : %7d.%02d MB - %7d.%02d MB/Sec\n", + whole (mbytes_recv), + hundredths (mbytes_recv), + whole (mbytes_recv / time_s), + hundredths (mbytes_recv / time_s)); + printf( "Total RDMA Read : %7d.%02d MB - %7d.%02d MB/Sec\n", + whole (mbytes_rdma_read), + hundredths (mbytes_rdma_read), + whole (mbytes_rdma_read / time_s), + hundredths (mbytes_rdma_read / time_s)); + printf( "Total RDMA Write : %7d.%02d MB - %7d.%02d MB/Sec\n", + whole (mbytes_rdma_write), + hundredths (mbytes_rdma_write), + whole (mbytes_rdma_write / time_s), + hundredths (mbytes_rdma_write / time_s)); + printf("========== End of Work -- Client Exiting\n"); +unlock_and_return: +return; + //DT_Mdep_Unlock (&transaction_stats->lock); +} Index: dapltest/mdep/linux/dapl_mdep_user.h =================================================================== --- dapltest/mdep/linux/dapl_mdep_user.h (revision 2257) +++ dapltest/mdep/linux/dapl_mdep_user.h (working copy) @@ -40,10 +40,17 @@ #include #include -#ifdef __ia64__ +#if defined(__ia64__) +/* In suse 2.6.5 cant use timex.h BUG of suse */ +#include +#if (LINUX_VERSION_CODE < KERNEL_VERSION(2,6,0)) #include +#else +#include +#endif #endif + /* Default Device Name */ #define DT_MdepDeviceName "ia0a" @@ -114,20 +121,31 @@ typedef unsigned long long int DT_Mdep static _INLINE_ DT_Mdep_TimeStamp DT_Mdep_GetTimeStamp ( void ) { -#if defined(__i386__) || defined(__x86_64__) +#if defined(__i386__) DT_Mdep_TimeStamp x; __asm__ volatile (".byte 0x0f, 0x31" : "=A" (x)); return x; -#else - -#ifdef __ia64__ +#elif defined(__x86_64__) + unsigned low, high; + unsigned long long val; + __asm__ __volatile__("rdtsc" : "=a" (low), "=d" (high)); + val = high; + val = (val << 32) | low; + return val; +#elif defined(__ia64__) +/* In suse 2.6.5 cant use timex.h BUG of suse */ +#if (LINUX_VERSION_CODE < KERNEL_VERSION(2,6,0)) unsigned long x; - x = get_cycles (); return x; #else -#error "Non-Pentium Linux - unimplemented" + u_int64_t ia64_intri_res; + asm volatile ("mov %0=ar%1" : "=r" (ia64_intri_res) \ + : "i"(_IA64_REG_AR_ITC - _IA64_REG_AR_KR0)); + return ia64_intri_res; #endif +#else +#error "Non-Pentium Linux - unimplemented" #endif } Index: dapltest/kdapl/kdapl_module.c =================================================================== --- dapltest/kdapl/kdapl_module.c (revision 2257) +++ dapltest/kdapl/kdapl_module.c (working copy) @@ -43,7 +43,8 @@ MODULE_AUTHOR("James Lentini"); static int kdapltest_major;/* device major number */ static DT_Mdep_LockType kdapltest_lock; /* lock global test structs */ static int kdapltest_num; /* unique test invocation number */ - +static Transaction_Stats2_t Client_Stats_T; +static Performance_Stats_t Client_Stats_P; static void DT_Tdep_Test_Thread (Params_t *user_params); @@ -87,6 +88,7 @@ static int kdapltest_ioctl (struct inode rval = -ENOMEM; break; } + memset((void *)local_params, 0, sizeof (Params_t)); if (copy_from_user (local_params, (void __user *)param, sizeof (Params_t))) @@ -108,13 +110,40 @@ static int kdapltest_ioctl (struct inode rval = -ENOMEM; break; } - + g_status = 0; kernel_thread ((void*)DT_Tdep_Test_Thread, (void*)local_params, 0); rval = instance; break; } + case KDAPL_IOCTL_END_TEST: + { + int status = g_status; + if (copy_to_user((int *) param, + &status, sizeof status)) { + rval = -EFAULT; + } + break; + } + case KDAPL_IOCTL_GET_STAT_T: + { + Transaction_Stats2_t Client_Stats_local = Client_Stats_T; + if (copy_to_user((Transaction_Stats2_t *) param, + &Client_Stats_local, sizeof(Transaction_Stats2_t))) { + rval = -EFAULT; + } + break; + } + case KDAPL_IOCTL_GET_STAT_P: + { + Performance_Stats_t Client_Stats_local = Client_Stats_P; + if (copy_to_user((Performance_Stats_t *) param, + &Client_Stats_local, sizeof(Performance_Stats_t))) { + rval = -EFAULT; + } + break; + } case KDAPL_IOCTL_GET_PRINTF: { if (copy_from_user (&print_ioctl, @@ -160,6 +189,12 @@ DT_Tdep_Test_Thread (Params_t *user_para MOD_INC_USE_COUNT; #endif DT_Execute_Test (user_params); + if (user_params->test_type == TRANSACTION_TEST) { + Client_Stats_T = user_params->Client_Stats_T; + } + if (user_params->test_type == PERFORMANCE_TEST) { + Client_Stats_P = user_params->Client_Stats_P; + } DT_Tdep_PT_Printf (user_params->phead, ""); /* indicate done */ DT_Mdep_Free (user_params); #if LINUX_VERSION_CODE < KERNEL_VERSION(2,6,0) Index: dapltest/kdapl/kdapl_tdep_user.c =================================================================== --- dapltest/kdapl/kdapl_tdep_user.c (revision 2257) +++ dapltest/kdapl/kdapl_tdep_user.c (working copy) @@ -55,12 +55,14 @@ DT_Tdep_End (void) close (fd); } -void +int DT_Tdep_Execute_Test (Params_t *params_ptr) { DT_get_printf_ioctl print_ioctl; char buffer[PRINT_MAX]; - + int status; + Transaction_Stats2_t Client_Stats_T; + Performance_Stats_t Client_Stats_P; print_ioctl.buffer = &buffer[0]; print_ioctl.cookie = ioctl (fd, KDAPL_IOCTL_START_TEST, params_ptr); if (print_ioctl.cookie < 0) @@ -72,5 +74,22 @@ DT_Tdep_Execute_Test (Params_t *params_p { printf ("%s", &buffer[0]); } + if (params_ptr->test_type == TRANSACTION_TEST) { + print_ioctl.cookie = ioctl (fd, KDAPL_IOCTL_GET_STAT_T, &Client_Stats_T); + DT_print_transaction_stats(&Client_Stats_T,params_ptr->u.Transaction_Cmd.num_iterations, + params_ptr->u.Transaction_Cmd.eps_per_thread); + } + + if (params_ptr->test_type == PERFORMANCE_TEST) { + print_ioctl.cookie = ioctl (fd, KDAPL_IOCTL_GET_STAT_P, &Client_Stats_P); + DT_performance_stats_print (&Client_Stats_P, &(params_ptr->u.Performance_Cmd)); + } + print_ioctl.cookie = ioctl (fd, KDAPL_IOCTL_END_TEST, &status); printf ("TEST INSTANCE %d\n", print_ioctl.cookie); + if (print_ioctl.cookie < 0) + { + fprintf (stderr, "IOCTL failed: errno %d\n", errno); + exit (1); + } + return status; } Index: dapltest/kdapl/kdapl_ioctl.h =================================================================== --- dapltest/kdapl/kdapl_ioctl.h (revision 2257) +++ dapltest/kdapl/kdapl_ioctl.h (working copy) @@ -31,7 +31,9 @@ #define KDAPL_IOCTL_BASE (0x2828) #define KDAPL_IOCTL_START_TEST (KDAPL_IOCTL_BASE + 0) #define KDAPL_IOCTL_GET_PRINTF (KDAPL_IOCTL_BASE + 1) - +#define KDAPL_IOCTL_END_TEST (KDAPL_IOCTL_BASE + 2) +#define KDAPL_IOCTL_GET_STAT_T (KDAPL_IOCTL_BASE + 3) +#define KDAPL_IOCTL_GET_STAT_P (KDAPL_IOCTL_BASE + 4) #define PRINT_MAX 256 typedef struct { Index: dapltest/kdapl/Makefile =================================================================== --- dapltest/kdapl/Makefile (revision 2257) +++ dapltest/kdapl/Makefile (working copy) @@ -61,12 +61,7 @@ clean : #-------------------------------------------------------------------------- else -#need to define flag hard-float in i686 arch only IS_i686 = $(shell uname -m) -ifeq (${IS_i686},i686) -# Override -msoft-float in arch/i386/Makefile -EXTRA_CFLAGS += -mhard-float -endif ifeq (${IS_i686},x86_64) EXTRA_CFLAGS += -msse endif -- Itamar From mshefty at ichips.intel.com Thu May 5 10:33:09 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Thu, 05 May 2005 10:33:09 -0700 Subject: [openib-general] [PATCH] [MAD] changes to ib_create_send_mad In-Reply-To: <1115313377.4476.278.camel@localhost.localdomain> References: <1115253041.4476.147.camel@localhost.localdomain> <1115300959.4476.189.camel@localhost.localdomain> <427A5379.6060503@ichips.intel.com> <1115313377.4476.278.camel@localhost.localdomain> Message-ID: <427A58D5.8050305@ichips.intel.com> Hal Rosenstock wrote: >>I will see if I can reproduce this using the numbers that you gave. What >>size did you specify for the sge? The total number of segments is >>calculated using the sge sizes. > > > Isn't that calculated and filled in by ib_create_send_mad ? > > buf_size = get_buf_length(hdr_len, data_len) > ... > send_buf->sge.length = buf_size; > > I pretty sure I leave this along after calling ib_create_send_mad. Is > there something that needs to be done here ? I didn't realize that you were calling this. No, there's nothing more that you should need to do. - Sean From robert.j.woodruff at intel.com Thu May 5 10:36:43 2005 From: robert.j.woodruff at intel.com (Bob Woodruff) Date: Thu, 5 May 2005 10:36:43 -0700 Subject: [openib-general] RE: [OOPS] user-mode verbs In-Reply-To: <52mzr9622e.fsf@topspin.com> Message-ID: Roland wrote, >I just checked in a new version of uverbs_cmd.c that should fix the >use-after-free bugs. > - R. Ok, I will give it a try. woody From mshefty at ichips.intel.com Thu May 5 11:10:04 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Thu, 05 May 2005 11:10:04 -0700 Subject: [openib-general] cm_id state information Message-ID: <427A617C.1060607@ichips.intel.com> This came up on a separate thread... Currently the cm_id structured defined in ib_cm.h exposes a state field. This (and a couple of other fields) were originally exposed only for debug purposes, but I've noticed that both kDAPL and SDP access this field. The issue is that access to the cm_id state must be protected by a lock in order to guarantee correct operation of the CM state machine. Because CM messages can be received at any time, the cm_id state could change if it is accessed outside of the locks. While in a CM callback, the reported event should be used to determine the correct operation to take, rather than reading the cm_id state. I will try to work up some patches to correct this. I will at the very least add comment to ib_cm.h about using these fields, or move them into the internal CM data structures. - Sean From mshefty at ichips.intel.com Thu May 5 11:23:32 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Thu, 05 May 2005 11:23:32 -0700 Subject: [openib-general] [PATCH] [MAD] changes to ib_create_send_mad In-Reply-To: <1115300959.4476.189.camel@localhost.localdomain> References: <1115253041.4476.147.camel@localhost.localdomain> <1115300959.4476.189.camel@localhost.localdomain> Message-ID: <427A64A4.60007@ichips.intel.com> Hal Rosenstock wrote: > On PayloadLength, single segment sends are fine. It is multisegment > sends which seem wrong to me. > > Case 1: ib_create_send_mad with hdr_len 0x38 data_len 0x278 > paylen_newwin stored in header is 0x28C which seems correct > this creates 4 segments > 1-3 segments paylen_newwin is 0x6E0, 4th segment is 0x34 > > Case 2: ib_create_send_mad with hdr_len 0x38 data_len 0x620 > paylen_newwin stored in header is 0x634 which seems correct > this creates 8 segments > 1-7 segments paylen_newwin is 0x6E0, 4th segment is 0xBC > > Last paylen_newwin in both cases appears to me to be correct but the > paylen_newwin in the 1-n segments (0x6E0) seems wrong to me. I modified grmpp to use management class 3, and I can reproduce this problem now. Thank you for your patience. Your problem is important to us. Please continue to hold. Your problem will be handled in the order in which it was received. - Sean From iod00d at hp.com Thu May 5 11:32:44 2005 From: iod00d at hp.com (Grant Grundler) Date: Thu, 5 May 2005 11:32:44 -0700 Subject: [openib-general] PF 5/4 Telecon Minutes In-Reply-To: References: <20050505001734.GD16887@esmail.cup.hp.com> Message-ID: <20050505183244.GB19523@esmail.cup.hp.com> On Thu, May 05, 2005 at 08:30:05AM -0700, Bob Woodruff wrote: > I just noticed this too and sent an email to AJ. Apparently, there > was a mix-up and even though I received a confirmation that the paper > had been accepted, it was suppose to be an acceptance of a BOF instead, > so they gave us a BOF time slot rather than a presentation time slot. Ok. > He did say that we can submit the paper for printing in the proceedings, Cool. That's halfway. If you are preparing to submit a white paper for publication, I'd be happy to review it. The proposed topic is a good one for openib.org "documentation". Seems to match the slideset OpenIB-Arch-041221.pdf. It would probably be good to update that slideset to match the paper as well. > so it really does not matter to much if we present it in a presentation > slot or a BOF time slot. Well, I think it does matter. There are *alot* of BOFs. Too many IMHO. And people usually have enough by the end of the day. So it will be interesting to see who actually shows up after dinner instead of hanging out at the Blackthorn or some other pub wander around town for some sight seeing. thanks, grant From mshefty at ichips.intel.com Thu May 5 11:48:08 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Thu, 05 May 2005 11:48:08 -0700 Subject: [openib-general] [PATCH] [MAD] changes to ib_create_send_mad In-Reply-To: <427A64A4.60007@ichips.intel.com> References: <1115253041.4476.147.camel@localhost.localdomain> <1115300959.4476.189.camel@localhost.localdomain> <427A64A4.60007@ichips.intel.com> Message-ID: <427A6A68.8030402@ichips.intel.com> Sean Hefty wrote: > Hal Rosenstock wrote: > >> On PayloadLength, single segment sends are fine. It is multisegment >> sends which seem wrong to me. >> >> Case 1: ib_create_send_mad with hdr_len 0x38 data_len 0x278 >> paylen_newwin stored in header is 0x28C which seems correct >> this creates 4 segments >> 1-3 segments paylen_newwin is 0x6E0, 4th segment is 0x34 >> >> Case 2: ib_create_send_mad with hdr_len 0x38 data_len 0x620 >> paylen_newwin stored in header is 0x634 which seems correct >> this creates 8 segments >> 1-7 segments paylen_newwin is 0x6E0, 4th segment is 0xBC >> >> Last paylen_newwin in both cases appears to me to be correct but the >> paylen_newwin in the 1-n segments (0x6E0) seems wrong to me. Err... I spoke too soon. Here's what I see on both the send and receive side: Using SA class, hdr_len = 56, data_len = 632 (x278) 1st payload: 0x370, last: 0x34 hdr_len = 56, data_len = 1568 (x620) 1st payload: 0x6e0, last: 0xbc The middle segments in both cases equal the 1st payload, which is what I would expect. The payload values in both cases are correct. - Sean Output from the two runs below... grmpp: sent: 1, received: 0 grmpp: starting client Madeye:sent GMP MAD version....0x1 Class..........0x3 (Subnet admin.) Class version..0x2 Method.........0x1 (Get) Status.........0x00 Class specific.0x00 Trans ID.......0x2a7bcba09000000 Attr ID........0x35 Attr modifier..0x0000 Madeye:recv GMP MAD version....0x1 Class..........0x3 (Subnet admin.) Class version..0x2 Method.........0x81 (Get response) Status.........0x00 Class specific.0x00 Trans ID.......0x2a7bcba09000000 Attr ID........0x35 Attr modifier..0x0000 Madeye:sent GMP MAD version....0x1 Class..........0x3 (Subnet admin.) Class version..0x1 Method.........0x12 (Get table) Status.........0x00 Class specific.0x00 Trans ID.......0xf000000 Attr ID........0x00 Attr modifier..0x0000 RMPP version...0x1 RMPP type......0x1 (Data) RMPP RRespTime.0x0 RMPP flags.....0x3 (Active - First) RMPP status....0x0 Seg number.....0x0001 Payload len....0x0370 Madeye:recv GMP MAD version....0x1 Class..........0x3 (Subnet admin.) Class version..0x1 Method.........0x92 (Get table response) Status.........0x00 Class specific.0x00 Trans ID.......0xf000000 Attr ID........0x00 Attr modifier..0x0000 RMPP version...0x1 RMPP type......0x2 (Ack) RMPP RRespTime.0x0 RMPP flags.....0x1 (Active) RMPP status....0x0 Seg number.....0x0001 New window.....0x0041 Madeye:sent GMP MAD version....0x1 Class..........0x3 (Subnet admin.) Class version..0x1 Method.........0x12 (Get table) Status.........0x00 Class specific.0x00 Trans ID.......0xf000000 Attr ID........0x00 Attr modifier..0x0000 RMPP version...0x1 RMPP type......0x1 (Data) RMPP RRespTime.0x0 RMPP flags.....0x1 (Active) RMPP status....0x0 Seg number.....0x0002 Payload len....0x0370 Madeye:sent GMP MAD version....0x1 Class..........0x3 (Subnet admin.) Class version..0x1 Method.........0x12 (Get table) Status.........0x00 Class specific.0x00 Trans ID.......0xf000000 Attr ID........0x00 Attr modifier..0x0000 RMPP version...0x1 RMPP type......0x1 (Data) RMPP RRespTime.0x0 RMPP flags.....0x1 (Active) RMPP status....0x0 Seg number.....0x0003 Payload len....0x0370 Madeye:sent GMP MAD version....0x1 Class..........0x3 (Subnet admin.) Class version..0x1 Method.........0x12 (Get table) Status.........0x00 Class specific.0x00 Trans ID.......0xf000000 Attr ID........0x00 Attr modifier..0x0000 RMPP version...0x1 RMPP type......0x1 (Data) RMPP RRespTime.0x0 RMPP flags.....0x5 (Active - Last) RMPP status....0x0 Seg number.....0x0004 Payload len....0x0034 Madeye:recv GMP MAD version....0x1 Class..........0x3 (Subnet admin.) Class version..0x1 Method.........0x92 (Get table response) Status.........0x00 Class specific.0x00 Trans ID.......0xf000000 Attr ID........0x00 Attr modifier..0x0000 RMPP version...0x1 RMPP type......0x2 (Ack) RMPP RRespTime.0x0 RMPP flags.....0x1 (Active) RMPP status....0x0 Seg number.....0x0004 New window.....0x0041 grmpp: test time: 1000 us grmpp: test complete grmpp: sent: 1, received: 0 grmpp: starting client Madeye:sent GMP MAD version....0x1 Class..........0x3 (Subnet admin.) Class version..0x2 Method.........0x1 (Get) Status.........0x00 Class specific.0x00 Trans ID.......0x3a7bcba09000000 Attr ID........0x35 Attr modifier..0x0000 Madeye:recv GMP MAD version....0x1 Class..........0x3 (Subnet admin.) Class version..0x2 Method.........0x81 (Get response) Status.........0x00 Class specific.0x00 Trans ID.......0x3a7bcba09000000 Attr ID........0x35 Attr modifier..0x0000 Madeye:sent GMP MAD version....0x1 Class..........0x3 (Subnet admin.) Class version..0x1 Method.........0x12 (Get table) Status.........0x00 Class specific.0x00 Trans ID.......0x10000000 Attr ID........0x00 Attr modifier..0x0000 RMPP version...0x1 RMPP type......0x1 (Data) RMPP RRespTime.0x0 RMPP flags.....0x3 (Active - First) RMPP status....0x0 Seg number.....0x0001 Payload len....0x06e0 Madeye:recv GMP MAD version....0x1 Class..........0x3 (Subnet admin.) Class version..0x1 Method.........0x92 (Get table response) Status.........0x00 Class specific.0x00 Trans ID.......0x10000000 Attr ID........0x00 Attr modifier..0x0000 RMPP version...0x1 RMPP type......0x2 (Ack) RMPP RRespTime.0x0 RMPP flags.....0x1 (Active) RMPP status....0x0 Seg number.....0x0001 New window.....0x0041 Madeye:sent GMP MAD version....0x1 Class..........0x3 (Subnet admin.) Class version..0x1 Method.........0x12 (Get table) Status.........0x00 Class specific.0x00 Trans ID.......0x10000000 Attr ID........0x00 Attr modifier..0x0000 RMPP version...0x1 RMPP type......0x1 (Data) RMPP RRespTime.0x0 RMPP flags.....0x1 (Active) RMPP status....0x0 Seg number.....0x0002 Payload len....0x06e0 Madeye:sent GMP MAD version....0x1 Class..........0x3 (Subnet admin.) Class version..0x1 Method.........0x12 (Get table) Status.........0x00 Class specific.0x00 Trans ID.......0x10000000 Attr ID........0x00 Attr modifier..0x0000 RMPP version...0x1 RMPP type......0x1 (Data) RMPP RRespTime.0x0 RMPP flags.....0x1 (Active) RMPP status....0x0 Seg number.....0x0003 Payload len....0x06e0 Madeye:sent GMP MAD version....0x1 Class..........0x3 (Subnet admin.) Class version..0x1 Method.........0x12 (Get table) Status.........0x00 Class specific.0x00 Trans ID.......0x10000000 Attr ID........0x00 Attr modifier..0x0000 RMPP version...0x1 RMPP type......0x1 (Data) RMPP RRespTime.0x0 RMPP flags.....0x1 (Active) RMPP status....0x0 Seg number.....0x0004 Payload len....0x06e0 Madeye:sent GMP MAD version....0x1 Class..........0x3 (Subnet admin.) Class version..0x1 Method.........0x12 (Get table) Status.........0x00 Class specific.0x00 Trans ID.......0x10000000 Attr ID........0x00 Attr modifier..0x0000 RMPP version...0x1 RMPP type......0x1 (Data) RMPP RRespTime.0x0 RMPP flags.....0x1 (Active) RMPP status....0x0 Seg number.....0x0005 Payload len....0x06e0 Madeye:sent GMP MAD version....0x1 Class..........0x3 (Subnet admin.) Class version..0x1 Method.........0x12 (Get table) Status.........0x00 Class specific.0x00 Trans ID.......0x10000000 Attr ID........0x00 Attr modifier..0x0000 RMPP version...0x1 RMPP type......0x1 (Data) RMPP RRespTime.0x0 RMPP flags.....0x1 (Active) RMPP status....0x0 Seg number.....0x0006 Payload len....0x06e0 Madeye:sent GMP MAD version....0x1 Class..........0x3 (Subnet admin.) Class version..0x1 Method.........0x12 (Get table) Status.........0x00 Class specific.0x00 Trans ID.......0x10000000 Attr ID........0x00 Attr modifier..0x0000 RMPP version...0x1 RMPP type......0x1 (Data) RMPP RRespTime.0x0 RMPP flags.....0x1 (Active) RMPP status....0x0 Seg number.....0x0007 Payload len....0x06e0 Madeye:sent GMP MAD version....0x1 Class..........0x3 (Subnet admin.) Class version..0x1 Method.........0x12 (Get table) Status.........0x00 Class specific.0x00 Trans ID.......0x10000000 Attr ID........0x00 Attr modifier..0x0000 RMPP version...0x1 RMPP type......0x1 (Data) RMPP RRespTime.0x0 RMPP flags.....0x5 (Active - Last) RMPP status....0x0 Seg number.....0x0008 Payload len....0x00bc Madeye:recv GMP MAD version....0x1 Class..........0x3 (Subnet admin.) Class version..0x1 Method.........0x92 (Get table response) Status.........0x00 Class specific.0x00 Trans ID.......0x10000000 Attr ID........0x00 Attr modifier..0x0000 RMPP version...0x1 RMPP type......0x2 (Ack) RMPP RRespTime.0x0 RMPP flags.....0x1 (Active) RMPP status....0x0 Seg number.....0x0008 New window.....0x0041 grmpp: test time: 1000 us grmpp: test complete From timur.tabi at ammasso.com Thu May 5 11:48:30 2005 From: timur.tabi at ammasso.com (Timur Tabi) Date: Thu, 05 May 2005 13:48:30 -0500 Subject: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation In-Reply-To: <4279142A.8050501@ammasso.com> References: <200544159.Ahk9l0puXy39U6u6@topspin.com> <20050411142213.GC26127@kalmia.hozed.org> <52mzs51g5g.fsf@topspin.com> <20050411163342.GE26127@kalmia.hozed.org> <5264yt1cbu.fsf@topspin.com> <20050411180107.GF26127@kalmia.hozed.org> <52oeclyyw3.fsf@topspin.com> <20050411171347.7e05859f.akpm@osdl.org> <20050412180447.E6958@topspin.com> <20050425203110.A9729@topspin.com> <4279142A.8050501@ammasso.com> Message-ID: <427A6A7E.8000604@ammasso.com> Timur Tabi wrote: > When you say "older", what exactly do you mean? I have different test > that normally fails with just get_user_pages(), but it works with 2.6.9 > and above. I haven't been able to get any kernel earlier than 2.6.9 to > compile or boot properly, so I'm having a hard time narrowing down the > actual point when get_user_pages() started working. I haven't gotten a reply to this question, but I've done my own research, and I think I found the answer. Using my own test of get_user_pages(), it appears that the fix was placed in 2.6.7. However, I would like to know specifically what the fix is. Unfortunately, tracking this stuff down is beyond my understanding of the Linux VM. Assuming that the fix is in try_to_unmap_one(), the only significant change I see between 2.6.6 and 2.6.7 is the addition of this code: pgd = pgd_offset(mm, address); if (!pgd_present(*pgd)) goto out_unlock; pmd = pmd_offset(pgd, address); if (!pmd_present(*pmd)) goto out_unlock; pte = pte_offset_map(pmd, address); if (!pte_present(*pte)) goto out_unmap; if (page_to_pfn(page) != pte_pfn(*pte)) goto out_unmap; Can anyone tell me if this is the actual fix, or at least a major part of the actual fix? -- Timur Tabi Staff Software Engineer timur.tabi at ammasso.com One thing a Southern boy will never say is, "I don't think duct tape will fix it." -- Ed Smylie, NASA engineer for Apollo 13 From tduffy at sun.com Thu May 5 11:51:21 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 05 May 2005 11:51:21 -0700 Subject: [openib-general] Re: [openib-commits] r2260 - in gen2/trunk/src/linux-kernel/infiniband: core include In-Reply-To: <20050505174645.3972C22834D@openib.ca.sandia.gov> References: <20050505174645.3972C22834D@openib.ca.sandia.gov> Message-ID: <1115319081.13489.4.camel@duffman> On Thu, 2005-05-05 at 10:46 -0700, sean.hefty at openib.org wrote: > Modified: > gen2/trunk/src/linux-kernel/infiniband/core/mad.c > gen2/trunk/src/linux-kernel/infiniband/include/ib_mad.h > Log: > Add formatting of RMPP header to ib_create_send_mad. Clear > allocated MAD data buffer. You probably already know this, but this checkin breaks the build: CC [M] drivers/infiniband/core/ping.o /build1/tduffy/openib-work/linux-2.6.12-rc3-openib/drivers/infiniband/core/ping.c: In function ‘pingd_recv_handler’: /build1/tduffy/openib-work/linux-2.6.12-rc3-openib/drivers/infiniband/core/ping.c:131: error: too few arguments to function ‘ib_create_send_mad’ make[4]: *** [drivers/infiniband/core/ping.o] Error 1 make[3]: *** [drivers/infiniband/core] Error 2 make[2]: *** [drivers/infiniband] Error 2 make[1]: *** [drivers] Error 2 make: *** [_all] Error 2 This patch fixes the build assuming ping does not need rmpp. Signed-off-by: Tom Duffy Index: drivers/infiniband/core/ping.c =================================================================== --- drivers/infiniband/core/ping.c (revision 2262) +++ drivers/infiniband/core/ping.c (working copy) @@ -124,7 +124,7 @@ static void pingd_recv_handler(struct ib } msg = ib_create_send_mad(mad_agent, mad_recv_wc->wc->src_qp, - mad_recv_wc->wc->pkey_index, ah, + mad_recv_wc->wc->pkey_index, ah, 0, offsetof(struct ib_vendor_mad, data), mad_recv_wc->mad_len - offsetof(struct ib_vendor_mad, data), From sean.hefty at intel.com Thu May 5 11:56:25 2005 From: sean.hefty at intel.com (Hefty, Sean) Date: Thu, 5 May 2005 11:56:25 -0700 Subject: [openib-general] Re: [openib-commits] r2260 - ingen2/trunk/src/linux-kernel/infiniband: core include Message-ID: >On Thu, 2005-05-05 at 10:46 -0700, sean.hefty at openib.org wrote: >> Modified: >> gen2/trunk/src/linux-kernel/infiniband/core/mad.c >> gen2/trunk/src/linux-kernel/infiniband/include/ib_mad.h >> Log: >> Add formatting of RMPP header to ib_create_send_mad. Clear >> allocated MAD data buffer. > >You probably already know this, but this checkin breaks the build: Actually, I didn't know this. I wasn't aware that anything else was calling this function, and I don't include ping as part of my build. Sorry about that, I'll start including it, and apply this patch. - Sean > CC [M] drivers/infiniband/core/ping.o >/build1/tduffy/openib-work/linux-2.6.12-rc3-openib/drivers/infiniband/c ore/ping.c: In function >'pingd_recv_handler': >/build1/tduffy/openib-work/linux-2.6.12-rc3-openib/drivers/infiniband/c ore/ping.c:131: error: too >few arguments to function 'ib_create_send_mad' >make[4]: *** [drivers/infiniband/core/ping.o] Error 1 >make[3]: *** [drivers/infiniband/core] Error 2 >make[2]: *** [drivers/infiniband] Error 2 >make[1]: *** [drivers] Error 2 >make: *** [_all] Error 2 > >This patch fixes the build assuming ping does not need rmpp. > >Signed-off-by: Tom Duffy > >Index: drivers/infiniband/core/ping.c >=================================================================== >--- drivers/infiniband/core/ping.c (revision 2262) >+++ drivers/infiniband/core/ping.c (working copy) >@@ -124,7 +124,7 @@ static void pingd_recv_handler(struct ib > } > > msg = ib_create_send_mad(mad_agent, mad_recv_wc->wc->src_qp, >- mad_recv_wc->wc->pkey_index, ah, >+ mad_recv_wc->wc->pkey_index, ah, 0, > offsetof(struct ib_vendor_mad, data), > mad_recv_wc->mad_len - > offsetof(struct ib_vendor_mad, data), > >_______________________________________________ >openib-general mailing list >openib-general at openib.org >http://openib.org/mailman/listinfo/openib-general > >To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From halr at voltaire.com Thu May 5 11:49:51 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 05 May 2005 14:49:51 -0400 Subject: [openib-general] Re: [openib-commits] r2260 - in gen2/trunk/src/linux-kernel/infiniband: core include In-Reply-To: <1115319081.13489.4.camel@duffman> References: <20050505174645.3972C22834D@openib.ca.sandia.gov> <1115319081.13489.4.camel@duffman> Message-ID: <1115318991.4695.5.camel@localhost.localdomain> On Thu, 2005-05-05 at 14:51, Tom Duffy wrote: > This patch fixes the build assuming ping does not need rmpp. Yes, (ib) ping does not currently support RMPP (yet). -- Hal From halr at voltaire.com Thu May 5 11:53:26 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 05 May 2005 14:53:26 -0400 Subject: [openib-general] [PATCH] [MAD] changes to ib_create_send_mad In-Reply-To: <427A6A68.8030402@ichips.intel.com> References: <1115253041.4476.147.camel@localhost.localdomain> <1115300959.4476.189.camel@localhost.localdomain> <427A64A4.60007@ichips.intel.com> <427A6A68.8030402@ichips.intel.com> Message-ID: <1115319121.4695.8.camel@localhost.localdomain> On Thu, 2005-05-05 at 14:48, Sean Hefty wrote: > Sean Hefty wrote: > > Hal Rosenstock wrote: > > > >> On PayloadLength, single segment sends are fine. It is multisegment > >> sends which seem wrong to me. > >> > >> Case 1: ib_create_send_mad with hdr_len 0x38 data_len 0x278 > >> paylen_newwin stored in header is 0x28C which seems correct > >> this creates 4 segments > >> 1-3 segments paylen_newwin is 0x6E0, 4th segment is 0x34 > >> > >> Case 2: ib_create_send_mad with hdr_len 0x38 data_len 0x620 > >> paylen_newwin stored in header is 0x634 which seems correct > >> this creates 8 segments > >> 1-7 segments paylen_newwin is 0x6E0, 4th segment is 0xBC > >> > >> Last paylen_newwin in both cases appears to me to be correct but the > >> paylen_newwin in the 1-n segments (0x6E0) seems wrong to me. > > Err... I spoke too soon. Here's what I see on both the send and receive > side: Using SA class, > > hdr_len = 56, data_len = 632 (x278) > 1st payload: 0x370, last: 0x34 > > hdr_len = 56, data_len = 1568 (x620) > 1st payload: 0x6e0, last: 0xbc > > The middle segments in both cases equal the 1st payload, which is what I > would expect. Me too. (They should be ignored anyhow). > The payload values in both cases are correct. Let me look again. I don't think I was hallucinating... -- Hal From halr at voltaire.com Thu May 5 12:07:48 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 05 May 2005 15:07:48 -0400 Subject: [openib-general] PF 5/4 Telecon Minutes In-Reply-To: <20050505183244.GB19523@esmail.cup.hp.com> References: <20050505001734.GD16887@esmail.cup.hp.com> <20050505183244.GB19523@esmail.cup.hp.com> Message-ID: <1115320068.4476.1.camel@localhost.localdomain> Hi Grant, On Thu, 2005-05-05 at 14:32, Grant Grundler wrote: > The proposed topic is a good one for openib.org "documentation". > Seems to match the slideset OpenIB-Arch-041221.pdf. It would probably > be good to update that slideset to match the paper as well. As you can well imagine, that slide deck is somewhat out of date. The question is when it needs to be updated to be in sync. I don't think it would take very long but want to minimize the number of times I do this. -- Hal From tduffy at sun.com Thu May 5 12:18:55 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 05 May 2005 12:18:55 -0700 Subject: [openib-general] [kDAPL][PATCH] fix panic in server side In-Reply-To: <1115272856.4476.52.camel@localhost.localdomain> References: <1115243969.14238.6.camel@duffman> <1115272856.4476.52.camel@localhost.localdomain> Message-ID: <1115320735.13489.9.camel@duffman> On Thu, 2005-05-05 at 02:00 -0400, Hal Rosenstock wrote: > Hmm, when I do this (on x86 and ), I get the following: > DT_cs_Client: Connect Endpoint > DT_cs_Client: Await connection ... > Warning: conn_event_wait DAT_CONNECTION_EVENT_DISCONNECTED > DT_cs_Client: bad connection event > DT_cs_Client: Cleaning Up ... > DT_cs_Client: IA mthca0a closed > DT_cs_Client: ========== End of Work -- Client Exiting > > This takes a while until REQ timeout/retries are exhausted. > > On x86_64, I do get the following (same as you in userland): > DT_cs_Client: Connect Endpoint > DT_cs_Client: Await connection ... > Warning: conn_event_wait DAT_CONNECTION_EVENT_DISCONNECTED > DT_cs_Client: bad connection event > DT_cs_Client: Cleaning Up ... > DT_cs_Client: IA mthca0a closed > DT_cs_Client: ========== End of Work -- Client Exiting > > *** glibc detected *** double free or corruption: 0x000000000050c010 *** > Aborted > > but no oops. That is a kdapltest test program issue as you noted. > > How long did you wait ? Did you ctl-C out ? Might have ;-) > I'm presuming you loaded IPoIB and are pointing the client at an IPoIB > which is pingable but does not have a kdapl server (connection listener) > started up. Yes it is pingable: [root at sins-stinger-10 ~]# !ping ping -b 192.168.0.0 WARNING: pinging broadcast address PING 192.168.0.0 (192.168.0.0) 56(84) bytes of data. 64 bytes from 192.168.0.233: icmp_seq=0 ttl=64 time=0.063 ms 64 bytes from 192.168.0.26: icmp_seq=0 ttl=64 time=0.210 ms (DUP!) > Would you describe your procedure some more so I can see if I can > recreate this ? Now I am getting something weird. I cannot even connect: [root at sins-stinger-10 ~]# ./kdapltest -T Q -s 192.168.0.26 -D mthca0a -d Server Name: 192.168.0.26 Server Net Address: 192.168.0.26 DT_cs_Client: Starting Test ... DT_cs_Client: IA mthca0a opened DT_cs_Client: EP created ***** DAPL Characteristics ***** Provider: mthca0a Version 1.0 DAPL 1.2 Adapter: Generic InfiniBand HCA by DAPL Reference Implementation Version 0.0 Supporting: 64512 EPs with 65535 DTOs and 0 in RDMA/RDs and 0ut RDMA/RDs each 65408 EVDs of up to 65535 entries (default S/R size is 16/16) IOVs of up to 28 elements 131056 LMRs (and 131056 RMRs) of up to 0xffffffffffffffff bytes Maximum MTU 0x80000000 bytes, RDMA 0x80000000 bytes Maximum Private data size 92 bytes ***** ***** ***** ***** ***** ***** DT_BpoolAlloc: dat_lmr_kcreate failed DAT_NOT_IMPLEMENTED DT_cs_Client: no memory for command buffer pool. DT_cs_Client: Cleaning Up ... DT_cs_Client: IA mthca0a closed TEST INSTANCE 0 TEST return code = 1 Grr.. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From blist at aon.at Thu May 5 12:26:14 2005 From: blist at aon.at (Bernhard Fischer) Date: Thu, 5 May 2005 21:26:14 +0200 Subject: [openib-general] [patch] misc trivia nitpicks 03 Message-ID: <20050505192614.GW15688@aon.at> Hi, What's the status of SDP_EXPECT ? will it be submitted upstream or cleaned up beforehand? Just curious. - s/conenct/connect/g - s/conect/connect/g - trim whitespace - fold sdp_conn_internal_lock into sdp_conn_lock. - fold sdp_conn_internal_unlock into sdp_conn_unlock. PS: those folds are completely untested.. Libor, anyone else? thank you, -------------- next part -------------- diff -X excl -rduNp gen2.2257.oorig/trunk/src/userspace/management/doc/diagtools.txt gen2.2257/trunk/src/userspace/management/doc/diagtools.txt --- gen2.2257.oorig/trunk/src/userspace/management/doc/diagtools.txt 2005-04-26 20:58:41.000000000 +0200 +++ gen2.2257/trunk/src/userspace/management/doc/diagtools.txt 2005-05-05 20:33:53.572745456 +0200 @@ -335,7 +335,7 @@ smpquery, smpquery output format, ibaddr 16. ibnetchecknode Description: -Check conectivity and do some simple sanity checks for the specified node. +Check connectivity and do some simple sanity checks for the specified node. Port address is lid unless -G option is used to specify a GUID address. Syntax: -------------- next part -------------- diff -X excl -rduNp gen2.2257.oorig/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_conn.c gen2.2257/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_conn.c --- gen2.2257.oorig/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_conn.c 2005-04-26 20:58:46.000000000 +0200 +++ gen2.2257/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_conn.c 2005-05-05 20:32:42.415562992 +0200 @@ -595,8 +595,8 @@ done: return conn; } -/* - * Functions to cancel IOCB requests in a conenctions queues. +/* + * Functions to cancel IOCB requests in a connections queues. */ static int sdp_desc_q_cancel_lookup_func(struct sdpc_desc *element, void *arg) { -------------- next part -------------- diff -X excl -rduNp gen2.2257.oorig/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_conn.c gen2.2257/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_conn.c --- gen2.2257.oorig/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_conn.c 2005-04-26 20:58:46.000000000 +0200 +++ gen2.2257/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_conn.c 2005-05-05 21:24:27.895458288 +0200 @@ -352,13 +352,13 @@ int sdp_inet_port_get(struct sdp_opt *co INADDR_ANY == look->src_addr || conn->src_addr == look->src_addr) { - sdp_dbg_warn(conn, + sdp_dbg_warn(conn, "port rejected. <%04x><%d:%d><%d:%d><%04x><%u:%u>", port, sk->sk_bound_dev_if, srch->sk_bound_dev_if, sk->sk_reuse, - srch->sk_reuse, + srch->sk_reuse, look->state, conn->src_addr, look->src_addr); @@ -670,6 +670,7 @@ void sdp_conn_destruct(struct sdp_opt *c result = sdp_conn_stat_dump(conn); SDP_EXPECT(result >= 0); + /* * really there shouldn't be anything in these tables, but it's * really bad if we leave a dangling reference here. @@ -748,7 +749,7 @@ void sdp_conn_destruct(struct sdp_opt *c result); } /* - * check consistancy + * check consistency */ if (atomic_read(&conn->refcnt) < 0) sdp_dbg_warn(conn, "destruct low ref count <%04x>", @@ -812,7 +789,7 @@ void sdp_conn_relock(struct sdp_opt *con if (1 == result_r) { result = sdp_cq_event_locked(&entry, conn); if (result < 0) - sdp_dbg_warn(conn, + sdp_dbg_warn(conn, "Error <%d> from event handler.", result); @@ -823,7 +800,7 @@ void sdp_conn_relock(struct sdp_opt *con if (1 == result_s) { result = sdp_cq_event_locked(&entry, conn); if (result < 0) - sdp_dbg_warn(conn, + sdp_dbg_warn(conn, "Error <%d> from event handler.", result); rearm = 1; @@ -836,17 +813,17 @@ void sdp_conn_relock(struct sdp_opt *con result = ib_req_notify_cq(conn->recv_cq, IB_CQ_NEXT_COMP); if (result) - sdp_dbg_warn(conn, + sdp_dbg_warn(conn, "Error <%d> rearming recv CQ", result); result = ib_req_notify_cq(conn->send_cq, IB_CQ_NEXT_COMP); if (result) - sdp_dbg_warn(conn, + sdp_dbg_warn(conn, "Error <%d> rearming send CQ", result); - + rearm = 0; } else break; /* exit CQ handler routine */ @@ -858,18 +835,20 @@ void sdp_conn_relock(struct sdp_opt *con } /* - * sdp_conn_cq_drain - drain one of the the connection's CQs + * sdp_conn_cq_drain - drain one of the connection's CQs */ +/* + * the function should only be called under the connection locks + * spinlock to ensure the call is serialized to avoid races. + */ + int sdp_conn_cq_drain(struct ib_cq *cq, struct sdp_opt *conn) { struct ib_wc entry; int result; int rearm = 1; int calls = 0; - /* - * the function should only be called under the connection locks - * spinlock to ensure the call is serialized to avoid races. - */ + for (;;) { /* * poll for a completion @@ -877,7 +856,7 @@ int sdp_conn_cq_drain(struct ib_cq *cq, result = ib_poll_cq(cq, 1, &entry); if (1 == result) { /* - * dispatch completion, and mark that the CQ needs + * dispatch completion, and mark that the CQ needs * to be armed. */ result = sdp_cq_event_locked(&entry, conn); @@ -895,7 +874,7 @@ int sdp_conn_cq_drain(struct ib_cq *cq, if (rearm > 0) { result = ib_req_notify_cq(cq, IB_CQ_NEXT_COMP); if (result) - sdp_dbg_warn(conn, + sdp_dbg_warn(conn, "Error <%d> rearming CQ", result); rearm = 0; @@ -973,13 +931,13 @@ int sdp_conn_alloc_ib(struct sdp_opt *co result = -ENOMEM; goto error_attr; } - + init_attr = kmalloc(sizeof(*init_attr), GFP_KERNEL); if (!init_attr) { result = -ENOMEM; goto error_param; } - + memset(qp_attr, 0, sizeof(*qp_attr)); memset(init_attr, 0, sizeof(*init_attr)); /* @@ -1121,7 +1079,7 @@ struct sdp_opt *sdp_conn_alloc(int prior struct sock *sk; int result; - sk = sk_alloc(dev_root_s.proto, priority, + sk = sk_alloc(dev_root_s.proto, priority, sizeof(struct inet_sock), dev_root_s.sock_cache); if (!sk) { sdp_dbg_warn(NULL, "socket alloc error for protocol. <%d:%d>", @@ -1316,12 +1274,12 @@ error: "dst address:port src address:port ID comm_id pid " \ " dst guid src guid dlid slid dqpn " \ "sqpn data sent buff'd data rcvd_buff'd " \ - " data written data read src_serv snk_serv\n" + " data written data read src_serv snk_serv\n" #define SDP_PROC_CONN_MAIN_SEP \ "---------------- ---------------- ---- -------- ---- " \ "---------------- ---------------- ---- ---- ------ " \ "------ ---------------- ---------------- " \ - "---------------- ---------------- -------- --------\n" + "---------------- ---------------- -------- --------\n" #define SDP_PROC_CONN_MAIN_FORM \ "%02x.%02x.%02x.%02x:%04x %02x.%02x.%02x.%02x:%04x " \ "%04x %08x %04x %08x%08x %08x%08x %04x %04x " \ @@ -1330,7 +1288,7 @@ error: /* * sdp_proc_dump_conn_main - dump the connection table to /proc */ -int sdp_proc_dump_conn_main(char *buffer, int max_size, off_t start_index, +int sdp_proc_dump_conn_main(char *buffer, int max_size, off_t start_index, long *end_index) { struct sdp_opt *conn; @@ -1360,7 +1318,7 @@ int sdp_proc_dump_conn_main(char *buffer /* * loop across connections. */ - for (counter = start_index; + for (counter = start_index; counter < dev_root_s.sk_size && !(SDP_CONN_PROC_MAIN_SIZE > (max_size - offset)); counter++) { @@ -1382,7 +1340,7 @@ int sdp_proc_dump_conn_main(char *buffer ((conn->src_addr >> 8) & 0xff), ((conn->src_addr >> 16) & 0xff), ((conn->src_addr >> 24) & 0xff), - conn->src_port, + conn->src_port, conn->hashent, conn->cm_id ? conn->cm_id->local_id : 0, conn->pid, @@ -1801,8 +1759,8 @@ static void sdp_device_init_one(struct i port->next = hca->port_list; hca->port_list = port; - result = ib_query_gid(hca->ca, - port->index, + result = ib_query_gid(hca->ca, + port->index, 0, /* index */ &port->gid); if (result) { @@ -1852,7 +1810,7 @@ static void sdp_device_remove_one(struct sdp_warn("Device <%s> has no HCA info.", device->name); return; } - + while (hca->port_list) { port = hca->port_list; hca->port_list = port->next; @@ -1909,7 +1867,7 @@ int sdp_conn_table_init(int proto_family dev_root_s.recv_buff_max = recv_buff_max; dev_root_s.send_post_max = send_post_max; dev_root_s.send_buff_max = send_buff_max; - + dev_root_s.send_usig_max = send_usig_max; /* * Get HCA/port list @@ -1967,7 +1925,7 @@ int sdp_conn_table_init(int proto_family } dev_root_s.sock_cache = kmem_cache_create("sdp_sock", - sizeof(struct inet_sock), + sizeof(struct inet_sock), 0, SLAB_HWCACHE_ALIGN, NULL, NULL); if (!dev_root_s.sock_cache) { -------------- next part -------------- diff -X excl -rduNp gen2.2257.oorig/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_conn.c gen2.2257/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_conn.c --- gen2.2257.oorig/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_conn.c 2005-04-26 20:58:46.000000000 +0200 +++ gen2.2257/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_conn.c 2005-05-05 21:24:27.895458288 +0200 @@ -770,30 +771,6 @@ void sdp_conn_destruct(struct sdp_opt *c } /* - * sdp_conn_internal_lock - lock the connection (use only from macro) - */ -void sdp_conn_internal_lock(struct sdp_opt *conn, unsigned long *flags) -{ - DECLARE_WAITQUEUE(wait, current); - unsigned long f = *flags; - - add_wait_queue_exclusive(&(conn->lock.waitq), &wait); - for (;;) { - current->state = TASK_UNINTERRUPTIBLE; - spin_unlock_irqrestore(&(conn->lock.slock), f); - schedule(); - spin_lock_irqsave(&(conn->lock.slock), f); - *flags = f; - - if (!conn->lock.users) - break; - } - - current->state = TASK_RUNNING; - remove_wait_queue(&(conn->lock.waitq), &wait); -} - -/* * sdp_conn_relock - test the connection (use only from macro) */ void sdp_conn_relock(struct sdp_opt *conn) @@ -905,28 +884,7 @@ int sdp_conn_cq_drain(struct ib_cq *cq, sdp_dbg_warn(conn, "Unexpected error <%d> from CQ", result); } - -return calls; -} - -/* - * sdp_conn_internal_unlock - lock the connection (use only from macro) - */ -void sdp_conn_internal_unlock(struct sdp_opt *conn) -{ - int calls = 0; - /* - * poll CQs for events. - */ - if (conn) { - if (conn->flags & SDP_CONN_F_RECV_CQ_PEND) - calls += sdp_conn_cq_drain(conn->recv_cq, conn); - - if (conn->flags & SDP_CONN_F_SEND_CQ_PEND) - calls += sdp_conn_cq_drain(conn->send_cq, conn); - - conn->flags &= ~SDP_CONN_F_MASK_EVENT; - } + return /* finished # of */calls; } /* diff -X excl -rduNp gen2.2257.oorig/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_conn.h gen2.2257/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_conn.h --- gen2.2257.oorig/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_conn.h 2005-04-26 20:58:46.000000000 +0200 +++ gen2.2257/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_conn.h 2005-05-05 21:21:53.964859296 +0200 @@ -462,8 +462,6 @@ struct sdp_opt { /* * SDP connection lock */ -extern void sdp_conn_internal_lock(struct sdp_opt *conn, unsigned long *flags); -extern void sdp_conn_internal_unlock(struct sdp_opt *conn); extern void sdp_conn_relock(struct sdp_opt *conn); extern void sdp_conn_destruct(struct sdp_opt *conn); extern int sdp_conn_cq_drain(struct ib_cq *cq, struct sdp_opt *conn); @@ -482,7 +480,18 @@ static inline void sdp_conn_lock(struct spin_lock_irqsave(&conn->lock.slock, flags); if (conn->lock.users != 0) { - sdp_conn_internal_lock(conn, &flags); + DECLARE_WAITQUEUE(wait, current); + add_wait_queue_exclusive(&(conn->lock.waitq), &wait); + for (;;) { + current->state = TASK_UNINTERRUPTIBLE; + spin_unlock_irqrestore(&(conn->lock.slock), flags); + schedule(); + spin_lock_irqsave(&(conn->lock.slock), flags); + if (!conn->lock.users) + break; + } + current->state = TASK_RUNNING; + remove_wait_queue(&(conn->lock.waitq), &wait); } conn->lock.users = 1; @@ -497,7 +506,16 @@ static inline void sdp_conn_unlock(struc if ((conn->flags & SDP_CONN_F_MASK_EVENT) && (SDP_ST_MASK_EVENTS & conn->state)) { - sdp_conn_internal_unlock(conn); + /* + * poll CQs for events. + */ + if (conn->flags & SDP_CONN_F_RECV_CQ_PEND) + sdp_conn_cq_drain(conn->recv_cq, conn); + + if (conn->flags & SDP_CONN_F_SEND_CQ_PEND) + sdp_conn_cq_drain(conn->send_cq, conn); + + conn->flags &= ~SDP_CONN_F_MASK_EVENT; } conn->lock.users = 0; -------------- next part -------------- diff -X excl -rduNp gen2.2257.oorig/trunk/src/userspace/management/doc/diagtools.txt gen2.2257/trunk/src/userspace/management/doc/diagtools.txt --- gen2.2257.oorig/trunk/src/userspace/management/doc/diagtools.txt 2005-04-26 20:58:41.000000000 +0200 +++ gen2.2257/trunk/src/userspace/management/doc/diagtools.txt 2005-05-05 20:33:53.572745456 +0200 @@ -335,7 +335,7 @@ smpquery, smpquery output format, ibaddr 16. ibnetchecknode Description: -Check conectivity and do some simple sanity checks for the specified node. +Check connectivity and do some simple sanity checks for the specified node. Port address is lid unless -G option is used to specify a GUID address. Syntax: From blist at aon.at Thu May 5 13:16:27 2005 From: blist at aon.at (Bernhard Fischer) Date: Thu, 5 May 2005 22:16:27 +0200 Subject: [openib-general] Re: [openib-commits] r2239 - gen2/trunk/src/linux-kernel/infiniband/include Message-ID: <20050505201626.GX15688@aon.at> Hal, - Fix commentary typo: s/intialized/initialized/g Would you mind also fixing src/userspace/libsdp/src/port.c Please, if you happen to find those, grep the whole tree for occurances. My current -ENOPARSE list looks like: conenctions esablished etablish defintions defaul\ a\ single\ buffers retreival recieve explict implict responsab listning conntension consistancy the\ the intia Furthermore it would be really, really nice if anyone who touches files would trim any whitespace as that is extremely annoying if you happen to be forced to look at. Also, whitespace just eats my precious (disc-)space for no good. TIA, From halr at voltaire.com Thu May 5 13:16:37 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 05 May 2005 16:16:37 -0400 Subject: [openib-general] Re: [openib-commits] r2239 - gen2/trunk/src/linux-kernel/infiniband/include In-Reply-To: <20050505201626.GX15688@aon.at> References: <20050505201626.GX15688@aon.at> Message-ID: <1115324197.4476.3.camel@localhost.localdomain> On Thu, 2005-05-05 at 16:16, Bernhard Fischer wrote: > Hal, > > - Fix commentary typo: s/intialized/initialized/g > > Would you mind also fixing > src/userspace/libsdp/src/port.c > > Please, if you happen to find those, grep the whole tree for occurances. > > My current -ENOPARSE list looks like: > > conenctions > esablished > etablish > defintions > defaul\ > a\ single\ buffers > retreival > recieve > explict > implict > responsab > listning > conntension > consistancy > the\ the > intia Libor is the maintainer for SDP so this is a request to him. -- Hal > Furthermore it would be really, really nice if anyone who touches files > would trim any whitespace as that is extremely annoying if you happen to > be forced to look at. Also, whitespace just eats my precious > (disc-)space for no good. > > TIA, > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From blist at aon.at Thu May 5 13:24:47 2005 From: blist at aon.at (Bernhard Fischer) Date: Thu, 5 May 2005 22:24:47 +0200 Subject: [openib-general] HCA Firmware Message-ID: <20050505202447.GY15688@aon.at> Hi, I may have not searched in the proper places, so may miss the obvious "standard" sink of vendors for firmware.. Still, where can i get current, 'blessed' (fwiw) firmware for infiniband HCA's? PS: On hardware manufactured by an unnamed, recently sold vendor, i get: Mellanox Tavor Device Driver is creating device "InfiniHost0" (bus=0a, devfn=00) THH(1): TAVOR_IF_CMD_STAT_BAD_OP - Operation/command not supported or opcode modifier not supported THH(1): infiniband/hw/mellanox-hca/thh/thh_hob.c[3793]: THH_hob_create: CMD_error in THH_cmd_SYS_EN (2) THH(1): infiniband/hw/mellanox-hca/thh/thh_mod_obj.c[292]: Failed creating THH_hob for InfiniHost0 THH(1): THH_init_hh_all_tavor: For all 1 Tavor devices initialization was not successful THH(1): infiniband/hw/mellanox-hca/thh/thh_mod_obj.c[329]: Failed initialization of all available InfiniHost devices where current firmware is: # cat /proc/infiniband/core/ca1/info name: mthca provider: mthca node GUID: 0005:ad00:0002:312c ports: 2 vendor ID: 0x5ad device ID: 0x5a44 HW revision: 0x0 FW revision: 0x300020000 which let's me suspect that that revision needs upgrading, no? Thanks in advance for any hint in this respect. From iod00d at hp.com Thu May 5 13:30:21 2005 From: iod00d at hp.com (Grant Grundler) Date: Thu, 5 May 2005 13:30:21 -0700 Subject: [openib-general] PF 5/4 Telecon Minutes In-Reply-To: <1115320068.4476.1.camel@localhost.localdomain> References: <20050505001734.GD16887@esmail.cup.hp.com> <20050505183244.GB19523@esmail.cup.hp.com> <1115320068.4476.1.camel@localhost.localdomain> Message-ID: <20050505203021.GC19523@esmail.cup.hp.com> On Thu, May 05, 2005 at 03:07:48PM -0400, Hal Rosenstock wrote: > Hi Grant, > > On Thu, 2005-05-05 at 14:32, Grant Grundler wrote: > > The proposed topic is a good one for openib.org "documentation". > > Seems to match the slideset OpenIB-Arch-041221.pdf. It would probably > > be good to update that slideset to match the paper as well. > > As you can well imagine, that slide deck is somewhat out of date. The > question is when it needs to be updated to be in sync. I don't think it > would take very long but want to minimize the number of times I do this. Hi Hal, Understood and I sympathize. I'm just pointing out there is (a) opportunity for woody to leverage and (b) it's of general interest. If you can commit the document source (.mgp?) to subversion, other people can submit/commit patches to so you don't have to. Woody has incentive since he will need a slideset for his OLS BOF. thanks, grant From blist at aon.at Thu May 5 13:29:38 2005 From: blist at aon.at (Bernhard Fischer) Date: Thu, 5 May 2005 22:29:38 +0200 Subject: [openib-general] Re: [openib-commits] r2239 - gen2/trunk/src/linux-kernel/infiniband/include In-Reply-To: <1115324197.4476.3.camel@localhost.localdomain> References: <20050505201626.GX15688@aon.at> <1115324197.4476.3.camel@localhost.localdomain> Message-ID: <20050505202938.GZ15688@aon.at> On Thu, May 05, 2005 at 04:16:37PM -0400, Hal Rosenstock wrote: >On Thu, 2005-05-05 at 16:16, Bernhard Fischer wrote: >> Hal, >> >> - Fix commentary typo: s/intialized/initialized/g >> >> Would you mind also fixing >> src/userspace/libsdp/src/port.c >> >> Please, if you happen to find those, grep the whole tree for occurances. >> >> My current -ENOPARSE list looks like: >> >> conenctions >> esablished >> etablish >> defintions >> defaul\ >> a\ single\ buffers >> retreival >> recieve >> explict >> implict >> responsab >> listning >> conntension >> consistancy >> the\ the >> intia > >Libor is the maintainer for SDP so this is a request to him. Libor is maintaining trunk/src/userspace/management/osm/opensm/osm_state_mgr.c ? Guess i'm looking at an old copy (i've got revision 2265), am i? > >-- Hal > >> Furthermore it would be really, really nice if anyone who touches files >> would trim any whitespace as that is extremely annoying if you happen to >> be forced to look at. Also, whitespace just eats my precious >> (disc-)space for no good. >> >> TIA, From halr at voltaire.com Thu May 5 13:26:17 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 05 May 2005 16:26:17 -0400 Subject: [openib-general] HCA Firmware In-Reply-To: <20050505202447.GY15688@aon.at> References: <20050505202447.GY15688@aon.at> Message-ID: <1115324777.4476.7.camel@localhost.localdomain> Hi Bernhard, On Thu, 2005-05-05 at 16:24, Bernhard Fischer wrote: > Hi, > > I may have not searched in the proper places, so may miss the obvious > "standard" sink of vendors for firmware.. > > Still, where can i get current, 'blessed' (fwiw) firmware for infiniband > HCA's? > > PS: > On hardware manufactured by an unnamed, recently sold vendor, You need to get it from the HCA vendor. > i get: > > Mellanox Tavor Device Driver is creating device "InfiniHost0" (bus=0a, > devfn=00) > > THH(1): TAVOR_IF_CMD_STAT_BAD_OP - Operation/command not supported or > opcode modifier not supported > THH(1): infiniband/hw/mellanox-hca/thh/thh_hob.c[3793]: THH_hob_create: > CMD_error in THH_cmd_SYS_EN (2) > THH(1): infiniband/hw/mellanox-hca/thh/thh_mod_obj.c[292]: Failed > creating THH_hob for InfiniHost0 > THH(1): THH_init_hh_all_tavor: For all 1 Tavor devices initialization > was not successful > THH(1): infiniband/hw/mellanox-hca/thh/thh_mod_obj.c[329]: Failed > initialization of all available InfiniHost devices > > where current firmware is: > # cat /proc/infiniband/core/ca1/info > name: mthca > provider: mthca > node GUID: 0005:ad00:0002:312c > ports: 2 > vendor ID: 0x5ad > device ID: 0x5a44 > HW revision: 0x0 > FW revision: 0x300020000 That looks like 3.2.0. The latest is 3.3.2. I don't recall if there are issues running with 3.2.0 or not. 3.2.0 or later is recommended in the IPoIB FAQ but I'm not sure this is sufficient anymore. > which let's me suspect that that revision needs upgrading, no? > Thanks in advance for any hint in this respect. -- Hal From halr at voltaire.com Thu May 5 13:29:45 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 05 May 2005 16:29:45 -0400 Subject: [openib-general] Re: [openib-commits] r2239 - gen2/trunk/src/linux-kernel/infiniband/include In-Reply-To: <20050505202938.GZ15688@aon.at> References: <20050505201626.GX15688@aon.at> <1115324197.4476.3.camel@localhost.localdomain> <20050505202938.GZ15688@aon.at> Message-ID: <1115324936.4476.10.camel@localhost.localdomain> On Thu, 2005-05-05 at 16:29, Bernhard Fischer wrote: > >> Please, if you happen to find those, grep the whole tree for occurances. > >> > >> My current -ENOPARSE list looks like: > >> > >> conenctions > >> esablished > >> etablish > >> defintions > >> defaul\ > >> a\ single\ buffers > >> retreival > >> recieve > >> explict > >> implict > >> responsab > >> listning > >> conntension > >> consistancy > >> the\ the > >> intia > > > >Libor is the maintainer for SDP so this is a request to him. > > Libor is maintaining > trunk/src/userspace/management/osm/opensm/osm_state_mgr.c ? > > Guess i'm looking at an old copy (i've got revision 2265), am i? OK. Are others in this list in OpenSM as well ? -- Hal From robert.j.woodruff at intel.com Thu May 5 13:38:52 2005 From: robert.j.woodruff at intel.com (Woodruff, Robert J) Date: Thu, 5 May 2005 13:38:52 -0700 Subject: [openib-general] PF 5/4 Telecon Minutes Message-ID: <1AC79F16F5C5284499BB9591B33D6F000450C634@orsmsx408> Grant wrote, >Hi Hal, >Understood and I sympathize. I'm just pointing out there is (a) opportunity >for woody to leverage and (b) it's of general interest. >If you can commit the document source (.mgp?) to subversion, >other people can submit/commit patches to so you don't have to. >Woody has incentive since he will need a slideset for his OLS BOF. >thanks, >grant Hal, don't feel that you need to update these just for me. I am sure that there is already enough information available for Sean and I to pull together something for OLS. Also, there is no rush to get the presentation part done right away. The paper will have to be submitted for printing pretty soon, but I think that we have up until the conference to work on the presentation. Ok coarse, having a good Arch presentation on openib in SVN is a good suggestion, but don't feel that you need to drop everything and work on it right now just for me. woody From halr at voltaire.com Thu May 5 13:33:23 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 05 May 2005 16:33:23 -0400 Subject: [openib-general] Re: [openib-commits] r2239 - gen2/trunk/src/linux-kernel/infiniband/include In-Reply-To: <20050505201626.GX15688@aon.at> References: <20050505201626.GX15688@aon.at> Message-ID: <1115325042.4476.15.camel@localhost.localdomain> On Thu, 2005-05-05 at 16:16, Bernhard Fischer wrote: > Furthermore it would be really, really nice if anyone who touches files > would trim any whitespace as that is extremely annoying if you happen to > be forced to look at. Also, whitespace just eats my precious > (disc-)space for no good. Are OpenSM files guilty of this ? -- Hal From tduffy at sun.com Thu May 5 13:42:07 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 05 May 2005 13:42:07 -0700 Subject: [openib-general] [PATCH] kDAPL: more fixes to kdapltest Makefile Message-ID: <1115325727.11449.2.camel@duffman> This patch renames the KERNEL_ROOT variable to KERNELDIR to be consistent with other out-of-kernel Makefiles. It also removes the -msse flag as this is no longer needed now that float is out of the kernel. Signed-off-by: Tom Duffy Index: linux-kernel/test/dapltest/kdapl/Makefile =================================================================== --- linux-kernel/test/dapltest/kdapl/Makefile (revision 2265) +++ linux-kernel/test/dapltest/kdapl/Makefile (working copy) @@ -37,7 +37,7 @@ ifeq ($(origin KERNELRELEASE), undefined) KDAPLTEST_ROOT = `pwd` # directory that contains this makefile -KERNEL_ROOT ?= /lib/modules/`uname -r`/build # root of kernel sources +KERNELDIR ?= /lib/modules/`uname -r`/build # root of kernel sources # Tools ECHO = /bin/echo @@ -45,7 +45,7 @@ RM = /bin/rm all : @$(ECHO) "--- Compiling kdapltest module ---" - $(MAKE) -C $(KERNEL_ROOT) M=$(KDAPLTEST_ROOT) modules + $(MAKE) -C $(KERNELDIR) M=$(KDAPLTEST_ROOT) modules clean : $(RM) -rf \ @@ -61,11 +61,6 @@ clean : #-------------------------------------------------------------------------- else -IS_i686 = $(shell uname -m) -ifeq (${IS_i686},x86_64) -EXTRA_CFLAGS += -msse -endif - EXTRA_CFLAGS += \ -I$(src)/../include \ -I$(src)/../mdep/linux \ From blist at aon.at Thu May 5 13:42:37 2005 From: blist at aon.at (Bernhard Fischer) Date: Thu, 5 May 2005 22:42:37 +0200 Subject: [openib-general] Re: [openib-commits] r2239 - gen2/trunk/src/linux-kernel/infiniband/include In-Reply-To: <1115324936.4476.10.camel@localhost.localdomain> References: <20050505201626.GX15688@aon.at> <1115324197.4476.3.camel@localhost.localdomain> <20050505202938.GZ15688@aon.at> <1115324936.4476.10.camel@localhost.localdomain> Message-ID: <20050505204237.GA15688@aon.at> On Thu, May 05, 2005 at 04:29:45PM -0400, Hal Rosenstock wrote: >On Thu, 2005-05-05 at 16:29, Bernhard Fischer wrote: > >> >> Please, if you happen to find those, grep the whole tree for occurances. >> >> >> >> My current -ENOPARSE list looks like: >> >> >> >> conenctions >> >> esablished >> >> etablish >> >> defintions >> >> defaul\ >> >> a\ single\ buffers >> >> retreival >> >> recieve >> >> explict >> >> implict >> >> responsab >> >> listning >> >> conntension >> >> consistancy >> >> the\ the >> >> intia >> > >> >Libor is the maintainer for SDP so this is a request to him. >> >> Libor is maintaining >> trunk/src/userspace/management/osm/opensm/osm_state_mgr.c ? >> >> Guess i'm looking at an old copy (i've got revision 2265), am i? > >OK. Are others in this list in OpenSM as well ? Haven't specifically grepped for occurances in osm, no. I'm not encouraged nor payed to take care of those at all ;) Nonetheless thanks for fixing those you're forced to be concerned about. kind regards, From halr at voltaire.com Thu May 5 13:39:49 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 05 May 2005 16:39:49 -0400 Subject: [openib-general] PF 5/4 Telecon Minutes In-Reply-To: <20050505203021.GC19523@esmail.cup.hp.com> References: <20050505001734.GD16887@esmail.cup.hp.com> <20050505183244.GB19523@esmail.cup.hp.com> <1115320068.4476.1.camel@localhost.localdomain> <20050505203021.GC19523@esmail.cup.hp.com> Message-ID: <1115325202.4476.24.camel@localhost.localdomain> On Thu, 2005-05-05 at 16:30, Grant Grundler wrote: > Understood and I sympathize. I'm just pointing out there is (a) opportunity > for woody to leverage and (b) it's of general interest. > > If you can commit the document source (.mgp?) to subversion, It's ppt. Is there some conversion or would I need to do this manually ? > other people can submit/commit patches to so you don't have to. > Woody has incentive since he will need a slideset for his OLS BOF. -- Hal From iod00d at hp.com Thu May 5 14:09:44 2005 From: iod00d at hp.com (Grant Grundler) Date: Thu, 5 May 2005 14:09:44 -0700 Subject: [openib-general] HCA Firmware In-Reply-To: <20050505202447.GY15688@aon.at> References: <20050505202447.GY15688@aon.at> Message-ID: <20050505210944.GE19523@esmail.cup.hp.com> On Thu, May 05, 2005 at 10:24:47PM +0200, Bernhard Fischer wrote: > Hi, > > I may have not searched in the proper places, so may miss the obvious > "standard" sink of vendors for firmware.. There isn't one. I've been griping about this for a while. But none of the vendors wants to even risk the possibility they might be supporting someone else's cards. > Still, where can i get current, 'blessed' (fwiw) firmware for infiniband > HCA's? At your own risk, you can try: http://www.mellanox.com/products/firmware.html > where current firmware is: > # cat /proc/infiniband/core/ca1/info AFAIK, gen2 stack doesn't support /proc . > HW revision: 0x0 > FW revision: 0x300020000 > which let's me suspect that that revision needs upgrading, no? Yes...that's either 3.0.2 or 3.2.0 (I forgot). grundler at gsyprf3:~$ ls -F /sys/class/infiniband/mthca0/ device@ fw_ver hw_rev node_type sys_image_guid driver@ hca_type node_guid ports/ grundler at gsyprf3:~$ cat /sys/class/infiniband/mthca0/* cat: /sys/class/infiniband/mthca0/device: Is a directory cat: /sys/class/infiniband/mthca0/driver: Is a directory 3.3.2 MT23108 a1 0002:c902:0000:03fc 1: CA cat: /sys/class/infiniband/mthca0/ports: Is a directory 0005:ad00:0100:d050 hth, grant From tduffy at sun.com Thu May 5 14:08:09 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 05 May 2005 14:08:09 -0700 Subject: [openib-general] [PATCH] kDAPL: remove DAT_UINT32 and DAT_UINT64 Message-ID: <1115327289.11449.7.camel@duffman> James, let's start some typedef cleanup. I will continue in this fashion if you apply. This patch removes the use of typedefs DAT_UINT32 and DAT_UINT64 and instead uses standard kernel u32 and u64 (except in stuff that is compiled userland where it uses uint32_t and uint64_t). Signed-off-by: Tom Duffy Index: linux-kernel/test/dapltest/test/dapl_limit.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_limit.c (revision 2265) +++ linux-kernel/test/dapltest/test/dapl_limit.c (working copy) @@ -1049,7 +1049,7 @@ limit_test ( DT_Tdep_Print_Head *phead, hdl_sets[i].lmr_buffer; iovp->segment_length = DFLT_BUFFSZ; iovp->lmr_context = hdl_sets[i].lmr_context; - cookie.as_64 = (DAT_UINT64)0UL; + cookie.as_64 = (u64)0UL; cookie.as_ptr = (DAT_PVOID) hdl_sets[i].lmr_buffer; DT_Tdep_PT_Printf (phead, "%s: dat_ep_post_recv #%d\n", module, Index: linux-kernel/test/dapltest/test/dapl_transaction_test.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_transaction_test.c (revision 2265) +++ linux-kernel/test/dapltest/test/dapl_transaction_test.c (working copy) @@ -230,7 +230,7 @@ DT_Transaction_Main (void *param) { Transaction_Test_t *test_ptr = (Transaction_Test_t *) param; DAT_RETURN ret; - DAT_UINT32 i, j; + u32 i, j; bool success = false; Per_Test_Data_t *pt_ptr; Thread *thread; @@ -345,7 +345,7 @@ DT_Transaction_Main (void *param) for (i = 0; i < test_ptr->cmd->eps_per_thread; i++) { DAT_EP_ATTR ep_attr; - DAT_UINT32 buff_size = MAX_OPS * sizeof (RemoteMemoryInfo); + u32 buff_size = MAX_OPS * sizeof (RemoteMemoryInfo); /* * Adjust default EP attributes to fit the requested test. @@ -517,7 +517,7 @@ DT_Transaction_Main (void *param) for (i = 0; i < test_ptr->cmd->eps_per_thread; i++) { - DAT_UINT32 buff_size = MAX_OPS * sizeof (RemoteMemoryInfo); + u32 buff_size = MAX_OPS * sizeof (RemoteMemoryInfo); RemoteMemoryInfo *RemoteMemInfo; DAT_DTO_COMPLETION_EVENT_DATA dto_stat; DAT_CR_ARRIVAL_EVENT_DATA cr_stat; @@ -704,7 +704,7 @@ DT_Transaction_Main (void *param) * Client - connect */ unsigned int retry_cnt = 0; - DAT_UINT32 buff_size = MAX_OPS * sizeof (RemoteMemoryInfo); + u32 buff_size = MAX_OPS * sizeof (RemoteMemoryInfo); DT_Tdep_PT_Debug (1,(phead,"Client[" F64x "]: Connect #%d on port 0x" F64x "\n", test_ptr->base_port, i, Index: linux-kernel/test/dapltest/test/dapl_bpool.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_bpool.c (revision 2265) +++ linux-kernel/test/dapltest/test/dapl_bpool.c (working copy) @@ -226,7 +226,7 @@ DT_BpoolAlloc ( iov.segment_length = bpool_ptr->reg_size; iov.lmr_context = bpool_ptr->lmr_context; - cookie.as_64 = (DAT_UINT64)0UL; + cookie.as_64 = (u64)0UL; cookie.as_ptr = (DAT_PVOID) (uintptr_t) bpool_ptr->reg_addr; mflags = (enable_rdma_write && enable_rdma_read ? DAT_MEM_PRIV_ALL_FLAG : (enable_rdma_write ? DAT_MEM_PRIV_WRITE_FLAG @@ -333,7 +333,7 @@ DT_Bpool_Destroy (Per_Test_Data_t * pt_p iov.virtual_address = bpool_ptr->reg_addr; iov.segment_length = 0; /* un-bind */ iov.lmr_context = bpool_ptr->lmr_context; - cookie.as_64 = (DAT_UINT64)0UL; + cookie.as_64 = (u64)0UL; cookie.as_ptr = (DAT_PVOID) (uintptr_t)bpool_ptr->reg_addr; /* Index: linux-kernel/test/dapltest/test/dapl_test_util.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_test_util.c (revision 2265) +++ linux-kernel/test/dapltest/test/dapl_test_util.c (working copy) @@ -178,7 +178,7 @@ DT_post_recv_buffer (DT_Tdep_Print_Head /* iov->virtual_address = (DAT_VADDR) (uintptr_t) buff; */ iov->segment_length = size; iov->lmr_context = lmr_c; - cookie.as_64 = (DAT_UINT64)0UL; + cookie.as_64 = (u64)0UL; cookie.as_ptr = (DAT_PVOID) buff; DT_Tdep_PT_Debug (3, (phead, "Post-Recv #%d [%p, %x]\n", index, buff, size)); @@ -223,7 +223,7 @@ DT_post_send_buffer (DT_Tdep_Print_Head /* iov->virtual_address = (DAT_VADDR) (uintptr_t) buff; */ iov->segment_length = size; iov->lmr_context = lmr_c; - cookie.as_64 = (DAT_UINT64)0UL; + cookie.as_64 = (u64)0UL; cookie.as_ptr = (DAT_PVOID) buff; DT_Tdep_PT_Debug (3, (phead, "Post-Send #%d [%p, %x]\n", index, buff, size)); Index: linux-kernel/test/dapltest/test/dapl_client.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_client.c (revision 2265) +++ linux-kernel/test/dapltest/test/dapl_client.c (working copy) @@ -38,7 +38,7 @@ int DT_cs_Client (Params_t * params_ptr, char *dapl_name, char *server_name, - DAT_UINT32 total_threads) + u32 total_threads) { Per_Test_Data_t *pt_ptr = NULL; DAT_IA_HANDLE ia_handle = DAT_HANDLE_NULL; Index: linux-kernel/test/dapltest/test/dapl_transaction_util.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_transaction_util.c (revision 2265) +++ linux-kernel/test/dapltest/test/dapl_transaction_util.c (working copy) @@ -55,7 +55,7 @@ DT_handle_post_recv_buf (DT_Tdep_Print_H iov[j].lmr_context = DT_Bpool_GetLMR (op->bp, j); } cookie.as_64 = - ((((DAT_UINT64) i) << 32) + ((((u64) i) << 32) | (((uintptr_t) DT_Bpool_GetBuffer (op->bp, 0)) & 0xffffffffUL)); /* Post the recv */ @@ -118,7 +118,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p iov[j].lmr_context = DT_Bpool_GetLMR (op->bp, j); } cookie.as_64 = - ((((DAT_UINT64) i) << 32) + ((((u64) i) << 32) | (((uintptr_t) DT_Bpool_GetBuffer (op->bp, 0)) & 0xffffffffUL)); /* Post the send */ @@ -183,7 +183,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p op = &ep_context[epnum].op[op_indx]; dto_cookie.as_64 = - ((((DAT_UINT64) epnum) << 32) + ((((u64) epnum) << 32) | (((uintptr_t) DT_Bpool_GetBuffer (op->bp, 0)) & 0xffffffffUL)); if (!DT_dto_check (phead, @@ -293,7 +293,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p op = &ep_context[epnum].op[op_indx]; dto_cookie.as_64 = - ((((DAT_UINT64) epnum) << 32) + ((((u64) epnum) << 32) | (((uintptr_t) DT_Bpool_GetBuffer (op->bp, 0)) & 0xffffffffUL)); if (!DT_dto_check (phead, @@ -368,7 +368,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p op = &ep_context[epnum].op[op_indx - 1]; dto_cookie.as_64 = - ((((DAT_UINT64) epnum) << 32) + ((((u64) epnum) << 32) | (((uintptr_t) DT_Bpool_GetBuffer (op->bp, 0)) & 0xffffffffUL)); @@ -498,7 +498,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p iov[j].lmr_context = DT_Bpool_GetLMR (op->bp, j); } cookie.as_64 = - ((((DAT_UINT64) i) << 32) + ((((u64) i) << 32) | (((uintptr_t) DT_Bpool_GetBuffer (op->bp, 0)) & 0xffffffffUL)); rmr_triplet.pad = 0U; @@ -583,7 +583,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p op = &ep_context[epnum].op[op_indx]; dto_cookie.as_64 = - ((((DAT_UINT64) epnum) << 32) + ((((u64) epnum) << 32) | (((uintptr_t) DT_Bpool_GetBuffer (op->bp, 0)) & 0xffffffffUL)); if (!DT_dto_check (phead, Index: linux-kernel/test/dapltest/include/dapl_client_info.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_client_info.h (revision 2265) +++ linux-kernel/test/dapltest/include/dapl_client_info.h (working copy) @@ -34,10 +34,10 @@ typedef struct { - DAT_UINT32 dapltest_version; - DAT_UINT32 is_little_endian; - DAT_UINT32 test_type; - DAT_UINT32 total_threads; + uint32_t dapltest_version; + uint32_t is_little_endian; + uint32_t test_type; + uint32_t total_threads; } Client_Info_t; #pragma pack () Index: linux-kernel/test/dapltest/include/dapl_performance_test.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_performance_test.h (revision 2265) +++ linux-kernel/test/dapltest/include/dapl_performance_test.h (working copy) @@ -41,11 +41,10 @@ #define DT_PERF_DFLT_EVD_LENGTH 8 -typedef struct -{ +typedef struct { DT_Transfer_Type transfer_type; - DAT_UINT32 num_segs; - DAT_UINT32 seg_size; + uint32_t num_segs; + uint32_t seg_size; Bpool *bp; /* RDMA info */ Index: linux-kernel/test/dapltest/include/dapl_limit_cmd.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_limit_cmd.h (revision 2265) +++ linux-kernel/test/dapltest/include/dapl_limit_cmd.h (working copy) @@ -53,10 +53,10 @@ typedef struct { char device_name[256]; /* -D */ DAT_QOS ReliabilityLevel; /* -R */ - DAT_UINT32 width; /* -w */ - DAT_UINT32 debug; /* -d */ - DAT_UINT32 maximum; /* -m */ - DAT_UINT32 Test_List[ LIM_NUM_TESTS ]; + uint32_t width; /* -w */ + uint32_t debug; /* -d */ + uint32_t maximum; /* -m */ + uint32_t Test_List[ LIM_NUM_TESTS ]; DAT_MEM_TYPE DT_mem_type; /* -M */ } Limit_Cmd_t; Index: linux-kernel/test/dapltest/include/dapl_proto.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_proto.h (revision 2265) +++ linux-kernel/test/dapltest/include/dapl_proto.h (working copy) @@ -112,7 +112,7 @@ int get_ep_connection_state int DT_cs_Client (Params_t * params_ptr, char *dapl_name, char *server_name, - DAT_UINT32 total_threads); + uint32_t total_threads); /* dapl_client_info.c */ void DT_Client_Info_Endian (Client_Info_t * client_info); @@ -143,10 +143,10 @@ void DT_print_transaction_sta /* dapl_endian.c */ void DT_Endian_Init (void); -DAT_UINT32 DT_Endian32 (DAT_UINT32 val); -DAT_UINT64 DT_Endian64 (DAT_UINT64 val); -DAT_UINT32 DT_EndianMemHandle (DAT_UINT32 val); -DAT_UINT64 DT_EndianMemAddress (DAT_UINT64 val); +uint32_t DT_Endian32 (uint32_t val); +uint64_t DT_Endian64 (uint64_t val); +uint32_t DT_EndianMemHandle (uint32_t val); +uint64_t DT_EndianMemAddress (uint64_t val); /* dapl_getopt.c */ void DT_mygetopt_init (mygetopt_t * opts); Index: linux-kernel/test/dapltest/include/dapl_quit_cmd.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_quit_cmd.h (revision 2265) +++ linux-kernel/test/dapltest/include/dapl_quit_cmd.h (working copy) @@ -33,7 +33,7 @@ typedef struct { char server_name[256]; /* -s */ char device_name[256]; /* -D */ - DAT_UINT32 debug; /* -d */ + uint32_t debug; /* -d */ DAT_QOS ReliabilityLevel; /* -R */ DAT_MEM_TYPE DT_mem_type; /* -M */ } Quit_Cmd_t; Index: linux-kernel/test/dapltest/include/dapl_transaction_cmd.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_transaction_cmd.h (revision 2265) +++ linux-kernel/test/dapltest/include/dapl_transaction_cmd.h (working copy) @@ -34,31 +34,29 @@ #define NAME_SZ 256 #pragma pack (2) -typedef struct -{ - DAT_UINT32 server_initiated; - DAT_UINT32 transfer_type; - DAT_UINT32 num_segs; - DAT_UINT32 seg_size; - DAT_UINT32 reap_send_on_recv; +typedef struct { + uint32_t server_initiated; + uint32_t transfer_type; + uint32_t num_segs; + uint32_t seg_size; + uint32_t reap_send_on_recv; } Transaction_Cmd_Op_t; -typedef struct -{ - DAT_UINT32 dapltest_version; - DAT_UINT32 client_is_little_endian; - char server_name[NAME_SZ]; /* -s */ - DAT_UINT32 num_iterations; /* -i */ - DAT_UINT32 num_threads; /* -t */ - DAT_UINT32 eps_per_thread; /* -w */ - DAT_UINT32 use_cno; /* NOT USED - remove and bump version*/ - DAT_UINT32 use_rsp; /* -r */ - DAT_UINT32 debug; /* -d */ - DAT_UINT32 validate; /* -V */ - DAT_UINT32 poll; /* -P */ +typedef struct { + uint32_t dapltest_version; + uint32_t client_is_little_endian; + char server_name[NAME_SZ]; /* -s */ + uint32_t num_iterations; /* -i */ + uint32_t num_threads; /* -t */ + uint32_t eps_per_thread; /* -w */ + uint32_t use_cno; /* NOT USED - remove and bump version*/ + uint32_t use_rsp; /* -r */ + uint32_t debug; /* -d */ + uint32_t validate; /* -V */ + uint32_t poll; /* -P */ char dapl_name[NAME_SZ]; /* -D */ DAT_QOS ReliabilityLevel; - DAT_UINT32 num_ops; + uint32_t num_ops; Transaction_Cmd_Op_t op[MAX_OPS]; DAT_MEM_TYPE DT_mem_type; /* -M */ } Transaction_Cmd_t; Index: linux-kernel/test/dapltest/include/dapl_bpool.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_bpool.h (revision 2265) +++ linux-kernel/test/dapltest/include/dapl_bpool.h (working copy) @@ -38,7 +38,7 @@ typedef struct Bpool_tag Bpool; struct Bpool_tag { unsigned char *alloc_ptr; - DAT_UINT32 alloc_size; + uint32_t alloc_size; DAT_PZ_HANDLE pz_handle; DAT_COUNT seg_size; DAT_COUNT num_segs; /* num segments */ Index: linux-kernel/test/dapltest/include/dapl_server_info.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_server_info.h (revision 2265) +++ linux-kernel/test/dapltest/include/dapl_server_info.h (working copy) @@ -44,11 +44,10 @@ extern Started_server_t *DT_started_serv #define SERVER_PORT_NUMBER ((DAT_CONN_QUAL)0xB0de) -typedef struct -{ - DAT_UINT32 dapltest_version; - DAT_UINT32 is_little_endian; - DAT_UINT32 first_port_number; +typedef struct { + uint32_t dapltest_version; + uint32_t is_little_endian; + uint32_t first_port_number; } Server_Info_t; #pragma pack () Index: linux-kernel/test/dapltest/include/dapl_performance_cmd.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_performance_cmd.h (revision 2265) +++ linux-kernel/test/dapltest/include/dapl_performance_cmd.h (working copy) @@ -39,26 +39,24 @@ typedef enum } Performance_Mode_Type; #pragma pack (2) -typedef struct -{ - DAT_UINT32 transfer_type; - DAT_UINT32 seg_size; - DAT_UINT32 num_segs; +typedef struct { + uint32_t transfer_type; + uint32_t seg_size; + uint32_t num_segs; } Performance_Cmd_Op_t; -typedef struct -{ - DAT_UINT32 dapltest_version; - DAT_UINT32 client_is_little_endian; +typedef struct { + uint32_t dapltest_version; + uint32_t client_is_little_endian; char server_name[NAME_SZ]; /* -s */ char dapl_name[NAME_SZ]; /* -D */ DAT_QOS qos; - DAT_UINT32 debug; /* -d */ + uint32_t debug; /* -d */ Performance_Mode_Type mode; /* -m */ - DAT_UINT32 num_iterations; /* -i */ - DAT_UINT32 pipeline_len; /* -p */ + uint32_t num_iterations; /* -i */ + uint32_t pipeline_len; /* -p */ Performance_Cmd_Op_t op; - DAT_UINT32 use_rsp; /* -r */ + uint32_t use_rsp; /* -r */ DAT_MEM_TYPE DT_mem_type; /* -M */ } Performance_Cmd_t; #pragma pack () Index: linux-kernel/test/dapltest/include/dapl_transaction_test.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_transaction_test.h (revision 2265) +++ linux-kernel/test/dapltest/include/dapl_transaction_test.h (working copy) @@ -37,8 +37,8 @@ typedef struct { DAT_BOOLEAN server_initiated; DT_Transfer_Type transfer_type; - DAT_UINT32 num_segs; - DAT_UINT32 seg_size; + uint32_t num_segs; + uint32_t seg_size; DAT_BOOLEAN reap_send_on_recv; Bpool *bp; Index: linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.c =================================================================== --- linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.c (revision 2265) +++ linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.c (working copy) @@ -175,7 +175,7 @@ DT_Mdep_GetTime (void) { struct tms ts; clock_t t = times (&ts); - return (unsigned long) ((DAT_UINT64) t * 1000 / CLK_TCK); + return (unsigned long) ((uint64_t) t * 1000 / CLK_TCK); } double Index: linux-kernel/test/dapltest/common/dapl_endian.c =================================================================== --- linux-kernel/test/dapltest/common/dapl_endian.c (revision 2265) +++ linux-kernel/test/dapltest/common/dapl_endian.c (working copy) @@ -39,21 +39,21 @@ DT_Endian_Init (void) * Big/Little Endian conversion functions */ -#define c1a32 ((DAT_UINT32)0x00FF00FF) -#define c1b32 ((DAT_UINT32)0xFF00FF00) -#define c2a32 ((DAT_UINT32)0x0000FFFF) -#define c2b32 ((DAT_UINT32)0xFFFF0000) -#define c164 ((DAT_UINT64)0x00FF00FF) +#define c1a32 ((uint32_t)0x00FF00FF) +#define c1b32 ((uint32_t)0xFF00FF00) +#define c2a32 ((uint32_t)0x0000FFFF) +#define c2b32 ((uint32_t)0xFFFF0000) +#define c164 ((uint64_t)0x00FF00FF) #define c1a64 (c164 | (c164 << 32)) #define c1b64 (c1a64 << 8) -#define c264 ((DAT_UINT64)0x0000FFFF) +#define c264 ((uint64_t)0x0000FFFF) #define c2a64 (c264 | (c264 << 32)) #define c2b64 (c2a64 << 16) -#define c3a64 ((DAT_UINT64)0xFFFFFFFF) +#define c3a64 ((uint64_t)0xFFFFFFFF) #define c3b64 (c3a64 << 32) -DAT_UINT32 -DT_Endian32 (DAT_UINT32 val) +uint32_t +DT_Endian32 (uint32_t val) { if (DT_local_is_little_endian) { @@ -64,8 +64,8 @@ DT_Endian32 (DAT_UINT32 val) return (val); } -DAT_UINT64 -DT_Endian64 (DAT_UINT64 val) +uint64_t +DT_Endian64 (uint64_t val) { if (DT_local_is_little_endian) { @@ -77,18 +77,18 @@ DT_Endian64 (DAT_UINT64 val) return (val); } -DAT_UINT32 -DT_EndianMemHandle (DAT_UINT32 val) +uint32_t +DT_EndianMemHandle (uint32_t val) { val = ((val & c1a32) << 8) | ((val & c1b32) >> 8); val = ((val & c2a32) << 16) | ((val & c2b32) >> 16); return (val); } -DAT_UINT64 -DT_EndianMemAddress (DAT_UINT64 val) +uint64_t +DT_EndianMemAddress (uint64_t val) { - DAT_UINT64 val64; + uint64_t val64; val64 = val; val64 = ((val64 & c1a64) << 8) | ((val64 & c1b64) >> 8); val64 = ((val64 & c2a64) << 16) | ((val64 & c2b64) >> 16); Index: linux-kernel/dat-provider/dapl_adapter_util.h =================================================================== --- linux-kernel/dat-provider/dapl_adapter_util.h (revision 2265) +++ linux-kernel/dat-provider/dapl_adapter_util.h (working copy) @@ -79,7 +79,7 @@ DAT_RETURN dapl_ib_connect(DAT_EP_HANDLE DAT_RETURN dapl_ib_disconnect(DAPL_EP * ep_ptr, DAT_CLOSE_FLAGS close_flags); DAT_RETURN dapl_ib_setup_conn_listener(DAPL_IA * ia_ptr, - DAT_UINT64 ServiceID, DAPL_SP * sp_ptr); + u64 ServiceID, DAPL_SP * sp_ptr); DAT_RETURN dapl_ib_remove_conn_listener(DAPL_IA * ia_ptr, DAPL_SP * sp_ptr); Index: linux-kernel/dat-provider/dapl_openib_cm.c =================================================================== --- linux-kernel/dat-provider/dapl_openib_cm.c (revision 2265) +++ linux-kernel/dat-provider/dapl_openib_cm.c (working copy) @@ -598,7 +598,7 @@ void dapl_ib_reinit_ep(DAPL_EP * ep_ptr) */ DAT_RETURN dapl_ib_setup_conn_listener(DAPL_IA * ia_ptr, - DAT_UINT64 ServiceID, DAPL_SP * sp_ptr) + u64 ServiceID, DAPL_SP * sp_ptr) { int status; Index: linux-kernel/dat-provider/dapl_util.c =================================================================== --- linux-kernel/dat-provider/dapl_util.c (revision 2265) +++ linux-kernel/dat-provider/dapl_util.c (working copy) @@ -60,7 +60,7 @@ DAT_RETURN dapl_os_get_time(DAPL_OS_TIME struct timeval tv; do_gettimeofday(&tv); - *loc = ((DAT_UINT64) (tv.tv_sec) * 1000000L) + (DAT_UINT64) tv.tv_usec; + *loc = ((u64) (tv.tv_sec) * 1000000L) + (u64) tv.tv_usec; return DAT_SUCCESS; } Index: linux-kernel/dat-provider/dapl_util.h =================================================================== --- linux-kernel/dat-provider/dapl_util.h (revision 2265) +++ linux-kernel/dat-provider/dapl_util.h (working copy) @@ -188,7 +188,7 @@ static __inline__ char *dapl_os_strdup(c * Timer Functions */ -typedef DAT_UINT64 DAPL_OS_TIMEVAL; +typedef u64 DAPL_OS_TIMEVAL; typedef struct dapl_timer_entry DAPL_OS_TIMER; typedef unsigned long long int DAPL_OS_TICKS; Index: linux-kernel/dat-provider/dapl.h =================================================================== --- linux-kernel/dat-provider/dapl.h (revision 2265) +++ linux-kernel/dat-provider/dapl.h (working copy) @@ -156,7 +156,7 @@ typedef struct dapl_cookie_buffer DAPL_C typedef struct dapl_hash_table DAPL_HASH_TABLE; typedef struct dapl_hash_table *DAPL_HASH_TABLEP; -typedef DAT_UINT64 DAPL_HASH_KEY; +typedef u64 DAPL_HASH_KEY; typedef void *DAPL_HASH_DATA; typedef struct dapl_hca DAPL_HCA; Index: linux-kernel/dat/consumer.c =================================================================== --- linux-kernel/dat/consumer.c (revision 2265) +++ linux-kernel/dat/consumer.c (working copy) @@ -579,7 +579,7 @@ dat_ia_openv(const DAT_NAME_PTR name, DAT_COUNT async_event_qlen, DAT_EVD_HANDLE * async_event_handle, DAT_IA_HANDLE * ia_handle, - DAT_UINT32 dat_major, DAT_UINT32 dat_minor, + u32 dat_major, u32 dat_minor, DAT_BOOLEAN thread_safety) { DAT_IA_OPEN_FUNC ia_open_func; Index: linux-kernel/dat/dat.h =================================================================== --- linux-kernel/dat/dat.h (revision 2265) +++ linux-kernel/dat/dat.h (working copy) @@ -37,18 +37,12 @@ #define DAT_VERSION_MINOR 2 #define DAT_THREADSAFE DAT_FALSE -/* unsigned host order, 32 bits */ -typedef u_int32_t DAT_UINT32; - -/* unsigned host order, 64 bits */ -typedef u_int64_t DAT_UINT64; - /* unsigned longest native to compiler */ typedef unsigned long long DAT_UVERYLONG; typedef void * DAT_PVOID; typedef int DAT_COUNT; -typedef DAT_UINT64 DAT_PADDR; +typedef u64 DAT_PADDR; typedef struct sockaddr DAT_SOCK_ADDR; typedef struct sockaddr_in6 DAT_SOCK_ADDR6; @@ -65,7 +59,7 @@ typedef struct sockaddr_in6 DAT_SOCK_ADD * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ */ -typedef DAT_UINT32 DAT_RETURN; +typedef u32 DAT_RETURN; typedef enum dat_return_class { DAT_CLASS_ERROR = 0x80000000, @@ -341,14 +335,14 @@ typedef enum dat_return_subtype { * Determining the success of an operation is best done with a macro; * each of these returns a boolean value. */ -#define DAT_IS_ERROR(status) ((DAT_UINT32)(status) & DAT_CLASS_ERROR) -#define DAT_IS_WARNING(status) ((DAT_UINT32)(status) & DAT_CLASS_WARNING) +#define DAT_IS_ERROR(status) ((u32)(status) & DAT_CLASS_ERROR) +#define DAT_IS_WARNING(status) ((u32)(status) & DAT_CLASS_WARNING) #define DAT_ERROR(Type, SubType) ((DAT_RETURN)(DAT_CLASS_ERROR | Type | SubType)) #define DAT_WARNING(Type, SubType) ((DAT_RETURN)(DAT_CLASS_WARNING | Type | SubType)) -#define DAT_GET_TYPE(status) ((DAT_UINT32)(status) & DAT_TYPE_MASK) -#define DAT_GET_SUBTYPE(status) ((DAT_UINT32)(status) & DAT_SUBTYPE_MASK) +#define DAT_GET_TYPE(status) ((u32)(status) & DAT_TYPE_MASK) +#define DAT_GET_SUBTYPE(status) ((u32)(status) & DAT_SUBTYPE_MASK) /* dat handle types */ typedef enum dat_handle_type { @@ -379,7 +373,7 @@ typedef enum dat_boolean { typedef union dat_context { DAT_PVOID as_ptr; - DAT_UINT64 as_64; + u64 as_64; DAT_UVERYLONG as_index; } DAT_CONTEXT; @@ -404,7 +398,7 @@ typedef enum dat_completion_flags { DAT_COMPLETION_EVD_THRESHOLD_FLAG = 0x10 } DAT_COMPLETION_FLAGS; -typedef DAT_UINT32 DAT_TIMEOUT; /* microseconds */ +typedef u32 DAT_TIMEOUT; /* microseconds */ /* timeout = infinity */ #define DAT_TIMEOUT_INFINITE ((DAT_TIMEOUT) ~0) @@ -426,8 +420,8 @@ typedef DAT_HANDLE DAT_SRQ_HANDLE; typedef DAT_SOCK_ADDR *DAT_IA_ADDRESS_PTR; -typedef DAT_UINT64 DAT_CONN_QUAL; -typedef DAT_UINT64 DAT_PORT_QUAL; +typedef u64 DAT_CONN_QUAL; +typedef u64 DAT_PORT_QUAL; /* QOS definitions */ typedef enum dat_qos { @@ -518,11 +512,11 @@ typedef enum dat_psp_flags { * For the best performance, Consumers should align each buffer segment to * the boundary specified by the dat_optimal_alignment. */ -typedef DAT_UINT32 DAT_LMR_CONTEXT; -typedef DAT_UINT32 DAT_RMR_CONTEXT; +typedef u32 DAT_LMR_CONTEXT; +typedef u32 DAT_RMR_CONTEXT; -typedef DAT_UINT64 DAT_VLEN; -typedef DAT_UINT64 DAT_VADDR; +typedef u64 DAT_VLEN; +typedef u64 DAT_VADDR; /* It is legal for the Consumer to specify zero for segment_length * of the dat_lmr_triplet. When 0 is specified for the @@ -532,14 +526,14 @@ typedef DAT_UINT64 DAT_VADDR; typedef struct dat_lmr_triplet { DAT_LMR_CONTEXT lmr_context; - DAT_UINT32 pad; /* align structure on 64-bit boundry */ + u32 pad; /* align structure on 64-bit boundry */ DAT_VADDR virtual_address; DAT_VLEN segment_length; } DAT_LMR_TRIPLET; typedef struct dat_rmr_triplet { DAT_RMR_CONTEXT rmr_context; - DAT_UINT32 pad; /* align structure on 64-bit boundry */ + u32 pad; /* align structure on 64-bit boundry */ DAT_VADDR target_address; DAT_VLEN segment_length; } DAT_RMR_TRIPLET; @@ -955,8 +949,8 @@ typedef struct dat_upcall_object { typedef struct dat_provider_info { char ia_name[DAT_NAME_MAX_LENGTH]; - DAT_UINT32 dat_version_major; - DAT_UINT32 dat_version_minor; + u32 dat_version_major; + u32 dat_version_minor; DAT_BOOLEAN is_thread_safe; } DAT_PROVIDER_INFO; @@ -1013,10 +1007,10 @@ typedef struct dat_lmr_param { typedef struct dat_ia_attr { char adapter_name[DAT_NAME_MAX_LENGTH]; char vendor_name[DAT_NAME_MAX_LENGTH]; - DAT_UINT32 hardware_version_major; - DAT_UINT32 hardware_version_minor; - DAT_UINT32 firmware_version_major; - DAT_UINT32 firmware_version_minor; + u32 hardware_version_major; + u32 hardware_version_minor; + u32 firmware_version_major; + u32 firmware_version_minor; DAT_IA_ADDRESS_PTR ia_address_ptr; DAT_COUNT max_eps; DAT_COUNT max_dto_per_ep; @@ -1078,10 +1072,10 @@ typedef enum dat_ep_creator_for_psp { typedef struct dat_provider_attr { char provider_name[DAT_NAME_MAX_LENGTH]; - DAT_UINT32 provider_version_major; - DAT_UINT32 provider_version_minor; - DAT_UINT32 dat_version_major; - DAT_UINT32 dat_version_minor; + u32 provider_version_major; + u32 provider_version_minor; + u32 dat_version_major; + u32 dat_version_minor; DAT_MEM_TYPE lmr_mem_types_supported; DAT_IOV_OWNERSHIP iov_ownership_on_return; DAT_QOS dat_qos_supported; @@ -1091,7 +1085,7 @@ typedef struct dat_provider_attr { DAT_BOOLEAN supports_multipath; DAT_EP_CREATOR_FOR_PSP ep_creator; DAT_UPCALL_POLICY upcall_policy; - DAT_UINT32 optimal_buffer_alignment; + u32 optimal_buffer_alignment; const DAT_BOOLEAN evd_stream_merging_supported[6][6]; DAT_BOOLEAN srq_supported; DAT_COUNT srq_watermarks_supported; @@ -1117,7 +1111,7 @@ typedef DAT_RETURN (*DAT_IA_OPEN_FUNC) ( typedef DAT_RETURN (*DAT_IA_OPENV_FUNC) ( const DAT_NAME_PTR, DAT_COUNT, DAT_EVD_HANDLE *, DAT_IA_HANDLE *, - DAT_UINT32, DAT_UINT32, DAT_BOOLEAN); + u32, u32, DAT_BOOLEAN); typedef DAT_RETURN (*DAT_IA_CLOSE_FUNC) (DAT_IA_HANDLE, DAT_CLOSE_FLAGS); @@ -1380,7 +1374,7 @@ extern DAT_RETURN dat_registry_remove_pr */ extern DAT_RETURN dat_ia_openv(const DAT_NAME_PTR, DAT_COUNT, DAT_EVD_HANDLE *, DAT_IA_HANDLE *, - DAT_UINT32, DAT_UINT32, DAT_BOOLEAN); + u32, u32, DAT_BOOLEAN); #define dat_ia_open(name, qlen, async_evd, ia) \ dat_ia_openv((name), (qlen), (async_evd), (ia), \ From halr at voltaire.com Thu May 5 14:04:14 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 05 May 2005 17:04:14 -0400 Subject: [openib-general] [PATCH] [MAD] changes to ib_create_send_mad In-Reply-To: <1115319121.4695.8.camel@localhost.localdomain> References: <1115253041.4476.147.camel@localhost.localdomain> <1115300959.4476.189.camel@localhost.localdomain> <427A64A4.60007@ichips.intel.com> <427A6A68.8030402@ichips.intel.com> <1115319121.4695.8.camel@localhost.localdomain> Message-ID: <1115327054.4476.32.camel@localhost.localdomain> On Thu, 2005-05-05 at 14:53, Hal Rosenstock wrote: > > The payload values in both cases are correct. > > Let me look again. I don't think I was hallucinating... I wasn't hallucinating. Just stupid code. I stepped on this field without noticing. Sorry for pointing in the wrong direction. But one point which perhaps can be improved: With rmpp_active, ib_create_send_mad formats certain MAD fields. This is before some copy occurs. So somehow the two need "merging" for the modified fields. I think it makes a straight copy impossible. So a separate call could do this after the copy or the header can be saved, the copy made, and then the header merged. Those are the 2 approaches I have come up with. Not sure I like either of them. I'm open to better ideas :-) -- Hal From iod00d at hp.com Thu May 5 14:19:13 2005 From: iod00d at hp.com (Grant Grundler) Date: Thu, 5 May 2005 14:19:13 -0700 Subject: [openib-general] HCA Firmware In-Reply-To: <1115324777.4476.7.camel@localhost.localdomain> References: <20050505202447.GY15688@aon.at> <1115324777.4476.7.camel@localhost.localdomain> Message-ID: <20050505211913.GF19523@esmail.cup.hp.com> On Thu, May 05, 2005 at 04:26:17PM -0400, Hal Rosenstock wrote: > That looks like 3.2.0. The latest is 3.3.2. I don't recall if there > are issues running with 3.2.0 or not. 3.2.0 or later is recommended > in the IPoIB FAQ but I'm not sure this is sufficient anymore. MSI/MSI-X support requires at least 3.3.2. ISTR something about FMRs too but are probably mangling that. grant From halr at voltaire.com Thu May 5 14:12:57 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 05 May 2005 17:12:57 -0400 Subject: [openib-general] [PATCH] [MAD] changes to ib_create_send_mad In-Reply-To: <1115327054.4476.32.camel@localhost.localdomain> References: <1115253041.4476.147.camel@localhost.localdomain> <1115300959.4476.189.camel@localhost.localdomain> <427A64A4.60007@ichips.intel.com> <427A6A68.8030402@ichips.intel.com> <1115319121.4695.8.camel@localhost.localdomain> <1115327054.4476.32.camel@localhost.localdomain> Message-ID: <1115327546.4476.36.camel@localhost.localdomain> On Thu, 2005-05-05 at 17:04, Hal Rosenstock wrote: > On Thu, 2005-05-05 at 14:53, Hal Rosenstock wrote: > > > The payload values in both cases are correct. > > > > Let me look again. I don't think I was hallucinating... > > I wasn't hallucinating. Just stupid code. I stepped on this field > without noticing. Sorry for pointing in the wrong direction. > > But one point which perhaps can be improved: > > With rmpp_active, ib_create_send_mad formats certain MAD fields. This is > before some copy occurs. So somehow the two need "merging" for the > modified fields. I think it makes a straight copy impossible. > > So a separate call could do this after the copy or the header can be > saved, the copy made, and then the header merged. Those are the 2 > approaches I have come up with. Not sure I like either of them. I'm open > to better ideas :-) Never mind. I didn't describe this fully and I think I have a reasonable solution. Now back to your regularly scheduled program :-) -- Hal From itamar at mellanox.co.il Thu May 5 14:42:14 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Fri, 6 May 2005 00:42:14 +0300 Subject: [openib-general] [PATCH] kDAPL: remove DAT_UINT32 and DAT_UIN T64 Message-ID: <91DB792C7985D411BEC300B40080D29CC35A4E@mtvex01.mtv.mtl.com> do you still see problems with dat_lmr_create() ? i am using x86 and dont see any problems can you send me the /var/log/messages with full debug enable modprobe ib_dat_provider g_dapl_dbg_type=0xffff Itamar > -----Original Message----- > From: Tom Duffy [mailto:tduffy at sun.com] > Sent: Friday, May 06, 2005 12:08 AM > To: openib-general at openib.org > Subject: [openib-general] [PATCH] kDAPL: remove DAT_UINT32 and > DAT_UINT64 > > > James, let's start some typedef cleanup. I will continue in this > fashion if you apply. > > This patch removes the use of typedefs DAT_UINT32 and DAT_UINT64 and > instead uses standard kernel u32 and u64 (except in stuff that is > compiled userland where it uses uint32_t and uint64_t). > > Signed-off-by: Tom Duffy > > Index: linux-kernel/test/dapltest/test/dapl_limit.c > =================================================================== > --- linux-kernel/test/dapltest/test/dapl_limit.c (revision 2265) > +++ linux-kernel/test/dapltest/test/dapl_limit.c (working copy) > @@ -1049,7 +1049,7 @@ limit_test ( DT_Tdep_Print_Head *phead, > > hdl_sets[i].lmr_buffer; > iovp->segment_length = DFLT_BUFFSZ; > iovp->lmr_context = hdl_sets[i].lmr_context; > - cookie.as_64 = (DAT_UINT64)0UL; > + cookie.as_64 = (u64)0UL; > cookie.as_ptr = (DAT_PVOID) > hdl_sets[i].lmr_buffer; > > DT_Tdep_PT_Printf (phead, "%s: > dat_ep_post_recv #%d\n", module, > Index: linux-kernel/test/dapltest/test/dapl_transaction_test.c > =================================================================== > --- linux-kernel/test/dapltest/test/dapl_transaction_test.c > (revision 2265) > +++ linux-kernel/test/dapltest/test/dapl_transaction_test.c > (working copy) > @@ -230,7 +230,7 @@ DT_Transaction_Main (void *param) > { > Transaction_Test_t *test_ptr = (Transaction_Test_t *) param; > DAT_RETURN ret; > - DAT_UINT32 i, j; > + u32 i, j; > bool success = false; > Per_Test_Data_t *pt_ptr; > Thread *thread; > @@ -345,7 +345,7 @@ DT_Transaction_Main (void *param) > for (i = 0; i < test_ptr->cmd->eps_per_thread; i++) > { > DAT_EP_ATTR ep_attr; > - DAT_UINT32 buff_size = MAX_OPS * sizeof > (RemoteMemoryInfo); > + u32 buff_size = MAX_OPS * sizeof (RemoteMemoryInfo); > > /* > * Adjust default EP attributes to fit the requested test. > @@ -517,7 +517,7 @@ DT_Transaction_Main (void *param) > > for (i = 0; i < test_ptr->cmd->eps_per_thread; i++) > { > - DAT_UINT32 buff_size = MAX_OPS * sizeof > (RemoteMemoryInfo); > + u32 buff_size = MAX_OPS * sizeof (RemoteMemoryInfo); > RemoteMemoryInfo *RemoteMemInfo; > DAT_DTO_COMPLETION_EVENT_DATA dto_stat; > DAT_CR_ARRIVAL_EVENT_DATA cr_stat; > @@ -704,7 +704,7 @@ DT_Transaction_Main (void *param) > * Client - connect > */ > unsigned int retry_cnt = 0; > - DAT_UINT32 buff_size = MAX_OPS * sizeof > (RemoteMemoryInfo); > + u32 buff_size = MAX_OPS * sizeof (RemoteMemoryInfo); > > DT_Tdep_PT_Debug (1,(phead,"Client[" F64x "]: > Connect #%d on port 0x" F64x "\n", > test_ptr->base_port, i, > Index: linux-kernel/test/dapltest/test/dapl_bpool.c > =================================================================== > --- linux-kernel/test/dapltest/test/dapl_bpool.c (revision 2265) > +++ linux-kernel/test/dapltest/test/dapl_bpool.c (working copy) > @@ -226,7 +226,7 @@ DT_BpoolAlloc ( > iov.segment_length = bpool_ptr->reg_size; > iov.lmr_context = bpool_ptr->lmr_context; > > - cookie.as_64 = (DAT_UINT64)0UL; > + cookie.as_64 = (u64)0UL; > cookie.as_ptr = (DAT_PVOID) (uintptr_t) > bpool_ptr->reg_addr; > mflags = (enable_rdma_write && enable_rdma_read ? > DAT_MEM_PRIV_ALL_FLAG > : (enable_rdma_write ? DAT_MEM_PRIV_WRITE_FLAG > @@ -333,7 +333,7 @@ DT_Bpool_Destroy (Per_Test_Data_t * pt_p > iov.virtual_address = bpool_ptr->reg_addr; > iov.segment_length = 0; /* un-bind */ > iov.lmr_context = bpool_ptr->lmr_context; > - cookie.as_64 = (DAT_UINT64)0UL; > + cookie.as_64 = (u64)0UL; > cookie.as_ptr = (DAT_PVOID) > (uintptr_t)bpool_ptr->reg_addr; > > /* > Index: linux-kernel/test/dapltest/test/dapl_test_util.c > =================================================================== > --- linux-kernel/test/dapltest/test/dapl_test_util.c (revision 2265) > +++ linux-kernel/test/dapltest/test/dapl_test_util.c (working copy) > @@ -178,7 +178,7 @@ DT_post_recv_buffer (DT_Tdep_Print_Head > /* iov->virtual_address = (DAT_VADDR) (uintptr_t) buff; */ > iov->segment_length = size; > iov->lmr_context = lmr_c; > - cookie.as_64 = (DAT_UINT64)0UL; > + cookie.as_64 = (u64)0UL; > cookie.as_ptr = (DAT_PVOID) buff; > > DT_Tdep_PT_Debug (3, (phead, "Post-Recv #%d [%p, %x]\n", > index, buff, size)); > @@ -223,7 +223,7 @@ DT_post_send_buffer (DT_Tdep_Print_Head > /* iov->virtual_address = (DAT_VADDR) (uintptr_t) buff; */ > iov->segment_length = size; > iov->lmr_context = lmr_c; > - cookie.as_64 = (DAT_UINT64)0UL; > + cookie.as_64 = (u64)0UL; > cookie.as_ptr = (DAT_PVOID) buff; > > DT_Tdep_PT_Debug (3, (phead, "Post-Send #%d [%p, %x]\n", > index, buff, size)); > Index: linux-kernel/test/dapltest/test/dapl_client.c > =================================================================== > --- linux-kernel/test/dapltest/test/dapl_client.c (revision 2265) > +++ linux-kernel/test/dapltest/test/dapl_client.c (working copy) > @@ -38,7 +38,7 @@ int > DT_cs_Client (Params_t * params_ptr, > char *dapl_name, > char *server_name, > - DAT_UINT32 total_threads) > + u32 total_threads) > { > Per_Test_Data_t *pt_ptr = NULL; > DAT_IA_HANDLE ia_handle = DAT_HANDLE_NULL; > Index: linux-kernel/test/dapltest/test/dapl_transaction_util.c > =================================================================== > --- linux-kernel/test/dapltest/test/dapl_transaction_util.c > (revision 2265) > +++ linux-kernel/test/dapltest/test/dapl_transaction_util.c > (working copy) > @@ -55,7 +55,7 @@ DT_handle_post_recv_buf (DT_Tdep_Print_H > iov[j].lmr_context = DT_Bpool_GetLMR (op->bp, j); > } > cookie.as_64 = > - ((((DAT_UINT64) i) << 32) > + ((((u64) i) << 32) > | (((uintptr_t) DT_Bpool_GetBuffer (op->bp, 0)) & > 0xffffffffUL)); > > /* Post the recv */ > @@ -118,7 +118,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p > iov[j].lmr_context = DT_Bpool_GetLMR (op->bp, j); > } > cookie.as_64 = > - ((((DAT_UINT64) i) << 32) > + ((((u64) i) << 32) > | (((uintptr_t) DT_Bpool_GetBuffer (op->bp, 0)) & > 0xffffffffUL)); > > /* Post the send */ > @@ -183,7 +183,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p > op = &ep_context[epnum].op[op_indx]; > > dto_cookie.as_64 = > - ((((DAT_UINT64) epnum) << 32) > + ((((u64) epnum) << 32) > | (((uintptr_t) DT_Bpool_GetBuffer (op->bp, 0)) & > 0xffffffffUL)); > > if (!DT_dto_check (phead, > @@ -293,7 +293,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p > > op = &ep_context[epnum].op[op_indx]; > dto_cookie.as_64 = > - ((((DAT_UINT64) epnum) << 32) > + ((((u64) epnum) << 32) > | (((uintptr_t) DT_Bpool_GetBuffer (op->bp, 0)) & > 0xffffffffUL)); > > if (!DT_dto_check (phead, > @@ -368,7 +368,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p > op = &ep_context[epnum].op[op_indx - 1]; > > dto_cookie.as_64 = > - ((((DAT_UINT64) epnum) << 32) > + ((((u64) epnum) << 32) > | (((uintptr_t) DT_Bpool_GetBuffer (op->bp, 0)) > & 0xffffffffUL)); > > @@ -498,7 +498,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p > iov[j].lmr_context = DT_Bpool_GetLMR (op->bp, j); > } > cookie.as_64 = > - ((((DAT_UINT64) i) << 32) > + ((((u64) i) << 32) > | (((uintptr_t) DT_Bpool_GetBuffer (op->bp, 0)) & > 0xffffffffUL)); > > rmr_triplet.pad = 0U; > @@ -583,7 +583,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p > op = &ep_context[epnum].op[op_indx]; > > dto_cookie.as_64 = > - ((((DAT_UINT64) epnum) << 32) > + ((((u64) epnum) << 32) > | (((uintptr_t) DT_Bpool_GetBuffer (op->bp, 0)) & > 0xffffffffUL)); > > if (!DT_dto_check (phead, > Index: linux-kernel/test/dapltest/include/dapl_client_info.h > =================================================================== > --- linux-kernel/test/dapltest/include/dapl_client_info.h > (revision 2265) > +++ linux-kernel/test/dapltest/include/dapl_client_info.h > (working copy) > @@ -34,10 +34,10 @@ > > typedef struct > { > - DAT_UINT32 dapltest_version; > - DAT_UINT32 is_little_endian; > - DAT_UINT32 test_type; > - DAT_UINT32 total_threads; > + uint32_t dapltest_version; > + uint32_t is_little_endian; > + uint32_t test_type; > + uint32_t total_threads; > } Client_Info_t; > #pragma pack () > > Index: linux-kernel/test/dapltest/include/dapl_performance_test.h > =================================================================== > --- > linux-kernel/test/dapltest/include/dapl_performance_test.h > (revision 2265) > +++ > linux-kernel/test/dapltest/include/dapl_performance_test.h > (working copy) > @@ -41,11 +41,10 @@ > #define DT_PERF_DFLT_EVD_LENGTH 8 > > > -typedef struct > -{ > +typedef struct { > DT_Transfer_Type transfer_type; > - DAT_UINT32 num_segs; > - DAT_UINT32 seg_size; > + uint32_t num_segs; > + uint32_t seg_size; > Bpool *bp; > > /* RDMA info */ > Index: linux-kernel/test/dapltest/include/dapl_limit_cmd.h > =================================================================== > --- linux-kernel/test/dapltest/include/dapl_limit_cmd.h > (revision 2265) > +++ linux-kernel/test/dapltest/include/dapl_limit_cmd.h > (working copy) > @@ -53,10 +53,10 @@ typedef struct > { > char device_name[256]; /* -D */ > DAT_QOS ReliabilityLevel; /* -R */ > - DAT_UINT32 width; /* -w */ > - DAT_UINT32 debug; /* -d */ > - DAT_UINT32 maximum; /* -m */ > - DAT_UINT32 Test_List[ LIM_NUM_TESTS ]; > + uint32_t width; /* -w */ > + uint32_t debug; /* -d */ > + uint32_t maximum; /* -m */ > + uint32_t Test_List[ LIM_NUM_TESTS ]; > DAT_MEM_TYPE DT_mem_type; /* -M */ > } Limit_Cmd_t; > > Index: linux-kernel/test/dapltest/include/dapl_proto.h > =================================================================== > --- linux-kernel/test/dapltest/include/dapl_proto.h (revision 2265) > +++ linux-kernel/test/dapltest/include/dapl_proto.h (working copy) > @@ -112,7 +112,7 @@ int get_ep_connection_state > int DT_cs_Client (Params_t * params_ptr, > char *dapl_name, > char *server_name, > - DAT_UINT32 total_threads); > + uint32_t total_threads); > > /* dapl_client_info.c */ > void DT_Client_Info_Endian (Client_Info_t * client_info); > @@ -143,10 +143,10 @@ void DT_print_transaction_sta > > /* dapl_endian.c */ > void DT_Endian_Init (void); > -DAT_UINT32 DT_Endian32 (DAT_UINT32 val); > -DAT_UINT64 DT_Endian64 (DAT_UINT64 val); > -DAT_UINT32 DT_EndianMemHandle (DAT_UINT32 val); > -DAT_UINT64 DT_EndianMemAddress (DAT_UINT64 val); > +uint32_t DT_Endian32 (uint32_t val); > +uint64_t DT_Endian64 (uint64_t val); > +uint32_t DT_EndianMemHandle (uint32_t val); > +uint64_t DT_EndianMemAddress (uint64_t val); > > /* dapl_getopt.c */ > void DT_mygetopt_init (mygetopt_t * opts); > Index: linux-kernel/test/dapltest/include/dapl_quit_cmd.h > =================================================================== > --- linux-kernel/test/dapltest/include/dapl_quit_cmd.h > (revision 2265) > +++ linux-kernel/test/dapltest/include/dapl_quit_cmd.h > (working copy) > @@ -33,7 +33,7 @@ typedef struct > { > char server_name[256]; /* -s */ > char device_name[256]; /* -D */ > - DAT_UINT32 debug; /* -d */ > + uint32_t debug; /* -d */ > DAT_QOS ReliabilityLevel; /* -R */ > DAT_MEM_TYPE DT_mem_type; /* -M */ > } Quit_Cmd_t; > Index: linux-kernel/test/dapltest/include/dapl_transaction_cmd.h > =================================================================== > --- linux-kernel/test/dapltest/include/dapl_transaction_cmd.h > (revision 2265) > +++ linux-kernel/test/dapltest/include/dapl_transaction_cmd.h > (working copy) > @@ -34,31 +34,29 @@ > #define NAME_SZ 256 > > #pragma pack (2) > -typedef struct > -{ > - DAT_UINT32 server_initiated; > - DAT_UINT32 transfer_type; > - DAT_UINT32 num_segs; > - DAT_UINT32 seg_size; > - DAT_UINT32 reap_send_on_recv; > +typedef struct { > + uint32_t server_initiated; > + uint32_t transfer_type; > + uint32_t num_segs; > + uint32_t seg_size; > + uint32_t reap_send_on_recv; > } Transaction_Cmd_Op_t; > > -typedef struct > -{ > - DAT_UINT32 dapltest_version; > - DAT_UINT32 client_is_little_endian; > - char server_name[NAME_SZ]; /* -s */ > - DAT_UINT32 num_iterations; /* -i */ > - DAT_UINT32 num_threads; /* -t */ > - DAT_UINT32 eps_per_thread; /* -w */ > - DAT_UINT32 use_cno; /* NOT USED - remove > and bump version*/ > - DAT_UINT32 use_rsp; /* -r */ > - DAT_UINT32 debug; /* -d */ > - DAT_UINT32 validate; /* -V */ > - DAT_UINT32 poll; /* -P */ > +typedef struct { > + uint32_t dapltest_version; > + uint32_t client_is_little_endian; > + char server_name[NAME_SZ]; /* -s */ > + uint32_t num_iterations; /* -i */ > + uint32_t num_threads; /* -t */ > + uint32_t eps_per_thread; /* -w */ > + uint32_t use_cno; /* NOT USED - remove > and bump version*/ > + uint32_t use_rsp; /* -r */ > + uint32_t debug; /* -d */ > + uint32_t validate; /* -V */ > + uint32_t poll; /* -P */ > char dapl_name[NAME_SZ]; /* -D */ > DAT_QOS ReliabilityLevel; > - DAT_UINT32 num_ops; > + uint32_t num_ops; > Transaction_Cmd_Op_t op[MAX_OPS]; > DAT_MEM_TYPE DT_mem_type; /* -M */ > } Transaction_Cmd_t; > Index: linux-kernel/test/dapltest/include/dapl_bpool.h > =================================================================== > --- linux-kernel/test/dapltest/include/dapl_bpool.h (revision 2265) > +++ linux-kernel/test/dapltest/include/dapl_bpool.h (working copy) > @@ -38,7 +38,7 @@ typedef struct Bpool_tag Bpool; > struct Bpool_tag > { > unsigned char *alloc_ptr; > - DAT_UINT32 alloc_size; > + uint32_t alloc_size; > DAT_PZ_HANDLE pz_handle; > DAT_COUNT seg_size; > DAT_COUNT num_segs; /* num segments */ > Index: linux-kernel/test/dapltest/include/dapl_server_info.h > =================================================================== > --- linux-kernel/test/dapltest/include/dapl_server_info.h > (revision 2265) > +++ linux-kernel/test/dapltest/include/dapl_server_info.h > (working copy) > @@ -44,11 +44,10 @@ extern Started_server_t *DT_started_serv > > #define SERVER_PORT_NUMBER ((DAT_CONN_QUAL)0xB0de) > > -typedef struct > -{ > - DAT_UINT32 dapltest_version; > - DAT_UINT32 is_little_endian; > - DAT_UINT32 first_port_number; > +typedef struct { > + uint32_t dapltest_version; > + uint32_t is_little_endian; > + uint32_t first_port_number; > } Server_Info_t; > #pragma pack () > > Index: linux-kernel/test/dapltest/include/dapl_performance_cmd.h > =================================================================== > --- linux-kernel/test/dapltest/include/dapl_performance_cmd.h > (revision 2265) > +++ linux-kernel/test/dapltest/include/dapl_performance_cmd.h > (working copy) > @@ -39,26 +39,24 @@ typedef enum > } Performance_Mode_Type; > > #pragma pack (2) > -typedef struct > -{ > - DAT_UINT32 transfer_type; > - DAT_UINT32 seg_size; > - DAT_UINT32 num_segs; > +typedef struct { > + uint32_t transfer_type; > + uint32_t seg_size; > + uint32_t num_segs; > } Performance_Cmd_Op_t; > > -typedef struct > -{ > - DAT_UINT32 dapltest_version; > - DAT_UINT32 client_is_little_endian; > +typedef struct { > + uint32_t dapltest_version; > + uint32_t client_is_little_endian; > char server_name[NAME_SZ]; /* -s */ > char dapl_name[NAME_SZ]; /* -D */ > DAT_QOS qos; > - DAT_UINT32 debug; /* -d */ > + uint32_t debug; /* -d */ > Performance_Mode_Type mode; /* -m */ > - DAT_UINT32 num_iterations; /* -i */ > - DAT_UINT32 pipeline_len; /* -p */ > + uint32_t num_iterations; /* -i */ > + uint32_t pipeline_len; /* -p */ > Performance_Cmd_Op_t op; > - DAT_UINT32 use_rsp; /* -r */ > + uint32_t use_rsp; /* -r */ > DAT_MEM_TYPE DT_mem_type; /* -M */ > } Performance_Cmd_t; > #pragma pack () > Index: linux-kernel/test/dapltest/include/dapl_transaction_test.h > =================================================================== > --- > linux-kernel/test/dapltest/include/dapl_transaction_test.h > (revision 2265) > +++ > linux-kernel/test/dapltest/include/dapl_transaction_test.h > (working copy) > @@ -37,8 +37,8 @@ typedef struct > { > DAT_BOOLEAN server_initiated; > DT_Transfer_Type transfer_type; > - DAT_UINT32 num_segs; > - DAT_UINT32 seg_size; > + uint32_t num_segs; > + uint32_t seg_size; > DAT_BOOLEAN reap_send_on_recv; > Bpool *bp; > > Index: linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.c > =================================================================== > --- linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.c > (revision 2265) > +++ linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.c > (working copy) > @@ -175,7 +175,7 @@ DT_Mdep_GetTime (void) > { > struct tms ts; > clock_t t = times (&ts); > - return (unsigned long) ((DAT_UINT64) t * 1000 / CLK_TCK); > + return (unsigned long) ((uint64_t) t * 1000 / CLK_TCK); > } > > double > Index: linux-kernel/test/dapltest/common/dapl_endian.c > =================================================================== > --- linux-kernel/test/dapltest/common/dapl_endian.c (revision 2265) > +++ linux-kernel/test/dapltest/common/dapl_endian.c (working copy) > @@ -39,21 +39,21 @@ DT_Endian_Init (void) > * Big/Little Endian conversion functions > */ > > -#define c1a32 ((DAT_UINT32)0x00FF00FF) > -#define c1b32 ((DAT_UINT32)0xFF00FF00) > -#define c2a32 ((DAT_UINT32)0x0000FFFF) > -#define c2b32 ((DAT_UINT32)0xFFFF0000) > -#define c164 ((DAT_UINT64)0x00FF00FF) > +#define c1a32 ((uint32_t)0x00FF00FF) > +#define c1b32 ((uint32_t)0xFF00FF00) > +#define c2a32 ((uint32_t)0x0000FFFF) > +#define c2b32 ((uint32_t)0xFFFF0000) > +#define c164 ((uint64_t)0x00FF00FF) > #define c1a64 (c164 | (c164 << 32)) > #define c1b64 (c1a64 << 8) > -#define c264 ((DAT_UINT64)0x0000FFFF) > +#define c264 ((uint64_t)0x0000FFFF) > #define c2a64 (c264 | (c264 << 32)) > #define c2b64 (c2a64 << 16) > -#define c3a64 ((DAT_UINT64)0xFFFFFFFF) > +#define c3a64 ((uint64_t)0xFFFFFFFF) > #define c3b64 (c3a64 << 32) > > -DAT_UINT32 > -DT_Endian32 (DAT_UINT32 val) > +uint32_t > +DT_Endian32 (uint32_t val) > { > if (DT_local_is_little_endian) > { > @@ -64,8 +64,8 @@ DT_Endian32 (DAT_UINT32 val) > return (val); > } > > -DAT_UINT64 > -DT_Endian64 (DAT_UINT64 val) > +uint64_t > +DT_Endian64 (uint64_t val) > { > if (DT_local_is_little_endian) > { > @@ -77,18 +77,18 @@ DT_Endian64 (DAT_UINT64 val) > return (val); > } > > -DAT_UINT32 > -DT_EndianMemHandle (DAT_UINT32 val) > +uint32_t > +DT_EndianMemHandle (uint32_t val) > { > val = ((val & c1a32) << 8) | ((val & c1b32) >> 8); > val = ((val & c2a32) << 16) | ((val & c2b32) >> 16); > return (val); > } > > -DAT_UINT64 > -DT_EndianMemAddress (DAT_UINT64 val) > +uint64_t > +DT_EndianMemAddress (uint64_t val) > { > - DAT_UINT64 val64; > + uint64_t val64; > val64 = val; > val64 = ((val64 & c1a64) << 8) | ((val64 & c1b64) >> 8); > val64 = ((val64 & c2a64) << 16) | ((val64 & c2b64) >> 16); > Index: linux-kernel/dat-provider/dapl_adapter_util.h > =================================================================== > --- linux-kernel/dat-provider/dapl_adapter_util.h (revision 2265) > +++ linux-kernel/dat-provider/dapl_adapter_util.h (working copy) > @@ -79,7 +79,7 @@ DAT_RETURN dapl_ib_connect(DAT_EP_HANDLE > DAT_RETURN dapl_ib_disconnect(DAPL_EP * ep_ptr, > DAT_CLOSE_FLAGS close_flags); > > DAT_RETURN dapl_ib_setup_conn_listener(DAPL_IA * ia_ptr, > - DAT_UINT64 ServiceID, > DAPL_SP * sp_ptr); > + u64 ServiceID, DAPL_SP * sp_ptr); > > DAT_RETURN dapl_ib_remove_conn_listener(DAPL_IA * ia_ptr, > DAPL_SP * sp_ptr); > > Index: linux-kernel/dat-provider/dapl_openib_cm.c > =================================================================== > --- linux-kernel/dat-provider/dapl_openib_cm.c (revision 2265) > +++ linux-kernel/dat-provider/dapl_openib_cm.c (working copy) > @@ -598,7 +598,7 @@ void dapl_ib_reinit_ep(DAPL_EP * ep_ptr) > */ > DAT_RETURN > dapl_ib_setup_conn_listener(DAPL_IA * ia_ptr, > - DAT_UINT64 ServiceID, DAPL_SP * sp_ptr) > + u64 ServiceID, DAPL_SP * sp_ptr) > { > int status; > > Index: linux-kernel/dat-provider/dapl_util.c > =================================================================== > --- linux-kernel/dat-provider/dapl_util.c (revision 2265) > +++ linux-kernel/dat-provider/dapl_util.c (working copy) > @@ -60,7 +60,7 @@ DAT_RETURN dapl_os_get_time(DAPL_OS_TIME > struct timeval tv; > > do_gettimeofday(&tv); > - *loc = ((DAT_UINT64) (tv.tv_sec) * 1000000L) + > (DAT_UINT64) tv.tv_usec; > + *loc = ((u64) (tv.tv_sec) * 1000000L) + (u64) tv.tv_usec; > > return DAT_SUCCESS; > } > Index: linux-kernel/dat-provider/dapl_util.h > =================================================================== > --- linux-kernel/dat-provider/dapl_util.h (revision 2265) > +++ linux-kernel/dat-provider/dapl_util.h (working copy) > @@ -188,7 +188,7 @@ static __inline__ char *dapl_os_strdup(c > * Timer Functions > */ > > -typedef DAT_UINT64 DAPL_OS_TIMEVAL; > +typedef u64 DAPL_OS_TIMEVAL; > typedef struct dapl_timer_entry DAPL_OS_TIMER; > typedef unsigned long long int DAPL_OS_TICKS; > > Index: linux-kernel/dat-provider/dapl.h > =================================================================== > --- linux-kernel/dat-provider/dapl.h (revision 2265) > +++ linux-kernel/dat-provider/dapl.h (working copy) > @@ -156,7 +156,7 @@ typedef struct dapl_cookie_buffer DAPL_C > > typedef struct dapl_hash_table DAPL_HASH_TABLE; > typedef struct dapl_hash_table *DAPL_HASH_TABLEP; > -typedef DAT_UINT64 DAPL_HASH_KEY; > +typedef u64 DAPL_HASH_KEY; > typedef void *DAPL_HASH_DATA; > > typedef struct dapl_hca DAPL_HCA; > Index: linux-kernel/dat/consumer.c > =================================================================== > --- linux-kernel/dat/consumer.c (revision 2265) > +++ linux-kernel/dat/consumer.c (working copy) > @@ -579,7 +579,7 @@ dat_ia_openv(const DAT_NAME_PTR name, > DAT_COUNT async_event_qlen, > DAT_EVD_HANDLE * async_event_handle, > DAT_IA_HANDLE * ia_handle, > - DAT_UINT32 dat_major, DAT_UINT32 dat_minor, > + u32 dat_major, u32 dat_minor, > DAT_BOOLEAN thread_safety) > { > DAT_IA_OPEN_FUNC ia_open_func; > Index: linux-kernel/dat/dat.h > =================================================================== > --- linux-kernel/dat/dat.h (revision 2265) > +++ linux-kernel/dat/dat.h (working copy) > @@ -37,18 +37,12 @@ > #define DAT_VERSION_MINOR 2 > #define DAT_THREADSAFE DAT_FALSE > > -/* unsigned host order, 32 bits */ > -typedef u_int32_t DAT_UINT32; > - > -/* unsigned host order, 64 bits */ > -typedef u_int64_t DAT_UINT64; > - > /* unsigned longest native to compiler */ > typedef unsigned long long DAT_UVERYLONG; > > typedef void * DAT_PVOID; > typedef int DAT_COUNT; > -typedef DAT_UINT64 DAT_PADDR; > +typedef u64 DAT_PADDR; > typedef struct sockaddr DAT_SOCK_ADDR; > typedef struct sockaddr_in6 DAT_SOCK_ADDR6; > > @@ -65,7 +59,7 @@ typedef struct sockaddr_in6 DAT_SOCK_ADD > * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ > */ > > -typedef DAT_UINT32 DAT_RETURN; > +typedef u32 DAT_RETURN; > > typedef enum dat_return_class { > DAT_CLASS_ERROR = 0x80000000, > @@ -341,14 +335,14 @@ typedef enum dat_return_subtype { > * Determining the success of an operation is best done with a macro; > * each of these returns a boolean value. > */ > -#define DAT_IS_ERROR(status) ((DAT_UINT32)(status) & > DAT_CLASS_ERROR) > -#define DAT_IS_WARNING(status) ((DAT_UINT32)(status) & > DAT_CLASS_WARNING) > +#define DAT_IS_ERROR(status) ((u32)(status) & DAT_CLASS_ERROR) > +#define DAT_IS_WARNING(status) ((u32)(status) & DAT_CLASS_WARNING) > > #define DAT_ERROR(Type, SubType) > ((DAT_RETURN)(DAT_CLASS_ERROR | Type | SubType)) > #define DAT_WARNING(Type, SubType) > ((DAT_RETURN)(DAT_CLASS_WARNING | Type | SubType)) > > -#define DAT_GET_TYPE(status) ((DAT_UINT32)(status) & > DAT_TYPE_MASK) > -#define DAT_GET_SUBTYPE(status) ((DAT_UINT32)(status) & > DAT_SUBTYPE_MASK) > +#define DAT_GET_TYPE(status) ((u32)(status) & DAT_TYPE_MASK) > +#define DAT_GET_SUBTYPE(status) ((u32)(status) & DAT_SUBTYPE_MASK) > > /* dat handle types */ > typedef enum dat_handle_type { > @@ -379,7 +373,7 @@ typedef enum dat_boolean { > > typedef union dat_context { > DAT_PVOID as_ptr; > - DAT_UINT64 as_64; > + u64 as_64; > DAT_UVERYLONG as_index; > } DAT_CONTEXT; > > @@ -404,7 +398,7 @@ typedef enum dat_completion_flags { > DAT_COMPLETION_EVD_THRESHOLD_FLAG = 0x10 > } DAT_COMPLETION_FLAGS; > > -typedef DAT_UINT32 DAT_TIMEOUT; /* microseconds */ > +typedef u32 DAT_TIMEOUT; /* microseconds */ > > /* timeout = infinity */ > #define DAT_TIMEOUT_INFINITE ((DAT_TIMEOUT) ~0) > @@ -426,8 +420,8 @@ typedef DAT_HANDLE DAT_SRQ_HANDLE; > > typedef DAT_SOCK_ADDR *DAT_IA_ADDRESS_PTR; > > -typedef DAT_UINT64 DAT_CONN_QUAL; > -typedef DAT_UINT64 DAT_PORT_QUAL; > +typedef u64 DAT_CONN_QUAL; > +typedef u64 DAT_PORT_QUAL; > > /* QOS definitions */ > typedef enum dat_qos { > @@ -518,11 +512,11 @@ typedef enum dat_psp_flags { > * For the best performance, Consumers should align each > buffer segment to > * the boundary specified by the dat_optimal_alignment. > */ > -typedef DAT_UINT32 DAT_LMR_CONTEXT; > -typedef DAT_UINT32 DAT_RMR_CONTEXT; > +typedef u32 DAT_LMR_CONTEXT; > +typedef u32 DAT_RMR_CONTEXT; > > -typedef DAT_UINT64 DAT_VLEN; > -typedef DAT_UINT64 DAT_VADDR; > +typedef u64 DAT_VLEN; > +typedef u64 DAT_VADDR; > > /* It is legal for the Consumer to specify zero for segment_length > * of the dat_lmr_triplet. When 0 is specified for the > @@ -532,14 +526,14 @@ typedef DAT_UINT64 DAT_VADDR; > > typedef struct dat_lmr_triplet { > DAT_LMR_CONTEXT lmr_context; > - DAT_UINT32 pad; /* align structure on 64-bit boundry */ > + u32 pad; /* align structure on 64-bit boundry */ > DAT_VADDR virtual_address; > DAT_VLEN segment_length; > } DAT_LMR_TRIPLET; > > typedef struct dat_rmr_triplet { > DAT_RMR_CONTEXT rmr_context; > - DAT_UINT32 pad; /* align structure on 64-bit boundry */ > + u32 pad; /* align structure on 64-bit boundry */ > DAT_VADDR target_address; > DAT_VLEN segment_length; > } DAT_RMR_TRIPLET; > @@ -955,8 +949,8 @@ typedef struct dat_upcall_object { > > typedef struct dat_provider_info { > char ia_name[DAT_NAME_MAX_LENGTH]; > - DAT_UINT32 dat_version_major; > - DAT_UINT32 dat_version_minor; > + u32 dat_version_major; > + u32 dat_version_minor; > DAT_BOOLEAN is_thread_safe; > } DAT_PROVIDER_INFO; > > @@ -1013,10 +1007,10 @@ typedef struct dat_lmr_param { > typedef struct dat_ia_attr { > char adapter_name[DAT_NAME_MAX_LENGTH]; > char vendor_name[DAT_NAME_MAX_LENGTH]; > - DAT_UINT32 hardware_version_major; > - DAT_UINT32 hardware_version_minor; > - DAT_UINT32 firmware_version_major; > - DAT_UINT32 firmware_version_minor; > + u32 hardware_version_major; > + u32 hardware_version_minor; > + u32 firmware_version_major; > + u32 firmware_version_minor; > DAT_IA_ADDRESS_PTR ia_address_ptr; > DAT_COUNT max_eps; > DAT_COUNT max_dto_per_ep; > @@ -1078,10 +1072,10 @@ typedef enum dat_ep_creator_for_psp { > > typedef struct dat_provider_attr { > char provider_name[DAT_NAME_MAX_LENGTH]; > - DAT_UINT32 provider_version_major; > - DAT_UINT32 provider_version_minor; > - DAT_UINT32 dat_version_major; > - DAT_UINT32 dat_version_minor; > + u32 provider_version_major; > + u32 provider_version_minor; > + u32 dat_version_major; > + u32 dat_version_minor; > DAT_MEM_TYPE lmr_mem_types_supported; > DAT_IOV_OWNERSHIP iov_ownership_on_return; > DAT_QOS dat_qos_supported; > @@ -1091,7 +1085,7 @@ typedef struct dat_provider_attr { > DAT_BOOLEAN supports_multipath; > DAT_EP_CREATOR_FOR_PSP ep_creator; > DAT_UPCALL_POLICY upcall_policy; > - DAT_UINT32 optimal_buffer_alignment; > + u32 optimal_buffer_alignment; > const DAT_BOOLEAN evd_stream_merging_supported[6][6]; > DAT_BOOLEAN srq_supported; > DAT_COUNT srq_watermarks_supported; > @@ -1117,7 +1111,7 @@ typedef DAT_RETURN (*DAT_IA_OPEN_FUNC) ( > > typedef DAT_RETURN (*DAT_IA_OPENV_FUNC) ( > const DAT_NAME_PTR, DAT_COUNT, DAT_EVD_HANDLE *, > DAT_IA_HANDLE *, > - DAT_UINT32, DAT_UINT32, DAT_BOOLEAN); > + u32, u32, DAT_BOOLEAN); > > typedef DAT_RETURN (*DAT_IA_CLOSE_FUNC) (DAT_IA_HANDLE, > DAT_CLOSE_FLAGS); > > @@ -1380,7 +1374,7 @@ extern DAT_RETURN dat_registry_remove_pr > */ > extern DAT_RETURN dat_ia_openv(const DAT_NAME_PTR, DAT_COUNT, > DAT_EVD_HANDLE *, DAT_IA_HANDLE *, > - DAT_UINT32, DAT_UINT32, DAT_BOOLEAN); > + u32, u32, DAT_BOOLEAN); > > #define dat_ia_open(name, qlen, async_evd, ia) \ > dat_ia_openv((name), (qlen), (async_evd), (ia), \ > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From iod00d at hp.com Thu May 5 14:51:58 2005 From: iod00d at hp.com (Grant Grundler) Date: Thu, 5 May 2005 14:51:58 -0700 Subject: [openib-general] PF 5/4 Telecon Minutes In-Reply-To: <1115325202.4476.24.camel@localhost.localdomain> References: <20050505001734.GD16887@esmail.cup.hp.com> <20050505183244.GB19523@esmail.cup.hp.com> <1115320068.4476.1.camel@localhost.localdomain> <20050505203021.GC19523@esmail.cup.hp.com> <1115325202.4476.24.camel@localhost.localdomain> Message-ID: <20050505215158.GG19523@esmail.cup.hp.com> On Thu, May 05, 2005 at 04:39:49PM -0400, Hal Rosenstock wrote: > On Thu, 2005-05-05 at 16:30, Grant Grundler wrote: > > Understood and I sympathize. I'm just pointing out there is (a) opportunity > > for woody to leverage and (b) it's of general interest. > > > > If you can commit the document source (.mgp?) to subversion, > > It's ppt. Is there some conversion or would I need to do this manually ? Hrm...I don't know how subversion deals with binary blobs or if we want to go down that path. I was hopeful this might be trivial since .mgp "source" files are just plain text. The pdf file *looks* like a magic point presentation. Should I create a .mgp file with the existing content for you? That won't solve the whole problem since it embeds the arch block diagram image that would need to go into subversion as well. And unless there has been a windows port, it will require a linux box to run magic point (of which there should be no shortage at OLS). thanks, grant From robert.j.woodruff at intel.com Thu May 5 15:28:43 2005 From: robert.j.woodruff at intel.com (Bob Woodruff) Date: Thu, 5 May 2005 15:28:43 -0700 Subject: [openib-general] RE: [OOPS] user-mode verbs In-Reply-To: <52mzr9622e.fsf@topspin.com> Message-ID: Roland wrote, > > Bob> Ok, let me know when you have the fixes checked in and I will > Bob> do a fresh pull from SVN. >I just checked in a new version of uverbs_cmd.c that should fix the >use-after-free bugs. >- R. Looks like this fixed my problem. I will run tests overnight to make sure, but so far (after 4 hours) it looks good. woody From tduffy at sun.com Thu May 5 15:30:22 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 05 May 2005 15:30:22 -0700 Subject: [openib-general] [PATCH] kDAPL: remove DAT_UINT32 and DAT_UINT64 In-Reply-To: <469958e00505051425a5fc1fd@mail.gmail.com> References: <1115327289.11449.7.camel@duffman> <469958e00505051425a5fc1fd@mail.gmail.com> Message-ID: <1115332222.11449.11.camel@duffman> On Thu, 2005-05-05 at 14:25 -0700, Caitlin Bestler wrote: > One of the strengths of dapltest is that it allows peer-to-peer > testing across dissimiliar platforms. I can run a peer-to-peer > test between Linux 2.4 and Linux 2.6, or between PPC and > x86, or between Linux and BSD. > > Even more to the point, it allows peer-to-peer testing between > uDAPL clients and kDAPL clients. > > And it accomplishes all of that while still letting its linux > kernel version be a loadable kernel module with GPL licensing. > > But forking the code to rewrite it under linux kernel standards > means that it will eventually be hard to state that you are > running the same application in all of the target environments. > It will also make it hard to have the overwhelming majority of > the code be applicable to both user and kernel modes (which > Steve and James did a very admirable job of on the sourceforge > version). > > Making a Linux native DAPL is one thing, but there are reasons > for dapltest code to be OS neutral other than minimizing coding > and maintenance efforts. sorry, but it will be necessary to fork kDAPL for inclusion in the Linux kernel. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From tduffy at sun.com Thu May 5 15:39:06 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 05 May 2005 15:39:06 -0700 Subject: [openib-general] [PATCH] kDAPL: remove DAT_UINT32 and DAT_UIN T64 In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35A4E@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35A4E@mtvex01.mtv.mtl.com> Message-ID: <1115332746.20452.2.camel@duffman> On Fri, 2005-05-06 at 00:42 +0300, Itamar Rabenstein wrote: > do you still see problems with dat_lmr_create() ? yes > i am using x86 and dont see any problems > can you send me the /var/log/messages with full debug enable > modprobe ib_dat_provider g_dapl_dbg_type=0xffff btw, it is now just dbg_type. kDAPL: DAPL: Started (dapl_init) kDAPL: Inside dapl_add_one called by mthca0 KDAPL : Added device mthca0a to the registry KDAPL : Added device mthca0b to the registry Kdapltest Module Loaded - major 253 kDAPL: dapl_ia_open (mthca0a, 8, ffff81003b011ed0, ffff81003b011ef8) kDAPL: dapl_ia_open () returns 0x0 kDAPL: dapl_pz_create (ffff81003aef2168, ffff81003b011ef0) kDAPL: dapl_evd_kcreate (ffff81003aef2168, 8, 1, upcall, 0x20, ffff81003b011ee8) kDAPL: dapl_evd_kcreate (ffff81003aef2168, 8, 1, upcall, 0xa0, ffff81003b011ee0) kDAPL: dapl_evd_kcreate (ffff81003aef2168, 8, 1, upcall, 0x40, ffff81003b011ed8) kDAPL: dapl_ep_create (ffff81003aef2168, ffff8100770c0e10, ffff81003aef2a28, ffff81003aef26e0, ffff81003b6a1b40, 0000000000000000, ffff81003b011ec8) kDAPL: created qp_hndl = ffff81007693b2d8, qpn = 170406 kDAPL: dapl_ia_query (ffff81003aef2168, ffff81003b011dc0, ffff81003a97cb88, ffff81003a97ce38) kDAPL: dapl_ia_query () returns 0x0 kDAPL: dapl_ep_query (ffff81007e3354f8, ffff81003b011d08) kDAPL: dapl_ep_free (ffff81007e3354f8) kDAPL: dapl_ep_disconnect (ffff81007e3354f8, 0) kDAPL: dapl_ep_disconnect () returns 0x8007002e kDAPL: dapl_ep_free: Free EP: 0, ep ffff81007e3354f8 qp_state 1 qp_handle 7693b2d8 kDAPL: dapl_evd_free (ffff81003b6a1b40) kDAPL: dapl_evd_free () returns 0x0 kDAPL: dapl_evd_free (ffff81003aef26e0) kDAPL: dapl_evd_free () returns 0x0 kDAPL: dapl_evd_free (ffff81003aef2a28) kDAPL: dapl_evd_free () returns 0x0 kDAPL: dapl_pz_free (ffff8100770c0e10) kDAPL: dapl_ia_query (ffff81003aef2168, 0000000000000000, 0000000000000000, ffff81003b011ab8) kDAPL: dapl_ia_query () returns 0x0 kDAPL: dapl_ia_close (ffff81003aef2168, 1) kDAPL: dapl_evd_free (ffff81003aef2d70) kDAPL: dapl_evd_free () returns 0x0 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From tduffy at sun.com Thu May 5 15:50:22 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 05 May 2005 15:50:22 -0700 Subject: [openib-general] PF 5/4 Telecon Minutes In-Reply-To: <20050505215158.GG19523@esmail.cup.hp.com> References: <20050505001734.GD16887@esmail.cup.hp.com> <20050505183244.GB19523@esmail.cup.hp.com> <1115320068.4476.1.camel@localhost.localdomain> <20050505203021.GC19523@esmail.cup.hp.com> <1115325202.4476.24.camel@localhost.localdomain> <20050505215158.GG19523@esmail.cup.hp.com> Message-ID: <1115333422.20452.7.camel@duffman> On Thu, 2005-05-05 at 14:51 -0700, Grant Grundler wrote: > Hrm...I don't know how subversion deals with binary blobs or if we > want to go down that path. > > I was hopeful this might be trivial since .mgp "source" files are > just plain text. The pdf file *looks* like a magic point presentation. > > Should I create a .mgp file with the existing content for you? > That won't solve the whole problem since it embeds the arch block > diagram image that would need to go into subversion as well. > And unless there has been a windows port, it will require a linux box > to run magic point (of which there should be no shortage at OLS). Bet you could take the ppt and convert it (trivially using openoffice 1.9) as Open Document Presentation format which is just zipped xml. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From halr at voltaire.com Thu May 5 15:45:02 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 05 May 2005 18:45:02 -0400 Subject: [openib-general] [PATCH] [TRIVIAL] kDAPL: Fix ib_dat_provider MODULE_PARM_DESC Message-ID: <1115333101.4476.8.camel@localhost.localdomain> Fix ib_dat_provider MODULE_PARM_DESC Signed-off-by: Hal Rosenstock Index: dapl_module.c =================================================================== --- dapl_module.c (revision 2265) +++ dapl_module.c (working copy) @@ -44,7 +44,7 @@ int g_dapl_dbg_type = 0; module_param_named(dbg_type, g_dapl_dbg_type, int, 0644); -MODULE_PARM_DESC(g_dapl_dbg_type, "Enable dapl debug types"); +MODULE_PARM_DESC(dbg_type, "Enable dapl debug types"); static int dapl_init(void); static void dapl_fini(void); From libor at topspin.com Thu May 5 16:34:39 2005 From: libor at topspin.com (Libor Michalek) Date: Thu, 5 May 2005 16:34:39 -0700 Subject: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation In-Reply-To: <4279142A.8050501@ammasso.com>; from timur.tabi@ammasso.com on Wed, May 04, 2005 at 01:27:54PM -0500 References: <20050411142213.GC26127@kalmia.hozed.org> <52mzs51g5g.fsf@topspin.com> <20050411163342.GE26127@kalmia.hozed.org> <5264yt1cbu.fsf@topspin.com> <20050411180107.GF26127@kalmia.hozed.org> <52oeclyyw3.fsf@topspin.com> <20050411171347.7e05859f.akpm@osdl.org> <20050412180447.E6958@topspin.com> <20050425203110.A9729@topspin.com> <4279142A.8050501@ammasso.com> Message-ID: <20050505163439.A26428@topspin.com> On Wed, May 04, 2005 at 01:27:54PM -0500, Timur Tabi wrote: > Libor Michalek wrote: > > > The program opens the charcter device file descriptor, pins the pages > > and waits for a signal, before checking the pages, which is sent to the > > process after running some other program which exercises the VM. On older > > kernels the check fails, on my 2.6.11 kernel the check succeeds. So > > mlock is not needed on top of get_user_pages() as it was before. > > When you say "older", what exactly do you mean? I have different test > that normally fails with just get_user_pages(), but it works with 2.6.9 > and above. I haven't been able to get any kernel earlier than 2.6.9 to > compile or boot properly, so I'm having a hard time narrowing down the > actual point when get_user_pages() started working. The older kernel I tried was one of the 2.4.21 RHEL 3 kernels. I hadn't spent much time investigating the issue since this was a new kernel, so it was a natural one for me to try. -Libor From libor at topspin.com Thu May 5 17:21:22 2005 From: libor at topspin.com (Libor Michalek) Date: Thu, 5 May 2005 17:21:22 -0700 Subject: [openib-general] 0 op factor In-Reply-To: <20050505155803.GS15688@aon.at>; from blist@aon.at on Thu, May 05, 2005 at 05:58:03PM +0200 References: <20050505155803.GS15688@aon.at> Message-ID: <20050505172122.B26428@topspin.com> On Thu, May 05, 2005 at 05:58:03PM +0200, Bernhard Fischer wrote: > Hi, > > I forgot, but what's the purpose of statements like: > > $ egrep -ri "[^[:digit:]|\.]0[[:space:]]*[-+*/]" *|egrep -v "(\.(svn-base|txt):|/\*)" > infiniband/ulp/sdp/sdp_inet.c: * flag: 0 - recv shutdown > infiniband/ulp/sdp/sdp_actv.c: sdp_iocb_q_cancel_all(conn, (0 - error)); > infiniband/ulp/sdp/sdp_actv.c: sdp_cm_actv_error(conn, (0 - result)); > infiniband/ulp/sdp/sdp_actv.c: sdp_cm_actv_error(conn, (0 - status)); > infiniband/ulp/sdp/sdp_pass.c: sdp_iocb_q_cancel_all(conn, (0 - error)); > infiniband/ulp/sdp/sdp_pass.c: sdp_cm_pass_error(conn, (0 - result)); > > Could somebody please elaborate? No point other then the obvious, changing the sign of the variables value. It's just a style convention that I have no problem changing. -Libor From blist at aon.at Thu May 5 17:30:18 2005 From: blist at aon.at (Bernhard Fischer) Date: Fri, 6 May 2005 02:30:18 +0200 Subject: [openib-general] [SDP] missing MODULE_PARM_DESC Message-ID: <20050506003018.GD15688@aon.at> Hi, in src/linux-kernel/infiniband/ulp/sdp/sdp_inet.c module_param seem to be missing MODULE_PARM_DESC please fix. From libor at topspin.com Thu May 5 17:32:57 2005 From: libor at topspin.com (Libor Michalek) Date: Thu, 5 May 2005 17:32:57 -0700 Subject: [openib-general] [patch] misc trivia nitpicks 01 In-Reply-To: <20050505170956.GU15688@aon.at>; from blist@aon.at on Thu, May 05, 2005 at 07:09:56PM +0200 References: <20050505170956.GU15688@aon.at> Message-ID: <20050505173257.C26428@topspin.com> On Thu, May 05, 2005 at 07:09:56PM +0200, Bernhard Fischer wrote: > Hi, > > please consider attached. > > diffstat ./gen2.2257.cow-20050505-1902.diff > linux-kernel/infiniband/ulp/sdp/sdp_actv.c | 27 +-- > userspace/libmthca/src/mthca.c | 4 > userspace/management/osm/include/opensm/osm_subnet.h | 79 +++++----- > userspace/management/osm/include/vendor/osm_vendor_mtl_transaction_mgr.h | 43 ++--- > userspace/management/osm/opensm/osm_sa_service_record.c | 8 - > userspace/management/osm/opensm/osm_sm_state_mgr.c | 38 ++-- > userspace/management/osm/opensm/osm_sw_info_rcv.c | 9 - > userspace/management/osm/opensm/osm_trap_rcv.c | 62 +++---- > 8 files changed, 134 insertions(+), 136 deletions(-) Bernhard, I have no problem with the content, but I think this would be easier if you submitted a patch per module/program, so each maintainer could treat this independently. -Libor From blist at aon.at Thu May 5 17:39:42 2005 From: blist at aon.at (Bernhard Fischer) Date: Fri, 6 May 2005 02:39:42 +0200 Subject: [openib-general] [patch] misc trivia nitpicks 04 Message-ID: <20050506003942.GE15688@aon.at> Hi, - s/incomm/incom/g -------------- next part -------------- diff -X excl -rduNp gen2.2257.oorig/trunk/src/linux-kernel/infiniband/core/ucm.c gen2.2257/trunk/src/linux-kernel/infiniband/core/ucm.c --- gen2.2257.oorig/trunk/src/linux-kernel/infiniband/core/ucm.c 2005-04-26 20:58:50.000000000 +0200 +++ gen2.2257/trunk/src/linux-kernel/infiniband/core/ucm.c 2005-05-06 02:46:18.055867696 +0200 @@ -99,7 +99,7 @@ static void ib_ucm_ctx_put(struct ib_ucm list_del(&uevent->file_list); list_del(&uevent->ctx_list); - /* clear incomming connections. */ + /* clear incoming connections. */ if (uevent->cm_id) (void)ib_destroy_cm_id(uevent->cm_id); diff -X excl -rduNp gen2.2257.oorig/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_inet.c gen2.2257/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_inet.c --- gen2.2257.oorig/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_inet.c 2005-04-26 20:58:46.000000000 +0200 +++ gen2.2257/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_inet.c 2005-05-06 02:42:17.965367000 +0200 @@ -37,7 +37,7 @@ #include "sdp_main.h" /* - * list of connections waiting for an incomming connection + * list of connections waiting for an incoming connection */ static int proto_family = AF_INET_SDP; static int buff_min = SDP_BUFF_POOL_COUNT_MIN; @@ -690,7 +690,7 @@ done: } /* - * sdp_inet_listen - listen on a socket for incomming addresses + * sdp_inet_listen - listen on a socket for incoming addresses */ static int sdp_inet_listen(struct socket *sock, int backlog) { diff -X excl -rduNp gen2.2257.oorig/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_pass.c gen2.2257/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_pass.c --- gen2.2257.oorig/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_pass.c 2005-04-26 20:58:46.000000000 +0200 +++ gen2.2257/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_pass.c 2005-05-06 02:45:48.961290744 +0200 @@ -35,7 +35,7 @@ #include "sdp_main.h" /* - * handle incomming passive connection error. (REJ) + * handle incoming passive connection error. (REJ) */ void sdp_cm_pass_error(struct sdp_opt *conn, int error) { @@ -67,7 +67,7 @@ void sdp_cm_pass_error(struct sdp_opt *c } /* - * handle incomming passive connection establishment. (RTU) + * handle incoming passive connection establishment. (RTU) */ int sdp_cm_pass_establish(struct sdp_opt *conn) { @@ -295,7 +295,7 @@ static int sdp_cm_listen_lookup(struct s struct sock *sk; int result; /* - * match a listener with an incomming conn, and generate + * match a listener with an incoming conn, and generate * accept message and state on success. */ sdp_dbg_ctrl(conn, diff -X excl -rduNp gen2.2257.oorig/trunk/src/userspace/examples/aio/README gen2.2257/trunk/src/userspace/examples/aio/README --- gen2.2257.oorig/trunk/src/userspace/examples/aio/README 2005-05-05 10:49:55.000000000 +0200 +++ gen2.2257/trunk/src/userspace/examples/aio/README 2005-05-06 02:46:34.885309232 +0200 @@ -17,7 +17,7 @@ ttcp.aio.c: gcc -I../../../linux-kernel/infiniband/ulp/sdp ttcp.aio.c \ -o ttcp.aio.x -laio - The program runs in two modes, a server which listens for incomming + The program runs in two modes, a server which listens for incoming connections and acts as a data sink, and a client which connects to a given address/port and acts as a data source. A decent help exists for available parameters, but here are some reasonable defaults: diff -X excl -rduNp gen2.2257.oorig/trunk/src/userspace/management/osm/include/iba/ib_types.h gen2.2257/trunk/src/userspace/management/osm/include/iba/ib_types.h --- gen2.2257.oorig/trunk/src/userspace/management/osm/include/iba/ib_types.h 2005-04-26 20:58:28.000000000 +0200 +++ gen2.2257/trunk/src/userspace/management/osm/include/iba/ib_types.h 2005-05-06 02:47:15.938068264 +0200 @@ -7403,7 +7403,7 @@ typedef struct _ib_ca_attr * * max_qp_resp_res * max_eec_resp_res -* Maximum limit on number of responder resources for incomming RDMA +* Maximum limit on number of responder resources for incoming RDMA * operations, on QPs and EEC's respectively. * * max_resp_res From blist at aon.at Thu May 5 17:49:05 2005 From: blist at aon.at (Bernhard Fischer) Date: Fri, 6 May 2005 02:49:05 +0200 Subject: [openib-general] [patch] misc trivia nitpicks 01 In-Reply-To: <20050505173257.C26428@topspin.com> References: <20050505170956.GU15688@aon.at> <20050505173257.C26428@topspin.com> Message-ID: <20050506004904.GF15688@aon.at> On Thu, May 05, 2005 at 05:32:57PM -0700, Libor Michalek wrote: >On Thu, May 05, 2005 at 07:09:56PM +0200, Bernhard Fischer wrote: >> Hi, >> >> please consider attached. >> >> diffstat ./gen2.2257.cow-20050505-1902.diff >> linux-kernel/infiniband/ulp/sdp/sdp_actv.c | 27 +-- >> userspace/libmthca/src/mthca.c | 4 >> userspace/management/osm/include/opensm/osm_subnet.h | 79 +++++----- >> userspace/management/osm/include/vendor/osm_vendor_mtl_transaction_mgr.h | 43 ++--- >> userspace/management/osm/opensm/osm_sa_service_record.c | 8 - >> userspace/management/osm/opensm/osm_sm_state_mgr.c | 38 ++-- >> userspace/management/osm/opensm/osm_sw_info_rcv.c | 9 - >> userspace/management/osm/opensm/osm_trap_rcv.c | 62 +++---- >> 8 files changed, 134 insertions(+), 136 deletions(-) > >Bernhard, > > I have no problem with the content, but I think this would be easier if >you submitted a patch per module/program, so each maintainer could treat >this independently. I tend to disagree. I'm an independant nitpicker. I'll try to separate those, but if a patch -- especially a relatively trivial one, which mostly touches obvious typos in the documentation -- comes in, i'd think that you in the sense of payed maintainers should be able to weight if you can apply them, regardless of who pays the 3 minutes to read through it. At any rate, i'll try to send them as separate patches per vendor in the future.. PS: cisco is core, mthca, sdp, and voltaire is osm, yes? Or would a split per individual be better? I really don't know. > >-Libor > > From jcarr at linuxmachines.com Thu May 5 18:02:16 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Thu, 05 May 2005 18:02:16 -0700 Subject: [openib-general] in need of a simple ulp In-Reply-To: <426FD098.9090300@ichips.intel.com> References: <426FCF0A.3070806@linuxmachines.com> <426FD098.9090300@ichips.intel.com> Message-ID: <427AC218.9070307@linuxmachines.com> Sean Hefty wrote: > Within the SVN repository, your best bet for finding things is staying > withing the gen2 branch. For a relatively simple example that does > what you mention above, try: > > https://openib.org/svn/gen2/utils/src/linux-kernel/infiniband/util/cmpost/ > > This is a simple CM test program for the kernel. Does your CM test module actually transfer data? I've set it up and run it; it runs fine, but when I put some data in the test.mem buffer, the test.mem buffer on the client never changes. Am I looking in the wrong place for where in memory the actual message is allocated and where it actually ends up? As far as I could tell that's what it seemed like you intended test.mem for. Other than that, the code has been very helpful for getting started with IB and was what I was looking for. Thanks for any suggestions, Jeff From libor at topspin.com Thu May 5 18:08:28 2005 From: libor at topspin.com (Libor Michalek) Date: Thu, 5 May 2005 18:08:28 -0700 Subject: [openib-general] [patch] misc trivia nitpicks 01 In-Reply-To: <20050506004904.GF15688@aon.at>; from blist@aon.at on Fri, May 06, 2005 at 02:49:05AM +0200 References: <20050505170956.GU15688@aon.at> <20050505173257.C26428@topspin.com> <20050506004904.GF15688@aon.at> Message-ID: <20050505180828.D26428@topspin.com> On Fri, May 06, 2005 at 02:49:05AM +0200, Bernhard Fischer wrote: > On Thu, May 05, 2005 at 05:32:57PM -0700, Libor Michalek wrote: > >On Thu, May 05, 2005 at 07:09:56PM +0200, Bernhard Fischer wrote: > >> Hi, > >> > >> please consider attached. > >> > >> diffstat ./gen2.2257.cow-20050505-1902.diff > >> linux-kernel/infiniband/ulp/sdp/sdp_actv.c | 27 +-- > >> userspace/libmthca/src/mthca.c | 4 > >> userspace/management/osm/include/opensm/osm_subnet.h | 79 +++++----- > >> userspace/management/osm/include/vendor/osm_vendor_mtl_transaction_mgr.h | 43 ++--- > >> userspace/management/osm/opensm/osm_sa_service_record.c | 8 - > >> userspace/management/osm/opensm/osm_sm_state_mgr.c | 38 ++-- > >> userspace/management/osm/opensm/osm_sw_info_rcv.c | 9 - > >> userspace/management/osm/opensm/osm_trap_rcv.c | 62 +++---- > >> 8 files changed, 134 insertions(+), 136 deletions(-) > > > >Bernhard, > > > > I have no problem with the content, but I think this would be easier if > >you submitted a patch per module/program, so each maintainer could treat > >this independently. > > At any rate, i'll try to send them as separate patches per vendor in the > future.. > > PS: > cisco is core, mthca, sdp, and voltaire is osm, yes? Or would a split > per individual be better? I really don't know. I was just suggesting breaking it up by module/program which would avoid a loaded value judgment like this yet still achieves the desired result. The only reason I suggested it was that I saw SDP and OpenSM in the same patch and I haven't even built OpenSM in a while, so I didn't want to check in changes to it. -Libor From mshefty at ichips.intel.com Thu May 5 18:08:21 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Thu, 05 May 2005 18:08:21 -0700 Subject: [openib-general] in need of a simple ulp In-Reply-To: <427AC218.9070307@linuxmachines.com> References: <426FCF0A.3070806@linuxmachines.com> <426FD098.9090300@ichips.intel.com> <427AC218.9070307@linuxmachines.com> Message-ID: <427AC385.5070308@ichips.intel.com> Jeff Carr wrote: > Does your CM test module actually transfer data? I've set it up and run > it; it runs fine, but when I put some data in the test.mem buffer, the > test.mem buffer on the client never changes. In theory, yes. Though I've never verified the data. Note that the code uses a single data buffer for the sends and receives, so it can get overwritten. > Am I looking in the wrong place for where in memory the actual message > is allocated and where it actually ends up? As far as I could tell > that's what it seemed like you intended test.mem for. You're looking in the right place. I'll add some code to format the data and verify that it is making it to the remote side, and see if I can track down any problems. (This will need to wait until tomorrow for me though...) - Sean From blist at aon.at Thu May 5 18:14:26 2005 From: blist at aon.at (Bernhard Fischer) Date: Fri, 6 May 2005 03:14:26 +0200 Subject: [openib-general] [patch] misc trivia nitpicks 01 In-Reply-To: <20050505180828.D26428@topspin.com> References: <20050505170956.GU15688@aon.at> <20050505173257.C26428@topspin.com> <20050506004904.GF15688@aon.at> <20050505180828.D26428@topspin.com> Message-ID: <20050506011426.GG15688@aon.at> On Thu, May 05, 2005 at 06:08:28PM -0700, Libor Michalek wrote: >On Fri, May 06, 2005 at 02:49:05AM +0200, Bernhard Fischer wrote: >> On Thu, May 05, 2005 at 05:32:57PM -0700, Libor Michalek wrote: >> >On Thu, May 05, 2005 at 07:09:56PM +0200, Bernhard Fischer wrote: >> >> Hi, >> >> >> >> please consider attached. >> >> >> >> diffstat ./gen2.2257.cow-20050505-1902.diff >> >> linux-kernel/infiniband/ulp/sdp/sdp_actv.c | 27 +-- >> >> userspace/libmthca/src/mthca.c | 4 >> >> userspace/management/osm/include/opensm/osm_subnet.h | 79 +++++----- >> >> userspace/management/osm/include/vendor/osm_vendor_mtl_transaction_mgr.h | 43 ++--- >> >> userspace/management/osm/opensm/osm_sa_service_record.c | 8 - >> >> userspace/management/osm/opensm/osm_sm_state_mgr.c | 38 ++-- >> >> userspace/management/osm/opensm/osm_sw_info_rcv.c | 9 - >> >> userspace/management/osm/opensm/osm_trap_rcv.c | 62 +++---- >> >> 8 files changed, 134 insertions(+), 136 deletions(-) [] >> At any rate, i'll try to send them as separate patches per vendor in the >> future.. [] > I was just suggesting breaking it up by module/program which would >avoid a loaded value judgment like this yet still achieves the desired Yes, i see. Thanks in advance for applying the parts you feel comfortable with :) cheers, >result. The only reason I suggested it was that I saw SDP and OpenSM >in the same patch and I haven't even built OpenSM in a while, so I >didn't want to check in changes to it. > >-Libor > > From blist at aon.at Thu May 5 18:21:41 2005 From: blist at aon.at (Bernhard Fischer) Date: Fri, 6 May 2005 03:21:41 +0200 Subject: [openib-general] Re: [openib-commits] r2239 - gen2/trunk/src/linux-kernel/infiniband/include In-Reply-To: <1115325042.4476.15.camel@localhost.localdomain> References: <20050505201626.GX15688@aon.at> <1115325042.4476.15.camel@localhost.localdomain> Message-ID: <20050506012140.GH15688@aon.at> On Thu, May 05, 2005 at 04:33:23PM -0400, Hal Rosenstock wrote: >On Thu, 2005-05-05 at 16:16, Bernhard Fischer wrote: >> Furthermore it would be really, really nice if anyone who touches files >> would trim any whitespace as that is extremely annoying if you happen to >> be forced to look at. Also, whitespace just eats my precious >> (disc-)space for no good. > >Are OpenSM files guilty of this ? Just looking at the headers (which are probably most likely to be installed) and within those, superfluous noise before eol. $ cat ~/huh4 find ./ -iname "*h" -exec grep -HinE "[[:space:]]$" {} \; gives the attached (file:line as of gen2 rev. 2257) I'd say yes. >From my POV, it would be good to trim at least those that might get propagated to installed images. YMMV. -------------- next part -------------- feeding /usr/local/src/obj/gen2/trunk/src/userspace into /home/cow/huh4 ./libibcm/include/infiniband/cm_abi.h:53: ./libibcm/include/infiniband/cm.h:267: * IB_CM_REQ_RECEIVED and IB_CM_SIDR_REQ_RECEIVED communication events ./libibcm/include/infiniband/cm.h:268: * generated as a result of listen requests result in the allocation of a ./libibcm/include/infiniband/cm.h:269: * new @cm_id. ./libibcm/include/infiniband/cm.h:328:int ib_cm_attr_id(uint32_t cm_id, ./libmthca/config/ltmain.sh:110:: ${IFS=" ./mstflint/mtcr.h:146: unsigned long long ./mstflint/mtcr.h:192: Binary file ./libibverbs/examples/.libs/ibv_asyncwatch matches ./libibverbs/config/ltmain.sh:110:: ${IFS=" ./libibverbs/include/infiniband/verbs.h:378: struct ibv_pd *pd; ./libibverbs/include/infiniband/verbs.h:522: ./libibverbs/include/infiniband/driver.h:63: ./management/libibcommon/include/infiniband/common.h:187:get_build_version(void) ./management/libibmad/include/infiniband/mad.h:242: IB_GS_DATA_F, ./management/osm/include/vendor/osm_vendor_mlx_sender.h:63: * ./management/osm/include/vendor/osm_vendor_mlx_sender.h:118: * NAME osmv_rmpp_snd_error ./management/osm/include/vendor/osm_vendor_mtl.h:91:* ./management/osm/include/vendor/osm_vendor_mtl.h:153:* Handle returned by MTL open call (ib_open_al). ./management/osm/include/vendor/osm_vendor_mtl.h:242: * Given the vendor obj and a guid ./management/osm/include/vendor/osm_vendor_mtl.h:282: * Fill in the array of port_attr with all available ports on ALL the ./management/osm/include/vendor/osm_vendor_mtl.h:284: * ALSO - ./management/osm/include/vendor/osm_vendor_mlx.h:61: ./management/osm/include/vendor/osm_vendor_mlx.h:68: uint32_t resp_timeout; ./management/osm/include/vendor/osm_vendor_mlx.h:69: ./management/osm/include/vendor/osm_vendor_mlx.h:75: ./management/osm/include/vendor/osm_vendor_mlx_transport.h:40: * ./management/osm/include/vendor/osm_vendor_mlx_transport.h:43: * that allows sending a single MAD/receiving a callback ./management/osm/include/vendor/osm_vendor_mlx_transport.h:44: * when a single MAD is received. ./management/osm/include/vendor/osm_vendor_mlx_inout.h:56:#ifndef OSM_VENDOR_INTF_ANAFA ./management/osm/include/vendor/osm_vendor_ts.h:139: * Points to the MAD pool used by OSM ./management/osm/include/vendor/osm_vendor_ts.h:160: * ./management/osm/include/vendor/osm_vendor_ts.h:220: * Handle returned by TS open call . ./management/osm/include/vendor/osm_vendor_ts.h:314: * Given the vendor obj and a guid ./management/osm/include/vendor/osm_vendor_ts.h:354: * Fill in the array of port_attr with all available ports on ALL the ./management/osm/include/vendor/osm_vendor_ts.h:356: * ALSO - ./management/osm/include/vendor/osm_vendor_mtl_hca_guid.h:40: * Provides interface over VAPI for obtaining the local ports guids or from guid ./management/osm/include/vendor/osm_vendor_mtl_hca_guid.h:134: * Given the vendor obj and a guid ./management/osm/include/vendor/osm_vendor_mtl_hca_guid.h:173: * Fill in the array of port_attr with all available ports on ALL the ./management/osm/include/vendor/osm_vendor_mtl_hca_guid.h:175: * ALSO - ./management/osm/include/vendor/osm_vendor_umadt.h:75:* ./management/osm/include/vendor/osm_vendor_umadt.h:104:* Umadt APIs. ./management/osm/include/vendor/osm_vendor_umadt.h:131:* Umadt mad structure to identify a mad. ./management/osm/include/vendor/osm_vendor_umadt.h:140: ./management/osm/include/vendor/osm_vendor_ibumad.h:68:* ./management/osm/include/vendor/osm_vendor_mlx_sar.h:54: void* p_arbt_mad; ./management/osm/include/vendor/osm_vendor_mlx_sar.h:58: uint32_t data_sz; /*typical data sz for this kind of mad (sa or regular */ ./management/osm/include/vendor/osm_vendor_mlx_sar.h:59: ./management/osm/include/vendor/osm_vendor_mlx_sar.h:71: * ./management/osm/include/vendor/osm_vendor_mlx_sar.h:74:osmv_rmpp_sar_init(osmv_rmpp_sar_t* p_sar, void* p_arbt_mad, ./management/osm/include/vendor/osm_vendor_mlx_sar.h:85: * ./management/osm/include/vendor/osm_vendor_mlx_sar.h:95: * segments the original mad buffer . returnes a mad with the data of the i-th segment ./management/osm/include/vendor/osm_vendor_mlx_sar.h:98: * ./management/osm/include/vendor/osm_vendor_mlx_sar.h:108: * gets a qlist of mads and reassmbles to one big mad buffer ./management/osm/include/vendor/osm_vendor_mlx_sar.h:112: * ./management/osm/include/vendor/osm_vendor_mlx_sar.h:114:ib_api_status_t ./management/osm/include/vendor/osm_vendor_mlx_transport_anafa.h:40: * ./management/osm/include/vendor/osm_vendor_mlx_transport_anafa.h:43: * that allows sending a single MAD/receiving a callback ./management/osm/include/vendor/osm_vendor_mlx_transport_anafa.h:44: * when a single MAD is received. ./management/osm/include/vendor/osm_vendor_mlx_txn.h:84: osmv_rmpp_txfr_t rmpp_txfr; ./management/osm/include/vendor/osm_vendor_mlx_txn.h:91: ./management/osm/include/vendor/osm_vendor_mlx_txn.h:103: /* Reference to a log to enable tracing */ ./management/osm/include/vendor/osm_vendor_mlx_txn.h:105: ./management/osm/include/vendor/osm_vendor_mlx_txn.h:115: * allocs & inits the osmv_txn_ctx obj and insert it into the db ./management/osm/include/vendor/osm_vendor_mlx_txn.h:117: * ./management/osm/include/vendor/osm_vendor_mlx_txn.h:131: * init the rmpp send ctx in the transaction ./management/osm/include/vendor/osm_vendor_mlx_txn.h:147: * init the rmpp recv ctx in the transaction ./management/osm/include/vendor/osm_vendor_mlx_txn.h:162: * destroys txn object and removes it from the db ./management/osm/include/vendor/osm_vendor_mlx_txn.h:176: * returns tid of the transaction ./management/osm/include/vendor/osm_vendor_mlx_txn.h:178: * ./management/osm/include/vendor/osm_vendor_mlx_txn.h:180:static inline uint64_t ./management/osm/include/vendor/osm_vendor_mlx_txn.h:192: * returns key of the transaction ./management/osm/include/vendor/osm_vendor_mlx_txn.h:194: * ./management/osm/include/vendor/osm_vendor_mlx_txn.h:197:static inline uint64_t ./management/osm/include/vendor/osm_vendor_mlx_txn.h:208: * returns whether the rmpp txfr was init by the peer ./management/osm/include/vendor/osm_vendor_mlx_txn.h:225: * returns osmv_rmpp_send_ctx obj ./management/osm/include/vendor/osm_vendor_mlx_txn.h:227: * ./management/osm/include/vendor/osm_vendor_mlx_txn.h:229:static inline osmv_rmpp_send_ctx_t* ./management/osm/include/vendor/osm_vendor_mlx_txn.h:241: * returns osmv_rmpp_recv_ctx obj ./management/osm/include/vendor/osm_vendor_mlx_txn.h:243: * ./management/osm/include/vendor/osm_vendor_mlx_txn.h:245:static inline osmv_rmpp_recv_ctx_t* ./management/osm/include/vendor/osm_vendor_mlx_txn.h:257: * returns the rmpp role of the transactino ( send/ recv) ./management/osm/include/vendor/osm_vendor_mlx_txn.h:259: * ./management/osm/include/vendor/osm_vendor_mlx_txn.h:273: * sets the rmpp role of the transaction (send/ recv) ./management/osm/include/vendor/osm_vendor_mlx_txn.h:275: * ./management/osm/include/vendor/osm_vendor_mlx_txn.h:292: * ./management/osm/include/vendor/osm_vendor_mlx_txn.h:308: * ./management/osm/include/vendor/osm_vendor_mlx_txn.h:320: * osmv_txn_set_timeout_ev ./management/osm/include/vendor/osm_vendor_mlx_txn.h:325: * ./management/osm/include/vendor/osm_vendor_mlx_txn.h:333: * osmv_txn_remove_timeout_ev ./management/osm/include/vendor/osm_vendor_mlx_txn.h:338: * ./management/osm/include/vendor/osm_vendor_mlx_txn.h:345: * osmv_txn_lookup ./management/osm/include/vendor/osm_vendor_mlx_txn.h:351: * ./management/osm/include/vendor/osm_vendor_mlx_txn.h:358:void ./management/osm/include/vendor/osm_vendor_mlx_txn.h:366: * osmv_txnmgr_init ./management/osm/include/vendor/osm_vendor_mlx_txn.h:369: * c'tor for txn mgr obj ./management/osm/include/vendor/osm_vendor_mlx_txn.h:371: * ./management/osm/include/vendor/osm_vendor_mlx_txn.h:380: * osmv_txnmgr_done ./management/osm/include/vendor/osm_vendor_mlx_txn.h:383: * c'tor for txn mgr obj ./management/osm/include/vendor/osm_vendor_mlx_txn.h:385: * ./management/osm/include/vendor/osm_vendor_sa_api.h:148:* Query for port information based on the port's LID and port num. ./management/osm/include/vendor/osm_vendor_sa_api.h:375:* osmv_query_sa, osmv_pfn_query_cb_t, ib_api_status_t, ./management/osm/include/vendor/osm_vendor_sa_api.h:387:* osmv_query_sa(). ./management/osm/include/vendor/osm_vendor_sa_api.h:712:* ./management/osm/include/vendor/osm_vendor_sa_api.h:716:osm_bind_handle_t ./management/osm/include/vendor/osm_vendor_sa_api.h:732:* ./management/osm/include/vendor/osm_vendor_sa_api.h:753:osmv_query_sa( ./management/osm/include/vendor/osm_vendor_sa_api.h:755: IN const osmv_query_req_t * const p_query_req ./management/osm/include/vendor/osm_vendor_sa_api.h:760:* [in] bind handle for this port. Should be previously ./management/osm/include/vendor/osm_vendor_sa_api.h:765:* ./management/osm/include/vendor/osm_vendor_sa_api.h:767:* IB_SUCCESS if completed successfuly (or in ASYNC mode ./management/osm/include/vendor/osm_vendor_mlx_rmpp_ctx.h:58: ./management/osm/include/vendor/osm_vendor_mlx_rmpp_ctx.h:70: ./management/osm/include/vendor/osm_vendor_mlx_rmpp_ctx.h:79: ./management/osm/include/vendor/osm_vendor_mlx_rmpp_ctx.h:99: * ./management/osm/include/vendor/osm_vendor_mlx_rmpp_ctx.h:114: * ./management/osm/include/vendor/osm_vendor_mlx_rmpp_ctx.h:127: * ./management/osm/include/vendor/osm_vendor_mlx_rmpp_ctx.h:129:static inline uint32_t ./management/osm/include/vendor/osm_vendor_mlx_rmpp_ctx.h:143: * ./management/osm/include/vendor/osm_vendor_mlx_rmpp_ctx.h:159: * ./management/osm/include/vendor/osm_vendor_mlx_rmpp_ctx.h:161:static inline uint32_t ./management/osm/include/vendor/osm_vendor_mlx_rmpp_ctx.h:175: * ./management/osm/include/vendor/osm_vendor_mlx_rmpp_ctx.h:189: * returns the total number of mad segments to send ./management/osm/include/vendor/osm_vendor_mlx_rmpp_ctx.h:191: * ./management/osm/include/vendor/osm_vendor_mlx_rmpp_ctx.h:193:uint32_t ./management/osm/include/vendor/osm_vendor_mlx_rmpp_ctx.h:201: * Retrieves the mad segment by seg number (including setting the mad relevant bits & hdrs) ./management/osm/include/vendor/osm_vendor_mlx_rmpp_ctx.h:203: * ./management/osm/include/vendor/osm_vendor_mlx_rmpp_ctx.h:220: * ./management/osm/include/vendor/osm_vendor_mlx_rmpp_ctx.h:232: * ./management/osm/include/vendor/osm_vendor_mlx_rmpp_ctx.h:243: * ./management/osm/include/vendor/osm_vendor_mlx_rmpp_ctx.h:258: * ./management/osm/include/vendor/osm_vendor_mlx_rmpp_ctx.h:274: * ./management/osm/include/vendor/osm_vendor_mlx_rmpp_ctx.h:281:uint32_t ./management/osm/include/vendor/osm_vendor_mlx_rmpp_ctx.h:284:uint32_t ./management/osm/include/vendor/osm_vendor_mlx_rmpp_ctx.h:287:uint32_t ./management/osm/include/vendor/osm_vendor_mlx_rmpp_ctx.h:297: * reassembles all rmpp buffs to one big arbitrary mad ./management/osm/include/vendor/osm_vendor_al.h:88:* ./management/osm/include/vendor/osm_vendor_al.h:291:* Handle returned by AL open call (ib_open_al). ./management/osm/include/vendor/osm_vendor_api.h:108:* a response are completed in error, most likely due to a ./management/osm/include/vendor/osm_vendor_api.h:173:* ./management/osm/include/vendor/osm_vendor_api.h:243: IN const uint32_t timeout ./management/osm/include/vendor/osm_vendor_api.h:260:* ./management/osm/include/vendor/osm_vendor_api.h:295:* [in] pointer to a mad wrappers pool to be used for allocating ./management/osm/include/vendor/osm_vendor_api.h:347:* Obtain a mad wrapper holding actual mad buffer to be sent via ./management/osm/include/vendor/osm_vendor_api.h:399:* ./management/osm/include/vendor/osm_vendor_api.h:402:* ./management/osm/include/vendor/osm_vendor_api.h:489:* [in] If TRUE - will set the is_sm to TRUE, if FALSE - will set the ./management/osm/include/opensm/osm_log.h:239: if (cl_spinlock_init( &p_log->lock ) == CL_SUCCESS) ./management/osm/include/opensm/osm_port_info_rcv.h:266: ./management/osm/include/opensm/osm_vl_arb_rcv_ctrl.h:41: * This object represents a controller that set or get resp the ./management/osm/include/opensm/cl_dispatcher.h:42: * ./management/osm/include/opensm/cl_dispatcher.h:150:typedef void ./management/osm/include/opensm/cl_dispatcher.h:191:typedef void ./management/osm/include/opensm/cl_dispatcher.h:266:* ./management/osm/include/opensm/cl_dispatcher.h:366:* ./management/osm/include/opensm/cl_dispatcher.h:449:* Dispatcher, cl_disp_destoy, cl_disp_register, cl_disp_unregister, ./management/osm/include/opensm/cl_dispatcher.h:600:* ./management/osm/include/opensm/osm_port_profile.h:119:* ./management/osm/include/opensm/osm_port_profile.h:149:* ./management/osm/include/opensm/osm_port_profile.h:214: IN const osm_subn_t *p_subn, ./management/osm/include/opensm/osm_port_profile.h:232:* port_guid ./management/osm/include/opensm/osm_port_profile.h:233:* [in] The port guid ./management/osm/include/opensm/osm_port_profile.h:254: IN osm_subn_t *p_subn, ./management/osm/include/opensm/osm_port_profile.h:262: ./management/osm/include/opensm/osm_port_profile.h:265: } ./management/osm/include/opensm/osm_port_profile.h:266: ./management/osm/include/opensm/osm_port_profile.h:277:* port_guid ./management/osm/include/opensm/osm_port_profile.h:278:* [in] The port guid ./management/osm/include/opensm/osm_ft_config_ctrl.h:41: * This object represents a controller that performs a ./management/osm/include/opensm/osm_base.h:97:* The version String for OpenSM ./management/osm/include/opensm/osm_base.h:145:* Default Maximal Operational VLs to be initialized on ./management/osm/include/opensm/osm_base.h:288:* Specifies the default number of VL15 SMP MADs allowed on ./management/osm/include/opensm/osm_base.h:400: * OSM_CAP ARE C15-0.1.7 Table 152 ./management/osm/include/opensm/osm_base.h:421:* Multicast is supported ./management/osm/include/opensm/osm_base.h:465: OSM_SM_STATE_IDLE, ./management/osm/include/opensm/osm_base.h:536:* Enumerates the possible state progressing codes used by the OSM ./management/osm/include/opensm/osm_base.h:548:* OSM_STATE_STEP_CONTINUOUS ./management/osm/include/opensm/osm_base.h:551:* OSM_STATE_STEP_TAKE_ONE ./management/osm/include/opensm/osm_base.h:552:* Do one step ./management/osm/include/opensm/osm_base.h:555:* Stop before taking next step (the while loop in the state ./management/osm/include/opensm/osm_mcast_mgr.h:383: ./management/osm/include/opensm/osm_mtree.h:340:* ./management/osm/include/opensm/osm_mtree.h:372:* ./management/osm/include/opensm/osm_vl15intf.h:172:* ./management/osm/include/opensm/osm_pkt_randomizer.h:78:* The Packet Randomizer object encapsulates the information needed for ./management/osm/include/opensm/osm_pkt_randomizer.h:94:* Packet randomizer sturcture. This structure contains the various ./management/osm/include/opensm/osm_pkt_randomizer.h:118:* The number of unstable links to be drawn. ./management/osm/include/opensm/osm_pkt_randomizer.h:127:* Array of osm_dr_path_t objects, that includes all the dr_paths ./management/osm/include/opensm/osm_pkt_randomizer.h:131:* Describes the number of paths from the fault_dr_paths array that ./management/osm/include/opensm/osm_pkt_randomizer.h:205:* randomizer. ./management/osm/include/opensm/osm_service.h:76:* SA to manage service registrations. ./management/osm/include/opensm/osm_service.h:108: ./management/osm/include/opensm/osm_service.h:123:* ./management/osm/include/opensm/osm_sa_lft_record_ctrl.h:231:* Linear Forwarding Table Receive Controller object, ./management/osm/include/opensm/osm_vl_arb_rcv.h:265: ./management/osm/include/opensm/osm_sa_portinfo_record.h:136:* ./management/osm/include/opensm/osm_multicast.h:195:* during the initialization of SM/SA and will be present even if ./management/osm/include/opensm/osm_errors.h:132: ./management/osm/include/opensm/osm_errors.h:147: ./management/osm/include/opensm/osm_errors.h:150: ./management/osm/include/opensm/osm_lid_mgr.h:280: ./management/osm/include/opensm/osm_mtl_bind.h:133:* Points to the MAD pool used by OSM ./management/osm/include/opensm/osm_sa_lft_record.h:149:* Linear Forwarding Table Receiver object, osm_lftr_rcv_init, ./management/osm/include/opensm/osm_sa_lft_record.h:227:* Linear Forwarding Table Receiver object, osm_lftr_rcv_construct, ./management/osm/include/opensm/osm_sa_slvl_record.h:136:* ./management/osm/include/opensm/osm_sa_sminfo_record.h:270:#define SMIR_ACTCOUNT_COMPMASK 0x10 ./management/osm/include/opensm/osm_slvl_map_rcv.h:265: ./management/osm/include/opensm/osm_mcast_tbl.h:390:* Returns true if there are more blocks necessary to ./management/osm/include/opensm/osm_sa_informinfo_ctrl.h:107: ./management/osm/include/opensm/osm_switch.h:617:* ./management/osm/include/opensm/osm_switch.h:644: /* ./management/osm/include/opensm/osm_switch.h:647: unreachable through a switch (we put a OSM_NO_PATH value at ./management/osm/include/opensm/osm_switch.h:649: at the fwd table but return NULL ./management/osm/include/opensm/osm_switch.h:670:* ./management/osm/include/opensm/osm_switch.h:699:* ./management/osm/include/opensm/osm_switch.h:729:* ./management/osm/include/opensm/osm_switch.h:760:* ./management/osm/include/opensm/osm_switch.h:789:* ./management/osm/include/opensm/osm_switch.h:881:* Returns true if there are more blocks necessary to ./management/osm/include/opensm/osm_switch.h:1080:* In cases where LMC > 0, the remote side system and node ./management/osm/include/opensm/osm_switch.h:1081:* used for the routing are tracked in the provided arrays ./management/osm/include/opensm/osm_switch.h:1114:* [in out] The array of remote system guids already used to route ./management/osm/include/opensm/osm_switch.h:1116:* ./management/osm/include/opensm/osm_switch.h:1121:* [in out] The array of remote node guids already used to route ./management/osm/include/opensm/osm_switch.h:1123:* ./management/osm/include/opensm/osm_switch.h:1128:* [in] The maximum allowed number of target lids routed through ./management/osm/include/opensm/osm_switch.h:1129:* a specific port of the switch. If the port already assigned ./management/osm/include/opensm/osm_switch.h:1137:* If FALSE - such function isn't defined. Do the minimum hop checks ./management/osm/include/opensm/osm_sm_info_get_ctrl.h:294: ./management/osm/include/opensm/osm_pi_config_ctrl.h:41: * This object represents a controller that performs a ./management/osm/include/opensm/osm_pi_config_ctrl.h:286: ./management/osm/include/opensm/osm_inform.h:82:* when SM receives Traps for registered LIDs. ./management/osm/include/opensm/osm_inform.h:224:* Find a matching osm_infr_t in the subnet DB by inform_info_record RID ./management/osm/include/opensm/osm_inform.h:235:* p_subn ./management/osm/include/opensm/osm_inform.h:266:* p_subn ./management/osm/include/opensm/osm_inform.h:273:* [in] Pointer to an inform_info record ./management/osm/include/opensm/osm_inform.h:298:* Once a Trap was received by the osm_trap_rcv, or a Trap sourced in ./management/osm/include/opensm/osm_inform.h:304:* ./management/osm/include/opensm/osm_node.h:303:* in which case all the Physical Ports share the same GUID. ./management/osm/include/opensm/osm_node.h:323:* in which case all the Physical Ports share the same GUID. ./management/osm/include/opensm/osm_attrib_req.h:71:* The Attribute Request structure encapsulates ./management/osm/include/opensm/cl_event_wheel.h:42: * ./management/osm/include/opensm/cl_event_wheel.h:156: ./management/osm/include/opensm/cl_event_wheel.h:175:* closing ./management/osm/include/opensm/cl_event_wheel.h:217:* map_item ./management/osm/include/opensm/cl_event_wheel.h:218:* The map item of this event ./management/osm/include/opensm/cl_event_wheel.h:300:* ./management/osm/include/opensm/cl_event_wheel.h:335:* ./management/osm/include/opensm/cl_event_wheel.h:389:* It doesn't aquire the lock by itself. ./management/osm/include/opensm/cl_event_wheel.h:463:* After the event has aged it is automatically removed from ./management/osm/include/opensm/cl_event_wheel.h:480:uint32_t ./management/osm/include/opensm/osm_trap_rcv.h:136:* Basically we can start a timer every time we receive a specific ./management/osm/include/opensm/osm_pkey_config_ctrl.h:41: * This object represents a controller that performs a ./management/osm/include/opensm/osm_pkey_config_ctrl.h:286: ./management/osm/include/opensm/osm_sa_mcmember_record.h:296: IN const ib_member_rec_t* const p_recvd_mcmember_rec, ./management/osm/include/opensm/osm_sa_mcmember_record.h:307:* ./management/osm/include/opensm/osm_sa_mcmember_record.h:315:* ./management/osm/include/opensm/osm_sa_mcmember_record.h:329: IB_MCR_COMPMASK_SL) ./management/osm/include/opensm/osm_port.h:150:* Tracks the health of the port. Normally should be TRUE but ./management/osm/include/opensm/osm_port.h:176:* want to send the request anyways - since every we need at least one ./management/osm/include/opensm/osm_port.h:349:* All physical ports are initialized as "healthy" but may be marked ./management/osm/include/opensm/osm_port.h:364:* and FALSE otherwise. Link is healthy if both its physical ports are ./management/osm/include/opensm/osm_port.h:379:* All physical ports are initialized as "healthy" but may be marked ./management/osm/include/opensm/osm_port.h:393:* Sets the port health flag. TRUE means the port is healthy and ./management/osm/include/opensm/osm_port.h:398:static inline void ./management/osm/include/opensm/osm_port.h:417:* NONE ./management/osm/include/opensm/osm_port.h:469:void ./management/osm/include/opensm/osm_port.h:552: IN ib_slvl_table_t *p_slvl_tbl, ./management/osm/include/opensm/osm_port.h:625: IN ib_vl_arb_table_t *p_vla_tbl, ./management/osm/include/opensm/osm_port.h:1928:* Calculate the Port MTU based on current and remote ./management/osm/include/opensm/osm_port.h:1959:* Calculate the Port OP_VLS based on current and remote ./management/osm/include/opensm/osm_ucast_updn.h:44: * Implementation of Up Down Algorithm using ranking & Min Hop ./management/osm/include/opensm/osm_ucast_updn.h:58:#include ./management/osm/include/opensm/osm_ucast_updn.h:90: * UP ./management/osm/include/opensm/osm_ucast_updn.h:92: * DOWN ./management/osm/include/opensm/osm_ucast_updn.h:176:* The number of occurences of the same hop value ./management/osm/include/opensm/osm_ucast_updn.h:188:* ./management/osm/include/opensm/osm_ucast_updn.h:208:* guids specified as an array (converted from a list given in the UI) ./management/osm/include/opensm/osm_ucast_updn.h:212:* ./management/osm/include/opensm/osm_ucast_updn.h:218:* ./management/osm/include/opensm/osm_ucast_updn.h:238:* state of the updn algorithm which basically should pass through Init ./management/osm/include/opensm/osm_ucast_updn.h:246:* ./management/osm/include/opensm/osm_ucast_updn.h:298:* ./management/osm/include/opensm/osm_ucast_updn.h:326:* ./management/osm/include/opensm/osm_ucast_updn.h:328:* ./management/osm/include/opensm/osm_ucast_updn.h:426:* osm_updn_reg_calc_min_hop_table ./management/osm/include/opensm/osm_ucast_updn.h:429:* Registration function to ucast routing manager (instead of ./management/osm/include/opensm/osm_ucast_updn.h:430:* Min Hop Algorithm) ./management/osm/include/opensm/osm_ucast_updn.h:465:* ./management/osm/include/opensm/osm_mcast_config_ctrl.h:41: * This object represents a controller that performs a ./management/osm/include/opensm/osm_mcast_config_ctrl.h:288: ./management/osm/include/opensm/osm_sm_state_mgr.h:293: ./management/osm/include/opensm/osm_sm_state_mgr.h:327:* Checks the legality of the signal received, according to the ./management/osm/include/opensm/osm_remote_sm.h:202:* ./management/osm/include/opensm/osm_link_mgr.h:271: ./management/osm/include/opensm/osm_node_info_rcv.h:305: ./management/osm/include/opensm/osm_sa.h:107:* Anil Keshavamurthy, Intel ./management/osm/include/opensm/osm_sa.h:168: ./management/osm/include/opensm/osm_sa.h:180: ./management/osm/include/opensm/osm_sa.h:188: ./management/osm/include/opensm/osm_sa.h:424:* Adds the well known Multicast group to the SA database. This ./management/osm/include/opensm/osm_sa.h:459:* Creates the well known MC record and calls osm_sa_add_well_known_mc_record. This ./management/osm/include/opensm/osm_msgdef.h:93:* ./management/osm/include/opensm/osm_msgdef.h:107:* ./management/osm/include/opensm/osm_msgdef.h:121:* ./management/osm/include/opensm/osm_msgdef.h:135:* ./management/osm/include/opensm/osm_msgdef.h:148:* ./management/osm/include/opensm/osm_msgdef.h:169: OSM_MSG_MAD_PORT_INFO, ./management/osm/include/opensm/osm_umadt.h:99: cl_spinlock_t trans_ctxt_lock; ./management/osm/include/opensm/osm_umadt.h:100: cl_qlist_t trans_ctxt_list; ./management/osm/include/opensm/osm_umadt.h:103: cl_qlist_t timeout_list; ./management/osm/include/opensm/osm_umadt.h:104: RegisterClassStruct umadt_reg_class; ./management/osm/include/opensm/osm_umadt.h:122:* Pointer to the MAD pool to be used by mads with this bind handle. ./management/osm/include/opensm/osm_ucast_mgr.h:274: ./management/osm/include/opensm/osm_state_mgr.h:197:* Idle item. ./management/osm/include/opensm/osm_state_mgr.h:218: osm_pfn_done_t pfn_done; ./management/osm/include/opensm/osm_state_mgr.h:246:* Formulates the osm_idle_item and inserts it into the queue and ./management/osm/include/opensm/osm_state_mgr.h:275:* [in] Pointer to void ./management/osm/include/opensm/osm_state_mgr.h:276:* ./management/osm/include/opensm/osm_state_mgr.h:278:* [in] Pointer to void ./management/osm/include/opensm/osm_state_mgr.h:527: ./management/osm/include/opensm/osm_helper.h:258:osm_dump_pkey_table( ./management/osm/include/opensm/osm_helper.h:260: IN uint64_t port_guid, ./management/osm/include/opensm/osm_helper.h:267:osm_dump_slvl_map_table( ./management/osm/include/opensm/osm_helper.h:269: IN uint64_t port_guid, ./management/osm/include/opensm/osm_helper.h:276:osm_dump_vl_arb_table( ./management/osm/include/opensm/osm_helper.h:278: IN uint64_t port_guid, ./management/osm/include/opensm/osm_helper.h:442: IN const ib_smp_t* const p_smp, ./management/osm/include/opensm/osm_sa_vlarb_record.h:136:* ./management/osm/include/opensm/osm_sm.h:448:* [in] Type of the MC request that caused this join ./management/osm/include/opensm/osm_sm.h:517:* wait_us ./management/osm/include/opensm/osm_sm.h:525:* CL_SUCCESS if the wait operation succeeded in response to the event ./management/osm/include/opensm/osm_pkey.h:83:* In the following, let M_P_Key(Message P_Key) be the P_Key in the incoming ./management/osm/include/opensm/osm_pkey.h:95:* then the P_Keys are said to match. ./management/osm/include/opensm/osm_pkey.h:131:* In the following, let M_P_Key(Message P_Key) be the P_Key in the incoming ./management/osm/include/opensm/osm_pkey.h:143:* then the P_Keys are said to match. ./management/osm/include/opensm/osm_pkey.h:179:* In the following, let M_P_Key(Message P_Key) be the P_Key in the incoming ./management/osm/include/opensm/osm_pkey.h:191:* then the P_Keys are said to match. ./management/osm/include/opensm/osm_pkey.h:239:* In the following, let M_P_Key(Message P_Key) be the P_Key in the incoming ./management/osm/include/opensm/osm_pkey.h:251:* then the P_Keys are said to match. ./management/osm/include/opensm/osm_pkey.h:289: IN osm_log_t *p_log, ./management/osm/include/opensm/osm_pkey.h:290: IN osm_req_t *p_req, ./management/osm/include/opensm/osm_madw.h:93:* PortGuid of local port ./management/osm/include/opensm/osm_madw.h:96:* Mgmt Class ID ./management/osm/include/opensm/osm_madw.h:106:* ./management/osm/include/opensm/osm_madw.h:115:* ./management/osm/include/opensm/osm_madw.h:165:* ./management/osm/include/opensm/osm_madw.h:358: ib_net16_t source_lid; ./management/osm/include/opensm/osm_madw.h:359: uint8_t port_num; ./management/osm/include/opensm/osm_madw.h:364: ib_net32_t remote_qp; ./management/osm/include/opensm/osm_madw.h:365: ib_net32_t remote_qkey; ./management/osm/include/opensm/osm_madw.h:367: uint8_t service_level; ./management/osm/include/opensm/osm_subnet_config_ctrl.h:287: ./management/osm/include/opensm/osm_opensm.h:265:* [in] Pointer to an osm_opensm_t object on which to ./management/osm/include/opensm/osm_opensm.h:362:* wait_us ./management/osm/include/opensm/osm_opensm.h:370:* CL_SUCCESS if the wait operation succeeded in response to the event ./management/osm/include/opensm/osm_opensm.h:384:/****v* OpenSM/osm_exit_flag ./management/osm/include/opensm/osm_opensm.h:387:/* ./management/osm/include/opensm/osm_opensm.h:388:* DESCRIPTION ./management/osm/include/opensm/osm_opensm.h:389:* Set to one to cause all threads to leave ./management/osm/include/opensm/osm_sa_node_record.h:134:* ./management/osm/include/opensm/osm_mcm_port.h:101:* If TRUE - Join was performed on behalf of the endport identified ./management/osm/include/iba/ib_types.h:535:* Indicitates if the Class Code if a vendor specific class from ./management/osm/include/iba/ib_types.h:553:* TRUE if the class is in the Low range of Vendor Specific MADs ./management/osm/include/iba/ib_types.h:567:* Indicitates if the Class Code if a vendor specific class from ./management/osm/include/iba/ib_types.h:585:* TRUE if the class is in the High range of Vendor Specific MADs ./management/osm/include/iba/ib_types.h:2040: uint8_t num_path; ./management/osm/include/iba/ib_types.h:4447:* Sets the Head of Queue Lifetime for which a packet can live in the head ./management/osm/include/iba/ib_types.h:4481:* Gets the Head of Queue Lifetime for which a packet can live in the head ./management/osm/include/iba/ib_types.h:4732: p_pi->error_threshold = ./management/osm/include/iba/ib_types.h:4741:* [in] Physical Link Errors Threshold above which Trap 129 is generated ./management/osm/include/iba/ib_types.h:4744:* [in] Credits overrun Errors Threshold above which Trap 129 is generated ./management/osm/include/iba/ib_types.h:5638: ./management/osm/include/iba/ib_types.h:5647: ./management/osm/include/iba/ib_types.h:5655: ./management/osm/include/iba/ib_types.h:5658: ./management/osm/include/iba/ib_types.h:5675: ./management/osm/include/iba/ib_types.h:5681: ./management/osm/include/iba/ib_types.h:5946: ./management/osm/include/iba/ib_types.h:5955: ./management/osm/include/iba/ib_types.h:5957: ./management/osm/include/iba/ib_types.h:5980: ./management/osm/include/iba/ib_types.h:6017: p_ii->g_or_v.generic.qpn_resp_time_val = ./management/osm/include/complib/cl_atomic_osd.h:72: ./management/osm/include/complib/cl_atomic_osd.h:73: if (spinlock.state != CL_INITIALIZED) ./management/osm/include/complib/cl_atomic_osd.h:93: ./management/osm/include/complib/cl_atomic_osd.h:94: if (spinlock.state != CL_INITIALIZED) ./management/osm/include/complib/cl_atomic_osd.h:113: ./management/osm/include/complib/cl_atomic_osd.h:115: ./management/osm/include/complib/cl_atomic_osd.h:116: if (spinlock.state != CL_INITIALIZED) ./management/osm/include/complib/cl_atomic_osd.h:137: ./management/osm/include/complib/cl_atomic_osd.h:138: if (spinlock.state != CL_INITIALIZED) ./management/osm/include/complib/cl_obj.h:189:cl_obj_mgr_create(void); ./management/osm/include/complib/cl_memory.h:348: IN const int32_t line_num, ./management/osm/include/complib/cl_debug_osd.h:81:#endif From 6th_yuki_ml_service at docomo.ne.jp Thu May 5 18:26:18 2005 From: 6th_yuki_ml_service at docomo.ne.jp (6th_yuki_ml_service at docomo.ne.jp) Date: Fri, 6 May 2005 10:26:18 +0900 Subject: [openib-general] =?utf-8?b?wpFTwojDtcKCwqrCgXnCkGzCjcOIwoF6woo=?= =?utf-8?b?w7nCjcKlwo/Cl8KQwqvCj2/CicOvwoLCosKDVMKBW8KDTsKDwovCgsOM?= =?utf-8?b?woLCssKIw4TCk8Og?= Message-ID: 20050506102330 皆様、突然のメールお許し下さい。 私は、既婚女性のための出会いサークル「Dear Again」の代表をしております吉田加奈と申します。 本日は、男性の皆様へご案内がございまして、メールをお送りさせていただきます。 当サークルは、平成２年に金銭的に恵まれている既婚女性数名が「主婦の自立サークル」として設立され、主に、既婚女性特有の満たされない欲求を満たす目的に活動を開始しました。 その後、平成１５年に私、吉田加奈が代表を引き継ぎ、サークル名も現在の「Dear Again」とし、設立当初の目的を今も引き継いで活動を行っております。 現在の主な会員さんは、上場企業や官公庁にお勤めのご主人を持つ主婦やご自分で事業をされている方などを中心にやはり金銭的に余裕のある方です。 今回、募集させていただくのはこのような女性の方々のお相手をしていただける１８歳以上の男性の方々です。 既婚女性という立場のある方々ですので、堂々と宣伝を行う事ができず、このような形でのご案内となっております。 ご希望の方がおられましたら、下記のフォームからご応募いただくか、専用ホームページのフォームをご利用の上、ご応募いただきたいと思います。 現在、会員さんは全国に１０５４名（４月１０日現在）在籍しておりますので、お住まいの近くで、ご希望条件に合う会員をご紹介させていただきます。 最後まで、ご高読ありがとうございました。 　　既婚女性のための出会いサークル 　　　　　　　　　　　「Dear Again」　代表　吉田加奈 　　 **************************************************** ご応募にあたり送信していただきたい内容 「お名前」　（ハンドルネーム・通称名等で結構です） 「居住地域」（都道府県名は必ずご記入下さい） 「希望する女性のタイプ」 次のホームページから送信いただけます http://www.dear-again.com/x13/ **************************************************** //////////////////////////////////////////////////// 会員女性の一部をご紹介いたします。 ※ホームページでは下記女性の顔写真付き詳しいプロフィールをご紹介しております。 ホームページアドレス http://www.dear-again.com/x13/ //////////////////////////////////////////////////// 涼子　　31歳　160cm　47Kg 　　　　主人との営みが淡白すぎて耐え・・・続きはHPで 美智江　42歳　156cm　50Kg 　　　　10年も結婚生活を送っていると・・・続きはHPで 綾乃　　26歳　163cm　49Kg 　　　　はっきり言って結婚は早まった・・・続きはHPで 紀美子　45歳　154cm　42Kg 　　　　こんな歳になってもまだ性欲が・・・続きはHPで 美佐江　35歳　159cm　51Kg 　　　　そろそろ倦怠期に入ってしまう・・・続きはHPで 亜佐美　28歳　170cm　53Kg 　　　　私たち夫婦はセックスレスなん・・・続きはHPで 美子　　22歳　155cm　50Kg 　　　　私は浮気されています。結婚す・・・続きはHPで 由紀子　51歳　160cm　54Kg 　　　　結婚したのが遅かったせいかす・・・続きはHPで 奈保美　24歳　168cm　53Kg 　　　　少し太ってきたせいなのか最近・・・続きはHPで 博美　　35歳　151cm　43Kg 　　　　身長が小さいせいで主人と並ん・・・続きはHPで 真奈美　31歳　163cm　50Kg 　　　　結婚が早かったんです。親の反・・・続きはHPで //////////////////////////////////////////////////// ホームページアドレス http://www.dear-again.com/x13/ //////////////////////////////////////////////////// 配信代行者　ユキＭＬサービス 配信拒否はこちらのアドレスまで、お申し付けください 配信拒否 → http://www.mail-kyohi.com/x13/ From jcarr at linuxmachines.com Thu May 5 18:36:25 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Thu, 05 May 2005 18:36:25 -0700 Subject: [openib-general] in need of a simple ulp In-Reply-To: <427AC385.5070308@ichips.intel.com> References: <426FCF0A.3070806@linuxmachines.com> <426FD098.9090300@ichips.intel.com> <427AC218.9070307@linuxmachines.com> <427AC385.5070308@ichips.intel.com> Message-ID: <427ACA19.7010708@linuxmachines.com> Sean Hefty wrote: > In theory, yes. Though I've never verified the data. Note that the > code uses a single data buffer for the sends and receives, so it can get > overwritten. Yep, noticed that. I was going to set about changing it to add a few more (I don't need that many) but then noticed that nothing was changing. > You're looking in the right place. Good to know. As far as I can tell, neither buffer on the client or server changes between the time it's kmalloc'd to the time it goes to destroy it after poll_cqs() is finished. Jeff From roland at topspin.com Thu May 5 19:26:08 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 05 May 2005 19:26:08 -0700 Subject: [openib-general] [patch] misc trivia nitpicks 01 In-Reply-To: <20050505170956.GU15688@aon.at> (Bernhard Fischer's message of "Thu, 5 May 2005 19:09:56 +0200") References: <20050505170956.GU15688@aon.at> Message-ID: <523bt15bun.fsf@topspin.com> Have you tested this patch? For example this change in libmthca/src/mthca.c: - sscanf(attr->value, "%i", &vendor); + sscanf(attr->value, "%u", &vendor); seems like it will break things, since this sscanf is reading the vendor attribute of a PCI device from sysfs, and that will be a string like "0x15b3" with current kernels. - R. From halr at voltaire.com Thu May 5 22:57:15 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 06 May 2005 01:57:15 -0400 Subject: [openib-general] [PATCH] kDAPL: remove DAT_UINT32 and DAT_UIN T64 In-Reply-To: <1115332746.20452.2.camel@duffman> References: <91DB792C7985D411BEC300B40080D29CC35A4E@mtvex01.mtv.mtl.com> <1115332746.20452.2.camel@duffman> Message-ID: <1115358822.4476.13.camel@localhost.localdomain> On Thu, 2005-05-05 at 18:39, Tom Duffy wrote: > kDAPL: DAPL: Started (dapl_init) > kDAPL: Inside dapl_add_one called by mthca0 > KDAPL : Added device mthca0a to the registry > KDAPL : Added device mthca0b to the registry > Kdapltest Module Loaded - major 253 > kDAPL: dapl_ia_open (mthca0a, 8, ffff81003b011ed0, ffff81003b011ef8) > kDAPL: dapl_ia_open () returns 0x0 > kDAPL: dapl_pz_create (ffff81003aef2168, ffff81003b011ef0) > kDAPL: dapl_evd_kcreate (ffff81003aef2168, 8, 1, upcall, 0x20, ffff81003b011ee8) > kDAPL: dapl_evd_kcreate (ffff81003aef2168, 8, 1, upcall, 0xa0, ffff81003b011ee0) > kDAPL: dapl_evd_kcreate (ffff81003aef2168, 8, 1, upcall, 0x40, ffff81003b011ed8) > kDAPL: dapl_ep_create (ffff81003aef2168, ffff8100770c0e10, ffff81003aef2a28, ffff81003aef26e0, ffff81003b6a1b40, 0000000000000000, ffff81003b011ec8) > kDAPL: created qp_hndl = ffff81007693b2d8, qpn = 170406 > kDAPL: dapl_ia_query (ffff81003aef2168, ffff81003b011dc0, ffff81003a97cb88, ffff81003a97ce38) > kDAPL: dapl_ia_query () returns 0x0 > kDAPL: dapl_ep_query (ffff81007e3354f8, ffff81003b011d08) Below is where things start shutting down... > kDAPL: dapl_ep_free (ffff81007e3354f8) > kDAPL: dapl_ep_disconnect (ffff81007e3354f8, 0) > kDAPL: dapl_ep_disconnect () returns 0x8007002e > kDAPL: dapl_ep_free: Free EP: 0, ep ffff81007e3354f8 qp_state 1 qp_handle 7693b2d8 > kDAPL: dapl_evd_free (ffff81003b6a1b40) > kDAPL: dapl_evd_free () returns 0x0 > kDAPL: dapl_evd_free (ffff81003aef26e0) > kDAPL: dapl_evd_free () returns 0x0 > kDAPL: dapl_evd_free (ffff81003aef2a28) > kDAPL: dapl_evd_free () returns 0x0 > kDAPL: dapl_pz_free (ffff8100770c0e10) > kDAPL: dapl_ia_query (ffff81003aef2168, 0000000000000000, 0000000000000000, ffff81003b011ab8) > kDAPL: dapl_ia_query () returns 0x0 > kDAPL: dapl_ia_close (ffff81003aef2168, 1) > kDAPL: dapl_evd_free (ffff81003aef2d70) > kDAPL: dapl_evd_free () returns 0x0 I think the key was in your previous email with the client error indicated as: DT_BpoolAlloc: dat_lmr_kcreate failed DAT_NOT_IMPLEMENTED I think this is a registry/provider mismatch somehow which makes me think that the recent build changes did this somehow. -- Hal From info at kqsv07.com Thu May 5 21:03:18 2005 From: info at kqsv07.com (info at kqsv07.com) Date: 6 May 2005 13:03:18 +0900 Subject: [openib-general] $BL5NA(B10000$B1_J,$GM7$s$G$_$h$&!*(B Message-ID: <20050506040318.4923.qmail@mail.kqsv07.com> _/_/_/_/_/$B5U1g!y?M:J!y3d at Z!y(BSM$B!yB(2q$$!yD>%aD>EE(B_/_/_/_/_/ _/_/_/_/_/_/ http://awg.webchu.com/?springg _/_/_/_/_/_/ $B!xBg9%I>!V(B10000$B1_!WL5NA%]%$%s%HB#Dh%-%c%s%Z!<%s7QB3@-$,Hs>o$K9b$/!"0l;~@$4V$rFx$o$;$?2M6u at A5a$H$bEvJ}$O0l at Z4X$o$j$,$"$j$^$;$s$N$G0B?4$7$F$4MxMQ2<$5$$!#(B $B!}L5NA$G$?$C$W$jM7$s$GD:$-!"!V$3$3$J$i!*!W$H;W$C$FD:$1$?J}$O0B?4$NA0J'$$$r$4MxMQ2<$5$$!#$?$C$?(B3000$B1_$+$i$4MxMQD:$1$^$9!#(B $B"(Ev%a!<%k$NG[?.5qH]$O$3$A$i"*(B csvbit7q8a9z at ok.kz $B$^$G$* References: <1115253041.4476.147.camel@localhost.localdomain> <1115300959.4476.189.camel@localhost.localdomain> <427A5379.6060503@ichips.intel.com> Message-ID: <1115383996.4476.40.camel@localhost.localdomain> On Thu, 2005-05-05 at 13:10, Sean Hefty wrote: > > Also, I did more investigation of send failures. In terms of send > > failures upon not receiving ACKs, SA is different from vendor class 2. I > > think the problem starts (and hopefully ends) with response_mad(). In > > the case of SA GetTableResp being sent, the non data packets (ACK, etc.) > > come back as SA GetTable so this is not currently considered a response > > but I think it needs to be. I'm not sure if there are other issues > > behind this as I didn't chase it further. Let me know if you want me to > > do this. > > The RMPP code formats ACKs by inverting the response bit. This is necessary > in order to route the ACK to the correct agent, and meets the SA > requirements. I wasn't saying otherwise :-> Just trying to describe the scenario. > Response MADs are routed to the correct agent using the TID. > Non-response MADs are routed by using the lookup tables. > > E.g. if you look at the SA, the ACK (GetTable) needs to go to the agent > using the lookup tables. The TID carried in the ACK was actually set by the > sender of the ACK, and cannot be used to route to the SA's agent. Right now, OpenSM registers the SA MAD agent with all methods set in the method mask (overkill) so I'm not sure what is not happening correctly yet. > Can you describe in more detail what the exact failure that you're seeing is > and what you were expecting? When the ACK is sent, I do not get a send failure. This is expected based on what you said at the SA side. I think it (TID matching) would only happen in the MAD layer automatically on the SA client side so you would get a send failure if the ACK didn't come back in the other direction. > Thanks for looking into this more. So it seems that something above the MAD layer needs to implement the matching of ACKs, etc. on the SA side :-( If we extended response_mad to handle RMPP non data packets, this might not be the case but I haven't thought about it enough. Just a thought as a possible way around the issue on the SA side with this. -- Hal From sean.hefty at intel.com Fri May 6 07:49:19 2005 From: sean.hefty at intel.com (Hefty, Sean) Date: Fri, 6 May 2005 07:49:19 -0700 Subject: [openib-general] [PATCH] [MAD] changes to ib_create_send_mad Message-ID: >> E.g. if you look at the SA, the ACK (GetTable) needs to go to the agent >> using the lookup tables. The TID carried in the ACK was actually set by the >> sender of the ACK, and cannot be used to route to the SA's agent. > >Right now, OpenSM registers the SA MAD agent with all methods set in the >method mask (overkill) so I'm not sure what is not happening correctly >yet. This sounds fine. >> Can you describe in more detail what the exact failure that you're seeing is >> and what you were expecting? > >When the ACK is sent, I do not get a send failure. This is expected >based on what you said at the SA side. I think it (TID matching) would >only happen in the MAD layer automatically on the SA client side so you >would get a send failure if the ACK didn't come back in the other >direction. Are you referring to RMPP direction switch? If so, that is not yet implemented. I.e. The code will not generate or process ACKs to ACKs, which are required for RMPP direction switch. What the code does now is to match RMPP requests with RMPP responses, but in a manner similar to receiver initiated RMPP. I don't think that the routing algorithms in the MAD layer will need to change in order to support direction switch. Also, I don't think that supporting direction switch on the initiator's side is difficult. It is a little more challenging on the responder side. On the responder side, the MAD layer would need to know if a received MAD was going to generate an RMPP response. I'm not sure how to do this in a generic fashion, without the MAD layer being protocol aware. - Sean From tduffy at sun.com Fri May 6 07:56:08 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 06 May 2005 07:56:08 -0700 Subject: [openib-general] [PATCH] kDAPL: remove DAT_UINT32 and DAT_UINT64 In-Reply-To: <469958e0050505231754857367@mail.gmail.com> References: <1115327289.11449.7.camel@duffman> <469958e00505051425a5fc1fd@mail.gmail.com> <1115332222.11449.11.camel@duffman> <469958e0050505231754857367@mail.gmail.com> Message-ID: <1115391368.6189.3.camel@duffman> On Thu, 2005-05-05 at 23:17 -0700, Caitlin Bestler wrote: > Why does kDAPLtest have to be pre-integrated with the kernel? > > It was in a distinct directory for a reason. You are correct. kdapltest does not need to be integrated and can be coded in whatever style you prefer. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From halr at voltaire.com Fri May 6 07:51:58 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 06 May 2005 10:51:58 -0400 Subject: [openib-general] [PATCH] [MAD] changes to ib_create_send_mad In-Reply-To: References: Message-ID: <1115391117.4476.131.camel@localhost.localdomain> On Fri, 2005-05-06 at 10:49, Hefty, Sean wrote: > >When the ACK is sent, I do not get a send failure. This is expected > >based on what you said at the SA side. I think it (TID matching) would > >only happen in the MAD layer automatically on the SA client side so you > >would get a send failure if the ACK didn't come back in the other > >direction. > > Are you referring to RMPP direction switch? No. I referring to the SA sending the SA GetTableResp as one RMPP send, the SA client at the other end not ACKing, but the send that the SA issued completing successfully rather than failing with some send status indicating something like timeout. > If so, that is not yet > implemented. > I.e. The code will not generate or process ACKs to ACKs, which are > required for > RMPP direction switch. What the code does now is to match RMPP requests > with > RMPP responses, but in a manner similar to receiver initiated RMPP. And what is the definition of request/response in terms of RMPP. I'm not sure it is method based if non DATA packets are included. > I don't think that the routing algorithms in the MAD layer will need to > change > in order to support direction switch. Also, I don't think that > supporting > direction switch on the initiator's side is difficult. It is a little > more > challenging on the responder side. On the responder side, the MAD layer > would need to know if a received MAD was going to generate an RMPP > response. > I'm not sure how to do this in a generic fashion, without the MAD layer > being protocol aware. Me neither. This is method specific. (The only case of this now is SA GetMulti). -- Hal From halr at voltaire.com Fri May 6 08:13:51 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 06 May 2005 11:13:51 -0400 Subject: [openib-general] [patch] misc trivia nitpicks 04 In-Reply-To: <20050506003942.GE15688@aon.at> References: <20050506003942.GE15688@aon.at> Message-ID: <1115392431.4476.135.camel@localhost.localdomain> On Thu, 2005-05-05 at 20:39, Bernhard Fischer wrote: > Hi, > > - s/incomm/incom/g Patch applied (osm/include/iba/ib_types.h) BTW, I saw niopicks 01, 03, and 04. Did I miss 02 ? -- Hal From halr at voltaire.com Fri May 6 08:17:18 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 06 May 2005 11:17:18 -0400 Subject: [openib-general] [patch] misc trivia nitpicks 01 In-Reply-To: <20050505170956.GU15688@aon.at> References: <20050505170956.GU15688@aon.at> Message-ID: <1115392638.4476.141.camel@localhost.localdomain> On Thu, 2005-05-05 at 13:09, Bernhard Fischer wrote: > Hi, > > please consider attached. > > diffstat ./gen2.2257.cow-20050505-1902.diff > linux-kernel/infiniband/ulp/sdp/sdp_actv.c | 27 +-- > userspace/libmthca/src/mthca.c | 4 > userspace/management/osm/include/opensm/osm_subnet.h | 79 +++++----- > userspace/management/osm/include/vendor/osm_vendor_mtl_transaction_mgr.h | 43 ++--- > userspace/management/osm/opensm/osm_sa_service_record.c | 8 - > userspace/management/osm/opensm/osm_sm_state_mgr.c | 38 ++-- > userspace/management/osm/opensm/osm_sw_info_rcv.c | 9 - > userspace/management/osm/opensm/osm_trap_rcv.c | 62 +++---- > 8 files changed, 134 insertions(+), 136 deletions(-) I incorporated the OpenSM ones. Please note that osm_vendor_mtl_transaction_mgr.h is not used by OpenIB (gen2) but is currently kept around for the time when gen1 development might be done on the same code base. -- Hal From halr at voltaire.com Fri May 6 08:21:32 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 06 May 2005 11:21:32 -0400 Subject: [openib-general] [patch] misc trivia nitpicks 03 In-Reply-To: <20050505192614.GW15688@aon.at> References: <20050505192614.GW15688@aon.at> Message-ID: <1115392892.4476.143.camel@localhost.localdomain> On Thu, 2005-05-05 at 15:26, Bernhard Fischer wrote: > - s/conect/connect/g Applied the one to diags/doc/diagtools.txt -- Hal From jlentini at netapp.com Fri May 6 08:33:49 2005 From: jlentini at netapp.com (James Lentini) Date: Fri, 6 May 2005 11:33:49 -0400 (EDT) Subject: [openib-general] [PATCH] kDAPL: remove DAT_UINT32 and DAT_UINT64 In-Reply-To: <1115391368.6189.3.camel@duffman> References: <1115327289.11449.7.camel@duffman> <469958e00505051425a5fc1fd@mail.gmail.com> <1115332222.11449.11.camel@duffman> <469958e0050505231754857367@mail.gmail.com> <1115391368.6189.3.camel@duffman> Message-ID: I intentionally created a separate kdapltest directory in anticipation of needing to modify it to support changes in the DAT API for Linux. I do expect some divergence from the original test program. While there isn't a need to make modifications to the coding style, I think the test could use a thorough review. In particular, I'd like to see a better separation between the client and server sides of the test. If anyone is interested in volunteering for that please let me know. james On Fri, 6 May 2005, Tom Duffy wrote: > On Thu, 2005-05-05 at 23:17 -0700, Caitlin Bestler wrote: >> Why does kDAPLtest have to be pre-integrated with the kernel? >> >> It was in a distinct directory for a reason. > > You are correct. kdapltest does not need to be integrated and can be > coded in whatever style you prefer. > > -tduffy > From jcarr at linuxmachines.com Fri May 6 09:51:57 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Fri, 06 May 2005 09:51:57 -0700 Subject: [openib-general] in need of a simple ulp In-Reply-To: <427AC385.5070308@ichips.intel.com> References: <426FCF0A.3070806@linuxmachines.com> <426FD098.9090300@ichips.intel.com> <427AC218.9070307@linuxmachines.com> <427AC385.5070308@ichips.intel.com> Message-ID: <427BA0AD.6010105@linuxmachines.com> Sean Hefty wrote: > You're looking in the right place. > > I'll add some code to format the data and verify that it is making it to > the remote side, and see if I can track down any problems. (This will > need to wait until tomorrow for me though...) > > - Sean May 5 16:31:50 localhost kernel: ib_mthca 0000:09:00.0: CQ overrun on CQN 1a0084 I didn't notice this error. In any case, it was something I did wrong; I went back and did a simple check with your code and it is ok. I do notice though that you can generate: May 5 16:31:50 localhost kernel: ib_mthca 0000:09:00.0: 1a0084/0: error CQE -> QPN 1a0406, WQE @ 00000042 May 5 16:31:50 localhost kernel: [ 0] 001a0406 May 5 16:31:50 localhost kernel: [ 4] 00001aed May 5 16:31:50 localhost kernel: [ 8] 00000004 May 5 16:31:50 localhost kernel: [ c] 00003800 May 5 16:31:50 localhost kernel: [10] 128a0000 May 5 16:31:50 localhost kernel: [14] 00000000 May 5 16:31:50 localhost kernel: [18] 00000042 May 5 16:31:50 localhost kernel: [1c] ff000000 if you up the message_count to 0x1000. I'm guessing this is just some normal overrun error though. In any case, thanks for your help; cmtest is correct. Jeff From halr at voltaire.com Fri May 6 09:52:24 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 06 May 2005 12:52:24 -0400 Subject: [openib-general] Re: slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects In-Reply-To: <20050504065759.GA9674@mellanox.co.il> References: <20050502084013.GR16918@mellanox.co.il> <1115036059.4477.5703.camel@localhost.localdomain> <42765027.10605@ichips.intel.com> <20050502163327.GB9939@mellanox.co.il> <1115066730.4476.139.camel@localhost.localdomain> <52u0llfiot.fsf@topspin.com> <20050503072727.GD9674@mellanox.co.il> <5264y0e49p.fsf@topspin.com> <1115134226.4476.618.camel@localhost.localdomain> <20050504065759.GA9674@mellanox.co.il> Message-ID: <1115398343.4476.154.camel@localhost.localdomain> On Wed, 2005-05-04 at 02:57, Michael S. Tsirkin wrote: > > If this only occurs on machines which are running OpenSM, it could be a > > problem with the local MAD handling where ib_free_recv_mad is not called > > for some case where it should. I will look into this if this is the > > case. > > > > -- Hal > > > > This is the case for me: the error only occurs on machines which are > running OpenSM. So assuming mthca sets the bits properly for local MAD handling (and I have no reason to think otherwise), this is likely a local MAD handling error in mad.c where some path does not return an allocation to the MAD cache. I inspected the code to try and find it but couldn't. I will need to dig deeper. This will take some more time. (I will add some accounting in and see what this shows up). Please bear with me. -- Hal From info at qsv09.com Thu May 5 21:40:06 2005 From: info at qsv09.com (info at qsv09.com) Date: 6 May 2005 13:40:06 +0900 Subject: [openib-general] $BNY$N!{!{$G$9!&!&!&(B Message-ID: <20050506044006.31900.qmail@mail.qsv09.com> $B!!!!!!!!!!!!"v!&!yL5NAEPO?%-%c%s%Z!<%sCf!y!&"v(B $B!!!!!!$d$C$Q$j=P0)$&$J$i$46a=j$G2q$($kAjR2pCW$7$^$9!#(B $B!!!!!!!!!!!!!!!!!!!!$^$:$O$*;n$7L5NAEPO?$+$i(B $B!!!!!!!!!!!!!!!!!!!!!!!!!!"-"-"-"-"-"-(B $B!!!!!!!!!!!!!!!!(Bhttp://www.jumpb2.net/?imasugu $B!!!!!!!!!!!!!!!!(B $B!!!!!!!!!!!!!!!!!!(B $B%U%j!<%a!<%kBP1~$G$9!*(B $B5qH](B iranai at jumpb2.net From steve at wooding.uklinux.net Fri May 6 10:28:42 2005 From: steve at wooding.uklinux.net (Steven Wooding) Date: Fri, 06 May 2005 18:28:42 +0100 Subject: [openib-general] [PATCH][SDP] AIO buffer corruption In-Reply-To: <20050503130243.C21837@topspin.com> References: <200505021548.52053.steve@wooding.uklinux.net> <20050503130243.C21837@topspin.com> Message-ID: <427BA94A.20106@wooding.uklinux.net> Libor, Here are the details of my setup: HCA: PCI-Express MT25208 (MT23108 compat mode) with 256MB memory HCA Fireware: 4.6.2 Command: ttcp.aio.c.x -r -fM -a 8 -l 2048 ttcp.aio.c.x -t -fM -a 8 -n 10000 -l 2048 {ip of receiver} Syslog: Get the following message with sucessful transfer, but not with -32/-104 error Kernel: ERR: IOCB <0> cancel <0> flag <00e4> size <{-l}:0:{-l}> SDP Debug: Get the following message with all ttcp.aio.c.x runs Kernel: WARN: <9> <0101:3b01> CM state <0> event <9> error <-2> I'm away for two weeks, so I'll get back to you with any further info you require when I get back. Cheers, Steve Wooding. Libor Michalek wrote: >On Mon, May 02, 2005 at 03:48:51PM +0100, Steven Wooding wrote: > > >>Hello Libor, >> >>I've tried your patch, but unfortunately it made no difference to >>the -32/-104 errors I get. I have observed the following features, >>which may help you diagnose my problem: >> >>Platform: 64 bit >>OS: RHEL 4 >>Kernel: 2.6.11.6 >>OpenIB gen2: 2225 >>SM: IO5000 switch >> >> > >Steve, > > I'm not seeing this issue on my x86_64 systems. Is there anything >in the syslog on either system? If you build SDP with debug, there >will be messages in the log, any errors? Which HCA are you using, and >which firmware? Also, can you send the exact command line? > >-Libor > > > > From jlentini at netapp.com Fri May 6 10:30:16 2005 From: jlentini at netapp.com (James Lentini) Date: Fri, 6 May 2005 13:30:16 -0400 (EDT) Subject: [openib-general] Re: [kDAPL][PATCH] fix panic in server side In-Reply-To: <1115243969.14238.6.camel@duffman> References: <1115243969.14238.6.camel@duffman> Message-ID: Tom, There were two changes in this patch: - check for a NULL comm_id pointer - add error returns My understanding of the subsequent discussion is that: - we should not have been using the comm_id->state value. We should have been using the event->event value instead - if we return a non 0 error code, our CM ID will be destroyed Given those pieces of information: Do we still want to check for a NULL comm_id pointer? Should we always return 0 from this function? I think the answer is yes to both of the above, but I want to make sure there is consensus. james On Wed, 4 May 2005, Tom Duffy wrote: > This occurred when I did a connect without first setting up a listen > using kdapltest. > > Unable to handle kernel NULL pointer dereference at 0000000000000020 RIP: > {:ib_dat_provider:dapl_cm_passive_cb_handler+27} > PGD 371ab067 PUD 36822067 PMD 0 > Oops: 0000 [1] SMP > CPU 2 > Modules linked in: kdapltest ib_dat_provider dat ib_at ib_cm ib_ipoib ib_sa ib_umad md5 ipv6 parport_pc lp parport autofs4 nfs lockd sunrpc pcmcia yenta_socket rsrc_nonstatic pcmcia_core ext3 jbd video container button battery ac uhci_hcd ehci_hcd hw_random i2c_i801 i2c_core ib_mthca ib_mad ib_core e1000 floppy dm_snapshot dm_zero dm_mirror xfs exportfs dm_mod mptscsih mptbase sd_mod scsi_mod > Pid: 11326, comm: ib_cm/2 Not tainted 2.6.12-rc3openib > RIP: 0010:[] {:ib_dat_provider:dapl_cm_passive_cb_handler+27} > RSP: 0018:ffff810038f65d90 EFLAGS: 00010202 > RAX: 0000000000000000 RBX: ffff81000b3764b0 RCX: ffff81000b376550 > RDX: 0000000000000056 RSI: ffff81000b376548 RDI: ffff81003f0cd8a0 > RBP: ffff81003f0cd8a0 R08: 0000000000000000 R09: 0000000000000000 > R10: 0000000000000020 R11: ffff81003925ae10 R12: ffff81003f0cd8a0 > R13: ffff81003dcb8a88 R14: 0000000000000000 R15: ffff81000b3765a0 > FS: 0000000000000000(0000) GS:ffffffff80487a00(0000) knlGS:0000000000000000 > CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b > CR2: 0000000000000020 CR3: 0000000038bbd000 CR4: 00000000000006e0 > Process ib_cm/2 (pid: 11326, threadinfo ffff810038f64000, task ffff81003e75af30)Stack: ffffffff882aae32 0100000000000293 0000000000000000 ffff81000b3764b0 > 000000003f0cd8a0 ffff81003dcb8a88 0000000000000000 ffff81000b3765a0 > ffffffff882abd1e ffff810001e1b680 > Call Trace:{:ib_cm:cm_process_work+50} {:ib_cm:cm_work_handler+1438} > {do_sigaction+520} {:ib_cm:cm_work_handler+0} > {worker_thread+476} {default_wake_function+0} > {__wake_up_common+67} {worker_thread+0} > {keventd_create_kthread+0} {kthread+217} > {schedule_tail+64} {child_rip+8} > {keventd_create_kthread+0} {kthread+0} > {child_rip+0} > > Code: 8b 40 20 89 44 24 04 83 7c 24 04 03 74 30 83 7c 24 04 03 77 > RIP {:ib_dat_provider:dapl_cm_passive_cb_handler+27} RSP > CR2: 0000000000000020 > > This cuteness happened in the kdapltest userland program: > > Warning: conn_event_wait DAT_CONNECTION_EVENT_DISCONNECTED > DT_cs_Client: bad connection event > DT_cs_Client: Cleaning Up ... > DT_cs_Client: IA mthca0a closed > DT_cs_Client: ========== End of Work -- Client Exiting > TEST INSTANCE 1 > *** glibc detected *** free(): invalid next size (fast): 0x000000000050ca50 *** > Aborted (core dumped) > > This patches fixes two things. It makes it so that > dapl_cm_passive_cb_handler won't dereference a potentially NULL ib_cm_id > and also that it returns ret instead of 0 all the time. > > Signed-off-by: Tom Duffy > > Index: dapl_openib_cm.c > =================================================================== > --- linux-kernel/dat-provider/dapl_openib_cm.c (revision 2257) > +++ linux-kernel/dat-provider/dapl_openib_cm.c (working copy) > @@ -227,6 +227,9 @@ int dapl_cm_passive_cb_handler(struct ib > { > int ret = 0; > > + if (!comm_id) > + return -1; > + > switch (comm_id->state) { > case IB_CM_IDLE: > ret = do_passive_idle(comm_id); > @@ -247,7 +250,7 @@ int dapl_cm_passive_cb_handler(struct ib > break; > } > > - return 0; > + return ret; > } > > static void dapl_ib_destroy_cm_id_work(void *data) > From jlentini at netapp.com Fri May 6 10:35:10 2005 From: jlentini at netapp.com (James Lentini) Date: Fri, 6 May 2005 13:35:10 -0400 (EDT) Subject: [openib-general] Re: [kDAPL][PATCH] fix panic in server side In-Reply-To: References: <1115243969.14238.6.camel@duffman> Message-ID: > Tom, > > There were two changes in this patch: > > - check for a NULL comm_id pointer > - add error returns > > My understanding of the subsequent discussion is that: > > - we should not have been using the comm_id->state value. We should > have been using the event->event value instead > > - if we return a non 0 error code, our CM ID will be destroyed > > Given those pieces of information: > > Do we still want to check for a NULL comm_id pointer? > Should we always return 0 from this function? > > I think the answer is yes to both of the above, but I want to make sure there > is consensus. On second thought, I'd like to retract that last statement. If the comm_id pointer is NULL, that strikes me as a bug somewhere that needs to be fixed. An debug assert to verify that the comm_id is not null seems like a better option. > > james > > On Wed, 4 May 2005, Tom Duffy wrote: > >> This occurred when I did a connect without first setting up a listen >> using kdapltest. >> >> Unable to handle kernel NULL pointer dereference at 0000000000000020 RIP: >> {:ib_dat_provider:dapl_cm_passive_cb_handler+27} >> PGD 371ab067 PUD 36822067 PMD 0 >> Oops: 0000 [1] SMP >> CPU 2 >> Modules linked in: kdapltest ib_dat_provider dat ib_at ib_cm ib_ipoib ib_sa >> ib_umad md5 ipv6 parport_pc lp parport autofs4 nfs lockd sunrpc pcmcia >> yenta_socket rsrc_nonstatic pcmcia_core ext3 jbd video container button >> battery ac uhci_hcd ehci_hcd hw_random i2c_i801 i2c_core ib_mthca ib_mad >> ib_core e1000 floppy dm_snapshot dm_zero dm_mirror xfs exportfs dm_mod >> mptscsih mptbase sd_mod scsi_mod >> Pid: 11326, comm: ib_cm/2 Not tainted 2.6.12-rc3openib >> RIP: 0010:[] >> {:ib_dat_provider:dapl_cm_passive_cb_handler+27} >> RSP: 0018:ffff810038f65d90 EFLAGS: 00010202 >> RAX: 0000000000000000 RBX: ffff81000b3764b0 RCX: ffff81000b376550 >> RDX: 0000000000000056 RSI: ffff81000b376548 RDI: ffff81003f0cd8a0 >> RBP: ffff81003f0cd8a0 R08: 0000000000000000 R09: 0000000000000000 >> R10: 0000000000000020 R11: ffff81003925ae10 R12: ffff81003f0cd8a0 >> R13: ffff81003dcb8a88 R14: 0000000000000000 R15: ffff81000b3765a0 >> FS: 0000000000000000(0000) GS:ffffffff80487a00(0000) >> knlGS:0000000000000000 >> CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b >> CR2: 0000000000000020 CR3: 0000000038bbd000 CR4: 00000000000006e0 >> Process ib_cm/2 (pid: 11326, threadinfo ffff810038f64000, task >> ffff81003e75af30)Stack: ffffffff882aae32 0100000000000293 0000000000000000 >> ffff81000b3764b0 >> 000000003f0cd8a0 ffff81003dcb8a88 0000000000000000 ffff81000b3765a0 >> ffffffff882abd1e ffff810001e1b680 >> Call Trace:{:ib_cm:cm_process_work+50} >> {:ib_cm:cm_work_handler+1438} >> {do_sigaction+520} >> {:ib_cm:cm_work_handler+0} >> {worker_thread+476} >> {default_wake_function+0} >> {__wake_up_common+67} >> {worker_thread+0} >> {keventd_create_kthread+0} >> {kthread+217} >> {schedule_tail+64} {child_rip+8} >> {keventd_create_kthread+0} >> {kthread+0} >> {child_rip+0} >> >> Code: 8b 40 20 89 44 24 04 83 7c 24 04 03 74 30 83 7c 24 04 03 77 >> RIP {:ib_dat_provider:dapl_cm_passive_cb_handler+27} RSP >> >> CR2: 0000000000000020 >> >> This cuteness happened in the kdapltest userland program: >> >> Warning: conn_event_wait DAT_CONNECTION_EVENT_DISCONNECTED >> DT_cs_Client: bad connection event >> DT_cs_Client: Cleaning Up ... >> DT_cs_Client: IA mthca0a closed >> DT_cs_Client: ========== End of Work -- Client Exiting >> TEST INSTANCE 1 >> *** glibc detected *** free(): invalid next size (fast): 0x000000000050ca50 >> *** >> Aborted (core dumped) >> >> This patches fixes two things. It makes it so that >> dapl_cm_passive_cb_handler won't dereference a potentially NULL ib_cm_id >> and also that it returns ret instead of 0 all the time. >> >> Signed-off-by: Tom Duffy >> >> Index: dapl_openib_cm.c >> =================================================================== >> --- linux-kernel/dat-provider/dapl_openib_cm.c (revision 2257) >> +++ linux-kernel/dat-provider/dapl_openib_cm.c (working copy) >> @@ -227,6 +227,9 @@ int dapl_cm_passive_cb_handler(struct ib >> { >> int ret = 0; >> >> + if (!comm_id) >> + return -1; >> + >> switch (comm_id->state) { >> case IB_CM_IDLE: >> ret = do_passive_idle(comm_id); >> @@ -247,7 +250,7 @@ int dapl_cm_passive_cb_handler(struct ib >> break; >> } >> >> - return 0; >> + return ret; >> } >> >> static void dapl_ib_destroy_cm_id_work(void *data) >> > From tduffy at sun.com Fri May 6 10:37:37 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 06 May 2005 10:37:37 -0700 Subject: [openib-general] Re: [kDAPL][PATCH] fix panic in server side In-Reply-To: References: <1115243969.14238.6.camel@duffman> Message-ID: <1115401057.6189.13.camel@duffman> On Fri, 2005-05-06 at 13:30 -0400, James Lentini wrote: > Tom, > > There were two changes in this patch: > > - check for a NULL comm_id pointer > - add error returns > > My understanding of the subsequent discussion is that: > > - we should not have been using the comm_id->state value. We should > have been using the event->event value instead > > - if we return a non 0 error code, our CM ID will be destroyed > > Given those pieces of information: > > Do we still want to check for a NULL comm_id pointer? > Should we always return 0 from this function? > > I think the answer is yes to both of the above, but I want to make > sure there is consensus. My patch was wrong as pointed out. The panic I saw did have a NULL comm_id. I am not sure how this happened. Apparently, this function should never be called with a null pointer. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From blist at aon.at Fri May 6 10:41:04 2005 From: blist at aon.at (Bernhard Fischer) Date: Fri, 6 May 2005 19:41:04 +0200 Subject: [openib-general] [patch] misc trivia nitpicks 04 In-Reply-To: <1115392431.4476.135.camel@localhost.localdomain> References: <20050506003942.GE15688@aon.at> <1115392431.4476.135.camel@localhost.localdomain> Message-ID: <20050506174104.GI15688@aon.at> On Fri, May 06, 2005 at 11:13:51AM -0400, Hal Rosenstock wrote: >On Thu, 2005-05-05 at 20:39, Bernhard Fischer wrote: >> Hi, >> >> - s/incomm/incom/g > >Patch applied (osm/include/iba/ib_types.h) Thank you. > >BTW, I saw niopicks 01, 03, and 04. Did I miss 02 ? 02 was get_clock.h: if mhz!=m set mhz=0.0 and break so the file is closed proper. I haven't sent that yet. Should i send a patch later or will MST fix this? From jlentini at netapp.com Fri May 6 10:43:50 2005 From: jlentini at netapp.com (James Lentini) Date: Fri, 6 May 2005 13:43:50 -0400 (EDT) Subject: [openib-general] cm_id state information In-Reply-To: <427A617C.1060607@ichips.intel.com> References: <427A617C.1060607@ichips.intel.com> Message-ID: Sean, Is there documentation or a piece of sample code that shows how to interact with the CM correctly? james On Thu, 5 May 2005, Sean Hefty wrote: > This came up on a separate thread... > > Currently the cm_id structured defined in ib_cm.h exposes a state field. This > (and a couple of other fields) were originally exposed only for debug > purposes, but I've noticed that both kDAPL and SDP access this field. > > The issue is that access to the cm_id state must be protected by a lock in > order to guarantee correct operation of the CM state machine. Because CM > messages can be received at any time, the cm_id state could change if it is > accessed outside of the locks. > > While in a CM callback, the reported event should be used to determine the > correct operation to take, rather than reading the cm_id state. I will try > to work up some patches to correct this. > > I will at the very least add comment to ib_cm.h about using these fields, or > move them into the internal CM data structures. > > - Sean > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From halr at voltaire.com Fri May 6 11:17:07 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 06 May 2005 14:17:07 -0400 Subject: [openib-general] Re: [kDAPL][PATCH] fix panic in server side In-Reply-To: References: <1115243969.14238.6.camel@duffman> Message-ID: <1115403427.4476.214.camel@localhost.localdomain> On Fri, 2005-05-06 at 13:30, James Lentini wrote: > - we should not have been using the comm_id->state value. We should > have been using the event->event value instead The DAPL CM is being reworked for this. > Do we still want to check for a NULL comm_id pointer? That should not occur and should be chased down. > Should we always return 0 from this function? Currently yes. -- Hal From mshefty at ichips.intel.com Fri May 6 11:28:01 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Fri, 06 May 2005 11:28:01 -0700 Subject: [openib-general] cm_id state information In-Reply-To: References: <427A617C.1060607@ichips.intel.com> Message-ID: <427BB731.50800@ichips.intel.com> James Lentini wrote: > Is there documentation or a piece of sample code that shows how to > interact with the CM correctly? The only sample code is cmpost, under the util directory. You could probably use SDP as well. Also, I'm currently working with Hal to make sure that kDAPL works properly with the CM. It will just take a little while before a patch is ready. - Sean From mshefty at ichips.intel.com Fri May 6 11:33:19 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Fri, 06 May 2005 11:33:19 -0700 Subject: [openib-general] [PATCH] [MAD] changes to ib_create_send_mad In-Reply-To: <1115391117.4476.131.camel@localhost.localdomain> References: <1115391117.4476.131.camel@localhost.localdomain> Message-ID: <427BB86F.5040909@ichips.intel.com> Hal Rosenstock wrote: > On Fri, 2005-05-06 at 10:49, Hefty, Sean wrote: > >>>When the ACK is sent, I do not get a send failure. This is expected >>>based on what you said at the SA side. I think it (TID matching) would >>>only happen in the MAD layer automatically on the SA client side so you >>>would get a send failure if the ACK didn't come back in the other >>>direction. >> >>Are you referring to RMPP direction switch? > > > No. I referring to the SA sending the SA GetTableResp as one RMPP send, > the SA client at the other end not ACKing, but the send that the SA > issued completing successfully rather than failing with some send status > indicating something like timeout. Ah... I think I finally understand now. I'll see if I can reproduce this. > >>If so, that is not yet >>implemented. >>I.e. The code will not generate or process ACKs to ACKs, which are >>required for >>RMPP direction switch. What the code does now is to match RMPP requests >>with >>RMPP responses, but in a manner similar to receiver initiated RMPP. > > > And what is the definition of request/response in terms of RMPP. I'm not > sure it is method based if non DATA packets are included. I was referring only to DATA packets and whether the response bit was set or not. - Sean From iod00d at hp.com Fri May 6 12:09:47 2005 From: iod00d at hp.com (Grant Grundler) Date: Fri, 6 May 2005 12:09:47 -0700 Subject: [openib-general] Re: Re: user-mode verbs on Itanium In-Reply-To: <20050504070910.GC9674@mellanox.co.il> References: <1AC79F16F5C5284499BB9591B33D6F00043B37C9@orsmsx408> <20050429051009.GT20957@esmail.cup.hp.com> <52pswdk5au.fsf@topspin.com> <20050429162127.GA24871@esmail.cup.hp.com> <52ll71k09w.fsf@topspin.com> <20050429183846.GH24871@esmail.cup.hp.com> <20050504005534.GA13574@esmail.cup.hp.com> <20050504070910.GC9674@mellanox.co.il> Message-ID: <20050506190947.GA23958@esmail.cup.hp.com> On Wed, May 04, 2005 at 10:09:10AM +0300, Michael S. Tsirkin wrote: > Grant, if you want a latency benchmark you may want to look at > contrib/mellanox/perftest. Notes on finding perftest ------------------------- [ ggg *hugs* google ] http://article.gmane.org/gmane.linux.drivers.openib/9614 The SVN layout is confusing me. I was expecting perftest in either svn/gen1/trunk or svn/gen2/trunk. [ And there is also trunk/openib/gen1. Is that the same thing as svn/gen1? ] Would it make sense to move rdma_lat.c to gen2/trunk/src/userspace/examples ? I only suggest it because the perftest/README says: This directory includes gen2 uverbs microbenchmarks - ... Building rmda_lat.c ------------------- In order to compile on ia64, I had to hack: o use asm/timex.h to get arch specific cycles_t and get_cycles o ditch get_clock and use get_cycles instead o renamed get_clock.h into get_clock.c since it only contains C code o rewrite Makefile so it knew to build/link get_clock.o I also rewrote the README so it's easier to read and list which libs/headers rdma_lat depends on to build. Since the diff is essentially the whole file for everything but rdma_lat.c, I've parked the entire mess here: http://gsyprf3.external.hp.com/openib/perftest-01.tgz BTW, Can I "leverage" code from ibv_pingpong to replace the issues the following comment in rdma_lat.c refers to? /* * pp_get_local_lid() uses a pretty bogus method for finding the LID * of a local port. Please don't copy this into your app (or if you * do, please rip it out soon). */ rdma_lat runtime error ---------------------- Unlike ibv_pingpong, rdma_lat doesn't work and I'm hoping it might be obvious to someone else why not. If it's not obvious, I can dig deeper into it. The two sides are able to identify each other but call to ibv_post_send() failed on the first attempt. "Server" side output: grundler at gsyprf3:/usr/src/trunk/contrib/mellanox/perftest$ ./rdma_lat -s 64 -n 10000 local address: LID 0x0b, QPN 0x1b0406, PSN 0xdcf5cc RKey 0x1100032 VAddr 0x6000000000014040 remote address: LID 0x10, QPN 0x30406, PSN 0x15f97c, RKey 0x1e0436 VAddr 0x6000000000014040 "Client" side output: grundler at iota:/usr/src/trunk/contrib/mellanox/perftest$ ./rdma_lat -s 64 -n 10000 10.0.0.51 local address: LID 0x10, QPN 0x30406, PSN 0x15f97c RKey 0x1e0436 VAddr 0x6000000000014040 remote address: LID 0x0b, QPN 0x1b0406, PSN 0xdcf5cc, RKey 0x1100032 VAddr 0x6000000000014040 Couldn't post send: scnt=1 grundler at iota:/usr/src/trunk/contrib/mellanox/perftest$ thanks, grant From libor at topspin.com Fri May 6 12:41:12 2005 From: libor at topspin.com (Libor Michalek) Date: Fri, 6 May 2005 12:41:12 -0700 Subject: [openib-general] [PATCH][SDP] AIO buffer corruption In-Reply-To: <427BA94A.20106@wooding.uklinux.net>; from steve@wooding.uklinux.net on Fri, May 06, 2005 at 06:28:42PM +0100 References: <200505021548.52053.steve@wooding.uklinux.net> <20050503130243.C21837@topspin.com> <427BA94A.20106@wooding.uklinux.net> Message-ID: <20050506124112.A26823@topspin.com> On Fri, May 06, 2005 at 06:28:42PM +0100, Steven Wooding wrote: > Libor, > > Command: ttcp.aio.c.x -r -fM -a 8 -l 2048 > ttcp.aio.c.x -t -fM -a 8 -n 10000 -l 2048 {ip of receiver} This command reproduces it very quickly, it just happens that the parameters I tried did not show the problem. > Syslog: Get the following message with sucessful transfer, but not with > -32/-104 error > Kernel: ERR: IOCB <0> cancel <0> flag <00e4> size <{-l}:0:{-l}> > > SDP Debug: Get the following message with all ttcp.aio.c.x runs > Kernel: WARN: <9> <0101:3b01> CM state <0> event <9> error <-2> Actually the IOCB cancels are not errors, I should change those to regular messages. Here are the real errors on the send side: kernel: WARN: <0> <050e:11b1> Error <-2048> post data during flush kernel: WARN: <0> <050e:11b1> Error <-2048> flushing data queue kernel: WARN: <0> <050e:11b1> Error <-2048> flushing send queue > I'm away for two weeks, so I'll get back to you with any further info > you require when I get back. Here is the patch to fix the problem, the fix exposes another problem, where the send pipeline stalls and is not restarted, which I'm working on. The problem was caused by copying more data from the iocb then it contained sending the size of the iocb to negative. Signed-off-by: Libor Michalek Index: sdp_send.c =================================================================== --- sdp_send.c (revision 2270) +++ sdp_send.c (working copy) @@ -742,11 +742,10 @@ copy = min((PAGE_SIZE - offset), (unsigned long)(buff->end - buff->tail)); - + copy = min((unsigned long)iocb->len, copy); #ifndef _SDP_DATA_PATH_NULL memcpy(buff->tail, (addr + offset), copy); #endif - buff->data_size += copy; buff->tail += copy; iocb->post += copy; @@ -805,7 +804,7 @@ /* * TODO: need to be checking OOB here. */ - result =sdp_send_iocb_buff_write(iocb, buff); + result = sdp_send_iocb_buff_write(iocb, buff); if (result < 0) { sdp_dbg_warn(conn, "Error <%d> copy from IOCB <%d>.", result, iocb->key); @@ -981,8 +980,9 @@ * error */ if (result < 0) { - sdp_dbg_warn(conn, "Error <%d> post data during flush", - result); + sdp_dbg_warn(conn, + "Error <%d> post data <%d> during flush", + result, element->type); /* * check for dangling element reference, * since called functions can dequeue the From iod00d at hp.com Fri May 6 12:43:30 2005 From: iod00d at hp.com (Grant Grundler) Date: Fri, 6 May 2005 12:43:30 -0700 Subject: [openib-general] [patch] misc trivia nitpicks 04 In-Reply-To: <20050506174104.GI15688@aon.at> References: <20050506003942.GE15688@aon.at> <1115392431.4476.135.camel@localhost.localdomain> <20050506174104.GI15688@aon.at> Message-ID: <20050506194330.GB23958@esmail.cup.hp.com> On Fri, May 06, 2005 at 07:41:04PM +0200, Bernhard Fischer wrote: > 02 was get_clock.h: if mhz!=m set mhz=0.0 and break so the file is > closed proper. I haven't sent that yet. I've fixed now too. I've added an option for the program to print out the results in cycles. And get_clock_khz() is silly - it should be called get_clock_mhz since the usage is: double mhz = get_cpu_khz()/1000; and all the arches I have report: ia64 reports "cpu MHz : 1500.000000" x86 reports "cpu MHz : 1000.814" parisc reports "cpu MHz : 650.000000" I also fixed a typo "meadian" (vs median). I'll post a -02 version shortly at: http://gsyprf3.external.hp.com/openib/perftest-02.tgz > Should i send a patch later or will MST fix this? I'll keep pestering MST with this bit. :^) thanks, grant From blist at aon.at Fri May 6 13:25:51 2005 From: blist at aon.at (Bernhard Fischer) Date: Fri, 6 May 2005 22:25:51 +0200 Subject: [openib-general] [patch] misc trivia nitpicks 04 In-Reply-To: <20050506194330.GB23958@esmail.cup.hp.com> References: <20050506003942.GE15688@aon.at> <1115392431.4476.135.camel@localhost.localdomain> <20050506174104.GI15688@aon.at> <20050506194330.GB23958@esmail.cup.hp.com> Message-ID: <20050506202551.GJ15688@aon.at> On Fri, May 06, 2005 at 12:43:30PM -0700, Grant Grundler wrote: >On Fri, May 06, 2005 at 07:41:04PM +0200, Bernhard Fischer wrote: >> 02 was get_clock.h: if mhz!=m set mhz=0.0 and break so the file is >> closed proper. I haven't sent that yet. > >I've fixed now too. > >I've added an option for the program to print out the results in cycles. > >And get_clock_khz() is silly - it should be called get_clock_mhz since >the usage is: > double mhz = get_cpu_khz()/1000; > >and all the arches I have report: > ia64 reports "cpu MHz : 1500.000000" > x86 reports "cpu MHz : 1000.814" > parisc reports "cpu MHz : 650.000000" > >I also fixed a typo "meadian" (vs median). > >I'll post a -02 version shortly at: > http://gsyprf3.external.hp.com/openib/perftest-02.tgz > > >> Should i send a patch later or will MST fix this? > >I'll keep pestering MST with this bit. :^) I've also inlined double_compar and inserted an else befor the second if to have a better if elif else chain. Cosmetic since it's fed into qsort, i assume. Even the original version just segfaults on me. I must be missing something.. Still looking > >thanks, >grant > From halr at voltaire.com Fri May 6 13:24:20 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 06 May 2005 16:24:20 -0400 Subject: [openib-general] [PATCH] kDAPL: remove DAT_UINT32 and DAT_UINT64 Message-ID: <1115411060.4476.346.camel@localhost.localdomain> Hi, I think I found what is going on with the failure Tom reported: DT_BpoolAlloc: dat_lmr_kcreate failed DAT_NOT_IMPLEMENTED kdapltest recently changed to support memory types. [-M ] One of VIR, PHY, IA, FMR, or BP. I think it uses virtual by default. That is not a good choice for OpenIB as: dat-provider/dapl_lmr_create: case DAT_MEM_TYPE_VIRTUAL: { dat_status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); break; so the various clients should use physical by default. I also couldn't seem to get the -M PHY to work: DT_BpoolAlloc: dat_lmr_kcreate failed DAT_SUCCESS DAT_INVALID_ARG7 -- Hal From jlentini at netapp.com Fri May 6 13:36:09 2005 From: jlentini at netapp.com (James Lentini) Date: Fri, 6 May 2005 16:36:09 -0400 (EDT) Subject: [openib-general] [PATCH] kDAPL: remove DAT_UINT32 and DAT_UINT64 In-Reply-To: <1115411060.4476.346.camel@localhost.localdomain> References: <1115411060.4476.346.camel@localhost.localdomain> Message-ID: This was the problem. I just committed the fix in revision 2272. Before committing the change to add the -M flag, I thought I had changed the default to physical. Let me know if you still have trouble. james > Hi, > > I think I found what is going on with the failure Tom reported: > DT_BpoolAlloc: dat_lmr_kcreate failed DAT_NOT_IMPLEMENTED > > kdapltest recently changed to support memory types. > > [-M ] One of VIR, PHY, IA, FMR, or BP. > > I think it uses virtual by default. That is not a good choice for OpenIB > as: > > dat-provider/dapl_lmr_create: > > case DAT_MEM_TYPE_VIRTUAL: > { > dat_status = > DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); > break; > > so the various clients should use physical by default. > > I also couldn't seem to get the -M PHY to work: > DT_BpoolAlloc: dat_lmr_kcreate failed DAT_SUCCESS DAT_INVALID_ARG7 > > -- Hal > From iod00d at hp.com Fri May 6 14:08:07 2005 From: iod00d at hp.com (Grant Grundler) Date: Fri, 6 May 2005 14:08:07 -0700 Subject: [openib-general] [patch] misc trivia nitpicks 04 In-Reply-To: <20050506202551.GJ15688@aon.at> References: <20050506003942.GE15688@aon.at> <1115392431.4476.135.camel@localhost.localdomain> <20050506174104.GI15688@aon.at> <20050506194330.GB23958@esmail.cup.hp.com> <20050506202551.GJ15688@aon.at> Message-ID: <20050506210807.GE23958@esmail.cup.hp.com> On Fri, May 06, 2005 at 10:25:51PM +0200, Bernhard Fischer wrote: > I've also inlined double_compar Eh? That's pointless. The double_compar() function pointer is passed to qsort(): qsort(delta, iters - 1, sizeof *delta, double_compar); > and inserted an else befor the second if to have a better > if elif else chain. Cosmetic since it's fed into qsort, i assume. I ditched the "else" in "else return 0". Pointless too. (This is inside double_compar()). > Even the original version just segfaults on me. > I must be missing something..Still looking Hrm...can you post output how far it gets? You verified your libibverbs matches your kernel modules? thanks, grant From halr at voltaire.com Fri May 6 14:12:01 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 06 May 2005 17:12:01 -0400 Subject: [openib-general] [PATCH] kDAPL: remove DAT_UINT32 and DAT_UINT64 In-Reply-To: References: <1115411060.4476.346.camel@localhost.localdomain> Message-ID: <1115413920.4476.1.camel@localhost.localdomain> On Fri, 2005-05-06 at 16:36, James Lentini wrote: > Let me know if you still have trouble. Works for me. Hope it does for Tom. > > I also couldn't seem to get the -M PHY to work: > > DT_BpoolAlloc: dat_lmr_kcreate failed DAT_SUCCESS DAT_INVALID_ARG7 Have you tried the -M option ? Do you see this ? -- Hal From blist at aon.at Fri May 6 15:19:19 2005 From: blist at aon.at (Bernhard Fischer) Date: Sat, 7 May 2005 00:19:19 +0200 Subject: [openib-general] [patch] misc trivia nitpicks 04 In-Reply-To: <20050506210807.GE23958@esmail.cup.hp.com> References: <20050506003942.GE15688@aon.at> <1115392431.4476.135.camel@localhost.localdomain> <20050506174104.GI15688@aon.at> <20050506194330.GB23958@esmail.cup.hp.com> <20050506202551.GJ15688@aon.at> <20050506210807.GE23958@esmail.cup.hp.com> Message-ID: <20050506221919.GK15688@aon.at> On Fri, May 06, 2005 at 02:08:07PM -0700, Grant Grundler wrote: >On Fri, May 06, 2005 at 10:25:51PM +0200, Bernhard Fischer wrote: >Hrm...can you post output how far it gets? > >You verified your libibverbs matches your kernel modules? Ah, yes it's an old kernel. thanks, From blist at aon.at Fri May 6 15:35:06 2005 From: blist at aon.at (Bernhard Fischer) Date: Sat, 7 May 2005 00:35:06 +0200 Subject: [openib-general] sdp_pass.c: sdp_cm_listen_lookup() Message-ID: <20050506223506.GL15688@aon.at> Hi, I get compile errors in sdp_cm_listen_lookup for these being unknown: sk_debug, sk_localroute, sk_rcvtstamp From libor at topspin.com Fri May 6 15:57:50 2005 From: libor at topspin.com (Libor Michalek) Date: Fri, 6 May 2005 15:57:50 -0700 Subject: [openib-general] sdp_pass.c: sdp_cm_listen_lookup() In-Reply-To: <20050506223506.GL15688@aon.at>; from blist@aon.at on Sat, May 07, 2005 at 12:35:06AM +0200 References: <20050506223506.GL15688@aon.at> Message-ID: <20050506155750.B26823@topspin.com> On Sat, May 07, 2005 at 12:35:06AM +0200, Bernhard Fischer wrote: > Hi, > > I get compile errors in sdp_cm_listen_lookup for these being unknown: > > sk_debug, sk_localroute, sk_rcvtstamp Which version of the kernel are you using, and is this the only error that you get? -Libor From timur.tabi at ammasso.com Fri May 6 16:08:17 2005 From: timur.tabi at ammasso.com (Timur Tabi) Date: Fri, 06 May 2005 18:08:17 -0500 Subject: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation In-Reply-To: <427A6A7E.8000604@ammasso.com> References: <200544159.Ahk9l0puXy39U6u6@topspin.com> <20050411142213.GC26127@kalmia.hozed.org> <52mzs51g5g.fsf@topspin.com> <20050411163342.GE26127@kalmia.hozed.org> <5264yt1cbu.fsf@topspin.com> <20050411180107.GF26127@kalmia.hozed.org> <52oeclyyw3.fsf@topspin.com> <20050411171347.7e05859f.akpm@osdl.org> <20050412180447.E6958@topspin.com> <20050425203110.A9729@topspin.com> <4279142A.8050501@ammasso.com> <427A6A7E.8000604@ammasso.com> Message-ID: <427BF8E1.2080006@ammasso.com> Timur Tabi wrote: > I haven't gotten a reply to this question, but I've done my own > research, and I think I found the answer. Using my own test of > get_user_pages(), it appears that the fix was placed in 2.6.7. However, > I would like to know specifically what the fix is. Unfortunately, > tracking this stuff down is beyond my understanding of the Linux VM. I'm also still waiting for a reply to this question. Anyone???? Upon doing some more research, I think the fix might be those code instead: /* * Don't pull an anonymous page out from under get_user_pages. * GUP carefully breaks COW and raises page count (while holding * page_table_lock, as we have here) to make sure that the page * cannot be freed. If we unmap that page here, a user write * access to the virtual address will bring back the page, but * its raised count will (ironically) be taken to mean it's not * an exclusive swap page, do_wp_page will replace it by a copy * page, and the user never get to see the data GUP was holding * the original page for. */ if (PageSwapCache(page) && page_count(page) != page->mapcount + 2) { ret = SWAP_FAIL; goto out_unmap; } Both this change and the other one I mentioned are new to 2.6.7. I suppose I could try applying these patches to the 2.6.6 kernel and see if anything improves, but that won't help me understand what's really going on. The above comment makes sounds almost like it's a fix, but it talks about copy-on-write, which is has nothing to do with the real problem. -- Timur Tabi Staff Software Engineer timur.tabi at ammasso.com One thing a Southern boy will never say is, "I don't think duct tape will fix it." -- Ed Smylie, NASA engineer for Apollo 13 From libor at topspin.com Fri May 6 16:27:13 2005 From: libor at topspin.com (Libor Michalek) Date: Fri, 6 May 2005 16:27:13 -0700 Subject: [openib-general] [PATCH][SDP] AIO buffer corruption In-Reply-To: <20050506124112.A26823@topspin.com>; from libor@topspin.com on Fri, May 06, 2005 at 12:41:12PM -0700 References: <200505021548.52053.steve@wooding.uklinux.net> <20050503130243.C21837@topspin.com> <427BA94A.20106@wooding.uklinux.net> <20050506124112.A26823@topspin.com> Message-ID: <20050506162713.C26823@topspin.com> On Fri, May 06, 2005 at 12:41:12PM -0700, Libor Michalek wrote: > On Fri, May 06, 2005 at 06:28:42PM +0100, Steven Wooding wrote: > > > I'm away for two weeks, so I'll get back to you with any further info > > you require when I get back. > > Here is the patch to fix the problem, the fix exposes another problem, > where the send pipeline stalls and is not restarted, which I'm working on. > The problem was caused by copying more data from the iocb then it contained > sending the size of the iocb to negative. Here is a patch to fix the pipeline stall I was seeing. It removes a check which should not be made. When moving data from the send_queue to the posted queue, there is no need to check the amount of queued data. Signed-off-by: Libor Michalek Index: sdp_send.c =================================================================== --- sdp_send.c (revision 2271) +++ sdp_send.c (working copy) @@ -768,16 +768,13 @@ { struct sdpc_buff *buff; int result; - int copy; if (conn->src_sent > 0) return ENOBUFS; /* * loop through queued buffers and copy them to the destination */ - copy = min(sdp_inet_write_space(conn, 0), iocb->len); - - while (copy > 0 && + while (iocb->len > 0 && conn->r_recv_bf > 2 && conn->send_cq_size > conn->s_wq_size) { /* @@ -812,8 +809,6 @@ } conn->send_qud += buff->data_size; - copy -= buff->data_size; - conn->nond_send++; /* * transmision time. An update of send_pipe is not needed, From blist at aon.at Fri May 6 16:51:35 2005 From: blist at aon.at (Bernhard Fischer) Date: Sat, 7 May 2005 01:51:35 +0200 Subject: [openib-general] [patch] misc trivia nitpicks 04 In-Reply-To: <20050506210807.GE23958@esmail.cup.hp.com> References: <20050506003942.GE15688@aon.at> <1115392431.4476.135.camel@localhost.localdomain> <20050506174104.GI15688@aon.at> <20050506194330.GB23958@esmail.cup.hp.com> <20050506202551.GJ15688@aon.at> <20050506210807.GE23958@esmail.cup.hp.com> Message-ID: <20050506235135.GM15688@aon.at> On Fri, May 06, 2005 at 02:08:07PM -0700, Grant Grundler wrote: >On Fri, May 06, 2005 at 10:25:51PM +0200, Bernhard Fischer wrote: >> I've also inlined double_compar > >Eh? That's pointless. erm, right. Other notes: getopt 'i' fills in ib_port but checks port, it seems. Didn't look closely. The segfault i was seeing did came from dlist_start(). iters is off by one (stops at 998 when asked for 1000). And the timings look odd towards the end: 987, 4.944600 988, 4.977098 989, 5.158333 990, 5.482057 991, 6.955688 992, 7.680629 993, 8.210586 994, 8.759292 995, 9.820456 996, 26.436613 997, 72.905357 998, 704572144.206260 From tduffy at sun.com Fri May 6 16:56:33 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 06 May 2005 16:56:33 -0700 Subject: [openib-general] sdp_pass.c: sdp_cm_listen_lookup() In-Reply-To: <20050506223506.GL15688@aon.at> References: <20050506223506.GL15688@aon.at> Message-ID: <1115423793.19661.4.camel@duffman> On Sat, 2005-05-07 at 00:35 +0200, Bernhard Fischer wrote: > Hi, > > I get compile errors in sdp_cm_listen_lookup for these being unknown: > > sk_debug, sk_localroute, sk_rcvtstamp I posted a patch to fix these to the list. It should be applied to the tree once 2.6.12 is out. In the mean time, the tree should compile cleanly against 2.6.11. Libor hinted at another problem all together and that is the new socket slab allocator. I posted a patch to update SDP to use the new mechanism as well. If you cannot find the patches in the archives, I will resend. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From blist at aon.at Fri May 6 16:58:36 2005 From: blist at aon.at (Bernhard Fischer) Date: Sat, 7 May 2005 01:58:36 +0200 Subject: [openib-general] sdp_pass.c: sdp_cm_listen_lookup() In-Reply-To: <20050506155750.B26823@topspin.com> References: <20050506223506.GL15688@aon.at> <20050506155750.B26823@topspin.com> Message-ID: <20050506235836.GN15688@aon.at> On Fri, May 06, 2005 at 03:57:50PM -0700, Libor Michalek wrote: >On Sat, May 07, 2005 at 12:35:06AM +0200, Bernhard Fischer wrote: >> Hi, >> >> I get compile errors in sdp_cm_listen_lookup for these being unknown: >> >> sk_debug, sk_localroute, sk_rcvtstamp > > Which version of the kernel are you using, and is this the only error >that you get? 2.6.12-rc, gen2 rev. 2270. I did not see any other obvious compile errors. I did not find any other occurance of the abovementioned three, is this correct? > >-Libor > From blist at aon.at Fri May 6 17:01:44 2005 From: blist at aon.at (Bernhard Fischer) Date: Sat, 7 May 2005 02:01:44 +0200 Subject: [openib-general] sdp_pass.c: sdp_cm_listen_lookup() In-Reply-To: <1115423793.19661.4.camel@duffman> References: <20050506223506.GL15688@aon.at> <1115423793.19661.4.camel@duffman> Message-ID: <20050507000144.GO15688@aon.at> On Fri, May 06, 2005 at 04:56:33PM -0700, Tom Duffy wrote: >On Sat, 2005-05-07 at 00:35 +0200, Bernhard Fischer wrote: >> Hi, >> >> I get compile errors in sdp_cm_listen_lookup for these being unknown: >> >> sk_debug, sk_localroute, sk_rcvtstamp > >I posted a patch to fix these to the list. It should be applied to the >tree once 2.6.12 is out. In the mean time, the tree should compile >cleanly against 2.6.11. Libor hinted at another problem all together >and that is the new socket slab allocator. I posted a patch to update >SDP to use the new mechanism as well. If you cannot find the patches in >the archives, I will resend. Thanks for the hint. I'll find them in the archives. From tduffy at sun.com Fri May 6 17:03:18 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 06 May 2005 17:03:18 -0700 Subject: [openib-general] [PATCH] kDAPL: remove typedef DAT_PROVIDER Message-ID: <1115424198.19661.11.camel@duffman> James, thanks for applying my last patch. You know where I am going with this... this is the first in what will be a huge amount of patches. I am happy to go through and send them all to the list, but it might be quicker and easier for you to do it directly to your tree for all the structs and enums. Sup to you. Remove the typedef DAT_PROVIDER and use struct dat_provider directly. Signed-off-by: Tom Duffy Index: dat-provider/dapl_module.c =================================================================== --- dat-provider/dapl_module.c (revision 2274) +++ dat-provider/dapl_module.c (working copy) @@ -126,7 +126,7 @@ module_exit(dapl_fini); void DAT_PROVIDER_INIT_FUNC_NAME(const DAT_PROVIDER_INFO * provider_info, struct ib_device *device, u8 port) { - DAT_PROVIDER *provider; + struct dat_provider *provider; DAPL_HCA *hca_ptr; DAT_RETURN dat_status; @@ -178,7 +178,7 @@ void DAT_PROVIDER_INIT_FUNC_NAME(const D */ void DAT_PROVIDER_FINI_FUNC_NAME(const DAT_PROVIDER_INFO * provider_info) { - DAT_PROVIDER *provider; + struct dat_provider *provider; DAT_RETURN dat_status; dat_status = Index: dat-provider/dapl_ia_util.c =================================================================== --- dat-provider/dapl_ia_util.c (revision 2274) +++ dat-provider/dapl_ia_util.c (working copy) @@ -61,7 +61,7 @@ void dapl_ia_release_hca(DAPL_HCA * hca_ * none * */ -DAPL_IA *dapl_ia_alloc(DAT_PROVIDER * provider, DAPL_HCA * hca_ptr) +DAPL_IA *dapl_ia_alloc(struct dat_provider *provider, DAPL_HCA * hca_ptr) { DAPL_IA *ia_ptr; Index: dat-provider/dapl_provider.c =================================================================== --- dat-provider/dapl_provider.c (revision 2274) +++ dat-provider/dapl_provider.c (working copy) @@ -51,7 +51,7 @@ DAPL_PROVIDER_LIST g_dapl_provider_list; * the function table for this provider */ -DAT_PROVIDER g_dapl_provider_template = { +struct dat_provider g_dapl_provider_template = { NULL, NULL, &dapl_ia_open, @@ -211,7 +211,8 @@ DAT_COUNT dapl_provider_list_size(void) return g_dapl_provider_list.size; } -DAT_RETURN dapl_provider_list_insert(const char *name, DAT_PROVIDER ** p_data) +DAT_RETURN dapl_provider_list_insert(const char *name, + struct dat_provider **p_data) { DAPL_PROVIDER_LIST_NODE *cur_node, *prev_node, *next_node; DAT_RETURN status; @@ -265,7 +266,8 @@ DAT_RETURN dapl_provider_list_insert(con return status; } -DAT_RETURN dapl_provider_list_search(const char *name, DAT_PROVIDER ** p_data) +DAT_RETURN dapl_provider_list_search(const char *name, + struct dat_provider **p_data) { DAPL_PROVIDER_LIST_NODE *cur_node; DAT_RETURN status; Index: dat-provider/dapl_ia_util.h =================================================================== --- dat-provider/dapl_ia_util.h (revision 2274) +++ dat-provider/dapl_ia_util.h (working copy) @@ -39,7 +39,7 @@ #include "dapl.h" -DAPL_IA *dapl_ia_alloc(DAT_PROVIDER * provider, DAPL_HCA * hca_ptr); +DAPL_IA *dapl_ia_alloc(struct dat_provider *provider, DAPL_HCA * hca_ptr); DAT_RETURN dapl_ia_abrupt_close(DAPL_IA * ia_ptr); Index: dat-provider/dapl_provider.h =================================================================== --- dat-provider/dapl_provider.h (revision 2274) +++ dat-provider/dapl_provider.h (working copy) @@ -48,7 +48,7 @@ typedef struct DAPL_PROVIDER_LIST_NODE { char name[DAT_NAME_MAX_LENGTH]; - DAT_PROVIDER data; + struct dat_provider data; struct DAPL_PROVIDER_LIST_NODE *next; struct DAPL_PROVIDER_LIST_NODE *prev; } DAPL_PROVIDER_LIST_NODE; @@ -66,7 +66,7 @@ typedef struct DAPL_PROVIDER_LIST { *********************************************************************/ extern DAPL_PROVIDER_LIST g_dapl_provider_list; -extern DAT_PROVIDER g_dapl_provider_template; +extern struct dat_provider g_dapl_provider_template; /********************************************************************* * * @@ -81,10 +81,10 @@ extern DAT_RETURN dapl_provider_list_des extern DAT_COUNT dapl_provider_list_size(void); extern DAT_RETURN -dapl_provider_list_insert(const char *name, DAT_PROVIDER ** p_data); +dapl_provider_list_insert(const char *name, struct dat_provider **p_data); extern DAT_RETURN -dapl_provider_list_search(const char *name, DAT_PROVIDER ** p_data); +dapl_provider_list_search(const char *name, struct dat_provider **p_data); extern DAT_RETURN dapl_provider_list_remove(const char *name); Index: dat-provider/dapl_ia_open.c =================================================================== --- dat-provider/dapl_ia_open.c (revision 2274) +++ dat-provider/dapl_ia_open.c (working copy) @@ -64,7 +64,7 @@ dapl_ia_open(const DAT_NAME_PTR name, DAT_IA_HANDLE * ia_handle_ptr) { DAT_RETURN dat_status; - DAT_PROVIDER *provider; + struct dat_provider *provider; DAPL_HCA *hca_ptr; DAPL_IA *ia_ptr; DAPL_EVD *evd_ptr; Index: dat-provider/dapl.h =================================================================== --- dat-provider/dapl.h (revision 2274) +++ dat-provider/dapl.h (working copy) @@ -228,7 +228,7 @@ struct dapl_hca { /* DAPL Objects always have the following header */ struct dapl_header { - DAT_PROVIDER *provider; /* required by DAT - must be first */ + struct dat_provider *provider; /* required by DAT - must be first */ DAPL_MAGIC magic; /* magic number for verification */ DAT_HANDLE_TYPE handle_type; /* struct type */ DAPL_IA *owner_ia; /* ia which owns this stuct */ Index: dat/api.c =================================================================== --- dat/api.c (revision 2274) +++ dat/api.c (working copy) @@ -627,7 +627,7 @@ EXPORT_SYMBOL(dat_ia_openv); DAT_RETURN dat_ia_close(DAT_IA_HANDLE ia_handle, DAT_CLOSE_FLAGS flags) { - DAT_PROVIDER *provider; + struct dat_provider *provider; DAT_PROVIDER_ATTR provider_attr; DAT_RETURN status; const char *ia_name; @@ -718,7 +718,7 @@ EXPORT_SYMBOL(dat_strerror); ***********************************************************************/ DAT_RETURN -dat_registry_add_provider(const DAT_PROVIDER * provider, +dat_registry_add_provider(const struct dat_provider *provider, const DAT_PROVIDER_INFO * provider_info) { DAT_DR_ENTRY entry; @@ -753,7 +753,7 @@ EXPORT_SYMBOL(dat_registry_remove_provid ***********************************************************************/ DAT_RETURN -dat_registry_remove_provider(const DAT_PROVIDER * provider, +dat_registry_remove_provider(const struct dat_provider *provider, const DAT_PROVIDER_INFO * provider_info) { dat_dbg_print(DAT_DBG_CLASS_PROVIDER_API, Index: dat/dat.h =================================================================== --- dat/dat.h (revision 2274) +++ dat/dat.h (working copy) @@ -1272,7 +1272,7 @@ typedef DAT_RETURN (*DAT_EVD_KCREATE_FUN typedef DAT_RETURN (*DAT_EVD_MODIFY_UPCALL_FUNC) ( DAT_EVD_HANDLE, DAT_UPCALL_POLICY, const DAT_UPCALL_OBJECT *); -typedef struct dat_provider { +struct dat_provider { const char *device_name; DAT_PVOID extension; @@ -1350,7 +1350,7 @@ typedef struct dat_provider { DAT_SRQ_QUERY_FUNC srq_query_func; DAT_SRQ_RESIZE_FUNC srq_resize_func; DAT_SRQ_SET_LW_FUNC srq_set_lw_func; -} DAT_PROVIDER; +}; /*************************************************************** * @@ -1364,10 +1364,10 @@ typedef struct dat_provider { */ extern DAT_RETURN dat_registry_add_provider( - const DAT_PROVIDER *, const DAT_PROVIDER_INFO *); + const struct dat_provider *, const DAT_PROVIDER_INFO *); extern DAT_RETURN dat_registry_remove_provider( - const DAT_PROVIDER *, const DAT_PROVIDER_INFO *); + const struct dat_provider *, const DAT_PROVIDER_INFO *); /* * DAT registry functions for consumers @@ -1391,7 +1391,7 @@ extern DAT_RETURN dat_registry_list_prov /* * inline functions for consumers */ -#define DAT_HANDLE_TO_PROVIDER(handle) (*(DAT_PROVIDER **)(handle)) +#define DAT_HANDLE_TO_PROVIDER(handle) (*(struct dat_provider **)(handle)) #define DAT_CALL_PROVIDER_FUNC(func, handle, ...) \ DAT_HANDLE_TO_PROVIDER(handle)->func(handle, ##__VA_ARGS__) From libor at topspin.com Fri May 6 17:03:44 2005 From: libor at topspin.com (Libor Michalek) Date: Fri, 6 May 2005 17:03:44 -0700 Subject: [openib-general] Re: [PATCH] sdp: get rid of in_atomic In-Reply-To: <20050505120455.GJ30653@mellanox.co.il>; from mst@mellanox.co.il on Thu, May 05, 2005 at 03:04:55PM +0300 References: <20050505120455.GJ30653@mellanox.co.il> Message-ID: <20050506170344.D26823@topspin.com> On Thu, May 05, 2005 at 03:04:55PM +0300, Michael S. Tsirkin wrote: > Hello, Libor! > Here's a rediff against the latest bits. Could you please comment on > this patch? > > sdp_kmap and sdp_kunmap use in_atomic and kmap_atomic in an > unsafe manner. The following patch replaces all uses of these macros with > kmap_atomic on IRQ0/IRQ1 slot, while disabling local interrupts. Thanks Michael, applied and checked in. I hadn't checked it in the first time around because I was chasing the corruption issue and didn't want to add yet another variable. -Libor From libor at topspin.com Fri May 6 17:22:46 2005 From: libor at topspin.com (Libor Michalek) Date: Fri, 6 May 2005 17:22:46 -0700 Subject: [openib-general] [patch] misc trivia nitpicks 01 In-Reply-To: <20050505170956.GU15688@aon.at>; from blist@aon.at on Thu, May 05, 2005 at 07:09:56PM +0200 References: <20050505170956.GU15688@aon.at> Message-ID: <20050506172246.E26823@topspin.com> On Thu, May 05, 2005 at 07:09:56PM +0200, Bernhard Fischer wrote: > Hi, > > please consider attached. > > diffstat ./gen2.2257.cow-20050505-1902.diff > linux-kernel/infiniband/ulp/sdp/sdp_actv.c | 27 +-- Thanks, applied and commited. -Libor From iod00d at hp.com Fri May 6 17:28:36 2005 From: iod00d at hp.com (Grant Grundler) Date: Fri, 6 May 2005 17:28:36 -0700 Subject: [openib-general] [patch] misc trivia nitpicks 04 In-Reply-To: <20050506235135.GM15688@aon.at> References: <20050506003942.GE15688@aon.at> <1115392431.4476.135.camel@localhost.localdomain> <20050506174104.GI15688@aon.at> <20050506194330.GB23958@esmail.cup.hp.com> <20050506202551.GJ15688@aon.at> <20050506210807.GE23958@esmail.cup.hp.com> <20050506235135.GM15688@aon.at> Message-ID: <20050507002836.GI23958@esmail.cup.hp.com> On Sat, May 07, 2005 at 01:51:35AM +0200, Bernhard Fischer wrote: > Other notes: > > getopt 'i' fills in ib_port but checks port, it seems. Didn't look > closely. Yeah, that looks wrong to me too. I think it's just trying to validate that ib_port is a reasonable number. > iters is off by one (stops at 998 when asked for 1000). Yes, but not like you think. "iters" runs from 0-998 - ie 999 samples. Is adding one to the -H output the right answer? (ie run 1-999) We asked for 1000 "exchanges". The measured the time between them results in one less sample. The usage says -n, --iters= number of exchanges (at least 2, default 1000) I'm inclined to leave that alone. ... > 994, 8.759292 > 995, 9.820456 > 996, 26.436613 > 997, 72.905357 > 998, 704572144.206260 998 is probably garbage. I'll sort that out on the ia64 box. Several things could be causing this besides program error: - process got moved to another cpu between samples. taskset should take care of this. - CPU clock rolled over and the conversion from unsigned long long to double happened before the difference was taken. And apologies for not posting the -02 tarball...got distracted by some other issues. Almost done... hth, grant From tduffy at sun.com Fri May 6 17:27:49 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 06 May 2005 17:27:49 -0700 Subject: [openib-general] kdapltest works again!, still having slab corruption In-Reply-To: <1115413920.4476.1.camel@localhost.localdomain> References: <1115411060.4476.346.camel@localhost.localdomain> <1115413920.4476.1.camel@localhost.localdomain> Message-ID: <1115425669.19661.14.camel@duffman> On Fri, 2005-05-06 at 17:12 -0400, Hal Rosenstock wrote: > On Fri, 2005-05-06 at 16:36, James Lentini wrote: > > Let me know if you still have trouble. > > Works for me. Hope it does for Tom. Quit test works for me now! But, I am still seeing this slab corruption every time I run the test, on the active side: Slab corruption: start=ffff81002353edb0, len=288 Redzone: 0x5a2cf071/0x5a2cf071. Last user: [](req_comp_work+0x54/0x90 [ib_at]) 040: 00 00 00 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b 110: 00 00 00 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b a5 Prev obj: start=ffff81002353ec78, len=288 Redzone: 0x5a2cf071/0x5a2cf071. Last user: [<0000000000000000>](0x0) 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b Slab corruption: start=ffff8100778b2eb8, len=312 Redzone: 0x5a2cf071/0x5a2cf071. Last user: [](req_comp_work+0x42/0x90 [ib_at]) 050: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 00 00 00 00 120: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 00 00 00 00 Prev obj: start=ffff8100778b2d68, len=312 Redzone: 0x5a2cf071/0x5a2cf071. Last user: [<0000000000000000>](0x0) 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From libor at topspin.com Fri May 6 17:30:50 2005 From: libor at topspin.com (Libor Michalek) Date: Fri, 6 May 2005 17:30:50 -0700 Subject: [openib-general] [patch] misc trivia nitpicks 03 In-Reply-To: <20050505192614.GW15688@aon.at>; from blist@aon.at on Thu, May 05, 2005 at 09:26:14PM +0200 References: <20050505192614.GW15688@aon.at> Message-ID: <20050506173050.F26823@topspin.com> On Thu, May 05, 2005 at 09:26:14PM +0200, Bernhard Fischer wrote: > Hi, > > What's the status of SDP_EXPECT ? > will it be submitted upstream or cleaned up beforehand? Just curious. It should be cleaned up before hand. Each one needs to be looked at, and either the return value should be used, with the error passed down the call stack, or the function should be a void return. > - fold sdp_conn_internal_lock into sdp_conn_lock. > - fold sdp_conn_internal_unlock into sdp_conn_unlock. > > PS: those folds are completely untested.. Libor, anyone else? > thank you, This was discussed earlier. I wasn't planning on folding internal_lock() into lock(), since the internal call is only made as an uncommon exception path when there is lock contention. This is just like lock_sock/release_sock in net/core/sock.c and include/net/sock.h. I would use those locks, but they are too closely tied to skb processing, specifically the sk_backlog queue. -Libor From libor at topspin.com Fri May 6 17:38:15 2005 From: libor at topspin.com (Libor Michalek) Date: Fri, 6 May 2005 17:38:15 -0700 Subject: [openib-general] [patch] misc trivia nitpicks 04 In-Reply-To: <20050506003942.GE15688@aon.at>; from blist@aon.at on Fri, May 06, 2005 at 02:39:42AM +0200 References: <20050506003942.GE15688@aon.at> Message-ID: <20050506173815.G26823@topspin.com> On Fri, May 06, 2005 at 02:39:42AM +0200, Bernhard Fischer wrote: > Hi, > > - s/incomm/incom/g > > +++ gen2.2257/trunk/src/linux-kernel/infiniband/core/ucm.c > +++ gen2.2257/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_inet.c > +++ gen2.2257/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_pass.c > +++ gen2.2257/trunk/src/userspace/examples/aio/README Applied and commited. -Libor From libor at topspin.com Fri May 6 17:43:10 2005 From: libor at topspin.com (Libor Michalek) Date: Fri, 6 May 2005 17:43:10 -0700 Subject: [openib-general] cm_id state information In-Reply-To: <427BB731.50800@ichips.intel.com>; from mshefty@ichips.intel.com on Fri, May 06, 2005 at 11:28:01AM -0700 References: <427A617C.1060607@ichips.intel.com> <427BB731.50800@ichips.intel.com> Message-ID: <20050506174310.H26823@topspin.com> On Fri, May 06, 2005 at 11:28:01AM -0700, Sean Hefty wrote: > James Lentini wrote: > > Is there documentation or a piece of sample code that shows how to > > interact with the CM correctly? > > The only sample code is cmpost, under the util directory. You could > probably use SDP as well. Also, I'm currently working with Hal to make sure > that kDAPL works properly with the CM. It will just take a little while > before a patch is ready. I thought James' question was in the context of not using cm_id->state to drive the CM state machine, which SDP is currently doing, so it would make a bad example. -Libor From libor at topspin.com Fri May 6 17:54:15 2005 From: libor at topspin.com (Libor Michalek) Date: Fri, 6 May 2005 17:54:15 -0700 Subject: [openib-general] Re: [PATCH] sdp: replace mlock with get_user_pages In-Reply-To: <20050505110158.GH30653@mellanox.co.il>; from mst@mellanox.co.il on Thu, May 05, 2005 at 02:01:58PM +0300 References: <20050505110158.GH30653@mellanox.co.il> Message-ID: <20050506175415.I26823@topspin.com> On Thu, May 05, 2005 at 02:01:58PM +0300, Michael S. Tsirkin wrote: > Hello, Libor! > The following patch replaces the mlock hack with call > to get_user_pages. Since the application could have forked > while an iocb is outstanding, when an iocb is done > I do get_user_pages for a second time and copy data if > the physical address has changed. > > Thus, changing ulimit is no longer required to get aio > working, processes are also allowed to fork and to call mlock/munlock > on the buffer. > > Tested by the ttcp.aio benchmark, and works fine for me. In the latest kernel what happens to a page that's held with a get_user_page reference on fork? Which is the only case left. Since there is an iocv flag parameter it would be prefered to use a bit there instead of is_receive. Also, please leave spaces around '='. Finally it would be nice if we could get rid of having to schedule a task to unlock the iocb in complete, I've noticed this having a negative effect on system CPU utilization on older kernels. -Libor > Signed-off-by: Michael S. Tsirkin > > Index: ulp/sdp/sdp_send.c > =================================================================== > --- ulp/sdp/sdp_send.c (revision 2235) > +++ ulp/sdp/sdp_send.c (working copy) > @@ -2197,6 +2202,7 @@ skip: /* entry point for IOCB based tran > iocb->req = req; > iocb->key = req->ki_key; > iocb->addr = (unsigned long)msg->msg_iov->iov_base - copied; > + iocb->is_receive = 0; > > req->ki_cancel = sdp_inet_write_cancel; > > Index: ulp/sdp/sdp_recv.c > =================================================================== > --- ulp/sdp/sdp_recv.c (revision 2235) > +++ ulp/sdp/sdp_recv.c (working copy) > @@ -1447,6 +1448,7 @@ int sdp_inet_recv(struct kiocb *req, st > iocb->key = req->ki_key; > iocb->addr = ((unsigned long)msg->msg_iov->iov_base - > copied); > + iocb->is_receive = 1; > > req->ki_cancel = sdp_inet_read_cancel; > > Index: ulp/sdp/sdp_iocb.c > =================================================================== > --- ulp/sdp/sdp_iocb.c (revision 2235) > +++ ulp/sdp/sdp_iocb.c (working copy) > @@ -1,5 +1,6 @@ > /* > * Copyright (c) 2005 Topspin Communications. All rights reserved. > + * Copyright (c) 2005 Mellanox Technologies Ltd. All rights reserved. > * > * This software is available to you under a choice of one of two > * licenses. You may choose to be licensed under the terms of the GNU > @@ -31,389 +32,197 @@ > * > * $Id$ > */ > - > +#include > #include "sdp_main.h" > > static kmem_cache_t *sdp_iocb_cache = NULL; > > -/* > - * memory locking functions > - */ > -#include > - > -typedef int (*do_mlock_ptr_t)(unsigned long, size_t, int); > -static do_mlock_ptr_t mlock_ptr = NULL; > - > -/* > - * do_iocb_unlock - unlock the memory for an IOCB > - */ > -static int do_iocb_unlock(struct sdpc_iocb *iocb) > -{ > - struct vm_area_struct *vma; > - > - vma = find_vma(iocb->mm, (iocb->addr & PAGE_MASK)); > - if (!vma) > - sdp_warn("No VMA for IOCB <%lx:%Zu> unlock", > - iocb->addr, iocb->size); > - > - while (vma) { > - sdp_dbg_data(NULL, > - "unmark <%lx> <%p> <%08lx:%08lx> <%08lx> <%ld>", > - iocb->addr, vma, vma->vm_start, vma->vm_end, > - vma->vm_flags, (long)vma->vm_private_data); > - > - spin_lock(&iocb->mm->page_table_lock); > - /* > - * if there are no more references to the vma > - */ > - vma->vm_private_data--; > - > - if (!vma->vm_private_data) { > - /* > - * modify VM flags. > - */ > - vma->vm_flags &= ~(VM_DONTCOPY|VM_LOCKED); > - /* > - * adjust locked page count > - */ > - vma->vm_mm->locked_vm -= ((vma->vm_end - > - vma->vm_start) >> > - PAGE_SHIFT); > - } > - > - spin_unlock(&iocb->mm->page_table_lock); > - /* > - * continue if the buffer continues onto the next vma > - */ > - if ((iocb->addr + iocb->size) > vma->vm_end) > - vma = vma->vm_next; > - else > - vma = NULL; > - } > - > - return 0; > +static void sdp_copy_one_page(struct page *from, struct page* to, > + unsigned long iocb_addr, size_t iocb_size, > + unsigned long uaddr) > +{ > + size_t size_left = iocb_addr + iocb_size - uaddr; > + size_t size = min(size_left,PAGE_SIZE); > + unsigned long offset = uaddr % PAGE_SIZE; > + unsigned long flags; > + > + void* fptr; > + void* tptr; > + > + local_irq_save(flags); > + fptr = kmap_atomic(from, KM_IRQ0); > + tptr = kmap_atomic(to, KM_IRQ1); > + > + memcpy(tptr + offset, fptr + offset, size); > + > + kunmap_atomic(tptr, KM_IRQ1); > + kunmap_atomic(fptr, KM_IRQ0); > + local_irq_restore(flags); > + set_page_dirty_lock(to); > } > > /* > * sdp_iocb_unlock - unlock the memory for an IOCB > + * Copy if pages moved since. > + * TODO: is this needed? > */ > int sdp_iocb_unlock(struct sdpc_iocb *iocb) > { > int result; > + struct page ** pages = NULL; > + unsigned long uaddr; > + int i; > + > > - /* > - * check if IOCB is locked. > - */ > if (!(iocb->flags & SDP_IOCB_F_LOCKED)) > return 0; > - /* > - * spin lock since this could be from interrupt context. > - */ > - down_write(&iocb->mm->mmap_sem); > - > - result = do_iocb_unlock(iocb); > > - up_write(&iocb->mm->mmap_sem); > + /* For read, unlock and we are done */ > + if (!iocb->is_receive) { > + for (i = 0;i < iocb->page_count; ++i) > + put_page(iocb->page_array[i]); > + goto done; > + } > > - kfree(iocb->page_array); > - kfree(iocb->addr_array); > + /* For write, we must check the virtual pages did not get remapped */ > > - iocb->page_array = NULL; > - iocb->addr_array = NULL; > - iocb->mm = NULL; > - /* > - * mark IOCB unlocked. > - */ > - iocb->flags &= ~SDP_IOCB_F_LOCKED; > + /* As an optimisation (to avoid scanning the vma tree each time), > + * try to get all pages in one go. */ > + /* TODO: use cache for allocations? Allocate by chunks? */ > > - return result; > -} > + pages = kmalloc((sizeof(struct page *) * iocb->page_count), GFP_KERNEL); > > -/* > - * sdp_iocb_page_save - save page information for an IOCB > - */ > -static int sdp_iocb_page_save(struct sdpc_iocb *iocb) > -{ > - unsigned int counter; > - unsigned long addr; > - size_t size; > - int result = -ENOMEM; > - struct page *page; > - unsigned long pfn; > - pgd_t *pgd; > - pud_t *pud; > - pmd_t *pmd; > - pte_t *ptep; > - pte_t pte; > + down_read(&iocb->mm->mmap_sem); > > - if (iocb->page_count <= 0 || iocb->size <= 0 || !iocb->addr) > - return -EINVAL; > - /* > - * create array to hold page value which are later needed to register > - * the buffer with the HCA > - */ > - iocb->addr_array = kmalloc((sizeof(u64) * iocb->page_count), > - GFP_KERNEL); > - if (!iocb->addr_array) > - goto err_addr; > + if (pages) { > + result=get_user_pages(iocb->tsk, iocb->mm, > + iocb->addr, > + iocb->page_count , iocb->is_receive, 0, > + pages, NULL); > > - iocb->page_array = kmalloc((sizeof(struct page *) * iocb->page_count), > - GFP_KERNEL); > - if (!iocb->page_array) > - goto err_page; > - /* > - * iocb->addr - buffer start address > - * iocb->size - buffer length > - * addr - page aligned > - * size - page multiple > - */ > - addr = iocb->addr & PAGE_MASK; > - size = PAGE_ALIGN(iocb->size + (iocb->addr & ~PAGE_MASK)); > + if (result != iocb->page_count) { > + kfree(pages); > + pages = NULL; > + } > + } > > - iocb->page_offset = iocb->addr - addr; > - /* > - * Find pages used within the buffer which will then be registered > - * for RDMA > - */ > - spin_lock(&iocb->mm->page_table_lock); > + for (i = 0, uaddr = iocb->addr; i < iocb->page_count; > + ++i, uaddr = (uaddr & PAGE_MASK) + PAGE_SIZE) > + { > + struct page* page; > + set_page_dirty_lock(iocb->page_array[i]); > + > + if (pages) > + page = pages[i]; > + else { > + result=get_user_pages(iocb->tsk, iocb->mm, > + uaddr & PAGE_MASK, > + 1 , 1, 0, &page, NULL); > + if (result != 1) { > + page = NULL; > + } > + } > > - for (counter = 0; > - size > 0; > - counter++, addr += PAGE_SIZE, size -= PAGE_SIZE) { > - pgd = pgd_offset_gate(iocb->mm, addr); > - if (!pgd || pgd_none(*pgd)) > - break; > - > - pud = pud_offset(pgd, addr); > - if (!pud || pud_none(*pud)) > - break; > - > - pmd = pmd_offset(pud, addr); > - if (!pmd || pmd_none(*pmd)) > - break; > - > - ptep = pte_offset_map(pmd, addr); > - if (!ptep) > - break; > - > - pte = *ptep; > - pte_unmap(ptep); > - > - if (!pte_present(pte)) > - break; > - > - pfn = pte_pfn(pte); > - if (!pfn_valid(pfn)) > - break; > - > - page = pfn_to_page(pfn); > + if (page && iocb->page_array[i] != page) > + sdp_copy_one_page(iocb->page_array[i], page, > + iocb->addr, iocb->size, uaddr); > > - iocb->page_array[counter] = page; > - iocb->addr_array[counter] = page_to_phys(page); > + if (page) > + put_page(page); > + put_page(iocb->page_array[i]); > } > > - spin_unlock(&iocb->mm->page_table_lock); > - > - if (size > 0) { > - result = -EFAULT; > - goto err_find; > - } > + up_read(&iocb->mm->mmap_sem); > > - return 0; > -err_find: > - > - kfree(iocb->page_array); > - iocb->page_array = NULL; > -err_page: > + if (pages) > + kfree(pages); > > +done: > + kfree(iocb->page_array); > kfree(iocb->addr_array); > + > + iocb->page_array = NULL; > iocb->addr_array = NULL; > -err_addr: > + iocb->mm = NULL; > + iocb->tsk = NULL; > > - return result; > + iocb->flags &= ~SDP_IOCB_F_LOCKED; > + > + return 0; > } > > /* > * sdp_iocb_lock - lock the memory for an IOCB > + * We do not take a reference on the mm, AIO handles this for us. > */ > int sdp_iocb_lock(struct sdpc_iocb *iocb) > { > - struct vm_area_struct *vma; > - kernel_cap_t real_cap; > - unsigned long limit; > int result = -ENOMEM; > unsigned long addr; > - size_t size; > - > - /* > - * mark IOCB as locked. We do not take a reference on the mm, AIO > - * handles this for us. > - */ > - iocb->flags |= SDP_IOCB_F_LOCKED; > - iocb->mm = current->mm; > + size_t size; > + int i; > /* > - * save and raise capabilities > + * iocb->addr - buffer start address > + * iocb->size - buffer length > + * addr - page aligned > + * size - page multiple > */ > - real_cap = cap_t(current->cap_effective); > - cap_raise(current->cap_effective, CAP_IPC_LOCK); > - > - size = PAGE_ALIGN(iocb->size + (iocb->addr & ~PAGE_MASK)); > addr = iocb->addr & PAGE_MASK; > + size = PAGE_ALIGN(iocb->size + (iocb->addr & ~PAGE_MASK)); > > + iocb->page_offset = iocb->addr - addr; > + > iocb->page_count = size >> PAGE_SHIFT; > - > - limit = current->signal->rlim[RLIMIT_MEMLOCK].rlim_cur; > - limit >>= PAGE_SHIFT; > - /* > - * lock the mm, if within the limit lock the address range. > - */ > - down_write(&iocb->mm->mmap_sem); > - > - if (!((iocb->page_count + current->mm->locked_vm) > limit)) > - result = (*mlock_ptr)(addr, size, 1); > /* > - * process result > - */ > - if (result) { > - sdp_dbg_err("VMA lock <%lx:%Zu> error <%d> <%d:%lu:%lu>", > - iocb->addr, iocb->size, result, > - iocb->page_count, iocb->mm->locked_vm, limit); > - goto err_lock; > - } > - /* > - * look up the head of the vma queue, loop through the vmas, marking > - * them do not copy, reference counting, and saving them. > + * create array to hold page value which are later needed to register > + * the buffer with the HCA > */ > - vma = find_vma(iocb->mm, addr); > - if (!vma) > - /* > - * sanity check. > - */ > - sdp_warn("No VMA for IOCB! <%lx:%Zu> lock", > - iocb->addr, iocb->size); > > - while (vma) { > - spin_lock(&iocb->mm->page_table_lock); > + /* TODO: use cache for allocations? Allocate by chunks? */ > + iocb->addr_array = kmalloc((sizeof(u64) * iocb->page_count), > + GFP_KERNEL); > + if (!iocb->addr_array) > + goto err_addr; > > - if (!(VM_LOCKED & vma->vm_flags)) > - sdp_warn("Unlocked vma! <%08lx>", vma->vm_flags); > + iocb->page_array = kmalloc((sizeof(struct page *) * iocb->page_count), > + GFP_KERNEL); > + if (!iocb->page_array) > + goto err_page; > > - if (PAGE_SIZE < (unsigned long)vma->vm_private_data) > - sdp_dbg_err("VMA: private daya in use! <%08lx>", > - (unsigned long)vma->vm_private_data); > - > - vma->vm_flags |= VM_DONTCOPY; > - vma->vm_private_data++; > + down_read(¤t->mm->mmap_sem); > > - spin_unlock(&iocb->mm->page_table_lock); > + result=get_user_pages(current, current->mm, iocb->addr, > + iocb->page_count , iocb->is_receive, 0, > + iocb->page_array, NULL); > + > + up_read(¤t->mm->mmap_sem); > + > + if (result != iocb->page_count) { > + sdp_dbg_err("unable to lock <%lx:%Zu> error <%d> <%d>", > + iocb->addr, iocb->size, result, iocb->page_count); > + goto err_get; > + } > > - sdp_dbg_data(NULL, > - "mark <%lx> <0x%p> <%08lx:%08lx> <%08lx> <%ld>", > - iocb->addr, vma, vma->vm_start, vma->vm_end, > - vma->vm_flags, (long)vma->vm_private_data); > + iocb->flags |= SDP_IOCB_F_LOCKED; > + iocb->mm = current->mm; > + iocb->tsk = current; > > - if ((addr + size) > vma->vm_end) > - vma = vma->vm_next; > - else > - vma = NULL; > - } > > - result = sdp_iocb_page_save(iocb); > - if (result) { > - sdp_dbg_err("Error <%d> saving pages for IOCB <%lx:%Zu>", > - result, iocb->addr, iocb->size); > - goto err_save; > + for (i = 0; i< iocb->page_count; ++i) { > + iocb->addr_array[i] = page_to_phys(iocb->page_array[i]); > } > > - up_write(&iocb->mm->mmap_sem); > - cap_t(current->cap_effective) = real_cap; > - > return 0; > -err_save: > - > - (void)do_iocb_unlock(iocb); > -err_lock: > - /* > - * unlock the mm and restore capabilities. > - */ > - up_write(&iocb->mm->mmap_sem); > - cap_t(current->cap_effective) = real_cap; > - > - iocb->flags &= ~SDP_IOCB_F_LOCKED; > - iocb->mm = NULL; > > +err_get: > + kfree(iocb->page_array); > +err_page: > + kfree(iocb->addr_array); > +err_addr: > return result; > } > > /* > - * IOCB memory locking init functions > - */ > -struct kallsym_iter { > - loff_t pos; > - struct module *owner; > - unsigned long value; > - unsigned int nameoff; /* If iterating in core kernel symbols */ > - char type; > - char name[128]; > -}; > - > -/* > - * sdp_mem_lock_init - initialize the userspace memory locking > - */ > -static int sdp_mem_lock_init(void) > -{ > - struct file *kallsyms; > - struct seq_file *seq; > - struct kallsym_iter *iter; > - loff_t pos = 0; > - int ret = -EINVAL; > - > - sdp_dbg_init("Memory Locking initialization."); > - > - kallsyms = filp_open("/proc/kallsyms", O_RDONLY, 0); > - if (!kallsyms) { > - sdp_warn("Failed to open /proc/kallsyms"); > - goto done; > - } > - > - seq = (struct seq_file *)kallsyms->private_data; > - if (!seq) { > - sdp_warn("Failed to fetch sequential file."); > - goto err_close; > - } > - > - for (iter = seq->op->start(seq, &pos); > - iter != NULL; > - iter = seq->op->next(seq, iter, &pos)) > - if (!strcmp(iter->name, "do_mlock")) > - mlock_ptr = (do_mlock_ptr_t)iter->value; > - > - if (!mlock_ptr) > - sdp_warn("Failed to find lock pointer."); > - else > - ret = 0; > - > -err_close: > - filp_close(kallsyms, NULL); > -done: > - return ret; > -} > - > -/* > - * sdp_mem_lock_cleanup - cleanup the memory locking tables > - */ > -static int sdp_mem_lock_cleanup(void) > -{ > - sdp_dbg_init("Memory Locking cleanup."); > - /* > - * null out entries. > - */ > - mlock_ptr = NULL; > - > - return 0; > -} > - > -/* > * IOCB memory registration functions > */ > > @@ -831,28 +640,12 @@ void sdp_iocb_q_clear(struct sdpc_iocb_q > } > > /* > - * primary initialization/cleanup functions > - */ > - > -/* > * sdp_main_iocb_init - initialize the advertisment caches > */ > int sdp_main_iocb_init(void) > { > - int result; > - > sdp_dbg_init("IOCB cache initialization."); > - /* > - * initialize locking code. > - */ > - result = sdp_mem_lock_init(); > - if (result < 0) { > - sdp_warn("Error <%d> initializing memory locking.", result); > - return result; > - } > - /* > - * initialize the caches only once. > - */ > + > if (sdp_iocb_cache) { > sdp_warn("IOCB caches already initialized."); > return -EINVAL; > @@ -862,15 +655,10 @@ int sdp_main_iocb_init(void) > sizeof(struct sdpc_iocb), > 0, SLAB_HWCACHE_ALIGN, NULL, > NULL); > - if (!sdp_iocb_cache) { > - result = -ENOMEM; > - goto error_iocb_c; > - } > + if (!sdp_iocb_cache) > + return -ENOMEM; > > return 0; > -error_iocb_c: > - (void)sdp_mem_lock_cleanup(); > - return result; > } > > /* > @@ -879,16 +667,6 @@ error_iocb_c: > void sdp_main_iocb_cleanup(void) > { > sdp_dbg_init("IOCB cache cleanup."); > - /* > - * cleanup the caches > - */ > kmem_cache_destroy(sdp_iocb_cache); > - /* > - * null out entries. > - */ > sdp_iocb_cache = NULL; > - /* > - * cleanup memory locking > - */ > - (void)sdp_mem_lock_cleanup(); > } > Index: ulp/sdp/sdp_iocb.h > =================================================================== > --- ulp/sdp/sdp_iocb.h (revision 2235) > +++ ulp/sdp/sdp_iocb.h (working copy) > @@ -99,9 +99,11 @@ struct sdpc_iocb { > /* > * page list. data for locking/registering userspace > */ > - struct mm_struct *mm; /* user mm struct */ > - unsigned long addr; /* user space address */ > - size_t size; /* total size of the user buffer */ > + struct mm_struct *mm; /* user mm struct */ > + struct task_struct *tsk; > + unsigned long addr; /* user space address */ > + size_t size; /* total size of the user buffer */ > + int is_receive; > > struct page **page_array; /* list of page structure pointers. */ > u64 *addr_array; /* list of physical page addresses. */ > > > -- > MST - Michael S. Tsirkin From sean.hefty at intel.com Fri May 6 20:19:20 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Fri, 6 May 2005 20:19:20 -0700 Subject: [openib-general] cm_id state information In-Reply-To: <20050506174310.H26823@topspin.com> Message-ID: >On Fri, May 06, 2005 at 11:28:01AM -0700, Sean Hefty wrote: >> James Lentini wrote: >> > Is there documentation or a piece of sample code that shows how to >> > interact with the CM correctly? >> >> The only sample code is cmpost, under the util directory. You could >> probably use SDP as well. Also, I'm currently working with Hal to make sure >> that kDAPL works properly with the CM. It will just take a little while >> before a patch is ready. > > I thought James' question was in the context of not using cm_id->state >to drive the CM state machine, which SDP is currently doing, so it would >make a bad example. I've been looking at a way to make the cm_id->state usable to consumers, and I think that it might be possible, but would require some rework in the CM. But if it would avoid consumers needing to maintain their own connection states, it could be worthwhile. Currently, the CM serializes all callbacks to a single client after it has processed received MADs. By moving the serialization higher up in the codepath, before the MADs are processed, I might be able serialize the cm_id state transitions with user callbacks. So that the cm_id->state would not change while in a user callback, unless the user invokes a routine that would naturally change it. I'm just not sure yet if such a change significantly simplifies CM clients any, particularly clients that are themselves midlayers. - Sean From iod00d at hp.com Fri May 6 21:03:32 2005 From: iod00d at hp.com (Grant Grundler) Date: Fri, 6 May 2005 21:03:32 -0700 Subject: [openib-general] [patch] misc trivia nitpicks 04 In-Reply-To: <20050507002836.GI23958@esmail.cup.hp.com> References: <20050506003942.GE15688@aon.at> <1115392431.4476.135.camel@localhost.localdomain> <20050506174104.GI15688@aon.at> <20050506194330.GB23958@esmail.cup.hp.com> <20050506202551.GJ15688@aon.at> <20050506210807.GE23958@esmail.cup.hp.com> <20050506235135.GM15688@aon.at> <20050507002836.GI23958@esmail.cup.hp.com> Message-ID: <20050507040332.GM23958@esmail.cup.hp.com> On Fri, May 06, 2005 at 05:28:36PM -0700, Grant Grundler wrote: > > 996, 26.436613 > > 997, 72.905357 > > 998, 704572144.206260 > > 998 is probably garbage. > I'll sort that out on the ia64 box. It was garbage. The code was doing math with tstamp[0] uninitialized. I've posted perftest-02.c with rdma_lat that works on ia64 and I *think* should work on most other arches as well: http://gsyprf3.external.hp.com/openib/perftest-02.tgz More gory details in another follow up to the original thread with MST. grant From iod00d at hp.com Fri May 6 22:37:06 2005 From: iod00d at hp.com (Grant Grundler) Date: Fri, 6 May 2005 22:37:06 -0700 Subject: [openib-general] user-mode verbs on Itanium In-Reply-To: <20050506190947.GA23958@esmail.cup.hp.com> References: <1AC79F16F5C5284499BB9591B33D6F00043B37C9@orsmsx408> <20050429051009.GT20957@esmail.cup.hp.com> <52pswdk5au.fsf@topspin.com> <20050429162127.GA24871@esmail.cup.hp.com> <52ll71k09w.fsf@topspin.com> <20050429183846.GH24871@esmail.cup.hp.com> <20050504005534.GA13574@esmail.cup.hp.com> <20050504070910.GC9674@mellanox.co.il> <20050506190947.GA23958@esmail.cup.hp.com> Message-ID: <20050507053706.GN23958@esmail.cup.hp.com> On Fri, May 06, 2005 at 12:09:47PM -0700, Grant Grundler wrote: > Since the diff is essentially the whole file for everything but > rdma_lat.c, I've parked the entire mess here: > http://gsyprf3.external.hp.com/openib/perftest-01.tgz perftest-02.tgz is now available. And rdma_lat.c has substantial changes: o -c option to display output in "raw" CPU cycles o -U option to dump unsorted data (MORE ON THIS BELOW) o fixed bugs and typos noted by Bernhard Fischer . o fixed a few more bugs I found. I'm consistently getting 4.74 usec median latency with 2.6.11 kernel + svn r2229 on HP ZX1 platform (PCI-X) + 1.5Ghz/6M Madison processors. This seems kinda low so I'm not 100% comfortable with the measurements yet. Maybe adding MPI on top of this will add the 2-3 usec that I think is missing. [ If it's correct, I guess I can go back to working on some tg3 driver tunes :^) Just kidding! It's late here... ] > BTW, Can I "leverage" code from ibv_pingpong to replace the issues > the following comment in rdma_lat.c refers to? > /* > * pp_get_local_lid() uses a pretty bogus method for finding the LID > * of a local port. Please don't copy this into your app (or if you > * do, please rip it out soon). > */ This is still outstanding. But I'd like to first see perftest-02 land in a sane place in openib.org Subversion tree. I can then submit patches against some stuff: o update the README with notes on how to use/interpret the data o stop replicating code and make subroutines o split up main() into bite-sized chunks so people know which part is "initialization" and "run time". o stop pretending there is no global state and get major variables off the stack and into .bss or respective subroutines. (I still want to hack a bit on mthca_cq.c:cqe_sw() too) > Couldn't post send: scnt=1 Another clue about this failure: I'm only seeing this if I specify "-s". 1,2,4,8,16,24,25,26,27,28 all worked. All the values > 29 failed. (I tried 29, 30, 31, 32, 33, 64, 65, 4096, 8192). I'm pretty busy next week...not sure I'll be able to track it down then. Why I like -U ------------- (Sorry, bad pun. ggg hides :^) "unsorted" output is useful to: o correlate client/server data (hiccups in algorithm/fabric) o recognize startup vs "steady state" behaviors. o recognize cyclical data (e.g. long runs where itimer interfers) The following is a discussion about the first two points. Extra credit question buried in the middle. The unsorted output in CPU cycles of "server": grundler at gsyprf3:/usr/src/trunk/contrib/mellanox/perftest$ ./rdma_lat -cU | head local address: LID 0x0b, QPN 0x320406, PSN 0xeb1f3e RKey 0x1f60032 VAddr 0x6000000000014001 remote address: LID 0x10, QPN 0x1a0406, PSN 0x6d8a50, RKey 0x1040436 VAddr 0x6000000000014001 #, cycles 1, 97779 2, 52050 3, 7028 4, 7318 5, 7201 ... And the same from the "client": grundler at iota:/usr/src/trunk/contrib/mellanox/perftest$ ./rdma_lat -cU 10.0.0.51 | head local address: LID 0x10, QPN 0x1a0406, PSN 0x6d8a50 RKey 0x1040436 VAddr 0x6000000000014001 remote address: LID 0x0b, QPN 0x320406, PSN 0xeb1f3e, RKey 0x1f60032 VAddr 0x6000000000014001 #, cycles 1, 611 2, 93491 3, 27129 4, 7142 5, 7260 ... The first sample (611) on the client is improbably small for a 1.5Ghz system (i.e ~0.4 usecs). That is a clue that delta[0] means something different than delta[{N>0}] on the client. It might be accurate if the client sends first and the next cycle count is taken right after telling the card data is ready to send. This could be useful data too. Michael (mst), is it obvious to you if I've understood that correctly? Either way, I'd like to leave the test as is and add the explanation to the README. The 2cd and 3rd sample sort-of match the first two samples from the server and are plausible (~63 usecs). The fact that client/server closely agree on the ~63 usec (93k cycles) sample is a good clue I can trust the measurement. That first sample on the server represents a tuning opportunity. A ~10x difference between startup and runtime is significant for short lived connections/regions. It might be in the switch (TS90), HCA, or host SW stack. I don't know. I'd need a PCI-X bus traces, HW stats from the HCA or Switch to determine how much each contributes to the latency. I hope it's obvious now why -U is interesting. thanks, grant From halr at voltaire.com Fri May 6 22:54:05 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 07 May 2005 01:54:05 -0400 Subject: [openib-general] Re: kdapltest works again!, still having slab corruption In-Reply-To: <1115425669.19661.14.camel@duffman> References: <1115411060.4476.346.camel@localhost.localdomain> <1115413920.4476.1.camel@localhost.localdomain> <1115425669.19661.14.camel@duffman> Message-ID: <1115445245.4733.26.camel@localhost.localdomain> On Fri, 2005-05-06 at 20:27, Tom Duffy wrote: > But, I am still seeing this slab corruption every time I run the test, > on the active side: > > Slab corruption: start=ffff81002353edb0, len=288 > Redzone: 0x5a2cf071/0x5a2cf071. > Last user: [](req_comp_work+0x54/0x90 [ib_at]) > 040: 00 00 00 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b > 110: 00 00 00 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b a5 > Prev obj: start=ffff81002353ec78, len=288 > Redzone: 0x5a2cf071/0x5a2cf071. > Last user: [<0000000000000000>](0x0) > 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > Slab corruption: start=ffff8100778b2eb8, len=312 > Redzone: 0x5a2cf071/0x5a2cf071. > Last user: [](req_comp_work+0x42/0x90 [ib_at]) > 050: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 00 00 00 00 > 120: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 00 00 00 00 > Prev obj: start=ffff8100778b2d68, len=312 > Redzone: 0x5a2cf071/0x5a2cf071. > Last user: [<0000000000000000>](0x0) > 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b Is the client connecting to a server which is present or not ? -- Hal From hugh at veritas.com Sat May 7 06:18:08 2005 From: hugh at veritas.com (Hugh Dickins) Date: Sat, 7 May 2005 14:18:08 +0100 (BST) Subject: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation In-Reply-To: <427BF8E1.2080006@ammasso.com> References: <200544159.Ahk9l0puXy39U6u6@topspin.com> <20050411142213.GC26127@kalmia.hozed.org> <52mzs51g5g.fsf@topspin.com> <20050411163342.GE26127@kalmia.hozed.org> <5264yt1cbu.fsf@topspin.com> <20050411180107.GF26127@kalmia.hozed.org> <52oeclyyw3.fsf@topspin.com> <20050411171347.7e05859f.akpm@osdl.org> <20050412180447.E6958@topspin.com> <20050425203110.A9729@topspin.com> <4279142A.8050501@ammasso.com> <427A6A7E.8000604@ammasso.com> <427BF8E1.2080006@ammasso.com> Message-ID: Sorry for not replying earlier (indeed, sorry for not joining in the wider RDMA pinning discussion), concentrating on other stuff at present. On Fri, 6 May 2005, Timur Tabi wrote: > Timur Tabi wrote: > > > I haven't gotten a reply to this question, but I've done my own research, > > and I think I found the answer. Using my own test of get_user_pages(), > > it appears that the fix was placed in 2.6.7. However, I would like to > > know specifically what the fix is. Unfortunately, tracking this stuff > > down is beyond my understanding of the Linux VM. > > I'm also still waiting for a reply to this question. Anyone???? > > Upon doing some more research, I think the fix might be those code instead: I believe you're right this time - I was rather puzzled by your earlier choice, then unhelpfully forgot to reply and point you a few lines further down to this comment, which does shout "get_user_pages fix" quite loudly. > /* > * Don't pull an anonymous page out from under get_user_pages. > * GUP carefully breaks COW and raises page count (while holding > * page_table_lock, as we have here) to make sure that the page > * cannot be freed. If we unmap that page here, a user write > * access to the virtual address will bring back the page, but > * its raised count will (ironically) be taken to mean it's not > * an exclusive swap page, do_wp_page will replace it by a copy > * page, and the user never get to see the data GUP was holding > * the original page for. > */ > if (PageSwapCache(page) && > page_count(page) != page->mapcount + 2) { > ret = SWAP_FAIL; > goto out_unmap; > } > > Both this change and the other one I mentioned are new to 2.6.7. I suppose I > could try applying these patches to the 2.6.6 kernel and see if anything > improves, but that won't help me understand what's really going on. There's a lot of change in the rmap area between 2.6.6 and 2.6.7, but you're right that this is an isolated fix, which could in principle be applied to earlier releases. Though I don't see it's worth doing now. > The above comment makes sounds almost like it's a fix, Almost? Sorry if my comment doesn't make it obvious it's a fix for a get_user_pages issue - I rewrote Andrea Arcangeli's original commment. The analysis and fix are his. > but it talks about copy-on-write, > which is has nothing to do with the real problem. Oh, well, maybe, but what is the real problem? Are you sure that copy-on-write doesn't come into it? I haven't reread through the whole thread, but my recollection is that you never quite said what the real problem is: you'd found some time ago that get_user_pages sometimes failed to pin the pages for your complex app, so were forced to mlock too; but couldn't provide any simple test case for the failure (which can indeed be a lot of work to devise), so we were all in the dark as to what went wrong. But you've now found that 2.6.7 and later kernels allow your app to work correctly without mlock, good. get_user_pages is certainly the right tool to use for such pinning. (On the question of whether mlock guarantees that user virtual addresses map to the same physical addresses, I prefer Arjan's view that it does not; but accept that there might prove to be difficulties in holding that position.) So, it works now, you've exonerated today's get_user_pages, and you've identified at least one get_user_pages fix which went in at that time: do we really need to chase this further? Oh, in writing of copy-on-write, I've just remembered another fix for get_user_pages which I made in 2.6.7 (though I've not heard of anyone seeing the problem fixed): call to do_wp_page in do_swap_page. get_user_pages assumes that the write fault it generates will break copy-on-write i.e. will make a private copy page when necessary, before returning to the caller; but that wasn't happening in the do_swap_page case. By the way, please don't be worried when soon the try_to_unmap_one comment and code that you identified above disappear. When I'm back in patch submission mode, I'll be sending Andrew a patch which removes it, instead reworking can_share_swap_page to rely on the page_mapcount instead of page_count, which avoids the ironical behaviour my comment refers to, and allows an awkward page migration case to proceed (once unpinned). Andrea and I now both prefer this page_mapcount approach. Hugh From timur.tabi at ammasso.com Sat May 7 07:45:50 2005 From: timur.tabi at ammasso.com (Timur Tabi) Date: Sat, 07 May 2005 09:45:50 -0500 Subject: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation In-Reply-To: References: <200544159.Ahk9l0puXy39U6u6@topspin.com> <20050411142213.GC26127@kalmia.hozed.org> <52mzs51g5g.fsf@topspin.com> <20050411163342.GE26127@kalmia.hozed.org> <5264yt1cbu.fsf@topspin.com> <20050411180107.GF26127@kalmia.hozed.org> <52oeclyyw3.fsf@topspin.com> <20050411171347.7e05859f.akpm@osdl.org> <20050412180447.E6958@topspin.com> <20050425203110.A9729@topspin.com> <4279142A.8050501@ammasso.com> <427A6A7E.8000604@ammasso.com> <427BF8E1.2080006@ammasso.com> Message-ID: <427CD49E.6080300@ammasso.com> Hugh Dickins wrote: > Oh, well, maybe, but what is the real problem? > Are you sure that copy-on-write doesn't come into it? No, but I do know that my test case doesn't call fork(), so it's reproducible without involving COW. Of course, I'm sure someone's going to tell me now that COW comes into effect even without fork(). If so, please explain. > I haven't reread through the whole thread, but my recollection is > that you never quite said what the real problem is: you'd found some > time ago that get_user_pages sometimes failed to pin the pages for > your complex app, so were forced to mlock too; but couldn't provide > any simple test case for the failure (which can indeed be a lot of > work to devise), so we were all in the dark as to what went wrong. The short answer: under "extreme" memory pressure, the data inside a page pinned by get_user_pages() is swapped out, moved, or deleted (I'm not sure which). Some other data is placed into that physical location. By extreme memory pressure, I mean having the process allocate and touch as much memory as possible. Something like this: num_bytes = get_amount_of_physical_ram(); char *p = malloc(num_bytes); for (i=0; i But you've now found that 2.6.7 and later kernels allow your app to > work correctly without mlock, good. get_user_pages is certainly the > right tool to use for such pinning. (On the question of whether > mlock guarantees that user virtual addresses map to the same physical > addresses, I prefer Arjan's view that it does not; but accept that > there might prove to be difficulties in holding that position.) My understanding is that mlock() could in theory allow the page to be moved, but that currently nothing in the kernel would actually move it. However, that could change in the future to allow hot-swapping of RAM. > So, it works now, you've exonerated today's get_user_pages, and you've > identified at least one get_user_pages fix which went in at that time: > do we really need to chase this further? My driver needs to support all 2.4 and 2.6 kernel versions. My makefile scans the kernel source tree with 'grep' to identify various characterists, and I use #ifdefs to conditionally compile code depending on what features are present in the kernel. I can't use the kernel version number, because that's not reliable - distros will incorporate patches from future kernels without changing the version ID. So I need to take into account distro vendors that use an earlier kernel, like 2.6.5, and back-port the patch from 2.6.7. The distro vendor will keep the 2.6.5 version number, which is why I can't rely on it. I need to know exactly what the fix is, so that when I scan mm/rmap.c, I know what to look for. Currently, I look for this regex: try_to_unmap_one.*vm_area_struct which seems to work. However, now I think it's just a coincidence. > By the way, please don't be worried when soon the try_to_unmap_one > comment and code that you identified above disappear. When I'm > back in patch submission mode, I'll be sending Andrew a patch which > removes it, instead reworking can_share_swap_page to rely on the > page_mapcount instead of page_count, which avoids the ironical > behaviour my comment refers to, and allows an awkward page migration > case to proceed (once unpinned). Andrea and I now both prefer this > page_mapcount approach. Ugh, that means my regex is probably going to break. Not only that, but I don't understand what you're saying either. Trying to understand the VM is really hard. I guess in this specific case, it doesn't really matter, because calling mlock() when I should be calling get_user_pages() is not a bad thing. From tduffy at sun.com Sat May 7 08:57:18 2005 From: tduffy at sun.com (Tom Duffy) Date: Sat, 07 May 2005 08:57:18 -0700 Subject: [openib-general] Re: kdapltest works again!, still having slab corruption In-Reply-To: <1115445245.4733.26.camel@localhost.localdomain> References: <1115411060.4476.346.camel@localhost.localdomain> <1115413920.4476.1.camel@localhost.localdomain> <1115425669.19661.14.camel@duffman> <1115445245.4733.26.camel@localhost.localdomain> Message-ID: <1115481438.5263.4.camel@duffman> On Sat, 2005-05-07 at 01:54 -0400, Hal Rosenstock wrote: > Is the client connecting to a server which is present or not ? Yes, the client *is* connecting to the server. Here is the full client side transaction: [root at sins-stinger-10 ~]# ./kdapltest -T Q -s 192.168.0.26 -D mthca0a -d Server Name: 192.168.0.26 Server Net Address: 192.168.0.26 DT_cs_Client: Starting Test ... DT_cs_Client: IA mthca0a opened DT_cs_Client: EP created ***** DAPL Characteristics ***** Provider: mthca0a Version 1.0 DAPL 1.2 Adapter: Generic InfiniBand HCA by DAPL Reference Implementation Version 0.0 Supporting: 64512 EPs with 65535 DTOs and 0 in RDMA/RDs and 0ut RDMA/RDs each 65408 EVDs of up to 65535 entries (default S/R size is 16/16) IOVs of up to 28 elements 131056 LMRs (and 131056 RMRs) of up to 0xffffffffffffffff bytes Maximum MTU 0x80000000 bytes, RDMA 0x80000000 bytes Maximum Private data size 92 bytes ***** ***** ***** ***** ***** ***** DT_cs_Client: Posting 1 recv buffer DT_cs_Client: Connect Endpoint DT_cs_Client: Await connection ... DT_cs_Client: Connected! DAT_STATE: DAT_EP_STATE_CONNECTED DAT_STATE: Inbound DTO Status: Active DAT_STATE: Outbound DTO Status: Idle DT_cs_Client: Sending Client_Info DT_cs_Client: Sent Client_Info - awaiting completion DT_cs_Client: Sending Command DT_cs_Client: Sent Command - awaiting completion DT_cs_Client: Waiting for Server_Info DT_cs_Client: Server_Info Received DT_cs_Client: Version OK! ------------------------------------- Server_Info.dapltest_version : 6 Server_Info.is_little_endian : 1 ------------------------------------- Client_Info.dapltest_version : 6 Client_Info.is_little_endian : 1 Client_Info.test_type : 4 Quit_Cmd.server_name: 192.168.0.26 Quit_Cmd.device_name: mthca0a DT_cs_Client: Cleaning Up ... DT_cs_Client: IA mthca0a closed TEST INSTANCE 0 TEST return code = 0 [root at sins-stinger-10 ~]# Slab corruption: start=ffff81003b2eddb0, len=288 Redzone: 0x5a2cf071/0x5a2cf071. Last user: [](req_comp_work+0x54/0x90 [ib_at]) 040: 00 00 00 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b 110: 00 00 00 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b a5 Prev obj: start=ffff81003b2edc78, len=288 Redzone: 0x5a2cf071/0x5a2cf071. Last user: [<0000000000000000>](stext+0x7feff0e8/0xe8) 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b Slab corruption: start=ffff81003a026eb8, len=312 Redzone: 0x5a2cf071/0x5a2cf071. Last user: [](req_comp_work+0x42/0x90 [ib_at]) 050: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 00 00 00 00 120: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 00 00 00 00 Prev obj: start=ffff81003a026d68, len=312 Redzone: 0x5a2cf071/0x5a2cf071. Last user: [<0000000000000000>](stext+0x7feff0e8/0xe8) 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From hugh at veritas.com Sat May 7 09:30:52 2005 From: hugh at veritas.com (Hugh Dickins) Date: Sat, 7 May 2005 17:30:52 +0100 (BST) Subject: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation In-Reply-To: <427CD49E.6080300@ammasso.com> References: <200544159.Ahk9l0puXy39U6u6@topspin.com> <20050411142213.GC26127@kalmia.hozed.org> <52mzs51g5g.fsf@topspin.com> <20050411163342.GE26127@kalmia.hozed.org> <5264yt1cbu.fsf@topspin.com> <20050411180107.GF26127@kalmia.hozed.org> <52oeclyyw3.fsf@topspin.com> <20050411171347.7e05859f.akpm@osdl.org> <20050412180447.E6958@topspin.com> <20050425203110.A9729@topspin.com> <4279142A.8050501@ammasso.com> <427A6A7E.8000604@ammasso.com> <427BF8E1.2080006@ammasso.com> <427CD49E.6080300@ammasso.com> Message-ID: On Sat, 7 May 2005, Timur Tabi wrote: > > > Oh, well, maybe, but what is the real problem? > > Are you sure that copy-on-write doesn't come into it? > > No, but I do know that my test case doesn't call fork(), so it's reproducible > without involving COW. Of course, I'm sure someone's going to tell me now > that COW comes into effect even without fork(). If so, please explain. I'll try. COW comes into effect whenever you're sharing a page and then need to make private changes to it. Fork is one way of sharing (with ancestor and descendant processes). Using the empty zero page is another way of sharing (with all other processes and parts of your own address space with a readonly page full of zeroes). Using a file page from the page cache is another way of sharing. None of those is actually your case, but our test for whether a page is shared has been inadequate: oversimplifying, if page_count is more than 1 then we have to assume it is shared and do the copy-on-write (if the modifications are to be private). But there are various places where the page_count is temporarily raised (e.g. while paging out), which we cannot distinguish, so occasionally we'll copy on write even when it's not necessary, but we lack the information to tell us so. In particular, of course, get_user_pages raises page_count to pin the page: so making a page appear shared when it's not shared at all. > The short answer: under "extreme" memory pressure, the data inside a page > pinned by get_user_pages() is swapped out, moved, or deleted (I'm not sure > which). Some other data is placed into that physical location. > > By extreme memory pressure, I mean having the process allocate and touch as > much memory as possible. Something like this: > > num_bytes = get_amount_of_physical_ram(); > char *p = malloc(num_bytes); > for (i=0; i p[i] = 0; > > The above over-simplified code fails on earlier 2.6 kernels (or earlier > versions of glibc that accompany most distros the use the earlier 2.6 > kernels). Either malloc() returns NULL, or the p[i]=0 part causes a segfault. > I haven't bothered to trace down why. But when it does work, the page pinned > by get_user_pages() changes. Which has to be a bug with get_user_pages, which has no other purpose than to pin the pages. I cannot criticize you for working around it to get your app working on lots of releases, but what _we_ have to do is fix get_user_pages - and it appears that Andrea did so a year ago. I'm surprised if it's as simple as you describe (you do say over- simplified, maybe the critical points have fallen out), since GUP users would have complained long ago if it wasn't doing the job in normal cases of memory pressure. Andrea's case does involve the process independently trying to touch a page it has pinned for I/O with get_user_pages. Or (and I've only just thought of this, suspect it might be exactly your case) not touch, but apply get_user_pages again to a page already so pinned (while memory pressure has caused try_to_unmap_one temporarily to detach it from the user address space - the aspect of the problem that Andrea's fix attacks). > My understanding is that mlock() could in theory allow the page to be moved, > but that currently nothing in the kernel would actually move it. However, > that could change in the future to allow hot-swapping of RAM. That's my understanding too, that nothing currently does so. Aside from hot-swapping RAM, there's also a need to be able to migrate pages around RAM, either to unfragment memory allowing higher-order allocations to succeed more often, or to get around extreme dmamem/normal-mem/highmem imbalances without dedicating huge reserves. Those would more often succeed if uninhibited by mlock. > So I need to take into account distro vendors that use an earlier kernel, like > 2.6.5, and back-port the patch from 2.6.7. The distro vendor will keep the > 2.6.5 version number, which is why I can't rely on it. > > I need to know exactly what the fix is, so that when I scan mm/rmap.c, I know > what to look for. Currently, I look for this regex: > > try_to_unmap_one.*vm_area_struct > > which seems to work. However, now I think it's just a coincidence. Perhaps any release based on 2.6.7 or above, or any release which mentions "get_user_pages" in its mm/rmap.c or mm/objrmap.c? > > By the way, please don't be worried when soon the try_to_unmap_one > > comment and code that you identified above disappear. When I'm > > back in patch submission mode, I'll be sending Andrew a patch which > > removes it, instead reworking can_share_swap_page to rely on the > > page_mapcount instead of page_count, which avoids the ironical > > behaviour my comment refers to, and allows an awkward page migration > > case to proceed (once unpinned). Andrea and I now both prefer this > > page_mapcount approach. > > Ugh, that means my regex is probably going to break. Not only that, but I > don't understand what you're saying either. Trying to understand the VM is > really hard. Sorry about that, but suiting your regex is low in our priorities for VM design! I was tempted to offer to keep a comment on get_user_pages in mm/rmap.c after the change, but that's really rather babyish: just assume 2.6.7 and upwards are fixed (or complain if you find not). Perhaps I'll manage a clearer explanation when I come to write the change description for the patch, we'll have to see. > I guess in this specific case, it doesn't really matter, because calling > mlock() when I should be calling get_user_pages() is not a bad thing. If you can afford to keep that amount of memory mlocked, and have to capability to do so, yes, it should do no harm. We were just upset to think that mlock was still needed to get around a get_user_pages bug which was fixed a year ago. Hugh From tduffy at sun.com Sat May 7 09:33:28 2005 From: tduffy at sun.com (Tom Duffy) Date: Sat, 07 May 2005 09:33:28 -0700 Subject: [openib-general] [PATCH] SDP: add missing MODULE_PARM_DESC In-Reply-To: <20050506003018.GD15688@aon.at> References: <20050506003018.GD15688@aon.at> Message-ID: <1115483608.17241.2.camel@duffman> On Fri, 2005-05-06 at 02:30 +0200, Bernhard Fischer wrote: > in src/linux-kernel/infiniband/ulp/sdp/sdp_inet.c > module_param seem to be missing MODULE_PARM_DESC Signed-off-by: Tom Duffy Index: linux/drivers/infiniband/ulp/sdp/sdp_inet.c =================================================================== --- linux/drivers/infiniband/ulp/sdp/sdp_inet.c (revision 2278) +++ linux/drivers/infiniband/ulp/sdp/sdp_inet.c (working copy) @@ -59,15 +59,34 @@ MODULE_DESCRIPTION("InfiniBand SDP modul MODULE_LICENSE("Dual BSD/GPL"); module_param(proto_family, int, 0); +MODULE_PARM_DESC(proto_family, "Override the default protocol family value of 27."); + module_param(buff_min, int, 0); +MODULE_PARM_DESC(buff_min, "Set the minimum number of buffers to allocate."); + module_param(buff_max, int, 0); +MODULE_PARM_DESC(buff_max, "Set the maximum number of buffers to allocate."); + module_param(conn_size, int, 0); +MODULE_PARM_DESC(conn_size, "Set the total number of active sockets."); + module_param(recv_post_max, int, 0); +MODULE_PARM_DESC(recv_post_max, "Set the receive completion queue size."); + module_param(recv_buff_max, int, 0); +MODULE_PARM_DESC(recv_buff_max, "Set the maximum number of receives buffered."); + module_param(send_post_max, int, 0); +MODULE_PARM_DESC(send_post_max, "Set the send completion queue size."); + module_param(send_buff_max, int, 0); +MODULE_PARM_DESC(send_buff_max, "Set the maximum number of sends buffered."); + module_param(send_usig_max, int, 0); +MODULE_PARM_DESC(send_usig_max, "Set the maximum consecutive unsignalled send events."); + module_param(sdp_debug_level, int, 0); +MODULE_PARM_DESC(sdp_debug_level, "Set the debug level 0-9."); /* * socket structure relevant fields: From blist at aon.at Sat May 7 10:47:18 2005 From: blist at aon.at (Bernhard Fischer) Date: Sat, 7 May 2005 19:47:18 +0200 Subject: [PATCH] Re: [openib-general] 0 op factor In-Reply-To: <20050505172122.B26428@topspin.com> References: <20050505155803.GS15688@aon.at> <20050505172122.B26428@topspin.com> Message-ID: <20050507174718.GQ15688@aon.at> On Thu, May 05, 2005 at 05:21:22PM -0700, Libor Michalek wrote: > No point other then the obvious, changing the sign of the variables >value. It's just a style convention that I have no problem changing. - remove '0 operator factor' statements. - a bit of whitespace removal. - remove return at end of void functions. - remove expect from _sdp_cm_path_complete(). - comment out empty if(var&FLAG){}else{} in _sdp_inet_release(). untested. hw/mthca/mthca_cmd.c | 2 - ulp/sdp/sdp_actv.c | 12 +++------- ulp/sdp/sdp_inet.c | 58 ++++++++++++++++++++++----------------------------- ulp/sdp/sdp_rcvd.c | 3 -- ulp/sdp/sdp_wall.c | 4 +-- 5 files changed, 34 insertions(+), 45 deletions(-) -------------- next part -------------- diff -rup infiniband.oorig/hw/mthca/mthca_cmd.c infiniband/hw/mthca/mthca_cmd.c --- infiniband.oorig/hw/mthca/mthca_cmd.c 2005-01-19 15:53:15.000000000 +0100 +++ infiniband/hw/mthca/mthca_cmd.c 2005-05-07 19:28:34.924018992 +0200 @@ -219,7 +219,7 @@ static int mthca_cmd_post(struct mthca_d * (and some architectures such as ia64 implement memcpy_toio * in terms of writeb). */ - __raw_writel(cpu_to_be32(in_param >> 32), dev->hcr + 0 * 4); + __raw_writel(cpu_to_be32(in_param >> 32), dev->hcr/*+0 * 4*/); __raw_writel(cpu_to_be32(in_param & 0xfffffffful), dev->hcr + 1 * 4); __raw_writel(cpu_to_be32(in_modifier), dev->hcr + 2 * 4); __raw_writel(cpu_to_be32(out_param >> 32), dev->hcr + 3 * 4); diff -rup infiniband.oorig/ulp/sdp/sdp_actv.c infiniband/ulp/sdp/sdp_actv.c --- infiniband.oorig/ulp/sdp/sdp_actv.c 2005-02-19 14:33:28.000000000 +0100 +++ infiniband/ulp/sdp/sdp_actv.c 2005-05-07 19:04:41.244971336 +0200 @@ -385,7 +385,6 @@ static void _sdp_cm_path_complete(u64 id struct sdp_opt *conn = (struct sdp_opt *) arg; struct sdpc_buff *buff; int result = 0; - int expect; /* * lock the socket */ @@ -491,8 +490,7 @@ static void _sdp_cm_path_complete(u64 id sdp_dbg_warn(conn, "Error <%d> buffering hello msg.", result); - expect = sdp_buff_pool_put(buff); - SDP_EXPECT(!(0 > expect)); + sdp_buff_pool_put(buff); goto failed; } @@ -560,20 +558,18 @@ static void _sdp_cm_path_complete(u64 id goto done; failed: - result = sdp_wall_recv_reject(conn, (0 - status)); + result = sdp_wall_recv_reject(conn, -status); if (0 > result) { sdp_dbg_warn(conn, "Error <%d> rejecting connection", result); - - expect = sdp_wall_recv_drop(conn); - SDP_EXPECT(!(0 > expect)); + + sdp_wall_recv_drop(conn); } /* if */ done: SDP_CONN_UNLOCK(conn); SDP_CONN_PUT(conn); - return; } /* _sdp_cm_path_complete */ /* diff -rup infiniband.oorig/ulp/sdp/sdp_inet.c infiniband/ulp/sdp/sdp_inet.c --- infiniband.oorig/ulp/sdp/sdp_inet.c 2005-02-19 14:33:28.000000000 +0100 +++ infiniband/ulp/sdp/sdp_inet.c 2005-05-07 19:21:34.928867960 +0200 @@ -103,11 +103,11 @@ void sdp_inet_wake_send(struct sock *sk) { struct sdp_opt *conn; - if (NULL == sk || + if (NULL == sk || NULL == (conn = SDP_GET_CONN(sk))) { return; - } + } if (NULL != sk->sk_socket && 0 < test_bit(SOCK_NOSPACE, &sk->sk_socket->flags) && @@ -128,7 +128,6 @@ void sdp_inet_wake_send(struct sock *sk) read_unlock(&sk->sk_callback_lock); } - return; } /* sdp_inet_wake_send */ /* @@ -149,7 +148,6 @@ void sdp_inet_wake_generic(struct sock * read_unlock(&sk->sk_callback_lock); } - return; } /* sdp_inet_wake_generic */ /* @@ -161,7 +159,7 @@ void sdp_inet_wake_recv(struct sock *sk, read_lock(&sk->sk_callback_lock); if (NULL != sk->sk_sleep) { - + wake_up_interruptible(sk->sk_sleep); } @@ -169,7 +167,6 @@ void sdp_inet_wake_recv(struct sock *sk, read_unlock(&sk->sk_callback_lock); } - return; } /* sdp_inet_wake_recv */ /* @@ -181,7 +178,7 @@ void sdp_inet_wake_error(struct sock *sk read_lock(&sk->sk_callback_lock); if (NULL != sk->sk_sleep) { - + wake_up_interruptible(sk->sk_sleep); } @@ -189,7 +186,6 @@ void sdp_inet_wake_error(struct sock *sk read_unlock(&sk->sk_callback_lock); } - return; } /* sdp_inet_wake_error */ /* @@ -205,7 +201,6 @@ void sdp_inet_wake_urg(struct sock *sk) sk_send_sigurg(sk); } - return; } /* sdp_inet_wake_urg */ /* @@ -384,7 +379,7 @@ static int _sdp_inet_release(struct sock */ result = sdp_inet_listen_stop(conn); if (0 > result) { - + sdp_dbg_warn(conn, "Error <%d> while releasing listen", result); } @@ -417,7 +412,7 @@ static int _sdp_inet_release(struct sock */ result = _sdp_inet_abort(conn); if (0 > result) { - + sdp_dbg_warn(conn, "Error <%d> while aborting socket", result); } @@ -429,7 +424,7 @@ static int _sdp_inet_release(struct sock */ result = _sdp_inet_disconnect(conn); if (0 > result) { - + sdp_dbg_warn(conn, "Error <%d> while disconnecting socket", result); goto done; @@ -449,7 +444,7 @@ static int _sdp_inet_release(struct sock DECLARE_WAITQUEUE(wait, current); timeout = sk->sk_lingertime; - + add_wait_queue(sk->sk_sleep, &wait); set_current_state(TASK_INTERRUPTIBLE); @@ -461,7 +456,7 @@ static int _sdp_inet_release(struct sock SDP_CONN_LOCK(conn); if (signal_pending(current)) { - + break; } } @@ -487,6 +482,7 @@ static int _sdp_inet_release(struct sock } /* if (blocking) */ done: +#if 0 if (0 < (SDP_ST_MASK_CLOSED & conn->istate)) { /* * pass @@ -495,6 +491,7 @@ done: else { } +#endif /* * finally drop socket reference. (socket API reference) */ @@ -878,7 +875,7 @@ done: * _sdp_inet_accept - accept a new socket from a listen socket */ static int _sdp_inet_accept(struct socket *listen_sock, - struct socket *accept_sock, + struct socket *accept_sock, int flags) { struct sock *listen_sk; @@ -1043,7 +1040,7 @@ static int _sdp_inet_accept(struct socke listen_done: SDP_CONN_UNLOCK(listen_conn); - sdp_dbg_ctrl(listen_conn, + sdp_dbg_ctrl(listen_conn, "ACCEPT: complete <%d> <%08x:%04x><%08x:%04x>", (NULL == accept_conn ? SDP_DEV_SK_INVALID : accept_conn->hashent), @@ -1074,7 +1071,7 @@ static int _sdp_inet_getname(struct sock conn = SDP_GET_CONN(sk); sdp_dbg_ctrl(conn, "GETNAME: src <%08x:%04x> dst <%08x:%04x>", - conn->src_addr, conn->src_port, + conn->src_addr, conn->src_port, conn->dst_addr, conn->dst_port); addr->sin_family = _proto_family; @@ -1163,8 +1160,8 @@ static unsigned int _sdp_inet_poll(struc mask |= POLLIN | POLLRDNORM; } /* - * send EOF _or_ send data space. - * (Some poll() Linux documentation says that POLLHUP is + * send EOF _or_ send data space. + * (Some poll() Linux documentation says that POLLHUP is * incompatible with the POLLOUT/POLLWR flags) */ if (0 < (SEND_SHUTDOWN & conn->shutdown)) { @@ -1174,7 +1171,7 @@ static unsigned int _sdp_inet_poll(struc else { /* * avoid race by setting flags, and only clearing - * them if the test is passed. Setting after the + * them if the test is passed. Setting after the * test, we can end up with them set and a passing * test. */ @@ -1196,10 +1193,9 @@ static unsigned int _sdp_inet_poll(struc } } - sdp_dbg_data(conn, "POLL: mask <%08x> flags <%08lx> <%d:%d:%d>", + sdp_dbg_data(conn, "POLL: mask <%08x> flags <%08lx> <%d:%d:%d>", mask, sock->flags, conn->send_buf, conn->send_qud, __sdp_inet_writable(conn)); - return mask; } /* _sdp_inet_poll */ @@ -1352,7 +1348,7 @@ static int _sdp_inet_ioctl(struct socket /* * _sdp_inet_setopt - set a socket option */ -static int _sdp_inet_setopt(struct socket *sock, int level, int optname, +static int _sdp_inet_setopt(struct socket *sock, int level, int optname, char __user *optval, int optlen) { struct sock *sk; @@ -1367,7 +1363,7 @@ static int _sdp_inet_setopt(struct socke sk = sock->sk; conn = SDP_GET_CONN(sk); - sdp_dbg_ctrl(conn, "SETSOCKOPT: level <%d> option <%d>", + sdp_dbg_ctrl(conn, "SETSOCKOPT: level <%d> option <%d>", level, optname); if (SOL_TCP != level && SOL_SDP != level) { @@ -1668,7 +1664,7 @@ static int _sdp_inet_create(struct socke sdp_dbg_ctrl(NULL, "SOCKET: type <%d> proto <%d> state <%u:%08lx>", sock->type, protocol, sock->state, sock->flags); - + if (SOCK_STREAM != sock->type || (IPPROTO_IP != protocol && IPPROTO_TCP != protocol)) { @@ -1680,7 +1676,7 @@ static int _sdp_inet_create(struct socke conn = sdp_conn_alloc(GFP_KERNEL); if (NULL == conn) { - + sdp_dbg_warn(conn, "SOCKET: failed to create socekt <%d:%d>", sock->type, protocol); return -ENOMEM; @@ -1758,9 +1754,9 @@ static int __init sdp_init(void) /* * buffer memory */ - result = sdp_buff_pool_init(_buff_min, - _buff_max, - _alloc_inc, + result = sdp_buff_pool_init(_buff_min, + _buff_max, + _alloc_inc, _free_mark); if (0 > result) { @@ -1771,7 +1767,7 @@ static int __init sdp_init(void) * connection table */ result = sdp_conn_table_init(_proto_family, - _conn_size, + _conn_size, _recv_post_max, _recv_buff_max, _send_post_max, @@ -1841,8 +1837,6 @@ static void __exit sdp_exit(void) * proc tables */ (void)sdp_main_proc_cleanup(); - - return; } /* sdp_exit */ module_init(sdp_init); diff -rup infiniband.oorig/ulp/sdp/sdp_rcvd.c infiniband/ulp/sdp/sdp_rcvd.c --- infiniband.oorig/ulp/sdp/sdp_rcvd.c 2005-02-19 14:33:28.000000000 +0100 +++ infiniband/ulp/sdp/sdp_rcvd.c 2005-05-07 19:31:40.217850048 +0200 @@ -1032,8 +1032,7 @@ static int _sdp_rcvd_src_avail(struct sd } advt->rkey = srcah->r_key; - advt->post = - 0 - ((SDP_SRC_AVAIL_RECV > size) ? size : 0); + advt->post = (SDP_SRC_AVAIL_RECV > size) ? -size : 0; advt->size = srcah->size - ((SDP_SRC_AVAIL_RECV > size) ? 0 : size); diff -rup infiniband.oorig/ulp/sdp/sdp_wall.c infiniband/ulp/sdp/sdp_wall.c --- infiniband.oorig/ulp/sdp/sdp_wall.c 2005-02-19 14:33:28.000000000 +0100 +++ infiniband/ulp/sdp/sdp_wall.c 2005-05-07 19:09:02.698224400 +0200 @@ -320,9 +320,9 @@ int sdp_wall_recv_reject(struct sdp_opt conn->istate = SDP_SOCK_ST_ERROR; conn->shutdown = SHUTDOWN_MASK; - result = sdp_iocb_q_cancel_all(conn, (0 - error)); + result = sdp_iocb_q_cancel_all(conn, -error); if (0 > result) { - + sdp_dbg_warn(conn, "Error <%d> canceling outstanding IOCBs", result); } From tduffy at sun.com Sat May 7 12:13:12 2005 From: tduffy at sun.com (Tom Duffy) Date: Sat, 07 May 2005 12:13:12 -0700 Subject: [openib-general] [PATCHv6] SDP: allow SDP to work on 2.6.12-rc4 Message-ID: <1115493192.23057.10.camel@duffman> This is an updated patch to SDP allowing it to work on 2.6.12-rc4 with the new sock slab allocator. This applies cleanly on SDP rev 2277. It also fixes a bug I had in v5 that would cause slab corruption on conn destroy. Please apply once 2.6.12-final is out. Signed-off-by: Tom Duffy Index: linux-2.6.12-rc4-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c =================================================================== --- linux-2.6.12-rc4-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c (revision 2278) +++ linux-2.6.12-rc4-openib/drivers/infiniband/ulp/sdp/sdp_rcvd.c (working copy) @@ -1236,7 +1236,7 @@ int sdp_event_recv(struct sdp_opt *conn, * If data was consumed by the protocol, signal * the user. */ - sdp_inet_wake_recv(conn->sk, conn->byte_strm); + sdp_inet_wake_recv(sk_sdp(conn), conn->byte_strm); /* * It's possible that a new recv buffer advertisment opened up the * recv window and we can flush buffered send data Index: linux-2.6.12-rc4-openib/drivers/infiniband/ulp/sdp/sdp_inet.c =================================================================== --- linux-2.6.12-rc4-openib/drivers/infiniband/ulp/sdp/sdp_inet.c (revision 2278) +++ linux-2.6.12-rc4-openib/drivers/infiniband/ulp/sdp/sdp_inet.c (working copy) @@ -101,9 +101,9 @@ module_param(sdp_debug_level, int, 0); */ void sdp_inet_wake_send(struct sock *sk) { - struct sdp_opt *conn; + struct sdp_opt *conn = sdp_sk(sk); - if (!sk || !(conn = SDP_GET_CONN(sk))) + if (!sk) return; if (sk->sk_socket && test_bit(SOCK_NOSPACE, &sk->sk_socket->flags) && @@ -300,27 +300,22 @@ static int sdp_inet_disconnect(struct sd static int sdp_inet_release(struct socket *sock) { struct sdp_opt *conn; - struct sock *sk; + struct sock *sk = sock->sk; int result; long timeout; u32 flags; - if (!sock->sk) { + if (!sk) { sdp_dbg_warn(NULL, "release empty <%d:%d> flags <%08lx>", sock->type, sock->state, sock->flags); return 0; } - sk = sock->sk; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); sdp_dbg_ctrl(conn, "RELEASE: linger <%d:%lu> data <%d:%d>", sock_flag(sk, SOCK_LINGER), sk->sk_lingertime, conn->byte_strm, conn->src_recv); - /* - * clear out sock, so we only do this once. - */ - sock->sk = NULL; sdp_conn_lock(conn); conn->shutdown = SHUTDOWN_MASK; @@ -427,6 +422,7 @@ done: * finally drop socket reference. (socket API reference) */ sock_orphan(sk); + sock->sk = NULL; sdp_conn_unlock(conn); sdp_conn_put(conn); @@ -446,7 +442,7 @@ static int sdp_inet_bind(struct socket * int result; sk = sock->sk; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); sdp_dbg_ctrl(conn, "BIND: family <%d> addr <%08x:%04x>", addr->sin_family, addr->sin_addr.s_addr, addr->sin_port); @@ -537,7 +533,7 @@ static int sdp_inet_connect(struct socke int result; sk = sock->sk; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); sdp_dbg_ctrl(conn, "CONNECT: family <%d> addr <%08x:%04x>", addr->sin_family, addr->sin_addr.s_addr, addr->sin_port); @@ -699,7 +695,7 @@ static int sdp_inet_listen(struct socket int result; sk = sock->sk; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); sdp_dbg_ctrl(conn, "LISTEN: addr <%08x:%04x> backlog <%04x>", conn->src_addr, conn->src_port, backlog); @@ -760,7 +756,7 @@ static int sdp_inet_accept(struct socket long timeout; listen_sk = listen_sock->sk; - listen_conn = SDP_GET_CONN(listen_sk); + listen_conn = sdp_sk(listen_sk); sdp_dbg_ctrl(listen_conn, "ACCEPT: addr <%08x:%04x>", listen_conn->src_addr, listen_conn->src_port); @@ -816,7 +812,7 @@ static int sdp_inet_accept(struct socket goto listen_done; } } else { - accept_sk = accept_conn->sk; + accept_sk = sk_sdp(accept_conn); switch (accept_conn->istate) { case SDP_SOCK_ST_ACCEPTED: @@ -913,7 +909,7 @@ static int sdp_inet_getname(struct socke struct sdp_opt *conn; sk = sock->sk; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); sdp_dbg_ctrl(conn, "GETNAME: src <%08x:%04x> dst <%08x:%04x>", conn->src_addr, conn->src_port, @@ -953,7 +949,7 @@ static unsigned int sdp_inet_poll(struct * recheck the falgs on being woken. */ sk = sock->sk; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); sdp_dbg_data(conn, "POLL: socket flags <%08lx>", sock->flags); @@ -1040,7 +1036,7 @@ static int sdp_inet_ioctl(struct socket int value; sk = sock->sk; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); sdp_dbg_ctrl(conn, "IOCTL: command <%d> argument <%08lx>", cmd, arg); /* @@ -1162,7 +1158,7 @@ static int sdp_inet_setopt(struct socket int result = 0; sk = sock->sk; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); sdp_dbg_ctrl(conn, "SETSOCKOPT: level <%d> option <%d>", level, optname); @@ -1229,7 +1225,7 @@ static int sdp_inet_getopt(struct socket int len; sk = sock->sk; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); sdp_dbg_ctrl(conn, "GETSOCKOPT: level <%d> option <%d>", level, optname); @@ -1287,7 +1283,7 @@ static int sdp_inet_shutdown(struct sock int result = 0; struct sdp_opt *conn; - conn = SDP_GET_CONN(sock->sk); + conn = sdp_sk(sock->sk); sdp_dbg_ctrl(conn, "SHUTDOWN: flag <%d>", flag); /* @@ -1422,7 +1418,7 @@ static int sdp_inet_create(struct socket sock->ops = &lnx_stream_ops; sock->state = SS_UNCONNECTED; - sock_graft(conn->sk, sock); + sock_graft(sk_sdp(conn), sock); conn->pid = current->pid; Index: linux-2.6.12-rc4-openib/drivers/infiniband/ulp/sdp/sdp_read.c =================================================================== --- linux-2.6.12-rc4-openib/drivers/infiniband/ulp/sdp/sdp_read.c (revision 2278) +++ linux-2.6.12-rc4-openib/drivers/infiniband/ulp/sdp/sdp_read.c (working copy) @@ -185,7 +185,7 @@ int sdp_event_read(struct sdp_opt *conn, */ conn->byte_strm += result; - sdp_inet_wake_recv(conn->sk, conn->byte_strm); + sdp_inet_wake_recv(sk_sdp(conn), conn->byte_strm); } else { if (result < 0) sdp_dbg_warn(conn, "Error <%d> receiving buff", @@ -229,7 +229,7 @@ int sdp_event_read(struct sdp_opt *conn, iocb->flags &= ~(SDP_IOCB_F_ACTIVE | SDP_IOCB_F_RDMA_R); - if (conn->sk->sk_rcvlowat > iocb->post) + if (sk_sdp(conn)->sk_rcvlowat > iocb->post) break; iocb = sdp_iocb_q_get_head(&conn->r_pend); Index: linux-2.6.12-rc4-openib/drivers/infiniband/ulp/sdp/sdp_send.c =================================================================== --- linux-2.6.12-rc4-openib/drivers/infiniband/ulp/sdp/sdp_send.c (revision 2278) +++ linux-2.6.12-rc4-openib/drivers/infiniband/ulp/sdp/sdp_send.c (working copy) @@ -1793,7 +1793,7 @@ static int sdp_inet_write_cancel(struct /* * lock the socket while we operate. */ - conn = SDP_GET_CONN(si->sock->sk); + conn = sdp_sk(si->sock->sk); sdp_conn_lock(conn); sdp_dbg_ctrl(conn, "Cancel Write IOCB. <%08x:%04x> <%08x:%04x>", @@ -2003,7 +2003,7 @@ int sdp_send_flush(struct sdp_opt *conn) /* * see if there is enough buffer to wake/notify writers */ - sdp_inet_wake_send(conn->sk); /* conn->sk->write_space(conn->sk); */ + sdp_inet_wake_send(sk_sdp(conn)); /* conn->sk->write_space(conn->sk); */ return 0; done: @@ -2032,7 +2032,7 @@ int sdp_inet_send(struct kiocb *req, str oob = (msg->msg_flags & MSG_OOB); sk = sock->sk; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); sdp_dbg_data(conn, "send state <%04x> size <%Zu> flags <%08x>", conn->state, size, msg->msg_flags); Index: linux-2.6.12-rc4-openib/drivers/infiniband/ulp/sdp/sdp_conn.c =================================================================== --- linux-2.6.12-rc4-openib/drivers/infiniband/ulp/sdp/sdp_conn.c (revision 2278) +++ linux-2.6.12-rc4-openib/drivers/infiniband/ulp/sdp/sdp_conn.c (working copy) @@ -312,7 +312,7 @@ int sdp_inet_port_get(struct sdp_opt *co static s32 rover = -1; unsigned long flags; - sk = conn->sk; + sk = sk_sdp(conn); /* * lock table */ @@ -323,7 +323,7 @@ int sdp_inet_port_get(struct sdp_opt *co if (port > 0) { for (look = dev_root_s.bind_list, port_ok = 1; look; look = look->bind_next) { - srch = look->sk; + srch = sk_sdp(look); /* * 1) same port * 2) linux force reuse is off. @@ -756,17 +756,8 @@ void sdp_conn_destruct(struct sdp_opt *c if (dump) sdp_conn_state_dump(conn); - /* - * free the OS socket structure - */ - if (!conn->sk) - sdp_dbg_warn(conn, "destruct, no socket! continuing."); - else { - sk_free(conn->sk); - conn->sk = NULL; - } - kmem_cache_free(dev_root_s.conn_cache, conn); + sk_free(sk_sdp(conn)); } /* @@ -1112,6 +1103,12 @@ error_attr: return result; } +static struct proto sdp_sk_proto = { + .name = "SDP", + .owner = THIS_MODULE, + .obj_size = sizeof(struct sdp_opt), +}; + /* * sdp_conn_alloc - allocate a new socket, and init. */ @@ -1121,8 +1118,7 @@ struct sdp_opt *sdp_conn_alloc(int prior struct sock *sk; int result; - sk = sk_alloc(dev_root_s.proto, priority, - sizeof(struct inet_sock), dev_root_s.sock_cache); + sk = sk_alloc(dev_root_s.proto, priority, &sdp_sk_proto, 1); if (!sk) { sdp_dbg_warn(NULL, "socket alloc error for protocol. <%d:%d>", dev_root_s.proto, priority); @@ -1146,23 +1142,8 @@ struct sdp_opt *sdp_conn_alloc(int prior sk->sk_state_change = sdp_inet_wake_generic; sk->sk_data_ready = sdp_inet_wake_recv; sk->sk_error_report = sdp_inet_wake_error; - /* - * Allocate must be called from process context, since QP - * create/modifies must be in that context. - */ - conn = kmem_cache_alloc(dev_root_s.conn_cache, priority); - if (!conn) { - sdp_dbg_warn(conn, "connection alloc error. <%d>", priority); - result = -ENOMEM; - goto error; - } - memset(conn, 0, sizeof(struct sdp_opt)); - /* - * The STRM interface specific data is map/cast over the TCP specific - * area of the sock. - */ - SDP_SET_CONN(sk, conn); + conn = sdp_sk(sk); SDP_CONN_ST_INIT(conn); conn->cm_id = NULL; @@ -1179,7 +1160,6 @@ struct sdp_opt *sdp_conn_alloc(int prior conn->parent = NULL; conn->pid = 0; - conn->sk = sk; conn->hashent = SDP_DEV_SK_INVALID; conn->istate = SDP_SOCK_ST_CLOSED; conn->flags = 0; @@ -1286,7 +1266,7 @@ struct sdp_opt *sdp_conn_alloc(int prior sdp_dbg_warn(conn, "Error <%d> conn table insert <%d:%d>", result, dev_root_s.sk_entry, dev_root_s.sk_size); - goto error_conn; + goto error; } /* * set reference @@ -1300,8 +1280,6 @@ struct sdp_opt *sdp_conn_alloc(int prior * done */ return conn; -error_conn: - kmem_cache_free(dev_root_s.conn_cache, conn); error: sk_free(sk); return NULL; @@ -1470,7 +1448,7 @@ int sdp_proc_dump_conn_data(char *buffer continue; conn = dev_root_s.sk_array[counter]; - sk = conn->sk; + sk = sk_sdp(conn); offset += sprintf((buffer + offset), SDP_PROC_CONN_DATA_FORM, conn->hashent, @@ -1956,26 +1934,13 @@ int sdp_conn_table_init(int proto_family goto error_iocb; } - dev_root_s.conn_cache = kmem_cache_create("sdp_conn", - sizeof(struct sdp_opt), - 0, SLAB_HWCACHE_ALIGN, - NULL, NULL); - if (!dev_root_s.conn_cache) { - sdp_warn("Failed to initialize connection cache."); + sdp_dbg_init("Registering socket proto."); + if (proto_register(&sdp_sk_proto, 1) != 0) { + sdp_warn("Failed to register sdp proto."); result = -ENOMEM; goto error_conn; } - dev_root_s.sock_cache = kmem_cache_create("sdp_sock", - sizeof(struct inet_sock), - 0, SLAB_HWCACHE_ALIGN, - NULL, NULL); - if (!dev_root_s.sock_cache) { - sdp_warn("Failed to initialize sock cache."); - result = -ENOMEM; - goto error_sock; - } - /* * start listening */ @@ -2002,9 +1967,7 @@ int sdp_conn_table_init(int proto_family error_listen: (void)ib_destroy_cm_id(dev_root_s.listen_id); error_cm_id: - kmem_cache_destroy(dev_root_s.sock_cache); -error_sock: - kmem_cache_destroy(dev_root_s.conn_cache); + proto_unregister(&sdp_sk_proto); error_conn: sdp_main_iocb_cleanup(); error_iocb: @@ -2045,14 +2008,7 @@ int sdp_conn_table_clear(void) * delete IOCB table */ sdp_main_iocb_cleanup(); - /* - * delete conn cache - */ - kmem_cache_destroy(dev_root_s.conn_cache); - /* - * delete sock cache - */ - kmem_cache_destroy(dev_root_s.sock_cache); + proto_unregister(&sdp_sk_proto); /* * stop listening */ Index: linux-2.6.12-rc4-openib/drivers/infiniband/ulp/sdp/sdp_actv.c =================================================================== --- linux-2.6.12-rc4-openib/drivers/infiniband/ulp/sdp/sdp_actv.c (revision 2278) +++ linux-2.6.12-rc4-openib/drivers/infiniband/ulp/sdp/sdp_actv.c (working copy) @@ -40,6 +40,7 @@ void sdp_cm_actv_error(struct sdp_opt *conn, int error) { int result; + struct sock *sk; /* * error value is positive error. * @@ -95,11 +96,12 @@ void sdp_cm_actv_error(struct sdp_opt *c conn->shutdown = SHUTDOWN_MASK; conn->send_buf = 0; - if (conn->sk->sk_socket) - conn->sk->sk_socket->state = SS_UNCONNECTED; + sk = sk_sdp(conn); + if (sk->sk_socket) + sk->sk_socket->state = SS_UNCONNECTED; sdp_iocb_q_cancel_all(conn, (0 - error)); - sdp_inet_wake_error(conn->sk); + sdp_inet_wake_error(sk); } /* @@ -116,7 +118,7 @@ static int sdp_cm_actv_establish(struct conn->src_addr, conn->src_port, conn->dst_addr, conn->dst_port); - sk = conn->sk; + sk = sk_sdp(conn); qp_attr = kmalloc(sizeof(*qp_attr), GFP_KERNEL); if (!qp_attr) @@ -549,7 +551,7 @@ int sdp_cm_connect(struct sdp_opt *conn) result = sdp_link_path_lookup(htonl(conn->dst_addr), htonl(conn->src_addr), - conn->sk->sk_bound_dev_if, + sk_sdp(conn)->sk_bound_dev_if, sdp_cm_path_complete, conn, &conn->plid); Index: linux-2.6.12-rc4-openib/drivers/infiniband/ulp/sdp/sdp_wall.c =================================================================== --- linux-2.6.12-rc4-openib/drivers/infiniband/ulp/sdp/sdp_wall.c (revision 2278) +++ linux-2.6.12-rc4-openib/drivers/infiniband/ulp/sdp/sdp_wall.c (working copy) @@ -294,8 +294,8 @@ int sdp_wall_recv_close(struct sdp_opt * /* * async notification. POLL_HUP on full duplex close only. */ - sdp_inet_wake_generic(conn->sk); - sk_wake_async(conn->sk, 1, POLL_IN); + sdp_inet_wake_generic(sk_sdp(conn)); + sk_wake_async(sk_sdp(conn), 1, POLL_IN); break; } @@ -327,8 +327,8 @@ int sdp_wall_recv_closing(struct sdp_opt /* * async notification. POLL_HUP on full duplex close only. */ - sdp_inet_wake_generic(conn->sk); - sk_wake_async(conn->sk, 1, POLL_HUP); + sdp_inet_wake_generic(sk_sdp(conn)); + sk_wake_async(sk_sdp(conn), 1, POLL_HUP); return 0; } @@ -368,7 +368,7 @@ int sdp_wall_recv_abort(struct sdp_opt * */ sdp_iocb_q_cancel_all(conn, -ECONNRESET); - sdp_inet_wake_error(conn->sk); + sdp_inet_wake_error(sk_sdp(conn)); return 0; } @@ -402,7 +402,7 @@ void sdp_wall_recv_drop(struct sdp_opt * break; case SDP_SOCK_ST_CLOSING: conn->istate = SDP_SOCK_ST_CLOSED; - sdp_inet_wake_generic(conn->sk); + sdp_inet_wake_generic(sk_sdp(conn)); break; default: @@ -418,7 +418,7 @@ void sdp_wall_recv_drop(struct sdp_opt * */ sdp_iocb_q_cancel_all(conn, -ECONNRESET); - sdp_inet_wake_error(conn->sk); + sdp_inet_wake_error(sk_sdp(conn)); break; } Index: linux-2.6.12-rc4-openib/drivers/infiniband/ulp/sdp/sdp_recv.c =================================================================== --- linux-2.6.12-rc4-openib/drivers/infiniband/ulp/sdp/sdp_recv.c (revision 2278) +++ linux-2.6.12-rc4-openib/drivers/infiniband/ulp/sdp/sdp_recv.c (working copy) @@ -783,7 +783,7 @@ static int sdp_recv_buff_iocb_pending(st */ if (!iocb->len || (!conn->src_recv && - !(conn->sk->sk_rcvlowat > iocb->post))) { + !(sk_sdp(conn)->sk_rcvlowat > iocb->post))) { /* * complete IOCB */ @@ -838,7 +838,7 @@ int sdp_recv_buff(struct sdp_opt *conn, */ if (buff->flags & SDP_BUFF_F_OOB_PEND) { conn->rcv_urg_cnt++; - sdp_inet_wake_urg(conn->sk); + sdp_inet_wake_urg(sk_sdp(conn)); } /* * loop while there are available IOCB's, break if there is no @@ -936,7 +936,7 @@ static int sdp_inet_read_cancel(struct k /* * lock the socket while we operate. */ - conn = SDP_GET_CONN(si->sock->sk); + conn = sdp_sk(si->sock->sk); sdp_conn_lock(conn); sdp_dbg_ctrl(conn, "Cancel Read IOCB. <%08x:%04x> <%08x:%04x>", @@ -1089,7 +1089,7 @@ static int sdp_inet_recv_urg(struct sock int result = 0; u8 value; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); if (sock_flag(sk, SOCK_URGINLINE) || !conn->rcv_urg_cnt) return -EINVAL; @@ -1176,7 +1176,7 @@ int sdp_inet_recv(struct kiocb *req, st struct sdpc_buff_q peek_queue; sk = sock->sk; - conn = SDP_GET_CONN(sk); + conn = sdp_sk(sk); sdp_dbg_data(conn, "state <%08x> size <%Zu> pending <%d> falgs <%08x>", conn->state, size, conn->byte_strm, flags); Index: linux-2.6.12-rc4-openib/drivers/infiniband/ulp/sdp/sdp_conn.h =================================================================== --- linux-2.6.12-rc4-openib/drivers/infiniband/ulp/sdp/sdp_conn.h (revision 2278) +++ linux-2.6.12-rc4-openib/drivers/infiniband/ulp/sdp/sdp_conn.h (working copy) @@ -146,16 +146,18 @@ enum sdp_mode { */ #define SDP_MSG_EVENT_TABLE_SIZE 0x20 -/* - * connection handle within a socket. - */ -#define SDP_GET_CONN(sk) \ - (*((struct sdp_opt **)&(sk)->sk_protinfo)) -#define SDP_SET_CONN(sk, conn) \ - (*((struct sdp_opt **)&(sk)->sk_protinfo) = (conn)) +static inline struct sdp_opt *sdp_sk(struct sock *sk) +{ + return (struct sdp_opt *)sk; +} + +static inline struct sock *sk_sdp(struct sdp_opt *conn) +{ + return (struct sock *)conn; +} #define SDP_CONN_SET_ERR(conn, val) \ - ((conn)->error = (conn)->sk->sk_err = (val)) + ((conn)->error = sk_sdp(conn)->sk_err = (val)) #define SDP_CONN_GET_ERR(conn) \ ((conn)->error) @@ -214,10 +216,15 @@ struct sdp_conn_lock { * SDP Connection structure. */ struct sdp_opt { + /* + * inet_sock must be first member of sdp_opt + * NOTE: this depends on inet_sock having struct sock as its + * first member + */ + struct inet_sock in; __s32 hashent; /* connection ID/hash entry */ atomic_t refcnt; /* connection reference count. */ - struct sock *sk; /* * SDP specific data */ @@ -530,7 +537,7 @@ static inline int sdp_conn_error(struct * lock, however the linux socket error, needs to be xchg'd since the * SO_ERROR getsockopt happens outside of the connection lock. */ - int error = xchg(&conn->sk->sk_err, 0); + int error = xchg(&sk_sdp(conn)->sk_err, 0); SDP_CONN_SET_ERR(conn, 0); return -error; Index: linux-2.6.12-rc4-openib/drivers/infiniband/ulp/sdp/sdp_pass.c =================================================================== --- linux-2.6.12-rc4-openib/drivers/infiniband/ulp/sdp/sdp_pass.c (revision 2278) +++ linux-2.6.12-rc4-openib/drivers/infiniband/ulp/sdp/sdp_pass.c (working copy) @@ -40,6 +40,7 @@ void sdp_cm_pass_error(struct sdp_opt *conn, int error) { int result; + struct sock *sk; sdp_dbg_ctrl(conn, "passive error. src <%08x:%04x> dst <%08x:%04x> <%d>", @@ -59,11 +60,12 @@ void sdp_cm_pass_error(struct sdp_opt *c conn->shutdown = SHUTDOWN_MASK; conn->send_buf = 0; - if (conn->sk->sk_socket) - conn->sk->sk_socket->state = SS_UNCONNECTED; + sk = sk_sdp(conn); + if (sk->sk_socket) + sk->sk_socket->state = SS_UNCONNECTED; sdp_iocb_q_cancel_all(conn, (0 - error)); - sdp_inet_wake_error(conn->sk); + sdp_inet_wake_error(sk); } /* @@ -130,7 +132,7 @@ int sdp_cm_pass_establish(struct sdp_opt goto error; } - sdp_inet_wake_send(conn->sk); + sdp_inet_wake_send(sk_sdp(conn)); kfree(qp_attr); return 0; @@ -320,8 +322,8 @@ static int sdp_cm_listen_lookup(struct s /* * check backlog */ - listen_sk = listen_conn->sk; - sk = conn->sk; + listen_sk = sk_sdp(listen_conn); + sk = sk_sdp(conn); if (listen_conn->backlog_cnt > listen_conn->backlog_max) { sdp_dbg_warn(listen_conn, @@ -356,13 +358,16 @@ static int sdp_cm_listen_lookup(struct s */ sk->sk_lingertime = listen_sk->sk_lingertime; sk->sk_rcvlowat = listen_sk->sk_rcvlowat; - sk->sk_debug = listen_sk->sk_debug; - sk->sk_localroute = listen_sk->sk_localroute; + if (sock_flag(listen_sk, SOCK_DBG)) + sock_set_flag(sk, SOCK_DBG); + if (sock_flag(listen_sk, SOCK_LOCALROUTE)) + sock_set_flag(sk, SOCK_LOCALROUTE); sk->sk_sndbuf = listen_sk->sk_sndbuf; sk->sk_rcvbuf = listen_sk->sk_rcvbuf; sk->sk_no_check = listen_sk->sk_no_check; sk->sk_priority = listen_sk->sk_priority; - sk->sk_rcvtstamp = listen_sk->sk_rcvtstamp; + if (sock_flag(listen_sk, SOCK_RCVTSTAMP)) + sock_set_flag(sk, SOCK_RCVTSTAMP); sk->sk_rcvtimeo = listen_sk->sk_rcvtimeo; sk->sk_sndtimeo = listen_sk->sk_sndtimeo; sk->sk_reuse = listen_sk->sk_reuse; @@ -501,7 +506,7 @@ int sdp_cm_req_handler(struct ib_cm_id * goto done; } /* - * Lock the new connection before modifyingg it into any tables. + * Lock the new connection before modifying it into any tables. */ sdp_conn_lock(conn); /* Index: linux-2.6.12-rc4-openib/drivers/infiniband/ulp/sdp/sdp_dev.h =================================================================== --- linux-2.6.12-rc4-openib/drivers/infiniband/ulp/sdp/sdp_dev.h (revision 2278) +++ linux-2.6.12-rc4-openib/drivers/infiniband/ulp/sdp/sdp_dev.h (working copy) @@ -197,11 +197,6 @@ struct sdev_root { * SDP wide listen */ struct ib_cm_id *listen_id; /* listen handle */ - /* - * cache's - */ - kmem_cache_t *conn_cache; - kmem_cache_t *sock_cache; }; #endif /* _SDP_DEV_H */ From tduffy at sun.com Sat May 7 16:41:09 2005 From: tduffy at sun.com (Tom Duffy) Date: Sat, 07 May 2005 16:41:09 -0700 Subject: [openib-general] [PATCH] SDP: remove redundant setting of sk_family Message-ID: <1115509269.23057.14.camel@duffman> sk_alloc() already takes care of setting sk_family to proto. No need to do it a second time in sdp. Signed-off-by: Tom Duffy Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_conn.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_conn.c (revision 2278) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_conn.c (working copy) @@ -1135,7 +1135,6 @@ struct sdp_opt *sdp_conn_alloc(int prior /* * other non-zero sock initialization. */ - sk->sk_family = dev_root_s.proto; sk->sk_protocol = IPPROTO_TCP; /* * some callbacks. higher levels of linux (e.g. setsockopt) call From mst at mellanox.co.il Sat May 7 22:23:42 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Sun, 8 May 2005 08:23:42 +0300 Subject: [openib-general] Re: [PATCH] sdp: replace mlock with get_user_pages In-Reply-To: <20050506175415.I26823@topspin.com> References: <20050505110158.GH30653@mellanox.co.il> <20050506175415.I26823@topspin.com> Message-ID: <20050508052342.GA14496@mellanox.co.il> Quoting r. Libor Michalek : > Subject: Re: [PATCH] sdp: replace mlock with get_user_pages > > On Thu, May 05, 2005 at 02:01:58PM +0300, Michael S. Tsirkin wrote: > > Hello, Libor! > > The following patch replaces the mlock hack with call > > to get_user_pages. Since the application could have forked > > while an iocb is outstanding, when an iocb is done > > I do get_user_pages for a second time and copy data if > > the physical address has changed. > > > > Thus, changing ulimit is no longer required to get aio > > working, processes are also allowed to fork and to call mlock/munlock > > on the buffer. > > > > Tested by the ttcp.aio benchmark, and works fine for me. > > In the latest kernel what happens to a page that's held > with a get_user_page reference on fork? Which is the only > case left. I dont think there's a change - the only other way to handle this is to block fork until an iocb is complete. > Since there is an iocv flag parameter it would > be prefered to use a bit there instead of is_receive. Also, > please leave spaces around '='. I'll try to address these. > Finally it would be nice > if we could get rid of having to schedule a task to unlock > the iocb in complete, I've noticed this having a negative > effect on system CPU utilization on older kernels. > > -Libor Even we get around the fact that get user pages needs an mm semaphore, its still nontrivial since set_page_dirty_lock might sleep. MST -- MST - Michael S. Tsirkin From mst at mellanox.co.il Sun May 8 05:49:16 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Sun, 8 May 2005 15:49:16 +0300 Subject: [openib-general] user-mode verbs on Itanium In-Reply-To: <20050507053706.GN23958@esmail.cup.hp.com> References: <1AC79F16F5C5284499BB9591B33D6F00043B37C9@orsmsx408> <20050429051009.GT20957@esmail.cup.hp.com> <52pswdk5au.fsf@topspin.com> <20050429162127.GA24871@esmail.cup.hp.com> <52ll71k09w.fsf@topspin.com> <20050429183846.GH24871@esmail.cup.hp.com> <20050504005534.GA13574@esmail.cup.hp.com> <20050504070910.GC9674@mellanox.co.il> <20050506190947.GA23958@esmail.cup.hp.com> <20050507053706.GN23958@esmail.cup.hp.com> Message-ID: <20050508124916.GM9941@mellanox.co.il> Quoting r. Grant Grundler : > Subject: Re: [openib-general] user-mode verbs on Itanium > > On Fri, May 06, 2005 at 12:09:47PM -0700, Grant Grundler wrote: > > Since the diff is essentially the whole file for everything but > > rdma_lat.c, I've parked the entire mess here: > > http://gsyprf3.external.hp.com/openib/perftest-01.tgz > > perftest-02.tgz is now available. > > And rdma_lat.c has substantial changes: > o -c option to display output in "raw" CPU cycles > o -U option to dump unsorted data (MORE ON THIS BELOW) I'll try to merge these two changes. > o fixed bugs and typos noted by Bernhard Fischer . > o fixed a few more bugs I found. Besides the new flags, I noticed the following changes: - asm/timex.h usage to get cycles. Unfortunately not portable to all platforms: on x86_64 asm/timex.h includes linux/config so its not legal for userspace to include. Please just implement get_cycles instead. - get_cpu_mhz() instead of get_cpu_khz() - is that important? Actually I planned to change get_cpu_khz() to cpu_khz to match linux in-kernel interface. Does this make sense to you? > I'm consistently getting 4.74 usec median latency with 2.6.11 kernel + > svn r2229 on HP ZX1 platform (PCI-X) + 1.5Ghz/6M Madison processors. Thats a bit higher latency than what I see with Intel, but at least in the right ballbark. > This seems kinda low so I'm not 100% comfortable with the > measurements yet. Maybe adding MPI on top of this will add > the 2-3 usec that I think is missing. > > [ If it's correct, I guess I can go back to working on some tg3 > driver tunes :^) Just kidding! It's late here... ] > > > BTW, Can I "leverage" code from ibv_pingpong to replace the issues > > the following comment in rdma_lat.c refers to? > > /* > > * pp_get_local_lid() uses a pretty bogus method for finding the LID > > * of a local port. Please don't copy this into your app (or if you > > * do, please rip it out soon). > > */ ibv_pingpong used to have the same comment too. I'll go back and look at ibv_pingpong. > This is still outstanding. But I'd like to first see perftest-02 > land in a sane place in openib.org Subversion tree. I could just copy stuff to say userspace/perftests. Is that OK with everyone? > I can then submit patches against some stuff: > o update the README with notes on how to use/interpret the data > o stop replicating code and make subroutines I considered this but I'm afraid that adding another layer on top of libibverbs would deduct from readability, and cause people to copy it wholesale, something which I would like to avoid. > o split up main() into bite-sized chunks so people > know which part is "initialization" and "run time". I guess adding more comments like /* Initialization */ is the way to go. > o stop pretending there is no global state and get major > variables off the stack and into .bss or respective > subroutines. Why is that good? Passing flags in global parameters would make the code less readable. > (I still want to hack a bit on mthca_cq.c:cqe_sw() too) > > > > Couldn't post send: scnt=1 > > Another clue about this failure: I'm only seeing this if I specify "-s". > 1,2,4,8,16,24,25,26,27,28 all worked. All the values > 29 failed. > (I tried 29, 30, 31, 32, 33, 64, 65, 4096, 8192). > I'm pretty busy next week...not sure I'll be able to > track it down then. I think thats because when I wrote the test qp attribute max_inline didnt exist. I'll update the code. > > Why I like -U > ------------- > (Sorry, bad pun. ggg hides :^) :) I'll merge the -U code, I see how it can be useful. > "unsorted" output is useful to: > o correlate client/server data (hiccups in algorithm/fabric) > o recognize startup vs "steady state" behaviors. > o recognize cyclical data (e.g. long runs where itimer interfers) > > The following is a discussion about the first two points. > Extra credit question buried in the middle. > > The unsorted output in CPU cycles of "server": > grundler at gsyprf3:/usr/src/trunk/contrib/mellanox/perftest$ ./rdma_lat -cU | head > local address: LID 0x0b, QPN 0x320406, PSN 0xeb1f3e RKey 0x1f60032 VAddr 0x6000000000014001 > remote address: LID 0x10, QPN 0x1a0406, PSN 0x6d8a50, RKey 0x1040436 VAddr 0x6000000000014001 > #, cycles > 1, 97779 > 2, 52050 > 3, 7028 > 4, 7318 > 5, 7201 > ... > > And the same from the "client": > grundler at iota:/usr/src/trunk/contrib/mellanox/perftest$ ./rdma_lat -cU 10.0.0.51 | head > local address: LID 0x10, QPN 0x1a0406, PSN 0x6d8a50 RKey 0x1040436 VAddr 0x6000000000014001 > remote address: LID 0x0b, QPN 0x320406, PSN 0xeb1f3e, RKey 0x1f60032 VAddr 0x6000000000014001 > #, cycles > 1, 611 > 2, 93491 > 3, 27129 > 4, 7142 > 5, 7260 > ... > > The first sample (611) on the client is improbably small for a 1.5Ghz > system (i.e ~0.4 usecs). That is a clue that delta[0] means > something different than delta[{N>0}] on the client. > It might be accurate if the client sends first and the next cycle count > is taken right after telling the card data is ready to send. > This could be useful data too. > Michael (mst), is it obvious to you if I've understood that correctly? > Either way, I'd like to leave the test as is and add the explanation > to the README. I'll look into this. > The 2cd and 3rd sample sort-of match the first two samples from the > server and are plausible (~63 usecs). The fact that client/server > closely agree on the ~63 usec (93k cycles) sample is a good clue > I can trust the measurement. > > That first sample on the server represents a tuning opportunity. > A ~10x difference between startup and runtime is significant > for short lived connections/regions. On the other hand, it could be negligible compared to connection setup time. > It might be in the switch (TS90), HCA, or host SW stack. > I don't know. I'd need a PCI-X bus traces, HW stats from > the HCA or Switch to determine how much each contributes > to the latency. > > I hope it's obvious now why -U is interesting. > > thanks, > grant > OK, so I see 3 changes there: 1. support for -U and -c I'll merge that 2. get_clock in assembly replaced with asm/timex.h: Unfortunately asm/timex.h was never intended for userspace, so this trick doesnt work on all platforms: specifically its broken on ppc,i386 and x86_64, so ppc64 and ia64 where it does work are kind of in the minority. And given that even these may be broken on some distros, I suggest we simply add an assembly implementation for now, rather than add this dependency. Grant, you mentioned some other fixes - what are they? Maybe I'll notice them when I do the actual merge. MST -- MST - Michael S. Tsirkin From mst at mellanox.co.il Sun May 8 05:52:00 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Sun, 8 May 2005 15:52:00 +0300 Subject: [openib-general] perftests location Message-ID: <20050508125200.GN9941@mellanox.co.il> Guys, I'd like to copy perftests to under src/userspace. It seems that multiple people want to work on it and its kind of inconvenient to work under contrib. I'll do this tomorrow unless I hear otherwise (we can always move it later if we want to). -- MST - Michael S. Tsirkin From mst at mellanox.co.il Sun May 8 05:59:43 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Sun, 8 May 2005 15:59:43 +0300 Subject: [openib-general] Re: [PATCH] sdp: replace mlock with get_user_pages In-Reply-To: <20050506175415.I26823@topspin.com> References: <20050505110158.GH30653@mellanox.co.il> <20050506175415.I26823@topspin.com> Message-ID: <20050508125943.GO9941@mellanox.co.il> Quoting r. Libor Michalek : > Subject: Re: [PATCH] sdp: replace mlock with get_user_pages > > On Thu, May 05, 2005 at 02:01:58PM +0300, Michael S. Tsirkin wrote: > > Hello, Libor! > > The following patch replaces the mlock hack with call > > to get_user_pages. Since the application could have forked > > while an iocb is outstanding, when an iocb is done > > I do get_user_pages for a second time and copy data if > > the physical address has changed. > > > > Thus, changing ulimit is no longer required to get aio > > working, processes are also allowed to fork and to call mlock/munlock > > on the buffer. > > > > Tested by the ttcp.aio benchmark, and works fine for me. > > In the latest kernel what happens to a page that's held > with a get_user_page reference on fork? Which is the only > case left. I looked in fork.c and I dont see anything related to outstanding aios in the fork flow. Do you? -- MST - Michael S. Tsirkin From mst at mellanox.co.il Sun May 8 07:23:45 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Sun, 8 May 2005 17:23:45 +0300 Subject: [openib-general] Re: user-mode verbs on Itanium In-Reply-To: <20050507053706.GN23958@esmail.cup.hp.com> References: <1AC79F16F5C5284499BB9591B33D6F00043B37C9@orsmsx408> <20050429051009.GT20957@esmail.cup.hp.com> <52pswdk5au.fsf@topspin.com> <20050429162127.GA24871@esmail.cup.hp.com> <52ll71k09w.fsf@topspin.com> <20050429183846.GH24871@esmail.cup.hp.com> <20050504005534.GA13574@esmail.cup.hp.com> <20050504070910.GC9674@mellanox.co.il> <20050506190947.GA23958@esmail.cup.hp.com> <20050507053706.GN23958@esmail.cup.hp.com> Message-ID: <20050508142345.GQ9941@mellanox.co.il> Quoting r. Grant Grundler : > Subject: Re: user-mode verbs on Itanium > > The first sample (611) on the client is improbably small for a 1.5Ghz > system (i.e ~0.4 usecs). That is a clue that delta[0] means > something different than delta[{N>0}] on the client. > It might be accurate if the client sends first and the next cycle count > is taken right after telling the card data is ready to send. > This could be useful data too. > > Michael (mst), is it obvious to you if I've understood that correctly? > Either way, I'd like to leave the test as is and add the explanation > to the README. Actually, the test had a bug - results were stored in tstamp[1]...tstamp[1001] instead of tstamp[0]...tstam[999]. Your patch initialized tstamp[0] to get_clock which still isnt right since its take before post_send and not after that. revision 2279 fixes this by storing the clock in tstamp[scnt - 1]. Thanks! MST -- MST - Michael S. Tsirkin From mst at mellanox.co.il Sun May 8 07:29:30 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Sun, 8 May 2005 17:29:30 +0300 Subject: [openib-general] [PATCH] libibverbs/examples: fix parameter parsing Message-ID: <20050508142930.GR9941@mellanox.co.il> Fix for two obvious typos in pingpong and us-pingpong parameter parsing (found by Grant in rdma_lat code). Signed-off-by: Michael S. Tsirkin Index: examples/pingpong.c =================================================================== --- examples/pingpong.c (revision 2278) +++ examples/pingpong.c (working copy) @@ -474,7 +474,7 @@ int main(int argc, char *argv[]) case 'i': ib_port = strtol(optarg, NULL, 0); - if (port < 0) { + if (ib_port < 0) { usage(argv[0]); return 1; } Index: examples/ud-pingpong.c =================================================================== --- examples/ud-pingpong.c (revision 2278) +++ examples/ud-pingpong.c (working copy) @@ -473,7 +473,7 @@ int main(int argc, char *argv[]) case 'i': ib_port = strtol(optarg, NULL, 0); - if (port < 0) { + if (ib_port < 0) { usage(argv[0]); return 1; } -- MST - Michael S. Tsirkin From mst at mellanox.co.il Sun May 8 07:52:42 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Sun, 8 May 2005 17:52:42 +0300 Subject: [openib-general] Re: user-mode verbs on Itanium In-Reply-To: <20050507053706.GN23958@esmail.cup.hp.com> References: <1AC79F16F5C5284499BB9591B33D6F00043B37C9@orsmsx408> <20050429051009.GT20957@esmail.cup.hp.com> <52pswdk5au.fsf@topspin.com> <20050429162127.GA24871@esmail.cup.hp.com> <52ll71k09w.fsf@topspin.com> <20050429183846.GH24871@esmail.cup.hp.com> <20050504005534.GA13574@esmail.cup.hp.com> <20050504070910.GC9674@mellanox.co.il> <20050506190947.GA23958@esmail.cup.hp.com> <20050507053706.GN23958@esmail.cup.hp.com> Message-ID: <20050508145242.GS9941@mellanox.co.il> Quoting r. Grant Grundler : > Subject: Re: user-mode verbs on Itanium > > > Couldn't post send: scnt=1 > > Another clue about this failure: I'm only seeing this if I specify "-s". > 1,2,4,8,16,24,25,26,27,28 all worked. All the values > 29 failed. > (I tried 29, 30, 31, 32, 33, 64, 65, 4096, 8192). > I'm pretty busy next week...not sure I'll be able to > track it down then. > Seems like a bug in mthca (kernel component): /usr/src/linux-2.6.11/drivers/infiniband/hw/mthca # grep max_inline *c mthca_provider.c: init_attr->cap.max_inline_data = 0; Roland - it seems like max_inline_data is being ignored instead of being used for calculating send wqe size. Want a patch to fix that? Grant - you can work around this bug for now by changing .max_send_sge to a bigger value. -- MST - Michael S. Tsirkin From info at kqsv10.com Sat May 7 19:28:43 2005 From: info at kqsv10.com (info at kqsv10.com) Date: 8 May 2005 11:28:43 +0900 Subject: [openib-general] $BL5NA(B10000$B1_J,$GM7$s$G$_$h$&!*(B Message-ID: <20050508022843.6115.qmail@mail.kqsv10.com> _/_/_/_/_/$B5U1g!y?M:J!y3d at Z!y(BSM$B!yB(2q$$!yD>%aD>EE(B_/_/_/_/_/ _/_/_/_/_/_/ http://awg.webchu.com/?springj _/_/_/_/_/_/ $B!xBg9%I>!V(B10000$B1_!WL5NA%]%$%s%HB#Dh%-%c%s%Z!<%s7QB3@-$,Hs>o$K9b$/!"0l;~@$4V$rFx$o$;$?2M6u at A5a$H$bEvJ}$O0l at Z4X$o$j$,$"$j$^$;$s$N$G0B?4$7$F$4MxMQ2<$5$$!#(B $B!}L5NA$G$?$C$W$jM7$s$GD:$-!"!V$3$3$J$i!*!W$H;W$C$FD:$1$?J}$O0B?4$NA0J'$$$r$4MxMQ2<$5$$!#$?$C$?(B3000$B1_$+$i$4MxMQD:$1$^$9!#(B $B"(Ev%a!<%k$NG[?.5qH]$O$3$A$i"*(B csvbit7q8a9z at ok.kz $B$^$G$* Hello, We tried contacting you awhile ago about your low interest morta(ge rate. You have qualified for the lowest rate in years... You could get over $380,000 for as little as $500 a month! Ba(d credit? Doesn't matter, low rates are fixed no matter what! To get a free, no obli,gation consultation click below: http://www.trust1ng.com/sign.asp Best Regards, Bethany Enriquez to be remov(ed: http://www.trust1ng.com/gone.asp this process takes one week, so please be patient. we do our best to take your email/s off but you have to fill out a rem/ove or else you will continue to recieve email/s. From mst at mellanox.co.il Mon May 9 07:14:56 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Mon, 9 May 2005 17:14:56 +0300 Subject: [openib-general] [PATCH 1/2] mthca: max_inline_data support Message-ID: <20050509141456.GH9941@mellanox.co.il> Kernel part of a patch: support max_inline_data capability in mthca. Signed-off-by: Michael S. Tsirkin Index: hw/mthca/mthca_dev.h =================================================================== --- hw/mthca/mthca_dev.h (revision 2281) +++ hw/mthca/mthca_dev.h (working copy) @@ -439,12 +439,14 @@ int mthca_alloc_qp(struct mthca_dev *dev struct mthca_cq *recv_cq, enum ib_qp_type type, enum ib_sig_type send_policy, + struct ib_qp_cap *cap, struct mthca_qp *qp); int mthca_alloc_sqp(struct mthca_dev *dev, struct mthca_pd *pd, struct mthca_cq *send_cq, struct mthca_cq *recv_cq, enum ib_sig_type send_policy, + struct ib_qp_cap *cap, int qpn, int port, struct mthca_sqp *sqp); Index: hw/mthca/mthca_provider.c =================================================================== --- hw/mthca/mthca_provider.c (revision 2281) +++ hw/mthca/mthca_provider.c (working copy) @@ -486,16 +486,11 @@ static struct ib_qp *mthca_create_qp(str qp->rq.db_index = ucmd.rq_db_index; } - qp->sq.max = init_attr->cap.max_send_wr; - qp->rq.max = init_attr->cap.max_recv_wr; - qp->sq.max_gs = init_attr->cap.max_send_sge; - qp->rq.max_gs = init_attr->cap.max_recv_sge; - err = mthca_alloc_qp(to_mdev(pd->device), to_mpd(pd), to_mcq(init_attr->send_cq), to_mcq(init_attr->recv_cq), init_attr->qp_type, init_attr->sq_sig_type, - qp); + &init_attr->cap, qp); if (err && pd->uobject) { context = to_mucontext(pd->uobject->context); @@ -524,17 +519,12 @@ static struct ib_qp *mthca_create_qp(str if (!qp) return ERR_PTR(-ENOMEM); - qp->sq.max = init_attr->cap.max_send_wr; - qp->rq.max = init_attr->cap.max_recv_wr; - qp->sq.max_gs = init_attr->cap.max_send_sge; - qp->rq.max_gs = init_attr->cap.max_recv_sge; - qp->ibqp.qp_num = init_attr->qp_type == IB_QPT_SMI ? 0 : 1; err = mthca_alloc_sqp(to_mdev(pd->device), to_mpd(pd), to_mcq(init_attr->send_cq), to_mcq(init_attr->recv_cq), - init_attr->sq_sig_type, + init_attr->sq_sig_type, &init_attr->cap, qp->ibqp.qp_num, init_attr->port_num, to_msqp(qp)); break; @@ -549,10 +539,6 @@ static struct ib_qp *mthca_create_qp(str return ERR_PTR(err); } - init_attr->cap.max_inline_data = 0; - init_attr->cap.max_send_wr = qp->sq.max; - init_attr->cap.max_recv_wr = qp->rq.max; - return &qp->ibqp; } Index: hw/mthca/mthca_qp.c =================================================================== --- hw/mthca/mthca_qp.c (revision 2281) +++ hw/mthca/mthca_qp.c (working copy) @@ -46,7 +46,9 @@ enum { MTHCA_MAX_DIRECT_QP_SIZE = 4 * PAGE_SIZE, MTHCA_ACK_REQ_FREQ = 10, MTHCA_FLIGHT_LIMIT = 9, - MTHCA_UD_HEADER_SIZE = 72 /* largest UD header possible */ + MTHCA_UD_HEADER_SIZE = 72, /* largest UD header possible */ + MTHCA_INLINE_HEADER_SIZE = 4, /* data segment overhead for inline */ + MTHCA_INLINE_CHUNK_SIZE = 16 /* inline data segment chunk */ }; enum { @@ -1205,22 +1207,42 @@ static int mthca_alloc_qp_common(struct return 0; } -static void mthca_align_qp_size(struct mthca_dev *dev, struct mthca_qp *qp) +static void mthca_set_qp_size(struct mthca_dev *dev, struct ib_qp_cap* cap, + struct mthca_qp *qp) { int i; + u32 d; - if (!mthca_is_memfree(dev)) - return; + if (mthca_is_memfree(dev)) { + for (i = 0; 1 << i < cap->max_recv_wr; ++i) + ; /* nothing */ - for (i = 0; 1 << i < qp->rq.max; ++i) - ; /* nothing */ + qp->rq.max = 1 << i; - qp->rq.max = 1 << i; + for (i = 0; 1 << i < cap->max_send_wr; ++i) + ; /* nothing */ - for (i = 0; 1 << i < qp->sq.max; ++i) - ; /* nothing */ + qp->sq.max = 1 << i; + } else { + qp->rq.max = cap->max_recv_wr; + qp->sq.max = cap->max_send_wr; + } + + qp->rq.max_gs = cap->max_recv_sge; + + d = ALIGN(cap->max_inline_data + MTHCA_INLINE_HEADER_SIZE, + MTHCA_INLINE_CHUNK_SIZE) / sizeof (struct mthca_data_seg); + qp->sq.max_gs = max(cap->max_send_sge, d); + + if (qp->transport == MLX) { + /* For MLX transport we need 2 extra S/G entries: + * for header and checksum */ + qp->sq.max_gs += 2; + } - qp->sq.max = 1 << i; + cap->max_inline_data = qp->sq.max_gs * sizeof (struct mthca_data_seg); + cap->max_send_wr = qp->sq.max; + cap->max_recv_wr = qp->rq.max; } int mthca_alloc_qp(struct mthca_dev *dev, @@ -1229,11 +1251,12 @@ int mthca_alloc_qp(struct mthca_dev *dev struct mthca_cq *recv_cq, enum ib_qp_type type, enum ib_sig_type send_policy, + struct ib_qp_cap *cap, struct mthca_qp *qp) { int err; - mthca_align_qp_size(dev, qp); + mthca_set_qp_size(dev, cap, qp); switch (type) { case IB_QPT_RC: qp->transport = RC; break; @@ -1266,6 +1289,7 @@ int mthca_alloc_sqp(struct mthca_dev *de struct mthca_cq *send_cq, struct mthca_cq *recv_cq, enum ib_sig_type send_policy, + struct ib_qp_cap *cap, int qpn, int port, struct mthca_sqp *sqp) @@ -1273,7 +1297,7 @@ int mthca_alloc_sqp(struct mthca_dev *de int err = 0; u32 mqpn = qpn * 2 + dev->qp_table.sqp_start + port - 1; - mthca_align_qp_size(dev, &sqp->qp); + mthca_set_qp_size(dev, cap, &sqp->qp); sqp->header_buf_size = sqp->qp.sq.max * MTHCA_UD_HEADER_SIZE; sqp->header_buf = dma_alloc_coherent(&dev->pdev->dev, sqp->header_buf_size, -- MST - Michael S. Tsirkin From mst at mellanox.co.il Mon May 9 07:16:31 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Mon, 9 May 2005 17:16:31 +0300 Subject: [openib-general] [PATCH 2/2] libmthca: max_inline_data support Message-ID: <20050509141631.GI9941@mellanox.co.il> Userspace part of a patch: support max_inline_data capability. Signed-off-by: Michael S. Tsirkin Index: libmthca/src/qp.c =================================================================== --- libmthca/src/qp.c (revision 2235) +++ libmthca/src/qp.c (working copy) @@ -718,10 +718,18 @@ out: return ret; } -int mthca_alloc_qp_buf(struct ibv_pd *pd, struct mthca_qp *qp) +int mthca_alloc_qp_buf(struct ibv_pd *pd, struct ibv_qp_cap *cap, + struct mthca_qp *qp) { int size; + qp->rq.max_gs = cap->max_recv_sge; + qp->sq.max_gs = align(cap->max_inline_data + + sizeof (struct mthca_inline_seg), + sizeof (struct mthca_data_seg)); + if (qp->sq.max_gs < cap->max_send_sge) + qp->sq.max_gs = cap->max_send_sge; + qp->wrid = malloc((qp->rq.max + qp->sq.max) * sizeof (uint64_t)); if (!qp->wrid) return -1; Index: libmthca/src/mthca.h =================================================================== --- libmthca/src/mthca.h (revision 2235) +++ libmthca/src/mthca.h (working copy) @@ -284,7 +284,8 @@ extern int mthca_arbel_post_send(struct struct ibv_send_wr **bad_wr); extern int mthca_arbel_post_recv(struct ibv_qp *ibqp, struct ibv_recv_wr *wr, struct ibv_recv_wr **bad_wr); -extern int mthca_alloc_qp_buf(struct ibv_pd *pd, struct mthca_qp *qp); +extern int mthca_alloc_qp_buf(struct ibv_pd *pd, struct ibv_qp_cap *, + struct mthca_qp *qp); extern struct mthca_qp *mthca_find_qp(struct mthca_context *ctx, uint32_t qpn); extern int mthca_store_qp(struct mthca_context *ctx, uint32_t qpn, struct mthca_qp *qp); extern void mthca_clear_qp(struct mthca_context *ctx, uint32_t qpn); Index: libmthca/src/verbs.c =================================================================== --- libmthca/src/verbs.c (revision 2235) +++ libmthca/src/verbs.c (working copy) @@ -270,7 +270,6 @@ struct ibv_qp *mthca_create_qp(struct ib qp->qpt = attr->qp_type; qp->sq.max = align_qp_size(pd->context, attr->cap.max_send_wr); - qp->sq.max_gs = attr->cap.max_send_sge; qp->sq.next_ind = 0; qp->sq.last_comp = qp->sq.max - 1; qp->sq.head = 0; @@ -278,14 +277,13 @@ struct ibv_qp *mthca_create_qp(struct ib qp->sq.last = NULL; qp->rq.max = align_qp_size(pd->context, attr->cap.max_recv_wr); - qp->rq.max_gs = attr->cap.max_recv_sge; qp->rq.next_ind = 0; qp->rq.last_comp = qp->rq.max - 1; qp->rq.head = 0; qp->rq.tail = 0; qp->rq.last = NULL; - if (mthca_alloc_qp_buf(pd, qp)) + if (mthca_alloc_qp_buf(pd, &attr->cap, qp)) goto err; if (pthread_spin_init(&qp->sq.lock, PTHREAD_PROCESS_PRIVATE) || -- MST - Michael S. Tsirkin From jlentini at netapp.com Mon May 9 07:29:40 2005 From: jlentini at netapp.com (James Lentini) Date: Mon, 9 May 2005 10:29:40 -0400 (EDT) Subject: [openib-general] [PATCH] kDAPL: remove DAT_UINT32 and DAT_UINT64 In-Reply-To: <1115413920.4476.1.camel@localhost.localdomain> References: <1115411060.4476.346.camel@localhost.localdomain> <1115413920.4476.1.camel@localhost.localdomain> Message-ID: >> Let me know if you still have trouble. > > Works for me. Hope it does for Tom. > >>> I also couldn't seem to get the -M PHY to work: >>> DT_BpoolAlloc: dat_lmr_kcreate failed DAT_SUCCESS DAT_INVALID_ARG7 > > Have you tried the -M option ? Do you see this ? What type of memory did you specify? -M PHY? From halr at voltaire.com Mon May 9 07:36:24 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 09 May 2005 10:36:24 -0400 Subject: [openib-general] [PATCH] kDAPL: remove DAT_UINT32 and DAT_UINT64 In-Reply-To: References: <1115411060.4476.346.camel@localhost.localdomain> <1115413920.4476.1.camel@localhost.localdomain> Message-ID: <1115649383.4490.1.camel@localhost.localdomain> On Mon, 2005-05-09 at 10:29, James Lentini wrote: > >>> I also couldn't seem to get the -M PHY to work: > >>> DT_BpoolAlloc: dat_lmr_kcreate failed DAT_SUCCESS DAT_INVALID_ARG7 > > > > Have you tried the -M option ? Do you see this ? > > What type of memory did you specify? -M PHY? -M PHY or -M VIRT. I expected a different error with -M VIR. It doesn't seem to like the -M option and says invalid arg7 always. -- Hal From jlentini at netapp.com Mon May 9 07:39:55 2005 From: jlentini at netapp.com (James Lentini) Date: Mon, 9 May 2005 10:39:55 -0400 (EDT) Subject: [openib-general] Re: kdapltest works again!, still having slab corruption In-Reply-To: <1115481438.5263.4.camel@duffman> References: <1115411060.4476.346.camel@localhost.localdomain> <1115413920.4476.1.camel@localhost.localdomain> <1115425669.19661.14.camel@duffman> <1115445245.4733.26.camel@localhost.localdomain> <1115481438.5263.4.camel@duffman> Message-ID: Shahar, The slab corruption message below seems to involve IBAT's req_comp_work function. Any ideas? james On Sat, 7 May 2005, Tom Duffy wrote: > On Sat, 2005-05-07 at 01:54 -0400, Hal Rosenstock wrote: >> Is the client connecting to a server which is present or not ? > > Yes, the client *is* connecting to the server. Here is the full client > side transaction: > > [root at sins-stinger-10 ~]# ./kdapltest -T Q -s 192.168.0.26 -D mthca0a -d > Server Name: 192.168.0.26 > Server Net Address: 192.168.0.26 > DT_cs_Client: Starting Test ... > DT_cs_Client: IA mthca0a opened > DT_cs_Client: EP created > ***** DAPL Characteristics ***** > Provider: mthca0a Version 1.0 DAPL 1.2 > Adapter: Generic InfiniBand HCA by DAPL Reference Implementation Version 0.0 > Supporting: > 64512 EPs with 65535 DTOs and 0 in RDMA/RDs and 0ut RDMA/RDs each > 65408 EVDs of up to 65535 entries (default S/R size is 16/16) > IOVs of up to 28 elements > 131056 LMRs (and 131056 RMRs) of up to 0xffffffffffffffff bytes > Maximum MTU 0x80000000 bytes, RDMA 0x80000000 bytes > Maximum Private data size 92 bytes > ***** ***** ***** ***** ***** ***** > DT_cs_Client: Posting 1 recv buffer > DT_cs_Client: Connect Endpoint > DT_cs_Client: Await connection ... > DT_cs_Client: Connected! > DAT_STATE: DAT_EP_STATE_CONNECTED > DAT_STATE: Inbound DTO Status: Active > DAT_STATE: Outbound DTO Status: Idle > DT_cs_Client: Sending Client_Info > DT_cs_Client: Sent Client_Info - awaiting completion > DT_cs_Client: Sending Command > DT_cs_Client: Sent Command - awaiting completion > DT_cs_Client: Waiting for Server_Info > DT_cs_Client: Server_Info Received > DT_cs_Client: Version OK! > ------------------------------------- > Server_Info.dapltest_version : 6 > Server_Info.is_little_endian : 1 > ------------------------------------- > Client_Info.dapltest_version : 6 > Client_Info.is_little_endian : 1 > Client_Info.test_type : 4 > Quit_Cmd.server_name: 192.168.0.26 > Quit_Cmd.device_name: mthca0a > DT_cs_Client: Cleaning Up ... > DT_cs_Client: IA mthca0a closed > TEST INSTANCE 0 > TEST return code = 0 > [root at sins-stinger-10 ~]# Slab corruption: start=ffff81003b2eddb0, len=288 > Redzone: 0x5a2cf071/0x5a2cf071. > Last user: [](req_comp_work+0x54/0x90 [ib_at]) > 040: 00 00 00 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b > 110: 00 00 00 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b a5 > Prev obj: start=ffff81003b2edc78, len=288 > Redzone: 0x5a2cf071/0x5a2cf071. > Last user: [<0000000000000000>](stext+0x7feff0e8/0xe8) > 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > Slab corruption: start=ffff81003a026eb8, len=312 > Redzone: 0x5a2cf071/0x5a2cf071. > Last user: [](req_comp_work+0x42/0x90 [ib_at]) > 050: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 00 00 00 00 > 120: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 00 00 00 00 > Prev obj: start=ffff81003a026d68, len=312 > Redzone: 0x5a2cf071/0x5a2cf071. > Last user: [<0000000000000000>](stext+0x7feff0e8/0xe8) > 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > > > From halr at voltaire.com Mon May 9 07:46:53 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 09 May 2005 10:46:53 -0400 Subject: [openib-general] Re: kdapltest works again!, still having slab corruption In-Reply-To: References: <1115411060.4476.346.camel@localhost.localdomain> <1115413920.4476.1.camel@localhost.localdomain> <1115425669.19661.14.camel@duffman> <1115445245.4733.26.camel@localhost.localdomain> <1115481438.5263.4.camel@duffman> Message-ID: <1115650013.4490.16.camel@localhost.localdomain> On Mon, 2005-05-09 at 10:39, James Lentini wrote: > The slab corruption message below seems to involve IBAT's > req_comp_work function. Any ideas? I looked at this over the weekend and still don't have a working theory. req_comp_work calls req_free which based on the pend->type frees the (in this case) path request (calls free_path_req). It is the same as the one which was allocated from the path request cache in ib_at_paths_by_route. We are still looking at this. -- Hal > > james > > On Sat, 7 May 2005, Tom Duffy wrote: > > > On Sat, 2005-05-07 at 01:54 -0400, Hal Rosenstock wrote: > >> Is the client connecting to a server which is present or not ? > > > > Yes, the client *is* connecting to the server. Here is the full client > > side transaction: > > > > [root at sins-stinger-10 ~]# ./kdapltest -T Q -s 192.168.0.26 -D mthca0a -d > > Server Name: 192.168.0.26 > > Server Net Address: 192.168.0.26 > > DT_cs_Client: Starting Test ... > > DT_cs_Client: IA mthca0a opened > > DT_cs_Client: EP created > > ***** DAPL Characteristics ***** > > Provider: mthca0a Version 1.0 DAPL 1.2 > > Adapter: Generic InfiniBand HCA by DAPL Reference Implementation Version 0.0 > > Supporting: > > 64512 EPs with 65535 DTOs and 0 in RDMA/RDs and 0ut RDMA/RDs each > > 65408 EVDs of up to 65535 entries (default S/R size is 16/16) > > IOVs of up to 28 elements > > 131056 LMRs (and 131056 RMRs) of up to 0xffffffffffffffff bytes > > Maximum MTU 0x80000000 bytes, RDMA 0x80000000 bytes > > Maximum Private data size 92 bytes > > ***** ***** ***** ***** ***** ***** > > DT_cs_Client: Posting 1 recv buffer > > DT_cs_Client: Connect Endpoint > > DT_cs_Client: Await connection ... > > DT_cs_Client: Connected! > > DAT_STATE: DAT_EP_STATE_CONNECTED > > DAT_STATE: Inbound DTO Status: Active > > DAT_STATE: Outbound DTO Status: Idle > > DT_cs_Client: Sending Client_Info > > DT_cs_Client: Sent Client_Info - awaiting completion > > DT_cs_Client: Sending Command > > DT_cs_Client: Sent Command - awaiting completion > > DT_cs_Client: Waiting for Server_Info > > DT_cs_Client: Server_Info Received > > DT_cs_Client: Version OK! > > ------------------------------------- > > Server_Info.dapltest_version : 6 > > Server_Info.is_little_endian : 1 > > ------------------------------------- > > Client_Info.dapltest_version : 6 > > Client_Info.is_little_endian : 1 > > Client_Info.test_type : 4 > > Quit_Cmd.server_name: 192.168.0.26 > > Quit_Cmd.device_name: mthca0a > > DT_cs_Client: Cleaning Up ... > > DT_cs_Client: IA mthca0a closed > > TEST INSTANCE 0 > > TEST return code = 0 > > [root at sins-stinger-10 ~]# Slab corruption: start=ffff81003b2eddb0, len=288 > > Redzone: 0x5a2cf071/0x5a2cf071. > > Last user: [](req_comp_work+0x54/0x90 [ib_at]) > > 040: 00 00 00 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b 6b > > 110: 00 00 00 00 00 00 00 00 6b 6b 6b 6b 6b 6b 6b a5 > > Prev obj: start=ffff81003b2edc78, len=288 > > Redzone: 0x5a2cf071/0x5a2cf071. > > Last user: [<0000000000000000>](stext+0x7feff0e8/0xe8) > > 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > > 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > > Slab corruption: start=ffff81003a026eb8, len=312 > > Redzone: 0x5a2cf071/0x5a2cf071. > > Last user: [](req_comp_work+0x42/0x90 [ib_at]) > > 050: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 00 00 00 00 > > 120: 6b 6b 6b 6b 6b 6b 6b 6b 00 00 00 00 00 00 00 00 > > Prev obj: start=ffff81003a026d68, len=312 > > Redzone: 0x5a2cf071/0x5a2cf071. > > Last user: [<0000000000000000>](stext+0x7feff0e8/0xe8) > > 000: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > > 010: 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b 6b > > > > > > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From jlentini at netapp.com Mon May 9 08:47:05 2005 From: jlentini at netapp.com (James Lentini) Date: Mon, 9 May 2005 11:47:05 -0400 (EDT) Subject: [kdapl] kdapltest -M flag (was RE: [openib-general] [PATCH] kDAPL: remove DAT_UINT32 and DAT_UINT64) In-Reply-To: <1115649383.4490.1.camel@localhost.localdomain> References: <1115411060.4476.346.camel@localhost.localdomain> <1115413920.4476.1.camel@localhost.localdomain> <1115649383.4490.1.camel@localhost.localdomain> Message-ID: What is the exact command line you are using? This works for me ./kdapltest -T T -D mthca0a -s 192.168.0.6 -M PHY client SR 256 server SR 256 On Mon, 9 May 2005, Hal Rosenstock wrote: > On Mon, 2005-05-09 at 10:29, James Lentini wrote: >>>>> I also couldn't seem to get the -M PHY to work: >>>>> DT_BpoolAlloc: dat_lmr_kcreate failed DAT_SUCCESS DAT_INVALID_ARG7 >>> >>> Have you tried the -M option ? Do you see this ? >> >> What type of memory did you specify? -M PHY? > > -M PHY or -M VIRT. I expected a different error with -M VIR. It doesn't > seem to like the -M option and says invalid arg7 always. > > -- Hal > From iod00d at hp.com Mon May 9 08:57:38 2005 From: iod00d at hp.com (Grant Grundler) Date: Mon, 9 May 2005 08:57:38 -0700 Subject: [openib-general] user-mode verbs on Itanium In-Reply-To: <20050508124916.GM9941@mellanox.co.il> References: <20050429051009.GT20957@esmail.cup.hp.com> <52pswdk5au.fsf@topspin.com> <20050429162127.GA24871@esmail.cup.hp.com> <52ll71k09w.fsf@topspin.com> <20050429183846.GH24871@esmail.cup.hp.com> <20050504005534.GA13574@esmail.cup.hp.com> <20050504070910.GC9674@mellanox.co.il> <20050506190947.GA23958@esmail.cup.hp.com> <20050507053706.GN23958@esmail.cup.hp.com> <20050508124916.GM9941@mellanox.co.il> Message-ID: <20050509155738.GA1319@esmail.cup.hp.com> On Sun, May 08, 2005 at 03:49:16PM +0300, Michael S. Tsirkin wrote: > Besides the new flags, I noticed the following changes: > - asm/timex.h usage to get cycles. > Unfortunately not portable to all platforms: > on x86_64 asm/timex.h includes linux/config so its not legal > for userspace to include. Please just implement get_cycles instead. That's a bug in the x86_64 header file. x86_64 doesn't need config.h. (as noted below, ppc doesn't either) But i386 does need config.h since TSC wasn't implemented until pentium boxes came around. Debian happens to do something that works: "linux/config.h" and "linux/autoconf.h" are installed in /usr/include/ and based on some standard kernel they build. So it "works for me" and feels like the right answer. IMHO it's just obviously stupid to replicate get_cycles in a private file when it's required (and available) in existing the arch specific header files. > - get_cpu_mhz() instead of get_cpu_khz() - is that important? > Actually I planned to change get_cpu_khz() to cpu_khz to match > linux in-kernel interface. Does this make sense to you? No. Only i386 defines cpu_khz. AFAICT, no other arch has ever defined it. (I'm assuming x86_64 is copying i386 so it's compatible.) I was just reflecting what the code actually was doing: reading Mhz from /proc/cpuinfo. That makes alot more sense to me. > > I'm consistently getting 4.74 usec median latency with 2.6.11 kernel + > > svn r2229 on HP ZX1 platform (PCI-X) + 1.5Ghz/6M Madison processors. > > Thats a bit higher latency than what I see with Intel, but at least > in the right ballbark. Ok. Not too surprising since the ZX1 chipset has two bridges between PCI-X device and memory. What values did you see? (which chipset/kernel/compiler too please) ... > > > /* > > > * pp_get_local_lid() uses a pretty bogus method for finding the LID > > > * of a local port. Please don't copy this into your app (or if you > > > * do, please rip it out soon). > > > */ > > ibv_pingpong used to have the same comment too. > I'll go back and look at ibv_pingpong. ok - thanks. > > This is still outstanding. But I'd like to first see perftest-02 > > land in a sane place in openib.org Subversion tree. > > I could just copy stuff to say userspace/perftests. > Is that OK with everyone? Sure. > > I can then submit patches against some stuff: > > o update the README with notes on how to use/interpret the data > > o stop replicating code and make subroutines > > I considered this but I'm afraid that adding another layer on top of > libibverbs would deduct from readability, and cause people > to copy it wholesale, something which I would like to avoid. I think you are reading more into this than I intended. I was thinking trivial things like the loop that dumps the values of delta[]. BTW, copying code wholesale is the whole point behind "Open Source". One measure of success is if people in fact do copy the code. > > o split up main() into bite-sized chunks so people > > know which part is "initialization" and "run time". > > I guess adding more comments like /* Initialization */ is the way to go. That might help, but it's not addressing the basic problem. People retrofitting an existing application will need at least three different peices of code: subsystem initialization, connection setup/tear down, and runtime communication. Just making those three code blocks subroutines is NOT creating a new layer. > > > o stop pretending there is no global state and get major > > variables off the stack and into .bss or respective > > subroutines. > > Why is that good? Passing flags in global parameters would > make the code less readable. I agree. But pulling out global state (relative to the program) is a first step to figuring out what needs to be a parameter and which variables are just local to certain bits of code. That's just my methodology for cleaning up stuff like this and not the end point. IMHO, the current code isn't readable either. main() has 268 lines and 22 (in my version, more in the original) local variables. Of those 22 variables, some are really local and not part of the global state but we can't easily tell which. ... > > Another clue about this failure: I'm only seeing this if I specify "-s". > > 1,2,4,8,16,24,25,26,27,28 all worked. All the values > 29 failed. > > (I tried 29, 30, 31, 32, 33, 64, 65, 4096, 8192). ... > I think thats because when I wrote the test qp attribute max_inline didnt > exist. I'll update the code. Ah good - I'm glad it's obvious to you since it sure wasnt' to me. :^) > :) > I'll merge the -U code, I see how it can be useful. thanks! ... > > That first sample on the server represents a tuning opportunity. > > A ~10x difference between startup and runtime is significant > > for short lived connections/regions. > > On the other hand, it could be negligible compared to connection > setup time. hrm...true. We should measure and report that too. A script could iterate the invocation with -n 1 to collect a reasonable sample. > OK, so I see 3 changes there: > > 1. support for -U and -c > I'll merge that thanks! > 2. get_clock in assembly replaced with asm/timex.h: > Unfortunately asm/timex.h was never intended for userspace, Exact copies from linux/include/asm* are not intended for userspace. But arch specific kernel headers are required to interact with the kernel on any given platform (e.g. syscalls). They just need to be "adjusted" by distro's so they are suitable for userspace consumption. > so this trick doesnt work on all platforms: specifically its > broken on ppc,i386 and x86_64, so ppc64 and ia64 where it does > work are kind of in the minority. x86_64 and ppc don't need to include config.h. I'll submit patches to fix this. i386 is the only one that has a reason to do so and I think one right answer is for the distro to include config.h/autoconf.h. > And given that even these may be broken on some distros, > I suggest we simply add an assembly implementation for now, > rather than add this dependency. I suggest define get_cycles() only for i386 and let everyone else use timex.h. > Grant, you mentioned some other fixes - what are they? > Maybe I'll notice them when I do the actual merge. Sorry - I made alot of changes to certain chunks of code and don't recall details. Main ones were off-by-one errors and the failure to initialize the first element of tstamp[]. thanks, grant From halr at voltaire.com Mon May 9 08:58:31 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 09 May 2005 11:58:31 -0400 Subject: [kdapl] kdapltest -M flag (was RE: [openib-general] [PATCH] kDAPL: remove DAT_UINT32 and DAT_UINT64) In-Reply-To: References: <1115411060.4476.346.camel@localhost.localdomain> <1115413920.4476.1.camel@localhost.localdomain> <1115649383.4490.1.camel@localhost.localdomain> Message-ID: <1115654311.4490.61.camel@localhost.localdomain> On Mon, 2005-05-09 at 11:47, James Lentini wrote: > What is the exact command line you are using? > > This works for me > > ./kdapltest -T T -D mthca0a -s 192.168.0.6 -M PHY client SR 256 server SR 256 This was from before the fix was made for the default memory type but I don't think that would explain why this works now. Anyhow, it is working now (for me too). -- Hal From roland at topspin.com Mon May 9 09:44:27 2005 From: roland at topspin.com (Roland Dreier) Date: Mon, 09 May 2005 09:44:27 -0700 Subject: [openib-general] [PATCH 1/2] mthca: max_inline_data support In-Reply-To: <20050509141456.GH9941@mellanox.co.il> (Michael S. Tsirkin's message of "Mon, 9 May 2005 17:14:56 +0300") References: <20050509141456.GH9941@mellanox.co.il> Message-ID: <52fyww1h90.fsf@topspin.com> Thanks... does it make sense to set the max_inline_data value for kernel QPs but then not support posting inline data? Also should we add some error checking against the HCA's maximum WQE size? - R. From roland at topspin.com Mon May 9 09:38:49 2005 From: roland at topspin.com (Roland Dreier) Date: Mon, 09 May 2005 09:38:49 -0700 Subject: [openib-general] [PATCH] libibverbs/examples: fix parameter parsing In-Reply-To: <20050508142930.GR9941@mellanox.co.il> (Michael S. Tsirkin's message of "Sun, 8 May 2005 17:29:30 +0300") References: <20050508142930.GR9941@mellanox.co.il> Message-ID: <52k6m81hie.fsf@topspin.com> Thanks, applied. From mst at mellanox.co.il Mon May 9 09:50:22 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Mon, 9 May 2005 19:50:22 +0300 Subject: [openib-general] Re: user-mode verbs on Itanium In-Reply-To: <20050509155738.GA1319@esmail.cup.hp.com> References: <52pswdk5au.fsf@topspin.com> <20050429162127.GA24871@esmail.cup.hp.com> <52ll71k09w.fsf@topspin.com> <20050429183846.GH24871@esmail.cup.hp.com> <20050504005534.GA13574@esmail.cup.hp.com> <20050504070910.GC9674@mellanox.co.il> <20050506190947.GA23958@esmail.cup.hp.com> <20050507053706.GN23958@esmail.cup.hp.com> <20050508124916.GM9941@mellanox.co.il> <20050509155738.GA1319@esmail.cup.hp.com> Message-ID: <20050509165022.GJ9941@mellanox.co.il> Hello, Grant! Thanks for comments on the perftest. I think fixed all bug you found. I just did >svn mv https://openib.org/svn/trunk/contrib/mellanox/perftest https://openib.org/svn/gen2/trunk/src/userspace Committed revision 2285. so we'll be able to exchange patches in the regular way. Quoting r. Grant Grundler : > Subject: Re: user-mode verbs on Itanium > > > I can then submit patches against some stuff: > > > o update the README with notes on how to use/interpret the data > > > o stop replicating code and make subroutines > > > > I considered this but I'm afraid that adding another layer on top of > > libibverbs would deduct from readability, and cause people > > to copy it wholesale, something which I would like to avoid. > > I think you are reading more into this than I intended. > I was thinking trivial things like the loop that dumps the values of delta[]. > > BTW, copying code wholesale is the whole point behind "Open Source". > One measure of success is if people in fact do copy the code. Right but why dont we put it in the core then? > > > o split up main() into bite-sized chunks so people > > > know which part is "initialization" and "run time". > > > > I guess adding more comments like /* Initialization */ is the way to go. > > That might help, but it's not addressing the basic problem. > People retrofitting an existing application will need at least > three different peices of code: subsystem initialization, connection > setup/tear down, and runtime communication. Just making those three > code blocks subroutines is NOT creating a new layer. But once these subroutines are moved into a library, it sure begins to look like one. > > > o stop pretending there is no global state and get major > > > variables off the stack and into .bss or respective > > > subroutines. > > > > Why is that good? Passing flags in global parameters would > > make the code less readable. > > I agree. But pulling out global state (relative to the program) > is a first step to figuring out what needs to be a parameter > and which variables are just local to certain bits of code. > That's just my methodology for cleaning up stuff like this > and not the end point. I see how separating things to subroutines would help here, but making things global will just hide who uses what, IMO. > IMHO, the current code isn't readable either. > main() has 268 lines and 22 (in my version, more in the original) > local variables. Of those 22 variables, some are really local > and not part of the global state but we can't easily tell which. Its a work in progress :) I'm not sure what do you call "global state". I am trying to think of an application that has many connections, global state would be something common to all of them? The original idea was to keep the connection state stuff inside the context structure. Maybe I departed from that. > ... > > > Another clue about this failure: I'm only seeing this if I specify "-s". > > > 1,2,4,8,16,24,25,26,27,28 all worked. All the values > 29 failed. > > > (I tried 29, 30, 31, 32, 33, 64, 65, 4096, 8192). > ... > > I think thats because when I wrote the test qp attribute max_inline didnt > > exist. I'll update the code. > > Ah good - I'm glad it's obvious to you since it sure wasnt' to me. :^) > > > :) > > I'll merge the -U code, I see how it can be useful. > > thanks! You'll need my two patches I sent previously to make it work though. As a work around, bump max_send_sge a bit. > ... > > > That first sample on the server represents a tuning opportunity. > > > A ~10x difference between startup and runtime is significant > > > for short lived connections/regions. > > > > On the other hand, it could be negligible compared to connection > > setup time. > > hrm...true. We should measure and report that too. > A script could iterate the invocation with -n 1 to collect > a reasonable sample. Problem is, connection setup involves the CM (or the socket kludge), so its not trivial to define what the overhead is. > > OK, so I see 3 changes there: > > > > 1. support for -U and -c > > I'll merge that > > thanks! Ok, I checked that in: rev 2283. I also renamed --histogram to --report-all. > > 2. get_clock in assembly replaced with asm/timex.h: > > Unfortunately asm/timex.h was never intended for userspace, > > Exact copies from linux/include/asm* are not intended for userspace. > But arch specific kernel headers are required to interact with the > kernel on any given platform (e.g. syscalls). They just need to be > "adjusted" by distro's so they are suitable for userspace consumption. IMO get_cycles is never needed to interact with the kernel. It could have been part of libc or something, but its not. > > so this trick doesnt work on all platforms: specifically its > > broken on ppc,i386 and x86_64, so ppc64 and ia64 where it does > > work are kind of in the minority. > > x86_64 and ppc don't need to include config.h. > I'll submit patches to fix this. > > i386 is the only one that has a reason to do so and I think one > right answer is for the distro to include config.h/autoconf.h. > > > And given that even these may be broken on some distros, > > I suggest we simply add an assembly implementation for now, > > rather than add this dependency. > > I suggest define get_cycles() only for i386 and let everyone else use timex.h. Sure. Still, I think I'll keep the one-line assembly till its fixed in most distros :) I added ia64 so everyone can make progress meanwhile. BTW, some architectures (notably sparc) define get_cycles to (0). Need to google for the right answer there. I also saw this errata for Itanium: 2. AR.ITC returns an incorrect value Implication: The 64-bit Interval Timer Counter (AR.ITC) register may return an incorrect value when the lower 32-bits are all F's. In this case, the value returned in the upper 32-bits is incremented by 1. For example, when the value returned is 0x1FFFFFFFF, the actual value should be 0x0FFFFFFFF. Implication: Software that utilizes the AR.ITC register will receive an incorrect value in this case. Workaround: The workaround is for software to re-read the AR.ITC register when it detects all F's in the lower 32-bits. which linux seems to ignore (possibly because high value of get_cycles does not matter there). Does this bug still affect Itanium systems in use out there? What about Itanium 2? -- MST - Michael S. Tsirkin From jlentini at netapp.com Mon May 9 10:01:08 2005 From: jlentini at netapp.com (James Lentini) Date: Mon, 9 May 2005 13:01:08 -0400 (EDT) Subject: [openib-general] Re: [PATCH][KDAPL] clean up code : evd flow In-Reply-To: <20050505143753.GA1517@mellanox.co.il> References: <20050505143753.GA1517@mellanox.co.il> Message-ID: Itamar, Could this patch have caused the error below? drivers/infiniband/ulp/dat-provider/dapl_evd_connection_callb.c:100: spin_lock(drivers/infiniband/ulp/dat-provider/dapl_ep_util.c:d740a428) already locked by drivers/infiniband/ulp/dat-provider/dapl_evd_connection_callb.c/100 drivers/infiniband/ulp/dat-provider/dapl_evd_connection_callb.c:152: spin_unlock(drivers/infiniband/ulp/dat-provider/dapl_ep_util.c:d740a428) not locked Unable to handle kernel NULL pointer dereference at virtual address 00000048 printing eip: e0a4715b *pde = 00000000 Oops: 0000 [#1] Modules linked in: kdapltest ib_dat_provider dat ib_cm ib_at ib_ipoib ib_sa md5 ipv6 parport_pc lp parport autofs4 nfs lockd sunrpc i2c_piix4 i2c_core ib_mthca ib_mad ib_core e100 mii floppy sg aic7xxx sd_mod scsi_mod CPU: 0 EIP: 0060:[] Not tainted VLI EFLAGS: 00010096 (2.6.11-openib) EIP is at ib_destroy_cm_id+0xb/0x790 [ib_cm] eax: 00000000 ebx: 00000000 ecx: d92f84e4 edx: d75234a0 esi: 00000296 edi: d92f84e0 ebp: e1ac6f4f esp: dfd9becc ds: 007b es: 007b ss: 0068 Process dapl_cm_wq/0 (pid: 5625, threadinfo=dfd9b000 task=d74268d0) Stack: 00000000 56360d6c 0004d7ea d7426350 d74268d0 d7426350 d74268d0 de14f9c0 dfd9bf40 c035fee4 d75234e0 00000001 00000003 d75234c8 00000001 00000003 dfd9bf40 00000096 00000000 00000292 000000c2 dbd75330 000000c1 d92f84e0 Call Trace: [] schedule+0x2e4/0x620 [] dapl_ib_destroy_cm_id_work+0x21/0x2e [ib_dat_provider] [] worker_thread+0x211/0x440 [] default_wake_function+0x0/0x10 [] default_wake_function+0x0/0x10 [] worker_thread+0x0/0x440 [] kthread+0x96/0xe0 [] kthread+0x0/0xe0 [] kernel_thread_helper+0x5/0x18 Code: 85 c0 75 02 5b c3 e8 f5 fc ff ff 8b 83 88 00 00 00 e8 ba 94 70 df 31 c0 89 83 88 00 00 00 5b c3 57 56 53 83 ec 5c 89 c3 9c 5e fa <81> 7b 48 3c 4b 24 1d 74 25 c7 04 24 d4 f3 a4 e0 8d 43 48 b9 6c On Thu, 5 May 2005, Itamar wrote: > Clean up kdapl : evd flow > > Signed-off-by: Itamar Rabenstein > > Index: dapl_evd_dto_callb.c > =================================================================== > --- dapl_evd_dto_callb.c (revision 2257) > +++ dapl_evd_dto_callb.c (working copy) > @@ -98,46 +98,26 @@ void dapl_evd_dto_callback(ib_cq_handle_ > * EVD is enabled. > */ > > - if (state == DAPL_EVD_STATE_WAITED) { > - /* > - * If we could, it would be best to avoid this wakeup > - * (and the context switch) unless the number of events/CQs > - * waiting for the waiter was its threshold. We don't > - * currently have the ability to determine that without > - * dequeueing the events, and we can't do that for > - * synchronization reasons (racing with the waiter waking > - * up and dequeuing, sparked by other callbacks). > - */ > + if (state == DAPL_EVD_STATE_OPEN && (evd_ptr->cno_ptr != NULL)) { > + /* > + * Re-enable callback, *then* trigger. > + * This guarantees we won't miss any events. > + */ > + dat_status = dapl_ib_completion_notify(evd_ptr, > + IB_NOTIFY_ON_NEXT_COMP); > > - /* > - * We don't need to worry about taking the lock for the > - * wakeup because wakeups are sticky. > - */ > - dapl_os_wait_object_wakeup(&evd_ptr->wait_object); > - } else if (state == DAPL_EVD_STATE_OPEN) { > - DAPL_CNO *cno = evd_ptr->cno_ptr; > - if (evd_ptr->evd_enabled && (evd_ptr->cno_ptr != NULL)) { > - /* > - * Re-enable callback, *then* trigger. > - * This guarantees we won't miss any events. > - */ > - dat_status = dapl_ib_completion_notify(evd_ptr, > - IB_NOTIFY_ON_NEXT_COMP); > - > - if (DAT_SUCCESS != dat_status) { > - (void)dapl_evd_post_async_error_event(evd_ptr-> > - header. > - owner_ia-> > + if (DAT_SUCCESS != dat_status) { > + (void)dapl_evd_post_async_error_event(evd_ptr-> > + header. > + owner_ia-> > async_error_evd, > DAT_ASYNC_ERROR_PROVIDER_INTERNAL_ERROR, > (DAT_IA_HANDLE) > evd_ptr-> > header. > owner_ia); > - } > - > - dapl_cno_trigger(cno, evd_ptr); > - } > - } > + } > + dapl_cno_trigger(evd_ptr->cno_ptr, evd_ptr); > + } > dapl_dbg_log(DAPL_DBG_TYPE_RTN, "dapl_evd_dto_callback () returns\n"); > } > Index: dapl_ep_create.c > =================================================================== > --- dapl_ep_create.c (revision 2257) > +++ dapl_ep_create.c (working copy) > @@ -74,7 +74,6 @@ dapl_ep_create(DAT_IA_HANDLE ia_handle, > DAPL_IA *ia_ptr; > DAPL_EP *ep_ptr; > DAT_EP_ATTR ep_attr_limit; > - DAPL_EVD *evd_ptr; > DAT_RETURN dat_status; > > dapl_dbg_log(DAPL_DBG_TYPE_API, > @@ -194,39 +193,6 @@ dapl_ep_create(DAT_IA_HANDLE ia_handle, > } > } > > - /* > - * Verify the completion flags for the EVD and the EP > - */ > - /* > - * XXX FIXME > - * XXX Need to make assign the EVD to the right completion type > - * XXX depending on the EP attributes. Fail if the types don't > - * XXX match, they are mutually exclusive. > - */ > - evd_ptr = (DAPL_EVD *) recv_evd_handle; > - if (evd_ptr != NULL && evd_ptr->completion_type == DAPL_EVD_STATE_INIT) { > - if (ep_attr != NULL && > - ep_attr->recv_completion_flags == > - DAT_COMPLETION_DEFAULT_FLAG) { > - evd_ptr->completion_type = DAPL_EVD_STATE_THRESHOLD; > - } else { > - evd_ptr->completion_type = > - ep_attr->recv_completion_flags; > - } > - } > - > - evd_ptr = (DAPL_EVD *) request_evd_handle; > - if (evd_ptr != NULL && evd_ptr->completion_type == DAPL_EVD_STATE_INIT) { > - if (ep_attr != NULL && > - ep_attr->recv_completion_flags == > - DAT_COMPLETION_DEFAULT_FLAG) { > - evd_ptr->completion_type = DAPL_EVD_STATE_THRESHOLD; > - } else { > - evd_ptr->completion_type = > - ep_attr->recv_completion_flags; > - } > - } > - > /* Allocate EP */ > ep_ptr = dapl_ep_alloc(ia_ptr, ep_attr); > if (ep_ptr == NULL) { > Index: dapl_openib_dto.h > =================================================================== > --- dapl_openib_dto.h (revision 2257) > +++ dapl_openib_dto.h (working copy) > @@ -167,5 +167,6 @@ dapl_ib_post_send(DAPL_EP * ep_ptr, > #define DAPL_GET_CQE_OPTYPE(cqe_p) ((ib_work_completion_t *)cqe_p)->opcode > #define DAPL_GET_CQE_BYTESNUM(cqe_p) ((ib_work_completion_t *)cqe_p)->byte_len > #define DAPL_GET_CQE_STATUS(cqe_p) ((ib_work_completion_t *)cqe_p)->status > +#define DAPL_GET_CQE_VND_SYND(cqe_p) ((ib_work_completion_t *)cqe_ptr)->vendor_err > > #endif /* DAPL_OPENIB_DTO_H */ > Index: dapl_evd_resize.c > =================================================================== > --- dapl_evd_resize.c (revision 2257) > +++ dapl_evd_resize.c (working copy) > @@ -93,13 +93,6 @@ DAT_RETURN dapl_evd_resize(DAT_EVD_HANDL > > spin_lock(&evd_ptr->header.lock); > > - /* Don't try to resize if we are actively waiting */ > - if (evd_ptr->evd_state == DAPL_EVD_STATE_WAITED) { > - spin_unlock(&evd_ptr->header.lock); > - dat_status = DAT_ERROR(DAT_INVALID_STATE, 0); > - goto bail; > - } > - > pend_cnt = dapl_rbuf_count(&evd_ptr->pending_event_queue); > if (pend_cnt > evd_qlen) { > spin_unlock(&evd_ptr->header.lock); > Index: dapl_evd_kcreate.c > =================================================================== > --- dapl_evd_kcreate.c (revision 2257) > +++ dapl_evd_kcreate.c (working copy) > @@ -96,16 +96,13 @@ DAT_RETURN dapl_evd_kcreate(DAT_IA_HANDL > goto bail; > } > > - cno_ptr = dapl_cno_alloc (ia_ptr, upcall); > - > - if (!cno_ptr) > - { > - dat_status = DAT_INSUFFICIENT_RESOURCES; > - goto bail; > - } > - > - cno_ptr->cno_upcall_policy = upcall_policy; > + cno_ptr = dapl_cno_alloc (ia_ptr, upcall); > > + if (!cno_ptr) { > + dat_status = DAT_INSUFFICIENT_RESOURCES; > + goto bail; > + } > + cno_ptr->cno_upcall_policy = upcall_policy; > > dat_status = dapl_evd_internal_create(ia_ptr, > cno_ptr, > @@ -115,9 +112,6 @@ DAT_RETURN dapl_evd_kcreate(DAT_IA_HANDL > goto bail; > } > > - evd_ptr->evd_state = DAPL_EVD_STATE_OPEN; > - evd_ptr->evd_enabled = DAT_TRUE; > - > evdhandle = *evd_handle = (DAT_EVD_HANDLE) evd_ptr; > > bail: > Index: dapl_openib_util.h > =================================================================== > --- dapl_openib_util.h (revision 2257) > +++ dapl_openib_util.h (working copy) > @@ -89,22 +89,6 @@ typedef struct ib_hca_transport { > u16 lid; > } ib_hca_transport_t; > > -/* > - * ib_shm_transport_t structure. Define fields specific to this > - * provider implementation necessary to maintain shared memory state > - * > - * OpenIB does not have an API to directly register memory as shared > - * memory. Instead, we must register memory, then register that memory > - * as shared. This gives us a new mr_handle, as well as a new l_key and > - * r_key, so we need to save the original handle until it can be > - * disposed of when the user does an lmr_free. The l_key and r_key are > - * not available and will disappear when we deallocate the mr_handle, so > - * we don't save them. > - */ > -typedef struct ib_shm_transport { > - ib_mr_handle_t mr_handle; > -} ib_shm_transport_t; > - > #define IB_INVALID_HANDLE NULL > > #define IB_MAX_REQ_PDATA_SIZE 92 > Index: dapl_ep_create_with_srq.c > =================================================================== > --- dapl_ep_create_with_srq.c (revision 2257) > +++ dapl_ep_create_with_srq.c (working copy) > @@ -81,7 +81,6 @@ dapl_ep_create_with_srq(DAT_IA_HANDLE ia > DAPL_IA *ia_ptr; > DAPL_EP *ep_ptr; > DAT_EP_ATTR ep_attr_limit; > - DAPL_EVD *evd_ptr; > DAT_RETURN dat_status; > > dat_status = DAT_SUCCESS; > @@ -209,39 +208,6 @@ dapl_ep_create_with_srq(DAT_IA_HANDLE ia > } > } > > - /* > - * Verify the completion flags for the EVD and the EP > - */ > - /* > - * XXX FIXME > - * XXX Need to make assign the EVD to the right completion type > - * XXX depending on the EP attributes. Fail if the types don't > - * XXX match, they are mutually exclusive. > - */ > - evd_ptr = (DAPL_EVD *) recv_evd_handle; > - if (evd_ptr != NULL && evd_ptr->completion_type == DAPL_EVD_STATE_INIT) { > - if (ep_attr != NULL && > - ep_attr->recv_completion_flags == > - DAT_COMPLETION_DEFAULT_FLAG) { > - evd_ptr->completion_type = DAPL_EVD_STATE_THRESHOLD; > - } else { > - evd_ptr->completion_type = > - ep_attr->recv_completion_flags; > - } > - } > - > - evd_ptr = (DAPL_EVD *) request_evd_handle; > - if (evd_ptr != NULL && evd_ptr->completion_type == DAPL_EVD_STATE_INIT) { > - if (ep_attr != NULL && > - ep_attr->recv_completion_flags == > - DAT_COMPLETION_DEFAULT_FLAG) { > - evd_ptr->completion_type = DAPL_EVD_STATE_THRESHOLD; > - } else { > - evd_ptr->completion_type = > - ep_attr->recv_completion_flags; > - } > - } > - > dat_status = DAT_NOT_IMPLEMENTED; > > /* > Index: dapl_evd_util.c > =================================================================== > --- dapl_evd_util.c (revision 2257) > +++ dapl_evd_util.c (working copy) > @@ -77,9 +77,6 @@ dapl_evd_internal_create(DAPL_IA * ia_pt > evd_ptr->evd_producer_locking_needed = > ((evd_flags & ~(DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG)) != 0); > > - /* Before we setup any callbacks, transition state to OPEN. */ > - evd_ptr->evd_state = DAPL_EVD_STATE_OPEN; > - > if (evd_flags & DAT_EVD_ASYNC_FLAG) { > /* > * There is no cq associate with async evd. Set it to invalid > @@ -161,19 +158,13 @@ DAPL_EVD *dapl_evd_alloc(DAPL_IA * ia_pt > /* > * Initialize the body > */ > - evd_ptr->evd_state = DAPL_EVD_STATE_INITIAL; > + evd_ptr->evd_state = DAPL_EVD_STATE_OPEN; > evd_ptr->evd_flags = evd_flags; > - evd_ptr->evd_enabled = DAT_TRUE; > - evd_ptr->evd_waitable = DAT_TRUE; > evd_ptr->evd_producer_locking_needed = 1; /* Conservative value. */ > evd_ptr->ib_cq_handle = IB_INVALID_HANDLE; > atomic_set(&evd_ptr->evd_ref_count, 0); > evd_ptr->catastrophic_overflow = DAT_FALSE; > evd_ptr->qlen = qlen; > - evd_ptr->completion_type = DAPL_EVD_STATE_THRESHOLD; /* FIXME: should be DAPL_EVD_STATE_INIT */ > - dapl_os_wait_object_init(&evd_ptr->wait_object); > - > - evd_ptr->cno_active_count = 0; > evd_ptr->cno_ptr = cno_ptr; > > bail: > @@ -219,10 +210,6 @@ DAT_RETURN dapl_evd_event_alloc(DAPL_EVD > event_ptr++; > } > > - evd_ptr->cq_notified = DAT_FALSE; > - evd_ptr->cq_notified_when = 0; > - evd_ptr->threshold = 0; > - > bail: > return dat_status; > } > @@ -331,7 +318,6 @@ DAT_RETURN dapl_evd_dealloc(DAPL_EVD * e > kfree(evd_ptr->events); > } > > - dapl_os_wait_object_destroy(&evd_ptr->wait_object); > kfree(evd_ptr); > > bail: > @@ -445,73 +431,20 @@ static void dapl_evd_post_event(DAPL_EVD > (void *)event_ptr); > dapl_os_assert(dat_status == DAT_SUCCESS); > > - dapl_os_assert(evd_ptr->evd_state == DAPL_EVD_STATE_WAITED > - || evd_ptr->evd_state == DAPL_EVD_STATE_OPEN); > + dapl_os_assert(evd_ptr->evd_state == DAPL_EVD_STATE_OPEN); > > if (evd_ptr->evd_state == DAPL_EVD_STATE_OPEN) { > /* No waiter. Arrange to trigger a CNO if it exists. */ > - > - if (evd_ptr->evd_enabled) { > - cno_to_trigger = evd_ptr->cno_ptr; > - } > - > - if (evd_ptr->evd_producer_locking_needed) { > - spin_unlock(&evd_ptr->header.lock); > - } > - } else { > - /* > - * We're in DAPL_EVD_STATE_WAITED. Take the lock if > - * we don't have it, recheck, and signal. > - */ > - if (!evd_ptr->evd_producer_locking_needed) { > - spin_lock(&evd_ptr->header.lock); > - } > - > - if (evd_ptr->evd_state == DAPL_EVD_STATE_WAITED > - && (dapl_rbuf_count(&evd_ptr->pending_event_queue) > - >= evd_ptr->threshold)) { > - spin_unlock(&evd_ptr->header.lock); > - dapl_os_wait_object_wakeup(&evd_ptr->wait_object); > - } else { > - spin_unlock(&evd_ptr->header.lock); > - } > + cno_to_trigger = evd_ptr->cno_ptr; > + if (evd_ptr->evd_producer_locking_needed) { > + spin_unlock(&evd_ptr->header.lock); > + } > } > - > if (cno_to_trigger != NULL) { > dapl_cno_trigger (cno_to_trigger, evd_ptr); > } > } > > -/* > - * Post the to the evd. Do not do any wakeup processing. > - * This function should only be called if it is known that there are > - * no waiters that it is appropriate to wakeup on this EVD. An example > - * of such a situation is during internal dat_evd_wait() processing. > - * > - * If producer side locking is required, the EVD lock must be held upon > - * entry to this function. > - */ > - > -static void > -dapl_evd_post_event_nosignal(DAPL_EVD * evd_ptr, const DAT_EVENT * event_ptr) > -{ > - DAT_RETURN dat_status; > - > - dapl_dbg_log(DAPL_DBG_TYPE_EVD, > - "dapl_evd_post_event_nonsignal: Called with event # %x\n", > - event_ptr->event_number); > - > - dat_status = dapl_rbuf_add(&evd_ptr->pending_event_queue, > - (void *)event_ptr); > - dapl_os_assert(dat_status == DAT_SUCCESS); > - > - dapl_os_assert(evd_ptr->evd_state == DAPL_EVD_STATE_WAITED > - || evd_ptr->evd_state == DAPL_EVD_STATE_OPEN); > - > - if (evd_ptr->evd_producer_locking_needed) { > - spin_unlock(&evd_ptr->header.lock); > - } > -} > > /* > * format an overflow event for posting > @@ -689,49 +622,6 @@ dapl_evd_post_software_event(DAPL_EVD * > } > > /* > - * dapl_evd_post_generic_event > - * > - * Post a generic event type. Not used by all providers > - * > - * Input: > - * evd_ptr > - * event_number > - * data > - * > - * Output: > - * none > - * > - * Returns: > - * DAT_SUCCESS > - * > - */ > -DAT_RETURN > -dapl_evd_post_generic_event(DAPL_EVD * evd_ptr, > - DAT_EVENT_NUMBER event_number, > - DAT_EVENT_DATA * data) > -{ > - DAT_EVENT *event_ptr; > - > - event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number); > - /* > - * Note event lock may be held on successful return > - * to be released by dapl_evd_post_event(), if provider side locking > - * is needed. > - */ > - > - if (event_ptr == NULL) { > - return DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, > - DAT_RESOURCE_MEMORY); > - } > - > - event_ptr->event_data = *data; > - > - dapl_evd_post_event(evd_ptr, event_ptr); > - > - return DAT_SUCCESS; > -} > - > -/* > * dapl_evd_cqe_to_event > * > * Convert a CQE into an event structure. > @@ -866,128 +756,13 @@ dapl_evd_cqe_to_event(DAPL_EVD * evd_ptr > } > } /* end switch */ > > - /* > - * Most error DTO ops result in disconnecting the EP. See > - * IBTA Vol 1.1, Chapter 10,Table 68, for expected effect on > - * state. > - */ > - if ((dto_status != DAT_DTO_SUCCESS) && > - (dto_status != DAT_DTO_ERR_FLUSHED)) { > -#ifdef CONN_EV_BROKEN_SUPPORTED > - DAPL_EVD *evd_ptr; > - > - /* > - * If we are connected, generate disconnect and generate an > - * event. We may be racing with other disconnect ops, so we > - * need to check. We may also be racing CM connection events, > - * requiring us to check for connection pending states too. > - */ > - spin_lock(&ep_ptr->header.lock); > - if (ep_ptr->param.ep_state == DAT_EP_STATE_CONNECTED || > - ep_ptr->param.ep_state == > - DAT_EP_STATE_ACTIVE_CONNECTION_PENDING > - || ep_ptr->param.ep_state == > - DAT_EP_STATE_PASSIVE_CONNECTION_PENDING > - || ep_ptr->param.ep_state == > - DAT_EP_STATE_COMPLETION_PENDING) { > - ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; > - spin_unlock(&ep_ptr->header.lock); > - > - /* Let the other side know we have disconnected */ > - // cant call to dapl_ib_disconnect besause it is in interrupt context > - // and this call is blocking need to fix ??? > - // (void) dapl_ib_disconnect (ep_ptr, DAT_CLOSE_ABRUPT_FLAG); > - printk("FIXME DAPL BUG %s %s %d \n", __FILE__, > - __FUNCTION__, __LINE__); > - /* ... and clean up the local side */ > - evd_ptr = (DAPL_EVD *) ep_ptr->param.connect_evd_handle; > - if (evd_ptr != NULL) { > - dapl_evd_post_connection_event(evd_ptr, > - DAT_CONNECTION_EVENT_BROKEN, > - (DAT_HANDLE) > - ep_ptr, 0, 0); > - } > - } else { > - spin_unlock(&ep_ptr->header.lock); > - } > - > - dapl_dbg_log(DAPL_DBG_TYPE_DTO_COMP_ERR, > - " DTO completion ERROR: %d: op %#x (ep disconnected)\n", > - DAPL_GET_CQE_STATUS(cqe_ptr), > - DAPL_GET_CQE_OPTYPE(cqe_ptr)); > -#else > - dapl_dbg_log(DAPL_DBG_TYPE_DTO_COMP_ERR, > - " DTO completion ERROR: %d: op %#x\n", > - DAPL_GET_CQE_STATUS(cqe_ptr), > - DAPL_GET_CQE_OPTYPE(cqe_ptr)); > -#endif > - } > -} > - > -/* > - * dapl_evd_copy_cq > - * > - * Copy all entries on a CQ associated with the EVD onto that EVD > - * Up to caller to handle races, if any. Note that no EVD waiters will > - * be awoken by this copy. > - * > - * Input: > - * evd_ptr > - * > - * Output: > - * None > - * > - * Returns: > - * none > - * > - */ > -void dapl_evd_copy_cq(DAPL_EVD * evd_ptr) > -{ > - ib_work_completion_t cur_cqe; > - DAT_RETURN dat_status; > - DAT_EVENT *event; > - > - if (evd_ptr->ib_cq_handle == IB_INVALID_HANDLE) { > - /* Nothing to do if no CQ. */ > - return; > - } > - > - while (1) { > - dat_status = > - dapl_ib_completion_poll(evd_ptr->header.owner_ia->hca_ptr, > - evd_ptr, &cur_cqe); > - > - if (dat_status != DAT_SUCCESS) { > - break; > - } > - > - /* For debugging. */ > - dapl_evd_eh_print_cqe(&cur_cqe); > - > - /* > - * Can use DAT_DTO_COMPLETION_EVENT because dapl_evd_cqe_to_event > - * will overwrite. > - */ > - > - event = > - dapl_evd_get_and_init_event(evd_ptr, > - DAT_DTO_COMPLETION_EVENT); > - if (event == NULL) { > - /* We've already attempted the overflow post; return. */ > - return; > - } > - > - dapl_evd_cqe_to_event(evd_ptr, &cur_cqe, event); > - > - dapl_evd_post_event_nosignal(evd_ptr, event); > - } > - > - if (DAT_GET_TYPE(dat_status) != DAT_QUEUE_EMPTY) { > - dapl_dbg_log(DAPL_DBG_TYPE_EVD, > - "dapl_evd_copy_cq: dapl_ib_completion_poll returned 0x%x\n", > - dat_status); > - dapl_os_assert(!"Bad return from dapl_ib_completion_poll"); > - } > + if (dto_status != DAT_DTO_SUCCESS) { > + dapl_dbg_log(DAPL_DBG_TYPE_DTO_COMP_ERR, > + " DTO completion ERROR:status %d: op %#x vendore synd: 0x%x\n", > + DAPL_GET_CQE_STATUS (cqe_ptr), > + DAPL_GET_CQE_OPTYPE (cqe_ptr), > + DAPL_GET_CQE_VND_SYND(cqe_ptr)); > + } > } > > /* > Index: dapl_evd_util.h > =================================================================== > --- dapl_evd_util.h (revision 2257) > +++ dapl_evd_util.h (working copy) > @@ -87,10 +87,6 @@ DAT_RETURN > dapl_evd_post_software_event(DAPL_EVD * evd_ptr, > DAT_EVENT_NUMBER event_number, DAT_PVOID pointer); > > -DAT_RETURN > -dapl_evd_post_generic_event(DAPL_EVD * evd_ptr, > - DAT_EVENT_NUMBER event_number, > - DAT_EVENT_DATA * data); > > /************************************* > * dapl internal callbacks functions * > @@ -115,8 +111,6 @@ extern void dapl_evd_cq_async_error_call > extern void dapl_evd_qp_async_error_callback(ib_error_record_t * cause_ptr, > void *context); > > -extern void dapl_evd_copy_cq(DAPL_EVD * evd_ptr); > - > extern DAT_RETURN dapl_evd_cq_poll_to_event(DAPL_EVD * evd_ptr, > DAT_EVENT * event); > > Index: dapl.h > =================================================================== > --- dapl.h (revision 2257) > +++ dapl.h (working copy) > @@ -64,10 +64,7 @@ typedef enum dapl_magic { > } DAPL_MAGIC; > > typedef enum dapl_evd_state { > - DAPL_EVD_STATE_TERMINAL, > - DAPL_EVD_STATE_INITIAL, > DAPL_EVD_STATE_OPEN, > - DAPL_EVD_STATE_WAITED, > DAPL_EVD_STATE_DEAD = 0xDEAD > } DAPL_EVD_STATE; > > @@ -270,8 +267,6 @@ struct dapl_evd > > DAPL_EVD_STATE evd_state; > DAT_EVD_FLAGS evd_flags; > - DAT_BOOLEAN evd_enabled; /* For attached CNO. */ > - DAT_BOOLEAN evd_waitable; /* EVD state. */ > > /* Derived from evd_flags; see dapls_evd_internal_create. */ > DAT_BOOLEAN evd_producer_locking_needed; > @@ -279,9 +274,6 @@ struct dapl_evd > /* Every EVD has a CQ unless it is a SOFTWARE_EVENT only EVD */ > ib_cq_handle_t ib_cq_handle; > > - /* Mellanox Specific completion handle for registration/de-registration */ > - ib_comp_handle_t ib_comp_handle; > - > /* An Event Dispatcher cannot be freed while > * it is referenced elsewhere. > */ > @@ -295,20 +287,7 @@ struct dapl_evd > DAT_EVENT *events; > DAPL_RING_BUFFER free_event_queue; > DAPL_RING_BUFFER pending_event_queue; > - > - /* CQ Completions are not placed into 'deferred_events' > - ** rather they are simply left on the Completion Queue > - ** and the fact that there was a notification is flagged. > - */ > - DAT_BOOLEAN cq_notified; > - DAPL_OS_TICKS cq_notified_when; > - > - DAT_COUNT cno_active_count; > DAPL_CNO *cno_ptr; > - > - DAPL_OS_WAIT_OBJECT wait_object; > - DAT_COUNT threshold; > - DAPL_EVD_COMPLETION completion_type; > }; > > /* uDAPL timer entry, used to queue timeouts */ > -- > Itamar > From iod00d at hp.com Mon May 9 10:06:54 2005 From: iod00d at hp.com (Grant Grundler) Date: Mon, 9 May 2005 10:06:54 -0700 Subject: [openib-general] [PATCH] libibverbs/examples: fix parameter parsing In-Reply-To: <20050508142930.GR9941@mellanox.co.il> References: <20050508142930.GR9941@mellanox.co.il> Message-ID: <20050509170654.GB1683@esmail.cup.hp.com> On Sun, May 08, 2005 at 05:29:30PM +0300, Michael S. Tsirkin wrote: > Fix for two obvious typos in pingpong and us-pingpong parameter parsing > (found by Grant in rdma_lat code). Actually found by Bernard Fischer in rdma_lat.c. Thanks for checking ibv_pingpong too. grant > > Signed-off-by: Michael S. Tsirkin > > Index: examples/pingpong.c > =================================================================== > --- examples/pingpong.c (revision 2278) > +++ examples/pingpong.c (working copy) > @@ -474,7 +474,7 @@ int main(int argc, char *argv[]) > > case 'i': > ib_port = strtol(optarg, NULL, 0); > - if (port < 0) { > + if (ib_port < 0) { > usage(argv[0]); > return 1; > } > Index: examples/ud-pingpong.c > =================================================================== > --- examples/ud-pingpong.c (revision 2278) > +++ examples/ud-pingpong.c (working copy) > @@ -473,7 +473,7 @@ int main(int argc, char *argv[]) > > case 'i': > ib_port = strtol(optarg, NULL, 0); > - if (port < 0) { > + if (ib_port < 0) { > usage(argv[0]); > return 1; > } > > -- > MST - Michael S. Tsirkin > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From halr at voltaire.com Mon May 9 10:22:17 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 09 May 2005 13:22:17 -0400 Subject: [openib-general] A Couple of User CM Questions Message-ID: <1115659336.4490.91.camel@localhost.localdomain> Hi Libor, I have a couple of questions pertaining to the user CM. 1. It appears that device (and port) are not supported either incoming or outgoing. So if that is correct, I presume only the first device (first port) is the only one which can be expected to work. Correct ? If so, that should be added to README as a current limitation. 2. Any idea on how device would be supported ? Would this just be with an IB device number or a string like "mthca0" ? Also note (nit) that struct ib_cm_sidr_req_event_param in userspace/libibcm/include/infiniband/cm.h still contains ib_device (and port). Thanks. -- Hal From mst at mellanox.co.il Mon May 9 10:31:57 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Mon, 9 May 2005 20:31:57 +0300 Subject: [openib-general] [PATCH 1/2] mthca: max_inline_data support In-Reply-To: <52fyww1h90.fsf@topspin.com> References: <20050509141456.GH9941@mellanox.co.il> <52fyww1h90.fsf@topspin.com> Message-ID: <20050509173157.GA24903@mellanox.co.il> Quoting r. Roland Dreier : > Subject: Re: [openib-general] [PATCH 1/2] mthca: max_inline_data support > > Thanks... does it make sense to set the max_inline_data value for > kernel QPs but then not support posting inline data? That will come next (I plan to add that so ULPs can get lower latency). But it seems kernel support is needed even for libibverbs since its the kernel that calculates wqe stride to put in cmd interface. No? > Also should we add some error checking against the HCA's maximum WQE > size? > > - R. And also max sge for memfree. May be a good idea, but seems to apply even without inline data. If you add this I'd prefer that you apply this patch first to avoid merging conflicts. -- MST - Michael S. Tsirkin From jlentini at netapp.com Mon May 9 10:58:51 2005 From: jlentini at netapp.com (James Lentini) Date: Mon, 9 May 2005 13:58:51 -0400 (EDT) Subject: [openib-general] Re: [PATCH][KDAPL] clean up code : evd flow In-Reply-To: References: <20050505143753.GA1517@mellanox.co.il> Message-ID: Since I only saw this error once, I decided to commit the patch (available in revision 2286). james On Mon, 9 May 2005, James Lentini wrote: > > Itamar, > > Could this patch have caused the error below? > > drivers/infiniband/ulp/dat-provider/dapl_evd_connection_callb.c:100: > spin_lock(drivers/infiniband/ulp/dat-provider/dapl_ep_util.c:d740a428) > already locked by > drivers/infiniband/ulp/dat-provider/dapl_evd_connection_callb.c/100 > drivers/infiniband/ulp/dat-provider/dapl_evd_connection_callb.c:152: > spin_unlock(drivers/infiniband/ulp/dat-provider/dapl_ep_util.c:d740a428) not > locked > Unable to handle kernel NULL pointer dereference at virtual address 00000048 > printing eip: > e0a4715b > *pde = 00000000 > Oops: 0000 [#1] > Modules linked in: kdapltest ib_dat_provider dat ib_cm ib_at ib_ipoib ib_sa > md5 ipv6 parport_pc lp parport autofs4 nfs lockd sunrpc i2c_piix4 i2c_core > ib_mthca ib_mad ib_core e100 mii floppy sg aic7xxx sd_mod scsi_mod > CPU: 0 > EIP: 0060:[] Not tainted VLI > EFLAGS: 00010096 (2.6.11-openib) > EIP is at ib_destroy_cm_id+0xb/0x790 [ib_cm] > eax: 00000000 ebx: 00000000 ecx: d92f84e4 edx: d75234a0 > esi: 00000296 edi: d92f84e0 ebp: e1ac6f4f esp: dfd9becc > ds: 007b es: 007b ss: 0068 > Process dapl_cm_wq/0 (pid: 5625, threadinfo=dfd9b000 task=d74268d0) > Stack: 00000000 56360d6c 0004d7ea d7426350 d74268d0 d7426350 d74268d0 > de14f9c0 > dfd9bf40 c035fee4 d75234e0 00000001 00000003 d75234c8 00000001 > 00000003 > dfd9bf40 00000096 00000000 00000292 000000c2 dbd75330 000000c1 > d92f84e0 > Call Trace: > [] schedule+0x2e4/0x620 > [] dapl_ib_destroy_cm_id_work+0x21/0x2e [ib_dat_provider] > [] worker_thread+0x211/0x440 > [] default_wake_function+0x0/0x10 > [] default_wake_function+0x0/0x10 > [] worker_thread+0x0/0x440 > [] kthread+0x96/0xe0 > [] kthread+0x0/0xe0 > [] kernel_thread_helper+0x5/0x18 > Code: 85 c0 75 02 5b c3 e8 f5 fc ff ff 8b 83 88 00 00 00 e8 ba 94 70 df 31 c0 > 89 83 88 00 00 00 5b c3 57 56 53 83 ec 5c 89 c3 9c 5e fa <81> 7b 48 3c 4b 24 > 1d 74 25 c7 04 24 d4 f3 a4 e0 8d 43 48 b9 6c > > > On Thu, 5 May 2005, Itamar wrote: > >> Clean up kdapl : evd flow >> >> Signed-off-by: Itamar Rabenstein >> >> Index: dapl_evd_dto_callb.c >> =================================================================== >> --- dapl_evd_dto_callb.c (revision 2257) >> +++ dapl_evd_dto_callb.c (working copy) >> @@ -98,46 +98,26 @@ void dapl_evd_dto_callback(ib_cq_handle_ >> * EVD is enabled. >> */ >> >> - if (state == DAPL_EVD_STATE_WAITED) { >> - /* >> - * If we could, it would be best to avoid this wakeup >> - * (and the context switch) unless the number of events/CQs >> - * waiting for the waiter was its threshold. We don't >> - * currently have the ability to determine that without >> - * dequeueing the events, and we can't do that for >> - * synchronization reasons (racing with the waiter waking >> - * up and dequeuing, sparked by other callbacks). >> - */ >> + if (state == DAPL_EVD_STATE_OPEN && (evd_ptr->cno_ptr != NULL)) { >> + /* >> + * Re-enable callback, *then* trigger. >> + * This guarantees we won't miss any events. >> + */ >> + dat_status = dapl_ib_completion_notify(evd_ptr, >> + IB_NOTIFY_ON_NEXT_COMP); >> >> - /* >> - * We don't need to worry about taking the lock for the >> - * wakeup because wakeups are sticky. >> - */ >> - dapl_os_wait_object_wakeup(&evd_ptr->wait_object); >> - } else if (state == DAPL_EVD_STATE_OPEN) { >> - DAPL_CNO *cno = evd_ptr->cno_ptr; >> - if (evd_ptr->evd_enabled && (evd_ptr->cno_ptr != NULL)) { >> - /* >> - * Re-enable callback, *then* trigger. >> - * This guarantees we won't miss any events. >> - */ >> - dat_status = dapl_ib_completion_notify(evd_ptr, >> - >> IB_NOTIFY_ON_NEXT_COMP); >> - >> - if (DAT_SUCCESS != dat_status) { >> - >> (void)dapl_evd_post_async_error_event(evd_ptr-> >> - header. >> - >> owner_ia-> >> + if (DAT_SUCCESS != dat_status) { >> + (void)dapl_evd_post_async_error_event(evd_ptr-> >> + header. >> + owner_ia-> >> async_error_evd, >> DAT_ASYNC_ERROR_PROVIDER_INTERNAL_ERROR, >> (DAT_IA_HANDLE) >> evd_ptr-> >> header. >> owner_ia); >> - } >> - >> - dapl_cno_trigger(cno, evd_ptr); >> - } >> - } >> + } >> + dapl_cno_trigger(evd_ptr->cno_ptr, evd_ptr); >> + } >> dapl_dbg_log(DAPL_DBG_TYPE_RTN, "dapl_evd_dto_callback () >> returns\n"); >> } >> Index: dapl_ep_create.c >> =================================================================== >> --- dapl_ep_create.c (revision 2257) >> +++ dapl_ep_create.c (working copy) >> @@ -74,7 +74,6 @@ dapl_ep_create(DAT_IA_HANDLE ia_handle, >> DAPL_IA *ia_ptr; >> DAPL_EP *ep_ptr; >> DAT_EP_ATTR ep_attr_limit; >> - DAPL_EVD *evd_ptr; >> DAT_RETURN dat_status; >> >> dapl_dbg_log(DAPL_DBG_TYPE_API, >> @@ -194,39 +193,6 @@ dapl_ep_create(DAT_IA_HANDLE ia_handle, >> } >> } >> >> - /* >> - * Verify the completion flags for the EVD and the EP >> - */ >> - /* >> - * XXX FIXME >> - * XXX Need to make assign the EVD to the right completion type >> - * XXX depending on the EP attributes. Fail if the types don't >> - * XXX match, they are mutually exclusive. >> - */ >> - evd_ptr = (DAPL_EVD *) recv_evd_handle; >> - if (evd_ptr != NULL && evd_ptr->completion_type == >> DAPL_EVD_STATE_INIT) { >> - if (ep_attr != NULL && >> - ep_attr->recv_completion_flags == >> - DAT_COMPLETION_DEFAULT_FLAG) { >> - evd_ptr->completion_type = DAPL_EVD_STATE_THRESHOLD; >> - } else { >> - evd_ptr->completion_type = >> - ep_attr->recv_completion_flags; >> - } >> - } >> - >> - evd_ptr = (DAPL_EVD *) request_evd_handle; >> - if (evd_ptr != NULL && evd_ptr->completion_type == >> DAPL_EVD_STATE_INIT) { >> - if (ep_attr != NULL && >> - ep_attr->recv_completion_flags == >> - DAT_COMPLETION_DEFAULT_FLAG) { >> - evd_ptr->completion_type = DAPL_EVD_STATE_THRESHOLD; >> - } else { >> - evd_ptr->completion_type = >> - ep_attr->recv_completion_flags; >> - } >> - } >> - >> /* Allocate EP */ >> ep_ptr = dapl_ep_alloc(ia_ptr, ep_attr); >> if (ep_ptr == NULL) { >> Index: dapl_openib_dto.h >> =================================================================== >> --- dapl_openib_dto.h (revision 2257) >> +++ dapl_openib_dto.h (working copy) >> @@ -167,5 +167,6 @@ dapl_ib_post_send(DAPL_EP * ep_ptr, >> #define DAPL_GET_CQE_OPTYPE(cqe_p) ((ib_work_completion_t *)cqe_p)->opcode >> #define DAPL_GET_CQE_BYTESNUM(cqe_p) ((ib_work_completion_t >> *)cqe_p)->byte_len >> #define DAPL_GET_CQE_STATUS(cqe_p) ((ib_work_completion_t *)cqe_p)->status >> +#define DAPL_GET_CQE_VND_SYND(cqe_p) ((ib_work_completion_t >> *)cqe_ptr)->vendor_err >> >> #endif /* DAPL_OPENIB_DTO_H */ >> Index: dapl_evd_resize.c >> =================================================================== >> --- dapl_evd_resize.c (revision 2257) >> +++ dapl_evd_resize.c (working copy) >> @@ -93,13 +93,6 @@ DAT_RETURN dapl_evd_resize(DAT_EVD_HANDL >> >> spin_lock(&evd_ptr->header.lock); >> >> - /* Don't try to resize if we are actively waiting */ >> - if (evd_ptr->evd_state == DAPL_EVD_STATE_WAITED) { >> - spin_unlock(&evd_ptr->header.lock); >> - dat_status = DAT_ERROR(DAT_INVALID_STATE, 0); >> - goto bail; >> - } >> - >> pend_cnt = dapl_rbuf_count(&evd_ptr->pending_event_queue); >> if (pend_cnt > evd_qlen) { >> spin_unlock(&evd_ptr->header.lock); >> Index: dapl_evd_kcreate.c >> =================================================================== >> --- dapl_evd_kcreate.c (revision 2257) >> +++ dapl_evd_kcreate.c (working copy) >> @@ -96,16 +96,13 @@ DAT_RETURN dapl_evd_kcreate(DAT_IA_HANDL >> goto bail; >> } >> >> - cno_ptr = dapl_cno_alloc (ia_ptr, upcall); >> - >> - if (!cno_ptr) >> - { >> - dat_status = DAT_INSUFFICIENT_RESOURCES; >> - goto bail; >> - } >> - >> - cno_ptr->cno_upcall_policy = upcall_policy; >> + cno_ptr = dapl_cno_alloc (ia_ptr, upcall); >> >> + if (!cno_ptr) { >> + dat_status = DAT_INSUFFICIENT_RESOURCES; >> + goto bail; >> + } >> + cno_ptr->cno_upcall_policy = upcall_policy; >> >> dat_status = dapl_evd_internal_create(ia_ptr, >> cno_ptr, >> @@ -115,9 +112,6 @@ DAT_RETURN dapl_evd_kcreate(DAT_IA_HANDL >> goto bail; >> } >> >> - evd_ptr->evd_state = DAPL_EVD_STATE_OPEN; >> - evd_ptr->evd_enabled = DAT_TRUE; >> - >> evdhandle = *evd_handle = (DAT_EVD_HANDLE) evd_ptr; >> >> bail: >> Index: dapl_openib_util.h >> =================================================================== >> --- dapl_openib_util.h (revision 2257) >> +++ dapl_openib_util.h (working copy) >> @@ -89,22 +89,6 @@ typedef struct ib_hca_transport { >> u16 lid; >> } ib_hca_transport_t; >> >> -/* >> - * ib_shm_transport_t structure. Define fields specific to this >> - * provider implementation necessary to maintain shared memory state >> - * >> - * OpenIB does not have an API to directly register memory as shared >> - * memory. Instead, we must register memory, then register that memory >> - * as shared. This gives us a new mr_handle, as well as a new l_key and >> - * r_key, so we need to save the original handle until it can be >> - * disposed of when the user does an lmr_free. The l_key and r_key are >> - * not available and will disappear when we deallocate the mr_handle, so >> - * we don't save them. >> - */ >> -typedef struct ib_shm_transport { >> - ib_mr_handle_t mr_handle; >> -} ib_shm_transport_t; >> - >> #define IB_INVALID_HANDLE NULL >> >> #define IB_MAX_REQ_PDATA_SIZE 92 >> Index: dapl_ep_create_with_srq.c >> =================================================================== >> --- dapl_ep_create_with_srq.c (revision 2257) >> +++ dapl_ep_create_with_srq.c (working copy) >> @@ -81,7 +81,6 @@ dapl_ep_create_with_srq(DAT_IA_HANDLE ia >> DAPL_IA *ia_ptr; >> DAPL_EP *ep_ptr; >> DAT_EP_ATTR ep_attr_limit; >> - DAPL_EVD *evd_ptr; >> DAT_RETURN dat_status; >> >> dat_status = DAT_SUCCESS; >> @@ -209,39 +208,6 @@ dapl_ep_create_with_srq(DAT_IA_HANDLE ia >> } >> } >> >> - /* >> - * Verify the completion flags for the EVD and the EP >> - */ >> - /* >> - * XXX FIXME >> - * XXX Need to make assign the EVD to the right completion type >> - * XXX depending on the EP attributes. Fail if the types don't >> - * XXX match, they are mutually exclusive. >> - */ >> - evd_ptr = (DAPL_EVD *) recv_evd_handle; >> - if (evd_ptr != NULL && evd_ptr->completion_type == >> DAPL_EVD_STATE_INIT) { >> - if (ep_attr != NULL && >> - ep_attr->recv_completion_flags == >> - DAT_COMPLETION_DEFAULT_FLAG) { >> - evd_ptr->completion_type = DAPL_EVD_STATE_THRESHOLD; >> - } else { >> - evd_ptr->completion_type = >> - ep_attr->recv_completion_flags; >> - } >> - } >> - >> - evd_ptr = (DAPL_EVD *) request_evd_handle; >> - if (evd_ptr != NULL && evd_ptr->completion_type == >> DAPL_EVD_STATE_INIT) { >> - if (ep_attr != NULL && >> - ep_attr->recv_completion_flags == >> - DAT_COMPLETION_DEFAULT_FLAG) { >> - evd_ptr->completion_type = DAPL_EVD_STATE_THRESHOLD; >> - } else { >> - evd_ptr->completion_type = >> - ep_attr->recv_completion_flags; >> - } >> - } >> - >> dat_status = DAT_NOT_IMPLEMENTED; >> >> /* >> Index: dapl_evd_util.c >> =================================================================== >> --- dapl_evd_util.c (revision 2257) >> +++ dapl_evd_util.c (working copy) >> @@ -77,9 +77,6 @@ dapl_evd_internal_create(DAPL_IA * ia_pt >> evd_ptr->evd_producer_locking_needed = >> ((evd_flags & ~(DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG)) != 0); >> >> - /* Before we setup any callbacks, transition state to OPEN. */ >> - evd_ptr->evd_state = DAPL_EVD_STATE_OPEN; >> - >> if (evd_flags & DAT_EVD_ASYNC_FLAG) { >> /* >> * There is no cq associate with async evd. Set it to invalid >> @@ -161,19 +158,13 @@ DAPL_EVD *dapl_evd_alloc(DAPL_IA * ia_pt >> /* >> * Initialize the body >> */ >> - evd_ptr->evd_state = DAPL_EVD_STATE_INITIAL; >> + evd_ptr->evd_state = DAPL_EVD_STATE_OPEN; >> evd_ptr->evd_flags = evd_flags; >> - evd_ptr->evd_enabled = DAT_TRUE; >> - evd_ptr->evd_waitable = DAT_TRUE; >> evd_ptr->evd_producer_locking_needed = 1; /* Conservative >> value. */ >> evd_ptr->ib_cq_handle = IB_INVALID_HANDLE; >> atomic_set(&evd_ptr->evd_ref_count, 0); >> evd_ptr->catastrophic_overflow = DAT_FALSE; >> evd_ptr->qlen = qlen; >> - evd_ptr->completion_type = DAPL_EVD_STATE_THRESHOLD; /* FIXME: >> should be DAPL_EVD_STATE_INIT */ >> - dapl_os_wait_object_init(&evd_ptr->wait_object); >> - >> - evd_ptr->cno_active_count = 0; >> evd_ptr->cno_ptr = cno_ptr; >> >> bail: >> @@ -219,10 +210,6 @@ DAT_RETURN dapl_evd_event_alloc(DAPL_EVD >> event_ptr++; >> } >> >> - evd_ptr->cq_notified = DAT_FALSE; >> - evd_ptr->cq_notified_when = 0; >> - evd_ptr->threshold = 0; >> - >> bail: >> return dat_status; >> } >> @@ -331,7 +318,6 @@ DAT_RETURN dapl_evd_dealloc(DAPL_EVD * e >> kfree(evd_ptr->events); >> } >> >> - dapl_os_wait_object_destroy(&evd_ptr->wait_object); >> kfree(evd_ptr); >> >> bail: >> @@ -445,73 +431,20 @@ static void dapl_evd_post_event(DAPL_EVD >> (void *)event_ptr); >> dapl_os_assert(dat_status == DAT_SUCCESS); >> >> - dapl_os_assert(evd_ptr->evd_state == DAPL_EVD_STATE_WAITED >> - || evd_ptr->evd_state == DAPL_EVD_STATE_OPEN); >> + dapl_os_assert(evd_ptr->evd_state == DAPL_EVD_STATE_OPEN); >> >> if (evd_ptr->evd_state == DAPL_EVD_STATE_OPEN) { >> /* No waiter. Arrange to trigger a CNO if it exists. */ >> - >> - if (evd_ptr->evd_enabled) { >> - cno_to_trigger = evd_ptr->cno_ptr; >> - } >> - >> - if (evd_ptr->evd_producer_locking_needed) { >> - spin_unlock(&evd_ptr->header.lock); >> - } >> - } else { >> - /* >> - * We're in DAPL_EVD_STATE_WAITED. Take the lock if >> - * we don't have it, recheck, and signal. >> - */ >> - if (!evd_ptr->evd_producer_locking_needed) { >> - spin_lock(&evd_ptr->header.lock); >> - } >> - >> - if (evd_ptr->evd_state == DAPL_EVD_STATE_WAITED >> - && (dapl_rbuf_count(&evd_ptr->pending_event_queue) >> - >= evd_ptr->threshold)) { >> - spin_unlock(&evd_ptr->header.lock); >> - dapl_os_wait_object_wakeup(&evd_ptr->wait_object); >> - } else { >> - spin_unlock(&evd_ptr->header.lock); >> - } >> + cno_to_trigger = evd_ptr->cno_ptr; >> + if (evd_ptr->evd_producer_locking_needed) { >> + spin_unlock(&evd_ptr->header.lock); >> + } >> } >> - >> if (cno_to_trigger != NULL) { >> dapl_cno_trigger (cno_to_trigger, evd_ptr); >> } >> } >> >> -/* >> - * Post the to the evd. Do not do any wakeup processing. >> - * This function should only be called if it is known that there are >> - * no waiters that it is appropriate to wakeup on this EVD. An example >> - * of such a situation is during internal dat_evd_wait() processing. >> - * >> - * If producer side locking is required, the EVD lock must be held upon >> - * entry to this function. >> - */ >> - >> -static void >> -dapl_evd_post_event_nosignal(DAPL_EVD * evd_ptr, const DAT_EVENT * >> event_ptr) >> -{ >> - DAT_RETURN dat_status; >> - >> - dapl_dbg_log(DAPL_DBG_TYPE_EVD, >> - "dapl_evd_post_event_nonsignal: Called with event # >> %x\n", >> - event_ptr->event_number); >> - >> - dat_status = dapl_rbuf_add(&evd_ptr->pending_event_queue, >> - (void *)event_ptr); >> - dapl_os_assert(dat_status == DAT_SUCCESS); >> - >> - dapl_os_assert(evd_ptr->evd_state == DAPL_EVD_STATE_WAITED >> - || evd_ptr->evd_state == DAPL_EVD_STATE_OPEN); >> - >> - if (evd_ptr->evd_producer_locking_needed) { >> - spin_unlock(&evd_ptr->header.lock); >> - } >> -} >> >> /* >> * format an overflow event for posting >> @@ -689,49 +622,6 @@ dapl_evd_post_software_event(DAPL_EVD * >> } >> >> /* >> - * dapl_evd_post_generic_event >> - * >> - * Post a generic event type. Not used by all providers >> - * >> - * Input: >> - * evd_ptr >> - * event_number >> - * data >> - * >> - * Output: >> - * none >> - * >> - * Returns: >> - * DAT_SUCCESS >> - * >> - */ >> -DAT_RETURN >> -dapl_evd_post_generic_event(DAPL_EVD * evd_ptr, >> - DAT_EVENT_NUMBER event_number, >> - DAT_EVENT_DATA * data) >> -{ >> - DAT_EVENT *event_ptr; >> - >> - event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number); >> - /* >> - * Note event lock may be held on successful return >> - * to be released by dapl_evd_post_event(), if provider side locking >> - * is needed. >> - */ >> - >> - if (event_ptr == NULL) { >> - return DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, >> - DAT_RESOURCE_MEMORY); >> - } >> - >> - event_ptr->event_data = *data; >> - >> - dapl_evd_post_event(evd_ptr, event_ptr); >> - >> - return DAT_SUCCESS; >> -} >> - >> -/* >> * dapl_evd_cqe_to_event >> * >> * Convert a CQE into an event structure. >> @@ -866,128 +756,13 @@ dapl_evd_cqe_to_event(DAPL_EVD * evd_ptr >> } >> } /* end switch */ >> >> - /* >> - * Most error DTO ops result in disconnecting the EP. See >> - * IBTA Vol 1.1, Chapter 10,Table 68, for expected effect on >> - * state. >> - */ >> - if ((dto_status != DAT_DTO_SUCCESS) && >> - (dto_status != DAT_DTO_ERR_FLUSHED)) { >> -#ifdef CONN_EV_BROKEN_SUPPORTED >> - DAPL_EVD *evd_ptr; >> - >> - /* >> - * If we are connected, generate disconnect and generate an >> - * event. We may be racing with other disconnect ops, so we >> - * need to check. We may also be racing CM connection events, >> - * requiring us to check for connection pending states too. >> - */ >> - spin_lock(&ep_ptr->header.lock); >> - if (ep_ptr->param.ep_state == DAT_EP_STATE_CONNECTED || >> - ep_ptr->param.ep_state == >> - DAT_EP_STATE_ACTIVE_CONNECTION_PENDING >> - || ep_ptr->param.ep_state == >> - DAT_EP_STATE_PASSIVE_CONNECTION_PENDING >> - || ep_ptr->param.ep_state == >> - DAT_EP_STATE_COMPLETION_PENDING) { >> - ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; >> - spin_unlock(&ep_ptr->header.lock); >> - >> - /* Let the other side know we have disconnected */ >> - // cant call to dapl_ib_disconnect besause it is in >> interrupt context >> - // and this call is blocking need to fix ??? >> - // (void) dapl_ib_disconnect (ep_ptr, >> DAT_CLOSE_ABRUPT_FLAG); >> - printk("FIXME DAPL BUG %s %s %d \n", __FILE__, >> - __FUNCTION__, __LINE__); >> - /* ... and clean up the local side */ >> - evd_ptr = (DAPL_EVD *) >> ep_ptr->param.connect_evd_handle; >> - if (evd_ptr != NULL) { >> - dapl_evd_post_connection_event(evd_ptr, >> - >> DAT_CONNECTION_EVENT_BROKEN, >> - (DAT_HANDLE) >> - ep_ptr, 0, 0); >> - } >> - } else { >> - spin_unlock(&ep_ptr->header.lock); >> - } >> - >> - dapl_dbg_log(DAPL_DBG_TYPE_DTO_COMP_ERR, >> - " DTO completion ERROR: %d: op %#x (ep >> disconnected)\n", >> - DAPL_GET_CQE_STATUS(cqe_ptr), >> - DAPL_GET_CQE_OPTYPE(cqe_ptr)); >> -#else >> - dapl_dbg_log(DAPL_DBG_TYPE_DTO_COMP_ERR, >> - " DTO completion ERROR: %d: op %#x\n", >> - DAPL_GET_CQE_STATUS(cqe_ptr), >> - DAPL_GET_CQE_OPTYPE(cqe_ptr)); >> -#endif >> - } >> -} >> - >> -/* >> - * dapl_evd_copy_cq >> - * >> - * Copy all entries on a CQ associated with the EVD onto that EVD >> - * Up to caller to handle races, if any. Note that no EVD waiters will >> - * be awoken by this copy. >> - * >> - * Input: >> - * evd_ptr >> - * >> - * Output: >> - * None >> - * >> - * Returns: >> - * none >> - * >> - */ >> -void dapl_evd_copy_cq(DAPL_EVD * evd_ptr) >> -{ >> - ib_work_completion_t cur_cqe; >> - DAT_RETURN dat_status; >> - DAT_EVENT *event; >> - >> - if (evd_ptr->ib_cq_handle == IB_INVALID_HANDLE) { >> - /* Nothing to do if no CQ. */ >> - return; >> - } >> - >> - while (1) { >> - dat_status = >> - >> dapl_ib_completion_poll(evd_ptr->header.owner_ia->hca_ptr, >> - evd_ptr, &cur_cqe); >> - >> - if (dat_status != DAT_SUCCESS) { >> - break; >> - } >> - >> - /* For debugging. */ >> - dapl_evd_eh_print_cqe(&cur_cqe); >> - >> - /* >> - * Can use DAT_DTO_COMPLETION_EVENT because >> dapl_evd_cqe_to_event >> - * will overwrite. >> - */ >> - >> - event = >> - dapl_evd_get_and_init_event(evd_ptr, >> - DAT_DTO_COMPLETION_EVENT); >> - if (event == NULL) { >> - /* We've already attempted the overflow post; return. >> */ >> - return; >> - } >> - >> - dapl_evd_cqe_to_event(evd_ptr, &cur_cqe, event); >> - >> - dapl_evd_post_event_nosignal(evd_ptr, event); >> - } >> - >> - if (DAT_GET_TYPE(dat_status) != DAT_QUEUE_EMPTY) { >> - dapl_dbg_log(DAPL_DBG_TYPE_EVD, >> - "dapl_evd_copy_cq: dapl_ib_completion_poll >> returned 0x%x\n", >> - dat_status); >> - dapl_os_assert(!"Bad return from dapl_ib_completion_poll"); >> - } >> + if (dto_status != DAT_DTO_SUCCESS) { >> + dapl_dbg_log(DAPL_DBG_TYPE_DTO_COMP_ERR, >> + " DTO completion ERROR:status %d: op %#x vendore synd: >> 0x%x\n", >> + DAPL_GET_CQE_STATUS (cqe_ptr), >> + DAPL_GET_CQE_OPTYPE (cqe_ptr), >> + DAPL_GET_CQE_VND_SYND(cqe_ptr)); >> + } >> } >> >> /* >> Index: dapl_evd_util.h >> =================================================================== >> --- dapl_evd_util.h (revision 2257) >> +++ dapl_evd_util.h (working copy) >> @@ -87,10 +87,6 @@ DAT_RETURN >> dapl_evd_post_software_event(DAPL_EVD * evd_ptr, >> DAT_EVENT_NUMBER event_number, DAT_PVOID >> pointer); >> >> -DAT_RETURN >> -dapl_evd_post_generic_event(DAPL_EVD * evd_ptr, >> - DAT_EVENT_NUMBER event_number, >> - DAT_EVENT_DATA * data); >> >> /************************************* >> * dapl internal callbacks functions * >> @@ -115,8 +111,6 @@ extern void dapl_evd_cq_async_error_call >> extern void dapl_evd_qp_async_error_callback(ib_error_record_t * cause_ptr, >> void *context); >> >> -extern void dapl_evd_copy_cq(DAPL_EVD * evd_ptr); >> - >> extern DAT_RETURN dapl_evd_cq_poll_to_event(DAPL_EVD * evd_ptr, >> DAT_EVENT * event); >> >> Index: dapl.h >> =================================================================== >> --- dapl.h (revision 2257) >> +++ dapl.h (working copy) >> @@ -64,10 +64,7 @@ typedef enum dapl_magic { >> } DAPL_MAGIC; >> >> typedef enum dapl_evd_state { >> - DAPL_EVD_STATE_TERMINAL, >> - DAPL_EVD_STATE_INITIAL, >> DAPL_EVD_STATE_OPEN, >> - DAPL_EVD_STATE_WAITED, >> DAPL_EVD_STATE_DEAD = 0xDEAD >> } DAPL_EVD_STATE; >> >> @@ -270,8 +267,6 @@ struct dapl_evd >> >> DAPL_EVD_STATE evd_state; >> DAT_EVD_FLAGS evd_flags; >> - DAT_BOOLEAN evd_enabled; /* For attached CNO. */ >> - DAT_BOOLEAN evd_waitable; /* EVD state. */ >> >> /* Derived from evd_flags; see dapls_evd_internal_create. */ >> DAT_BOOLEAN evd_producer_locking_needed; >> @@ -279,9 +274,6 @@ struct dapl_evd >> /* Every EVD has a CQ unless it is a SOFTWARE_EVENT only EVD */ >> ib_cq_handle_t ib_cq_handle; >> >> - /* Mellanox Specific completion handle for >> registration/de-registration */ >> - ib_comp_handle_t ib_comp_handle; >> - >> /* An Event Dispatcher cannot be freed while >> * it is referenced elsewhere. >> */ >> @@ -295,20 +287,7 @@ struct dapl_evd >> DAT_EVENT *events; >> DAPL_RING_BUFFER free_event_queue; >> DAPL_RING_BUFFER pending_event_queue; >> - >> - /* CQ Completions are not placed into 'deferred_events' >> - ** rather they are simply left on the Completion Queue >> - ** and the fact that there was a notification is flagged. >> - */ >> - DAT_BOOLEAN cq_notified; >> - DAPL_OS_TICKS cq_notified_when; >> - >> - DAT_COUNT cno_active_count; >> DAPL_CNO *cno_ptr; >> - >> - DAPL_OS_WAIT_OBJECT wait_object; >> - DAT_COUNT threshold; >> - DAPL_EVD_COMPLETION completion_type; >> }; >> >> /* uDAPL timer entry, used to queue timeouts */ >> -- >> Itamar >> > From tduffy at sun.com Mon May 9 11:09:55 2005 From: tduffy at sun.com (Tom Duffy) Date: Mon, 09 May 2005 11:09:55 -0700 Subject: [openib-general] Re: [PATCH] kDAPL: remove typedef DAT_PROVIDER In-Reply-To: References: <1115424198.19661.11.camel@duffman> Message-ID: <1115662195.9691.7.camel@duffman> On Mon, 2005-05-09 at 14:06 -0400, James Lentini wrote: > Committed revision 2287. > > On Fri, 6 May 2005, Tom Duffy wrote: > > tduffy> James, thanks for applying my last patch. > tduffy> > tduffy> You know where I am going with this... this is the first in what will be > tduffy> a huge amount of patches. I am happy to go through and send them all to > tduffy> the list, but it might be quicker and easier for you to do it directly > tduffy> to your tree for all the structs and enums. Sup to you. > tduffy> So, did you want me to continue with sending these type of patches? -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From jlentini at netapp.com Mon May 9 11:42:33 2005 From: jlentini at netapp.com (James Lentini) Date: Mon, 9 May 2005 14:42:33 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: remove typedef DAT_PROVIDER In-Reply-To: <1115662195.9691.7.camel@duffman> References: <1115424198.19661.11.camel@duffman> <1115662195.9691.7.camel@duffman> Message-ID: Definitely. On Mon, 9 May 2005, Tom Duffy wrote: > On Mon, 2005-05-09 at 14:06 -0400, James Lentini wrote: >> Committed revision 2287. >> >> On Fri, 6 May 2005, Tom Duffy wrote: >> >> tduffy> James, thanks for applying my last patch. >> tduffy> >> tduffy> You know where I am going with this... this is the first in what will be >> tduffy> a huge amount of patches. I am happy to go through and send them all to >> tduffy> the list, but it might be quicker and easier for you to do it directly >> tduffy> to your tree for all the structs and enums. Sup to you. >> tduffy> > > So, did you want me to continue with sending these type of patches? > > -tduffy > From libor at topspin.com Mon May 9 11:39:43 2005 From: libor at topspin.com (Libor Michalek) Date: Mon, 9 May 2005 11:39:43 -0700 Subject: [openib-general] Re: A Couple of User CM Questions In-Reply-To: <1115659336.4490.91.camel@localhost.localdomain>; from halr@voltaire.com on Mon, May 09, 2005 at 01:22:17PM -0400 References: <1115659336.4490.91.camel@localhost.localdomain> Message-ID: <20050509113943.A2462@topspin.com> On Mon, May 09, 2005 at 01:22:17PM -0400, Hal Rosenstock wrote: > Hi Libor, > > I have a couple of questions pertaining to the user CM. > > 1. It appears that device (and port) are not supported either incoming > or outgoing. So if that is correct, I presume only the first device > (first port) is the only one which can be expected to work. Correct ? > If so, that should be added to README as a current limitation. > > 2. Any idea on how device would be supported ? Would this just be with > an IB device number or a string like "mthca0" ? I was thinking that the destination GID in the req_event path record would be sufficient to identify the local device/port, but I had not taken a look at sidr_req_event. I think this is the correct GID to use for creating the QP, but I have not taken a look at how easy it is to turn a GID into a suitable device/portin user verbs. Thoughts? -Libor From jlentini at netapp.com Mon May 9 11:54:06 2005 From: jlentini at netapp.com (James Lentini) Date: Mon, 9 May 2005 14:54:06 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: remove typedef DAT_PROVIDER In-Reply-To: <1115662195.9691.7.camel@duffman> References: <1115424198.19661.11.camel@duffman> <1115662195.9691.7.camel@duffman> Message-ID: Tom, If you are interested, there is one more thing you could look into/give me advice on: The dat module currently keeps track of its "state" (see the DAT_REGISTRY_STATE enumeration). The registry uses this information to detect the case when a provider or consumer calls a dat registry function before the registry's init function (dat_init) has run. Do we need to protect against this in the kernel? This situation could occur in usersapce when the registry, providers, and consumers could be shared libraries and the library initialization functions were invoked in an arbitrary order. I suspect that we can remove the code, but I wanted to make sure. If the dat registry, dat provider, and a consumer (e.g. NFS-RDMA) were built statically as part of the kernel, would the initialization functions be automatically run in the correct order? james On Mon, 9 May 2005, Tom Duffy wrote: tduffy> On Mon, 2005-05-09 at 14:06 -0400, James Lentini wrote: tduffy> > Committed revision 2287. tduffy> > tduffy> > On Fri, 6 May 2005, Tom Duffy wrote: tduffy> > tduffy> > tduffy> James, thanks for applying my last patch. tduffy> > tduffy> tduffy> > tduffy> You know where I am going with this... this is the first in what will be tduffy> > tduffy> a huge amount of patches. I am happy to go through and send them all to tduffy> > tduffy> the list, but it might be quicker and easier for you to do it directly tduffy> > tduffy> to your tree for all the structs and enums. Sup to you. tduffy> > tduffy> tduffy> tduffy> So, did you want me to continue with sending these type of patches? tduffy> tduffy> -tduffy tduffy> From halr at voltaire.com Mon May 9 11:57:00 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 09 May 2005 14:57:00 -0400 Subject: [openib-general] Re: [openib-commits] r2275 - gen2/trunk/src/linux-kernel/infiniband/ulp/sdp In-Reply-To: <20050507111710.GP15688@aon.at> References: <20050506233401.438A922834D@openib.ca.sandia.gov> <20050507111710.GP15688@aon.at> Message-ID: <1115665020.4490.209.camel@localhost.localdomain> On Sat, 2005-05-07 at 07:17, Bernhard Fischer wrote: > s/seperat/separat/g > in: > trunk/src/userspace/mstflint/flint.cpp:4644 > trunk/src/userspace/management/osm/opensm/osm_lid_mgr.c:272: > trunk/src/userspace/management/osm/include/opensm/osm_port.h:139 > utils/src/linux-user/IBMgtSim/config/share/libtool/ltmain.sh:2516 > The last one should probably just be regenerated or removed, at least > i don't see 'seperate' in libtool any more. > > hm. I should probably better just send a patch instead, sorry. The first one is for Michael. The last one is for Eitan. I just took care of the 2 OpenSM ones. -- Hal From halr at voltaire.com Mon May 9 12:32:20 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 09 May 2005 15:32:20 -0400 Subject: [openib-general] Re: A Couple of User CM Questions In-Reply-To: <20050509113943.A2462@topspin.com> References: <1115659336.4490.91.camel@localhost.localdomain> <20050509113943.A2462@topspin.com> Message-ID: <1115667140.4490.244.camel@localhost.localdomain> On Mon, 2005-05-09 at 14:39, Libor Michalek wrote: > On Mon, May 09, 2005 at 01:22:17PM -0400, Hal Rosenstock wrote: > > Hi Libor, > > > > I have a couple of questions pertaining to the user CM. > > > > 1. It appears that device (and port) are not supported either incoming > > or outgoing. So if that is correct, I presume only the first device > > (first port) is the only one which can be expected to work. Correct ? > > If so, that should be added to README as a current limitation. So I am presuming this is a current limitation. > > 2. Any idea on how device would be supported ? Would this just be with > > an IB device number or a string like "mthca0" ? > > I was thinking that the destination GID in the req_event path record > would be sufficient to identify the local device/port, but I had not > taken a look at sidr_req_event. I think this is the correct GID to use > for creating the QP, but I have not taken a look at how easy it is to > turn a GID into a suitable device/portin user verbs. Thoughts? and source GID (in the REQ primary path) on the active side ? That seems like it would work but wouldn't you need to walk the GID tables on each port on each IB device until you find the match ? Sounds a little expensive but would only be done once per connection. The advantage of GID is this would be common across all IB devices. Not sure what else could be other than a deterministic numbering of IB devices locally which I believe does not currently exist. -- Hal From trimmer at infiniconsys.com Mon May 9 13:06:09 2005 From: trimmer at infiniconsys.com (Rimmer, Todd) Date: Mon, 9 May 2005 16:06:09 -0400 Subject: [openib-general] Re: A Couple of User CM Questions Message-ID: <5D78D28F88822E4D8702BB9EEF1A43670A0679@mercury.infiniconsys.com> > > > 2. Any idea on how device would be supported ? Would this > just be with > > > an IB device number or a string like "mthca0" ? > > > > I was thinking that the destination GID in the req_event > path record > > would be sufficient to identify the local device/port, but I had not > > taken a look at sidr_req_event. I think this is the correct > GID to use > > for creating the QP, but I have not taken a look at how > easy it is to > > turn a GID into a suitable device/portin user verbs. Thoughts? > > and source GID (in the REQ primary path) on the active side ? > > That seems like it would work but wouldn't you need to walk the GID > tables on each port on each IB device until you find the > match ? Sounds > a little expensive but would only be done once per connection. > > The advantage of GID is this would be common across all IB > devices. Not > sure what else could be other than a deterministic numbering of IB > devices locally which I believe does not currently exist. > > -- Hal We found it useful to allow a flexible listener bind and matched REQ based on a comparison of the following fields: // Field is Used by models as follows: // Listen Peer Connect Sidr Register // SID Y Y Y // local GID option Y future option // local LID option Y future option // QPN wildcard Y wildcard // EECN wildcard Y wildcard // CaGUID wildcard Y wildcard // remote GID option Y future option // remote LID option Y future option // private data discriminator length option option option // private data discriminator value option option option The search was done against listening CEPs, no search of port GIDs was necessary for the receiver. The listener (if it chose to listen on a specific set of GIDs or ports), would need to query the CA to obtain the GID list. The sender is already required to supply GIDs in order to formulate a proper REQ (GIDs should come from Path Record). We found the private data discriminator to be necessary to handle multiple instances of the same app on the same node, such as MPI processes, in which case the listener rank was used as the private data which must be matched. Fields marked as option could be omitted by the listener, in which case all values for that would match (hence a listener could listen on all ports very easily). Fields marked as wildcard were always ignored for the given connection type. These fields were required per IBTA in Peer Connect REQ handling. We have found limited use for SIDR so did not add the LID/GID capabilities there. LID is easily possible, GID is probably not worth the work. Unlike REQ, SIDR_REQ does not include the GIDs, so for SIDR GID would only apply to packets with GRH. Todd Rimmer From halr at voltaire.com Mon May 9 13:05:22 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 09 May 2005 16:05:22 -0400 Subject: [openib-general] Re: A Couple of User CM Questions In-Reply-To: <1115667140.4490.244.camel@localhost.localdomain> References: <1115659336.4490.91.camel@localhost.localdomain> <20050509113943.A2462@topspin.com> <1115667140.4490.244.camel@localhost.localdomain> Message-ID: <1115668929.4490.275.camel@localhost.localdomain> On Mon, 2005-05-09 at 15:32, Hal Rosenstock wrote: > On Mon, 2005-05-09 at 14:39, Libor Michalek wrote: > > On Mon, May 09, 2005 at 01:22:17PM -0400, Hal Rosenstock wrote: > > > Hi Libor, > > > > > > I have a couple of questions pertaining to the user CM. > > > > > > 1. It appears that device (and port) are not supported either incoming > > > or outgoing. So if that is correct, I presume only the first device > > > (first port) is the only one which can be expected to work. Correct ? > > > If so, that should be added to README as a current limitation. > > So I am presuming this is a current limitation. > > > > 2. Any idea on how device would be supported ? Would this just be with > > > an IB device number or a string like "mthca0" ? > > > > I was thinking that the destination GID in the req_event path record > > would be sufficient to identify the local device/port, but I had not > > taken a look at sidr_req_event. I think this is the correct GID to use > > for creating the QP, but I have not taken a look at how easy it is to > > turn a GID into a suitable device/portin user verbs. Thoughts? > > and source GID (in the REQ primary path) on the active side ? > > That seems like it would work but wouldn't you need to walk the GID > tables on each port on each IB device until you find the match ? Sounds > a little expensive but would only be done once per connection. > > The advantage of GID is this would be common across all IB devices. Not > sure what else could be other than a deterministic numbering of IB > devices locally which I believe does not currently exist. ib_verbs does have a way to get_devices (ibv_get_devices) and specify expose a pointer to a struct ibv_device. That would be another alternative to using GIDs. That would make it similar to the kernel CM. -- Hal From trimmer at infiniconsys.com Mon May 9 13:10:07 2005 From: trimmer at infiniconsys.com (Rimmer, Todd) Date: Mon, 9 May 2005 16:10:07 -0400 Subject: [openib-general] Re: A Couple of User CM Questions Message-ID: <5D78D28F88822E4D8702BB9EEF1A436773DCD9@mercury.infiniconsys.com> > On Mon, 2005-05-09 at 15:32, Hal Rosenstock wrote: > > On Mon, 2005-05-09 at 14:39, Libor Michalek wrote: > > > On Mon, May 09, 2005 at 01:22:17PM -0400, Hal Rosenstock wrote: > > > > Hi Libor, > > > > > > > > I have a couple of questions pertaining to the user CM. > > > > > > > > 1. It appears that device (and port) are not supported > either incoming > > > > or outgoing. So if that is correct, I presume only the > first device > > > > (first port) is the only one which can be expected to > work. Correct ? > > > > If so, that should be added to README as a current limitation. > > > > So I am presuming this is a current limitation. > > > > > > 2. Any idea on how device would be supported ? Would > this just be with > > > > an IB device number or a string like "mthca0" ? > > > > > > I was thinking that the destination GID in the > req_event path record > > > would be sufficient to identify the local device/port, > but I had not > > > taken a look at sidr_req_event. I think this is the > correct GID to use > > > for creating the QP, but I have not taken a look at how > easy it is to > > > turn a GID into a suitable device/portin user verbs. Thoughts? > > > > and source GID (in the REQ primary path) on the active side ? > > > > That seems like it would work but wouldn't you need to walk the GID > > tables on each port on each IB device until you find the > match ? Sounds > > a little expensive but would only be done once per connection. > > > > The advantage of GID is this would be common across all IB > devices. Not > > sure what else could be other than a deterministic numbering of IB > > devices locally which I believe does not currently exist. > > ib_verbs does have a way to get_devices (ibv_get_devices) and specify > expose a pointer to a struct ibv_device. That would be another > alternative to using GIDs. That would make it similar to the > kernel CM. > > -- Hal I would recommend GIDs, no searching is necessary and a GID based approach would work for both user and kernel CM. ibv_device would have challenges for user space listeners. Todd Rimmer From xma at us.ibm.com Mon May 9 13:36:17 2005 From: xma at us.ibm.com (Shirley Ma) Date: Mon, 9 May 2005 20:36:17 +0000 Subject: [openib-general] HCA Firmware In-Reply-To: <1115324777.4476.7.camel@localhost.localdomain> Message-ID: > I don't recall if there are issues running with 3.2.0 or not. 3.2.0 has no problem. Even 3.1.0 works fine for me. Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From xma at us.ibm.com Mon May 9 13:43:44 2005 From: xma at us.ibm.com (Shirley Ma) Date: Mon, 9 May 2005 20:43:44 +0000 Subject: [openib-general] panic: ib_sa_mcmember_rec_callback Message-ID: Did anyone see below panic before? I hit this panic, after I've updated the stack to the most recent one, it couldn't be reproducible. I guess some recent checkin has fixed this problem. ============================== Oops: Kernel access of bad area, sig: 11 [#1] SMP NR_CPUS=128 NUMA PSERIES LPAR Modules linked in: ib_ipoib ib_sa tg3 evdev joydev st sd_mod sr_mod ipv6 ohci_hcd ib_mthca ib_mad ib_core usbcore sg scsi_mod dm_mod NIP: D0000000002BF204 XER: 20000000 LR: D0000000002BE948 CTR: D00000 00002BF198 REGS: c00000001d4c76a0 TRAP: 0300 Not tainted (2.6.11.7) MSR: 8000000000009032 EE: 1 PR: 0 FP: 0 ME: 1 IR/DR: 11 CR: 24004042 DAR: 0000000000000000 DSISR: 0000000040000000 TASK: c00000000f212030[3964] 'ib_mad1' THREAD: c00000001d4c4000 CPU: 0 GPR00: D0000000002BE948 C00000001D4C7920 D0000000002CEAD8 FFFFFFFFFF FFFFFC GPR04: 0000000000000000 C00000001A9D8680 C00000001D4C7990 C00000000F 6E4AC0 GPR08: C000000001429998 0000000000000000 0000000000000000 0000000000 000000 GPR12: D0000000002BF8A0 C000000000499C00 0000000000000000 0000000000 000000 GPR16: 0000000000000000 0000000000000000 0000000001C00000 0000000002 033C28 GPR20: 0000000002033C28 BFFFFFFFFE400000 0000000001A1F9F0 C000000007 068040 GPR24: 0000000000000001 8000000000009032 C000000002855410 FFFFFFFFFF FFFFFC GPR28: C0000000131C4690 C0000000131C4680 D0000000002CE880 C00000001D 4C7920 NIP [d0000000002bf204] .ib_sa_mcmember_rec_callback+0x6c/0xc4 [ib_sa ] LR [d0000000002be948] .send_handler+0xb8/0x190 [ib_sa] Call Trace: [c00000001d4c7920] [c00000001d4c7970] 0xc00000001d4c7970 (unreliable ) [c00000001d4c7a10] [d0000000002be948] .send_handler+0xb8/0x190 [ib_s a] [c00000001d4c7ab0] [d0000000002d640c] .ib_mad_complete_send_wr+0x31c /0x324 [ib_mad] [c00000001d4c7b60] [d0000000002d6478] .ib_mad_send_done_handler+0x64 /0x1f0 [ib_mad] [c00000001d4c7c20] [d0000000002d6ad0] .ib_mad_completion_handler+0x2 28/0xa48 [ib_mad] [c00000001d4c7da0] [c00000000006d150] .worker_thread+0x258/0x32c [c00000001d4c7ed0] [c000000000074640] .kthread+0x18c/0x1e0 [c00000001d4c7f90] [c0000000000142b8] .kernel_thread+0x4c/0x6c Instruction dump: 38800013 7f63db78 3bbcfff0 38df0070 419e0068 e87e8088 480008f5 e8410 028 e93cfff0 e8bd0008 7f63db78 389f0070 f8410028 60000000 600 00000 Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From Tom.Duffy at Sun.COM Mon May 9 14:39:39 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 09 May 2005 14:39:39 -0700 Subject: [openib-general] [PATCH][0/26] kDAPL: Remove typedefs in dat.h Message-ID: <11156747793273@sun.com> The following set of patches removes the use of typedef for all the structs defined in dat/dat.h. Please apply in order. Signed-off-by: Tom Duffy Your friendly openib janitor. From Tom.Duffy at Sun.COM Mon May 9 14:39:40 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 09 May 2005 14:39:40 -0700 Subject: [openib-general] [PATCH][2/26] kDAPL: Remove typedef DAT_PROVIDER_INFO In-Reply-To: <11156747791462@sun.com> References: <11156747793273@sun.com> <11156747791462@sun.com> Message-ID: <1115674780270@sun.com> Signed-off-by: Tom Duffy diff -Nur -X /home/tduffy/dontdiff linux-kernel/dat/api.c linux-kernel2/dat/api.c --- linux-kernel/dat/api.c 2005-05-09 13:44:25.023960000 -0700 +++ linux-kernel2/dat/api.c 2005-05-09 13:47:18.873960000 -0700 @@ -583,7 +583,7 @@ DAT_BOOLEAN thread_safety) { DAT_IA_OPEN_FUNC ia_open_func; - DAT_PROVIDER_INFO info; + struct dat_provider_info info; DAT_RETURN status; size_t len; @@ -658,7 +658,7 @@ dat_dbg_print(DAT_DBG_CLASS_CONSUMER_API, "close function for IA %s failed\n", ia_name); } else { - DAT_PROVIDER_INFO info; + struct dat_provider_info info; size_t len; len = strlen(ia_name); @@ -724,7 +724,7 @@ ***********************************************************************/ DAT_RETURN dat_registry_add_provider(const struct dat_provider *provider, - const DAT_PROVIDER_INFO *provider_info) + const struct dat_provider_info *provider_info) { dat_dbg_print(DAT_DBG_CLASS_PROVIDER_API, "%s(ia:%s, ver:%x.%x, thrd_safe:%x)\n", @@ -752,7 +752,7 @@ ***********************************************************************/ DAT_RETURN dat_registry_remove_provider(const struct dat_provider *provider, - const DAT_PROVIDER_INFO *provider_info) + const struct dat_provider_info *provider_info) { dat_dbg_print(DAT_DBG_CLASS_PROVIDER_API, "%s(ia:%s)\n", @@ -775,7 +775,7 @@ DAT_RETURN dat_registry_list_providers(DAT_COUNT max_to_return, DAT_COUNT *entries_returned, - DAT_PROVIDER_INFO *(dat_provider_list[])) + struct dat_provider_info *(dat_provider_list[])) { DAT_RETURN status; diff -Nur -X /home/tduffy/dontdiff linux-kernel/dat/consumer.c linux-kernel2/dat/consumer.c --- linux-kernel/dat/consumer.c 2005-05-09 11:15:26.447001000 -0700 +++ linux-kernel2/dat/consumer.c 2005-05-09 11:21:32.399003000 -0700 @@ -583,7 +583,7 @@ DAT_BOOLEAN thread_safety) { DAT_IA_OPEN_FUNC ia_open_func; - DAT_PROVIDER_INFO info; + struct dat_provider_info info; DAT_RETURN status; size_t len; @@ -653,7 +653,7 @@ dat_dbg_print(DAT_DBG_CLASS_CONSUMER_API, "close function for IA %s failed\n", ia_name); } else { - DAT_PROVIDER_INFO info; + struct dat_provider_info info; size_t len; len = strlen(ia_name); diff -Nur -X /home/tduffy/dontdiff linux-kernel/dat/dat.h linux-kernel2/dat/dat.h --- linux-kernel/dat/dat.h 2005-05-09 11:19:35.554001000 -0700 +++ linux-kernel2/dat/dat.h 2005-05-09 11:24:22.839026000 -0700 @@ -947,12 +947,12 @@ /* Provider/registration info */ -typedef struct dat_provider_info { +struct dat_provider_info { char ia_name[DAT_NAME_MAX_LENGTH]; u32 dat_version_major; u32 dat_version_minor; DAT_BOOLEAN is_thread_safe; -} DAT_PROVIDER_INFO; +}; typedef struct dat_evd_param { DAT_IA_HANDLE ia_handle; @@ -1365,10 +1365,10 @@ */ extern DAT_RETURN dat_registry_add_provider( - const struct dat_provider *, const DAT_PROVIDER_INFO *); + const struct dat_provider *, const struct dat_provider_info *); extern DAT_RETURN dat_registry_remove_provider( - const struct dat_provider *, const DAT_PROVIDER_INFO *); + const struct dat_provider *, const struct dat_provider_info *); /* * DAT registry functions for consumers @@ -1387,7 +1387,7 @@ extern DAT_RETURN dat_strerror(DAT_RETURN, const char **, const char **); extern DAT_RETURN dat_registry_list_providers( - DAT_COUNT, DAT_COUNT *, DAT_PROVIDER_INFO * dat_provider_list[]); + DAT_COUNT, DAT_COUNT *, struct dat_provider_info *dat_provider_list[]); /* * inline functions for consumers diff -Nur -X /home/tduffy/dontdiff linux-kernel/dat/dictionary.c linux-kernel2/dat/dictionary.c --- linux-kernel/dat/dictionary.c 2005-05-06 16:16:19.665002000 -0700 +++ linux-kernel2/dat/dictionary.c 2005-05-09 11:25:49.359020000 -0700 @@ -40,7 +40,7 @@ *********************************************************************/ typedef struct DAT_DICTIONARY_NODE { - DAT_PROVIDER_INFO key; + struct dat_provider_info key; DAT_DICTIONARY_DATA data; struct DAT_DICTIONARY_NODE *prev; struct DAT_DICTIONARY_NODE *next; @@ -58,11 +58,13 @@ * * *********************************************************************/ -static DAT_RETURN dat_dictionary_key_dup(const DAT_PROVIDER_INFO * old_key, - DAT_PROVIDER_INFO * new_key); - -static DAT_BOOLEAN dat_dictionary_key_is_equal(const DAT_PROVIDER_INFO * key_a, - const DAT_PROVIDER_INFO * key_b); +static DAT_RETURN +dat_dictionary_key_dup(const struct dat_provider_info *old_key, + struct dat_provider_info *new_key); + +static DAT_BOOLEAN +dat_dictionary_key_is_equal(const struct dat_provider_info *key_a, + const struct dat_provider_info *key_b); /********************************************************************* * * @@ -215,7 +217,7 @@ DAT_RETURN dat_dictionary_insert(DAT_DICTIONARY * p_dictionary, DAT_DICTIONARY_ENTRY entry, - const DAT_PROVIDER_INFO * key, + const struct dat_provider_info *key, DAT_DICTIONARY_DATA data) { DAT_RETURN status; @@ -258,7 +260,7 @@ ***********************************************************************/ DAT_RETURN dat_dictionary_search(DAT_DICTIONARY * p_dictionary, - const DAT_PROVIDER_INFO * key, + const struct dat_provider_info *key, DAT_DICTIONARY_DATA * p_data) { DAT_DICTIONARY_NODE *cur_node; @@ -323,7 +325,7 @@ DAT_RETURN dat_dictionary_remove(DAT_DICTIONARY * p_dictionary, DAT_DICTIONARY_ENTRY * p_entry, - const DAT_PROVIDER_INFO * key, + const struct dat_provider_info *key, DAT_DICTIONARY_DATA * p_data) { DAT_DICTIONARY_NODE *cur_node, *prev_node, *next_node; @@ -372,8 +374,8 @@ * Function: dat_dictionary_key_create ***********************************************************************/ -DAT_RETURN dat_dictionary_key_dup(const DAT_PROVIDER_INFO * old_key, - DAT_PROVIDER_INFO * new_key) +DAT_RETURN dat_dictionary_key_dup(const struct dat_provider_info *old_key, + struct dat_provider_info *new_key) { BUG_ON(NULL == old_key); BUG_ON(NULL == new_key); @@ -390,8 +392,8 @@ * Function: dat_dictionary_key_is_equal ***********************************************************************/ -DAT_BOOLEAN dat_dictionary_key_is_equal(const DAT_PROVIDER_INFO * key_a, - const DAT_PROVIDER_INFO * key_b) +DAT_BOOLEAN dat_dictionary_key_is_equal(const struct dat_provider_info *key_a, + const struct dat_provider_info *key_b) { BUG_ON(NULL == key_a); BUG_ON(NULL == key_b); diff -Nur -X /home/tduffy/dontdiff linux-kernel/dat/dictionary.h linux-kernel2/dat/dictionary.h --- linux-kernel/dat/dictionary.h 2005-05-09 13:44:25.045967000 -0700 +++ linux-kernel2/dat/dictionary.h 2005-05-09 13:44:39.482982000 -0700 @@ -65,11 +65,11 @@ extern DAT_RETURN dat_dictionary_insert(DAT_DICTIONARY * p_dictionary, DAT_DICTIONARY_ENTRY entry, - const DAT_PROVIDER_INFO * key, + const struct dat_provider_info *key, DAT_DICTIONARY_DATA data); extern DAT_RETURN dat_dictionary_search(DAT_DICTIONARY * p_dictionary, - const DAT_PROVIDER_INFO * key, + const struct dat_provider_info *key, DAT_DICTIONARY_DATA * p_data); extern DAT_RETURN dat_dictionary_enumerate(DAT_DICTIONARY * p_dictionary, @@ -78,7 +78,7 @@ extern DAT_RETURN dat_dictionary_remove(DAT_DICTIONARY * p_dictionary, DAT_DICTIONARY_ENTRY * p_entry, - const DAT_PROVIDER_INFO * key, + const struct dat_provider_info *key, DAT_DICTIONARY_DATA * p_data); #endif /* DICTIONARY_H */ diff -Nur -X /home/tduffy/dontdiff linux-kernel/dat/dr.c linux-kernel2/dat/dr.c --- linux-kernel/dat/dr.c 2005-05-09 13:44:24.969974000 -0700 +++ linux-kernel2/dat/dr.c 2005-05-09 13:45:51.683961000 -0700 @@ -98,7 +98,7 @@ * Function: dat_dr_insert ************************************************************************/ -DAT_RETURN dat_dr_insert(const DAT_PROVIDER_INFO *info, +DAT_RETURN dat_dr_insert(const struct dat_provider_info *info, const DAT_IA_OPEN_FUNC ia_open_func) { DAT_RETURN status; @@ -144,7 +144,7 @@ * Function: dat_dr_remove ************************************************************************/ -DAT_RETURN dat_dr_remove(const DAT_PROVIDER_INFO *info) +DAT_RETURN dat_dr_remove(const struct dat_provider_info *info) { DAT_DR_ENTRY *data; DAT_DICTIONARY_ENTRY dict_entry; @@ -184,7 +184,7 @@ * Function: dat_dr_provider_open ************************************************************************/ -DAT_RETURN dat_dr_provider_open(const DAT_PROVIDER_INFO *info, +DAT_RETURN dat_dr_provider_open(const struct dat_provider_info *info, DAT_IA_OPEN_FUNC *p_ia_open_func) { DAT_RETURN status; @@ -209,7 +209,7 @@ * Function: dat_dr_provider_close ************************************************************************/ -DAT_RETURN dat_dr_provider_close(const DAT_PROVIDER_INFO * info) +DAT_RETURN dat_dr_provider_close(const struct dat_provider_info *info) { DAT_RETURN status; DAT_DR_ENTRY *data; @@ -242,7 +242,7 @@ DAT_RETURN dat_dr_list(DAT_COUNT max_to_return, DAT_COUNT *entries_returned, - DAT_PROVIDER_INFO *(dat_provider_list[])) + struct dat_provider_info *(dat_provider_list[])) { DAT_DR_ENTRY **array; DAT_COUNT array_size; diff -Nur -X /home/tduffy/dontdiff linux-kernel/dat/dr.h linux-kernel2/dat/dr.h --- linux-kernel/dat/dr.h 2005-05-09 13:44:25.063962000 -0700 +++ linux-kernel2/dat/dr.h 2005-05-09 13:48:49.034963000 -0700 @@ -46,20 +46,20 @@ extern DAT_RETURN dat_dr_fini(void); -extern DAT_RETURN dat_dr_insert(const DAT_PROVIDER_INFO *info, +extern DAT_RETURN dat_dr_insert(const struct dat_provider_info *info, const DAT_IA_OPEN_FUNC ia_open_func); -extern DAT_RETURN dat_dr_remove(const DAT_PROVIDER_INFO *info); +extern DAT_RETURN dat_dr_remove(const struct dat_provider_info *info); -extern DAT_RETURN dat_dr_provider_open(const DAT_PROVIDER_INFO *info, +extern DAT_RETURN dat_dr_provider_open(const struct dat_provider_info *info, DAT_IA_OPEN_FUNC *p_ia_open_func); -extern DAT_RETURN dat_dr_provider_close(const DAT_PROVIDER_INFO *info); +extern DAT_RETURN dat_dr_provider_close(const struct dat_provider_info *info); extern DAT_RETURN dat_dr_size(DAT_COUNT *size); extern DAT_RETURN dat_dr_list(DAT_COUNT max_to_return, DAT_COUNT *entries_returned, - DAT_PROVIDER_INFO *(dat_provider_list[])); + struct dat_provider_info *(dat_provider_list[])); #endif /* DR_H */ diff -Nur -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_module.c linux-kernel2/dat-provider/dapl_module.c --- linux-kernel/dat-provider/dapl_module.c 2005-05-06 16:39:48.195002000 -0700 +++ linux-kernel2/dat-provider/dapl_module.c 2005-05-09 11:27:27.003079000 -0700 @@ -123,7 +123,7 @@ * This function is called by the registry to initialize a provider * */ -void DAT_PROVIDER_INIT_FUNC_NAME(const DAT_PROVIDER_INFO * provider_info, +void DAT_PROVIDER_INIT_FUNC_NAME(const struct dat_provider_info *provider_info, struct ib_device *device, u8 port) { struct dat_provider *provider; @@ -176,7 +176,7 @@ * This function is called by the registry to de-initialize a provider * */ -void DAT_PROVIDER_FINI_FUNC_NAME(const DAT_PROVIDER_INFO * provider_info) +void DAT_PROVIDER_FINI_FUNC_NAME(const struct dat_provider_info *provider_info) { struct dat_provider *provider; DAT_RETURN dat_status; @@ -223,7 +223,7 @@ void dapl_add_one(struct ib_device *device) { DAT_COUNT i; - DAT_PROVIDER_INFO provider_info; + struct dat_provider_info provider_info; int length; dapl_dbg_log(DAPL_DBG_TYPE_UTIL, "Inside dapl_add_one called by %s\n", device->name); @@ -245,7 +245,7 @@ void dapl_remove_one(struct ib_device *device) { DAT_COUNT i; - DAT_PROVIDER_INFO provider_info; + struct dat_provider_info provider_info; int length; dapl_dbg_log(DAPL_DBG_TYPE_UTIL, "Inside dapl_remove_one called by %s\n", device->name); diff -Nur -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_module.h linux-kernel2/dat-provider/dapl_module.h --- linux-kernel/dat-provider/dapl_module.h 2005-04-20 12:42:06.812006000 -0700 +++ linux-kernel2/dat-provider/dapl_module.h 2005-05-09 11:21:32.386004000 -0700 @@ -43,9 +43,9 @@ extern int g_dapl_dbg_type; extern void -DAT_PROVIDER_INIT_FUNC_NAME(const DAT_PROVIDER_INFO *, +DAT_PROVIDER_INIT_FUNC_NAME(const struct dat_provider_info *, struct ib_device *, u8 port); -extern void DAT_PROVIDER_FINI_FUNC_NAME(const DAT_PROVIDER_INFO *); +extern void DAT_PROVIDER_FINI_FUNC_NAME(const struct dat_provider_info *); #endif /* DAPL_MODULE_H */ From Tom.Duffy at Sun.COM Mon May 9 14:39:39 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 09 May 2005 14:39:39 -0700 Subject: [openib-general] [PATCH][1/26] kDAPL: Remove typedef DAT_PROVIDER_ATTR In-Reply-To: <11156747793273@sun.com> References: <11156747793273@sun.com> Message-ID: <11156747791462@sun.com> Signed-off-by: Tom Duffy Index: linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c (revision 2290) +++ linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c (working copy) @@ -46,7 +46,7 @@ int DT_queryinfo_basic (Params_t *params char *dev_name; DAT_IA_HANDLE ia_handle; DAT_IA_ATTR ia_attributes; - DAT_PROVIDER_ATTR provider_attributes; + struct dat_provider_attr provider_attributes; DAT_EVD_HANDLE evd_handle; DAT_EVD_HANDLE conn_evd_handle; DAT_EVD_HANDLE cr_evd_handle; Index: linux-kernel/test/dapltest/include/dapl_test_data.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_test_data.h (revision 2290) +++ linux-kernel/test/dapltest/include/dapl_test_data.h (working copy) @@ -81,7 +81,7 @@ typedef struct Client_Info_t Client_Info; Params_t Params; DAT_IA_ATTR ia_attr; - DAT_PROVIDER_ATTR provider_attr; + struct dat_provider_attr provider_attr; DAT_EP_ATTR ep_attr; Per_Server_Data_t *ps_ptr; Transaction_Stats_t Client_Stats; Index: linux-kernel/dat-provider/dapl_vendor.h =================================================================== --- linux-kernel/dat-provider/dapl_vendor.h (revision 2290) +++ linux-kernel/dat-provider/dapl_vendor.h (working copy) @@ -72,8 +72,8 @@ /* * Provider Versions - * Returned in DAT_PROVIDER_ATTR.provider_version_major and - * DAT_PROVIDER_ATTR.provider_version_minor + * Returned in struct dat_provider_attr.provider_version_major and + * struct dat_provider_attr.provider_version_minor */ #define VN_PROVIDER_MAJOR 1 @@ -85,7 +85,7 @@ * the vendor must indicate if they support DAT_MEM_TYPE_SHARED_VIRTUAL. * Set this value to '1' if DAT_MEM_TYPE_SHARED_VIRTUAL is supported. * - * Returned in DAT_PROVIDER_ATTR.lmr_mem_types_supported + * Returned in struct dat_provider_attr.lmr_mem_types_supported */ #define VN_MEM_SHARED_VIRTUAL_SUPPORT 1 Index: linux-kernel/dat-provider/dapl_ia_query.c =================================================================== --- linux-kernel/dat-provider/dapl_ia_query.c (revision 2290) +++ linux-kernel/dat-provider/dapl_ia_query.c (working copy) @@ -63,7 +63,7 @@ DAT_RETURN dapl_ia_query(DAT_IA_HANDLE ia_handle, DAT_EVD_HANDLE * async_evd_handle, DAT_IA_ATTR * ia_attr, - DAT_PROVIDER_ATTR * provider_attr) + struct dat_provider_attr *provider_attr) { DAPL_IA *ia_ptr; DAT_RETURN dat_status; Index: linux-kernel/dat-provider/dapl.h =================================================================== --- linux-kernel/dat-provider/dapl.h (revision 2290) +++ linux-kernel/dat-provider/dapl.h (working copy) @@ -487,7 +487,7 @@ extern DAT_RETURN dapl_ia_close(DAT_IA_H extern DAT_RETURN dapl_ia_query(DAT_IA_HANDLE, DAT_EVD_HANDLE *, DAT_IA_ATTR *, - DAT_PROVIDER_ATTR *); + struct dat_provider_attr *); /* helper functions */ Index: linux-kernel/dat/api.c =================================================================== --- linux-kernel/dat/api.c (revision 2290) +++ linux-kernel/dat/api.c (working copy) @@ -635,7 +635,7 @@ EXPORT_SYMBOL(dat_ia_openv); DAT_RETURN dat_ia_close(DAT_IA_HANDLE ia_handle, DAT_CLOSE_FLAGS flags) { struct dat_provider *provider; - DAT_PROVIDER_ATTR provider_attr; + struct dat_provider_attr provider_attr; DAT_RETURN status; const char *ia_name; Index: linux-kernel/dat/dat.h =================================================================== --- linux-kernel/dat/dat.h (revision 2290) +++ linux-kernel/dat/dat.h (working copy) @@ -1070,7 +1070,7 @@ typedef enum dat_ep_creator_for_psp { * that Consumer passes. */ -typedef struct dat_provider_attr { +struct dat_provider_attr { char provider_name[DAT_NAME_MAX_LENGTH]; u32 provider_version_major; u32 provider_version_minor; @@ -1097,7 +1097,7 @@ typedef struct dat_provider_attr { DAT_BOOLEAN rdma_write_for_rdma_read_req; DAT_COUNT num_provider_specific_attr; DAT_NAMED_ATTR *provider_specific_attr; -} DAT_PROVIDER_ATTR; +}; /*************************************************************** @@ -1115,8 +1115,9 @@ typedef DAT_RETURN (*DAT_IA_OPENV_FUNC) typedef DAT_RETURN (*DAT_IA_CLOSE_FUNC) (DAT_IA_HANDLE, DAT_CLOSE_FLAGS); -typedef DAT_RETURN (*DAT_IA_QUERY_FUNC) ( - DAT_IA_HANDLE, DAT_EVD_HANDLE *, DAT_IA_ATTR *, DAT_PROVIDER_ATTR *); +typedef DAT_RETURN (*DAT_IA_QUERY_FUNC) (DAT_IA_HANDLE, DAT_EVD_HANDLE *, + DAT_IA_ATTR *, + struct dat_provider_attr *); typedef DAT_RETURN (*DAT_SET_CONSUMER_CONTEXT_FUNC) (DAT_HANDLE, DAT_CONTEXT); @@ -1431,7 +1432,7 @@ static inline DAT_RETURN dat_ia_memtype_ static inline DAT_RETURN dat_ia_query(DAT_IA_HANDLE ia, DAT_EVD_HANDLE * async_evd, DAT_IA_ATTR * ia_attr, - DAT_PROVIDER_ATTR * provider_attr) + struct dat_provider_attr *provider_attr) { return DAT_CALL_PROVIDER_FUNC( ia_query_func, ia, async_evd, ia_attr, provider_attr); From Tom.Duffy at Sun.COM Mon May 9 14:39:40 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 09 May 2005 14:39:40 -0700 Subject: [openib-general] [PATCH][3/26] kDAPL: Remove typedef DAT_LMR_TRIPLET In-Reply-To: <1115674780270@sun.com> References: <11156747793273@sun.com> <11156747791462@sun.com> <1115674780270@sun.com> Message-ID: <11156747803410@sun.com> Signed-off-by: Tom Duffy diff -ur -X /home/tduffy/dontdiff linux-kernel2/dat/dat.h linux-kernel3/dat/dat.h --- linux-kernel2/dat/dat.h 2005-05-09 11:24:22.839026000 -0700 +++ linux-kernel3/dat/dat.h 2005-05-09 11:36:42.184000000 -0700 @@ -524,12 +524,12 @@ * dat_lmr_triplet are irrelevant and can be invalid. */ -typedef struct dat_lmr_triplet { +struct dat_lmr_triplet { DAT_LMR_CONTEXT lmr_context; u32 pad; /* align structure on 64-bit boundry */ DAT_VADDR virtual_address; DAT_VLEN segment_length; -} DAT_LMR_TRIPLET; +}; typedef struct dat_rmr_triplet { DAT_RMR_CONTEXT rmr_context; @@ -566,7 +566,7 @@ typedef struct dat_rmr_param { DAT_IA_HANDLE ia_handle; DAT_PZ_HANDLE pz_handle; - DAT_LMR_TRIPLET lmr_triplet; + struct dat_lmr_triplet lmr_triplet; DAT_MEM_PRIV_FLAGS mem_priv; DAT_RMR_CONTEXT rmr_context; } DAT_RMR_PARAM; @@ -1168,19 +1168,19 @@ typedef DAT_RETURN (*DAT_EP_DISCONNECT_FUNC) (DAT_EP_HANDLE, DAT_CLOSE_FLAGS); typedef DAT_RETURN (*DAT_EP_POST_SEND_FUNC) ( - DAT_EP_HANDLE, DAT_COUNT, DAT_LMR_TRIPLET *, DAT_DTO_COOKIE, + DAT_EP_HANDLE, DAT_COUNT, struct dat_lmr_triplet *, DAT_DTO_COOKIE, DAT_COMPLETION_FLAGS); typedef DAT_RETURN (*DAT_EP_POST_RECV_FUNC) ( - DAT_EP_HANDLE, DAT_COUNT, DAT_LMR_TRIPLET *, DAT_DTO_COOKIE, + DAT_EP_HANDLE, DAT_COUNT, struct dat_lmr_triplet *, DAT_DTO_COOKIE, DAT_COMPLETION_FLAGS); typedef DAT_RETURN (*DAT_EP_POST_RDMA_READ_FUNC) ( - DAT_EP_HANDLE, DAT_COUNT, DAT_LMR_TRIPLET *, DAT_DTO_COOKIE, + DAT_EP_HANDLE, DAT_COUNT, struct dat_lmr_triplet *, DAT_DTO_COOKIE, const DAT_RMR_TRIPLET *, DAT_COMPLETION_FLAGS); typedef DAT_RETURN (*DAT_EP_POST_RDMA_WRITE_FUNC) ( - DAT_EP_HANDLE, DAT_COUNT, DAT_LMR_TRIPLET *, DAT_DTO_COOKIE, + DAT_EP_HANDLE, DAT_COUNT, struct dat_lmr_triplet *, DAT_DTO_COOKIE, const DAT_RMR_TRIPLET *, DAT_COMPLETION_FLAGS); typedef DAT_RETURN (*DAT_EP_GET_STATUS_FUNC) ( @@ -1201,17 +1201,17 @@ typedef DAT_RETURN (*DAT_LMR_QUERY_FUNC) (DAT_LMR_HANDLE, DAT_LMR_PARAM *); typedef DAT_RETURN (*DAT_LMR_SYNC_RDMA_READ_FUNC) ( - DAT_IA_HANDLE, const DAT_LMR_TRIPLET *, DAT_VLEN); + DAT_IA_HANDLE, const struct dat_lmr_triplet *, DAT_VLEN); typedef DAT_RETURN (*DAT_LMR_SYNC_RDMA_WRITE_FUNC) ( - DAT_IA_HANDLE, const DAT_LMR_TRIPLET *, DAT_VLEN); + DAT_IA_HANDLE, const struct dat_lmr_triplet *, DAT_VLEN); typedef DAT_RETURN (*DAT_RMR_CREATE_FUNC) (DAT_PZ_HANDLE, DAT_RMR_HANDLE *); typedef DAT_RETURN (*DAT_RMR_QUERY_FUNC) (DAT_RMR_HANDLE, DAT_RMR_PARAM *); typedef DAT_RETURN (*DAT_RMR_BIND_FUNC) ( - DAT_RMR_HANDLE, const DAT_LMR_TRIPLET *, DAT_MEM_PRIV_FLAGS, + DAT_RMR_HANDLE, const struct dat_lmr_triplet *, DAT_MEM_PRIV_FLAGS, DAT_EP_HANDLE, DAT_RMR_COOKIE, DAT_COMPLETION_FLAGS, DAT_RMR_CONTEXT *); typedef DAT_RETURN (*DAT_RMR_FREE_FUNC) (DAT_RMR_HANDLE); @@ -1254,7 +1254,7 @@ typedef DAT_RETURN (*DAT_SRQ_RESIZE_FUNC) (DAT_SRQ_HANDLE, DAT_COUNT); typedef DAT_RETURN (*DAT_SRQ_POST_RECV_FUNC) ( - DAT_SRQ_HANDLE, DAT_COUNT, DAT_LMR_TRIPLET *, DAT_DTO_COOKIE); + DAT_SRQ_HANDLE, DAT_COUNT, struct dat_lmr_triplet *, DAT_DTO_COOKIE); typedef DAT_RETURN (*DAT_LMR_KCREATE_FUNC) ( DAT_IA_HANDLE, DAT_MEM_TYPE, DAT_REGION_DESCRIPTION, DAT_VLEN, @@ -1591,7 +1591,7 @@ static inline DAT_RETURN dat_ep_post_rdma_read(DAT_EP_HANDLE ep, DAT_COUNT size, - DAT_LMR_TRIPLET * local_iov, + struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, const DAT_RMR_TRIPLET * remote_iov, DAT_COMPLETION_FLAGS flags) @@ -1602,7 +1602,7 @@ static inline DAT_RETURN dat_ep_post_rdma_write(DAT_EP_HANDLE ep, DAT_COUNT size, - DAT_LMR_TRIPLET * local_iov, + struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, const DAT_RMR_TRIPLET * remote_iov, DAT_COMPLETION_FLAGS flags) @@ -1613,7 +1613,7 @@ static inline DAT_RETURN dat_ep_post_recv(DAT_EP_HANDLE ep, DAT_COUNT size, - DAT_LMR_TRIPLET * local_iov, + struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, DAT_COMPLETION_FLAGS flags) { @@ -1623,7 +1623,7 @@ static inline DAT_RETURN dat_ep_post_send(DAT_EP_HANDLE ep, DAT_COUNT size, - DAT_LMR_TRIPLET * local_iov, + struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, DAT_COMPLETION_FLAGS flags) { @@ -1668,7 +1668,7 @@ } static inline DAT_RETURN dat_lmr_sync_rdma_read(DAT_IA_HANDLE ia, - const DAT_LMR_TRIPLET * iovs, + const struct dat_lmr_triplet *iovs, DAT_VLEN num_iovs) { return DAT_CALL_PROVIDER_FUNC( @@ -1676,7 +1676,7 @@ } static inline DAT_RETURN dat_lmr_sync_rdma_write(DAT_IA_HANDLE ia, - const DAT_LMR_TRIPLET * iovs, + const struct dat_lmr_triplet *iovs, DAT_VLEN num_iovs) { return DAT_CALL_PROVIDER_FUNC( @@ -1694,7 +1694,7 @@ } static inline DAT_RETURN dat_rmr_bind(DAT_RMR_HANDLE rmr, - const DAT_LMR_TRIPLET * iov, + const struct dat_lmr_triplet *iov, DAT_MEM_PRIV_FLAGS mem_flags, DAT_EP_HANDLE ep, DAT_RMR_COOKIE cookie, @@ -1790,7 +1790,7 @@ static inline DAT_RETURN dat_srq_post_recv(DAT_SRQ_HANDLE srq, DAT_COUNT num_iovs, - DAT_LMR_TRIPLET * iovs, + struct dat_lmr_triplet *iovs, DAT_DTO_COOKIE cookie) { return DAT_CALL_PROVIDER_FUNC( diff -ur -X /home/tduffy/dontdiff linux-kernel2/dat-provider/dapl_ep_post_rdma_read.c linux-kernel3/dat-provider/dapl_ep_post_rdma_read.c --- linux-kernel2/dat-provider/dapl_ep_post_rdma_read.c 2005-04-20 12:42:08.299025000 -0700 +++ linux-kernel3/dat-provider/dapl_ep_post_rdma_read.c 2005-05-09 11:37:00.776002000 -0700 @@ -69,7 +69,7 @@ DAT_RETURN dapl_ep_post_rdma_read(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, - DAT_LMR_TRIPLET * local_iov, + struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const DAT_RMR_TRIPLET * remote_iov, DAT_COMPLETION_FLAGS completion_flags) diff -ur -X /home/tduffy/dontdiff linux-kernel2/dat-provider/dapl_ep_post_rdma_write.c linux-kernel3/dat-provider/dapl_ep_post_rdma_write.c --- linux-kernel2/dat-provider/dapl_ep_post_rdma_write.c 2005-04-20 12:42:06.702005000 -0700 +++ linux-kernel3/dat-provider/dapl_ep_post_rdma_write.c 2005-05-09 11:37:10.118025000 -0700 @@ -69,7 +69,7 @@ DAT_RETURN dapl_ep_post_rdma_write(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, - DAT_LMR_TRIPLET * local_iov, + struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const DAT_RMR_TRIPLET * remote_iov, DAT_COMPLETION_FLAGS completion_flags) diff -ur -X /home/tduffy/dontdiff linux-kernel2/dat-provider/dapl_ep_post_recv.c linux-kernel3/dat-provider/dapl_ep_post_recv.c --- linux-kernel2/dat-provider/dapl_ep_post_recv.c 2005-04-27 11:38:35.803083000 -0700 +++ linux-kernel3/dat-provider/dapl_ep_post_recv.c 2005-05-09 11:37:22.915007000 -0700 @@ -69,7 +69,7 @@ DAT_RETURN dapl_ep_post_recv(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, - DAT_LMR_TRIPLET * local_iov, + struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, DAT_COMPLETION_FLAGS completion_flags) { diff -ur -X /home/tduffy/dontdiff linux-kernel2/dat-provider/dapl_ep_post_send.c linux-kernel3/dat-provider/dapl_ep_post_send.c --- linux-kernel2/dat-provider/dapl_ep_post_send.c 2005-04-20 12:42:07.360001000 -0700 +++ linux-kernel3/dat-provider/dapl_ep_post_send.c 2005-05-09 11:37:33.432060000 -0700 @@ -66,7 +66,7 @@ DAT_RETURN dapl_ep_post_send(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, - DAT_LMR_TRIPLET * local_iov, + struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, DAT_COMPLETION_FLAGS completion_flags) { diff -ur -X /home/tduffy/dontdiff linux-kernel2/dat-provider/dapl_ep_util.c linux-kernel3/dat-provider/dapl_ep_util.c --- linux-kernel2/dat-provider/dapl_ep_util.c 2005-05-04 08:19:32.791989000 -0700 +++ linux-kernel3/dat-provider/dapl_ep_util.c 2005-05-09 11:37:43.486003000 -0700 @@ -287,7 +287,7 @@ DAT_RETURN dapl_ep_post_send_req(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, - DAT_LMR_TRIPLET * local_iov, + struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const DAT_RMR_TRIPLET * remote_iov, DAT_COMPLETION_FLAGS completion_flags, diff -ur -X /home/tduffy/dontdiff linux-kernel2/dat-provider/dapl_ep_util.h linux-kernel3/dat-provider/dapl_ep_util.h --- linux-kernel2/dat-provider/dapl_ep_util.h 2005-04-20 12:42:08.254017000 -0700 +++ linux-kernel3/dat-provider/dapl_ep_util.h 2005-05-09 11:37:53.155002000 -0700 @@ -55,7 +55,7 @@ extern DAT_RETURN dapl_ep_post_send_req(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, - DAT_LMR_TRIPLET * local_iov, + struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const DAT_RMR_TRIPLET * remote_iov, DAT_COMPLETION_FLAGS completion_flags, diff -ur -X /home/tduffy/dontdiff linux-kernel2/dat-provider/dapl.h linux-kernel3/dat-provider/dapl.h --- linux-kernel2/dat-provider/dapl.h 2005-05-09 11:15:26.428002000 -0700 +++ linux-kernel3/dat-provider/dapl.h 2005-05-09 11:34:28.653042000 -0700 @@ -591,26 +591,26 @@ extern DAT_RETURN dapl_ep_post_send(DAT_EP_HANDLE, /* ep_handle */ DAT_COUNT, /* num_segments */ - DAT_LMR_TRIPLET *, /* local_iov */ + struct dat_lmr_triplet *, /* local_iov */ DAT_DTO_COOKIE, /* user_cookie */ DAT_COMPLETION_FLAGS); /* completion_flags */ extern DAT_RETURN dapl_ep_post_recv(DAT_EP_HANDLE, /* ep_handle */ DAT_COUNT, /* num_segments */ - DAT_LMR_TRIPLET *, /* local_iov */ + struct dat_lmr_triplet *, /* local_iov */ DAT_DTO_COOKIE, /* user_cookie */ DAT_COMPLETION_FLAGS); /* completion_flags */ extern DAT_RETURN dapl_ep_post_rdma_read(DAT_EP_HANDLE, /* ep_handle */ DAT_COUNT, /* num_segments */ - DAT_LMR_TRIPLET *, /* local_iov */ + struct dat_lmr_triplet *, /* local_iov */ DAT_DTO_COOKIE, /* user_cookie */ const DAT_RMR_TRIPLET *, /* remote_iov */ DAT_COMPLETION_FLAGS); /* completion_flags */ extern DAT_RETURN dapl_ep_post_rdma_write(DAT_EP_HANDLE, /* ep_handle */ DAT_COUNT, /* num_segments */ - DAT_LMR_TRIPLET *, /* local_iov */ + struct dat_lmr_triplet *, /* local_iov */ DAT_DTO_COOKIE, /* user_cookie */ const DAT_RMR_TRIPLET *, /* remote_iov */ DAT_COMPLETION_FLAGS); /* completion_flags */ @@ -661,11 +661,11 @@ extern DAT_RETURN dapl_lmr_free(DAT_LMR_HANDLE); extern DAT_RETURN dapl_lmr_sync_rdma_read(DAT_IA_HANDLE, /* ia_handle */ - const DAT_LMR_TRIPLET *, /* local_segments */ + const struct dat_lmr_triplet *, /* local_segments */ DAT_VLEN); /* num_segments */ extern DAT_RETURN dapl_lmr_sync_rdma_write(DAT_IA_HANDLE, /* ia_handle */ - const DAT_LMR_TRIPLET *, /* local_segments */ + const struct dat_lmr_triplet *, /* local_segments */ DAT_VLEN); /* num_segments */ /* RMR Functions */ @@ -677,7 +677,7 @@ DAT_RMR_PARAM *); /* rmr_args */ extern DAT_RETURN dapl_rmr_bind(DAT_RMR_HANDLE, /* rmr_handle */ - const DAT_LMR_TRIPLET *, /* lmr_triplet */ + const struct dat_lmr_triplet *, /* lmr_triplet */ DAT_MEM_PRIV_FLAGS, /* mem_priv */ DAT_EP_HANDLE, /* ep_handle */ DAT_RMR_COOKIE, /* user_cookie */ @@ -737,7 +737,7 @@ extern DAT_RETURN dapl_srq_post_recv(DAT_SRQ_HANDLE, /* srq_handle */ DAT_COUNT, /* num_segments */ - DAT_LMR_TRIPLET *, /* local_iov */ + struct dat_lmr_triplet *, /* local_iov */ DAT_DTO_COOKIE); /* user_cookie */ extern DAT_RETURN dapl_srq_query(DAT_SRQ_HANDLE, /* srq_handle */ diff -ur -X /home/tduffy/dontdiff linux-kernel2/dat-provider/dapl_lmr_sync_rdma_read.c linux-kernel3/dat-provider/dapl_lmr_sync_rdma_read.c --- linux-kernel2/dat-provider/dapl_lmr_sync_rdma_read.c 2005-04-20 12:42:06.609025000 -0700 +++ linux-kernel3/dat-provider/dapl_lmr_sync_rdma_read.c 2005-05-09 11:38:23.940013000 -0700 @@ -58,7 +58,7 @@ */ DAT_RETURN dapl_lmr_sync_rdma_read(DAT_IA_HANDLE ia_handle, - const DAT_LMR_TRIPLET * local_segments, + const struct dat_lmr_triplet *local_segments, DAT_VLEN num_segments) { DAPL_IA *ia_ptr; diff -ur -X /home/tduffy/dontdiff linux-kernel2/dat-provider/dapl_lmr_sync_rdma_write.c linux-kernel3/dat-provider/dapl_lmr_sync_rdma_write.c --- linux-kernel2/dat-provider/dapl_lmr_sync_rdma_write.c 2005-04-20 12:42:08.144018000 -0700 +++ linux-kernel3/dat-provider/dapl_lmr_sync_rdma_write.c 2005-05-09 11:38:36.655003000 -0700 @@ -43,7 +43,7 @@ */ DAT_RETURN dapl_lmr_sync_rdma_write(DAT_IA_HANDLE ia_handle, - const DAT_LMR_TRIPLET * local_segments, + const struct dat_lmr_triplet *local_segments, DAT_VLEN num_segments) { DAPL_IA *ia_ptr; diff -ur -X /home/tduffy/dontdiff linux-kernel2/dat-provider/dapl_openib_dto.h linux-kernel3/dat-provider/dapl_openib_dto.h --- linux-kernel2/dat-provider/dapl_openib_dto.h 2005-05-09 11:07:01.719014000 -0700 +++ linux-kernel3/dat-provider/dapl_openib_dto.h 2005-05-09 11:38:52.342013000 -0700 @@ -53,7 +53,7 @@ static __inline__ DAT_RETURN dapl_ib_post_recv(DAPL_EP * ep_ptr, DAPL_COOKIE * cookie, - DAT_COUNT num_segments, DAT_LMR_TRIPLET * local_iov) + DAT_COUNT num_segments, struct dat_lmr_triplet *local_iov) { struct ib_recv_wr rr_desc = { NULL }; struct ib_recv_wr *rr_desc_fail; @@ -105,7 +105,7 @@ int op_type, DAPL_COOKIE * cookie, DAT_COUNT num_segments, - DAT_LMR_TRIPLET * local_iov, + struct dat_lmr_triplet *local_iov, const DAT_RMR_TRIPLET * remote_iov, DAT_COMPLETION_FLAGS completion_flags) { diff -ur -X /home/tduffy/dontdiff linux-kernel2/dat-provider/dapl_rmr_bind.c linux-kernel3/dat-provider/dapl_rmr_bind.c --- linux-kernel2/dat-provider/dapl_rmr_bind.c 2005-04-27 11:38:35.313024000 -0700 +++ linux-kernel3/dat-provider/dapl_rmr_bind.c 2005-05-09 11:39:36.436004000 -0700 @@ -48,7 +48,7 @@ static __inline__ DAT_RETURN dapl_rmr_bind_fuse(DAPL_RMR * rmr, - const DAT_LMR_TRIPLET * lmr_triplet, + const struct dat_lmr_triplet *lmr_triplet, DAT_MEM_PRIV_FLAGS mem_priv, DAPL_EP * ep_ptr, DAT_RMR_COOKIE user_cookie, @@ -69,7 +69,7 @@ DAT_RETURN dapl_rmr_bind_fuse(DAPL_RMR * rmr, - const DAT_LMR_TRIPLET * lmr_triplet, + const struct dat_lmr_triplet *lmr_triplet, DAT_MEM_PRIV_FLAGS mem_priv, DAPL_EP * ep_ptr, DAT_RMR_COOKIE user_cookie, @@ -291,7 +291,7 @@ */ DAT_RETURN dapl_rmr_bind(DAT_RMR_HANDLE rmr_handle, - const DAT_LMR_TRIPLET * lmr_triplet, + const struct dat_lmr_triplet *lmr_triplet, DAT_MEM_PRIV_FLAGS mem_priv, DAT_EP_HANDLE ep_handle, DAT_RMR_COOKIE user_cookie, diff -ur -X /home/tduffy/dontdiff linux-kernel2/dat-provider/dapl_srq_post_recv.c linux-kernel3/dat-provider/dapl_srq_post_recv.c --- linux-kernel2/dat-provider/dapl_srq_post_recv.c 2005-04-27 11:38:35.467007000 -0700 +++ linux-kernel3/dat-provider/dapl_srq_post_recv.c 2005-05-09 11:39:54.162012000 -0700 @@ -70,7 +70,8 @@ DAT_RETURN dapl_srq_post_recv(DAT_SRQ_HANDLE srq_handle, DAT_COUNT num_segments, - DAT_LMR_TRIPLET * local_iov, DAT_DTO_COOKIE user_cookie) + struct dat_lmr_triplet *local_iov, + DAT_DTO_COOKIE user_cookie) { DAPL_SRQ *srq_ptr; DAPL_COOKIE *cookie; diff -ur -X /home/tduffy/dontdiff linux-kernel2/test/dapltest/include/dapl_bpool.h linux-kernel3/test/dapltest/include/dapl_bpool.h --- linux-kernel2/test/dapltest/include/dapl_bpool.h 2005-05-05 13:50:25.453960000 -0700 +++ linux-kernel3/test/dapltest/include/dapl_bpool.h 2005-05-09 11:42:16.577006000 -0700 @@ -35,25 +35,24 @@ * struct Bpool */ -struct Bpool_tag -{ - unsigned char *alloc_ptr; - uint32_t alloc_size; - DAT_PZ_HANDLE pz_handle; - DAT_COUNT seg_size; - DAT_COUNT num_segs; /* num segments */ - unsigned char *buffer_start; /* Start of buffer area */ - DAT_VLEN buffer_size; /* Size of data buffer (rounded) */ - DAT_VADDR reg_addr; /* start of registered area */ - DAT_VLEN reg_size; /* size of registered area */ - DAT_EP_HANDLE ep_handle; /* EP area is registered to */ - DAT_LMR_HANDLE lmr_handle; /* local access */ - DAT_LMR_CONTEXT lmr_context; - DAT_LMR_TRIPLET*tripl_start; /* local IOV */ - DAT_BOOLEAN enable_rdma_write; /* remote access */ - DAT_BOOLEAN enable_rdma_read; - DAT_RMR_HANDLE rmr_handle; - DAT_RMR_CONTEXT rmr_context; - DAT_EVD_HANDLE rmr_evd_handle; +struct Bpool_tag { + unsigned char *alloc_ptr; + uint32_t alloc_size; + DAT_PZ_HANDLE pz_handle; + DAT_COUNT seg_size; + DAT_COUNT num_segs; /* num segments */ + unsigned char *buffer_start; /* Start of buffer area */ + DAT_VLEN buffer_size; /* Size of data buffer (rounded) */ + DAT_VADDR reg_addr; /* start of registered area */ + DAT_VLEN reg_size; /* size of registered area */ + DAT_EP_HANDLE ep_handle; /* EP area is registered to */ + DAT_LMR_HANDLE lmr_handle; /* local access */ + DAT_LMR_CONTEXT lmr_context; + struct dat_lmr_triplet *tripl_start; /* local IOV */ + DAT_BOOLEAN enable_rdma_write; /* remote access */ + DAT_BOOLEAN enable_rdma_read; + DAT_RMR_HANDLE rmr_handle; + DAT_RMR_CONTEXT rmr_context; + DAT_EVD_HANDLE rmr_evd_handle; }; #endif diff -ur -X /home/tduffy/dontdiff linux-kernel2/test/dapltest/include/dapl_proto.h linux-kernel3/test/dapltest/include/dapl_proto.h --- linux-kernel2/test/dapltest/include/dapl_proto.h 2005-05-06 16:22:25.178003000 -0700 +++ linux-kernel3/test/dapltest/include/dapl_proto.h 2005-05-09 11:34:28.736009000 -0700 @@ -98,7 +98,7 @@ unsigned char *DT_Bpool_GetBuffer (Bpool * bpool_ptr, int index); DAT_VADDR DT_Bpool_GetRegBuff (Bpool * bpool_ptr, int index); DAT_COUNT DT_Bpool_GetBuffSize (Bpool * bpool_ptr, int index); -DAT_LMR_TRIPLET *DT_Bpool_GetIOV (Bpool * bpool_ptr, int index); +struct dat_lmr_triplet *DT_Bpool_GetIOV (Bpool * bpool_ptr, int index); DAT_LMR_CONTEXT DT_Bpool_GetLMR (Bpool * bpool_ptr, int index); DAT_RMR_CONTEXT DT_Bpool_GetRMR (Bpool * bpool_ptr, int index); diff -ur -X /home/tduffy/dontdiff linux-kernel2/test/dapltest/test/dapl_bpool.c linux-kernel3/test/dapltest/test/dapl_bpool.c --- linux-kernel2/test/dapltest/test/dapl_bpool.c 2005-05-05 13:45:32.607002000 -0700 +++ linux-kernel3/test/dapltest/test/dapl_bpool.c 2005-05-09 11:34:28.741036000 -0700 @@ -120,7 +120,7 @@ goto err; } - bpool_size = sizeof (Bpool) + num_segs * sizeof (DAT_LMR_TRIPLET); + bpool_size = sizeof (Bpool) + num_segs * sizeof (struct dat_lmr_triplet); bpool_ptr = (Bpool *) DT_MemListAlloc (pt_ptr, "bpool", BPOOL, bpool_size); if (!bpool_ptr) @@ -136,7 +136,7 @@ bpool_ptr->ep_handle = ep_handle; bpool_ptr->buffer_size = seg_size * num_segs; bpool_ptr->buffer_start = DT_AlignPtr (alloc_ptr, alignment) + GG_ALLOC_OFFSET; - bpool_ptr->tripl_start = (DAT_LMR_TRIPLET *) (bpool_ptr + 1); + bpool_ptr->tripl_start = (struct dat_lmr_triplet *) (bpool_ptr + 1); bpool_ptr->seg_size = seg_size; bpool_ptr->enable_rdma_write = enable_rdma_write; bpool_ptr->enable_rdma_read = enable_rdma_read; @@ -207,7 +207,7 @@ #endif if (enable_rdma_write || enable_rdma_read) { - DAT_LMR_TRIPLET iov; + struct dat_lmr_triplet iov; DAT_RMR_COOKIE cookie; DAT_MEM_PRIV_FLAGS mflags; DAT_RMR_BIND_COMPLETION_EVENT_DATA rmr_stat; @@ -326,7 +326,7 @@ { if (bpool_ptr->rmr_handle) { - DAT_LMR_TRIPLET iov; + struct dat_lmr_triplet iov; DAT_RMR_COOKIE cookie; DAT_RETURN ret; @@ -399,7 +399,7 @@ } /*****************************************************************************/ -DAT_LMR_TRIPLET * +struct dat_lmr_triplet * DT_Bpool_GetIOV (Bpool * bpool_ptr, int index) { return ( bpool_ptr->tripl_start + index ); diff -ur -X /home/tduffy/dontdiff linux-kernel2/test/dapltest/test/dapl_limit.c linux-kernel3/test/dapltest/test/dapl_limit.c --- linux-kernel2/test/dapltest/test/dapl_limit.c 2005-05-05 13:45:32.614001000 -0700 +++ linux-kernel3/test/dapltest/test/dapl_limit.c 2005-05-09 11:44:35.444001000 -0700 @@ -1011,7 +1011,7 @@ * but that should be OK. */ unsigned int count = START_COUNT; - DAT_LMR_TRIPLET *hdlptr = (DAT_LMR_TRIPLET *) + struct dat_lmr_triplet *hdlptr = (struct dat_lmr_triplet *) DT_Mdep_Malloc (count * cmd->width * sizeof (*hdlptr)); /* Recv-Post Exhaustion test loop */ @@ -1042,7 +1042,7 @@ } for (i = 0; i < cmd->width; i++) { - DAT_LMR_TRIPLET *iovp = &hdlptr[w * cmd->width + i]; + struct dat_lmr_triplet *iovp = &hdlptr[w * cmd->width + i]; DAT_DTO_COOKIE cookie; iovp->virtual_address = (DAT_VADDR) (uintptr_t) diff -ur -X /home/tduffy/dontdiff linux-kernel2/test/dapltest/test/dapl_performance_client.c linux-kernel3/test/dapltest/test/dapl_performance_client.c --- linux-kernel2/test/dapltest/test/dapl_performance_client.c 2005-05-05 09:42:06.336001000 -0700 +++ linux-kernel3/test/dapltest/test/dapl_performance_client.c 2005-05-09 11:45:05.677016000 -0700 @@ -252,7 +252,7 @@ Performance_Test_t *test_ptr, Performance_Stats_t *stats ) { - DAT_LMR_TRIPLET *iov; + struct dat_lmr_triplet *iov; DAT_RMR_TRIPLET rmr_triplet; DAT_DTO_COOKIE cookie; DAT_EVENT event; diff -ur -X /home/tduffy/dontdiff linux-kernel2/test/dapltest/test/dapl_performance_util.c linux-kernel3/test/dapltest/test/dapl_performance_util.c --- linux-kernel2/test/dapltest/test/dapl_performance_util.c 2005-05-05 13:35:54.404000000 -0700 +++ linux-kernel3/test/dapltest/test/dapl_performance_util.c 2005-05-09 11:45:34.287003000 -0700 @@ -429,7 +429,7 @@ DAT_DTO_COOKIE cookie; DAT_RETURN ret; Performance_Test_Op_t *op = &ep_context->op; - DAT_LMR_TRIPLET *iov = DT_Bpool_GetIOV (op->bp, 0); + struct dat_lmr_triplet *iov = DT_Bpool_GetIOV(op->bp, 0); DAT_RMR_TRIPLET rmr_triplet; bytes = op->seg_size * op->num_segs; diff -ur -X /home/tduffy/dontdiff linux-kernel2/test/dapltest/test/dapl_test_util.c linux-kernel3/test/dapltest/test/dapl_test_util.c --- linux-kernel2/test/dapltest/test/dapl_test_util.c 2005-05-05 13:45:32.619003000 -0700 +++ linux-kernel3/test/dapltest/test/dapl_test_util.c 2005-05-09 11:45:59.822013000 -0700 @@ -166,7 +166,7 @@ int size) { unsigned char *buff = DT_Bpool_GetBuffer (bp, index); - DAT_LMR_TRIPLET *iov = DT_Bpool_GetIOV (bp, index); + struct dat_lmr_triplet *iov = DT_Bpool_GetIOV(bp, index); DAT_LMR_CONTEXT lmr_c = DT_Bpool_GetLMR (bp, index); DAT_DTO_COOKIE cookie; DAT_RETURN ret; @@ -211,7 +211,7 @@ int size) { unsigned char *buff = DT_Bpool_GetBuffer (bp, index); - DAT_LMR_TRIPLET *iov = DT_Bpool_GetIOV (bp, index); + struct dat_lmr_triplet *iov = DT_Bpool_GetIOV(bp, index); DAT_LMR_CONTEXT lmr_c = DT_Bpool_GetLMR (bp, index); DAT_DTO_COOKIE cookie; DAT_RETURN ret; diff -ur -X /home/tduffy/dontdiff linux-kernel2/test/dapltest/test/dapl_transaction_util.c linux-kernel3/test/dapltest/test/dapl_transaction_util.c --- linux-kernel2/test/dapltest/test/dapl_transaction_util.c 2005-05-05 13:45:32.630004000 -0700 +++ linux-kernel3/test/dapltest/test/dapl_transaction_util.c 2005-05-09 11:46:47.629003000 -0700 @@ -41,7 +41,7 @@ for (i = 0; i < num_eps; i++) { Transaction_Test_Op_t *op = &ep_context[i].op[op_indx]; - DAT_LMR_TRIPLET *iov = DT_Bpool_GetIOV (op->bp, 0); + struct dat_lmr_triplet *iov = DT_Bpool_GetIOV(op->bp, 0); DAT_DTO_COOKIE cookie; DAT_RETURN ret; @@ -103,7 +103,7 @@ for (i = 0; i < num_eps; i++) { Transaction_Test_Op_t *op = &ep_context[i].op[op_indx]; - DAT_LMR_TRIPLET *iov = DT_Bpool_GetIOV (op->bp, 0); + struct dat_lmr_triplet *iov = DT_Bpool_GetIOV(op->bp, 0); DAT_DTO_COOKIE cookie; DAT_RETURN ret; @@ -484,7 +484,7 @@ for (i = 0; i < num_eps; i++) { Transaction_Test_Op_t *op = &ep_context[i].op[op_indx]; - DAT_LMR_TRIPLET *iov = DT_Bpool_GetIOV (op->bp, 0); + struct dat_lmr_triplet *iov = DT_Bpool_GetIOV (op->bp, 0); DAT_DTO_COOKIE cookie; DAT_RMR_TRIPLET rmr_triplet; From Tom.Duffy at Sun.COM Mon May 9 14:39:40 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 09 May 2005 14:39:40 -0700 Subject: [openib-general] [PATCH][4/26] kDAPL: Remove typedef DAT_RMR_TRIPLET In-Reply-To: <11156747803410@sun.com> References: <11156747793273@sun.com> <11156747791462@sun.com> <1115674780270@sun.com> <11156747803410@sun.com> Message-ID: <11156747802252@sun.com> Signed-off-by: Tom Duffy diff -Nur -X /home/tduffy/dontdiff linux-kernel3/dat/dat.h linux-kernel4/dat/dat.h --- linux-kernel3/dat/dat.h 2005-05-09 11:36:42.184000000 -0700 +++ linux-kernel4/dat/dat.h 2005-05-09 11:50:52.631016000 -0700 @@ -531,12 +531,12 @@ DAT_VLEN segment_length; }; -typedef struct dat_rmr_triplet { +struct dat_rmr_triplet { DAT_RMR_CONTEXT rmr_context; u32 pad; /* align structure on 64-bit boundry */ DAT_VADDR target_address; DAT_VLEN segment_length; -} DAT_RMR_TRIPLET; +}; typedef enum dat_mem_type { DAT_MEM_TYPE_VIRTUAL = 0x00, @@ -1177,11 +1177,11 @@ typedef DAT_RETURN (*DAT_EP_POST_RDMA_READ_FUNC) ( DAT_EP_HANDLE, DAT_COUNT, struct dat_lmr_triplet *, DAT_DTO_COOKIE, - const DAT_RMR_TRIPLET *, DAT_COMPLETION_FLAGS); + const struct dat_rmr_triplet *, DAT_COMPLETION_FLAGS); typedef DAT_RETURN (*DAT_EP_POST_RDMA_WRITE_FUNC) ( DAT_EP_HANDLE, DAT_COUNT, struct dat_lmr_triplet *, DAT_DTO_COOKIE, - const DAT_RMR_TRIPLET *, DAT_COMPLETION_FLAGS); + const struct dat_rmr_triplet *, DAT_COMPLETION_FLAGS); typedef DAT_RETURN (*DAT_EP_GET_STATUS_FUNC) ( DAT_EP_HANDLE, DAT_EP_STATE *, DAT_BOOLEAN *, DAT_BOOLEAN *); @@ -1593,7 +1593,7 @@ DAT_COUNT size, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, - const DAT_RMR_TRIPLET * remote_iov, + const struct dat_rmr_triplet *remote_iov, DAT_COMPLETION_FLAGS flags) { return DAT_CALL_PROVIDER_FUNC(ep_post_rdma_read_func, ep, size, @@ -1604,7 +1604,7 @@ DAT_COUNT size, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, - const DAT_RMR_TRIPLET * remote_iov, + const struct dat_rmr_triplet *remote_iov, DAT_COMPLETION_FLAGS flags) { return DAT_CALL_PROVIDER_FUNC(ep_post_rdma_write_func, ep, size, diff -Nur -X /home/tduffy/dontdiff linux-kernel3/dat-provider/dapl_ep_post_rdma_read.c linux-kernel4/dat-provider/dapl_ep_post_rdma_read.c --- linux-kernel3/dat-provider/dapl_ep_post_rdma_read.c 2005-05-09 11:37:00.776002000 -0700 +++ linux-kernel4/dat-provider/dapl_ep_post_rdma_read.c 2005-05-09 11:51:08.142014000 -0700 @@ -71,7 +71,7 @@ DAT_COUNT num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, - const DAT_RMR_TRIPLET * remote_iov, + const struct dat_rmr_triplet *remote_iov, DAT_COMPLETION_FLAGS completion_flags) { diff -Nur -X /home/tduffy/dontdiff linux-kernel3/dat-provider/dapl_ep_post_rdma_write.c linux-kernel4/dat-provider/dapl_ep_post_rdma_write.c --- linux-kernel3/dat-provider/dapl_ep_post_rdma_write.c 2005-05-09 11:37:10.118025000 -0700 +++ linux-kernel4/dat-provider/dapl_ep_post_rdma_write.c 2005-05-09 11:51:17.541031000 -0700 @@ -71,7 +71,7 @@ DAT_COUNT num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, - const DAT_RMR_TRIPLET * remote_iov, + const struct dat_rmr_triplet *remote_iov, DAT_COMPLETION_FLAGS completion_flags) { DAT_RETURN dat_status; diff -Nur -X /home/tduffy/dontdiff linux-kernel3/dat-provider/dapl_ep_post_send.c linux-kernel4/dat-provider/dapl_ep_post_send.c --- linux-kernel3/dat-provider/dapl_ep_post_send.c 2005-05-09 11:37:33.432060000 -0700 +++ linux-kernel4/dat-provider/dapl_ep_post_send.c 2005-05-09 11:49:45.248005000 -0700 @@ -70,7 +70,7 @@ DAT_DTO_COOKIE user_cookie, DAT_COMPLETION_FLAGS completion_flags) { - DAT_RMR_TRIPLET remote_iov = { 0, 0, 0, 0 }; + struct dat_rmr_triplet remote_iov = { 0, 0, 0, 0 }; DAT_RETURN dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, diff -Nur -X /home/tduffy/dontdiff linux-kernel3/dat-provider/dapl_ep_util.c linux-kernel4/dat-provider/dapl_ep_util.c --- linux-kernel3/dat-provider/dapl_ep_util.c 2005-05-09 11:37:43.486003000 -0700 +++ linux-kernel4/dat-provider/dapl_ep_util.c 2005-05-09 11:51:34.946002000 -0700 @@ -289,7 +289,7 @@ DAT_COUNT num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, - const DAT_RMR_TRIPLET * remote_iov, + const struct dat_rmr_triplet *remote_iov, DAT_COMPLETION_FLAGS completion_flags, DAPL_DTO_TYPE dto_type, int op_type) { diff -Nur -X /home/tduffy/dontdiff linux-kernel3/dat-provider/dapl_ep_util.h linux-kernel4/dat-provider/dapl_ep_util.h --- linux-kernel3/dat-provider/dapl_ep_util.h 2005-05-09 11:37:53.155002000 -0700 +++ linux-kernel4/dat-provider/dapl_ep_util.h 2005-05-09 11:51:45.071013000 -0700 @@ -57,7 +57,7 @@ DAT_COUNT num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, - const DAT_RMR_TRIPLET * remote_iov, + const struct dat_rmr_triplet *remote_iov, DAT_COMPLETION_FLAGS completion_flags, DAPL_DTO_TYPE dto_type, int op_type); diff -Nur -X /home/tduffy/dontdiff linux-kernel3/dat-provider/dapl.h linux-kernel4/dat-provider/dapl.h --- linux-kernel3/dat-provider/dapl.h 2005-05-09 11:34:28.653042000 -0700 +++ linux-kernel4/dat-provider/dapl.h 2005-05-09 11:49:45.236005000 -0700 @@ -605,14 +605,14 @@ DAT_COUNT, /* num_segments */ struct dat_lmr_triplet *, /* local_iov */ DAT_DTO_COOKIE, /* user_cookie */ - const DAT_RMR_TRIPLET *, /* remote_iov */ + const struct dat_rmr_triplet *, /* remote_iov */ DAT_COMPLETION_FLAGS); /* completion_flags */ extern DAT_RETURN dapl_ep_post_rdma_write(DAT_EP_HANDLE, /* ep_handle */ DAT_COUNT, /* num_segments */ struct dat_lmr_triplet *, /* local_iov */ DAT_DTO_COOKIE, /* user_cookie */ - const DAT_RMR_TRIPLET *, /* remote_iov */ + const struct dat_rmr_triplet *, /* remote_iov */ DAT_COMPLETION_FLAGS); /* completion_flags */ extern DAT_RETURN dapl_ep_get_status(DAT_EP_HANDLE, /* ep_handle */ diff -Nur -X /home/tduffy/dontdiff linux-kernel3/dat-provider/dapl_openib_dto.h linux-kernel4/dat-provider/dapl_openib_dto.h --- linux-kernel3/dat-provider/dapl_openib_dto.h 2005-05-09 11:38:52.342013000 -0700 +++ linux-kernel4/dat-provider/dapl_openib_dto.h 2005-05-09 11:52:00.134000000 -0700 @@ -106,7 +106,7 @@ DAPL_COOKIE * cookie, DAT_COUNT num_segments, struct dat_lmr_triplet *local_iov, - const DAT_RMR_TRIPLET * remote_iov, + const struct dat_rmr_triplet *remote_iov, DAT_COMPLETION_FLAGS completion_flags) { struct ib_send_wr send_desc = { NULL }; diff -Nur -X /home/tduffy/dontdiff linux-kernel3/test/dapltest/test/dapl_performance_client.c linux-kernel4/test/dapltest/test/dapl_performance_client.c --- linux-kernel3/test/dapltest/test/dapl_performance_client.c 2005-05-09 11:45:05.677016000 -0700 +++ linux-kernel4/test/dapltest/test/dapl_performance_client.c 2005-05-09 11:52:15.258003000 -0700 @@ -253,7 +253,7 @@ Performance_Stats_t *stats ) { struct dat_lmr_triplet *iov; - DAT_RMR_TRIPLET rmr_triplet; + struct dat_rmr_triplet rmr_triplet; DAT_DTO_COOKIE cookie; DAT_EVENT event; DAT_RETURN ret; diff -Nur -X /home/tduffy/dontdiff linux-kernel3/test/dapltest/test/dapl_performance_util.c linux-kernel4/test/dapltest/test/dapl_performance_util.c --- linux-kernel3/test/dapltest/test/dapl_performance_util.c 2005-05-09 11:45:34.287003000 -0700 +++ linux-kernel4/test/dapltest/test/dapl_performance_util.c 2005-05-09 11:52:38.934000000 -0700 @@ -430,7 +430,7 @@ DAT_RETURN ret; Performance_Test_Op_t *op = &ep_context->op; struct dat_lmr_triplet *iov = DT_Bpool_GetIOV(op->bp, 0); - DAT_RMR_TRIPLET rmr_triplet; + struct dat_rmr_triplet rmr_triplet; bytes = op->seg_size * op->num_segs; diff -Nur -X /home/tduffy/dontdiff linux-kernel3/test/dapltest/test/dapl_transaction_util.c linux-kernel4/test/dapltest/test/dapl_transaction_util.c --- linux-kernel3/test/dapltest/test/dapl_transaction_util.c 2005-05-09 11:46:47.629003000 -0700 +++ linux-kernel4/test/dapltest/test/dapl_transaction_util.c 2005-05-09 11:52:52.357004000 -0700 @@ -486,7 +486,7 @@ Transaction_Test_Op_t *op = &ep_context[i].op[op_indx]; struct dat_lmr_triplet *iov = DT_Bpool_GetIOV (op->bp, 0); DAT_DTO_COOKIE cookie; - DAT_RMR_TRIPLET rmr_triplet; + struct dat_rmr_triplet rmr_triplet; /* Prep the inputs */ for (j = 0; j < op->num_segs; j++) From Tom.Duffy at Sun.COM Mon May 9 14:39:40 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 09 May 2005 14:39:40 -0700 Subject: [openib-general] [PATCH][5/26] kDAPL: Remove typedef DAT_NAMED_ATTR In-Reply-To: <11156747802252@sun.com> References: <11156747793273@sun.com> <11156747791462@sun.com> <1115674780270@sun.com> <11156747803410@sun.com> <11156747802252@sun.com> Message-ID: <11156747801195@sun.com> Signed-off-by: Tom Duffy diff -Nur -X /home/tduffy/dontdiff linux-kernel4/dat/dat.h linux-kernel5/dat/dat.h --- linux-kernel4/dat/dat.h 2005-05-09 11:50:52.631016000 -0700 +++ linux-kernel5/dat/dat.h 2005-05-09 11:56:32.977020000 -0700 @@ -361,10 +361,10 @@ typedef char *DAT_NAME_PTR; #define DAT_NAME_MAX_LENGTH 256 -typedef struct dat_named_attr { +struct dat_named_attr { const char *name; const char *value; -} DAT_NAMED_ATTR; +}; typedef enum dat_boolean { DAT_FALSE = 0, @@ -588,9 +588,9 @@ DAT_COUNT max_rdma_read_iov; DAT_COUNT max_rdma_write_iov; DAT_COUNT ep_transport_specific_count; - DAT_NAMED_ATTR *ep_transport_specific; + struct dat_named_attr *ep_transport_specific; DAT_COUNT ep_provider_specific_count; - DAT_NAMED_ATTR *ep_provider_specific; + struct dat_named_attr *ep_provider_specific; } DAT_EP_ATTR; /* Endpoint Parameters */ @@ -1037,9 +1037,9 @@ DAT_BOOLEAN max_rdma_read_per_ep_in_guaranteed; DAT_BOOLEAN max_rdma_read_per_ep_out_guaranteed; DAT_COUNT num_transport_attr; - DAT_NAMED_ATTR *transport_attr; + struct dat_named_attr *transport_attr; DAT_COUNT num_vendor_attr; - DAT_NAMED_ATTR *vendor_attr; + struct dat_named_attr *vendor_attr; } DAT_IA_ATTR; /* Provider attributes */ @@ -1096,7 +1096,7 @@ DAT_BOOLEAN dto_async_return_guaranteed; DAT_BOOLEAN rdma_write_for_rdma_read_req; DAT_COUNT num_provider_specific_attr; - DAT_NAMED_ATTR *provider_specific_attr; + struct dat_named_attr *provider_specific_attr; }; diff -Nur -X /home/tduffy/dontdiff linux-kernel4/dat-provider/dapl_adapter_util.h linux-kernel5/dat-provider/dapl_adapter_util.h --- linux-kernel4/dat-provider/dapl_adapter_util.h 2005-05-06 16:22:02.936005000 -0700 +++ linux-kernel5/dat-provider/dapl_adapter_util.h 2005-05-09 11:56:04.963050000 -0700 @@ -178,7 +178,7 @@ /* - * Values for provider DAT_NAMED_ATTR + * Values for provider struct dat_named_attr */ #define IB_QP_STATE 1 /* QP state change request */ diff -Nur -X /home/tduffy/dontdiff linux-kernel4/dat-provider/dapl_ia_util.c linux-kernel5/dat-provider/dapl_ia_util.c --- linux-kernel4/dat-provider/dapl_ia_util.c 2005-05-09 11:07:01.789010000 -0700 +++ linux-kernel5/dat-provider/dapl_ia_util.c 2005-05-09 11:56:04.969021000 -0700 @@ -191,7 +191,7 @@ /* force the EP into error state to force flush all posted DTOs. */ { DAT_EP_ATTR ep_attr; - DAT_NAMED_ATTR ep_state; + struct dat_named_attr ep_state; memset(&ep_attr, 0, sizeof(DAT_EP_ATTR)); ep_state.name = (char *)IB_QP_STATE; diff -Nur -X /home/tduffy/dontdiff linux-kernel4/dat-provider/dapl_openib_util.c linux-kernel5/dat-provider/dapl_openib_util.c --- linux-kernel4/dat-provider/dapl_openib_util.c 2005-05-04 08:19:32.732990000 -0700 +++ linux-kernel5/dat-provider/dapl_openib_util.c 2005-05-09 11:58:35.028003000 -0700 @@ -799,11 +799,10 @@ * * Assign transport specific and vendor specific values here. * Both attributes are of type: - * typedef struct dat_named_attr - * { - * const char * name; * Name of attribute * - * const char * value; * Value of attribute * - * } DAT_NAMED_ATTR; + * struct dat_named_attr { + * const char *name; * Name of attribute * + * const char *value; * Value of attribute * + * }; */ ia_attr->num_transport_attr = 0; ia_attr->transport_attr = NULL; From Tom.Duffy at Sun.COM Mon May 9 14:39:41 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 09 May 2005 14:39:41 -0700 Subject: [openib-general] [PATCH][6/26] kDAPL: Remove typedef DAT_RMR_PARAM In-Reply-To: <11156747801195@sun.com> References: <11156747793273@sun.com> <11156747791462@sun.com> <1115674780270@sun.com> <11156747803410@sun.com> <11156747802252@sun.com> <11156747801195@sun.com> Message-ID: <11156747812210@sun.com> Signed-off-by: Tom Duffy diff -Nur -X /home/tduffy/dontdiff linux-kernel5/dat/dat.h linux-kernel6/dat/dat.h --- linux-kernel5/dat/dat.h 2005-05-09 11:56:32.977020000 -0700 +++ linux-kernel6/dat/dat.h 2005-05-09 12:09:01.001003000 -0700 @@ -563,13 +563,13 @@ (DAT_MEM_PRIV_LOCAL_WRITE_FLAG | DAT_MEM_PRIV_REMOTE_WRITE_FLAG) /* RMR Arguments */ -typedef struct dat_rmr_param { +struct dat_rmr_param { DAT_IA_HANDLE ia_handle; DAT_PZ_HANDLE pz_handle; struct dat_lmr_triplet lmr_triplet; DAT_MEM_PRIV_FLAGS mem_priv; DAT_RMR_CONTEXT rmr_context; -} DAT_RMR_PARAM; +}; /* Endpoint attributes */ typedef struct dat_ep_attr { @@ -1208,7 +1208,7 @@ typedef DAT_RETURN (*DAT_RMR_CREATE_FUNC) (DAT_PZ_HANDLE, DAT_RMR_HANDLE *); -typedef DAT_RETURN (*DAT_RMR_QUERY_FUNC) (DAT_RMR_HANDLE, DAT_RMR_PARAM *); +typedef DAT_RETURN (*DAT_RMR_QUERY_FUNC) (DAT_RMR_HANDLE, struct dat_rmr_param *); typedef DAT_RETURN (*DAT_RMR_BIND_FUNC) ( DAT_RMR_HANDLE, const struct dat_lmr_triplet *, DAT_MEM_PRIV_FLAGS, @@ -1688,7 +1688,7 @@ return DAT_CALL_PROVIDER_FUNC(rmr_create_func, pz, rmr); } -static inline DAT_RETURN dat_rmr_query(DAT_RMR_HANDLE rmr, DAT_RMR_PARAM * param) +static inline DAT_RETURN dat_rmr_query(DAT_RMR_HANDLE rmr, struct dat_rmr_param *param) { return DAT_CALL_PROVIDER_FUNC(rmr_query_func, rmr, param); } diff -Nur -X /home/tduffy/dontdiff linux-kernel5/dat-provider/dapl.h linux-kernel6/dat-provider/dapl.h --- linux-kernel5/dat-provider/dapl.h 2005-05-09 11:49:45.236005000 -0700 +++ linux-kernel6/dat-provider/dapl.h 2005-05-09 12:08:08.092003000 -0700 @@ -366,7 +366,7 @@ /* DAPL_RMR maps to DAT_RMR_HANDLE */ struct dapl_rmr { DAPL_HEADER header; - DAT_RMR_PARAM param; + struct dat_rmr_param param; DAPL_EP *ep; DAPL_PZ *pz; DAPL_LMR *lmr; @@ -674,7 +674,7 @@ DAT_RMR_HANDLE *); /* rmr_handle */ extern DAT_RETURN dapl_rmr_query(DAT_RMR_HANDLE, /* rmr_handle */ - DAT_RMR_PARAM *); /* rmr_args */ + struct dat_rmr_param *); /* rmr_args */ extern DAT_RETURN dapl_rmr_bind(DAT_RMR_HANDLE, /* rmr_handle */ const struct dat_lmr_triplet *, /* lmr_triplet */ diff -Nur -X /home/tduffy/dontdiff linux-kernel5/dat-provider/dapl_rmr_query.c linux-kernel6/dat-provider/dapl_rmr_query.c --- linux-kernel5/dat-provider/dapl_rmr_query.c 2005-05-04 08:19:32.691008000 -0700 +++ linux-kernel6/dat-provider/dapl_rmr_query.c 2005-05-09 12:10:37.270017000 -0700 @@ -38,7 +38,8 @@ #include "dapl.h" -DAT_RETURN dapl_rmr_query(DAT_RMR_HANDLE rmr_handle, DAT_RMR_PARAM * rmr_param) +DAT_RETURN dapl_rmr_query(DAT_RMR_HANDLE rmr_handle, + struct dat_rmr_param *rmr_param) { DAPL_RMR *rmr; DAT_RETURN dat_status; @@ -57,7 +58,7 @@ rmr = (DAPL_RMR *) rmr_handle; - memcpy(rmr_param, &rmr->param, sizeof(DAT_RMR_PARAM)); + memcpy(rmr_param, &rmr->param, sizeof *rmr_param); bail: return dat_status; diff -Nur -X /home/tduffy/dontdiff linux-kernel5/test/dapltest/test/dapl_fft_queryinfo.c linux-kernel6/test/dapltest/test/dapl_fft_queryinfo.c --- linux-kernel5/test/dapltest/test/dapl_fft_queryinfo.c 2005-05-09 11:15:26.469002000 -0700 +++ linux-kernel6/test/dapltest/test/dapl_fft_queryinfo.c 2005-05-09 12:10:49.778020000 -0700 @@ -69,7 +69,7 @@ DAT_LMR_PARAM lmr_param; DAT_LMR_CONTEXT lmr_context; DAT_RMR_HANDLE rmr_handle; - DAT_RMR_PARAM rmr_param; + struct dat_rmr_param rmr_param; DAT_REGION_DESCRIPTION region; DAT_VLEN reg_size; DAT_VADDR reg_addr; From Tom.Duffy at Sun.COM Mon May 9 14:39:41 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 09 May 2005 14:39:41 -0700 Subject: [openib-general] [PATCH][7/26] kDAPL: Remove typedef DAT_EP_ATTR In-Reply-To: <11156747812210@sun.com> References: <11156747793273@sun.com> <11156747791462@sun.com> <1115674780270@sun.com> <11156747803410@sun.com> <11156747802252@sun.com> <11156747801195@sun.com> <11156747812210@sun.com> Message-ID: <11156747812290@sun.com> Signed-off-by: Tom Duffy diff -Nur -X /home/tduffy/dontdiff linux-kernel6/dat/dat.h linux-kernel7/dat/dat.h --- linux-kernel6/dat/dat.h 2005-05-09 12:09:01.001003000 -0700 +++ linux-kernel7/dat/dat.h 2005-05-09 12:17:29.401004000 -0700 @@ -572,7 +572,7 @@ }; /* Endpoint attributes */ -typedef struct dat_ep_attr { +struct dat_ep_attr { DAT_VLEN max_message_size; DAT_VLEN max_rdma_size; DAT_QOS qos; @@ -591,7 +591,7 @@ struct dat_named_attr *ep_transport_specific; DAT_COUNT ep_provider_specific_count; struct dat_named_attr *ep_provider_specific; -} DAT_EP_ATTR; +}; /* Endpoint Parameters */ typedef enum dat_ep_state { @@ -622,7 +622,7 @@ DAT_EVD_HANDLE request_evd_handle; DAT_EVD_HANDLE connect_evd_handle; DAT_SRQ_HANDLE srq_handle; - DAT_EP_ATTR ep_attr; + struct dat_ep_attr ep_attr; } DAT_EP_PARAM; typedef enum dat_ep_param_mask { @@ -1146,11 +1146,11 @@ typedef DAT_RETURN (*DAT_EP_CREATE_FUNC) ( DAT_IA_HANDLE, DAT_PZ_HANDLE, DAT_EVD_HANDLE, DAT_EVD_HANDLE, - DAT_EVD_HANDLE, const DAT_EP_ATTR *, DAT_EP_HANDLE *); + DAT_EVD_HANDLE, const struct dat_ep_attr *, DAT_EP_HANDLE *); typedef DAT_RETURN (*DAT_EP_CREATE_WITH_SRQ_FUNC) ( DAT_IA_HANDLE, DAT_PZ_HANDLE, DAT_EVD_HANDLE, DAT_EVD_HANDLE, - DAT_EVD_HANDLE, DAT_SRQ_HANDLE, const DAT_EP_ATTR *, DAT_EP_HANDLE *); + DAT_EVD_HANDLE, DAT_SRQ_HANDLE, const struct dat_ep_attr *, DAT_EP_HANDLE *); typedef DAT_RETURN (*DAT_EP_QUERY_FUNC) (DAT_EP_HANDLE, DAT_EP_PARAM *); @@ -1530,7 +1530,7 @@ DAT_EVD_HANDLE in_evd, DAT_EVD_HANDLE out_evd, DAT_EVD_HANDLE connect_evd, - const DAT_EP_ATTR * attr, + const struct dat_ep_attr *attr, DAT_EP_HANDLE * ep) { return DAT_CALL_PROVIDER_FUNC( @@ -1544,7 +1544,7 @@ DAT_EVD_HANDLE out_evd, DAT_EVD_HANDLE connect_evd, DAT_SRQ_HANDLE srq, - const DAT_EP_ATTR * attr, + const struct dat_ep_attr *attr, DAT_EP_HANDLE * ep) { return DAT_CALL_PROVIDER_FUNC(ep_create_with_srq_func, ia, pz, in_evd, diff -Nur -X /home/tduffy/dontdiff linux-kernel6/dat-provider/dapl_adapter_util.h linux-kernel7/dat-provider/dapl_adapter_util.h --- linux-kernel6/dat-provider/dapl_adapter_util.h 2005-05-09 11:56:04.963050000 -0700 +++ linux-kernel7/dat-provider/dapl_adapter_util.h 2005-05-09 12:17:54.079004000 -0700 @@ -69,7 +69,7 @@ DAT_RETURN dapl_ib_qp_free(DAPL_IA * ia_ptr, DAPL_EP * ep_ptr); DAT_RETURN dapl_ib_qp_modify(DAPL_IA * ia_ptr, - DAPL_EP * ep_ptr, DAT_EP_ATTR * ep_attr); + DAPL_EP * ep_ptr, struct dat_ep_attr *ep_attr); DAT_RETURN dapl_ib_connect(DAT_EP_HANDLE ep_handle, DAT_IA_ADDRESS_PTR remote_ia_address, @@ -147,7 +147,8 @@ DAT_RETURN dapl_ib_query_hca(DAPL_HCA * hca_ptr, DAT_IA_ATTR * ia_attr, - DAT_EP_ATTR * ep_attr, DAT_SOCK_ADDR6 * ip_addr); + struct dat_ep_attr *ep_attr, + DAT_SOCK_ADDR6 * ip_addr); DAT_RETURN dapl_ib_completion_poll(DAPL_HCA * hca_ptr, DAPL_EVD * evd_ptr, diff -Nur -X /home/tduffy/dontdiff linux-kernel6/dat-provider/dapl_ep_create.c linux-kernel7/dat-provider/dapl_ep_create.c --- linux-kernel6/dat-provider/dapl_ep_create.c 2005-05-09 11:07:01.700016000 -0700 +++ linux-kernel7/dat-provider/dapl_ep_create.c 2005-05-09 12:18:42.251015000 -0700 @@ -69,11 +69,11 @@ DAT_EVD_HANDLE recv_evd_handle, DAT_EVD_HANDLE request_evd_handle, DAT_EVD_HANDLE connect_evd_handle, - const DAT_EP_ATTR * ep_attr, DAT_EP_HANDLE * ep_handle) + const struct dat_ep_attr *ep_attr, DAT_EP_HANDLE * ep_handle) { DAPL_IA *ia_ptr; DAPL_EP *ep_ptr; - DAT_EP_ATTR ep_attr_limit; + struct dat_ep_attr ep_attr_limit; DAT_RETURN dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, @@ -172,7 +172,7 @@ /* Verify the attributes against the transport */ if (ep_attr != NULL) { - memset(&ep_attr_limit, 0, sizeof(DAT_EP_ATTR)); + memset(&ep_attr_limit, 0, sizeof ep_attr_limit); dat_status = dapl_ib_query_hca(ia_ptr->hca_ptr, NULL, &ep_attr_limit, NULL); if (dat_status != DAT_SUCCESS) { diff -Nur -X /home/tduffy/dontdiff linux-kernel6/dat-provider/dapl_ep_create_with_srq.c linux-kernel7/dat-provider/dapl_ep_create_with_srq.c --- linux-kernel6/dat-provider/dapl_ep_create_with_srq.c 2005-05-09 11:07:01.993022000 -0700 +++ linux-kernel7/dat-provider/dapl_ep_create_with_srq.c 2005-05-09 12:19:41.681004000 -0700 @@ -76,11 +76,12 @@ DAT_EVD_HANDLE request_evd_handle, DAT_EVD_HANDLE connect_evd_handle, DAT_SRQ_HANDLE srq_handle, - const DAT_EP_ATTR * ep_attr, DAT_EP_HANDLE * ep_handle) + const struct dat_ep_attr *ep_attr, + DAT_EP_HANDLE * ep_handle) { DAPL_IA *ia_ptr; DAPL_EP *ep_ptr; - DAT_EP_ATTR ep_attr_limit; + struct dat_ep_attr ep_attr_limit; DAT_RETURN dat_status; dat_status = DAT_SUCCESS; @@ -187,7 +188,7 @@ /* Verify the attributes against the transport */ if (ep_attr != NULL) { - memset(&ep_attr_limit, 0, sizeof(DAT_EP_ATTR)); + memset(&ep_attr_limit, 0, sizeof ep_attr_limit); dat_status = dapl_ib_query_hca(ia_ptr->hca_ptr, NULL, &ep_attr_limit, NULL); if (dat_status != DAT_SUCCESS) { diff -Nur -X /home/tduffy/dontdiff linux-kernel6/dat-provider/dapl_ep_modify.c linux-kernel7/dat-provider/dapl_ep_modify.c --- linux-kernel6/dat-provider/dapl_ep_modify.c 2005-05-04 08:19:32.748971000 -0700 +++ linux-kernel7/dat-provider/dapl_ep_modify.c 2005-05-09 12:20:20.503018000 -0700 @@ -57,13 +57,14 @@ DAT_EP_PARAM_MASK mask, const DAT_EP_PARAM * ep_param, DAPL_IA ** ia_ptr, - DAPL_EP ** ep_ptr, DAT_EP_ATTR * ep_attr_ptr) + DAPL_EP ** ep_ptr, + struct dat_ep_attr *ep_attr_ptr) { DAPL_IA *ia; DAPL_EP *ep; - DAT_EP_ATTR ep_attr; - DAT_EP_ATTR ep_attr_limit; - DAT_EP_ATTR ep_attr_request; + struct dat_ep_attr ep_attr; + struct dat_ep_attr ep_attr_limit; + struct dat_ep_attr ep_attr_request; DAT_RETURN dat_status; *ia_ptr = NULL; @@ -174,7 +175,7 @@ */ ep_attr = ep->param.ep_attr; - memset(&ep_attr_limit, 0, sizeof(DAT_EP_ATTR)); + memset(&ep_attr_limit, 0, sizeof ep_attr_limit); dat_status = dapl_ib_query_hca(ia->hca_ptr, NULL, &ep_attr_limit, NULL); if (dat_status != DAT_SUCCESS) { goto bail; @@ -313,7 +314,7 @@ { DAPL_IA *ia; DAPL_EP *ep1, *ep2; - DAT_EP_ATTR ep_attr1, ep_attr2; + struct dat_ep_attr ep_attr1, ep_attr2; DAPL_EP new_ep, copy_of_old_ep; DAPL_EP alloc_ep; /* Holder for resources. */ DAPL_PZ *tmp_pz; diff -Nur -X /home/tduffy/dontdiff linux-kernel6/dat-provider/dapl_ep_util.c linux-kernel7/dat-provider/dapl_ep_util.c --- linux-kernel6/dat-provider/dapl_ep_util.c 2005-05-09 11:51:34.946002000 -0700 +++ linux-kernel7/dat-provider/dapl_ep_util.c 2005-05-09 12:21:06.403014000 -0700 @@ -75,7 +75,7 @@ * none * */ -DAPL_EP *dapl_ep_alloc(DAPL_IA * ia_ptr, const DAT_EP_ATTR * ep_attr) +DAPL_EP *dapl_ep_alloc(DAPL_IA * ia_ptr, const struct dat_ep_attr *ep_attr) { DAPL_EP *ep_ptr; @@ -223,11 +223,11 @@ */ void dapl_ep_default_attrs(DAPL_EP * ep_ptr) { - DAT_EP_ATTR *ep_attr; + struct dat_ep_attr *ep_attr; ep_attr = &ep_ptr->param.ep_attr; /* Set up defaults */ - memset(ep_attr, 0, sizeof(DAT_EP_ATTR)); + memset(ep_attr, 0, sizeof *ep_attr); /* mtu and rdma sizes fixed in IB as per IBTA 1.1, 9.4.3, 9.4.4, 9.7.7. */ ep_attr->max_message_size = 0x80000000; diff -Nur -X /home/tduffy/dontdiff linux-kernel6/dat-provider/dapl_ep_util.h linux-kernel7/dat-provider/dapl_ep_util.h --- linux-kernel6/dat-provider/dapl_ep_util.h 2005-05-09 11:51:45.071013000 -0700 +++ linux-kernel7/dat-provider/dapl_ep_util.h 2005-05-09 12:21:22.479009000 -0700 @@ -42,7 +42,7 @@ /* function prototypes */ -extern DAPL_EP *dapl_ep_alloc(DAPL_IA * ia, const DAT_EP_ATTR * ep_attr); +extern DAPL_EP *dapl_ep_alloc(DAPL_IA * ia, const struct dat_ep_attr *ep_attr); extern void dapl_ep_dealloc(DAPL_EP * ep_ptr); diff -Nur -X /home/tduffy/dontdiff linux-kernel6/dat-provider/dapl.h linux-kernel7/dat-provider/dapl.h --- linux-kernel6/dat-provider/dapl.h 2005-05-09 12:08:08.092003000 -0700 +++ linux-kernel7/dat-provider/dapl.h 2005-05-09 12:16:33.808001000 -0700 @@ -561,7 +561,7 @@ DAT_EVD_HANDLE, /* in_dto_completion_evd_handle */ DAT_EVD_HANDLE, /* out_dto_completion_evd_handle */ DAT_EVD_HANDLE, /* connect_evd_handle */ - const DAT_EP_ATTR *, /* ep_parameters */ + const struct dat_ep_attr *, /* ep_parameters */ DAT_EP_HANDLE *); /* ep_handle */ extern DAT_RETURN dapl_ep_query(DAT_EP_HANDLE, DAT_EP_PARAM *); @@ -630,7 +630,7 @@ DAT_EVD_HANDLE, /* request_evd_handle */ DAT_EVD_HANDLE, /* connect_evd_handle */ DAT_SRQ_HANDLE, /* srq_handle */ - const DAT_EP_ATTR *, /* ep_attributes */ + const struct dat_ep_attr *, /* ep_attributes */ DAT_EP_HANDLE *); /* ep_handle */ extern DAT_RETURN dapl_ep_recv_query(DAT_EP_HANDLE, /* ep_handle */ diff -Nur -X /home/tduffy/dontdiff linux-kernel6/dat-provider/dapl_ia_util.c linux-kernel7/dat-provider/dapl_ia_util.c --- linux-kernel6/dat-provider/dapl_ia_util.c 2005-05-09 11:56:04.969021000 -0700 +++ linux-kernel7/dat-provider/dapl_ia_util.c 2005-05-09 12:21:52.042013000 -0700 @@ -190,10 +190,10 @@ } /* force the EP into error state to force flush all posted DTOs. */ { - DAT_EP_ATTR ep_attr; + struct dat_ep_attr ep_attr; struct dat_named_attr ep_state; - memset(&ep_attr, 0, sizeof(DAT_EP_ATTR)); + memset(&ep_attr, 0, sizeof ep_attr); ep_state.name = (char *)IB_QP_STATE; ep_state.value = (char *)DAPL_QP_STATE_ERROR; ep_attr.ep_provider_specific_count = 1; diff -Nur -X /home/tduffy/dontdiff linux-kernel6/dat-provider/dapl_openib_qp.c linux-kernel7/dat-provider/dapl_openib_qp.c --- linux-kernel6/dat-provider/dapl_openib_qp.c 2005-04-11 13:49:41.248021000 -0700 +++ linux-kernel7/dat-provider/dapl_openib_qp.c 2005-05-09 12:22:24.354000000 -0700 @@ -70,7 +70,7 @@ DAT_RETURN dapl_ib_qp_alloc(DAPL_IA * ia_ptr, DAPL_EP * ep_ptr, DAPL_EP * ep_ctx_ptr) { - DAT_EP_ATTR *attr; + struct dat_ep_attr *attr; int ib_status; struct ib_qp_init_attr qp_attr; struct ib_pd *ib_pd_handle; @@ -232,7 +232,8 @@ * */ DAT_RETURN -dapl_ib_qp_modify(DAPL_IA * ia_ptr, DAPL_EP * ep_ptr, DAT_EP_ATTR * ep_attr) +dapl_ib_qp_modify(DAPL_IA * ia_ptr, DAPL_EP * ep_ptr, + struct dat_ep_attr *ep_attr) { struct ib_qp_attr qp_attr; struct ib_qp *qp_handle; diff -Nur -X /home/tduffy/dontdiff linux-kernel6/dat-provider/dapl_openib_util.c linux-kernel7/dat-provider/dapl_openib_util.c --- linux-kernel6/dat-provider/dapl_openib_util.c 2005-05-09 11:58:35.028003000 -0700 +++ linux-kernel7/dat-provider/dapl_openib_util.c 2005-05-09 12:22:39.554000000 -0700 @@ -747,7 +747,8 @@ */ DAT_RETURN dapl_ib_query_hca(DAPL_HCA * hca_ptr, DAT_IA_ATTR * ia_attr, - DAT_EP_ATTR * ep_attr, DAT_SOCK_ADDR6 * ip_addr) + struct dat_ep_attr *ep_attr, + DAT_SOCK_ADDR6 * ip_addr) { int ib_status; struct ib_device_attr device_attr; diff -Nur -X /home/tduffy/dontdiff linux-kernel6/test/dapltest/include/dapl_performance_test.h linux-kernel7/test/dapltest/include/dapl_performance_test.h --- linux-kernel6/test/dapltest/include/dapl_performance_test.h 2005-05-05 13:56:36.275962000 -0700 +++ linux-kernel7/test/dapltest/include/dapl_performance_test.h 2005-05-09 12:22:53.931012000 -0700 @@ -55,7 +55,7 @@ typedef struct { DAT_EP_HANDLE ep_handle; - DAT_EP_ATTR ep_attr; + struct dat_ep_attr ep_attr; DAT_CONN_QUAL port; DAT_COUNT pipeline_len; Bpool *bp; diff -Nur -X /home/tduffy/dontdiff linux-kernel6/test/dapltest/include/dapl_test_data.h linux-kernel7/test/dapltest/include/dapl_test_data.h --- linux-kernel6/test/dapltest/include/dapl_test_data.h 2005-05-09 11:15:26.458003000 -0700 +++ linux-kernel7/test/dapltest/include/dapl_test_data.h 2005-05-09 12:16:33.875003000 -0700 @@ -82,7 +82,7 @@ Params_t Params; DAT_IA_ATTR ia_attr; struct dat_provider_attr provider_attr; - DAT_EP_ATTR ep_attr; + struct dat_ep_attr ep_attr; Per_Server_Data_t *ps_ptr; Transaction_Stats_t Client_Stats; diff -Nur -X /home/tduffy/dontdiff linux-kernel6/test/dapltest/include/dapl_transaction_test.h linux-kernel7/test/dapltest/include/dapl_transaction_test.h --- linux-kernel6/test/dapltest/include/dapl_transaction_test.h 2005-05-05 13:57:04.196965000 -0700 +++ linux-kernel7/test/dapltest/include/dapl_transaction_test.h 2005-05-09 12:23:07.735003000 -0700 @@ -50,7 +50,7 @@ typedef struct { DAT_EP_HANDLE ep_handle; - DAT_EP_ATTR ep_attr; + struct dat_ep_attr ep_attr; DAT_CONN_QUAL ia_port; Bpool *bp; Transaction_Test_Op_t op[ MAX_OPS ]; diff -Nur -X /home/tduffy/dontdiff linux-kernel6/test/dapltest/test/dapl_client.c linux-kernel7/test/dapltest/test/dapl_client.c --- linux-kernel6/test/dapltest/test/dapl_client.c 2005-05-05 13:45:21.784000000 -0700 +++ linux-kernel7/test/dapltest/test/dapl_client.c 2005-05-09 12:16:33.887002000 -0700 @@ -179,7 +179,7 @@ recv_evd_hdl, /* recv */ reqt_evd_hdl, /* request */ conn_evd_hdl, /* connect */ - (DAT_EP_ATTR *) NULL, + (struct dat_ep_attr *) NULL, &ep_handle); if (ret != DAT_SUCCESS) { diff -Nur -X /home/tduffy/dontdiff linux-kernel6/test/dapltest/test/dapl_limit.c linux-kernel7/test/dapltest/test/dapl_limit.c --- linux-kernel6/test/dapltest/test/dapl_limit.c 2005-05-09 11:44:35.444001000 -0700 +++ linux-kernel7/test/dapltest/test/dapl_limit.c 2005-05-09 12:16:33.897001000 -0700 @@ -583,7 +583,7 @@ hdl_sets[w].evd_handle, /* recv */ hdl_sets[w].evd_handle, /* request */ conn_handle, /* connect */ - (DAT_EP_ATTR *) NULL, + (struct dat_ep_attr *) NULL, &hdl_sets[w].ep_handle); if (ret != DAT_SUCCESS) { @@ -628,7 +628,7 @@ hdl_sets[w % cmd->width].evd_handle, hdl_sets[w % cmd->width].evd_handle, conn_handle, /* connect */ - (DAT_EP_ATTR *) NULL, + (struct dat_ep_attr *) NULL, &hdlptr[w]); if (ret != DAT_SUCCESS) { @@ -728,7 +728,7 @@ hdl_sets[w % cmd->width].evd_handle, hdl_sets[w % cmd->width].evd_handle, conn_handle, - (DAT_EP_ATTR *) NULL, + (struct dat_ep_attr *) NULL, &epptr[w]); if (ret != DAT_SUCCESS) { diff -Nur -X /home/tduffy/dontdiff linux-kernel6/test/dapltest/test/dapl_server.c linux-kernel7/test/dapltest/test/dapl_server.c --- linux-kernel6/test/dapltest/test/dapl_server.c 2005-05-05 09:42:06.394000000 -0700 +++ linux-kernel7/test/dapltest/test/dapl_server.c 2005-05-09 12:16:33.904000000 -0700 @@ -195,7 +195,7 @@ ps_ptr->recv_evd_hdl, /* recv */ ps_ptr->reqt_evd_hdl, /* request */ ps_ptr->conn_evd_hdl, /* connect */ - (DAT_EP_ATTR *) NULL, + (struct dat_ep_attr *) NULL, &ps_ptr->ep_handle); if (ret != DAT_SUCCESS) { diff -Nur -X /home/tduffy/dontdiff linux-kernel6/test/dapltest/test/dapl_transaction_test.c linux-kernel7/test/dapltest/test/dapl_transaction_test.c --- linux-kernel6/test/dapltest/test/dapl_transaction_test.c 2005-05-05 13:45:21.794000000 -0700 +++ linux-kernel7/test/dapltest/test/dapl_transaction_test.c 2005-05-09 12:23:41.376001000 -0700 @@ -344,7 +344,7 @@ */ for (i = 0; i < test_ptr->cmd->eps_per_thread; i++) { - DAT_EP_ATTR ep_attr; + struct dat_ep_attr ep_attr; u32 buff_size = MAX_OPS * sizeof (RemoteMemoryInfo); /* From Tom.Duffy at Sun.COM Mon May 9 14:39:41 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 09 May 2005 14:39:41 -0700 Subject: [openib-general] [PATCH][8/26] kDAPL: Remove typedef DAT_EP_PARAM In-Reply-To: <11156747812290@sun.com> References: <11156747793273@sun.com> <11156747791462@sun.com> <1115674780270@sun.com> <11156747803410@sun.com> <11156747802252@sun.com> <11156747801195@sun.com> <11156747812210@sun.com> <11156747812290@sun.com> Message-ID: <11156747812388@sun.com> Signed-off-by: Tom Duffy diff -Nur -X /home/tduffy/dontdiff linux-kernel7/dat/dat.h linux-kernel8/dat/dat.h --- linux-kernel7/dat/dat.h 2005-05-09 12:17:29.401004000 -0700 +++ linux-kernel8/dat/dat.h 2005-05-09 12:30:15.364000000 -0700 @@ -610,7 +610,7 @@ DAT_EP_STATE_COMPLETION_PENDING } DAT_EP_STATE; -typedef struct dat_ep_param { +struct dat_ep_param { DAT_IA_HANDLE ia_handle; DAT_EP_STATE ep_state; DAT_IA_ADDRESS_PTR local_ia_address_ptr; @@ -623,7 +623,7 @@ DAT_EVD_HANDLE connect_evd_handle; DAT_SRQ_HANDLE srq_handle; struct dat_ep_attr ep_attr; -} DAT_EP_PARAM; +}; typedef enum dat_ep_param_mask { DAT_EP_FIELD_IA_HANDLE = (1 << 0), @@ -1152,10 +1152,10 @@ DAT_IA_HANDLE, DAT_PZ_HANDLE, DAT_EVD_HANDLE, DAT_EVD_HANDLE, DAT_EVD_HANDLE, DAT_SRQ_HANDLE, const struct dat_ep_attr *, DAT_EP_HANDLE *); -typedef DAT_RETURN (*DAT_EP_QUERY_FUNC) (DAT_EP_HANDLE, DAT_EP_PARAM *); +typedef DAT_RETURN (*DAT_EP_QUERY_FUNC) (DAT_EP_HANDLE, struct dat_ep_param *); typedef DAT_RETURN (*DAT_EP_MODIFY_FUNC) ( - DAT_EP_HANDLE, DAT_EP_PARAM_MASK, const DAT_EP_PARAM *); + DAT_EP_HANDLE, DAT_EP_PARAM_MASK, const struct dat_ep_param *); typedef DAT_RETURN (*DAT_EP_CONNECT_FUNC) ( DAT_EP_HANDLE, DAT_IA_ADDRESS_PTR, DAT_CONN_QUAL, DAT_TIMEOUT, DAT_COUNT, @@ -1584,7 +1584,7 @@ static inline DAT_RETURN dat_ep_modify(DAT_EP_HANDLE ep, DAT_EP_PARAM_MASK mask, - const DAT_EP_PARAM * param) + const struct dat_ep_param *param) { return DAT_CALL_PROVIDER_FUNC(ep_modify_func, ep, mask, param); } @@ -1631,7 +1631,7 @@ ep_post_send_func, ep, size, local_iov, cookie, flags); } -static inline DAT_RETURN dat_ep_query(DAT_EP_HANDLE ep, DAT_EP_PARAM * param) +static inline DAT_RETURN dat_ep_query(DAT_EP_HANDLE ep, struct dat_ep_param *param) { return DAT_CALL_PROVIDER_FUNC(ep_query_func, ep, param); } diff -Nur -X /home/tduffy/dontdiff linux-kernel7/dat-provider/dapl_ep_free.c linux-kernel8/dat-provider/dapl_ep_free.c --- linux-kernel7/dat-provider/dapl_ep_free.c 2005-04-27 11:38:35.113023000 -0700 +++ linux-kernel8/dat-provider/dapl_ep_free.c 2005-05-09 12:29:25.407003000 -0700 @@ -65,7 +65,7 @@ { DAPL_EP *ep_ptr; DAPL_IA *ia_ptr; - DAT_EP_PARAM *param; + struct dat_ep_param *param; ib_qp_state_t save_qp_state; DAT_RETURN dat_status = DAT_SUCCESS; diff -Nur -X /home/tduffy/dontdiff linux-kernel7/dat-provider/dapl_ep_modify.c linux-kernel8/dat-provider/dapl_ep_modify.c --- linux-kernel7/dat-provider/dapl_ep_modify.c 2005-05-09 12:20:20.503018000 -0700 +++ linux-kernel8/dat-provider/dapl_ep_modify.c 2005-05-09 12:31:12.072006000 -0700 @@ -55,7 +55,7 @@ static inline DAT_RETURN dapl_ep_modify_validate_parameters(DAT_EP_HANDLE ep_handle, DAT_EP_PARAM_MASK mask, - const DAT_EP_PARAM * ep_param, + const struct dat_ep_param *ep_param, DAPL_IA ** ia_ptr, DAPL_EP ** ep_ptr, struct dat_ep_attr *ep_attr_ptr) @@ -310,7 +310,7 @@ * DAT_INVALID_STATE */ DAT_RETURN dapl_ep_modify(DAT_EP_HANDLE ep_handle, DAT_EP_PARAM_MASK mask, - const DAT_EP_PARAM * ep_param) + const struct dat_ep_param *ep_param) { DAPL_IA *ia; DAPL_EP *ep1, *ep2; diff -Nur -X /home/tduffy/dontdiff linux-kernel7/dat-provider/dapl_ep_query.c linux-kernel8/dat-provider/dapl_ep_query.c --- linux-kernel7/dat-provider/dapl_ep_query.c 2005-05-04 08:19:32.703961000 -0700 +++ linux-kernel8/dat-provider/dapl_ep_query.c 2005-05-09 12:31:30.215002000 -0700 @@ -58,7 +58,7 @@ * DAT_INVALID_PARAMETER */ DAT_RETURN -dapl_ep_query(DAT_EP_HANDLE ep_handle, DAT_EP_PARAM * ep_param) +dapl_ep_query(DAT_EP_HANDLE ep_handle, struct dat_ep_param *ep_param) { DAPL_EP *ep_ptr; DAT_RETURN status; diff -Nur -X /home/tduffy/dontdiff linux-kernel7/dat-provider/dapl.h linux-kernel8/dat-provider/dapl.h --- linux-kernel7/dat-provider/dapl.h 2005-05-09 12:16:33.808001000 -0700 +++ linux-kernel8/dat-provider/dapl.h 2005-05-09 12:29:25.404000000 -0700 @@ -302,7 +302,7 @@ struct dapl_ep { DAPL_HEADER header; /* What the DAT Consumer asked for */ - DAT_EP_PARAM param; + struct dat_ep_param param; /* The RC Queue Pair (IBM OS API) */ ib_qp_handle_t qp_handle; @@ -564,11 +564,11 @@ const struct dat_ep_attr *, /* ep_parameters */ DAT_EP_HANDLE *); /* ep_handle */ -extern DAT_RETURN dapl_ep_query(DAT_EP_HANDLE, DAT_EP_PARAM *); +extern DAT_RETURN dapl_ep_query(DAT_EP_HANDLE, struct dat_ep_param *); extern DAT_RETURN dapl_ep_modify(DAT_EP_HANDLE, /* ep_handle */ DAT_EP_PARAM_MASK, /* ep_args_mask */ - const DAT_EP_PARAM *); /* ep_args */ + const struct dat_ep_param *); /* ep_args */ extern DAT_RETURN dapl_ep_connect(DAT_EP_HANDLE, /* ep_handle */ DAT_IA_ADDRESS_PTR, /* remote_ia_address */ diff -Nur -X /home/tduffy/dontdiff linux-kernel7/test/dapltest/test/dapl_fft_queryinfo.c linux-kernel8/test/dapltest/test/dapl_fft_queryinfo.c --- linux-kernel7/test/dapltest/test/dapl_fft_queryinfo.c 2005-05-09 12:10:49.778020000 -0700 +++ linux-kernel8/test/dapltest/test/dapl_fft_queryinfo.c 2005-05-09 12:29:25.437003000 -0700 @@ -53,7 +53,7 @@ DAT_EVD_HANDLE send_evd_handle; DAT_EVD_HANDLE recv_evd_handle; DAT_EP_HANDLE ep_handle; - DAT_EP_PARAM ep_param; + struct dat_ep_param ep_param; DAT_CNO_HANDLE cno_handle; #ifndef __KDAPLTEST__ DAT_CNO_PARAM cno_param; diff -Nur -X /home/tduffy/dontdiff linux-kernel7/test/dapltest/test/dapl_test_util.c linux-kernel8/test/dapltest/test/dapl_test_util.c --- linux-kernel7/test/dapltest/test/dapl_test_util.c 2005-05-09 11:45:59.822013000 -0700 +++ linux-kernel8/test/dapltest/test/dapl_test_util.c 2005-05-09 12:31:59.724000000 -0700 @@ -37,7 +37,7 @@ { unsigned char *module = "DT_query"; DAT_EVD_HANDLE async_evd_hdl; /* not used */ - DAT_EP_PARAM ep_params; + struct dat_ep_param ep_params; DAT_RETURN ret; DT_Tdep_Print_Head *phead; From Tom.Duffy at Sun.COM Mon May 9 14:39:41 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 09 May 2005 14:39:41 -0700 Subject: [openib-general] [PATCH][9/26] kDAPL: Remove typedef DAT_SRQ_ATTR In-Reply-To: <11156747812388@sun.com> References: <11156747793273@sun.com> <11156747791462@sun.com> <1115674780270@sun.com> <11156747803410@sun.com> <11156747802252@sun.com> <11156747801195@sun.com> <11156747812210@sun.com> <11156747812290@sun.com> <11156747812388@sun.com> Message-ID: <11156747811508@sun.com> Signed-off-by: Tom Duffy diff -Nur -X /home/tduffy/dontdiff linux-kernel8/dat/dat.h linux-kernel9/dat/dat.h --- linux-kernel8/dat/dat.h 2005-05-09 12:30:15.364000000 -0700 +++ linux-kernel9/dat/dat.h 2005-05-09 12:37:03.881002000 -0700 @@ -670,11 +670,11 @@ #define DAT_VALUE_UNKNOWN (((DAT_COUNT) ~0)-1) -typedef struct dat_srq_attr { +struct dat_srq_attr { DAT_COUNT max_recv_dtos; DAT_COUNT max_recv_iov; DAT_COUNT low_watermark; -} DAT_SRQ_ATTR; +}; typedef struct dat_srq_param { DAT_IA_HANDLE ia_handle; @@ -1243,7 +1243,7 @@ typedef DAT_RETURN (*DAT_PZ_FREE_FUNC) (DAT_PZ_HANDLE); typedef DAT_RETURN (*DAT_SRQ_CREATE_FUNC) ( - DAT_IA_HANDLE, DAT_PZ_HANDLE, DAT_SRQ_ATTR *, DAT_SRQ_HANDLE *); + DAT_IA_HANDLE, DAT_PZ_HANDLE, struct dat_srq_attr *, DAT_SRQ_HANDLE *); typedef DAT_RETURN (*DAT_SRQ_SET_LW_FUNC) (DAT_SRQ_HANDLE, DAT_COUNT); @@ -1777,7 +1777,7 @@ static inline DAT_RETURN dat_srq_create(DAT_IA_HANDLE ia, DAT_PZ_HANDLE pz, - DAT_SRQ_ATTR * attr, + struct dat_srq_attr *attr, DAT_SRQ_HANDLE * srq) { return DAT_CALL_PROVIDER_FUNC(srq_create_func, ia, pz, attr, srq); diff -Nur -X /home/tduffy/dontdiff linux-kernel8/dat-provider/dapl.h linux-kernel9/dat-provider/dapl.h --- linux-kernel8/dat-provider/dapl.h 2005-05-09 12:29:25.404000000 -0700 +++ linux-kernel9/dat-provider/dapl.h 2005-05-09 12:36:45.218002000 -0700 @@ -730,7 +730,7 @@ extern DAT_RETURN dapl_srq_create(DAT_IA_HANDLE, /* ia_handle */ DAT_PZ_HANDLE, /* pz_handle */ - DAT_SRQ_ATTR *, /* srq_attr */ + struct dat_srq_attr *, /* srq_attr */ DAT_SRQ_HANDLE *); /* srq_handle */ extern DAT_RETURN dapl_srq_free(DAT_SRQ_HANDLE); /* srq_handle */ diff -Nur -X /home/tduffy/dontdiff linux-kernel8/dat-provider/dapl_srq_create.c linux-kernel9/dat-provider/dapl_srq_create.c --- linux-kernel8/dat-provider/dapl_srq_create.c 2005-04-20 12:42:06.794011000 -0700 +++ linux-kernel9/dat-provider/dapl_srq_create.c 2005-05-09 12:37:34.949031000 -0700 @@ -68,7 +68,7 @@ DAT_RETURN dapl_srq_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, - DAT_SRQ_ATTR * srq_attr, DAT_SRQ_HANDLE * srq_handle) + struct dat_srq_attr *srq_attr, DAT_SRQ_HANDLE * srq_handle) { DAPL_IA *ia_ptr; DAPL_SRQ *srq_ptr; diff -Nur -X /home/tduffy/dontdiff linux-kernel8/dat-provider/dapl_srq_util.c linux-kernel9/dat-provider/dapl_srq_util.c --- linux-kernel8/dat-provider/dapl_srq_util.c 2005-04-27 11:38:35.075003000 -0700 +++ linux-kernel9/dat-provider/dapl_srq_util.c 2005-05-09 12:37:47.386013000 -0700 @@ -54,7 +54,7 @@ * pointer to srq * */ -DAPL_SRQ *dapl_srq_alloc(DAPL_IA * ia_ptr, const DAT_SRQ_ATTR * srq_attr) +DAPL_SRQ *dapl_srq_alloc(DAPL_IA * ia_ptr, const struct dat_srq_attr *srq_attr) { DAPL_SRQ *srq_ptr; diff -Nur -X /home/tduffy/dontdiff linux-kernel8/dat-provider/dapl_srq_util.h linux-kernel9/dat-provider/dapl_srq_util.h --- linux-kernel8/dat-provider/dapl_srq_util.h 2005-04-20 12:42:07.197020000 -0700 +++ linux-kernel9/dat-provider/dapl_srq_util.h 2005-05-09 12:38:01.695005000 -0700 @@ -41,7 +41,8 @@ /* function prototypes */ -extern DAPL_SRQ *dapl_srq_alloc(DAPL_IA * ia, const DAT_SRQ_ATTR * srq_attr); +extern DAPL_SRQ *dapl_srq_alloc(DAPL_IA * ia, + const struct dat_srq_attr *srq_attr); extern void dapl_srq_dealloc(DAPL_SRQ * srq_ptr); From Tom.Duffy at Sun.COM Mon May 9 14:39:42 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 09 May 2005 14:39:42 -0700 Subject: [openib-general] [PATCH][11/26] kDAPL: Remove typedef DAT_PZ_PARAM In-Reply-To: <11156747821301@sun.com> References: <11156747793273@sun.com> <11156747791462@sun.com> <1115674780270@sun.com> <11156747803410@sun.com> <11156747802252@sun.com> <11156747801195@sun.com> <11156747812210@sun.com> <11156747812290@sun.com> <11156747812388@sun.com> <11156747811508@sun.com> <11156747821301@sun.com> Message-ID: <1115674782460@sun.com> Signed-off-by: Tom Duffy diff -Nur -X /home/tduffy/dontdiff linux-kernel10/dat/dat.h linux-kernel11/dat/dat.h --- linux-kernel10/dat/dat.h 2005-05-09 12:40:19.541004000 -0700 +++ linux-kernel11/dat/dat.h 2005-05-09 12:47:11.494001000 -0700 @@ -689,9 +689,9 @@ /* PZ Parameters */ -typedef struct dat_pz_param { +struct dat_pz_param { DAT_IA_HANDLE ia_handle; -} DAT_PZ_PARAM; +}; /* PSP Parameters */ @@ -1238,7 +1238,7 @@ typedef DAT_RETURN (*DAT_PZ_CREATE_FUNC) (DAT_IA_HANDLE, DAT_PZ_HANDLE *); -typedef DAT_RETURN (*DAT_PZ_QUERY_FUNC) (DAT_PZ_HANDLE, DAT_PZ_PARAM *); +typedef DAT_RETURN (*DAT_PZ_QUERY_FUNC) (DAT_PZ_HANDLE, struct dat_pz_param *); typedef DAT_RETURN (*DAT_PZ_FREE_FUNC) (DAT_PZ_HANDLE); @@ -1766,7 +1766,8 @@ return DAT_CALL_PROVIDER_FUNC(pz_create_func, ia, pz); } -static inline DAT_RETURN dat_pz_query(DAT_PZ_HANDLE pz, DAT_PZ_PARAM * param) +static inline DAT_RETURN dat_pz_query(DAT_PZ_HANDLE pz, + struct dat_pz_param *param) { return DAT_CALL_PROVIDER_FUNC(pz_query_func, pz, param); } diff -Nur -X /home/tduffy/dontdiff linux-kernel10/dat-provider/dapl.h linux-kernel11/dat-provider/dapl.h --- linux-kernel10/dat-provider/dapl.h 2005-05-09 12:39:52.484000000 -0700 +++ linux-kernel11/dat-provider/dapl.h 2005-05-09 12:46:47.275001000 -0700 @@ -722,7 +722,7 @@ DAT_PZ_HANDLE *); /* pz_handle */ extern DAT_RETURN dapl_pz_query(DAT_PZ_HANDLE, /* pz_handle */ - DAT_PZ_PARAM *); /* pz_args */ + struct dat_pz_param *); /* pz_args */ extern DAT_RETURN dapl_pz_free(DAT_PZ_HANDLE); /* pz_handle */ diff -Nur -X /home/tduffy/dontdiff linux-kernel10/dat-provider/dapl_pz_query.c linux-kernel11/dat-provider/dapl_pz_query.c --- linux-kernel10/dat-provider/dapl_pz_query.c 2005-05-04 08:19:32.637978000 -0700 +++ linux-kernel11/dat-provider/dapl_pz_query.c 2005-05-09 12:47:42.795003000 -0700 @@ -36,7 +36,7 @@ #include "dapl.h" -DAT_RETURN dapl_pz_query(DAT_PZ_HANDLE pz_handle, DAT_PZ_PARAM * pz_param) +DAT_RETURN dapl_pz_query(DAT_PZ_HANDLE pz_handle, struct dat_pz_param *pz_param) { DAPL_PZ *pz; DAT_RETURN status; diff -Nur -X /home/tduffy/dontdiff linux-kernel10/test/dapltest/test/dapl_fft_queryinfo.c linux-kernel11/test/dapltest/test/dapl_fft_queryinfo.c --- linux-kernel10/test/dapltest/test/dapl_fft_queryinfo.c 2005-05-09 12:29:25.437003000 -0700 +++ linux-kernel11/test/dapltest/test/dapl_fft_queryinfo.c 2005-05-09 12:47:58.187003000 -0700 @@ -64,7 +64,7 @@ DAT_RSP_HANDLE rsp_handle; DAT_RSP_PARAM rsp_param; DAT_PZ_HANDLE pz_handle; - DAT_PZ_PARAM pz_param; + struct dat_pz_param pz_param; DAT_LMR_HANDLE lmr_handle; DAT_LMR_PARAM lmr_param; DAT_LMR_CONTEXT lmr_context; From Tom.Duffy at Sun.COM Mon May 9 14:39:42 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 09 May 2005 14:39:42 -0700 Subject: [openib-general] [PATCH][10/26] kDAPL: Remove typedef DAT_SRQ_PARAM In-Reply-To: <11156747811508@sun.com> References: <11156747793273@sun.com> <11156747791462@sun.com> <1115674780270@sun.com> <11156747803410@sun.com> <11156747802252@sun.com> <11156747801195@sun.com> <11156747812210@sun.com> <11156747812290@sun.com> <11156747812388@sun.com> <11156747811508@sun.com> Message-ID: <11156747821301@sun.com> Signed-off-by: Tom Duffy diff -Nur -X /home/tduffy/dontdiff linux-kernel9/dat/dat.h linux-kernel10/dat/dat.h --- linux-kernel9/dat/dat.h 2005-05-09 12:37:03.881002000 -0700 +++ linux-kernel10/dat/dat.h 2005-05-09 12:40:19.541004000 -0700 @@ -676,7 +676,7 @@ DAT_COUNT low_watermark; }; -typedef struct dat_srq_param { +struct dat_srq_param { DAT_IA_HANDLE ia_handle; DAT_SRQ_STATE srq_state; DAT_PZ_HANDLE pz_handle; @@ -685,7 +685,7 @@ DAT_COUNT low_watermark; DAT_COUNT available_dto_count; DAT_COUNT outstanding_dto_count; -} DAT_SRQ_PARAM; +}; /* PZ Parameters */ @@ -1249,7 +1249,8 @@ typedef DAT_RETURN (*DAT_SRQ_FREE_FUNC) (DAT_SRQ_HANDLE); -typedef DAT_RETURN (*DAT_SRQ_QUERY_FUNC) (DAT_SRQ_HANDLE, DAT_SRQ_PARAM *); +typedef DAT_RETURN (*DAT_SRQ_QUERY_FUNC) (DAT_SRQ_HANDLE, + struct dat_srq_param *); typedef DAT_RETURN (*DAT_SRQ_RESIZE_FUNC) (DAT_SRQ_HANDLE, DAT_COUNT); @@ -1797,7 +1798,8 @@ srq_post_recv_func, srq, num_iovs, iovs, cookie); } -static inline DAT_RETURN dat_srq_query(DAT_SRQ_HANDLE srq, DAT_SRQ_PARAM * param) +static inline DAT_RETURN dat_srq_query(DAT_SRQ_HANDLE srq, + struct dat_srq_param *param) { return DAT_CALL_PROVIDER_FUNC(srq_query_func, srq, param); } diff -Nur -X /home/tduffy/dontdiff linux-kernel9/dat-provider/dapl.h linux-kernel10/dat-provider/dapl.h --- linux-kernel9/dat-provider/dapl.h 2005-05-09 12:36:45.218002000 -0700 +++ linux-kernel10/dat-provider/dapl.h 2005-05-09 12:39:52.484000000 -0700 @@ -342,7 +342,7 @@ /* DAPL_SRQ maps to DAT_SRQ_HANDLE */ struct dapl_srq { DAPL_HEADER header; - DAT_SRQ_PARAM param; + struct dat_srq_param param; atomic_t srq_ref_count; DAPL_COOKIE_BUFFER recv_buffer; atomic_t recv_count; @@ -741,7 +741,7 @@ DAT_DTO_COOKIE); /* user_cookie */ extern DAT_RETURN dapl_srq_query(DAT_SRQ_HANDLE, /* srq_handle */ - DAT_SRQ_PARAM *); /* srq_param */ + struct dat_srq_param *); /* srq_param */ extern DAT_RETURN dapl_srq_resize(DAT_SRQ_HANDLE, /* srq_handle */ DAT_COUNT); /* srq_max_recv_dto */ diff -Nur -X /home/tduffy/dontdiff linux-kernel9/dat-provider/dapl_srq_free.c linux-kernel10/dat-provider/dapl_srq_free.c --- linux-kernel9/dat-provider/dapl_srq_free.c 2005-04-27 11:38:35.594003000 -0700 +++ linux-kernel10/dat-provider/dapl_srq_free.c 2005-05-09 12:39:52.487003000 -0700 @@ -63,7 +63,7 @@ { DAPL_SRQ *srq_ptr; DAPL_IA *ia_ptr; - DAT_SRQ_PARAM *param; + struct dat_srq_param *param; DAT_RETURN dat_status; dat_status = DAT_SUCCESS; diff -Nur -X /home/tduffy/dontdiff linux-kernel9/dat-provider/dapl_srq_query.c linux-kernel10/dat-provider/dapl_srq_query.c --- linux-kernel9/dat-provider/dapl_srq_query.c 2005-05-04 08:19:32.624965000 -0700 +++ linux-kernel10/dat-provider/dapl_srq_query.c 2005-05-09 12:40:58.336002000 -0700 @@ -39,7 +39,8 @@ #include "dapl.h" -DAT_RETURN dapl_srq_query(DAT_SRQ_HANDLE srq_handle, DAT_SRQ_PARAM * srq_param) +DAT_RETURN dapl_srq_query(DAT_SRQ_HANDLE srq_handle, + struct dat_srq_param *srq_param) { DAPL_SRQ *srq_ptr; DAT_RETURN dat_status; From Tom.Duffy at Sun.COM Mon May 9 14:39:43 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 09 May 2005 14:39:43 -0700 Subject: [openib-general] [PATCH][14/26] kDAPL: Remove typedef DAT_CR_PARAM In-Reply-To: <11156747823890@sun.com> References: <11156747793273@sun.com> <11156747791462@sun.com> <1115674780270@sun.com> <11156747803410@sun.com> <11156747802252@sun.com> <11156747801195@sun.com> <11156747812210@sun.com> <11156747812290@sun.com> <11156747812388@sun.com> <11156747811508@sun.com> <11156747821301@sun.com> <1115674782460@sun.com> <11156747821275@sun.com> <11156747823890@sun.com> Message-ID: <11156747831715@sun.com> Signed-off-by: Tom Duffy diff -Nur -X /home/tduffy/dontdiff linux-kernel13/dat/dat.h linux-kernel14/dat/dat.h --- linux-kernel13/dat/dat.h 2005-05-09 12:54:15.334001000 -0700 +++ linux-kernel14/dat/dat.h 2005-05-09 12:58:35.516881000 -0700 @@ -718,7 +718,7 @@ * encode it into Private Data. */ -typedef struct dat_cr_param { +struct dat_cr_param { /* Remote IA whose Endpoint requested the connection. */ DAT_IA_ADDRESS_PTR remote_ia_address_ptr; @@ -739,7 +739,7 @@ * requested connection. */ DAT_EP_HANDLE local_ep_handle; -} DAT_CR_PARAM; +}; /************************** Events ******************************/ @@ -1125,7 +1125,7 @@ typedef DAT_RETURN (*DAT_GET_HANDLE_TYPE_FUNC) (DAT_HANDLE, DAT_HANDLE_TYPE *); -typedef DAT_RETURN (*DAT_CR_QUERY_FUNC) (DAT_CR_HANDLE, DAT_CR_PARAM *); +typedef DAT_RETURN (*DAT_CR_QUERY_FUNC) (DAT_CR_HANDLE, struct dat_cr_param *); typedef DAT_RETURN (*DAT_CR_ACCEPT_FUNC) ( DAT_CR_HANDLE, DAT_EP_HANDLE, DAT_COUNT, const DAT_PVOID); @@ -1457,7 +1457,7 @@ } static inline DAT_RETURN dat_cr_query(DAT_CR_HANDLE cr, - DAT_CR_PARAM * param) + struct dat_cr_param *param) { return DAT_CALL_PROVIDER_FUNC(cr_query_func, cr, param); } diff -Nur -X /home/tduffy/dontdiff linux-kernel13/dat-provider/dapl_cr_query.c linux-kernel14/dat-provider/dapl_cr_query.c --- linux-kernel13/dat-provider/dapl_cr_query.c 2005-05-04 08:19:32.847977000 -0700 +++ linux-kernel14/dat-provider/dapl_cr_query.c 2005-05-09 12:59:17.398973000 -0700 @@ -34,7 +34,7 @@ #include "dapl.h" #include "dapl_adapter_util.h" -DAT_RETURN dapl_cr_query(DAT_CR_HANDLE cr_handle, DAT_CR_PARAM * cr_param) +DAT_RETURN dapl_cr_query(DAT_CR_HANDLE cr_handle, struct dat_cr_param *cr_param) { DAPL_CR *cr_ptr; DAT_RETURN status; @@ -60,7 +60,7 @@ &cr_ptr->remote_ia_address); /* since the arguments are easily accessible, ignore the mask */ - memcpy(cr_param, &cr_ptr->param, sizeof(DAT_CR_PARAM)); + memcpy(cr_param, &cr_ptr->param, sizeof *cr_param); status = DAT_SUCCESS; diff -Nur -X /home/tduffy/dontdiff linux-kernel13/dat-provider/dapl.h linux-kernel14/dat-provider/dapl.h --- linux-kernel13/dat-provider/dapl.h 2005-05-09 12:53:48.318003000 -0700 +++ linux-kernel14/dat-provider/dapl.h 2005-05-09 12:57:28.613033000 -0700 @@ -405,12 +405,12 @@ struct dapl_cr { DAPL_HEADER header; - /* for convenience the data is kept as a DAT_CR_PARAM. + /* for convenience the data is kept as a struct dat_cr_param. * however, the "local_endpoint" field is always NULL * so this wastes a pointer. This is probably ok to * simplify code, espedially dat_cr_query. */ - DAT_CR_PARAM param; + struct dat_cr_param param; /* IB specific fields */ ib_cm_handle_t ib_cm_handle; @@ -501,7 +501,7 @@ /* CR Functions */ -extern DAT_RETURN dapl_cr_query(DAT_CR_HANDLE, DAT_CR_PARAM *); +extern DAT_RETURN dapl_cr_query(DAT_CR_HANDLE, struct dat_cr_param *); extern DAT_RETURN dapl_cr_accept(DAT_CR_HANDLE, /* cr_handle */ DAT_EP_HANDLE, /* ep_handle */ diff -Nur -X /home/tduffy/dontdiff linux-kernel13/test/dapltest/test/dapl_transaction_test.c linux-kernel14/test/dapltest/test/dapl_transaction_test.c --- linux-kernel13/test/dapltest/test/dapl_transaction_test.c 2005-05-09 12:23:41.376001000 -0700 +++ linux-kernel14/test/dapltest/test/dapl_transaction_test.c 2005-05-09 12:59:53.159964000 -0700 @@ -529,7 +529,7 @@ if (pt_ptr->local_is_server) { - DAT_CR_PARAM cr_param; + struct dat_cr_param cr_param; if (test_ptr->cmd->use_rsp) { From Tom.Duffy at Sun.COM Mon May 9 14:39:42 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 09 May 2005 14:39:42 -0700 Subject: [openib-general] [PATCH][13/26] kDAPL: Remove typedef DAT_RSP_PARAM In-Reply-To: <11156747821275@sun.com> References: <11156747793273@sun.com> <11156747791462@sun.com> <1115674780270@sun.com> <11156747803410@sun.com> <11156747802252@sun.com> <11156747801195@sun.com> <11156747812210@sun.com> <11156747812290@sun.com> <11156747812388@sun.com> <11156747811508@sun.com> <11156747821301@sun.com> <1115674782460@sun.com> <11156747821275@sun.com> Message-ID: <11156747823890@sun.com> Signed-off-by: Tom Duffy diff -Nur -X /home/tduffy/dontdiff linux-kernel12/dat/dat.h linux-kernel13/dat/dat.h --- linux-kernel12/dat/dat.h 2005-05-09 12:50:54.204001000 -0700 +++ linux-kernel13/dat/dat.h 2005-05-09 12:54:15.334001000 -0700 @@ -704,12 +704,12 @@ /* RSP Parameters */ -typedef struct dat_rsp_param { +struct dat_rsp_param { DAT_IA_HANDLE ia_handle; DAT_CONN_QUAL conn_qual; DAT_EVD_HANDLE evd_handle; DAT_EP_HANDLE ep_handle; -} DAT_RSP_PARAM; +}; /* Connection Request Parameters. * @@ -1233,7 +1233,8 @@ DAT_IA_HANDLE, DAT_CONN_QUAL, DAT_EP_HANDLE, DAT_EVD_HANDLE, DAT_RSP_HANDLE *); -typedef DAT_RETURN (*DAT_RSP_QUERY_FUNC) (DAT_RSP_HANDLE, DAT_RSP_PARAM *); +typedef DAT_RETURN (*DAT_RSP_QUERY_FUNC) (DAT_RSP_HANDLE, + struct dat_rsp_param *); typedef DAT_RETURN (*DAT_RSP_FREE_FUNC) (DAT_RSP_HANDLE); @@ -1753,7 +1754,8 @@ rsp_create_func, ia, conn_qual, ep, evd, rsp); } -static inline DAT_RETURN dat_rsp_query(DAT_RSP_HANDLE rsp, DAT_RSP_PARAM * param) +static inline DAT_RETURN dat_rsp_query(DAT_RSP_HANDLE rsp, + struct dat_rsp_param *param) { return DAT_CALL_PROVIDER_FUNC(rsp_query_func, rsp, param); } diff -Nur -X /home/tduffy/dontdiff linux-kernel12/dat-provider/dapl.h linux-kernel13/dat-provider/dapl.h --- linux-kernel12/dat-provider/dapl.h 2005-05-09 12:49:51.381002000 -0700 +++ linux-kernel13/dat-provider/dapl.h 2005-05-09 12:53:48.318003000 -0700 @@ -712,7 +712,7 @@ DAT_EVD_HANDLE, /* evd_handle */ DAT_RSP_HANDLE *); /* rsp_handle */ -extern DAT_RETURN dapl_rsp_query(DAT_RSP_HANDLE, DAT_RSP_PARAM *); +extern DAT_RETURN dapl_rsp_query(DAT_RSP_HANDLE, struct dat_rsp_param *); extern DAT_RETURN dapl_rsp_free(DAT_RSP_HANDLE); /* rsp_handle */ diff -Nur -X /home/tduffy/dontdiff linux-kernel12/dat-provider/dapl_rsp_query.c linux-kernel13/dat-provider/dapl_rsp_query.c --- linux-kernel12/dat-provider/dapl_rsp_query.c 2005-05-04 08:19:32.860993000 -0700 +++ linux-kernel13/dat-provider/dapl_rsp_query.c 2005-05-09 12:54:54.612015000 -0700 @@ -38,7 +38,8 @@ #include "dapl.h" -DAT_RETURN dapl_rsp_query(DAT_RSP_HANDLE rsp_handle, DAT_RSP_PARAM * rsp_param) +DAT_RETURN dapl_rsp_query(DAT_RSP_HANDLE rsp_handle, + struct dat_rsp_param *rsp_param) { DAPL_SP *sp_ptr; DAT_RETURN status; diff -Nur -X /home/tduffy/dontdiff linux-kernel12/test/dapltest/test/dapl_fft_queryinfo.c linux-kernel13/test/dapltest/test/dapl_fft_queryinfo.c --- linux-kernel12/test/dapltest/test/dapl_fft_queryinfo.c 2005-05-09 12:51:27.837003000 -0700 +++ linux-kernel13/test/dapltest/test/dapl_fft_queryinfo.c 2005-05-09 12:55:09.772004000 -0700 @@ -62,7 +62,7 @@ DAT_PSP_HANDLE psp_handle; struct dat_psp_param psp_param; DAT_RSP_HANDLE rsp_handle; - DAT_RSP_PARAM rsp_param; + struct dat_rsp_param rsp_param; DAT_PZ_HANDLE pz_handle; struct dat_pz_param pz_param; DAT_LMR_HANDLE lmr_handle; From Tom.Duffy at Sun.COM Mon May 9 14:39:42 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 09 May 2005 14:39:42 -0700 Subject: [openib-general] [PATCH][12/26] kDAPL: Remove typedef DAT_PSP_PARAM In-Reply-To: <1115674782460@sun.com> References: <11156747793273@sun.com> <11156747791462@sun.com> <1115674780270@sun.com> <11156747803410@sun.com> <11156747802252@sun.com> <11156747801195@sun.com> <11156747812210@sun.com> <11156747812290@sun.com> <11156747812388@sun.com> <11156747811508@sun.com> <11156747821301@sun.com> <1115674782460@sun.com> Message-ID: <11156747821275@sun.com> Signed-off-by: Tom Duffy diff -Nur -X /home/tduffy/dontdiff linux-kernel11/dat/dat.h linux-kernel12/dat/dat.h --- linux-kernel11/dat/dat.h 2005-05-09 12:47:11.494001000 -0700 +++ linux-kernel12/dat/dat.h 2005-05-09 12:50:54.204001000 -0700 @@ -695,12 +695,12 @@ /* PSP Parameters */ -typedef struct dat_psp_param { +struct dat_psp_param { DAT_IA_HANDLE ia_handle; DAT_CONN_QUAL conn_qual; DAT_EVD_HANDLE evd_handle; DAT_PSP_FLAGS psp_flags; -} DAT_PSP_PARAM; +}; /* RSP Parameters */ @@ -1224,7 +1224,8 @@ DAT_IA_HANDLE, DAT_CONN_QUAL *, DAT_EVD_HANDLE, DAT_PSP_FLAGS, DAT_PSP_HANDLE *); -typedef DAT_RETURN (*DAT_PSP_QUERY_FUNC) (DAT_PSP_HANDLE, DAT_PSP_PARAM *); +typedef DAT_RETURN (*DAT_PSP_QUERY_FUNC) (DAT_PSP_HANDLE, + struct dat_psp_param *); typedef DAT_RETURN (*DAT_PSP_FREE_FUNC) (DAT_PSP_HANDLE); @@ -1731,7 +1732,8 @@ psp_create_any_func, ia, conn_qual, evd, flags, psp); } -static inline DAT_RETURN dat_psp_query(DAT_PSP_HANDLE psp, DAT_PSP_PARAM * param) +static inline DAT_RETURN dat_psp_query(DAT_PSP_HANDLE psp, + struct dat_psp_param *param) { return DAT_CALL_PROVIDER_FUNC(psp_query_func, psp, param); } diff -Nur -X /home/tduffy/dontdiff linux-kernel11/dat-provider/dapl.h linux-kernel12/dat-provider/dapl.h --- linux-kernel11/dat-provider/dapl.h 2005-05-09 12:46:47.275001000 -0700 +++ linux-kernel12/dat-provider/dapl.h 2005-05-09 12:49:51.381002000 -0700 @@ -700,7 +700,7 @@ DAT_PSP_FLAGS, /* psp_flags */ DAT_PSP_HANDLE *); /* psp_handle */ -extern DAT_RETURN dapl_psp_query(DAT_PSP_HANDLE, DAT_PSP_PARAM *); +extern DAT_RETURN dapl_psp_query(DAT_PSP_HANDLE, struct dat_psp_param *); extern DAT_RETURN dapl_psp_free(DAT_PSP_HANDLE); /* psp_handle */ diff -Nur -X /home/tduffy/dontdiff linux-kernel11/dat-provider/dapl_psp_query.c linux-kernel12/dat-provider/dapl_psp_query.c --- linux-kernel11/dat-provider/dapl_psp_query.c 2005-05-04 08:19:32.834965000 -0700 +++ linux-kernel12/dat-provider/dapl_psp_query.c 2005-05-09 12:51:13.571014000 -0700 @@ -38,7 +38,8 @@ #include "dapl.h" -DAT_RETURN dapl_psp_query(DAT_PSP_HANDLE psp_handle, DAT_PSP_PARAM * psp_param) +DAT_RETURN dapl_psp_query(DAT_PSP_HANDLE psp_handle, + struct dat_psp_param *psp_param) { DAPL_SP *sp_ptr; DAT_RETURN status; diff -Nur -X /home/tduffy/dontdiff linux-kernel11/test/dapltest/test/dapl_fft_queryinfo.c linux-kernel12/test/dapltest/test/dapl_fft_queryinfo.c --- linux-kernel11/test/dapltest/test/dapl_fft_queryinfo.c 2005-05-09 12:47:58.187003000 -0700 +++ linux-kernel12/test/dapltest/test/dapl_fft_queryinfo.c 2005-05-09 12:51:27.837003000 -0700 @@ -60,7 +60,7 @@ #endif DAT_EVD_PARAM evd_param; DAT_PSP_HANDLE psp_handle; - DAT_PSP_PARAM psp_param; + struct dat_psp_param psp_param; DAT_RSP_HANDLE rsp_handle; DAT_RSP_PARAM rsp_param; DAT_PZ_HANDLE pz_handle; From Tom.Duffy at Sun.COM Mon May 9 14:39:43 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 09 May 2005 14:39:43 -0700 Subject: [openib-general] [PATCH][15/26] kDAPL: Remove typedef DAT_DTO_COMPLETION_EVENT_DATA In-Reply-To: <11156747831715@sun.com> References: <11156747793273@sun.com> <11156747791462@sun.com> <1115674780270@sun.com> <11156747803410@sun.com> <11156747802252@sun.com> <11156747801195@sun.com> <11156747812210@sun.com> <11156747812290@sun.com> <11156747812388@sun.com> <11156747811508@sun.com> <11156747821301@sun.com> <1115674782460@sun.com> <11156747821275@sun.com> <11156747823890@sun.com> <11156747831715@sun.com> Message-ID: <11156747831729@sun.com> Signed-off-by: Tom Duffy diff -Nur -X /home/tduffy/dontdiff linux-kernel14/dat/dat.h linux-kernel15/dat/dat.h --- linux-kernel14/dat/dat.h 2005-05-09 12:58:35.516881000 -0700 +++ linux-kernel15/dat/dat.h 2005-05-09 13:15:04.793961000 -0700 @@ -774,12 +774,12 @@ /* DTO completion event data */ /* transfered_length is not defined if status is not DAT_SUCCESS */ -typedef struct dat_dto_completion_event_data { +struct dat_dto_completion_event_data { DAT_EP_HANDLE ep_handle; DAT_DTO_COOKIE user_cookie; DAT_DTO_COMPLETION_STATUS status; DAT_VLEN transfered_length; -} DAT_DTO_COMPLETION_EVENT_DATA; +}; /* RMR bind completion event data */ typedef struct dat_rmr_bind_completion_event_data { @@ -904,7 +904,7 @@ /* Union for event Data */ typedef union dat_event_data { - DAT_DTO_COMPLETION_EVENT_DATA dto_completion_event_data; + struct dat_dto_completion_event_data dto_completion_event_data; DAT_RMR_BIND_COMPLETION_EVENT_DATA rmr_completion_event_data; DAT_CR_ARRIVAL_EVENT_DATA cr_arrival_event_data; DAT_CONNECTION_EVENT_DATA connect_event_data; diff -Nur -X /home/tduffy/dontdiff linux-kernel14/test/dapltest/include/dapl_proto.h linux-kernel15/test/dapltest/include/dapl_proto.h --- linux-kernel14/test/dapltest/include/dapl_proto.h 2005-05-09 11:34:28.736009000 -0700 +++ linux-kernel15/test/dapltest/include/dapl_proto.h 2005-05-09 13:16:08.366963000 -0700 @@ -379,24 +379,24 @@ DAT_CR_ARRIVAL_EVENT_DATA *cr_stat_p); bool DT_dto_event_reap (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, - bool poll, - DAT_DTO_COMPLETION_EVENT_DATA *dtop); + DAT_EVD_HANDLE evd_handle, + bool poll, + struct dat_dto_completion_event_data *dtop); bool DT_dto_event_wait (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, - DAT_DTO_COMPLETION_EVENT_DATA *dtop); + DAT_EVD_HANDLE evd_handle, + struct dat_dto_completion_event_data *dtop); bool DT_dto_event_poll (DT_Tdep_Print_Head* phead, - DAT_EVD_HANDLE evd_handle, - DAT_DTO_COMPLETION_EVENT_DATA *dtop); + DAT_EVD_HANDLE evd_handle, + struct dat_dto_completion_event_data *dtop); bool DT_rmr_event_wait (DT_Tdep_Print_Head* phead, DAT_EVD_HANDLE evd_handle, DAT_RMR_BIND_COMPLETION_EVENT_DATA *rmr_ptr); bool DT_dto_check ( DT_Tdep_Print_Head* phead, - DAT_DTO_COMPLETION_EVENT_DATA *dto_p, + struct dat_dto_completion_event_data *dto_p, DAT_EP_HANDLE ep_expected, DAT_COUNT len_expected, DAT_DTO_COOKIE cookie_expected, diff -Nur -X /home/tduffy/dontdiff linux-kernel14/test/dapltest/test/dapl_client.c linux-kernel15/test/dapltest/test/dapl_client.c --- linux-kernel14/test/dapltest/test/dapl_client.c 2005-05-09 12:16:33.887002000 -0700 +++ linux-kernel15/test/dapltest/test/dapl_client.c 2005-05-09 13:14:52.046961000 -0700 @@ -60,7 +60,7 @@ unsigned int retry_cnt = 0; DAT_DTO_COOKIE dto_cookie; - DAT_DTO_COMPLETION_EVENT_DATA dto_stat; + struct dat_dto_completion_event_data dto_stat; DAT_EVENT_NUMBER event_num; unsigned char * buffp; DAT_RETURN ret; diff -Nur -X /home/tduffy/dontdiff linux-kernel14/test/dapltest/test/dapl_performance_client.c linux-kernel15/test/dapltest/test/dapl_performance_client.c --- linux-kernel14/test/dapltest/test/dapl_performance_client.c 2005-05-09 11:52:15.258003000 -0700 +++ linux-kernel15/test/dapltest/test/dapl_performance_client.c 2005-05-09 13:16:26.034961000 -0700 @@ -367,7 +367,7 @@ DT_Tdep_Print_Head *phead, Performance_Test_t *test_ptr) { - DAT_DTO_COMPLETION_EVENT_DATA dto_stat; + struct dat_dto_completion_event_data dto_stat; DAT_DTO_COOKIE dto_cookie; Performance_Stats_t stats; RemoteMemoryInfo *rmi; diff -Nur -X /home/tduffy/dontdiff linux-kernel14/test/dapltest/test/dapl_performance_server.c linux-kernel15/test/dapltest/test/dapl_performance_server.c --- linux-kernel14/test/dapltest/test/dapl_performance_server.c 2005-05-05 09:42:06.502020000 -0700 +++ linux-kernel15/test/dapltest/test/dapl_performance_server.c 2005-05-09 13:16:40.541971000 -0700 @@ -279,7 +279,7 @@ DT_Tdep_Print_Head *phead, Performance_Test_t *test_ptr) { - DAT_DTO_COMPLETION_EVENT_DATA dto_stat; + struct dat_dto_completion_event_data dto_stat; RemoteMemoryInfo *rmi; DAT_DTO_COOKIE dto_cookie; diff -Nur -X /home/tduffy/dontdiff linux-kernel14/test/dapltest/test/dapl_server.c linux-kernel15/test/dapltest/test/dapl_server.c --- linux-kernel14/test/dapltest/test/dapl_server.c 2005-05-09 12:16:33.904000000 -0700 +++ linux-kernel15/test/dapltest/test/dapl_server.c 2005-05-09 13:14:52.066962000 -0700 @@ -51,7 +51,7 @@ int status = 0; DAT_DTO_COOKIE dto_cookie; - DAT_DTO_COMPLETION_EVENT_DATA dto_stat; + struct dat_dto_completion_event_data dto_stat; DAT_RETURN ret; DT_Tdep_Print_Head *phead; @@ -870,7 +870,7 @@ { unsigned char *module = "send_control_data"; DAT_DTO_COOKIE dto_cookie; - DAT_DTO_COMPLETION_EVENT_DATA dto_stat; + struct dat_dto_completion_event_data dto_stat; buffp = DT_Bpool_GetBuffer (ps_ptr->bpool, 2); /* 3rd buffer */ memcpy ( (void *)buffp, diff -Nur -X /home/tduffy/dontdiff linux-kernel14/test/dapltest/test/dapl_test_util.c linux-kernel15/test/dapltest/test/dapl_test_util.c --- linux-kernel14/test/dapltest/test/dapl_test_util.c 2005-05-09 12:31:59.724000000 -0700 +++ linux-kernel15/test/dapltest/test/dapl_test_util.c 2005-05-09 13:14:52.072963000 -0700 @@ -414,7 +414,7 @@ DT_dto_event_reap (DT_Tdep_Print_Head *phead, DAT_EVD_HANDLE evd_handle, bool poll, - DAT_DTO_COMPLETION_EVENT_DATA *dto_statusp) + struct dat_dto_completion_event_data *dto_statusp) { if (poll) { @@ -433,7 +433,7 @@ bool DT_dto_event_poll (DT_Tdep_Print_Head *phead, DAT_EVD_HANDLE evd_handle, - DAT_DTO_COMPLETION_EVENT_DATA *dto_statusp) + struct dat_dto_completion_event_data *dto_statusp) { for (;;) { @@ -485,7 +485,7 @@ bool DT_dto_event_wait (DT_Tdep_Print_Head *phead, DAT_EVD_HANDLE evd_handle, - DAT_DTO_COMPLETION_EVENT_DATA *dto_statusp) + struct dat_dto_completion_event_data *dto_statusp) { for (;;) { @@ -571,7 +571,7 @@ */ bool DT_dto_check ( DT_Tdep_Print_Head *phead, - DAT_DTO_COMPLETION_EVENT_DATA *dto_p, + struct dat_dto_completion_event_data *dto_p, DAT_EP_HANDLE ep_expected, DAT_COUNT len_expected, DAT_DTO_COOKIE cookie_expected, diff -Nur -X /home/tduffy/dontdiff linux-kernel14/test/dapltest/test/dapl_transaction_test.c linux-kernel15/test/dapltest/test/dapl_transaction_test.c --- linux-kernel14/test/dapltest/test/dapl_transaction_test.c 2005-05-09 12:59:53.159964000 -0700 +++ linux-kernel15/test/dapltest/test/dapl_transaction_test.c 2005-05-09 13:14:52.085962000 -0700 @@ -519,7 +519,7 @@ { u32 buff_size = MAX_OPS * sizeof (RemoteMemoryInfo); RemoteMemoryInfo *RemoteMemInfo; - DAT_DTO_COMPLETION_EVENT_DATA dto_stat; + struct dat_dto_completion_event_data dto_stat; DAT_CR_ARRIVAL_EVENT_DATA cr_stat; DAT_CR_HANDLE cr_handle; @@ -1371,7 +1371,7 @@ } for (i = 0; i < test_ptr->cmd->eps_per_thread; i++) { - DAT_DTO_COMPLETION_EVENT_DATA dto_stat; + struct dat_dto_completion_event_data dto_stat; if ( !DT_dto_event_wait (phead, test_ptr->reqt_evd_hdl, @@ -1388,7 +1388,7 @@ test_ptr->base_port)); for (i = 0; i < test_ptr->cmd->eps_per_thread; i++) { - DAT_DTO_COMPLETION_EVENT_DATA dto_stat; + struct dat_dto_completion_event_data dto_stat; if ( !DT_dto_event_wait (phead, test_ptr->recv_evd_hdl, @@ -1409,7 +1409,7 @@ test_ptr->base_port)); for (i = 0; i < test_ptr->cmd->eps_per_thread; i++) { - DAT_DTO_COMPLETION_EVENT_DATA dto_stat; + struct dat_dto_completion_event_data dto_stat; if ( !DT_dto_event_wait (phead, test_ptr->recv_evd_hdl, @@ -1446,7 +1446,7 @@ } for (i = 0; i < test_ptr->cmd->eps_per_thread; i++) { - DAT_DTO_COMPLETION_EVENT_DATA dto_stat; + struct dat_dto_completion_event_data dto_stat; if ( !DT_dto_event_wait (phead, test_ptr->reqt_evd_hdl, diff -Nur -X /home/tduffy/dontdiff linux-kernel14/test/dapltest/test/dapl_transaction_util.c linux-kernel15/test/dapltest/test/dapl_transaction_util.c --- linux-kernel14/test/dapltest/test/dapl_transaction_util.c 2005-05-09 11:52:52.357004000 -0700 +++ linux-kernel15/test/dapltest/test/dapl_transaction_util.c 2005-05-09 13:14:52.091965000 -0700 @@ -157,7 +157,7 @@ for (i = 0; i < num_eps; i++) { Transaction_Test_Op_t *op; - DAT_DTO_COMPLETION_EVENT_DATA dto_stat; + struct dat_dto_completion_event_data dto_stat; DAT_DTO_COOKIE dto_cookie; unsigned int epnum; @@ -265,7 +265,7 @@ for (i = 0; i < num_eps; i++) { Transaction_Test_Op_t *op; - DAT_DTO_COMPLETION_EVENT_DATA dto_stat; + struct dat_dto_completion_event_data dto_stat; DAT_DTO_COOKIE dto_cookie; unsigned int epnum; @@ -558,7 +558,7 @@ for (i = 0; i < num_eps; i++) { Transaction_Test_Op_t *op; - DAT_DTO_COMPLETION_EVENT_DATA dto_stat; + struct dat_dto_completion_event_data dto_stat; DAT_DTO_COOKIE dto_cookie; unsigned int epnum; From Tom.Duffy at Sun.COM Mon May 9 14:39:44 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 09 May 2005 14:39:44 -0700 Subject: [openib-general] [PATCH][18/26] kDAPL: Remove typedef DAT_CONNECTION_EVENT_DATA In-Reply-To: <11156747832991@sun.com> References: <11156747793273@sun.com> <11156747791462@sun.com> <1115674780270@sun.com> <11156747803410@sun.com> <11156747802252@sun.com> <11156747801195@sun.com> <11156747812210@sun.com> <11156747812290@sun.com> <11156747812388@sun.com> <11156747811508@sun.com> <11156747821301@sun.com> <1115674782460@sun.com> <11156747821275@sun.com> <11156747823890@sun.com> <11156747831715@sun.com> <11156747831729@sun.com> <11156747831783@sun.com> <11156747832991@sun.com> Message-ID: <11156747841537@sun.com> Signed-off-by: Tom Duffy diff -Nur -X /home/tduffy/dontdiff linux-kernel17/dat/dat.h linux-kernel18/dat/dat.h --- linux-kernel17/dat/dat.h 2005-05-09 13:23:46.473960000 -0700 +++ linux-kernel18/dat/dat.h 2005-05-09 13:26:21.135961000 -0700 @@ -820,11 +820,11 @@ }; /* Connection event data */ -typedef struct dat_connection_event_data { +struct dat_connection_event_data { DAT_EP_HANDLE ep_handle; DAT_COUNT private_data_size; DAT_PVOID private_data; -} DAT_CONNECTION_EVENT_DATA; +}; /* Async Error event data */ @@ -907,7 +907,7 @@ struct dat_dto_completion_event_data dto_completion_event_data; struct dat_rmr_bind_completion_event_data rmr_completion_event_data; struct dat_cr_arrival_event_data cr_arrival_event_data; - DAT_CONNECTION_EVENT_DATA connect_event_data; + struct dat_connection_event_data connect_event_data; DAT_ASYNCH_ERROR_EVENT_DATA asynch_error_event_data; DAT_SOFTWARE_EVENT_DATA software_event_data; } DAT_EVENT_DATA; diff -Nur -X /home/tduffy/dontdiff linux-kernel17/test/dapltest/test/dapl_test_util.c linux-kernel18/test/dapltest/test/dapl_test_util.c --- linux-kernel17/test/dapltest/test/dapl_test_util.c 2005-05-09 13:23:19.258962000 -0700 +++ linux-kernel18/test/dapltest/test/dapl_test_util.c 2005-05-09 13:25:54.690963000 -0700 @@ -340,7 +340,7 @@ if (event.event_number == DAT_CONNECTION_EVENT_ESTABLISHED) { /* - * Could return DAT_CONNECTION_EVENT_DATA and verify: + * Could return struct dat_connection_event_data and verify: * event.event_data.connect_event_data.ep_handle * event.event_data.connect_event_data.private_data_size * event.event_data.connect_event_data.private_data From Tom.Duffy at Sun.COM Mon May 9 14:39:44 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 09 May 2005 14:39:44 -0700 Subject: [openib-general] [PATCH][19/26] kDAPL: Remove typedef DAT_ASYNCH_ERROR_EVENT_DATA In-Reply-To: <11156747841537@sun.com> References: <11156747793273@sun.com> <11156747791462@sun.com> <1115674780270@sun.com> <11156747803410@sun.com> <11156747802252@sun.com> <11156747801195@sun.com> <11156747812210@sun.com> <11156747812290@sun.com> <11156747812388@sun.com> <11156747811508@sun.com> <11156747821301@sun.com> <1115674782460@sun.com> <11156747821275@sun.com> <11156747823890@sun.com> <11156747831715@sun.com> <11156747831729@sun.com> <11156747831783@sun.com> <11156747832991@sun.com> <11156747841537@sun.com> Message-ID: <11156747843960@sun.com> Signed-off-by: Tom Duffy diff -Nur -X /home/tduffy/dontdiff linux-kernel18/dat/dat.h linux-kernel19/dat/dat.h --- linux-kernel18/dat/dat.h 2005-05-09 13:26:21.135961000 -0700 +++ linux-kernel19/dat/dat.h 2005-05-09 13:28:13.273961000 -0700 @@ -835,11 +835,11 @@ * affiliated asynchronous event dat_handle is either lmr_handle, * rmr_handle or pz_handle. */ -typedef struct dat_asynch_error_event_data { +struct dat_asynch_error_event_data { DAT_HANDLE dat_handle; /* either IA, EP, EVD, SRQ, */ /* LMR, RMR, or PZ handle */ DAT_COUNT reason; /* object specific */ -} DAT_ASYNCH_ERROR_EVENT_DATA; +}; /* The reason is object type specific and its values are defined below. */ typedef enum ia_async_error_reason { @@ -908,7 +908,7 @@ struct dat_rmr_bind_completion_event_data rmr_completion_event_data; struct dat_cr_arrival_event_data cr_arrival_event_data; struct dat_connection_event_data connect_event_data; - DAT_ASYNCH_ERROR_EVENT_DATA asynch_error_event_data; + struct dat_asynch_error_event_data asynch_error_event_data; DAT_SOFTWARE_EVENT_DATA software_event_data; } DAT_EVENT_DATA; From Tom.Duffy at Sun.COM Mon May 9 14:39:43 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 09 May 2005 14:39:43 -0700 Subject: [openib-general] [PATCH][16/26] kDAPL: Remove typedef DAT_RMR_BIND_COMPLETION_EVENT_DATA In-Reply-To: <11156747831729@sun.com> References: <11156747793273@sun.com> <11156747791462@sun.com> <1115674780270@sun.com> <11156747803410@sun.com> <11156747802252@sun.com> <11156747801195@sun.com> <11156747812210@sun.com> <11156747812290@sun.com> <11156747812388@sun.com> <11156747811508@sun.com> <11156747821301@sun.com> <1115674782460@sun.com> <11156747821275@sun.com> <11156747823890@sun.com> <11156747831715@sun.com> <11156747831729@sun.com> Message-ID: <11156747831783@sun.com> Signed-off-by: Tom Duffy diff -Nur -X /home/tduffy/dontdiff linux-kernel15/dat/dat.h linux-kernel16/dat/dat.h --- linux-kernel15/dat/dat.h 2005-05-09 13:15:04.793961000 -0700 +++ linux-kernel16/dat/dat.h 2005-05-09 13:19:22.189961000 -0700 @@ -782,11 +782,11 @@ }; /* RMR bind completion event data */ -typedef struct dat_rmr_bind_completion_event_data { +struct dat_rmr_bind_completion_event_data { DAT_RMR_HANDLE rmr_handle; DAT_RMR_COOKIE user_cookie; DAT_RMR_BIND_COMPLETION_STATUS status; -} DAT_RMR_BIND_COMPLETION_EVENT_DATA; +}; typedef union dat_sp_handle { DAT_RSP_HANDLE rsp_handle; @@ -905,7 +905,7 @@ typedef union dat_event_data { struct dat_dto_completion_event_data dto_completion_event_data; - DAT_RMR_BIND_COMPLETION_EVENT_DATA rmr_completion_event_data; + struct dat_rmr_bind_completion_event_data rmr_completion_event_data; DAT_CR_ARRIVAL_EVENT_DATA cr_arrival_event_data; DAT_CONNECTION_EVENT_DATA connect_event_data; DAT_ASYNCH_ERROR_EVENT_DATA asynch_error_event_data; diff -Nur -X /home/tduffy/dontdiff linux-kernel15/test/dapltest/include/dapl_proto.h linux-kernel16/test/dapltest/include/dapl_proto.h --- linux-kernel15/test/dapltest/include/dapl_proto.h 2005-05-09 13:16:08.366963000 -0700 +++ linux-kernel16/test/dapltest/include/dapl_proto.h 2005-05-09 13:20:14.136964000 -0700 @@ -393,7 +393,7 @@ bool DT_rmr_event_wait (DT_Tdep_Print_Head* phead, DAT_EVD_HANDLE evd_handle, - DAT_RMR_BIND_COMPLETION_EVENT_DATA *rmr_ptr); + struct dat_rmr_bind_completion_event_data *rmr_ptr); bool DT_dto_check ( DT_Tdep_Print_Head* phead, struct dat_dto_completion_event_data *dto_p, @@ -403,7 +403,7 @@ char *message); bool DT_rmr_check ( DT_Tdep_Print_Head* phead, - DAT_RMR_BIND_COMPLETION_EVENT_DATA*rmr_p, + struct dat_rmr_bind_completion_event_data *rmr_p, DAT_RMR_HANDLE rmr_expected, DAT_PVOID cookie_expected, char *message); diff -Nur -X /home/tduffy/dontdiff linux-kernel15/test/dapltest/test/dapl_bpool.c linux-kernel16/test/dapltest/test/dapl_bpool.c --- linux-kernel15/test/dapltest/test/dapl_bpool.c 2005-05-09 11:34:28.741036000 -0700 +++ linux-kernel16/test/dapltest/test/dapl_bpool.c 2005-05-09 13:19:10.558962000 -0700 @@ -210,7 +210,7 @@ struct dat_lmr_triplet iov; DAT_RMR_COOKIE cookie; DAT_MEM_PRIV_FLAGS mflags; - DAT_RMR_BIND_COMPLETION_EVENT_DATA rmr_stat; + struct dat_rmr_bind_completion_event_data rmr_stat; /* create the RMR */ ret = dat_rmr_create (pz_handle, &bpool_ptr->rmr_handle); diff -Nur -X /home/tduffy/dontdiff linux-kernel15/test/dapltest/test/dapl_test_util.c linux-kernel16/test/dapltest/test/dapl_test_util.c --- linux-kernel15/test/dapltest/test/dapl_test_util.c 2005-05-09 13:14:52.072963000 -0700 +++ linux-kernel16/test/dapltest/test/dapl_test_util.c 2005-05-09 13:19:10.563961000 -0700 @@ -529,7 +529,7 @@ bool DT_rmr_event_wait (DT_Tdep_Print_Head *phead, DAT_EVD_HANDLE evd_handle, - DAT_RMR_BIND_COMPLETION_EVENT_DATA *rmr_statusp) + struct dat_rmr_bind_completion_event_data *rmr_statusp) { for (;;) { @@ -619,7 +619,7 @@ */ bool DT_rmr_check ( DT_Tdep_Print_Head *phead, - DAT_RMR_BIND_COMPLETION_EVENT_DATA *rmr_p, + struct dat_rmr_bind_completion_event_data *rmr_p, DAT_RMR_HANDLE rmr_expected, DAT_PVOID cookie_expected, char *message) From Tom.Duffy at Sun.COM Mon May 9 14:39:44 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 09 May 2005 14:39:44 -0700 Subject: [openib-general] [PATCH][21/26] kDAPL: Remove typedef DAT_EVENT In-Reply-To: <11156747844081@sun.com> References: <11156747793273@sun.com> <11156747791462@sun.com> <1115674780270@sun.com> <11156747803410@sun.com> <11156747802252@sun.com> <11156747801195@sun.com> <11156747812210@sun.com> <11156747812290@sun.com> <11156747812388@sun.com> <11156747811508@sun.com> <11156747821301@sun.com> <1115674782460@sun.com> <11156747821275@sun.com> <11156747823890@sun.com> <11156747831715@sun.com> <11156747831729@sun.com> <11156747831783@sun.com> <11156747832991@sun.com> <11156747841537@sun.com> <11156747843960@sun.com> <11156747844081@sun.com> Message-ID: <11156747842198@sun.com> Signed-off-by: Tom Duffy diff -Nur -X /home/tduffy/dontdiff linux-kernel20/dat/dat.h linux-kernel21/dat/dat.h --- linux-kernel20/dat/dat.h 2005-05-09 13:29:50.153961000 -0700 +++ linux-kernel21/dat/dat.h 2005-05-09 13:32:35.923960000 -0700 @@ -914,11 +914,11 @@ /* Event struct that holds all event information */ -typedef struct dat_event { +struct dat_event { DAT_EVENT_NUMBER event_number; DAT_EVD_HANDLE evd_handle; DAT_EVENT_DATA event_data; -} DAT_EVENT; +}; /* Upcall support */ @@ -931,7 +931,8 @@ /* UpCalls return */ } DAT_UPCALL_POLICY; -typedef void (*DAT_UPCALL_FUNC) (DAT_PVOID, const DAT_EVENT *, DAT_BOOLEAN); +typedef void (*DAT_UPCALL_FUNC) (DAT_PVOID, const struct dat_event *, + DAT_BOOLEAN); typedef struct dat_upcall_object { DAT_PVOID instance_data; @@ -1136,9 +1137,10 @@ typedef DAT_RETURN (*DAT_EVD_RESIZE_FUNC) (DAT_EVD_HANDLE, DAT_COUNT); -typedef DAT_RETURN (*DAT_EVD_POST_SE_FUNC) (DAT_EVD_HANDLE, const DAT_EVENT *); +typedef DAT_RETURN (*DAT_EVD_POST_SE_FUNC) (DAT_EVD_HANDLE, + const struct dat_event *); -typedef DAT_RETURN (*DAT_EVD_DEQUEUE_FUNC) (DAT_EVD_HANDLE, DAT_EVENT *); +typedef DAT_RETURN (*DAT_EVD_DEQUEUE_FUNC) (DAT_EVD_HANDLE, struct dat_event *); typedef DAT_RETURN (*DAT_EVD_FREE_FUNC) (DAT_EVD_HANDLE); @@ -1468,7 +1470,7 @@ } static inline DAT_RETURN dat_evd_dequeue(DAT_EVD_HANDLE evd, - DAT_EVENT * event) + struct dat_event *event) { return DAT_CALL_PROVIDER_FUNC(evd_dequeue_func, evd, event); } @@ -1498,7 +1500,7 @@ } static inline DAT_RETURN dat_evd_post_se(DAT_EVD_HANDLE evd, - const DAT_EVENT * event) + const struct dat_event *event) { return DAT_CALL_PROVIDER_FUNC(evd_post_se_func, evd, event); } diff -Nur -X /home/tduffy/dontdiff linux-kernel20/dat-provider/dapl_cno_util.c linux-kernel21/dat-provider/dapl_cno_util.c --- linux-kernel20/dat-provider/dapl_cno_util.c 2005-05-04 13:38:41.244002000 -0700 +++ linux-kernel21/dat-provider/dapl_cno_util.c 2005-05-09 13:39:21.136968000 -0700 @@ -114,7 +114,7 @@ DAPL_EVD *evd_ptr) { DAT_RETURN dat_status; - DAT_EVENT event; + struct dat_event event; dat_status = DAT_SUCCESS; diff -Nur -X /home/tduffy/dontdiff linux-kernel20/dat-provider/dapl_evd_dequeue.c linux-kernel21/dat-provider/dapl_evd_dequeue.c --- linux-kernel20/dat-provider/dapl_evd_dequeue.c 2005-04-20 12:42:07.511003000 -0700 +++ linux-kernel21/dat-provider/dapl_evd_dequeue.c 2005-05-09 13:33:41.965966000 -0700 @@ -62,10 +62,10 @@ * DAT_QUEUE_EMPTY */ -DAT_RETURN dapl_evd_dequeue(DAT_EVD_HANDLE evd_handle, DAT_EVENT * event) +DAT_RETURN dapl_evd_dequeue(DAT_EVD_HANDLE evd_handle, struct dat_event *event) { DAPL_EVD *evd_ptr; - DAT_EVENT *local_event; + struct dat_event *local_event; DAT_RETURN dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, @@ -108,7 +108,7 @@ * from CQ to EVD. */ local_event = - (DAT_EVENT *) dapl_rbuf_remove(&evd_ptr->pending_event_queue); + (struct dat_event *)dapl_rbuf_remove(&evd_ptr->pending_event_queue); if (local_event != NULL) { *event = *local_event; dat_status = dapl_rbuf_add(&evd_ptr->free_event_queue, diff -Nur -X /home/tduffy/dontdiff linux-kernel20/dat-provider/dapl_evd_post_se.c linux-kernel21/dat-provider/dapl_evd_post_se.c --- linux-kernel20/dat-provider/dapl_evd_post_se.c 2005-04-20 12:42:07.215014000 -0700 +++ linux-kernel21/dat-provider/dapl_evd_post_se.c 2005-05-09 13:34:05.073960000 -0700 @@ -59,7 +59,8 @@ * DAT_INVALID_PARAMETER */ -DAT_RETURN dapl_evd_post_se(DAT_EVD_HANDLE evd_handle, const DAT_EVENT * event) +DAT_RETURN dapl_evd_post_se(DAT_EVD_HANDLE evd_handle, + const struct dat_event *event) { DAPL_EVD *evd_ptr; DAT_RETURN dat_status; diff -Nur -X /home/tduffy/dontdiff linux-kernel20/dat-provider/dapl_evd_util.c linux-kernel21/dat-provider/dapl_evd_util.c --- linux-kernel20/dat-provider/dapl_evd_util.c 2005-05-09 11:07:02.062017000 -0700 +++ linux-kernel21/dat-provider/dapl_evd_util.c 2005-05-09 13:41:09.036964000 -0700 @@ -176,7 +176,7 @@ */ DAT_RETURN dapl_evd_event_alloc(DAPL_EVD * evd_ptr, DAT_COUNT qlen) { - DAT_EVENT *event_ptr; + struct dat_event *event_ptr; DAT_COUNT i; DAT_RETURN dat_status; @@ -184,7 +184,8 @@ /* Allocate EVENTs */ event_ptr = - (DAT_EVENT *) kmalloc(evd_ptr->qlen * sizeof(DAT_EVENT), GFP_ATOMIC); + (struct dat_event *)kmalloc(evd_ptr->qlen * sizeof *event_ptr, + GFP_ATOMIC); if (event_ptr == NULL) { dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); @@ -219,15 +220,15 @@ */ DAT_RETURN dapl_evd_event_realloc(DAPL_EVD * evd_ptr, DAT_COUNT qlen) { - DAT_EVENT *events; + struct dat_event *events; DAT_COUNT old_qlen; DAT_COUNT i; intptr_t diff; DAT_RETURN dat_status; /* Allocate EVENTs */ - events = (DAT_EVENT *) dapl_os_realloc(evd_ptr->events, - qlen * sizeof (DAT_EVENT)); + events = (struct dat_event *)dapl_os_realloc(evd_ptr->events, + qlen * sizeof *events); if ( NULL == events ) { dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); @@ -392,15 +393,15 @@ * that the lock is held. */ -static DAT_EVENT *dapl_evd_get_event(DAPL_EVD * evd_ptr) +static struct dat_event *dapl_evd_get_event(DAPL_EVD * evd_ptr) { - DAT_EVENT *event; + struct dat_event *event; if (evd_ptr->evd_producer_locking_needed) { spin_lock(&evd_ptr->header.lock); } - event = (DAT_EVENT *) dapl_rbuf_remove(&evd_ptr->free_event_queue); + event = (struct dat_event *) dapl_rbuf_remove(&evd_ptr->free_event_queue); /* Release the lock if it was taken and the call failed. */ if (!event && evd_ptr->evd_producer_locking_needed) { @@ -418,7 +419,7 @@ * entry to this function. */ -static void dapl_evd_post_event(DAPL_EVD * evd_ptr, const DAT_EVENT * event_ptr) +static void dapl_evd_post_event(DAPL_EVD * evd_ptr, const struct dat_event * event_ptr) { DAT_RETURN dat_status; DAPL_CNO * cno_to_trigger = NULL; @@ -450,7 +451,7 @@ * format an overflow event for posting */ static void -dapl_evd_format_overflow_event(DAPL_EVD * evd_ptr, DAT_EVENT * event_ptr) +dapl_evd_format_overflow_event(DAPL_EVD * evd_ptr, struct dat_event * event_ptr) { DAPL_IA *ia_ptr; @@ -469,7 +470,7 @@ dapl_evd_post_overflow_event(DAPL_EVD * async_evd_ptr, DAPL_EVD * overflow_evd_ptr) { - DAT_EVENT *overflow_event; + struct dat_event *overflow_event; /* The overflow_evd_ptr mght be the same as evd. * In that case we've got a catastrophic overflow. @@ -493,10 +494,10 @@ return; } -static DAT_EVENT *dapl_evd_get_and_init_event(DAPL_EVD * evd_ptr, +static struct dat_event *dapl_evd_get_and_init_event(DAPL_EVD * evd_ptr, DAT_EVENT_NUMBER event_number) { - DAT_EVENT *event_ptr; + struct dat_event *event_ptr; event_ptr = dapl_evd_get_event(evd_ptr); if (NULL == event_ptr) { @@ -517,7 +518,7 @@ DAT_IA_ADDRESS_PTR ia_address_ptr, DAT_CONN_QUAL conn_qual, DAT_CR_HANDLE cr_handle) { - DAT_EVENT *event_ptr; + struct dat_event *event_ptr; event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number); /* * Note event lock may be held on successful return @@ -548,7 +549,7 @@ DAT_COUNT private_data_size, DAT_PVOID private_data) { - DAT_EVENT *event_ptr; + struct dat_event *event_ptr; event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number); /* * Note event lock may be held on successful return @@ -576,7 +577,7 @@ DAT_EVENT_NUMBER event_number, DAT_IA_HANDLE ia_handle) { - DAT_EVENT *event_ptr; + struct dat_event *event_ptr; event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number); /* * Note event lock may be held on successful return @@ -601,7 +602,7 @@ dapl_evd_post_software_event(DAPL_EVD * evd_ptr, DAT_EVENT_NUMBER event_number, DAT_PVOID pointer) { - DAT_EVENT *event_ptr; + struct dat_event *event_ptr; event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number); /* * Note event lock may be held on successful return @@ -638,7 +639,8 @@ * */ static void -dapl_evd_cqe_to_event(DAPL_EVD * evd_ptr, void *cqe_ptr, DAT_EVENT * event_ptr) +dapl_evd_cqe_to_event(DAPL_EVD * evd_ptr, void *cqe_ptr, + struct dat_event *event_ptr) { DAPL_EP *ep_ptr; DAPL_COOKIE *cookie; @@ -773,7 +775,8 @@ * Status of operation * */ -DAT_RETURN dapl_evd_cq_poll_to_event(DAPL_EVD * evd_ptr, DAT_EVENT * event) +DAT_RETURN dapl_evd_cq_poll_to_event(DAPL_EVD * evd_ptr, + struct dat_event *event) { DAT_RETURN dat_status; ib_work_completion_t cur_cqe; diff -Nur -X /home/tduffy/dontdiff linux-kernel20/dat-provider/dapl_evd_util.h linux-kernel21/dat-provider/dapl_evd_util.h --- linux-kernel20/dat-provider/dapl_evd_util.h 2005-05-09 11:07:02.080018000 -0700 +++ linux-kernel21/dat-provider/dapl_evd_util.h 2005-05-09 13:36:32.543960000 -0700 @@ -112,6 +112,6 @@ void *context); extern DAT_RETURN dapl_evd_cq_poll_to_event(DAPL_EVD * evd_ptr, - DAT_EVENT * event); + struct dat_event *event); #endif diff -Nur -X /home/tduffy/dontdiff linux-kernel20/dat-provider/dapl.h linux-kernel21/dat-provider/dapl.h --- linux-kernel20/dat-provider/dapl.h 2005-05-09 12:57:28.613033000 -0700 +++ linux-kernel21/dat-provider/dapl.h 2005-05-09 13:37:15.817963000 -0700 @@ -177,7 +177,8 @@ typedef void (*DAPL_CONNECTION_STATE_HANDLER) (DAPL_EP *, ib_cm_events_t, - const void *, DAT_EVENT *); + const void *, + struct dat_event *); /********************************************************************* * * @@ -284,7 +285,7 @@ /* the actual events */ DAT_COUNT qlen; - DAT_EVENT *events; + struct dat_event *events; DAPL_RING_BUFFER free_event_queue; DAPL_RING_BUFFER pending_event_queue; DAPL_CNO *cno_ptr; @@ -543,10 +544,10 @@ DAT_COUNT); /* evd_qlen */ extern DAT_RETURN dapl_evd_post_se(DAT_EVD_HANDLE, /* evd_handle */ - const DAT_EVENT *); /* event */ + const struct dat_event *); /* event */ extern DAT_RETURN dapl_evd_dequeue(DAT_EVD_HANDLE, /* evd_handle */ - DAT_EVENT *); /* event */ + struct dat_event *); /* event */ extern DAT_RETURN dapl_evd_free(DAT_EVD_HANDLE); diff -Nur -X /home/tduffy/dontdiff linux-kernel20/test/dapltest/include/dapl_fft_util.h linux-kernel21/test/dapltest/include/dapl_fft_util.h --- linux-kernel20/test/dapltest/include/dapl_fft_util.h 2005-05-09 13:23:19.226964000 -0700 +++ linux-kernel21/test/dapltest/include/dapl_fft_util.h 2005-05-09 13:31:50.574963000 -0700 @@ -60,7 +60,7 @@ DAT_PSP_HANDLE psp_handle; DAT_EP_HANDLE ep_handle; DAT_EVD_HANDLE cr_evd, conn_evd, send_evd, recv_evd; - DAT_EVENT event; + struct dat_event event; DAT_COUNT count; DAT_CR_HANDLE cr_handle; Bpool *bpool; diff -Nur -X /home/tduffy/dontdiff linux-kernel20/test/dapltest/include/dapl_tdep.h linux-kernel21/test/dapltest/include/dapl_tdep.h --- linux-kernel20/test/dapltest/include/dapl_tdep.h 2005-05-05 09:42:06.664001000 -0700 +++ linux-kernel21/test/dapltest/include/dapl_tdep.h 2005-05-09 13:42:35.094961000 -0700 @@ -57,9 +57,9 @@ DAT_RETURN DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_handle, DAT_TIMEOUT timeout, - DAT_EVENT *event); + struct dat_event *event); DAT_RETURN DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_handle, - DAT_EVENT *event); + struct dat_event *event); #endif diff -Nur -X /home/tduffy/dontdiff linux-kernel20/test/dapltest/kdapl/kdapl_tdep_evd.c linux-kernel21/test/dapltest/kdapl/kdapl_tdep_evd.c --- linux-kernel20/test/dapltest/kdapl/kdapl_tdep_evd.c 2005-05-05 13:35:54.888008000 -0700 +++ linux-kernel21/test/dapltest/kdapl/kdapl_tdep_evd.c 2005-05-09 13:38:10.908963000 -0700 @@ -30,7 +30,7 @@ typedef struct Tdep_Event_Tag { - DAT_EVENT dat_event; + struct dat_event dat_event; struct Tdep_Event_Tag *event_next; } Tdep_Event; @@ -46,7 +46,7 @@ /* static function prototypes */ static void DT_Tdep_Event_Callback ( DAT_PVOID instance_data, - const DAT_EVENT * dat_event, + const struct dat_event *dat_event, DAT_BOOLEAN bool_arg); /* static data */ @@ -156,7 +156,7 @@ } DAT_RETURN DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_handle, - DAT_EVENT *dat_event) + struct dat_event *dat_event) { DAT_RETURN dat_status; Tdep_Evd *evd_ptr; @@ -215,7 +215,7 @@ DAT_RETURN DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_handle, DAT_TIMEOUT timeout, - DAT_EVENT *dat_event) + struct dat_event *dat_event) { DAT_RETURN dat_status; Tdep_Evd *evd_ptr; @@ -323,7 +323,7 @@ static void DT_Tdep_Event_Callback ( DAT_PVOID instance_data, - const DAT_EVENT * dat_event, + const struct dat_event * dat_event, DAT_BOOLEAN bool_arg) { Tdep_Event *event; diff -Nur -X /home/tduffy/dontdiff linux-kernel20/test/dapltest/test/dapl_client.c linux-kernel21/test/dapltest/test/dapl_client.c --- linux-kernel20/test/dapltest/test/dapl_client.c 2005-05-09 13:14:52.046961000 -0700 +++ linux-kernel21/test/dapltest/test/dapl_client.c 2005-05-09 13:31:50.593960000 -0700 @@ -275,7 +275,7 @@ { if ( event_num == DAT_CONNECTION_EVENT_PEER_REJECTED ) { - DAT_EVENT event; + struct dat_event event; DAT_COUNT drained = 0; DT_Mdep_Sleep (1000); @@ -549,7 +549,7 @@ /* Free the EP */ if (ep_handle) { - DAT_EVENT event; + struct dat_event event; /* * Drain off outstanding DTOs that may have been * generated by racing disconnects diff -Nur -X /home/tduffy/dontdiff linux-kernel20/test/dapltest/test/dapl_fft_endpoint.c linux-kernel21/test/dapltest/test/dapl_fft_endpoint.c --- linux-kernel20/test/dapltest/test/dapl_fft_endpoint.c 2005-05-04 09:45:07.529005000 -0700 +++ linux-kernel21/test/dapltest/test/dapl_fft_endpoint.c 2005-05-09 13:31:50.597962000 -0700 @@ -168,7 +168,7 @@ DAT_EP_HANDLE ep_handle; DAT_EVD_HANDLE send_evd, conn_evd, recv_evd, cr_evd; DAT_PZ_HANDLE pz_handle; - DAT_EVENT event; + struct dat_event event; Bpool *bpool; int res; DAT_RETURN rc; diff -Nur -X /home/tduffy/dontdiff linux-kernel20/test/dapltest/test/dapl_limit.c linux-kernel21/test/dapltest/test/dapl_limit.c --- linux-kernel20/test/dapltest/test/dapl_limit.c 2005-05-09 12:16:33.897001000 -0700 +++ linux-kernel21/test/dapltest/test/dapl_limit.c 2005-05-09 13:31:50.606962000 -0700 @@ -1084,7 +1084,7 @@ /* Rpost Cleanup loop */ for (i = 0; i < cmd->width; i++) { - DAT_EVENT event; + struct dat_event event; /* * Disconnecting an unconnected EP should complete diff -Nur -X /home/tduffy/dontdiff linux-kernel20/test/dapltest/test/dapl_performance_client.c linux-kernel21/test/dapltest/test/dapl_performance_client.c --- linux-kernel20/test/dapltest/test/dapl_performance_client.c 2005-05-09 13:16:26.034961000 -0700 +++ linux-kernel21/test/dapltest/test/dapl_performance_client.c 2005-05-09 13:31:50.611964000 -0700 @@ -255,7 +255,7 @@ struct dat_lmr_triplet *iov; struct dat_rmr_triplet rmr_triplet; DAT_DTO_COOKIE cookie; - DAT_EVENT event; + struct dat_event event; DAT_RETURN ret; Performance_Ep_Context_t *ep_context; Performance_Test_Op_t *op; diff -Nur -X /home/tduffy/dontdiff linux-kernel20/test/dapltest/test/dapl_performance_util.c linux-kernel21/test/dapltest/test/dapl_performance_util.c --- linux-kernel20/test/dapltest/test/dapl_performance_util.c 2005-05-09 11:52:38.934000000 -0700 +++ linux-kernel21/test/dapltest/test/dapl_performance_util.c 2005-05-09 13:31:50.617962000 -0700 @@ -526,7 +526,7 @@ DAT_COUNT i; DAT_COUNT queue_size; DAT_RETURN ret; - DAT_EVENT event; + struct dat_event event; unsigned long pre_ctxt_num; unsigned long post_ctxt_num; DT_Mdep_TimeStamp pre_ts; @@ -615,7 +615,7 @@ Performance_Stats_t *stats) { DAT_RETURN ret; - DAT_EVENT event; + struct dat_event event; unsigned long pre_ctxt_num; unsigned long post_ctxt_num; DT_Mdep_TimeStamp pre_ts; diff -Nur -X /home/tduffy/dontdiff linux-kernel20/test/dapltest/test/dapl_test_util.c linux-kernel21/test/dapltest/test/dapl_test_util.c --- linux-kernel20/test/dapltest/test/dapl_test_util.c 2005-05-09 13:25:54.690963000 -0700 +++ linux-kernel21/test/dapltest/test/dapl_test_util.c 2005-05-09 13:31:50.623960000 -0700 @@ -259,7 +259,7 @@ for (;;) { DAT_RETURN ret; - DAT_EVENT event; + struct dat_event event; ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); if (ret != DAT_SUCCESS) @@ -314,7 +314,7 @@ for (;;) { DAT_RETURN ret; - DAT_EVENT event; + struct dat_event event; ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); if (ret != DAT_SUCCESS) @@ -367,7 +367,7 @@ for (;;) { DAT_RETURN ret; - DAT_EVENT event; + struct dat_event event; ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); if (ret != DAT_SUCCESS) @@ -438,7 +438,7 @@ for (;;) { DAT_RETURN ret; - DAT_EVENT event; + struct dat_event event; ret = DT_Tdep_evd_dequeue ( evd_handle, &event); @@ -490,7 +490,7 @@ for (;;) { DAT_RETURN ret; - DAT_EVENT event; + struct dat_event event; ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); if (ret != DAT_SUCCESS) @@ -534,7 +534,7 @@ for (;;) { DAT_RETURN ret; - DAT_EVENT event; + struct dat_event event; ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); if (ret != DAT_SUCCESS) diff -Nur -X /home/tduffy/dontdiff linux-kernel20/test/dapltest/test/dapl_transaction_test.c linux-kernel21/test/dapltest/test/dapl_transaction_test.c --- linux-kernel20/test/dapltest/test/dapl_transaction_test.c 2005-05-09 13:23:19.269962000 -0700 +++ linux-kernel21/test/dapltest/test/dapl_transaction_test.c 2005-05-09 13:31:50.633961000 -0700 @@ -741,7 +741,7 @@ * the REJECT; clean them up and repost if so */ { - DAT_EVENT event; + struct dat_event event; DAT_COUNT drained = 0; dat_ep_reset (test_ptr->ep_context[i].ep_handle); @@ -1186,7 +1186,7 @@ */ if ( DAT_HANDLE_NULL != ep_handle) { - DAT_EVENT event; + struct dat_event event; /* * Drain off outstanding DTOs that may have been * generated by racing disconnects From Tom.Duffy at Sun.COM Mon May 9 14:39:43 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 09 May 2005 14:39:43 -0700 Subject: [openib-general] [PATCH][17/26] kDAPL: Remove typedef DAT_CR_ARRIVAL_EVENT_DATA In-Reply-To: <11156747831783@sun.com> References: <11156747793273@sun.com> <11156747791462@sun.com> <1115674780270@sun.com> <11156747803410@sun.com> <11156747802252@sun.com> <11156747801195@sun.com> <11156747812210@sun.com> <11156747812290@sun.com> <11156747812388@sun.com> <11156747811508@sun.com> <11156747821301@sun.com> <1115674782460@sun.com> <11156747821275@sun.com> <11156747823890@sun.com> <11156747831715@sun.com> <11156747831729@sun.com> <11156747831783@sun.com> Message-ID: <11156747832991@sun.com> Signed-off-by: Tom Duffy diff -Nur -X /home/tduffy/dontdiff linux-kernel16/dat/dat.h linux-kernel17/dat/dat.h --- linux-kernel16/dat/dat.h 2005-05-09 13:19:22.189961000 -0700 +++ linux-kernel17/dat/dat.h 2005-05-09 13:23:46.473960000 -0700 @@ -794,7 +794,7 @@ } DAT_SP_HANDLE; /* Connection Request Arrival event data */ -typedef struct dat_cr_arrival_event_data { +struct dat_cr_arrival_event_data { /* Handle to the Service Point that received the Connection Request * from the remote side. If the Service Point was Reserved, sp is * DAT_HANDLE_NULL because the reserved Service Point is @@ -817,7 +817,7 @@ * to dat_cr_accept or dat_cr_reject of the Connection Request. */ DAT_CR_HANDLE cr_handle; -} DAT_CR_ARRIVAL_EVENT_DATA; +}; /* Connection event data */ typedef struct dat_connection_event_data { @@ -906,7 +906,7 @@ typedef union dat_event_data { struct dat_dto_completion_event_data dto_completion_event_data; struct dat_rmr_bind_completion_event_data rmr_completion_event_data; - DAT_CR_ARRIVAL_EVENT_DATA cr_arrival_event_data; + struct dat_cr_arrival_event_data cr_arrival_event_data; DAT_CONNECTION_EVENT_DATA connect_event_data; DAT_ASYNCH_ERROR_EVENT_DATA asynch_error_event_data; DAT_SOFTWARE_EVENT_DATA software_event_data; diff -Nur -X /home/tduffy/dontdiff linux-kernel16/test/dapltest/include/dapl_fft_util.h linux-kernel17/test/dapltest/include/dapl_fft_util.h --- linux-kernel16/test/dapltest/include/dapl_fft_util.h 2005-04-06 22:02:36.589960000 -0700 +++ linux-kernel17/test/dapltest/include/dapl_fft_util.h 2005-05-09 13:23:19.226964000 -0700 @@ -64,7 +64,7 @@ DAT_COUNT count; DAT_CR_HANDLE cr_handle; Bpool *bpool; - DAT_CR_ARRIVAL_EVENT_DATA cr_stat; + struct dat_cr_arrival_event_data cr_stat; DAT_EVENT_NUMBER event_num; DAT_IA_ADDRESS_PTR remote_netaddr; Per_Test_Data_t *pt_ptr; diff -Nur -X /home/tduffy/dontdiff linux-kernel16/test/dapltest/include/dapl_proto.h linux-kernel17/test/dapltest/include/dapl_proto.h --- linux-kernel16/test/dapltest/include/dapl_proto.h 2005-05-09 13:20:14.136964000 -0700 +++ linux-kernel17/test/dapltest/include/dapl_proto.h 2005-05-09 13:24:33.942968000 -0700 @@ -376,7 +376,7 @@ bool DT_cr_event_wait (DT_Tdep_Print_Head* phead, DAT_EVD_HANDLE evd_handle, - DAT_CR_ARRIVAL_EVENT_DATA *cr_stat_p); + struct dat_cr_arrival_event_data *cr_stat_p); bool DT_dto_event_reap (DT_Tdep_Print_Head* phead, DAT_EVD_HANDLE evd_handle, @@ -409,7 +409,7 @@ char *message); bool DT_cr_check (DT_Tdep_Print_Head* phead, - DAT_CR_ARRIVAL_EVENT_DATA *cr_stat_p, + struct dat_cr_arrival_event_data *cr_stat_p, DAT_PSP_HANDLE psp_handle_expected, DAT_CONN_QUAL port_expected, DAT_CR_HANDLE *cr_handlep, diff -Nur -X /home/tduffy/dontdiff linux-kernel16/test/dapltest/test/dapl_performance_server.c linux-kernel17/test/dapltest/test/dapl_performance_server.c --- linux-kernel16/test/dapltest/test/dapl_performance_server.c 2005-05-09 13:16:40.541971000 -0700 +++ linux-kernel17/test/dapltest/test/dapl_performance_server.c 2005-05-09 13:23:19.245962000 -0700 @@ -104,7 +104,7 @@ DAT_RSP_HANDLE rsp_handle; DAT_PSP_HANDLE psp_handle; - DAT_CR_ARRIVAL_EVENT_DATA cr_stat; + struct dat_cr_arrival_event_data cr_stat; DAT_CR_HANDLE cr_handle; DAT_EVENT_NUMBER event_num; diff -Nur -X /home/tduffy/dontdiff linux-kernel16/test/dapltest/test/dapl_server.c linux-kernel17/test/dapltest/test/dapl_server.c --- linux-kernel16/test/dapltest/test/dapl_server.c 2005-05-09 13:14:52.066962000 -0700 +++ linux-kernel17/test/dapltest/test/dapl_server.c 2005-05-09 13:23:19.253960000 -0700 @@ -276,7 +276,7 @@ for (;/* EVER */;) { DAT_CR_HANDLE cr_handle; - DAT_CR_ARRIVAL_EVENT_DATA cr_stat; + struct dat_cr_arrival_event_data cr_stat; DAT_EVENT_NUMBER event_num; /* Set up the Per_Test_Data */ diff -Nur -X /home/tduffy/dontdiff linux-kernel16/test/dapltest/test/dapl_test_util.c linux-kernel17/test/dapltest/test/dapl_test_util.c --- linux-kernel16/test/dapltest/test/dapl_test_util.c 2005-05-09 13:19:10.563961000 -0700 +++ linux-kernel17/test/dapltest/test/dapl_test_util.c 2005-05-09 13:23:19.258962000 -0700 @@ -250,7 +250,7 @@ bool DT_cr_event_wait ( DT_Tdep_Print_Head *phead, DAT_EVD_HANDLE evd_handle, - DAT_CR_ARRIVAL_EVENT_DATA *cr_stat_p) + struct dat_cr_arrival_event_data *cr_stat_p) { int err_cnt; @@ -658,7 +658,7 @@ */ bool DT_cr_check ( DT_Tdep_Print_Head *phead, - DAT_CR_ARRIVAL_EVENT_DATA *cr_stat_p, + struct dat_cr_arrival_event_data *cr_stat_p, DAT_PSP_HANDLE psp_handle_expected, DAT_CONN_QUAL port_expected, DAT_CR_HANDLE *cr_handlep, diff -Nur -X /home/tduffy/dontdiff linux-kernel16/test/dapltest/test/dapl_transaction_test.c linux-kernel17/test/dapltest/test/dapl_transaction_test.c --- linux-kernel16/test/dapltest/test/dapl_transaction_test.c 2005-05-09 13:14:52.085962000 -0700 +++ linux-kernel17/test/dapltest/test/dapl_transaction_test.c 2005-05-09 13:23:19.269962000 -0700 @@ -520,7 +520,7 @@ u32 buff_size = MAX_OPS * sizeof (RemoteMemoryInfo); RemoteMemoryInfo *RemoteMemInfo; struct dat_dto_completion_event_data dto_stat; - DAT_CR_ARRIVAL_EVENT_DATA cr_stat; + struct dat_cr_arrival_event_data cr_stat; DAT_CR_HANDLE cr_handle; /* From Tom.Duffy at Sun.COM Mon May 9 14:39:44 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 09 May 2005 14:39:44 -0700 Subject: [openib-general] [PATCH][20/26] kDAPL: Remove typedef DAT_SOFTWARE_EVENT_DATA In-Reply-To: <11156747843960@sun.com> References: <11156747793273@sun.com> <11156747791462@sun.com> <1115674780270@sun.com> <11156747803410@sun.com> <11156747802252@sun.com> <11156747801195@sun.com> <11156747812210@sun.com> <11156747812290@sun.com> <11156747812388@sun.com> <11156747811508@sun.com> <11156747821301@sun.com> <1115674782460@sun.com> <11156747821275@sun.com> <11156747823890@sun.com> <11156747831715@sun.com> <11156747831729@sun.com> <11156747831783@sun.com> <11156747832991@sun.com> <11156747841537@sun.com> <11156747843960@sun.com> Message-ID: <11156747844081@sun.com> Signed-off-by: Tom Duffy diff -Nur -X /home/tduffy/dontdiff linux-kernel19/dat/dat.h linux-kernel20/dat/dat.h --- linux-kernel19/dat/dat.h 2005-05-09 13:28:13.273961000 -0700 +++ linux-kernel20/dat/dat.h 2005-05-09 13:29:50.153961000 -0700 @@ -877,9 +877,9 @@ } DAT_PZ_ASYNC_ERROR_REASON; /* Software event data */ -typedef struct dat_software_event_data { +struct dat_software_event_data { DAT_PVOID pointer; -} DAT_SOFTWARE_EVENT_DATA; +}; typedef enum dat_event_number { DAT_DTO_COMPLETION_EVENT = 0x00001, @@ -909,7 +909,7 @@ struct dat_cr_arrival_event_data cr_arrival_event_data; struct dat_connection_event_data connect_event_data; struct dat_asynch_error_event_data asynch_error_event_data; - DAT_SOFTWARE_EVENT_DATA software_event_data; + struct dat_software_event_data software_event_data; } DAT_EVENT_DATA; /* Event struct that holds all event information */ From Tom.Duffy at Sun.COM Mon May 9 14:39:45 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 09 May 2005 14:39:45 -0700 Subject: [openib-general] [PATCH][24/26] kDAPL: Remove typedef DAT_LMR_PARAM In-Reply-To: <11156747851027@sun.com> References: <11156747793273@sun.com> <11156747791462@sun.com> <1115674780270@sun.com> <11156747803410@sun.com> <11156747802252@sun.com> <11156747801195@sun.com> <11156747812210@sun.com> <11156747812290@sun.com> <11156747812388@sun.com> <11156747811508@sun.com> <11156747821301@sun.com> <1115674782460@sun.com> <11156747821275@sun.com> <11156747823890@sun.com> <11156747831715@sun.com> <11156747831729@sun.com> <11156747831783@sun.com> <11156747832991@sun.com> <11156747841537@sun.com> <11156747843960@sun.com> <11156747844081@sun.com> <11156747842198@sun.com> <11156747851961@sun.com> <11156747851027@sun.com> Message-ID: <11156747853473@sun.com> Signed-off-by: Tom Duffy diff -Nur -X /home/tduffy/dontdiff linux-kernel23/dat/dat.h linux-kernel24/dat/dat.h --- linux-kernel23/dat/dat.h 2005-05-09 14:02:35.073961000 -0700 +++ linux-kernel24/dat/dat.h 2005-05-09 14:05:28.655967000 -0700 @@ -991,7 +991,7 @@ } DAT_REGION_DESCRIPTION; /* LMR Arguments */ -typedef struct dat_lmr_param { +struct dat_lmr_param { DAT_IA_HANDLE ia_handle; DAT_MEM_TYPE mem_type; DAT_REGION_DESCRIPTION region_desc; @@ -1002,7 +1002,7 @@ DAT_RMR_CONTEXT rmr_context; DAT_VLEN registered_size; DAT_VADDR registered_address; -} DAT_LMR_PARAM; +}; /* General Interface Adapter attributes.*/ typedef struct dat_ia_attr { @@ -1201,7 +1201,8 @@ typedef DAT_RETURN (*DAT_LMR_FREE_FUNC) (DAT_LMR_HANDLE); -typedef DAT_RETURN (*DAT_LMR_QUERY_FUNC) (DAT_LMR_HANDLE, DAT_LMR_PARAM *); +typedef DAT_RETURN (*DAT_LMR_QUERY_FUNC) (DAT_LMR_HANDLE, + struct dat_lmr_param *); typedef DAT_RETURN (*DAT_LMR_SYNC_RDMA_READ_FUNC) ( DAT_IA_HANDLE, const struct dat_lmr_triplet *, DAT_VLEN); @@ -1664,7 +1665,8 @@ soft_high_watermark, hard_high_watermark); } -static inline DAT_RETURN dat_lmr_query(DAT_LMR_HANDLE lmr, DAT_LMR_PARAM * param) +static inline DAT_RETURN dat_lmr_query(DAT_LMR_HANDLE lmr, + struct dat_lmr_param *param) { return DAT_CALL_PROVIDER_FUNC(lmr_query_func, lmr, param); } diff -Nur -X /home/tduffy/dontdiff linux-kernel23/dat-provider/dapl.h linux-kernel24/dat-provider/dapl.h --- linux-kernel23/dat-provider/dapl.h 2005-05-09 14:01:14.740963000 -0700 +++ linux-kernel24/dat-provider/dapl.h 2005-05-09 14:05:04.217962000 -0700 @@ -358,7 +358,7 @@ /* DAPL_LMR maps to DAT_LMR_HANDLE */ struct dapl_lmr { DAPL_HEADER header; - DAT_LMR_PARAM param; + struct dat_lmr_param param; ib_mr_handle_t mr_handle; atomic_t lmr_ref_count; }; @@ -656,7 +656,7 @@ DAT_VLEN *, /* registered_length */ DAT_VADDR *); /* registered_address */ -extern DAT_RETURN dapl_lmr_query(DAT_LMR_HANDLE, DAT_LMR_PARAM *); +extern DAT_RETURN dapl_lmr_query(DAT_LMR_HANDLE, struct dat_lmr_param *); extern DAT_RETURN dapl_lmr_free(DAT_LMR_HANDLE); diff -Nur -X /home/tduffy/dontdiff linux-kernel23/dat-provider/dapl_lmr_query.c linux-kernel24/dat-provider/dapl_lmr_query.c --- linux-kernel23/dat-provider/dapl_lmr_query.c 2005-05-04 08:19:32.665015000 -0700 +++ linux-kernel24/dat-provider/dapl_lmr_query.c 2005-05-09 14:06:13.807966000 -0700 @@ -36,7 +36,8 @@ #include "dapl.h" -DAT_RETURN dapl_lmr_query(DAT_LMR_HANDLE lmr_handle, DAT_LMR_PARAM * lmr_param) +DAT_RETURN dapl_lmr_query(DAT_LMR_HANDLE lmr_handle, + struct dat_lmr_param *lmr_param) { DAPL_LMR *lmr; DAT_RETURN dat_status; @@ -58,7 +59,7 @@ dat_status = DAT_SUCCESS; lmr = (DAPL_LMR *) lmr_handle; - memcpy(lmr_param, &lmr->param, sizeof(DAT_LMR_PARAM)); + memcpy(lmr_param, &lmr->param, sizeof *lmr_param); bail: return dat_status; diff -Nur -X /home/tduffy/dontdiff linux-kernel23/test/dapltest/test/dapl_fft_queryinfo.c linux-kernel24/test/dapltest/test/dapl_fft_queryinfo.c --- linux-kernel23/test/dapltest/test/dapl_fft_queryinfo.c 2005-05-09 14:01:14.763960000 -0700 +++ linux-kernel24/test/dapltest/test/dapl_fft_queryinfo.c 2005-05-09 14:05:04.246964000 -0700 @@ -66,7 +66,7 @@ DAT_PZ_HANDLE pz_handle; struct dat_pz_param pz_param; DAT_LMR_HANDLE lmr_handle; - DAT_LMR_PARAM lmr_param; + struct dat_lmr_param lmr_param; DAT_LMR_CONTEXT lmr_context; DAT_RMR_HANDLE rmr_handle; struct dat_rmr_param rmr_param; From Tom.Duffy at Sun.COM Mon May 9 14:39:45 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 09 May 2005 14:39:45 -0700 Subject: [openib-general] [PATCH][23/26] kDAPL: Remove typedef DAT_EVD_PARAM In-Reply-To: <11156747851961@sun.com> References: <11156747793273@sun.com> <11156747791462@sun.com> <1115674780270@sun.com> <11156747803410@sun.com> <11156747802252@sun.com> <11156747801195@sun.com> <11156747812210@sun.com> <11156747812290@sun.com> <11156747812388@sun.com> <11156747811508@sun.com> <11156747821301@sun.com> <1115674782460@sun.com> <11156747821275@sun.com> <11156747823890@sun.com> <11156747831715@sun.com> <11156747831729@sun.com> <11156747831783@sun.com> <11156747832991@sun.com> <11156747841537@sun.com> <11156747843960@sun.com> <11156747844081@sun.com> <11156747842198@sun.com> <11156747851961@sun.com> Message-ID: <11156747851027@sun.com> Signed-off-by: Tom Duffy diff -Nur -X /home/tduffy/dontdiff linux-kernel22/dat/dat.h linux-kernel23/dat/dat.h --- linux-kernel22/dat/dat.h 2005-05-09 13:55:55.040963000 -0700 +++ linux-kernel23/dat/dat.h 2005-05-09 14:02:35.073961000 -0700 @@ -955,13 +955,13 @@ DAT_BOOLEAN is_thread_safe; }; -typedef struct dat_evd_param { +struct dat_evd_param { DAT_IA_HANDLE ia_handle; DAT_COUNT evd_qlen; DAT_UPCALL_POLICY upcall_policy; struct dat_upcall_object upcall; DAT_EVD_FLAGS evd_flags; -} DAT_EVD_PARAM; +}; /* * Memory types @@ -1144,7 +1144,8 @@ typedef DAT_RETURN (*DAT_EVD_FREE_FUNC) (DAT_EVD_HANDLE); -typedef DAT_RETURN (*DAT_EVD_QUERY_FUNC) (DAT_EVD_HANDLE, DAT_EVD_PARAM *); +typedef DAT_RETURN (*DAT_EVD_QUERY_FUNC) (DAT_EVD_HANDLE, + struct dat_evd_param *); typedef DAT_RETURN (*DAT_EP_CREATE_FUNC) ( DAT_IA_HANDLE, DAT_PZ_HANDLE, DAT_EVD_HANDLE, DAT_EVD_HANDLE, @@ -1507,7 +1508,7 @@ } static inline DAT_RETURN dat_evd_query(DAT_EVD_HANDLE evd, - DAT_EVD_PARAM * param) + struct dat_evd_param *param) { return DAT_CALL_PROVIDER_FUNC(evd_query_func, evd, param); } diff -Nur -X /home/tduffy/dontdiff linux-kernel22/dat-provider/dapl_evd_kquery.c linux-kernel23/dat-provider/dapl_evd_kquery.c --- linux-kernel22/dat-provider/dapl_evd_kquery.c 2005-05-04 08:19:32.761994000 -0700 +++ linux-kernel23/dat-provider/dapl_evd_kquery.c 2005-05-09 14:03:05.011028000 -0700 @@ -38,7 +38,8 @@ #include "dapl.h" -DAT_RETURN dapl_evd_kquery(DAT_EVD_HANDLE evd_handle, DAT_EVD_PARAM * evd_param) +DAT_RETURN dapl_evd_kquery(DAT_EVD_HANDLE evd_handle, + struct dat_evd_param *evd_param) { DAPL_EVD *evd_ptr; DAT_RETURN dat_status; @@ -53,7 +54,7 @@ /* Note: the spec. allows for events to be directed to a NULL EVD */ /* with handle of type DAT_HANDLE_NULL. See 6.3.1 */ if (DAT_HANDLE_NULL == evd_handle) { - memset(evd_param, 0, sizeof(DAT_EVD_PARAM)); + memset(evd_param, 0, sizeof *evd_param); } else { if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, 0); diff -Nur -X /home/tduffy/dontdiff linux-kernel22/dat-provider/dapl.h linux-kernel23/dat-provider/dapl.h --- linux-kernel22/dat-provider/dapl.h 2005-05-09 13:59:10.723960000 -0700 +++ linux-kernel23/dat-provider/dapl.h 2005-05-09 14:01:14.740963000 -0700 @@ -529,7 +529,7 @@ DAT_EVD_HANDLE *); /* evd_handle */ extern DAT_RETURN dapl_evd_kquery(DAT_EVD_HANDLE, /* evd_handle */ - DAT_EVD_PARAM *); /* evd_args */ + struct dat_evd_param *); /* evd_args */ extern DAT_RETURN dapl_evd_modify_upcall(DAT_EVD_HANDLE, /* evd_handle */ DAT_UPCALL_POLICY, /* upcall_policy */ diff -Nur -X /home/tduffy/dontdiff linux-kernel22/test/dapltest/test/dapl_fft_queryinfo.c linux-kernel23/test/dapltest/test/dapl_fft_queryinfo.c --- linux-kernel22/test/dapltest/test/dapl_fft_queryinfo.c 2005-05-09 12:55:09.772004000 -0700 +++ linux-kernel23/test/dapltest/test/dapl_fft_queryinfo.c 2005-05-09 14:01:14.763960000 -0700 @@ -58,7 +58,7 @@ #ifndef __KDAPLTEST__ DAT_CNO_PARAM cno_param; #endif - DAT_EVD_PARAM evd_param; + struct dat_evd_param evd_param; DAT_PSP_HANDLE psp_handle; struct dat_psp_param psp_param; DAT_RSP_HANDLE rsp_handle; From Tom.Duffy at Sun.COM Mon May 9 14:39:45 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 09 May 2005 14:39:45 -0700 Subject: [openib-general] [PATCH][22/26] kDAPL: Remove typedef DAT_UPCALL_OBJECT In-Reply-To: <11156747842198@sun.com> References: <11156747793273@sun.com> <11156747791462@sun.com> <1115674780270@sun.com> <11156747803410@sun.com> <11156747802252@sun.com> <11156747801195@sun.com> <11156747812210@sun.com> <11156747812290@sun.com> <11156747812388@sun.com> <11156747811508@sun.com> <11156747821301@sun.com> <1115674782460@sun.com> <11156747821275@sun.com> <11156747823890@sun.com> <11156747831715@sun.com> <11156747831729@sun.com> <11156747831783@sun.com> <11156747832991@sun.com> <11156747841537@sun.com> <11156747843960@sun.com> <11156747844081@sun.com> <11156747842198@sun.com> Message-ID: <11156747851961@sun.com> Signed-off-by: Tom Duffy diff -Nur -X /home/tduffy/dontdiff linux-kernel21/dat/dat.h linux-kernel22/dat/dat.h --- linux-kernel21/dat/dat.h 2005-05-09 13:32:35.923960000 -0700 +++ linux-kernel22/dat/dat.h 2005-05-09 13:55:55.040963000 -0700 @@ -934,17 +934,17 @@ typedef void (*DAT_UPCALL_FUNC) (DAT_PVOID, const struct dat_event *, DAT_BOOLEAN); -typedef struct dat_upcall_object { +struct dat_upcall_object { DAT_PVOID instance_data; DAT_UPCALL_FUNC upcall_func; -} DAT_UPCALL_OBJECT; +}; /* Define NULL upcall */ #define DAT_UPCALL_NULL \ - ((DAT_UPCALL_OBJECT) { (DAT_PVOID) NULL, (DAT_UPCALL_FUNC) NULL }) + ((struct dat_upcall_object) { (DAT_PVOID) NULL, (DAT_UPCALL_FUNC) NULL }) -#define DAT_UPCALL_SAME ((DAT_UPCALL_OBJECT *) NULL) +#define DAT_UPCALL_SAME ((struct dat_upcall_object *) NULL) /* Provider/registration info */ @@ -959,7 +959,7 @@ DAT_IA_HANDLE ia_handle; DAT_COUNT evd_qlen; DAT_UPCALL_POLICY upcall_policy; - DAT_UPCALL_OBJECT upcall; + struct dat_upcall_object upcall; DAT_EVD_FLAGS evd_flags; } DAT_EVD_PARAM; @@ -1272,11 +1272,12 @@ DAT_VLEN *, DAT_VADDR *); typedef DAT_RETURN (*DAT_EVD_KCREATE_FUNC) ( - DAT_IA_HANDLE, DAT_COUNT, DAT_UPCALL_POLICY, const DAT_UPCALL_OBJECT *, + DAT_IA_HANDLE, DAT_COUNT, DAT_UPCALL_POLICY, + const struct dat_upcall_object *, DAT_EVD_FLAGS, DAT_EVD_HANDLE *); typedef DAT_RETURN (*DAT_EVD_MODIFY_UPCALL_FUNC) ( - DAT_EVD_HANDLE, DAT_UPCALL_POLICY, const DAT_UPCALL_OBJECT *); + DAT_EVD_HANDLE, DAT_UPCALL_POLICY, const struct dat_upcall_object *); struct dat_provider { const char *device_name; @@ -1483,7 +1484,7 @@ static inline DAT_RETURN dat_evd_kcreate(DAT_IA_HANDLE ia, DAT_COUNT qlen, DAT_UPCALL_POLICY policy, - const DAT_UPCALL_OBJECT * upcall, + const struct dat_upcall_object *upcall, DAT_EVD_FLAGS flags, DAT_EVD_HANDLE * evd) { @@ -1493,7 +1494,7 @@ static inline DAT_RETURN dat_evd_modify_upcall(DAT_EVD_HANDLE evd, DAT_UPCALL_POLICY policy, - const DAT_UPCALL_OBJECT * upcall) + const struct dat_upcall_object *upcall) { return DAT_CALL_PROVIDER_FUNC( evd_modify_upcall_func, evd, policy, upcall); diff -Nur -X /home/tduffy/dontdiff linux-kernel21/dat-provider/dapl_cno_util.c linux-kernel22/dat-provider/dapl_cno_util.c --- linux-kernel21/dat-provider/dapl_cno_util.c 2005-05-09 13:39:21.136968000 -0700 +++ linux-kernel22/dat-provider/dapl_cno_util.c 2005-05-09 13:57:23.215962000 -0700 @@ -41,9 +41,7 @@ * alloc and initialize an EVD struct */ DAPL_CNO * -dapl_cno_alloc ( - DAPL_IA *ia_ptr, - const DAT_UPCALL_OBJECT *upcall) +dapl_cno_alloc(DAPL_IA *ia_ptr, const struct dat_upcall_object *upcall) { DAPL_CNO *cno_ptr; diff -Nur -X /home/tduffy/dontdiff linux-kernel21/dat-provider/dapl_cno_util.h linux-kernel22/dat-provider/dapl_cno_util.h --- linux-kernel21/dat-provider/dapl_cno_util.h 2005-04-20 12:42:07.836018000 -0700 +++ linux-kernel22/dat-provider/dapl_cno_util.h 2005-05-09 13:57:53.648969000 -0700 @@ -37,9 +37,7 @@ #include "dapl.h" DAPL_CNO * -dapl_cno_alloc ( - DAPL_IA *ia_ptr, - const DAT_UPCALL_OBJECT *upcall) ; +dapl_cno_alloc(DAPL_IA *ia_ptr, const struct dat_upcall_object *upcall); void dapl_cno_dealloc ( diff -Nur -X /home/tduffy/dontdiff linux-kernel21/dat-provider/dapl_evd_kcreate.c linux-kernel22/dat-provider/dapl_evd_kcreate.c --- linux-kernel21/dat-provider/dapl_evd_kcreate.c 2005-05-09 11:07:01.955004000 -0700 +++ linux-kernel22/dat-provider/dapl_evd_kcreate.c 2005-05-09 13:58:06.581973000 -0700 @@ -50,7 +50,7 @@ DAT_RETURN dapl_evd_kcreate(DAT_IA_HANDLE ia_handle, DAT_COUNT evd_min_qlen, DAT_UPCALL_POLICY upcall_policy, /* ignore */ - const DAT_UPCALL_OBJECT * upcall, + const struct dat_upcall_object *upcall, DAT_EVD_FLAGS evd_flags, DAT_EVD_HANDLE * evd_handle) { diff -Nur -X /home/tduffy/dontdiff linux-kernel21/dat-provider/dapl_evd_modify_upcall.c linux-kernel22/dat-provider/dapl_evd_modify_upcall.c --- linux-kernel21/dat-provider/dapl_evd_modify_upcall.c 2005-04-20 12:42:07.397019000 -0700 +++ linux-kernel22/dat-provider/dapl_evd_modify_upcall.c 2005-05-09 13:58:27.721975000 -0700 @@ -43,8 +43,9 @@ * Modify the upcall instance on an EVD */ -DAT_RETURN dapl_evd_modify_upcall(DAT_EVD_HANDLE evd_handle, DAT_UPCALL_POLICY upcall_policy, /* ignore */ - const DAT_UPCALL_OBJECT * upcall) +DAT_RETURN dapl_evd_modify_upcall(DAT_EVD_HANDLE evd_handle, + DAT_UPCALL_POLICY upcall_policy, /* ignore */ + const struct dat_upcall_object *upcall) { DAPL_EVD *evd_ptr; DAPL_CNO *cno_ptr; diff -Nur -X /home/tduffy/dontdiff linux-kernel21/dat-provider/dapl.h linux-kernel22/dat-provider/dapl.h --- linux-kernel21/dat-provider/dapl.h 2005-05-09 13:37:15.817963000 -0700 +++ linux-kernel22/dat-provider/dapl.h 2005-05-09 13:59:10.723960000 -0700 @@ -254,11 +254,10 @@ }; /* DAPL_CNO maps to DAT_CNO_HANDLE */ -struct dapl_cno -{ +struct dapl_cno { DAPL_HEADER header; - DAT_UPCALL_OBJECT cno_upcall; - DAT_UPCALL_POLICY cno_upcall_policy; + struct dat_upcall_object cno_upcall; + DAT_UPCALL_POLICY cno_upcall_policy; }; /* DAPL_EVD maps to DAT_EVD_HANDLE */ @@ -525,7 +524,7 @@ extern DAT_RETURN dapl_evd_kcreate(DAT_IA_HANDLE, /* ia_handle */ DAT_COUNT, /* evd_min_qlen */ DAT_UPCALL_POLICY, /* upcall_policy */ - const DAT_UPCALL_OBJECT *, /* upcall */ + const struct dat_upcall_object *,/* upcall */ DAT_EVD_FLAGS, /* evd_flags */ DAT_EVD_HANDLE *); /* evd_handle */ @@ -534,7 +533,7 @@ extern DAT_RETURN dapl_evd_modify_upcall(DAT_EVD_HANDLE, /* evd_handle */ DAT_UPCALL_POLICY, /* upcall_policy */ - const DAT_UPCALL_OBJECT *); /* upcall */ + const struct dat_upcall_object *); /* upcall */ extern DAT_RETURN dapl_evd_enable(DAT_EVD_HANDLE); /* evd_handle */ diff -Nur -X /home/tduffy/dontdiff linux-kernel21/test/dapltest/kdapl/kdapl_tdep_evd.c linux-kernel22/test/dapltest/kdapl/kdapl_tdep_evd.c --- linux-kernel21/test/dapltest/kdapl/kdapl_tdep_evd.c 2005-05-09 13:38:10.908963000 -0700 +++ linux-kernel22/test/dapltest/kdapl/kdapl_tdep_evd.c 2005-05-09 13:59:26.957963000 -0700 @@ -103,7 +103,7 @@ { DAT_RETURN dat_status; - DAT_UPCALL_OBJECT upcall; + struct dat_upcall_object upcall; Tdep_Evd *evd_ptr; evd_ptr = NULL; From Tom.Duffy at Sun.COM Mon May 9 14:39:46 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 09 May 2005 14:39:46 -0700 Subject: [openib-general] [PATCH][26/26] kDAPL: Remove typedef DAT_SOCK_ADDR and DAT_SOCK_ADDR6 In-Reply-To: <11156747851643@sun.com> References: <11156747793273@sun.com> <11156747791462@sun.com> <1115674780270@sun.com> <11156747803410@sun.com> <11156747802252@sun.com> <11156747801195@sun.com> <11156747812210@sun.com> <11156747812290@sun.com> <11156747812388@sun.com> <11156747811508@sun.com> <11156747821301@sun.com> <1115674782460@sun.com> <11156747821275@sun.com> <11156747823890@sun.com> <11156747831715@sun.com> <11156747831729@sun.com> <11156747831783@sun.com> <11156747832991@sun.com> <11156747841537@sun.com> <11156747843960@sun.com> <11156747844081@sun.com> <11156747842198@sun.com> <11156747851961@sun.com> <11156747851027@sun.com> <11156747853473@sun.com> <11156747851643@sun.com> Message-ID: <11156747862748@sun.com> Signed-off-by: Tom Duffy diff -Nur -X /home/tduffy/dontdiff linux-kernel25/dat/dat.h linux-kernel26/dat/dat.h --- linux-kernel25/dat/dat.h 2005-05-09 14:08:31.214971000 -0700 +++ linux-kernel26/dat/dat.h 2005-05-09 14:12:56.760962000 -0700 @@ -43,8 +43,6 @@ typedef void * DAT_PVOID; typedef int DAT_COUNT; typedef u64 DAT_PADDR; -typedef struct sockaddr DAT_SOCK_ADDR; -typedef struct sockaddr_in6 DAT_SOCK_ADDR6; /* * All return codes are actually a 3-way tuple: @@ -418,7 +416,7 @@ #define DAT_HANDLE_NULL ((DAT_HANDLE)NULL) -typedef DAT_SOCK_ADDR *DAT_IA_ADDRESS_PTR; +typedef struct sockaddr *DAT_IA_ADDRESS_PTR; typedef u64 DAT_CONN_QUAL; typedef u64 DAT_PORT_QUAL; diff -Nur -X /home/tduffy/dontdiff linux-kernel25/dat-provider/dapl_adapter_util.h linux-kernel26/dat-provider/dapl_adapter_util.h --- linux-kernel25/dat-provider/dapl_adapter_util.h 2005-05-09 14:09:01.530972000 -0700 +++ linux-kernel26/dat-provider/dapl_adapter_util.h 2005-05-09 14:13:10.187965000 -0700 @@ -148,7 +148,7 @@ DAT_RETURN dapl_ib_query_hca(DAPL_HCA * hca_ptr, struct dat_ia_attr *ia_attr, struct dat_ep_attr *ep_attr, - DAT_SOCK_ADDR6 * ip_addr); + struct sockaddr_in6 *ip_addr); DAT_RETURN dapl_ib_completion_poll(DAPL_HCA * hca_ptr, DAPL_EVD * evd_ptr, @@ -175,7 +175,7 @@ ib_cm_events_t dapl_ib_get_cm_event(DAT_EVENT_NUMBER dat_event_num); DAT_RETURN dapl_ib_cm_remote_addr(DAT_HANDLE dat_handle, - DAT_SOCK_ADDR6 * remote_ia_address); + struct sockaddr_in6 *remote_ia_address); /* diff -Nur -X /home/tduffy/dontdiff linux-kernel25/dat-provider/dapl_ep_connect.c linux-kernel26/dat-provider/dapl_ep_connect.c --- linux-kernel25/dat-provider/dapl_ep_connect.c 2005-05-02 08:48:16.984961000 -0700 +++ linux-kernel26/dat-provider/dapl_ep_connect.c 2005-05-09 14:14:07.867963000 -0700 @@ -237,8 +237,8 @@ spin_unlock(&ep_ptr->header.lock); /* Copy the connection qualifiers */ - memcpy(ep_ptr->param.remote_ia_address_ptr, - remote_ia_address, sizeof(DAT_SOCK_ADDR)); + memcpy(ep_ptr->param.remote_ia_address_ptr, remote_ia_address, + sizeof *(ep_ptr->param.remote_ia_address_ptr)); ep_ptr->param.remote_port_qual = remote_conn_qual; memcpy(ep_ptr->private_data,private_data,private_data_size); diff -Nur -X /home/tduffy/dontdiff linux-kernel25/dat-provider/dapl_ep_util.c linux-kernel26/dat-provider/dapl_ep_util.c --- linux-kernel25/dat-provider/dapl_ep_util.c 2005-05-09 12:21:06.403014000 -0700 +++ linux-kernel26/dat-provider/dapl_ep_util.c 2005-05-09 14:12:24.073960000 -0700 @@ -80,14 +80,14 @@ DAPL_EP *ep_ptr; /* Allocate EP */ - ep_ptr = (DAPL_EP *) kmalloc(sizeof(DAPL_EP) + sizeof(DAT_SOCK_ADDR), + ep_ptr = (DAPL_EP *) kmalloc(sizeof(DAPL_EP) + sizeof(struct sockaddr), GFP_ATOMIC); if (ep_ptr == NULL) { goto bail; } /* zero the structure */ - memset(ep_ptr, 0, sizeof(DAPL_EP) + sizeof(DAT_SOCK_ADDR)); + memset(ep_ptr, 0, sizeof(DAPL_EP) + sizeof(struct sockaddr)); /* * initialize the header diff -Nur -X /home/tduffy/dontdiff linux-kernel25/dat-provider/dapl.h linux-kernel26/dat-provider/dapl.h --- linux-kernel25/dat-provider/dapl.h 2005-05-09 14:08:06.217043000 -0700 +++ linux-kernel26/dat-provider/dapl.h 2005-05-09 14:11:58.543960000 -0700 @@ -213,7 +213,7 @@ atomic_t handle_ref_count; /* count of ia_opens on handle */ DAPL_EVD *async_evd; DAPL_EVD *async_error_evd; - DAT_SOCK_ADDR6 hca_address; /* local address of HCA */ + struct sockaddr_in6 hca_address; /* local address of HCA */ char *name; /* provider name */ ib_hca_handle_t ib_hca_handle; u8 port_num; /* number of physical port */ @@ -314,7 +314,7 @@ /* store the remote IA address here, reference from the param * struct which only has a pointer, no storage */ - DAT_SOCK_ADDR6 remote_ia_address; + struct sockaddr_in6 remote_ia_address; /* For passive connections we maintain a back pointer to the CR */ void *cr_ptr; @@ -414,7 +414,7 @@ /* IB specific fields */ ib_cm_handle_t ib_cm_handle; - DAT_SOCK_ADDR6 remote_ia_address; + struct sockaddr_in6 remote_ia_address; /* Assuming that the maximum private data size is small. * If it gets large, use of a pointer may be appropriate. */ diff -Nur -X /home/tduffy/dontdiff linux-kernel25/dat-provider/dapl_openib_cm.c linux-kernel26/dat-provider/dapl_openib_cm.c --- linux-kernel25/dat-provider/dapl_openib_cm.c 2005-05-06 16:22:03.000009000 -0700 +++ linux-kernel26/dat-provider/dapl_openib_cm.c 2005-05-09 14:15:24.898964000 -0700 @@ -812,7 +812,7 @@ * */ DAT_RETURN dapl_ib_cm_remote_addr(DAT_HANDLE dat_handle, - DAT_SOCK_ADDR6 * remote_ia_address) + struct sockaddr_in6 *remote_ia_address) { DAPL_IA *ia_ptr; int rc; @@ -828,7 +828,7 @@ if (!memcmp(&ia_ptr->hca_ptr->ib_trans.gid, &remote_gid, 16)) { memcpy(remote_ia_address, &ia_ptr->hca_ptr->hca_address, - sizeof(DAT_SOCK_ADDR)); + sizeof(struct sockaddr)); return DAT_SUCCESS; } diff -Nur -X /home/tduffy/dontdiff linux-kernel25/dat-provider/dapl_openib_util.c linux-kernel26/dat-provider/dapl_openib_util.c --- linux-kernel25/dat-provider/dapl_openib_util.c 2005-05-09 14:09:32.103960000 -0700 +++ linux-kernel26/dat-provider/dapl_openib_util.c 2005-05-09 14:15:36.673960000 -0700 @@ -748,7 +748,7 @@ DAT_RETURN dapl_ib_query_hca(DAPL_HCA * hca_ptr, struct dat_ia_attr *ia_attr, struct dat_ep_attr *ep_attr, - DAT_SOCK_ADDR6 * ip_addr) + struct sockaddr_in6 *ip_addr) { int ib_status; struct ib_device_attr device_attr; diff -Nur -X /home/tduffy/dontdiff linux-kernel25/test/dapltest/cmd/dapl_netaddr.c linux-kernel26/test/dapltest/cmd/dapl_netaddr.c --- linux-kernel25/test/dapltest/cmd/dapl_netaddr.c 2005-04-06 22:02:37.114981000 -0700 +++ linux-kernel26/test/dapltest/cmd/dapl_netaddr.c 2005-05-09 14:12:24.095963000 -0700 @@ -32,7 +32,7 @@ { DAT_IA_ADDRESS_PTR netaddr; - netaddr = (DAT_IA_ADDRESS_PTR) DT_Mdep_Malloc (sizeof (DAT_SOCK_ADDR)); + netaddr = (DAT_IA_ADDRESS_PTR) DT_Mdep_Malloc (sizeof (struct sockaddr)); if (!netaddr) { DT_Mdep_printf ("dapltest: No Memory to create netaddr!\n"); diff -Nur -X /home/tduffy/dontdiff linux-kernel25/test/dapltest/include/dapl_params.h linux-kernel26/test/dapltest/include/dapl_params.h --- linux-kernel25/test/dapltest/include/dapl_params.h 2005-05-05 09:42:06.763028000 -0700 +++ linux-kernel26/test/dapltest/include/dapl_params.h 2005-05-09 14:12:24.103960000 -0700 @@ -62,7 +62,7 @@ /* Needed here due to structure of command processing */ DAT_QOS ReliabilityLevel; - DAT_SOCK_ADDR server_netaddr; + struct sockaddr server_netaddr; void * phead; bool local_is_little_endian; bool debug; diff -Nur -X /home/tduffy/dontdiff linux-kernel25/test/dapltest/test/dapl_test_util.c linux-kernel26/test/dapltest/test/dapl_test_util.c --- linux-kernel25/test/dapltest/test/dapl_test_util.c 2005-05-09 13:31:50.623960000 -0700 +++ linux-kernel26/test/dapltest/test/dapl_test_util.c 2005-05-09 14:11:58.578962000 -0700 @@ -72,7 +72,7 @@ */ if (DT_dapltest_debug) { - DAT_SOCK_ADDR6 *ip6_addr; + struct sockaddr_in6 *ip6_addr; struct sockaddr_in *ip_addr; DT_Tdep_PT_Printf (phead, "***** DAPL Characteristics *****\n"); @@ -111,7 +111,7 @@ DT_Tdep_PT_Printf (phead, "\tMaximum Private data size %d bytes\n", pt_ptr->provider_attr.max_private_data_size ); - ip6_addr = (DAT_SOCK_ADDR6 *)pt_ptr->ia_attr.ia_address_ptr; + ip6_addr = (struct sockaddr_in6 *)pt_ptr->ia_attr.ia_address_ptr; if (ip6_addr->sin6_family == AF_INET6 ) { DT_Tdep_PT_Printf (phead, "\tLocal IP address %x:%x:%x:%x:%x:%x:%x:%x:\n", From mshefty at ichips.intel.com Mon May 9 14:48:13 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Mon, 09 May 2005 14:48:13 -0700 Subject: [openib-general] in need of a simple ulp In-Reply-To: <427BA0AD.6010105@linuxmachines.com> References: <426FCF0A.3070806@linuxmachines.com> <426FD098.9090300@ichips.intel.com> <427AC218.9070307@linuxmachines.com> <427AC385.5070308@ichips.intel.com> <427BA0AD.6010105@linuxmachines.com> Message-ID: <427FDA9D.7030905@ichips.intel.com> Jeff Carr wrote: > I didn't notice this error. In any case, it was something I did wrong; I > went back and did a simple check with your code and it is ok. I do > notice though that you can generate: > > May 5 16:31:50 localhost kernel: ib_mthca 0000:09:00.0: 1a0084/0: error > CQE -> QPN 1a0406, WQE @ 00000042 > May 5 16:31:50 localhost kernel: [ 0] 001a0406 > May 5 16:31:50 localhost kernel: [ 4] 00001aed > May 5 16:31:50 localhost kernel: [ 8] 00000004 > May 5 16:31:50 localhost kernel: [ c] 00003800 > May 5 16:31:50 localhost kernel: [10] 128a0000 > May 5 16:31:50 localhost kernel: [14] 00000000 > May 5 16:31:50 localhost kernel: [18] 00000042 > May 5 16:31:50 localhost kernel: [1c] ff000000 > > if you up the message_count to 0x1000. I'm guessing this is just some > normal overrun error though. It's taken me a while to look at this, but I think that this is a real error. Cmpost is setting the CQ size too small, which can lead to the CQ overrun. The number of cqe's should have been message_count * 2, rather than just message_count. Message_count is fine on the client side, which receives all messages before sending. But on the server side, receives could begin coming in before all sends are done. Thanks for the info. I've submitted a change that should fix this. - Sean From Tom.Duffy at Sun.COM Mon May 9 14:39:45 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Mon, 09 May 2005 14:39:45 -0700 Subject: [openib-general] [PATCH][25/26] kDAPL: Remove typedef DAT_IA_ATTR In-Reply-To: <11156747853473@sun.com> References: <11156747793273@sun.com> <11156747791462@sun.com> <1115674780270@sun.com> <11156747803410@sun.com> <11156747802252@sun.com> <11156747801195@sun.com> <11156747812210@sun.com> <11156747812290@sun.com> <11156747812388@sun.com> <11156747811508@sun.com> <11156747821301@sun.com> <1115674782460@sun.com> <11156747821275@sun.com> <11156747823890@sun.com> <11156747831715@sun.com> <11156747831729@sun.com> <11156747831783@sun.com> <11156747832991@sun.com> <11156747841537@sun.com> <11156747843960@sun.com> <11156747844081@sun.com> <11156747842198@sun.com> <11156747851961@sun.com> <11156747851027@sun.com> <11156747853473@sun.com> Message-ID: <11156747851643@sun.com> Signed-off-by: Tom Duffy diff -Nur -X /home/tduffy/dontdiff linux-kernel24/dat/dat.h linux-kernel25/dat/dat.h --- linux-kernel24/dat/dat.h 2005-05-09 14:05:28.655967000 -0700 +++ linux-kernel25/dat/dat.h 2005-05-09 14:08:31.214971000 -0700 @@ -1005,7 +1005,7 @@ }; /* General Interface Adapter attributes.*/ -typedef struct dat_ia_attr { +struct dat_ia_attr { char adapter_name[DAT_NAME_MAX_LENGTH]; char vendor_name[DAT_NAME_MAX_LENGTH]; u32 hardware_version_major; @@ -1041,7 +1041,7 @@ struct dat_named_attr *transport_attr; DAT_COUNT num_vendor_attr; struct dat_named_attr *vendor_attr; -} DAT_IA_ATTR; +}; /* Provider attributes */ typedef enum dat_iov_ownership { @@ -1117,7 +1117,7 @@ typedef DAT_RETURN (*DAT_IA_CLOSE_FUNC) (DAT_IA_HANDLE, DAT_CLOSE_FLAGS); typedef DAT_RETURN (*DAT_IA_QUERY_FUNC) (DAT_IA_HANDLE, DAT_EVD_HANDLE *, - DAT_IA_ATTR *, + struct dat_ia_attr *, struct dat_provider_attr *); typedef DAT_RETURN (*DAT_SET_CONSUMER_CONTEXT_FUNC) (DAT_HANDLE, DAT_CONTEXT); @@ -1439,7 +1439,7 @@ static inline DAT_RETURN dat_ia_query(DAT_IA_HANDLE ia, DAT_EVD_HANDLE * async_evd, - DAT_IA_ATTR * ia_attr, + struct dat_ia_attr *ia_attr, struct dat_provider_attr *provider_attr) { return DAT_CALL_PROVIDER_FUNC( diff -Nur -X /home/tduffy/dontdiff linux-kernel24/dat-provider/dapl_adapter_util.h linux-kernel25/dat-provider/dapl_adapter_util.h --- linux-kernel24/dat-provider/dapl_adapter_util.h 2005-05-09 12:17:54.079004000 -0700 +++ linux-kernel25/dat-provider/dapl_adapter_util.h 2005-05-09 14:09:01.530972000 -0700 @@ -146,7 +146,7 @@ DAPL_COOKIE * cookie, DAT_BOOLEAN is_signaled); DAT_RETURN dapl_ib_query_hca(DAPL_HCA * hca_ptr, - DAT_IA_ATTR * ia_attr, + struct dat_ia_attr *ia_attr, struct dat_ep_attr *ep_attr, DAT_SOCK_ADDR6 * ip_addr); diff -Nur -X /home/tduffy/dontdiff linux-kernel24/dat-provider/dapl.h linux-kernel25/dat-provider/dapl.h --- linux-kernel24/dat-provider/dapl.h 2005-05-09 14:05:04.217962000 -0700 +++ linux-kernel25/dat-provider/dapl.h 2005-05-09 14:08:06.217043000 -0700 @@ -221,7 +221,7 @@ /* Memory Subsystem Support */ DAPL_HASH_TABLE *lmr_hash_table; /* Limits & useful HCA attributes */ - DAT_IA_ATTR ia_attr; + struct dat_ia_attr ia_attr; }; /* DAPL Objects always have the following header */ @@ -486,7 +486,7 @@ extern DAT_RETURN dapl_ia_query(DAT_IA_HANDLE, DAT_EVD_HANDLE *, - DAT_IA_ATTR *, + struct dat_ia_attr *, struct dat_provider_attr *); /* helper functions */ diff -Nur -X /home/tduffy/dontdiff linux-kernel24/dat-provider/dapl_ia_query.c linux-kernel25/dat-provider/dapl_ia_query.c --- linux-kernel24/dat-provider/dapl_ia_query.c 2005-05-09 11:18:08.146002000 -0700 +++ linux-kernel25/dat-provider/dapl_ia_query.c 2005-05-09 14:09:22.845962000 -0700 @@ -62,7 +62,7 @@ DAT_RETURN dapl_ia_query(DAT_IA_HANDLE ia_handle, DAT_EVD_HANDLE * async_evd_handle, - DAT_IA_ATTR * ia_attr, + struct dat_ia_attr *ia_attr, struct dat_provider_attr *provider_attr) { DAPL_IA *ia_ptr; diff -Nur -X /home/tduffy/dontdiff linux-kernel24/dat-provider/dapl_openib_util.c linux-kernel25/dat-provider/dapl_openib_util.c --- linux-kernel24/dat-provider/dapl_openib_util.c 2005-05-09 12:22:39.554000000 -0700 +++ linux-kernel25/dat-provider/dapl_openib_util.c 2005-05-09 14:09:32.103960000 -0700 @@ -746,7 +746,7 @@ * DAT_INVALID_PARAMETER */ DAT_RETURN dapl_ib_query_hca(DAPL_HCA * hca_ptr, - DAT_IA_ATTR * ia_attr, + struct dat_ia_attr *ia_attr, struct dat_ep_attr *ep_attr, DAT_SOCK_ADDR6 * ip_addr) { diff -Nur -X /home/tduffy/dontdiff linux-kernel24/dat-provider/dapl_vendor.h linux-kernel25/dat-provider/dapl_vendor.h --- linux-kernel24/dat-provider/dapl_vendor.h 2005-05-09 11:15:26.435002000 -0700 +++ linux-kernel25/dat-provider/dapl_vendor.h 2005-05-09 14:08:06.235963000 -0700 @@ -31,14 +31,14 @@ * * PURPOSE: * Vendor provides values for their implementation. Most of - * these values are returned in the DAT_IA_ATTR parameter of + * these values are returned in the struct dat_ia_attr parameter of * dat_ia_query() * * $Id: dapl_vendor.h 2190 2005-04-19 20:01:29Z jlentini $ **********************************************************************/ /********************************************************************** - * DAT_IA_ATTR attributes + * struct dat_ia_attr attributes * * These values are used in the provider support routine * dapl_ib_query_hca (). Many of the values there are HW @@ -50,13 +50,13 @@ /* * Product name of the adapter. - * Returned in DAT_IA_ATTR.adapter_name + * Returned in struct dat_ia_attr.adapter_name */ #define VN_ADAPTER_NAME "Generic InfiniBand HCA" /* * Vendor name - * Returned in DAT_IA_ATTR.vendor_name + * Returned in struct dat_ia_attr.vendor_name */ #define VN_VENDOR_NAME "DAPL Reference Implementation" diff -Nur -X /home/tduffy/dontdiff linux-kernel24/test/dapltest/include/dapl_performance_test.h linux-kernel25/test/dapltest/include/dapl_performance_test.h --- linux-kernel24/test/dapltest/include/dapl_performance_test.h 2005-05-09 12:22:53.931012000 -0700 +++ linux-kernel25/test/dapltest/include/dapl_performance_test.h 2005-05-09 14:09:53.078968000 -0700 @@ -69,7 +69,7 @@ DAT_IA_ADDRESS_PTR remote_ia_addr; DAT_BOOLEAN is_remote_little_endian; DAT_CONN_QUAL base_port; - DAT_IA_ATTR ia_attr; + struct dat_ia_attr ia_attr; DAT_IA_HANDLE ia_handle; DAT_PZ_HANDLE pz_handle; DAT_CNO_HANDLE cno_handle; diff -Nur -X /home/tduffy/dontdiff linux-kernel24/test/dapltest/include/dapl_test_data.h linux-kernel25/test/dapltest/include/dapl_test_data.h --- linux-kernel24/test/dapltest/include/dapl_test_data.h 2005-05-09 12:16:33.875003000 -0700 +++ linux-kernel25/test/dapltest/include/dapl_test_data.h 2005-05-09 14:08:06.256979000 -0700 @@ -80,7 +80,7 @@ Server_Info_t Server_Info; Client_Info_t Client_Info; Params_t Params; - DAT_IA_ATTR ia_attr; + struct dat_ia_attr ia_attr; struct dat_provider_attr provider_attr; struct dat_ep_attr ep_attr; Per_Server_Data_t *ps_ptr; diff -Nur -X /home/tduffy/dontdiff linux-kernel24/test/dapltest/test/dapl_fft_queryinfo.c linux-kernel25/test/dapltest/test/dapl_fft_queryinfo.c --- linux-kernel24/test/dapltest/test/dapl_fft_queryinfo.c 2005-05-09 14:05:04.246964000 -0700 +++ linux-kernel25/test/dapltest/test/dapl_fft_queryinfo.c 2005-05-09 14:08:06.261014000 -0700 @@ -45,7 +45,7 @@ { char *dev_name; DAT_IA_HANDLE ia_handle; - DAT_IA_ATTR ia_attributes; + struct dat_ia_attr ia_attributes; struct dat_provider_attr provider_attributes; DAT_EVD_HANDLE evd_handle; DAT_EVD_HANDLE conn_evd_handle; From libor at topspin.com Mon May 9 15:39:29 2005 From: libor at topspin.com (Libor Michalek) Date: Mon, 9 May 2005 15:39:29 -0700 Subject: [openib-general] [PATCH] SDP: remove redundant setting of sk_family In-Reply-To: <1115509269.23057.14.camel@duffman>; from tduffy@sun.com on Sat, May 07, 2005 at 04:41:09PM -0700 References: <1115509269.23057.14.camel@duffman> Message-ID: <20050509153929.B2462@topspin.com> On Sat, May 07, 2005 at 04:41:09PM -0700, Tom Duffy wrote: > sk_alloc() already takes care of setting sk_family to proto. No need to > do it a second time in sdp. Thanks, applied and commited. -Libor From iod00d at hp.com Mon May 9 19:03:41 2005 From: iod00d at hp.com (Grant Grundler) Date: Mon, 9 May 2005 19:03:41 -0700 Subject: [openib-general] Re: user-mode verbs on Itanium In-Reply-To: <20050509165022.GJ9941@mellanox.co.il> References: <20050429162127.GA24871@esmail.cup.hp.com> <52ll71k09w.fsf@topspin.com> <20050429183846.GH24871@esmail.cup.hp.com> <20050504005534.GA13574@esmail.cup.hp.com> <20050504070910.GC9674@mellanox.co.il> <20050506190947.GA23958@esmail.cup.hp.com> <20050507053706.GN23958@esmail.cup.hp.com> <20050508124916.GM9941@mellanox.co.il> <20050509155738.GA1319@esmail.cup.hp.com> <20050509165022.GJ9941@mellanox.co.il> Message-ID: <20050510020341.GA2633@esmail.cup.hp.com> On Mon, May 09, 2005 at 07:50:22PM +0300, Michael S. Tsirkin wrote: > Hello, Grant! > Thanks for comments on the perftest. > I think fixed all bug you found. probably - welcome > I just did > > >svn mv https://openib.org/svn/trunk/contrib/mellanox/perftest https://openib.org/svn/gen2/trunk/src/userspace > Committed revision 2285. > > so we'll be able to exchange patches in the regular way. Excellent - thanks! ... > > BTW, copying code wholesale is the whole point behind "Open Source". > > One measure of success is if people in fact do copy the code. > > Right but why dont we put it in the core then? Isn't "userspace/perftest" part of "core"? What do you mean by "core"? I was just looking for a place more visible than trunk/contrib/mellanox which on the surface has nothing to do with Openib Gen2. > > That might help, but it's not addressing the basic problem. > > People retrofitting an existing application will need at least > > three different peices of code: subsystem initialization, connection > > setup/tear down, and runtime communication. Just making those three > > code blocks subroutines is NOT creating a new layer. > > But once these subroutines are moved into a library, it sure > begins to look like one. Sure, but one where people can take the code and adapt it for their own use. In effect implementing a custom layer each time. Some people *need* another layer (e.g. MPI) and don't want to program directly to verbs. They won't look for this example. ... > > That's just my methodology for cleaning up stuff like this > > and not the end point. > > I see how separating things to subroutines would help here, > but making things global will just hide who uses what, IMO. Yes, but I don't want to make everything global in the end. I mostly want to seperate parameters from local variables. > > IMHO, the current code isn't readable either. > > main() has 268 lines and 22 (in my version, more in the original) > > local variables. Of those 22 variables, some are really local > > and not part of the global state but we can't easily tell which. > > Its a work in progress :) hehe...exactly :^) > I'm not sure what do you call "global state". I am trying to think > of an application that has many connections, global state would > be something common to all of them? e.g. for rdma_lat global state would be: o the connection since only one is allocated o tstamp[] since it only runs on one CPU o etc. Ideally though, we want the connection state to be a parameter in most normal apps. So I'd like to sort that out after seperating "local variables" (e.g. unsigned int i;). > You'll need my two patches I sent previously to make it work though. > As a work around, bump max_send_sge a bit. *nod* I'll added that to my code. I need to submit patches against src/userspace/perftest/ so it looks more like what I have now. :^) > > > On the other hand, it could be negligible compared to connection > > > setup time. > > > > hrm...true. We should measure and report that too. > > A script could iterate the invocation with -n 1 to collect > > a reasonable sample. > > Problem is, connection setup involves the CM (or the socket kludge), > so its not trivial to define what the overhead is. Well, just invent a definition that works for that program. The time the rdma_lat uses to setup *before* hitting the main loop would be a reasonable definition. > Ok, I checked that in: rev 2283. I also renamed --histogram to --report-all. Yeah, I'm still wondering if anyone will want both in the same report. The histogram can be derived from the unsorted listing. (obviously not the converse). The histogram is interesting from statiscal evalaulation of the data (standard deviation sort of stuff). But I'm only interested in the histogram to validate the test environment is reliable enough. After that I mostly want unsorted list. > > > 2. get_clock in assembly replaced with asm/timex.h: > > > Unfortunately asm/timex.h was never intended for userspace, > > > > Exact copies from linux/include/asm* are not intended for userspace. > > But arch specific kernel headers are required to interact with the > > kernel on any given platform (e.g. syscalls). They just need to be > > "adjusted" by distro's so they are suitable for userspace consumption. > > IMO get_cycles is never needed to interact with the kernel. Right. That's because it's referencing a read-only source of data. And accessing TSC through a system call would defeat the point of implementing it as an on-chip register readable by unprivileged apps. > It could have been part of libc or something, but its not. Not every interaction with the kernel goes through libc. (Thank $DEITY, glibc would be even more bloated than it already is). It would be nice if get_cycles were a gcc instric. gcc knows which arch/model CPU it's building code for. > > x86_64 and ppc don't need to include config.h. > > I'll submit patches to fix this. x86_64 patch submitted. ppc is harder to fix since they've wrapped __KERNEL__ around it. Seems like not all PPC CPUs have implemented cycle counters. PPC would be a good candidate for private one line assembly too. And some folks will just get undefined behavior. > > I suggest define get_cycles() only for i386 and let everyone else use timex.h. > > Sure. Still, I think I'll keep the one-line assembly till its fixed in > most distros :) Which distro is shipping on PPC? Waiting for any commercial x86-64 distro's to update to 2.6.12 (assuming my patch is accepted) will be a long wait. Then they have to update the kernel headers package too... > I added ia64 so everyone can make progress meanwhile. ok. That's me. :^) > BTW, some architectures (notably sparc) define get_cycles to (0). > Need to google for the right answer there. That would suggest it's not implemented for those architectures. Just like TSC was first implemented on pentium. > I also saw this errata for Itanium: > > 2. AR.ITC returns an incorrect value Yeah. And I'll take those chances (1 in 4 billion) if it keeps the code simpler and the machine won't crash because of it. > Does this bug still affect Itanium systems in use out there? Not many people liked Intanium 1. Noisy and fairly slow. If anyone *must* use Itanium 1 to measure performance, my deepest sympathies. > What about Itanium 2? It's not mentioned in the current Itanium 2 "Spec Updates". http://www.intel.com/design/itanium2/specupdt/251141.htm I'll assume it's fixed. grant From halr at voltaire.com Mon May 9 21:08:37 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 10 May 2005 00:08:37 -0400 Subject: [openib-general] CM ID destruction callback for kernel CM ? Message-ID: <1115698117.4490.1578.camel@localhost.localdomain> Hi Sean, The kernel CM is currently capable of destroying the CM ID at the end of the callback by setting ret to other than 0: * Users may not call ib_destroy_cm_id while in the context of this callback; * however, returning a non-zero value instructs the communication manager to * destroy the @cm_id after the callback completes. Perhaps also a CM ID destruction callback could be added to: struct ib_cm_id *ib_create_cm_id(ib_cm_handler cm_handler, void *context); which would also be called (if non NULL) on CM ID destruction at the end of the CM callback user invoked CM ID destruction. In that way, a client could cause some additional related destruction immediately subsequent to the CM ID destruction on its behalf. That might be useful to CM clients to simplify them. -- Hal From tziporet at mellanox.co.il Mon May 9 22:45:10 2005 From: tziporet at mellanox.co.il (Tziporet Koren) Date: Tue, 10 May 2005 08:45:10 +0300 Subject: [openib-general] HCA Firmware Message-ID: <506C3D7B14CDD411A52C00025558DED6064BF53D@mtlex01.yok.mtl.com> The last official release from Mellanox is 3.3.2 and one of the major new features is MSIX support. Tziporet -----Original Message----- From: Shirley Ma [mailto:xma at us.ibm.com] Sent: Monday, May 09, 2005 11:36 PM To: Hal Rosenstock Cc: openib-general-bounces at openib.org; openib-general at openib.org Subject: Re: [openib-general] HCA Firmware > I don't recall if there are issues running with 3.2.0 or not. 3.2.0 has no problem. Even 3.1.0 works fine for me. Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From mst at mellanox.co.il Tue May 10 03:00:21 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Tue, 10 May 2005 13:00:21 +0300 Subject: [openib-general] [PATCH] sdpc_buff trivial changes Message-ID: <20050510100021.GA2474@mellanox.co.il> struct sdpc_buff has a copy of ib_sge and uses casts to turn that into a real ib_sge. The right thing is obviously to keep ib_sge as part of sdpc_buff. Signed-off-by: Michael S. Tsirkin Index: ulp/sdp/sdp_buff.h =================================================================== --- ulp/sdp/sdp_buff.h (revision 2292) +++ ulp/sdp/sdp_buff.h (working copy) @@ -71,9 +71,7 @@ struct sdpc_buff { * IB specific data (The main buffer pool sets the lkey when * it is created) */ - u64 real; /* component of scather/gather list (address) */ - u32 size; /* component of scather/gather list (lenght) */ - u32 lkey; /* component of scather/gather list (key) */ + struct ib_sge sge; }; struct sdpc_buff_root { @@ -112,13 +110,7 @@ struct sdpc_buff_root { #define SDP_BUFF_F_GET_UNSIG(buff) ((buff)->flags & SDP_BUFF_F_UNSIG) #define SDP_BUFF_F_SET_UNSIG(buff) ((buff)->flags |= SDP_BUFF_F_UNSIG) #define SDP_BUFF_F_CLR_UNSIG(buff) ((buff)->flags &= (~SDP_BUFF_F_UNSIG)) -/* - * data accessors. - */ -#define SDP_BUFF_GAT_SCAT(buff) \ - ({ (buff)->real = virt_to_phys((buff)->data); \ - (buff)->size = (buff)->tail - (buff)->data; \ - (struct ib_sge *)(&(buff)->real); }) + /* * pool size */ Index: ulp/sdp/sdp_rcvd.c =================================================================== --- ulp/sdp/sdp_rcvd.c (revision 2292) +++ ulp/sdp/sdp_rcvd.c (working copy) @@ -1137,7 +1137,7 @@ int sdp_event_recv(struct sdp_opt *conn, } dma_unmap_single(conn->ca->dma_device, - buff->real, + buff->sge.addr, buff->tail - buff->data, PCI_DMA_FROMDEVICE); Index: ulp/sdp/sdp_send.c =================================================================== --- ulp/sdp/sdp_send.c (revision 2292) +++ ulp/sdp/sdp_send.c (working copy) @@ -57,7 +57,7 @@ static int sdp_send_buff_post(struct sdp conn->oob_offset -= (conn->oob_offset > 0) ? buff->data_size : 0; buff->wrid = conn->send_wrid++; - buff->lkey = conn->l_key; + buff->sge.lkey = conn->l_key; buff->bsdh_hdr->recv_bufs = conn->l_advt_bf; buff->bsdh_hdr->size = buff->tail - buff->data; buff->bsdh_hdr->seq_num = ++conn->send_seq; @@ -136,14 +136,14 @@ static int sdp_send_buff_post(struct sdp /* * post send */ - buff->size = buff->tail - buff->data; - buff->real = dma_map_single(conn->ca->dma_device, + buff->sge.length = buff->tail - buff->data; + buff->sge.addr = dma_map_single(conn->ca->dma_device, buff->data, - buff->size, + buff->sge.length, PCI_DMA_TODEVICE); send_param.next = NULL; send_param.wr_id = buff->wrid; - send_param.sg_list = (struct ib_sge *)&buff->real; + send_param.sg_list = &buff->sge; send_param.num_sge = 1; send_param.opcode = IB_WR_SEND; @@ -350,7 +350,7 @@ static int sdp_send_data_buff_snk(struct send_param.send_flags = IB_SEND_SIGNALED; buff->wrid = conn->send_wrid++; - buff->lkey = conn->l_key; + buff->sge.lkey = conn->l_key; advt->wrid = buff->wrid; advt->size -= (buff->tail - buff->data); @@ -382,9 +382,12 @@ static int sdp_send_data_buff_snk(struct /* * post RDMA */ + buff->sge.addr = virt_to_phys(buff->data); + buff->sge.length = buff->tail - buff->data; + send_param.next = NULL; send_param.wr_id = buff->wrid; - send_param.sg_list = SDP_BUFF_GAT_SCAT(buff); + send_param.sg_list = &buff->sge; send_param.num_sge = 1; result = ib_post_send(conn->qp, &send_param, &bad_wr); Index: ulp/sdp/sdp_recv.c =================================================================== --- ulp/sdp/sdp_recv.c (revision 2292) +++ ulp/sdp/sdp_recv.c (working copy) @@ -65,7 +65,7 @@ static int sdp_post_recv_buff(struct sdp */ buff->tail = buff->end; buff->data = buff->tail - conn->recv_size; - buff->lkey = conn->l_key; + buff->sge.lkey = conn->l_key; buff->wrid = conn->recv_wrid++; conn->l_recv_bf++; @@ -86,14 +86,14 @@ static int sdp_post_recv_buff(struct sdp /* * post recv */ - buff->size = buff->tail - buff->data; - buff->real = dma_map_single(conn->ca->dma_device, + buff->sge.length = buff->tail - buff->data; + buff->sge.addr = dma_map_single(conn->ca->dma_device, buff->data, - buff->size, + buff->sge.length, PCI_DMA_FROMDEVICE); receive_param.next = NULL; receive_param.wr_id = buff->wrid; - receive_param.sg_list = (struct ib_sge *)&buff->real; + receive_param.sg_list = &buff->sge; receive_param.num_sge = 1; result = ib_post_recv(conn->qp, &receive_param, &bad_wr); @@ -155,7 +155,7 @@ static int sdp_post_rdma_buff(struct sdp */ buff->tail = buff->end; buff->data = buff->tail - min((s32)conn->recv_size, advt->size); - buff->lkey = conn->l_key; + buff->sge.lkey = conn->l_key; buff->wrid = conn->send_wrid++; send_param.opcode = IB_WR_RDMA_READ; @@ -196,9 +196,12 @@ static int sdp_post_rdma_buff(struct sdp /* * post rdma */ - send_param.next = NULL; - send_param.wr_id = buff->wrid; - send_param.sg_list = SDP_BUFF_GAT_SCAT(buff); + buff->sge.addr = virt_to_phys(buff->data); + buff->sge.length = buff->tail - buff->data; + + send_param.next = NULL; + send_param.wr_id = buff->wrid; + send_param.sg_list = &buff->sge; send_param.num_sge = 1; result = ib_post_send(conn->qp, &send_param, &bad_wr); Index: ulp/sdp/sdp_sent.c =================================================================== --- ulp/sdp/sdp_sent.c (revision 2292) +++ ulp/sdp/sdp_sent.c (working copy) @@ -380,7 +380,7 @@ int sdp_event_send(struct sdp_opt *conn, } dma_unmap_single(conn->ca->dma_device, - buff->real, buff->tail - buff->data, + buff->sge.addr, buff->tail - buff->data, PCI_DMA_TODEVICE); /* Index: ulp/sdp/sdp_buff.c =================================================================== --- ulp/sdp/sdp_buff.c (revision 2292) +++ ulp/sdp/sdp_buff.c (working copy) @@ -321,7 +321,7 @@ void sdp_buff_q_clear_unmap(struct sdpc_ while ((buff = do_buff_q_get(pool, 0, NULL, NULL))) { if (dev) - dma_unmap_single(dev, buff->real, + dma_unmap_single(dev, buff->sge.addr, buff->tail - buff->data, direction); result = sdp_buff_pool_put(buff); @@ -418,15 +418,15 @@ static int sdp_buff_pool_alloc(struct sd break; } - buff->end = buff->head + PAGE_SIZE; - buff->data = buff->head; - buff->tail = buff->head; - buff->lkey = 0; - buff->real = 0; - buff->size = 0; - buff->pool = NULL; - buff->type = SDP_DESC_TYPE_BUFF; - buff->release = sdp_buff_pool_put; + buff->end = buff->head + PAGE_SIZE; + buff->data = buff->head; + buff->tail = buff->head; + buff->sge.lkey = 0; + buff->sge.addr = 0; + buff->sge.length = 0; + buff->pool = NULL; + buff->type = SDP_DESC_TYPE_BUFF; + buff->release = sdp_buff_pool_put; result = sdp_buff_q_put(&m_pool->pool, buff); if (result < 0) { @@ -619,7 +619,7 @@ struct sdpc_buff *sdp_buff_pool_get(void */ buff->bsdh_hdr = NULL; buff->flags = 0; - buff->lkey = 0; + buff->sge.lkey = 0; buff->data_size = 0; buff->wrid = 0; -- MST - Michael S. Tsirkin From halr at voltaire.com Tue May 10 05:51:16 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 10 May 2005 08:51:16 -0400 Subject: [openib-general] [PATCH] kdapl: Eliminate vendor specific comments Message-ID: <1115729328.4493.753.camel@localhost.localdomain> Eliminate vendor specific comments Also, fix some typos Signed-off-by: Hal Rosenstock Index: dapl.h =================================================================== --- dapl.h (revision 2272) +++ dapl.h (working copy) @@ -279,7 +279,7 @@ /* Every EVD has a CQ unless it is a SOFTWARE_EVENT only EVD */ ib_cq_handle_t ib_cq_handle; - /* Mellanox Specific completion handle for registration/de-registration */ + /* Completion handle for registration/de-registration */ ib_comp_handle_t ib_comp_handle; /* An Event Dispatcher cannot be freed while @@ -325,12 +325,12 @@ /* What the DAT Consumer asked for */ DAT_EP_PARAM param; - /* The RC Queue Pair (IBM OS API) */ + /* The RC Queue Pair */ ib_qp_handle_t qp_handle; unsigned int qpn; /* qp number */ ib_qp_state_t qp_state; - /* communications manager handle (IBM OS API) */ + /* communications manager handle */ ib_cm_handle_t cm_handle; /* store the remote IA address here, reference from the param * struct which only has a pointer, no storage @@ -417,7 +417,7 @@ /* maintenence fields */ DAT_BOOLEAN listening; /* PSP is registered & active */ - ib_cm_srvc_handle_t cm_srvc_handle; /* Used by Mellanox CM */ + ib_cm_srvc_handle_t cm_srvc_handle; /* Used by CM */ DAPL_LLIST_HEAD cr_list_head; /* CR pending queue */ DAT_COUNT cr_list_count; /* count of CRs on queue */ }; Index: dapl_vendor.h =================================================================== --- dapl_vendor.h (revision 2272) +++ dapl_vendor.h (working copy) @@ -105,8 +105,8 @@ * * Values that might be used: * #define VN_PREFIX "ia" (generic prefix) - * #define VN_PREFIX "jni" (JNI: OS Acces API) - * #define VN_PREFIX "ibm" (IBM: OS Acces API) + * #define VN_PREFIX "jni" (JNI: OS Access API) + * #define VN_PREFIX "ibm" (IBM: OS Access API) * #define VN_PREFIX "" (Mellanox: VAPI) * #define VN_PREFIX "" (Intel: IB Common API) */ Index: dapl_ep_util.c =================================================================== --- dapl_ep_util.c (revision 2272) +++ dapl_ep_util.c (working copy) @@ -114,9 +114,6 @@ ep_ptr->param.ep_attr = *ep_attr; } - /* - * IBM OS API specific fields - */ ep_ptr->qp_handle = IB_INVALID_HANDLE; ep_ptr->qpn = 0; ep_ptr->qp_state = DAPL_QP_STATE_UNATTACHED; From mst at mellanox.co.il Tue May 10 07:32:32 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Tue, 10 May 2005 17:32:32 +0300 Subject: [openib-general] [PATCH] (repost) sdp: replace mlock with get_user_pages In-Reply-To: <20050506175415.I26823@topspin.com> References: <20050505110158.GH30653@mellanox.co.il> <20050506175415.I26823@topspin.com> Message-ID: <20050510143232.GF2474@mellanox.co.il> Quoting r. Libor Michalek : > Subject: Re: [PATCH] sdp: replace mlock with get_user_pages > > On Thu, May 05, 2005 at 02:01:58PM +0300, Michael S. Tsirkin wrote: > > Hello, Libor! > > The following patch replaces the mlock hack with call > > to get_user_pages. Since the application could have forked > > while an iocb is outstanding, when an iocb is done > > I do get_user_pages for a second time and copy data if > > the physical address has changed. > > > > Thus, changing ulimit is no longer required to get aio > > working, processes are also allowed to fork and to call mlock/munlock > > on the buffer. > > > > Tested by the ttcp.aio benchmark, and works fine for me. > > In the latest kernel what happens to a page that's held > with a get_user_page reference on fork? Which is the only > case left. Since there is an iocv flag parameter it would > be prefered to use a bit there instead of is_receive. Also, > please leave spaces around '='. Finally it would be nice > if we could get rid of having to schedule a task to unlock > the iocb in complete, I've noticed this having a negative > effect on system CPU utilization on older kernels. > > -Libor I dont see what would prevent the page from being copied on a COW event. Do you? Certainly the patch in 2.6.7 that fixed the swap-out problem does not affect COW. With regard to avoiding task scheduling, since current code already needs this, so we are not breaking anything. I put in comments so that we remember to do this research at some point in time. In the patch below the style issues are addressed: whitespace fixed, is_receive changed to flag. OK to check in? Signed-off-by: Michael S. Tsirkin Index: ulp/sdp/sdp_send.c =================================================================== --- ulp/sdp/sdp_send.c (revision 2292) +++ ulp/sdp/sdp_send.c (working copy) @@ -2197,7 +2197,7 @@ skip: /* entry point for IOCB based tran iocb->req = req; iocb->key = req->ki_key; iocb->addr = (unsigned long)msg->msg_iov->iov_base - copied; - + req->ki_cancel = sdp_inet_write_cancel; result = sdp_iocb_lock(iocb); Index: ulp/sdp/sdp_recv.c =================================================================== --- ulp/sdp/sdp_recv.c (revision 2292) +++ ulp/sdp/sdp_recv.c (working copy) @@ -1449,6 +1449,8 @@ int sdp_inet_recv(struct kiocb *req, st iocb->addr = ((unsigned long)msg->msg_iov->iov_base - copied); + iocb->flags |= SDP_IOCB_F_RECV; + req->ki_cancel = sdp_inet_read_cancel; result = sdp_iocb_lock(iocb); Index: ulp/sdp/sdp_iocb.c =================================================================== --- ulp/sdp/sdp_iocb.c (revision 2292) +++ ulp/sdp/sdp_iocb.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Mellanox Technologies Ltd. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -31,389 +32,196 @@ * * $Id$ */ - +#include #include "sdp_main.h" static kmem_cache_t *sdp_iocb_cache = NULL; -/* - * memory locking functions - */ -#include - -typedef int (*do_mlock_ptr_t)(unsigned long, size_t, int); -static do_mlock_ptr_t mlock_ptr = NULL; - -/* - * do_iocb_unlock - unlock the memory for an IOCB - */ -static int do_iocb_unlock(struct sdpc_iocb *iocb) -{ - struct vm_area_struct *vma; - - vma = find_vma(iocb->mm, (iocb->addr & PAGE_MASK)); - if (!vma) - sdp_warn("No VMA for IOCB <%lx:%Zu> unlock", - iocb->addr, iocb->size); - - while (vma) { - sdp_dbg_data(NULL, - "unmark <%lx> <%p> <%08lx:%08lx> <%08lx> <%ld>", - iocb->addr, vma, vma->vm_start, vma->vm_end, - vma->vm_flags, (long)vma->vm_private_data); - - spin_lock(&iocb->mm->page_table_lock); - /* - * if there are no more references to the vma - */ - vma->vm_private_data--; - - if (!vma->vm_private_data) { - /* - * modify VM flags. - */ - vma->vm_flags &= ~(VM_DONTCOPY|VM_LOCKED); - /* - * adjust locked page count - */ - vma->vm_mm->locked_vm -= ((vma->vm_end - - vma->vm_start) >> - PAGE_SHIFT); - } - - spin_unlock(&iocb->mm->page_table_lock); - /* - * continue if the buffer continues onto the next vma - */ - if ((iocb->addr + iocb->size) > vma->vm_end) - vma = vma->vm_next; - else - vma = NULL; - } - - return 0; +static void sdp_copy_one_page(struct page *from, struct page* to, + unsigned long iocb_addr, size_t iocb_size, + unsigned long uaddr) +{ + size_t size_left = iocb_addr + iocb_size - uaddr; + size_t size = min(size_left,PAGE_SIZE); + unsigned long offset = uaddr % PAGE_SIZE; + unsigned long flags; + + void* fptr; + void* tptr; + + local_irq_save(flags); + fptr = kmap_atomic(from, KM_IRQ0); + tptr = kmap_atomic(to, KM_IRQ1); + + memcpy(tptr + offset, fptr + offset, size); + + kunmap_atomic(tptr, KM_IRQ1); + kunmap_atomic(fptr, KM_IRQ0); + local_irq_restore(flags); + set_page_dirty_lock(to); } /* * sdp_iocb_unlock - unlock the memory for an IOCB + * Copy if pages moved since. + * TODO: is this needed? */ int sdp_iocb_unlock(struct sdpc_iocb *iocb) { int result; + struct page ** pages = NULL; + unsigned long uaddr; + int i; + - /* - * check if IOCB is locked. - */ if (!(iocb->flags & SDP_IOCB_F_LOCKED)) return 0; - /* - * spin lock since this could be from interrupt context. - */ - down_write(&iocb->mm->mmap_sem); - - result = do_iocb_unlock(iocb); - up_write(&iocb->mm->mmap_sem); + /* For read, unlock and we are done */ + if (!(iocb->flags & SDP_IOCB_F_RECV)) { + for (i = 0;i < iocb->page_count; ++i) + put_page(iocb->page_array[i]); + goto done; + } - kfree(iocb->page_array); - kfree(iocb->addr_array); + /* For write, we must check the virtual pages did not get remapped */ - iocb->page_array = NULL; - iocb->addr_array = NULL; - iocb->mm = NULL; - /* - * mark IOCB unlocked. - */ - iocb->flags &= ~SDP_IOCB_F_LOCKED; + /* As an optimisation (to avoid scanning the vma tree each time), + * try to get all pages in one go. */ + /* TODO: use cache for allocations? Allocate by chunks? */ - return result; -} + pages = kmalloc((sizeof(struct page *) * iocb->page_count), GFP_KERNEL); -/* - * sdp_iocb_page_save - save page information for an IOCB - */ -static int sdp_iocb_page_save(struct sdpc_iocb *iocb) -{ - unsigned int counter; - unsigned long addr; - size_t size; - int result = -ENOMEM; - struct page *page; - unsigned long pfn; - pgd_t *pgd; - pud_t *pud; - pmd_t *pmd; - pte_t *ptep; - pte_t pte; + down_read(&iocb->mm->mmap_sem); - if (iocb->page_count <= 0 || iocb->size <= 0 || !iocb->addr) - return -EINVAL; - /* - * create array to hold page value which are later needed to register - * the buffer with the HCA - */ - iocb->addr_array = kmalloc((sizeof(u64) * iocb->page_count), - GFP_KERNEL); - if (!iocb->addr_array) - goto err_addr; + if (pages) { + result = get_user_pages(iocb->tsk, iocb->mm, iocb->addr, + iocb->page_count, 1, 0, pages, NULL); - iocb->page_array = kmalloc((sizeof(struct page *) * iocb->page_count), - GFP_KERNEL); - if (!iocb->page_array) - goto err_page; - /* - * iocb->addr - buffer start address - * iocb->size - buffer length - * addr - page aligned - * size - page multiple - */ - addr = iocb->addr & PAGE_MASK; - size = PAGE_ALIGN(iocb->size + (iocb->addr & ~PAGE_MASK)); + if (result != iocb->page_count) { + kfree(pages); + pages = NULL; + } + } - iocb->page_offset = iocb->addr - addr; - /* - * Find pages used within the buffer which will then be registered - * for RDMA - */ - spin_lock(&iocb->mm->page_table_lock); + for (i = 0, uaddr = iocb->addr; i < iocb->page_count; + ++i, uaddr = (uaddr & PAGE_MASK) + PAGE_SIZE) + { + struct page* page; + set_page_dirty_lock(iocb->page_array[i]); + + if (pages) + page = pages[i]; + else { + result = get_user_pages(iocb->tsk, iocb->mm, + uaddr & PAGE_MASK, + 1 , 1, 0, &page, NULL); + if (result != 1) { + page = NULL; + } + } - for (counter = 0; - size > 0; - counter++, addr += PAGE_SIZE, size -= PAGE_SIZE) { - pgd = pgd_offset_gate(iocb->mm, addr); - if (!pgd || pgd_none(*pgd)) - break; - - pud = pud_offset(pgd, addr); - if (!pud || pud_none(*pud)) - break; - - pmd = pmd_offset(pud, addr); - if (!pmd || pmd_none(*pmd)) - break; - - ptep = pte_offset_map(pmd, addr); - if (!ptep) - break; - - pte = *ptep; - pte_unmap(ptep); - - if (!pte_present(pte)) - break; - - pfn = pte_pfn(pte); - if (!pfn_valid(pfn)) - break; - - page = pfn_to_page(pfn); + if (page && iocb->page_array[i] != page) + sdp_copy_one_page(iocb->page_array[i], page, + iocb->addr, iocb->size, uaddr); - iocb->page_array[counter] = page; - iocb->addr_array[counter] = page_to_phys(page); + if (page) + put_page(page); + put_page(iocb->page_array[i]); } - spin_unlock(&iocb->mm->page_table_lock); - - if (size > 0) { - result = -EFAULT; - goto err_find; - } + up_read(&iocb->mm->mmap_sem); - return 0; -err_find: - - kfree(iocb->page_array); - iocb->page_array = NULL; -err_page: + if (pages) + kfree(pages); +done: + kfree(iocb->page_array); kfree(iocb->addr_array); + + iocb->page_array = NULL; iocb->addr_array = NULL; -err_addr: + iocb->mm = NULL; + iocb->tsk = NULL; - return result; + iocb->flags &= ~SDP_IOCB_F_LOCKED; + + return 0; } /* * sdp_iocb_lock - lock the memory for an IOCB + * We do not take a reference on the mm, AIO handles this for us. */ int sdp_iocb_lock(struct sdpc_iocb *iocb) { - struct vm_area_struct *vma; - kernel_cap_t real_cap; - unsigned long limit; int result = -ENOMEM; unsigned long addr; - size_t size; - - /* - * mark IOCB as locked. We do not take a reference on the mm, AIO - * handles this for us. - */ - iocb->flags |= SDP_IOCB_F_LOCKED; - iocb->mm = current->mm; + size_t size; + int i; /* - * save and raise capabilities + * iocb->addr - buffer start address + * iocb->size - buffer length + * addr - page aligned + * size - page multiple */ - real_cap = cap_t(current->cap_effective); - cap_raise(current->cap_effective, CAP_IPC_LOCK); - - size = PAGE_ALIGN(iocb->size + (iocb->addr & ~PAGE_MASK)); addr = iocb->addr & PAGE_MASK; + size = PAGE_ALIGN(iocb->size + (iocb->addr & ~PAGE_MASK)); + iocb->page_offset = iocb->addr - addr; + iocb->page_count = size >> PAGE_SHIFT; - - limit = current->signal->rlim[RLIMIT_MEMLOCK].rlim_cur; - limit >>= PAGE_SHIFT; - /* - * lock the mm, if within the limit lock the address range. - */ - down_write(&iocb->mm->mmap_sem); - - if (!((iocb->page_count + current->mm->locked_vm) > limit)) - result = (*mlock_ptr)(addr, size, 1); /* - * process result - */ - if (result) { - sdp_dbg_err("VMA lock <%lx:%Zu> error <%d> <%d:%lu:%lu>", - iocb->addr, iocb->size, result, - iocb->page_count, iocb->mm->locked_vm, limit); - goto err_lock; - } - /* - * look up the head of the vma queue, loop through the vmas, marking - * them do not copy, reference counting, and saving them. + * create array to hold page value which are later needed to register + * the buffer with the HCA */ - vma = find_vma(iocb->mm, addr); - if (!vma) - /* - * sanity check. - */ - sdp_warn("No VMA for IOCB! <%lx:%Zu> lock", - iocb->addr, iocb->size); - while (vma) { - spin_lock(&iocb->mm->page_table_lock); + /* TODO: use cache for allocations? Allocate by chunks? */ + iocb->addr_array = kmalloc((sizeof(u64) * iocb->page_count), + GFP_KERNEL); + if (!iocb->addr_array) + goto err_addr; - if (!(VM_LOCKED & vma->vm_flags)) - sdp_warn("Unlocked vma! <%08lx>", vma->vm_flags); + iocb->page_array = kmalloc((sizeof(struct page *) * iocb->page_count), + GFP_KERNEL); + if (!iocb->page_array) + goto err_page; - if (PAGE_SIZE < (unsigned long)vma->vm_private_data) - sdp_dbg_err("VMA: private daya in use! <%08lx>", - (unsigned long)vma->vm_private_data); - - vma->vm_flags |= VM_DONTCOPY; - vma->vm_private_data++; + down_read(¤t->mm->mmap_sem); - spin_unlock(&iocb->mm->page_table_lock); + result = get_user_pages(current, current->mm, + iocb->addr, iocb->page_count, + !!(iocb->flags & SDP_IOCB_F_RECV), 0, + iocb->page_array, NULL); + + up_read(¤t->mm->mmap_sem); + + if (result != iocb->page_count) { + sdp_dbg_err("unable to lock <%lx:%Zu> error <%d> <%d>", + iocb->addr, iocb->size, result, iocb->page_count); + goto err_get; + } - sdp_dbg_data(NULL, - "mark <%lx> <0x%p> <%08lx:%08lx> <%08lx> <%ld>", - iocb->addr, vma, vma->vm_start, vma->vm_end, - vma->vm_flags, (long)vma->vm_private_data); + iocb->flags |= SDP_IOCB_F_LOCKED; + iocb->mm = current->mm; + iocb->tsk = current; - if ((addr + size) > vma->vm_end) - vma = vma->vm_next; - else - vma = NULL; - } - result = sdp_iocb_page_save(iocb); - if (result) { - sdp_dbg_err("Error <%d> saving pages for IOCB <%lx:%Zu>", - result, iocb->addr, iocb->size); - goto err_save; + for (i = 0; i< iocb->page_count; ++i) { + iocb->addr_array[i] = page_to_phys(iocb->page_array[i]); } - up_write(&iocb->mm->mmap_sem); - cap_t(current->cap_effective) = real_cap; - return 0; -err_save: - - (void)do_iocb_unlock(iocb); -err_lock: - /* - * unlock the mm and restore capabilities. - */ - up_write(&iocb->mm->mmap_sem); - cap_t(current->cap_effective) = real_cap; - - iocb->flags &= ~SDP_IOCB_F_LOCKED; - iocb->mm = NULL; +err_get: + kfree(iocb->page_array); +err_page: + kfree(iocb->addr_array); +err_addr: return result; } /* - * IOCB memory locking init functions - */ -struct kallsym_iter { - loff_t pos; - struct module *owner; - unsigned long value; - unsigned int nameoff; /* If iterating in core kernel symbols */ - char type; - char name[128]; -}; - -/* - * sdp_mem_lock_init - initialize the userspace memory locking - */ -static int sdp_mem_lock_init(void) -{ - struct file *kallsyms; - struct seq_file *seq; - struct kallsym_iter *iter; - loff_t pos = 0; - int ret = -EINVAL; - - sdp_dbg_init("Memory Locking initialization."); - - kallsyms = filp_open("/proc/kallsyms", O_RDONLY, 0); - if (!kallsyms) { - sdp_warn("Failed to open /proc/kallsyms"); - goto done; - } - - seq = (struct seq_file *)kallsyms->private_data; - if (!seq) { - sdp_warn("Failed to fetch sequential file."); - goto err_close; - } - - for (iter = seq->op->start(seq, &pos); - iter != NULL; - iter = seq->op->next(seq, iter, &pos)) - if (!strcmp(iter->name, "do_mlock")) - mlock_ptr = (do_mlock_ptr_t)iter->value; - - if (!mlock_ptr) - sdp_warn("Failed to find lock pointer."); - else - ret = 0; - -err_close: - filp_close(kallsyms, NULL); -done: - return ret; -} - -/* - * sdp_mem_lock_cleanup - cleanup the memory locking tables - */ -static int sdp_mem_lock_cleanup(void) -{ - sdp_dbg_init("Memory Locking cleanup."); - /* - * null out entries. - */ - mlock_ptr = NULL; - - return 0; -} - -/* * IOCB memory registration functions */ @@ -831,28 +639,12 @@ void sdp_iocb_q_clear(struct sdpc_iocb_q } /* - * primary initialization/cleanup functions - */ - -/* * sdp_main_iocb_init - initialize the advertisment caches */ int sdp_main_iocb_init(void) { - int result; - sdp_dbg_init("IOCB cache initialization."); - /* - * initialize locking code. - */ - result = sdp_mem_lock_init(); - if (result < 0) { - sdp_warn("Error <%d> initializing memory locking.", result); - return result; - } - /* - * initialize the caches only once. - */ + if (sdp_iocb_cache) { sdp_warn("IOCB caches already initialized."); return -EINVAL; @@ -862,15 +654,10 @@ int sdp_main_iocb_init(void) sizeof(struct sdpc_iocb), 0, SLAB_HWCACHE_ALIGN, NULL, NULL); - if (!sdp_iocb_cache) { - result = -ENOMEM; - goto error_iocb_c; - } + if (!sdp_iocb_cache) + return -ENOMEM; return 0; -error_iocb_c: - (void)sdp_mem_lock_cleanup(); - return result; } /* @@ -879,16 +666,6 @@ error_iocb_c: void sdp_main_iocb_cleanup(void) { sdp_dbg_init("IOCB cache cleanup."); - /* - * cleanup the caches - */ kmem_cache_destroy(sdp_iocb_cache); - /* - * null out entries. - */ sdp_iocb_cache = NULL; - /* - * cleanup memory locking - */ - (void)sdp_mem_lock_cleanup(); } Index: ulp/sdp/sdp_iocb.h =================================================================== --- ulp/sdp/sdp_iocb.h (revision 2292) +++ ulp/sdp/sdp_iocb.h (working copy) @@ -52,7 +52,8 @@ #define SDP_IOCB_F_RDMA_R 0x00000010 /* IOCB is in RDMA read processing */ #define SDP_IOCB_F_RDMA_W 0x00000020 /* IOCB is in RDMA write processing */ #define SDP_IOCB_F_LOCKED 0x00000040 /* IOCB is locked in memory */ -#define SDP_IOCB_F_REG 0x00000080 /* IOCB is memory is registered */ +#define SDP_IOCB_F_REG 0x00000080 /* IOCB memory is registered */ +#define SDP_IOCB_F_RECV 0x00000100 /* IOCB is for a receive request */ #define SDP_IOCB_F_ALL 0xFFFFFFFF /* IOCB all mask */ /* * zcopy constants. @@ -99,9 +100,10 @@ struct sdpc_iocb { /* * page list. data for locking/registering userspace */ - struct mm_struct *mm; /* user mm struct */ - unsigned long addr; /* user space address */ - size_t size; /* total size of the user buffer */ + struct mm_struct *mm; /* user mm struct */ + struct task_struct *tsk; + unsigned long addr; /* user space address */ + size_t size; /* total size of the user buffer */ struct page **page_array; /* list of page structure pointers. */ u64 *addr_array; /* list of physical page addresses. */ -- MST - Michael S. Tsirkin From halr at voltaire.com Tue May 10 07:45:01 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 10 May 2005 10:45:01 -0400 Subject: [openib-general] [PATCH] kdapl: fix private data copying on incoming REQ Message-ID: <1115736301.4493.814.camel@localhost.localdomain> Fix private data copying on incoming REQ Also, fix a typo Signed-off-by: Hal Rosenstock Index: dapl_cr_callback.c =================================================================== --- dapl_cr_callback.c (revision 2292) +++ dapl_cr_callback.c (working copy) @@ -124,7 +124,7 @@ case DAT_CONNECTION_REQUEST_EVENT: { /* - * Requests arriving on a disabled SP are immediatly rejected + * Requests arriving on a disabled SP are immediately rejected */ spin_lock(&sp_ptr->header.lock); @@ -298,10 +298,10 @@ if (prd_ptr == NULL) { cr_ptr->param.private_data_size = 0; } else { - cr_ptr->param.private_data_size = IB_MAX_REP_PDATA_SIZE; - memcpy(cr_ptr->private_data, - prd_ptr, - IB_MAX_REP_PDATA_SIZE); + cr_ptr->param.private_data_size = IB_MAX_REQ_PDATA_SIZE; + memcpy(cr_ptr->private_data, + prd_ptr, + IB_MAX_REQ_PDATA_SIZE); } /* EP will be NULL unless RSP service point */ From sexton at mc.com Tue May 10 08:17:59 2005 From: sexton at mc.com (Sexton, Matt) Date: Tue, 10 May 2005 11:17:59 -0400 Subject: [openib-general] openib/MPI + bigphysarea data corruption problem Message-ID: <927DD5EB74B7114EAF78D7D40DDE10F003725E@ad-email1.ad.mc.com> Hi, We are attempting to share memory between a PCI IO device and MPI running over IB. We want the IO device to write to a memory buffer, and then have MPI transfer the data from the buffer out over IB (with no copies), but we are having data corruption problems with what is received on the other end of the IB link. The IO device requires hundreds of MBs of physically contiguous memory, so we applied the bigphysarea patch to 2.6. The bigphysarea patch allocates a large block of memory at boot time and then has permanent possession of it (so no worries about it being swapped out). We wrote a small kernel module to map the bigphysarea physical memory into user space via mmap. The user space process can work with the buffer just fine. But when we pass this buffer to MPI_Send(), data is not reliably received at the destination. In some cases data arrives correctly the first N times, but then incorrect data is received. On other cases (differing in buffer size, system configuration, etc.), the correct data is never received. If we used malloced or shared memory, the MPI data transmission works fine. We've been looking into how openib does the mapping, pinning, and freeing of the user space buffer, but we're getting a little lost between get_user_pages(), put_page(), and how it all relates to the page cache. Can someone please comment on whether there's someway for openib to accept this unusual memory? Thanks, Matt From iod00d at hp.com Tue May 10 08:24:28 2005 From: iod00d at hp.com (Grant Grundler) Date: Tue, 10 May 2005 08:24:28 -0700 Subject: [openib-general] Re: user-mode verbs on Itanium In-Reply-To: <20050508142345.GQ9941@mellanox.co.il> References: <20050429051009.GT20957@esmail.cup.hp.com> <52pswdk5au.fsf@topspin.com> <20050429162127.GA24871@esmail.cup.hp.com> <52ll71k09w.fsf@topspin.com> <20050429183846.GH24871@esmail.cup.hp.com> <20050504005534.GA13574@esmail.cup.hp.com> <20050504070910.GC9674@mellanox.co.il> <20050506190947.GA23958@esmail.cup.hp.com> <20050507053706.GN23958@esmail.cup.hp.com> <20050508142345.GQ9941@mellanox.co.il> Message-ID: <20050510152428.GA5495@esmail.cup.hp.com> On Sun, May 08, 2005 at 05:23:45PM +0300, Michael S. Tsirkin wrote: > Your patch initialized tstamp[0] to get_clock which still isnt right > since its take before post_send and not after that. Yes, but I think I want the first measurement *before* post_send since it seems to measure something different that's part of the code path too. I suspect it's measuring the first "TLB" miss of HCA. I found it very interesting that the first send (or two) was very expensive. grant From mst at mellanox.co.il Tue May 10 08:35:43 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Tue, 10 May 2005 18:35:43 +0300 Subject: [openib-general] [PATCH 1/2] (fixed) mthca: max_inline_data support In-Reply-To: <52fyww1h90.fsf@topspin.com> References: <20050509141456.GH9941@mellanox.co.il> <52fyww1h90.fsf@topspin.com> Message-ID: <20050510153543.GG2474@mellanox.co.il> Here's an updated patch - I'll be offline till Sunday and I hope it's good to go. Please note there's a companion patch for libmthca, ([PATCH 2/2]) which is also needed to actually use this stuff. Quoting r. Roland Dreier : > Subject: Re: [openib-general] [PATCH 1/2] mthca: max_inline_data support > > Thanks... does it make sense to set the max_inline_data value for > kernel QPs but then not support posting inline data? You are right: max_inline_data in init_attr shall be left 0 until posting inline data in kernel is supported. Fixed. > Also should we add some error checking against the HCA's maximum WQE > size? I added such a check. Support max_inline_data parameter for userspace QPs. Check max_send_sge/max_recv_sge/max_inline_data against the HCA's limits. Signed-off-by: Michael S. Tsirkin Index: hw/mthca/mthca_dev.h =================================================================== --- hw/mthca/mthca_dev.h (revision 2292) +++ hw/mthca/mthca_dev.h (working copy) @@ -439,12 +439,14 @@ int mthca_alloc_qp(struct mthca_dev *dev struct mthca_cq *recv_cq, enum ib_qp_type type, enum ib_sig_type send_policy, + struct ib_qp_cap *cap, struct mthca_qp *qp); int mthca_alloc_sqp(struct mthca_dev *dev, struct mthca_pd *pd, struct mthca_cq *send_cq, struct mthca_cq *recv_cq, enum ib_sig_type send_policy, + struct ib_qp_cap *cap, int qpn, int port, struct mthca_sqp *sqp); Index: hw/mthca/mthca_provider.c =================================================================== --- hw/mthca/mthca_provider.c (revision 2292) +++ hw/mthca/mthca_provider.c (working copy) @@ -486,16 +486,11 @@ static struct ib_qp *mthca_create_qp(str qp->rq.db_index = ucmd.rq_db_index; } - qp->sq.max = init_attr->cap.max_send_wr; - qp->rq.max = init_attr->cap.max_recv_wr; - qp->sq.max_gs = init_attr->cap.max_send_sge; - qp->rq.max_gs = init_attr->cap.max_recv_sge; - err = mthca_alloc_qp(to_mdev(pd->device), to_mpd(pd), to_mcq(init_attr->send_cq), to_mcq(init_attr->recv_cq), init_attr->qp_type, init_attr->sq_sig_type, - qp); + &init_attr->cap, qp); if (err && pd->uobject) { context = to_mucontext(pd->uobject->context); @@ -524,17 +519,12 @@ static struct ib_qp *mthca_create_qp(str if (!qp) return ERR_PTR(-ENOMEM); - qp->sq.max = init_attr->cap.max_send_wr; - qp->rq.max = init_attr->cap.max_recv_wr; - qp->sq.max_gs = init_attr->cap.max_send_sge; - qp->rq.max_gs = init_attr->cap.max_recv_sge; - qp->ibqp.qp_num = init_attr->qp_type == IB_QPT_SMI ? 0 : 1; err = mthca_alloc_sqp(to_mdev(pd->device), to_mpd(pd), to_mcq(init_attr->send_cq), to_mcq(init_attr->recv_cq), - init_attr->sq_sig_type, + init_attr->sq_sig_type, &init_attr->cap, qp->ibqp.qp_num, init_attr->port_num, to_msqp(qp)); break; Index: hw/mthca/mthca_qp.c =================================================================== --- hw/mthca/mthca_qp.c (revision 2292) +++ hw/mthca/mthca_qp.c (working copy) @@ -46,7 +46,9 @@ enum { MTHCA_MAX_DIRECT_QP_SIZE = 4 * PAGE_SIZE, MTHCA_ACK_REQ_FREQ = 10, MTHCA_FLIGHT_LIMIT = 9, - MTHCA_UD_HEADER_SIZE = 72 /* largest UD header possible */ + MTHCA_UD_HEADER_SIZE = 72, /* largest UD header possible */ + MTHCA_INLINE_HEADER_SIZE = 4, /* data segment overhead for inline */ + MTHCA_INLINE_CHUNK_SIZE = 16 /* inline data segment chunk */ }; enum { @@ -1205,22 +1207,41 @@ static int mthca_alloc_qp_common(struct return 0; } -static void mthca_align_qp_size(struct mthca_dev *dev, struct mthca_qp *qp) +static int mthca_set_qp_size(struct mthca_dev *dev, struct ib_qp_cap* cap, + struct mthca_qp *qp) { int i; + u32 d; - if (!mthca_is_memfree(dev)) - return; + if (mthca_is_memfree(dev)) { + for (i = 0; 1 << i < cap->max_recv_wr; ++i) + ; /* nothing */ - for (i = 0; 1 << i < qp->rq.max; ++i) - ; /* nothing */ + qp->rq.max = 1 << i; - qp->rq.max = 1 << i; + for (i = 0; 1 << i < cap->max_send_wr; ++i) + ; /* nothing */ - for (i = 0; 1 << i < qp->sq.max; ++i) - ; /* nothing */ + qp->sq.max = 1 << i; + } else { + qp->rq.max = cap->max_recv_wr; + qp->sq.max = cap->max_send_wr; + } + + qp->rq.max_gs = cap->max_recv_sge; - qp->sq.max = 1 << i; + d = ALIGN(cap->max_inline_data + MTHCA_INLINE_HEADER_SIZE, + MTHCA_INLINE_CHUNK_SIZE) / sizeof (struct mthca_data_seg); + qp->sq.max_gs = max(cap->max_send_sge, d); + + if (qp->transport == MLX) { + /* For MLX transport we need 2 extra S/G entries: + * for header and checksum */ + qp->sq.max_gs += 2; + } + + return (qp->sq.max_gs > dev->limits.max_sg || + qp->rq.max_gs > dev->limits.max_sg) ? -EINVAL : 0; } int mthca_alloc_qp(struct mthca_dev *dev, @@ -1229,11 +1250,14 @@ int mthca_alloc_qp(struct mthca_dev *dev struct mthca_cq *recv_cq, enum ib_qp_type type, enum ib_sig_type send_policy, + struct ib_qp_cap *cap, struct mthca_qp *qp) { int err; - mthca_align_qp_size(dev, qp); + err = mthca_set_qp_size(dev, cap, qp); + if (err) + return err; switch (type) { case IB_QPT_RC: qp->transport = RC; break; @@ -1266,14 +1290,17 @@ int mthca_alloc_sqp(struct mthca_dev *de struct mthca_cq *send_cq, struct mthca_cq *recv_cq, enum ib_sig_type send_policy, + struct ib_qp_cap *cap, int qpn, int port, struct mthca_sqp *sqp) { - int err = 0; u32 mqpn = qpn * 2 + dev->qp_table.sqp_start + port - 1; + int err; - mthca_align_qp_size(dev, &sqp->qp); + err = mthca_set_qp_size(dev, cap, &sqp->qp); + if (err) + return err; sqp->header_buf_size = sqp->qp.sq.max * MTHCA_UD_HEADER_SIZE; sqp->header_buf = dma_alloc_coherent(&dev->pdev->dev, sqp->header_buf_size, From mst at mellanox.co.il Tue May 10 08:40:30 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Tue, 10 May 2005 18:40:30 +0300 Subject: [openib-general] Re: user-mode verbs on Itanium In-Reply-To: <20050510152428.GA5495@esmail.cup.hp.com> References: <52pswdk5au.fsf@topspin.com> <20050429162127.GA24871@esmail.cup.hp.com> <52ll71k09w.fsf@topspin.com> <20050429183846.GH24871@esmail.cup.hp.com> <20050504005534.GA13574@esmail.cup.hp.com> <20050504070910.GC9674@mellanox.co.il> <20050506190947.GA23958@esmail.cup.hp.com> <20050507053706.GN23958@esmail.cup.hp.com> <20050508142345.GQ9941@mellanox.co.il> <20050510152428.GA5495@esmail.cup.hp.com> Message-ID: <20050510154030.GH2474@mellanox.co.il> Quoting r. Grant Grundler : > Subject: Re: user-mode verbs on Itanium > > On Sun, May 08, 2005 at 05:23:45PM +0300, Michael S. Tsirkin wrote: > > Your patch initialized tstamp[0] to get_clock which still isnt right > > since its take before post_send and not after that. > > Yes, but I think I want the first measurement *before* post_send > since it seems to measure something different that's part of the > code path too. I'd prefer all deltas to measure the same thing. > I suspect it's measuring the first "TLB" miss of HCA. I dont think so. > I found it very interesting that the first send (or two) > was very expensive. > > grant > You still see this cache warm-up with the new code. -- MST - Michael S. Tsirkin From trimmer at infiniconsys.com Tue May 10 08:49:07 2005 From: trimmer at infiniconsys.com (Rimmer, Todd) Date: Tue, 10 May 2005 11:49:07 -0400 Subject: [openib-general] [PATCH 1/2] (fixed) mthca: max_inline_data support Message-ID: <5D78D28F88822E4D8702BB9EEF1A43670A067D@mercury.infiniconsys.com> > Here's an updated patch - I'll be offline till Sunday and I hope > it's good to go. Please note there's a companion patch for libmthca, > ([PATCH 2/2]) which is also needed to actually use this stuff. > > Quoting r. Roland Dreier : > > Subject: Re: [openib-general] [PATCH 1/2] mthca: > max_inline_data support > > > > Thanks... does it make sense to set the max_inline_data value for > > kernel QPs but then not support posting inline data? > > You are right: max_inline_data in init_attr shall be left 0 > until posting inline data in kernel is supported. Fixed. > > > Also should we add some error checking against the HCA's maximum WQE > > size? > > I added such a check. > > > Support max_inline_data parameter for userspace QPs. > Check max_send_sge/max_recv_sge/max_inline_data against the HCA's > limits. > Regarding inline data in posted wqes, this decision can be automatically made for user QPs and for some kernel QPs. Hence the API need not be exposed in user space. If all MRs in the Protection domain have CPU Virtual == IO Virtual, then the post_wqe routine can automatically inline the data as appropriate. Given this situation, the IO Virtual address in the work request scatter gather list can be used as a direct CPU address to copy data into the WQE. For user space QPs, CPU Virtual should always == IO Virtual. This is because: register mr and register shared mr should use the CPU virtual as the IO Virtual. In the kernel, a flag can be kept per PD indicating if CPU Virtual == IO Virtual. Only for PDs using physical memory regions, would this condition be false. This approach relieves applications from dealing with the inline decisions, and it can permit the verbs driver to inline data in some additional cases (such as when there are fewer scatter gather entries in a WQE, hence making more room for inline data). Todd Rimmer From roland at topspin.com Tue May 10 09:08:05 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 10 May 2005 09:08:05 -0700 Subject: [openib-general] openib/MPI + bigphysarea data corruption problem In-Reply-To: <927DD5EB74B7114EAF78D7D40DDE10F003725E@ad-email1.ad.mc.com> (Matt Sexton's message of "Tue, 10 May 2005 11:17:59 -0400") References: <927DD5EB74B7114EAF78D7D40DDE10F003725E@ad-email1.ad.mc.com> Message-ID: <52vf5rxdwa.fsf@topspin.com> Matt> We've been looking into how openib does the mapping, Matt> pinning, and freeing of the user space buffer, but we're Matt> getting a little lost between get_user_pages(), put_page(), Matt> and how it all relates to the page cache. Matt> Can someone please comment on whether there's someway for Matt> openib to accept this unusual memory? The userspace verbs code calls get_user_pages() to get a list of page structs for the user memory being registered. I'm not familiar with how the bigphysarea patch works, so I'm not sure if get_user_pages() will work on memory allocated this way. By the way, what MPI library are you using? I was not aware that an MPI binding for the openib userspace verbs was available yet. - R. From mshefty at ichips.intel.com Tue May 10 09:27:14 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Tue, 10 May 2005 09:27:14 -0700 Subject: [openib-general] Re: CM ID destruction callback for kernel CM ? In-Reply-To: <1115698117.4490.1578.camel@localhost.localdomain> References: <1115698117.4490.1578.camel@localhost.localdomain> Message-ID: <4280E0E2.2040001@ichips.intel.com> Hal Rosenstock wrote: > Hi Sean, > > The kernel CM is currently capable of destroying the CM ID at the end of > the callback by setting ret to other than 0: > * Users may not call ib_destroy_cm_id while in the context of this callback; > * however, returning a non-zero value instructs the communication manager to > * destroy the @cm_id after the callback completes. > > Perhaps also a CM ID destruction callback could be added to: > struct ib_cm_id *ib_create_cm_id(ib_cm_handler cm_handler, > void *context); > which would also be called (if non NULL) on CM ID destruction at the end > of the CM callback user invoked CM ID destruction. In that way, a client > could cause some additional related destruction immediately subsequent > to the CM ID destruction on its behalf. That might be useful to CM > clients to simplify them. I don't think this is needed. The CM will serialize all callbacks relative to a single cm_id, and by returning a non-zero value, no additional callbacks will be generated. The user should be able to cleanup all resources before returning from the CM callback. - Sean From mst at mellanox.co.il Tue May 10 09:31:16 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Tue, 10 May 2005 19:31:16 +0300 Subject: [openib-general] [PATCH 1/2] (fixed) mthca: max_inline_data support In-Reply-To: <5D78D28F88822E4D8702BB9EEF1A43670A067D@mercury.infiniconsys.com> References: <5D78D28F88822E4D8702BB9EEF1A43670A067D@mercury.infiniconsys.com> Message-ID: <20050510163116.GA3572@mellanox.co.il> Quoting r. Rimmer, Todd : > Subject: RE: [openib-general] [PATCH 1/2] (fixed) mthca: max_inline_data support > > > Here's an updated patch - I'll be offline till Sunday and I hope > > it's good to go. Please note there's a companion patch for libmthca, > > ([PATCH 2/2]) which is also needed to actually use this stuff. > > > > Quoting r. Roland Dreier : > > > Subject: Re: [openib-general] [PATCH 1/2] mthca: > > max_inline_data support > > > > > > Thanks... does it make sense to set the max_inline_data value for > > > kernel QPs but then not support posting inline data? > > > > You are right: max_inline_data in init_attr shall be left 0 > > until posting inline data in kernel is supported. Fixed. > > > > > Also should we add some error checking against the HCA's maximum WQE > > > size? > > > > I added such a check. > > > > > > Support max_inline_data parameter for userspace QPs. > > Check max_send_sge/max_recv_sge/max_inline_data against the HCA's > > limits. > > > > Regarding inline data in posted wqes, this decision can be automatically made for user QPs and for some kernel QPs. Hence the API need not be exposed in user space. > > If all MRs in the Protection domain have CPU Virtual == IO Virtual, then the post_wqe routine can automatically inline the data as appropriate. Given this situation, the IO Virtual address in the work request scatter gather list can be used as a direct CPU address to copy data into the WQE. > > For user space QPs, CPU Virtual should always == IO Virtual. This is because: > register mr and register shared mr should use the CPU virtual as the IO Virtual. > > In the kernel, a flag can be kept per PD indicating if CPU Virtual == IO Virtual. Only for PDs using physical memory regions, would this condition be false. > > This approach relieves applications from dealing with the inline decisions, and it can permit the verbs driver to inline data in some additional cases (such as when there are fewer scatter gather entries in a WQE, hence making more room for inline data). > > Todd Rimmer > There are 3 reasons this is not done this way, in order of relevance: 1. posting inline data is trading off higher CPU utilixation for lower latency, may be bad for some applications 2. inline data need not be registered, application can make use of this fact 3. driver would need to know whether the data will fit inline ahead of the time, which means an extra scan of the scatter/gather list, application typically knows the total size since it builds the list -- MST - Michael S. Tsirkin From roland at topspin.com Tue May 10 09:34:06 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 10 May 2005 09:34:06 -0700 Subject: [openib-general] Re: [PATCH 1/2] (fixed) mthca: max_inline_data support In-Reply-To: <20050510153543.GG2474@mellanox.co.il> (Michael S. Tsirkin's message of "Tue, 10 May 2005 18:35:43 +0300") References: <20050509141456.GH9941@mellanox.co.il> <52fyww1h90.fsf@topspin.com> <20050510153543.GG2474@mellanox.co.il> Message-ID: <52is1rxcox.fsf@topspin.com> Thanks, I'll merge this stuff up today. - R. From roland at topspin.com Tue May 10 09:33:07 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 10 May 2005 09:33:07 -0700 Subject: [openib-general] [PATCH 1/2] (fixed) mthca: max_inline_data support In-Reply-To: <5D78D28F88822E4D8702BB9EEF1A43670A067D@mercury.infiniconsys.com> (Todd Rimmer's message of "Tue, 10 May 2005 11:49:07 -0400") References: <5D78D28F88822E4D8702BB9EEF1A43670A067D@mercury.infiniconsys.com> Message-ID: <52mzr3xcqk.fsf@topspin.com> Todd> Regarding inline data in posted wqes, this decision can be Todd> automatically made for user QPs and for some kernel QPs. Todd> Hence the API need not be exposed in user space. Todd> If all MRs in the Protection domain have CPU Virtual == IO Todd> Virtual, then the post_wqe routine can automatically inline Todd> the data as appropriate. Given this situation, the IO Todd> Virtual address in the work request scatter gather list can Todd> be used as a direct CPU address to copy data into the WQE. I don't think this sort of silent semantic change is a good idea at all. For example, automatically inlining data will fail in a very subtle way if an application relies on ordering between work requests on a send queue to post an RDMA read followed by a send of the same buffer. Inlining should only happen if the verbs consumer asks for it. - R. From robert.j.woodruff at intel.com Tue May 10 10:11:36 2005 From: robert.j.woodruff at intel.com (Woodruff, Robert J) Date: Tue, 10 May 2005 10:11:36 -0700 Subject: [openib-general] HCA Firmware Message-ID: <1AC79F16F5C5284499BB9591B33D6F00045BCC37@orsmsx408> Shirley Ma wrote, >> I don't recall if there are issues running with 3.2.0 or not. >3.2.0 has no problem. Even 3.1.0 works fine for me. FYI - I found that on Itainium, the user-mode verbs support does not appear to work unless you are running 3.3.2. With the older firmware 3.2 and even 3.0, IPoIB seemed to work fine, but the user-verbs pingpong test would hang or return an error of retries exceeded with the older firmware. woody From jlentini at netapp.com Tue May 10 10:20:49 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 10 May 2005 13:20:49 -0400 (EDT) Subject: [openib-general] [KDAPL] module initialization order (was Re: [PATCH] kDAPL: remove typedef DAT_PROVIDER) In-Reply-To: References: <1115424198.19661.11.camel@duffman> <1115662195.9691.7.camel@duffman> Message-ID: After reading through the kernel sources, I believe that builtin module initialization functions are called by the do_initcalls function in init/main.c. I haven't conclusively determined how the initialization order is specified. From what I've gleamed from mailing list archives, the module initialization order was determined by link order. As a result the relative position of the obj-$(CONFIG_XXX) lines in Makefiles determined the order of module initialization. Often when I found discussion of this topic, it was in the context of changing this scheme to make the initialization dependencies explicit. However, it does not appear that this was ever done. One final piece of information that I came across is that the order of initialization functions in the System.map file is the order that they will be called in. Can anyone confirm any of the information above? james On Mon, 9 May 2005, James Lentini wrote: > > Tom, > > If you are interested, there is one more thing you could look > into/give me advice on: > > The dat module currently keeps track of its "state" (see the > DAT_REGISTRY_STATE enumeration). The registry uses this information to > detect the case when a provider or consumer calls a dat registry > function before the registry's init function (dat_init) has run. > > Do we need to protect against this in the kernel? > > This situation could occur in usersapce when the registry, providers, > and consumers could be shared libraries and the library initialization > functions were invoked in an arbitrary order. > > I suspect that we can remove the code, but I wanted to make sure. If > the dat registry, dat provider, and a consumer (e.g. NFS-RDMA) were > built statically as part of the kernel, would the initialization > functions be automatically run in the correct order? > > james > > On Mon, 9 May 2005, Tom Duffy wrote: > > tduffy> On Mon, 2005-05-09 at 14:06 -0400, James Lentini wrote: > tduffy> > Committed revision 2287. > tduffy> > > tduffy> > On Fri, 6 May 2005, Tom Duffy wrote: > tduffy> > > tduffy> > tduffy> James, thanks for applying my last patch. > tduffy> > tduffy> > tduffy> > tduffy> You know where I am going with this... this is the first in what will be > tduffy> > tduffy> a huge amount of patches. I am happy to go through and send them all to > tduffy> > tduffy> the list, but it might be quicker and easier for you to do it directly > tduffy> > tduffy> to your tree for all the structs and enums. Sup to you. > tduffy> > tduffy> > tduffy> > tduffy> So, did you want me to continue with sending these type of patches? > tduffy> > tduffy> -tduffy > tduffy> From roland at topspin.com Tue May 10 10:25:35 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 10 May 2005 10:25:35 -0700 Subject: [openib-general] [KDAPL] module initialization order In-Reply-To: (James Lentini's message of "Tue, 10 May 2005 13:20:49 -0400 (EDT)") References: <1115424198.19661.11.camel@duffman> <1115662195.9691.7.camel@duffman> Message-ID: <523bsvxab4.fsf@topspin.com> Yes, link order is the order that initialization functions will be called in. However (and I'm not sure if this really affects you) devices can be added to or removed from the system at any time because of PCI hotplug etc., which can cause device driver init/cleanup functions to be called at arbitrary times. - R. From roland at topspin.com Tue May 10 10:22:16 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 10 May 2005 10:22:16 -0700 Subject: [openib-general] HCA Firmware In-Reply-To: <1AC79F16F5C5284499BB9591B33D6F00045BCC37@orsmsx408> (Robert J. Woodruff's message of "Tue, 10 May 2005 10:11:36 -0700") References: <1AC79F16F5C5284499BB9591B33D6F00045BCC37@orsmsx408> Message-ID: <52acn3xagn.fsf@topspin.com> Just for completeness, for PCI-X HCAs: . 3.3.2 is the first firmware release with working MSI-X . No released versions of firmware will work on systems with 64K pages. (This is a possible config for ia64 kernels at least) - R. From itamar at mellanox.co.il Tue May 10 10:49:04 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Tue, 10 May 2005 20:49:04 +0300 Subject: [openib-general] [KDAPL] module initialization order (was Re: [PATCH] kDAPL: remove typedef DAT_PROVIDER) Message-ID: <91DB792C7985D411BEC300B40080D29CC35AB6@mtvex01.mtv.mtl.com> I still see problem when NFSoRDMA or any other dat consumer and dat will be build statically but the dat_provider (ib,iwarp,...) will be build as a module. who will protect the dat consumer ? how will the dat consumer know that there is no dat_provider available when dat_provider modules are not loaded? Itamar > -----Original Message----- > From: James Lentini [mailto:jlentini at netapp.com] > Sent: Tuesday, May 10, 2005 8:21 PM > To: openib-general at openib.org > Cc: Tom Duffy > Subject: [openib-general] [KDAPL] module initialization order (was Re: > [PATCH] kDAPL: remove typedef DAT_PROVIDER) > > > > After reading through the kernel sources, I believe that builtin > module initialization functions are called by the do_initcalls > function in init/main.c. > > I haven't conclusively determined how the initialization order is > specified. From what I've gleamed from mailing list archives, the > module initialization order was determined by link order. As a result > the relative position of the obj-$(CONFIG_XXX) lines in Makefiles > determined the order of module initialization. Often when I found > discussion of this topic, it was in the context of changing this > scheme to make the initialization dependencies explicit. However, it > does not appear that this was ever done. > > One final piece of information that I came across is that the > order of > initialization functions in the System.map file is the order > that they > will be called in. > > Can anyone confirm any of the information above? > > james > > On Mon, 9 May 2005, James Lentini wrote: > > > > > Tom, > > > > If you are interested, there is one more thing you could look > > into/give me advice on: > > > > The dat module currently keeps track of its "state" (see the > > DAT_REGISTRY_STATE enumeration). The registry uses this > information to > > detect the case when a provider or consumer calls a dat registry > > function before the registry's init function (dat_init) has run. > > > > Do we need to protect against this in the kernel? > > > > This situation could occur in usersapce when the registry, > providers, > > and consumers could be shared libraries and the library > initialization > > functions were invoked in an arbitrary order. > > > > I suspect that we can remove the code, but I wanted to make sure. If > > the dat registry, dat provider, and a consumer (e.g. NFS-RDMA) were > > built statically as part of the kernel, would the initialization > > functions be automatically run in the correct order? > > > > james > > > > On Mon, 9 May 2005, Tom Duffy wrote: > > > > tduffy> On Mon, 2005-05-09 at 14:06 -0400, James Lentini wrote: > > tduffy> > Committed revision 2287. > > tduffy> > > > tduffy> > On Fri, 6 May 2005, Tom Duffy wrote: > > tduffy> > > > tduffy> > tduffy> James, thanks for applying my last patch. > > tduffy> > tduffy> > > tduffy> > tduffy> You know where I am going with this... > this is the first in what will be > > tduffy> > tduffy> a huge amount of patches. I am happy to > go through and send them all to > > tduffy> > tduffy> the list, but it might be quicker and > easier for you to do it directly > > tduffy> > tduffy> to your tree for all the structs and > enums. Sup to you. > > tduffy> > tduffy> > > tduffy> > > tduffy> So, did you want me to continue with sending these > type of patches? > > tduffy> > > tduffy> -tduffy > > tduffy> > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From robert.j.woodruff at intel.com Tue May 10 10:31:48 2005 From: robert.j.woodruff at intel.com (Woodruff, Robert J) Date: Tue, 10 May 2005 10:31:48 -0700 Subject: [openib-general] [OOPS] user-mode verbs on IPF Message-ID: <1AC79F16F5C5284499BB9591B33D6F00045BCC98@orsmsx408> Hi Roland, I was running the ibv_pingpong test in a loop over night and got this oops. The system is an older Itanium system 700Mhz, with 512 Mbytes of memory. The test was running on one system, in loopback. The IB code is SVN2281. It is running on a RedHat 2.6.9-5ELsmp kernel with the IB code backported to that kernel. I do not think it is IPF specific as I saw a similar oops on my X86_64 box, but did not save the messages. log on that one. woody ay 9 13:07:14 iclust-tiger2 su(pam_unix)[18393]: session opened for user root by woody(uid=500) May 9 13:43:01 iclust-tiger2 kernel: Unable to handle kernel paging request at virtual address 0016001900400005 May 9 13:43:01 iclust-tiger2 kernel: lt-ibv_pingpong[6525]: Oops 8821862825984 [1] May 9 13:43:01 iclust-tiger2 kernel: Modules linked in: ib_uverbs(U) ib_sdp(U) ib_cm(U) ib_ipoib(U) ib_sa(U) nls_utf8(U) parport_pc(U) lp(U) parport(U) autofs4(U) sunrpc(U) ds(U) yenta_socket(U) pcmcia_core(U) vfat(U) fat(U) button(U) md5(U) ipv6(U) joydev(U) uhci_hcd(U) ib_mthca(U) ib_mad(U) ib_core(U) e1000(U) sg(U) dm_snapshot(U) dm_zero(U) dm_mirror(U) ext3(U) jbd(U) dm_mod(U) mptscsih(U) mptbase(U) sd_mod(U) scsi_mod(U) May 9 13:43:01 iclust-tiger2 kernel: May 9 13:43:01 iclust-tiger2 kernel: Pid: 6525, CPU 1, comm: lt-ibv_pingpong May 9 13:43:01 iclust-tiger2 kernel: psr : 0000101008126010 ifs : 8000000000000289 ip : [] Not tainted May 9 13:43:01 iclust-tiger2 kernel: ip is at kref_put+0x90/0x1e0 May 9 13:43:01 iclust-tiger2 kernel: unat: 0000000000000000 pfs : 000000000000048b rsc : 0000000000000003 May 9 13:43:01 iclust-tiger2 kernel: rnat: 0000000000000000 bsps: 0000000000000000 pr : 00000000056a9955 May 9 13:43:01 iclust-tiger2 kernel: ldrs: 0000000000000000 ccv : 0000000000000000 fpsr: 0009804c8a70433f May 9 13:43:01 iclust-tiger2 kernel: csd : 0000000000000000 ssd : 0000000000000000 May 9 13:43:01 iclust-tiger2 kernel: b0 : a0000002004791e0 b6 : a0000001005561d0 b7 : a00000010000ed10 May 9 13:43:01 iclust-tiger2 kernel: f6 : 0fff7fffffffff0000000 f7 : 0ffe9c000000000000000 May 9 13:43:01 iclust-tiger2 kernel: f8 : 1003e00000000503ac43c f9 : 1003e000002a5875ff812 May 9 13:43:01 iclust-tiger2 kernel: f10 : 1003e55d603b5087fec38 f11 : 1003e00000000000000d4 May 9 13:43:01 iclust-tiger2 kernel: r1 : a00000010096a750 r2 : a00000010076aae0 r3 : a00000010076f1f0 May 9 13:43:01 iclust-tiger2 kernel: r8 : a000000100779820 r9 : a00000010076f1e8 r10 : e00000002e6c6db0 May 9 13:43:01 iclust-tiger2 kernel: r11 : e00000002d837110 r12 : e00000001e03fe20 r13 : e00000001e038000 May 9 13:43:01 iclust-tiger2 kernel: r14 : a000000100624f60 r15 : e00000003c6bbfa8 r16 : e00000002e40ee48 May 9 13:43:01 iclust-tiger2 kernel: r17 : e00000001e038dd0 r18 : 0000000000000000 r19 : 0009804c8a70433f May 9 13:43:01 iclust-tiger2 kernel: r20 : 000000001e038000 r21 : 00000000ffffffff r22 : e00000001e038028 May 9 13:43:01 iclust-tiger2 kernel: r23 : 0000000000000000 r24 : 0000000000000000 r25 : 0000000000000000 May 9 13:43:01 iclust-tiger2 kernel: r26 : 0000000000000000 r27 : 0000000000000000 r28 : 00000000056a6695 May 9 13:43:01 iclust-tiger2 kernel: r29 : 0000000000000000 r30 : 0000000000000000 r31 : e00000003c6bbf80 May 9 13:43:01 iclust-tiger2 kernel: May 9 13:43:01 iclust-tiger2 kernel: Call Trace: May 9 13:43:01 iclust-tiger2 kernel: [] show_stack+0x80/0xa0 May 9 13:43:01 iclust-tiger2 kernel: sp=e00000001e03f9d0 bsp=e00000001e0392b8 May 9 13:43:01 iclust-tiger2 kernel: [] show_regs+0x890/0x8c0 May 9 13:43:02 iclust-tiger2 kernel: sp=e00000001e03fba0 bsp=e00000001e039270 May 9 13:43:02 iclust-tiger2 kernel: [] die+0x150/0x240 May 9 13:43:02 iclust-tiger2 kernel: sp=e00000001e03fbc0 bsp=e00000001e039230 May 9 13:43:02 iclust-tiger2 kernel: [] ia64_do_page_fault+0x9f0/0xba0 May 9 13:43:02 iclust-tiger2 kernel: sp=e00000001e03fbc0 bsp=e00000001e0391c0 May 9 13:43:02 iclust-tiger2 kernel: [] ia64_leave_kernel+0x0/0x260 May 9 13:43:02 iclust-tiger2 kernel: sp=e00000001e03fc50 bsp=e00000001e0391c0 May 9 13:43:02 iclust-tiger2 kernel: [] kref_put+0x90/0x1e0 May 9 13:43:02 iclust-tiger2 kernel: sp=e00000001e03fe20 bsp=e00000001e039178 May 9 13:43:02 iclust-tiger2 kernel: [] ib_uverbs_event_release+0x140/0x160 [ib_uverbs] May 9 13:43:02 iclust-tiger2 kernel: sp=e00000001e03fe20 bsp=e00000001e039130 May 9 13:43:02 iclust-tiger2 kernel: [] ib_uverbs_event_close+0x30/0x60 [ib_uverbs] May 9 13:43:02 iclust-tiger2 kernel: sp=e00000001e03fe20 bsp=e00000001e039108 May 9 13:43:02 iclust-tiger2 kernel: [] __fput+0x340/0x360 May 9 13:43:02 iclust-tiger2 kernel: sp=e00000001e03fe20 bsp=e00000001e0390b8 May 9 13:43:02 iclust-tiger2 kernel: [] fput+0x40/0x60 May 9 13:43:02 iclust-tiger2 kernel: sp=e00000001e03fe20 bsp=e00000001e039098 May 9 13:43:02 iclust-tiger2 kernel: [] filp_close+0xc0/0x1a0 May 9 13:43:02 iclust-tiger2 kernel: sp=e00000001e03fe20 bsp=e00000001e039068 May 9 13:43:02 iclust-tiger2 kernel: [] put_files_struct+0x230/0x300 May 9 13:43:02 iclust-tiger2 kernel: sp=e00000001e03fe20 bsp=e00000001e039008 May 9 13:43:02 iclust-tiger2 kernel: [] do_exit+0x230/0x1ac0 May 9 13:43:02 iclust-tiger2 kernel: sp=e00000001e03fe20 bsp=e00000001e038f30 May 9 13:43:02 iclust-tiger2 kernel: [] do_group_exit+0x90/0x1c0 May 9 13:43:02 iclust-tiger2 kernel: sp=e00000001e03fe30 bsp=e00000001e038f00 May 9 13:43:02 iclust-tiger2 kernel: [] sys_exit_group+0x20/0x40 May 9 13:43:02 iclust-tiger2 kernel: sp=e00000001e03fe30 bsp=e00000001e038ea8 May 9 13:43:02 iclust-tiger2 kernel: [] ia64_ret_from_syscall+0x0/0x20 May 9 13:43:02 iclust-tiger2 kernel: sp=e00000001e03fe30 bsp=e00000001e038ea8 May 9 13:43:02 iclust-tiger2 kernel: [] 0xa000000000010640 May 9 13:43:02 iclust-tiger2 kernel: sp=e00000001e040000 bsp=e00000001e038ea8 May 9 13:43:02 iclust-tiger2 kernel: <1>Unable to handle kernel NULL pointer dereference (addressong[6525]: Oops 8813272891392 [4] May 9 13:43:02 iclust-tiger2 kernel: Modules linked in: ib_uverbs(U) ib_sdp(U) ib_cm(U) ib_ipoib(U) ib_sa(U) nls_utf8(U) parport_pc(U) lp(U) parport(U) autofs4(U) sunrpc(U) ds(U) yenta_socket(U) pcmcia_core(U) vfat(U) fat(U) button(U) md5(U) ipv6(U) joydev(U) uhci_hcd(U) ib_mthca(U) ib_mad(U) ib_core(U) e1000(U) sg(U) dm_snapshot(U) dm_zero(U) dm_mirror(U) ext3(U) jbd(U) dm_mod(U) mptscsih(U) mptbase(U) sd_mod(U) scsi_mod(U) May 9 13:43:02 iclust-tiger2 kernel: May 9 13:43:02 iclust-tiger2 kernel: Pid: 6525, CPU 1, comm: lt-ibv_pingpong May 9 13:43:02 iclust-tiger2 kernel: psr : 0000101008026018 ifs : 800000000000028b ip : [] Not tainted May 9 13:43:02 iclust-tiger2 kernel: ip is at mm_release+0x60/0x120 May 9 13:43:02 iclust-tiger2 kernel: unat: 0000000000000000 pfs : 000000000000040b rsc : 0000000000000003 May 9 13:43:02 iclust-tiger2 kernel: rnat: 000000000000040e bsps: 0000000000000003 pr : 00000000056a5595 May 9 13:43:02 iclust-tiger2 kernel: ldrs: 0000000000000000 ccv : 0000000000000000 fpsr: 0009804c8a70033f May 9 13:43:02 iclust-tiger2 kernel: csd : 0000000000000000 ssd : 0000000000000000 May 9 13:43:02 iclust-tiger2 kernel: b0 : a0000001000749d0 b6 : a0000001002fb3c0 b7 : a00000010000fab0 May 9 13:43:02 iclust-tiger2 kernel: f6 : 0fffafffffffff0000000 f7 : 0ffdea000000000000000 May 9 13:43:02 iclust-tiger2 kernel: f8 : 10002a000000000000000 f9 : 100038000000000000000 May 9 13:43:02 iclust-tiger2 kernel: f10 : 0fffe9ffffffff6000000 f11 : 1003e0000000000000000 May 9 13:43:02 iclust-tiger2 kernel: r1 : a00000010096a750 r2 : a0000001007783a0 r3 : ffffffffffffffff May 9 13:43:02 iclust-tiger2 kernel: r8 : 0000000000000000 r9 : e000000001015900 r10 : a000000100785270 May 9 13:43:02 iclust-tiger2 kernel: r11 : 0000000000000000 r12 : e00000001e03f6d0 r13 : e00000001e038000 May 9 13:43:02 iclust-tiger2 kernel: r14 : 0000000000000048 r15 : e00000001e0381f8 r16 : e00000001e038d38 May 9 13:43:02 iclust-tiger2 kernel: r17 : 0000000000000001 r18 : 0000000000000000 r19 : e00000001e038020 May 9 13:43:02 iclust-tiger2 kernel: r20 : e00000001e038de0 r21 : 0000000000400104 r22 : 0000000000400104 May 9 13:43:02 iclust-tiger2 kernel: r23 : 0000000000000000 r24 : e00000001e038d90 r25 : 000000000000784d May 9 13:43:02 iclust-tiger2 kernel: r26 : 000000000000003e r27 : 0000001008022018 r28 : a0000001007b7e56 May 9 13:43:02 iclust-tiger2 kernel: r29 : ffffffffffffffff r30 : a000000100770d90 r31 : a00000010077d0c0 May 9 13:43:02 iclust-tiger2 kernel: May 9 13:43:02 iclust-tiger2 kernel: Call Trace: May 9 13:43:02 iclust-tiger2 kernel: [] show_stack+0x80/0xa0 May 9 13:43:02 iclust-tiger2 kernel: sp=e00000001e03f280 bsp=e00000001e039900 May 9 13:43:02 iclust-tiger2 kernel: [] show_regs+0x890/0x8c0 May 9 13:43:02 iclust-tiger2 kernel: sp=e00000001e03f450 bsp=e00000001e0398b8 May 9 13:43:02 iclust-tiger2 kernel: [] die+0x150/0x240 May 9 13:43:02 iclust-tiger2 kernel: sp=e00000001e03f470 bsp=e00000001e039878 May 9 13:43:02 iclust-tiger2 kernel: [] ia64_do_page_fault+0x9f0/0xba0 May 9 13:43:02 iclust-tiger2 kernel: sp=e00000001e03f470 bsp=e00000001e039810 May 9 13:43:02 iclust-tiger2 kernel: [] ia64_leave_kernel+0x0/0x260 May 9 13:43:02 iclust-tiger2 kernel: sp=e00000001e03f500 bsp=e00000001e039810 May 9 13:43:02 iclust-tiger2 kernel: [] mm_release+0x60/0x120 May 9 13:43:02 iclust-tiger2 kernel: sp=e00000001e03f6d0 bsp=e00000001e0397b0 May 9 13:43:02 iclust-tiger2 kernel: [] __exit_mm+0x30/0x460 May 9 13:43:02 iclust-tiger2 kernel: sp=e00000001e03f6d0 bsp=e00000001e039770 May 9 13:43:02 iclust-tiger2 kernel: [] do_exit+0x1c0/0x1ac0 May 9 13:43:02 iclust-tiger2 kernel: sp=e00000001e03f6d0 bsp=e00000001e0396a0 May 9 13:43:02 iclust-tiger2 kernel: [] die+0x230/0x240 May 9 13:43:02 iclust-tiger2 kernel: sp=e00000001e03f6e0 bsp=e00000001e039660 May 9 13:43:02 iclust-tiger2 kernel: [] ia64_do_page_fault+0x9f0/0xba0 May 9 13:43:02 iclust-tiger2 kernel: sp=e00000001e03f6e0 bsp=e00000001e0395f0 May 9 13:43:02 iclust-tiger2 kernel: [] ia64_leave_kernel+0x0/0x260 May 9 13:43:02 iclust-tiger2 kernel: sp=e00000001e03f770 bsp=e00000001e0395f0 May 9 13:43:02 iclust-tiger2 kernel: [] mm_release+0x60/0x120 May 9 13:43:02 iclust-tiger2 kernel: sp=e00000001e03f940 bsp=e00000001e039598 May 9 13:43:02 iclust-tiger2 kernel: [] __exit_mm+0x30/0x460 May 9 13:49:05 iclust-tiger2 syslogd 1.4.1: restart. From roland at topspin.com Tue May 10 10:50:00 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 10 May 2005 10:50:00 -0700 Subject: [openib-general] [KDAPL] module initialization order In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35AB6@mtvex01.mtv.mtl.com> (Itamar Rabenstein's message of "Tue, 10 May 2005 20:49:04 +0300") References: <91DB792C7985D411BEC300B40080D29CC35AB6@mtvex01.mtv.mtl.com> Message-ID: <52y8anvulz.fsf@topspin.com> Itamar> I still see problem when NFSoRDMA or any other dat Itamar> consumer and dat will be build statically but the Itamar> dat_provider (ib,iwarp,...) will be build as a module. Itamar> who will protect the dat consumer ? how will the dat Itamar> consumer know that there is no dat_provider available when Itamar> dat_provider modules are not loaded? It seems that the kDAPL design of having consumers call a function to get a list of adapters does not fit very well with the Linux model of adapters appearing asynchronously due to drivers being loaded or even due to device hotplug. - R. From tduffy at sun.com Tue May 10 10:51:05 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 10 May 2005 10:51:05 -0700 Subject: [openib-general] [PATCH] kDAPL: remove DAT_BOOLEAN Message-ID: <1115747465.8222.15.camel@duffman> I am sending this as a separate patch because I am not sure what the "right" thing to do is. Looking around the kernel, it seems booleans are handled in different ways. Some places, they are enums, others just #defines. In e1000, ixgb, compaq fc, DAT960, and xfs, they are enums. In jfs and arch parisc, a bool is an int and TRUE and FALSE are #define'd. TRUE and FALSE are defined so many times in the kernel, it seems rather ridiculous. Maybe the right thing to do is fix the whole kernel in a consistent way and submit an upstream patch, but this will require quite a bit of janitorial work. I don't mind using either way in kDAPL (or some third -- getting rid of bool's all together?), but this patch uses the enum method. At least we need to get rid of the DAT_ prefix. Singed-off-by: Tom Duffy Index: linux-kernel/test/dapltest/test/dapl_performance_util.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_performance_util.c (revision 2295) +++ linux-kernel/test/dapltest/test/dapl_performance_util.c (working copy) @@ -35,8 +35,8 @@ DT_Performance_Test_Create ( Per_Test_Data_t *pt_ptr, DAT_IA_HANDLE *ia_handle, DAT_IA_ADDRESS_PTR remote_ia_addr, - DAT_BOOLEAN is_server, - DAT_BOOLEAN is_remote_little_endian, + boolean_t is_server, + boolean_t is_remote_little_endian, Performance_Test_t **perf_test) { Performance_Test_t *test_ptr; @@ -266,7 +266,7 @@ int DT_Performance_Test_Destroy ( Per_Test_Data_t *pt_ptr, Performance_Test_t *test_ptr, - DAT_BOOLEAN is_server) + boolean_t is_server) { DAT_RETURN ret; DAT_EP_HANDLE ep_handle; Index: linux-kernel/test/dapltest/test/dapl_cnxn.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_cnxn.c (revision 2295) +++ linux-kernel/test/dapltest/test/dapl_cnxn.c (working copy) @@ -32,8 +32,8 @@ int get_ep_connection_state (DT_Tdep_Print_Head *phead, DAT_EP_HANDLE ep_handle) { DAT_EP_STATE ep_state; - DAT_BOOLEAN in_dto_idle; - DAT_BOOLEAN out_dto_idle; + boolean_t in_dto_idle; + boolean_t out_dto_idle; DAT_RETURN ret; char *recv_status = "Idle"; char *req_status = "Idle"; Index: linux-kernel/test/dapltest/test/dapl_transaction_test.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_transaction_test.c (revision 2295) +++ linux-kernel/test/dapltest/test/dapl_transaction_test.c (working copy) @@ -159,9 +159,9 @@ DT_Transaction_Test_Server (void *params bool DT_Transaction_Create_Test (Per_Test_Data_t * pt_ptr, DAT_IA_HANDLE *ia_handle, - DAT_BOOLEAN is_server, + boolean_t is_server, unsigned int port_num, - DAT_BOOLEAN remote_is_little_endian, + boolean_t remote_is_little_endian, DAT_IA_ADDRESS_PTR remote_ia_addr) { Transaction_Test_t *test_ptr; @@ -820,7 +820,7 @@ retry: */ for (j = 0; j < test_ptr->cmd->num_ops; j++) { - DAT_BOOLEAN us; + boolean_t us; us = (pt_ptr->local_is_server && test_ptr->ep_context[i].op[j].server_initiated) || @@ -1039,7 +1039,7 @@ retry: RMI_RECV_BUFFER_ID); for (j = 0; j < test_ptr->cmd->num_ops; j++) { - DAT_BOOLEAN us; + boolean_t us; us = (pt_ptr->local_is_server && test_ptr->ep_context[i].op[j].server_initiated) || Index: linux-kernel/test/dapltest/test/dapl_bpool.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_bpool.c (revision 2295) +++ linux-kernel/test/dapltest/test/dapl_bpool.c (working copy) @@ -93,8 +93,8 @@ DT_BpoolAlloc ( DAT_COUNT seg_size, DAT_COUNT num_segs, DAT_COUNT alignment, - DAT_BOOLEAN enable_rdma_write, - DAT_BOOLEAN enable_rdma_read) + boolean_t enable_rdma_write, + boolean_t enable_rdma_read) { unsigned char *module = "DT_BpoolAlloc"; unsigned char *alloc_ptr = NULL; Index: linux-kernel/test/dapltest/test/dapl_test_util.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_test_util.c (revision 2295) +++ linux-kernel/test/dapltest/test/dapl_test_util.c (working copy) @@ -30,7 +30,7 @@ /* ----------------------------------------------------------- * Gather info about default attributes */ -DAT_BOOLEAN +boolean_t DT_query ( Per_Test_Data_t *pt_ptr, DAT_IA_HANDLE ia_handle, DAT_EP_HANDLE ep_handle) @@ -158,7 +158,7 @@ DT_query ( Per_Test_Data_t *pt_ptr, /* ----------------------------------------------------------- * Post a recv buffer */ -DAT_BOOLEAN +boolean_t DT_post_recv_buffer (DT_Tdep_Print_Head *phead, DAT_EP_HANDLE ep_handle, Bpool * bp, @@ -203,7 +203,7 @@ DT_post_recv_buffer (DT_Tdep_Print_Head /* ----------------------------------------------------------- * Post a send buffer */ -DAT_BOOLEAN +boolean_t DT_post_send_buffer (DT_Tdep_Print_Head *phead, DAT_EP_HANDLE ep_handle, Bpool * bp, Index: linux-kernel/test/dapltest/include/dapl_performance_test.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_performance_test.h (revision 2295) +++ linux-kernel/test/dapltest/include/dapl_performance_test.h (working copy) @@ -67,7 +67,7 @@ typedef struct Per_Test_Data_t *pt_ptr; Performance_Cmd_t *cmd; DAT_IA_ADDRESS_PTR remote_ia_addr; - DAT_BOOLEAN is_remote_little_endian; + boolean_t is_remote_little_endian; DAT_CONN_QUAL base_port; struct dat_ia_attr ia_attr; DAT_IA_HANDLE ia_handle; Index: linux-kernel/test/dapltest/include/dapl_proto.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_proto.h (revision 2295) +++ linux-kernel/test/dapltest/include/dapl_proto.h (working copy) @@ -88,8 +88,8 @@ Bpool * DT_BpoolAlloc (Per_Test_ DAT_COUNT seg_size, DAT_COUNT num_segs, DAT_COUNT alignment, - DAT_BOOLEAN enable_rdma_write, - DAT_BOOLEAN enable_rdma_read); + boolean_t enable_rdma_write, + boolean_t enable_rdma_read); bool DT_Bpool_Destroy (Per_Test_Data_t * pt_ptr, DT_Tdep_Print_Head *phead, @@ -267,13 +267,13 @@ bool DT_Performance_Test_Serv bool DT_Performance_Test_Create (Per_Test_Data_t * pt_ptr, DAT_IA_HANDLE * ia_handle, DAT_IA_ADDRESS_PTR remote_ia_addr, - DAT_BOOLEAN is_server, - DAT_BOOLEAN is_remote_little_endian, + boolean_t is_server, + boolean_t is_remote_little_endian, Performance_Test_t **perf_test); int DT_Performance_Test_Destroy (Per_Test_Data_t * pt_ptr, Performance_Test_t *test_ptr, - DAT_BOOLEAN is_server); + boolean_t is_server); bool DT_performance_post_rdma_op (Performance_Ep_Context_t *ep_context, DAT_EVD_HANDLE reqt_evd_hdl, @@ -349,17 +349,17 @@ Per_Test_Data_t *DT_Alloc_Per_Test_Data void DT_Free_Per_Test_Data (Per_Test_Data_t * pt_ptr); /* dapl_test_util.c */ -DAT_BOOLEAN DT_query (Per_Test_Data_t *pt_ptr, +boolean_t DT_query (Per_Test_Data_t *pt_ptr, DAT_IA_HANDLE ia_handle, DAT_EP_HANDLE ep_handle); -DAT_BOOLEAN DT_post_recv_buffer (DT_Tdep_Print_Head* phead, +boolean_t DT_post_recv_buffer (DT_Tdep_Print_Head* phead, DAT_EP_HANDLE ep_handle, Bpool * bp, int index, int size); -DAT_BOOLEAN DT_post_send_buffer (DT_Tdep_Print_Head* phead, +boolean_t DT_post_send_buffer (DT_Tdep_Print_Head* phead, DAT_EP_HANDLE ep_handle, Bpool * bp, int index, @@ -471,9 +471,9 @@ void DT_Transaction_Test_Serv bool DT_Transaction_Create_Test (Per_Test_Data_t * pt_ptr, DAT_IA_HANDLE * ia_handle, - DAT_BOOLEAN is_server, + boolean_t is_server, unsigned int port_num, - DAT_BOOLEAN remote_is_little_endian, + boolean_t remote_is_little_endian, DAT_IA_ADDRESS_PTR remote_ia_addr); void DT_Transaction_Main (void *param); Index: linux-kernel/test/dapltest/include/dapl_bpool.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_bpool.h (revision 2295) +++ linux-kernel/test/dapltest/include/dapl_bpool.h (working copy) @@ -49,8 +49,8 @@ struct Bpool_tag { DAT_LMR_HANDLE lmr_handle; /* local access */ DAT_LMR_CONTEXT lmr_context; struct dat_lmr_triplet *tripl_start; /* local IOV */ - DAT_BOOLEAN enable_rdma_write; /* remote access */ - DAT_BOOLEAN enable_rdma_read; + boolean_t enable_rdma_write; /* remote access */ + boolean_t enable_rdma_read; DAT_RMR_HANDLE rmr_handle; DAT_RMR_CONTEXT rmr_context; DAT_EVD_HANDLE rmr_evd_handle; Index: linux-kernel/test/dapltest/include/dapl_transaction_test.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_transaction_test.h (revision 2295) +++ linux-kernel/test/dapltest/include/dapl_transaction_test.h (working copy) @@ -35,11 +35,11 @@ typedef struct { - DAT_BOOLEAN server_initiated; + boolean_t server_initiated; DT_Transfer_Type transfer_type; uint32_t num_segs; uint32_t seg_size; - DAT_BOOLEAN reap_send_on_recv; + boolean_t reap_send_on_recv; Bpool *bp; /* RDMA info */ @@ -72,8 +72,8 @@ typedef struct typedef struct { /* This group set up by DT_Transaction_Create_Test() */ - DAT_BOOLEAN is_server; - DAT_BOOLEAN remote_is_little_endian; + boolean_t is_server; + boolean_t remote_is_little_endian; Per_Test_Data_t *pt_ptr; DAT_IA_HANDLE ia_handle; Transaction_Cmd_t *cmd; Index: linux-kernel/test/dapltest/cmd/dapl_netaddr.c =================================================================== --- linux-kernel/test/dapltest/cmd/dapl_netaddr.c (revision 2295) +++ linux-kernel/test/dapltest/cmd/dapl_netaddr.c (working copy) @@ -125,7 +125,7 @@ DT_NetAddrLookupHostAddress (DAT_IA_ADDR DT_Mdep_printf ("getaddrinfo (%s) failed (%s)\n", hostname, whatzit); - return DAT_FALSE; + return FALSE; } /* Pull out IP address and print it as a sanity check */ @@ -139,6 +139,6 @@ DT_NetAddrLookupHostAddress (DAT_IA_ADDR *to_netaddr = * ((DAT_IA_ADDRESS_PTR) target->ai_addr); - return ( DAT_TRUE ); + return ( TRUE ); } Index: linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.c =================================================================== --- linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.c (revision 2295) +++ linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.c (working copy) @@ -431,7 +431,7 @@ DT_Mdep_wait_object_init ( DT_WAIT_OBJECT *wait_obj) { - wait_obj->signaled = DAT_FALSE; + wait_obj->signaled = FALSE; if ( 0 != pthread_cond_init ( &wait_obj->cv, NULL ) ) { return (-1); @@ -492,7 +492,7 @@ DT_Mdep_wait_object_wait ( future.tv_nsec = now.tv_usec * 1000; pthread_mutex_lock (&wait_obj->lock); - while ( wait_obj->signaled == DAT_FALSE && pthread_status == 0) + while ( wait_obj->signaled == FALSE && pthread_status == 0) { pthread_status = pthread_cond_timedwait ( &wait_obj->cv, &wait_obj->lock, &future ); @@ -512,7 +512,7 @@ DT_Mdep_wait_object_wait ( else { pthread_mutex_lock (&wait_obj->lock); - while ( wait_obj->signaled == DAT_FALSE && pthread_status == 0) + while ( wait_obj->signaled == FALSE && pthread_status == 0) { pthread_status = pthread_cond_wait ( &wait_obj->cv, &wait_obj->lock ); Index: linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c =================================================================== --- linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c (revision 2295) +++ linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c (working copy) @@ -47,7 +47,7 @@ typedef struct Tdep_Evd_Tag static void DT_Tdep_Event_Callback ( DAT_PVOID instance_data, const struct dat_event *dat_event, - DAT_BOOLEAN bool_arg); + boolean_t bool_arg); /* static data */ @@ -324,7 +324,7 @@ DT_Tdep_evd_free (DAT_EVD_HANDLE evd_han static void DT_Tdep_Event_Callback ( DAT_PVOID instance_data, const struct dat_event * dat_event, - DAT_BOOLEAN bool_arg) + boolean_t bool_arg) { Tdep_Event *event; Tdep_Evd *evd_ptr; Index: linux-kernel/dat-provider/dapl_cr_callback.c =================================================================== --- linux-kernel/dat-provider/dapl_cr_callback.c (revision 2295) +++ linux-kernel/dat-provider/dapl_cr_callback.c (working copy) @@ -87,7 +87,7 @@ void dapl_cr_callback(ib_cm_handle_t ib_ sp_ptr->header.magic == DAPL_MAGIC_RSP); /* Obtain the event number from the provider layer */ - dat_event_num = dapl_ib_get_dat_event(ib_cm_event, DAT_FALSE); + dat_event_num = dapl_ib_get_dat_event(ib_cm_event, FALSE); /* * CONNECT_REQUEST events create an event on the PSP @@ -128,7 +128,7 @@ void dapl_cr_callback(ib_cm_handle_t ib_ */ spin_lock(&sp_ptr->header.lock); - if (sp_ptr->listening == DAT_FALSE) { + if (sp_ptr->listening == FALSE) { spin_unlock(&sp_ptr->header.lock); dapl_dbg_log(DAPL_DBG_TYPE_CM, "---> dapl_cr_callback: conn event on down SP\n"); @@ -143,7 +143,7 @@ void dapl_cr_callback(ib_cm_handle_t ib_ * RSP connections only allow a single connection. Close * it down NOW so we reject any further connections. */ - sp_ptr->listening = DAT_FALSE; + sp_ptr->listening = FALSE; } spin_unlock(&sp_ptr->header.lock); @@ -205,7 +205,7 @@ void dapl_cr_callback(ib_cm_handle_t ib_ return; } ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; - dapl_ib_disconnect_clean(ep_ptr, DAT_FALSE, + dapl_ib_disconnect_clean(ep_ptr, FALSE, ib_cm_event); spin_unlock(&ep_ptr->header.lock); @@ -221,7 +221,7 @@ void dapl_cr_callback(ib_cm_handle_t ib_ */ spin_lock(&ep_ptr->header.lock); ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; - dapl_ib_disconnect_clean(ep_ptr, DAT_FALSE, + dapl_ib_disconnect_clean(ep_ptr, FALSE, ib_cm_event); spin_unlock(&ep_ptr->header.lock); @@ -231,7 +231,7 @@ void dapl_cr_callback(ib_cm_handle_t ib_ { spin_lock(&ep_ptr->header.lock); ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; - dapl_ib_disconnect_clean(ep_ptr, DAT_FALSE, + dapl_ib_disconnect_clean(ep_ptr, FALSE, ib_cm_event); spin_unlock(&ep_ptr->header.lock); @@ -433,7 +433,7 @@ DAPL_EP *dapl_get_sp_ep(ib_cm_handle_t i * If this SP has been removed from service, free it * up after the last CR is removed */ - if (sp_ptr->listening != DAT_TRUE && sp_ptr->cr_list_count == 0 + if (sp_ptr->listening != TRUE && sp_ptr->cr_list_count == 0 && sp_ptr->state != DAPL_SP_STATE_FREE) { dapl_dbg_log(DAPL_DBG_TYPE_CM, "--> dapl_get_sp_ep! disconnect dump sp: %p \n", Index: linux-kernel/dat-provider/dapl_module.c =================================================================== --- linux-kernel/dat-provider/dapl_module.c (revision 2295) +++ linux-kernel/dat-provider/dapl_module.c (working copy) @@ -236,7 +236,7 @@ void dapl_add_one(struct ib_device *devi dev_name_suffix_table[i + 1]); provider_info.dat_version_major = DAT_VERSION_MAJOR; provider_info.dat_version_minor = DAT_VERSION_MINOR; - provider_info.is_thread_safe = DAT_FALSE; + provider_info.is_thread_safe = FALSE; DAT_PROVIDER_INIT_FUNC_NAME(&provider_info, device, i); } return; @@ -258,7 +258,7 @@ void dapl_remove_one(struct ib_device *d dev_name_suffix_table[i + 1]); provider_info.dat_version_major = DAT_VERSION_MAJOR; provider_info.dat_version_minor = DAT_VERSION_MINOR; - provider_info.is_thread_safe = DAT_FALSE; + provider_info.is_thread_safe = FALSE; DAT_PROVIDER_FINI_FUNC_NAME(&provider_info); } return; Index: linux-kernel/dat-provider/dapl_psp_create.c =================================================================== --- linux-kernel/dat-provider/dapl_psp_create.c (revision 2295) +++ linux-kernel/dat-provider/dapl_psp_create.c (working copy) @@ -76,7 +76,7 @@ dapl_psp_create(DAT_IA_HANDLE ia_handle, DAPL_IA *ia_ptr; DAPL_SP *sp_ptr; DAPL_EVD *evd_ptr; - DAT_BOOLEAN sp_found; + boolean_t sp_found; DAT_RETURN dat_status; ia_ptr = (DAPL_IA *) ia_handle; @@ -115,19 +115,19 @@ dapl_psp_create(DAT_IA_HANDLE ia_handle, * See if we have a quiescent listener to use for this PSP, else * create one and set it listening */ - sp_ptr = dapl_ia_sp_search(ia_ptr, conn_qual, DAT_TRUE); - sp_found = DAT_TRUE; + sp_ptr = dapl_ia_sp_search(ia_ptr, conn_qual, TRUE); + sp_found = TRUE; if (sp_ptr == NULL) { /* Allocate PSP */ - sp_found = DAT_FALSE; - sp_ptr = dapl_sp_alloc(ia_ptr, DAT_TRUE); + sp_found = FALSE; + sp_ptr = dapl_sp_alloc(ia_ptr, TRUE); if (sp_ptr == NULL) { dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto bail; } - } else if (sp_ptr->listening == DAT_TRUE) { + } else if (sp_ptr->listening == TRUE) { dat_status = DAT_ERROR(DAT_CONN_QUAL_IN_USE, 0); goto bail; } @@ -151,13 +151,13 @@ dapl_psp_create(DAT_IA_HANDLE ia_handle, * even before this call returns! */ sp_ptr->state = DAPL_SP_STATE_PSP_LISTENING; - sp_ptr->listening = DAT_TRUE; + sp_ptr->listening = TRUE; /* * If this is a new sp we need to add it to the IA queue, and set up * a conn_listener. */ - if (sp_found == DAT_FALSE) { + if (sp_found == FALSE) { /* Link it onto the IA before enabling it to receive conn * requests */ Index: linux-kernel/dat-provider/dapl_evd_connection_callb.c =================================================================== --- linux-kernel/dat-provider/dapl_evd_connection_callb.c (revision 2295) +++ linux-kernel/dat-provider/dapl_evd_connection_callb.c (working copy) @@ -109,7 +109,7 @@ dapl_evd_connection_callback(ib_cm_handl } /* Obtain the event number from the provider layer */ - dat_event_num = dapl_ib_get_dat_event(ib_cm_event, DAT_FALSE); + dat_event_num = dapl_ib_get_dat_event(ib_cm_event, FALSE); switch (dat_event_num) { case DAT_CONNECTION_EVENT_ESTABLISHED: @@ -148,7 +148,7 @@ dapl_evd_connection_callback(ib_cm_handl * another connection */ ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; - dapl_ib_disconnect_clean(ep_ptr, DAT_TRUE, ib_cm_event); + dapl_ib_disconnect_clean(ep_ptr, TRUE, ib_cm_event); spin_unlock(&ep_ptr->header.lock); break; @@ -156,7 +156,7 @@ dapl_evd_connection_callback(ib_cm_handl case DAT_CONNECTION_EVENT_PEER_REJECTED: { ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; - dapl_ib_disconnect_clean(ep_ptr, DAT_TRUE, ib_cm_event); + dapl_ib_disconnect_clean(ep_ptr, TRUE, ib_cm_event); spin_unlock(&ep_ptr->header.lock); break; @@ -164,7 +164,7 @@ dapl_evd_connection_callback(ib_cm_handl case DAT_CONNECTION_EVENT_UNREACHABLE: { ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; - dapl_ib_disconnect_clean(ep_ptr, DAT_TRUE, ib_cm_event); + dapl_ib_disconnect_clean(ep_ptr, TRUE, ib_cm_event); spin_unlock(&ep_ptr->header.lock); break; @@ -172,7 +172,7 @@ dapl_evd_connection_callback(ib_cm_handl case DAT_CONNECTION_EVENT_NON_PEER_REJECTED: { ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; - dapl_ib_disconnect_clean(ep_ptr, DAT_TRUE, ib_cm_event); + dapl_ib_disconnect_clean(ep_ptr, TRUE, ib_cm_event); spin_unlock(&ep_ptr->header.lock); break; @@ -180,7 +180,7 @@ dapl_evd_connection_callback(ib_cm_handl case DAT_CONNECTION_EVENT_BROKEN: { ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; - dapl_ib_disconnect_clean(ep_ptr, DAT_FALSE, + dapl_ib_disconnect_clean(ep_ptr, FALSE, ib_cm_event); spin_unlock(&ep_ptr->header.lock); Index: linux-kernel/dat-provider/dapl_psp_create_any.c =================================================================== --- linux-kernel/dat-provider/dapl_psp_create_any.c (revision 2295) +++ linux-kernel/dat-provider/dapl_psp_create_any.c (working copy) @@ -121,7 +121,7 @@ dapl_psp_create_any(DAT_IA_HANDLE ia_han } /* Allocate PSP */ - sp_ptr = dapl_sp_alloc(ia_ptr, DAT_TRUE); + sp_ptr = dapl_sp_alloc(ia_ptr, TRUE); if (sp_ptr == NULL) { dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); @@ -149,7 +149,7 @@ dapl_psp_create_any(DAT_IA_HANDLE ia_han * even before this call returns! */ sp_ptr->state = DAPL_SP_STATE_PSP_LISTENING; - sp_ptr->listening = DAT_TRUE; + sp_ptr->listening = TRUE; limit_conn_qual = 0; lcl_conn_qual = hint_conn_qual; Index: linux-kernel/dat-provider/dapl_psp_free.c =================================================================== --- linux-kernel/dat-provider/dapl_psp_free.c (revision 2295) +++ linux-kernel/dat-provider/dapl_psp_free.c (working copy) @@ -89,7 +89,7 @@ DAT_RETURN dapl_psp_free(DAT_PSP_HANDLE */ spin_lock(&sp_ptr->header.lock); - sp_ptr->listening = DAT_FALSE; + sp_ptr->listening = FALSE; /* Release reference on EVD. If an error was encountered in a previous * free the evd_handle will be NULL Index: linux-kernel/dat-provider/dapl_ep_get_status.c =================================================================== --- linux-kernel/dat-provider/dapl_ep_get_status.c (revision 2295) +++ linux-kernel/dat-provider/dapl_ep_get_status.c (working copy) @@ -62,7 +62,7 @@ DAT_RETURN dapl_ep_get_status(DAT_EP_HANDLE ep_handle, DAT_EP_STATE * ep_state, - DAT_BOOLEAN * in_dto_idle, DAT_BOOLEAN * out_dto_idle) + boolean_t *in_dto_idle, boolean_t *out_dto_idle) { DAPL_EP *ep_ptr; DAT_RETURN dat_status; @@ -92,14 +92,12 @@ dapl_ep_get_status(DAT_EP_HANDLE ep_hand if (in_dto_idle != NULL) { *in_dto_idle = - (atomic_read(&ep_ptr->recv_count)) ? DAT_FALSE : - DAT_TRUE; + (atomic_read(&ep_ptr->recv_count)) ? FALSE : TRUE; } if (out_dto_idle != NULL) { *out_dto_idle = - (atomic_read(&ep_ptr->req_count)) ? DAT_FALSE : - DAT_TRUE; + (atomic_read(&ep_ptr->req_count)) ? FALSE : TRUE; } bail: Index: linux-kernel/dat-provider/dapl_ia_query.c =================================================================== --- linux-kernel/dat-provider/dapl_ia_query.c (revision 2295) +++ linux-kernel/dat-provider/dapl_ia_query.c (working copy) @@ -68,9 +68,9 @@ dapl_ia_query(DAT_IA_HANDLE ia_handle, DAPL_IA *ia_ptr; DAT_RETURN dat_status; struct evd_merge_type { - DAT_BOOLEAN array[6][6]; + boolean_t array[6][6]; } *evd_merge; - DAT_BOOLEAN val; + boolean_t val; int i; int j; @@ -120,9 +120,9 @@ dapl_ia_query(DAT_IA_HANDLE ia_handle, provider_attr->dat_qos_supported = DAT_QOS_BEST_EFFORT; provider_attr->completion_flags_supported = DAT_COMPLETION_DEFAULT_FLAG; - provider_attr->is_thread_safe = DAT_FALSE; + provider_attr->is_thread_safe = FALSE; provider_attr->max_private_data_size = IB_MAX_REQ_PDATA_SIZE; - provider_attr->supports_multipath = DAT_FALSE; + provider_attr->supports_multipath = FALSE; provider_attr->ep_creator = DAT_PSP_CREATES_EP_NEVER; provider_attr->optimal_buffer_alignment = DAPL_OPTIMAL_ALIGNMENT; /* The value of pz_support may vary by transport */ @@ -149,17 +149,17 @@ dapl_ia_query(DAT_IA_HANDLE ia_handle, evd_merge = (struct evd_merge_type *)&provider_attr-> evd_stream_merging_supported[0][0]; - val = DAT_TRUE; + val = TRUE; for (i = 0; i < 6; i++) { if (i > 4) { /* ASYNC EVD is 5, so entire row will be 0 */ - val = DAT_FALSE; + val = FALSE; } for (j = 0; j < 5; j++) { evd_merge->array[i][j] = val; } /* Set the ASYNC_EVD column to FALSE */ - evd_merge->array[i][5] = DAT_FALSE; + evd_merge->array[i][5] = FALSE; } #ifndef DAPL_MERGE_CM_DTO @@ -169,9 +169,9 @@ dapl_ia_query(DAT_IA_HANDLE ia_handle, * skip the following code */ /* DAT_EVD_DTO_FLAG | DAT_EVD_CONNECTION_FLAG */ - evd_merge->array[2][3] = DAT_FALSE; + evd_merge->array[2][3] = FALSE; /* DAT_EVD_CONNECTION_FLAG | DAT_EVD_DTO_FLAG */ - evd_merge->array[3][2] = DAT_FALSE; + evd_merge->array[3][2] = FALSE; #endif /* DAPL_MERGE_CM_DTO */ } Index: linux-kernel/dat-provider/dapl_mr_util.h =================================================================== --- linux-kernel/dat-provider/dapl_mr_util.h (revision 2295) +++ linux-kernel/dat-provider/dapl_mr_util.h (working copy) @@ -49,7 +49,7 @@ extern DAT_VADDR dapl_mr_get_address(DAT_REGION_DESCRIPTION desc, DAT_MEM_TYPE type); -static __inline__ DAT_BOOLEAN +static __inline__ boolean_t dapl_mr_bounds_check(DAT_VADDR addr_a, DAT_VLEN length_a, DAT_VADDR addr_b, DAT_VLEN length_b); @@ -67,14 +67,14 @@ dapl_mr_bounds_check(DAT_VADDR addr_a, * */ -static __inline__ DAT_BOOLEAN +static __inline__ boolean_t dapl_mr_bounds_check(DAT_VADDR addr_a, DAT_VLEN length_a, DAT_VADDR addr_b, DAT_VLEN length_b) { if ((addr_a <= addr_b) && (addr_b + length_b) <= (addr_a + length_a)) { - return DAT_TRUE; + return TRUE; } else { - return DAT_FALSE; + return FALSE; } } Index: linux-kernel/dat-provider/dapl_adapter_util.h =================================================================== --- linux-kernel/dat-provider/dapl_adapter_util.h (revision 2295) +++ linux-kernel/dat-provider/dapl_adapter_util.h (working copy) @@ -139,11 +139,11 @@ DAT_RETURN dapl_ib_mw_bind(DAPL_RMR * rm DAT_VADDR virtual_address, DAT_VLEN length, DAT_MEM_PRIV_FLAGS mem_priv, - DAT_BOOLEAN is_signaled); + boolean_t is_signaled); DAT_RETURN dapl_ib_mw_unbind(DAPL_RMR * rmr, DAPL_EP * ep, - DAPL_COOKIE * cookie, DAT_BOOLEAN is_signaled); + DAPL_COOKIE * cookie, boolean_t is_signaled); DAT_RETURN dapl_ib_query_hca(DAPL_HCA * hca_ptr, struct dat_ia_attr *ia_attr, @@ -163,14 +163,14 @@ DAT_DTO_COMPLETION_STATUS dapl_ib_get_dt void dapl_ib_reinit_ep(DAPL_EP * ep_ptr); void dapl_ib_disconnect_clean(DAPL_EP * ep_ptr, - DAT_BOOLEAN passive, + boolean_t passive, const ib_cm_events_t ib_cm_event); DAT_RETURN dapl_ib_get_async_event(ib_error_record_t * cause_ptr, DAT_EVENT_NUMBER * async_event); DAT_EVENT_NUMBER dapl_ib_get_dat_event(const ib_cm_events_t ib_cm_event, - DAT_BOOLEAN active); + boolean_t active); ib_cm_events_t dapl_ib_get_cm_event(DAT_EVENT_NUMBER dat_event_num); Index: linux-kernel/dat-provider/dapl_ia_util.c =================================================================== --- linux-kernel/dat-provider/dapl_ia_util.c (revision 2295) +++ linux-kernel/dat-provider/dapl_ia_util.c (working copy) @@ -91,7 +91,7 @@ DAPL_IA *dapl_ia_alloc(struct dat_provid */ ia_ptr->hca_ptr = hca_ptr; ia_ptr->async_error_evd = NULL; - ia_ptr->cleanup_async_error_evd = DAT_FALSE; + ia_ptr->cleanup_async_error_evd = FALSE; dapl_llist_init_entry(&ia_ptr->hca_ia_list_entry); dapl_llist_init_head(&ia_ptr->ep_list_head); dapl_llist_init_head(&ia_ptr->lmr_list_head); @@ -316,7 +316,7 @@ DAT_RETURN dapl_ia_abrupt_close(DAPL_IA * Free the async EVD, shutting down callbacks from the HCA. */ if (ia_ptr->async_error_evd && - (DAT_TRUE == ia_ptr->cleanup_async_error_evd)) { + (TRUE == ia_ptr->cleanup_async_error_evd)) { dat_status = dapl_ia_teardown_callbacks(ia_ptr); atomic_dec(&ia_ptr->async_error_evd->evd_ref_count); @@ -382,7 +382,7 @@ DAT_RETURN dapl_ia_graceful_close(DAPL_I /* if the async evd does not need to be cleaned up */ /* (ie. it was not created by dapl_ia_open) */ /* then the evd list should be empty */ - if (DAT_FALSE == ia_ptr->cleanup_async_error_evd) { + if (FALSE == ia_ptr->cleanup_async_error_evd) { if (!dapl_llist_is_empty(&ia_ptr->evd_list_head)) { dat_status = DAT_ERROR(DAT_INVALID_STATE, @@ -437,7 +437,7 @@ DAT_RETURN dapl_ia_graceful_close(DAPL_I /* Tear down the async EVD if needed, first shutting down callbacks. */ if (ia_ptr->async_error_evd && - (DAT_TRUE == ia_ptr->cleanup_async_error_evd)) { + (TRUE == ia_ptr->cleanup_async_error_evd)) { cur_dat_status = dapl_ia_teardown_callbacks(ia_ptr); if (DAT_SUCCESS != cur_dat_status) { dat_status = cur_dat_status; @@ -683,7 +683,7 @@ void dapl_ia_unlink_sp(DAPL_IA * ia_ptr, * */ DAPL_SP *dapl_ia_sp_search(DAPL_IA * ia_ptr, - DAT_CONN_QUAL conn_qual, DAT_BOOLEAN is_psp) + DAT_CONN_QUAL conn_qual, boolean_t is_psp) { DAPL_SP *sp_ptr; DAPL_LLIST_HEAD *list_head; Index: linux-kernel/dat-provider/dapl_provider.c =================================================================== --- linux-kernel/dat-provider/dapl_provider.c (revision 2295) +++ linux-kernel/dat-provider/dapl_provider.c (working copy) @@ -135,7 +135,7 @@ struct dat_provider g_dapl_provider_temp * * *********************************************************************/ -static DAT_BOOLEAN +static boolean_t dapl_provider_list_key_cmp(const char *name_a, const char *name_b); /********************************************************************* @@ -319,17 +319,17 @@ DAT_RETURN dapl_provider_list_remove(con return status; } -DAT_BOOLEAN dapl_provider_list_key_cmp(const char *name_a, const char *name_b) +boolean_t dapl_provider_list_key_cmp(const char *name_a, const char *name_b) { unsigned int len; len = strlen(name_a); if (strlen(name_b) != len) { - return DAT_FALSE; + return FALSE; } else if (memcmp(name_a, name_b, len)) { - return DAT_FALSE; + return FALSE; } else { - return DAT_TRUE; + return TRUE; } } Index: linux-kernel/dat-provider/dapl_openib_cm.c =================================================================== --- linux-kernel/dat-provider/dapl_openib_cm.c (revision 2295) +++ linux-kernel/dat-provider/dapl_openib_cm.c (working copy) @@ -511,7 +511,7 @@ DAT_RETURN dapl_ib_disconnect(DAPL_EP * */ void dapl_ib_disconnect_clean(DAPL_EP * ep_ptr, - DAT_BOOLEAN active, + boolean_t active, const ib_cm_events_t ib_cm_event) { int status; Index: linux-kernel/dat-provider/dapl_ia_util.h =================================================================== --- linux-kernel/dat-provider/dapl_ia_util.h (revision 2295) +++ linux-kernel/dat-provider/dapl_ia_util.h (working copy) @@ -78,7 +78,7 @@ void dapl_ia_unlink_sp(DAPL_IA * ia_ptr, void dapl_ia_link_rsp(DAPL_IA * ia_ptr, DAPL_SP * sp_info); DAPL_SP *dapl_ia_sp_search(DAPL_IA * ia_ptr, - DAT_CONN_QUAL conn_qual, DAT_BOOLEAN is_psp); + DAT_CONN_QUAL conn_qual, boolean_t is_psp); DAT_RETURN dapl_ia_setup_callbacks(DAPL_IA * ia_ptr, DAPL_EVD * async_evd_ptr); Index: linux-kernel/dat-provider/dapl_rmr_bind.c =================================================================== --- linux-kernel/dat-provider/dapl_rmr_bind.c (revision 2295) +++ linux-kernel/dat-provider/dapl_rmr_bind.c (working copy) @@ -79,7 +79,7 @@ dapl_rmr_bind_fuse(DAPL_RMR * rmr, DAPL_LMR *lmr; DAPL_COOKIE *cookie; DAT_RETURN dat_status; - DAT_BOOLEAN is_signaled; + boolean_t is_signaled; dat_status = dapl_hash_search(rmr->header.owner_ia->hca_ptr->lmr_hash_table, @@ -104,7 +104,7 @@ dapl_rmr_bind_fuse(DAPL_RMR * rmr, goto bail; } - if (DAT_FALSE == + if (FALSE == dapl_mr_bounds_check(dapl_mr_get_address (lmr->param.region_desc, lmr->param.mem_type), lmr->param.length, @@ -146,8 +146,8 @@ dapl_rmr_bind_fuse(DAPL_RMR * rmr, } is_signaled = - (completion_flags & DAT_COMPLETION_SUPPRESS_FLAG) ? DAT_FALSE : - DAT_TRUE; + (completion_flags & DAT_COMPLETION_SUPPRESS_FLAG) ? FALSE : + TRUE; /* * Take reference before posting to avoid race conditions with @@ -196,7 +196,7 @@ dapl_rmr_bind_unfuse(DAPL_RMR * rmr, { DAPL_COOKIE *cookie; DAT_RETURN dat_status; - DAT_BOOLEAN is_signaled; + boolean_t is_signaled; dat_status = DAT_SUCCESS; /* @@ -246,8 +246,8 @@ dapl_rmr_bind_unfuse(DAPL_RMR * rmr, } is_signaled = - (completion_flags & DAT_COMPLETION_UNSIGNALLED_FLAG) ? DAT_FALSE : - DAT_TRUE; + (completion_flags & DAT_COMPLETION_UNSIGNALLED_FLAG) ? FALSE : + TRUE; /* * Take reference before posting to avoid race conditions with Index: linux-kernel/dat-provider/dapl_openib_util.c =================================================================== --- linux-kernel/dat-provider/dapl_openib_util.c (revision 2295) +++ linux-kernel/dat-provider/dapl_openib_util.c (working copy) @@ -623,7 +623,7 @@ dapl_ib_mw_bind(DAPL_RMR * rmr, DAPL_COOKIE * cookie, DAT_VADDR virtual_address, DAT_VLEN length, - DAT_MEM_PRIV_FLAGS mem_priv, DAT_BOOLEAN is_signaled) + DAT_MEM_PRIV_FLAGS mem_priv, boolean_t is_signaled) { int ib_status; struct ib_mw_bind mw_bind_prop; @@ -632,7 +632,7 @@ dapl_ib_mw_bind(DAPL_RMR * rmr, mw_bind_prop.length = length; mw_bind_prop.mw_access_flags = dapl_ib_convert_mem_privileges(mem_priv); mw_bind_prop.send_flags = - (is_signaled == DAT_TRUE) ? IB_SEND_SIGNALED : 0; + (is_signaled == TRUE) ? IB_SEND_SIGNALED : 0; mw_bind_prop.mr = lmr->mr_handle; mw_bind_prop.wr_id = (u64) (uintptr_t) cookie; dapl_dbg_log(DAPL_DBG_TYPE_UTIL, @@ -673,7 +673,7 @@ dapl_ib_mw_bind(DAPL_RMR * rmr, */ DAT_RETURN dapl_ib_mw_unbind(DAPL_RMR * rmr, - DAPL_EP * ep, DAPL_COOKIE * cookie, DAT_BOOLEAN is_signaled) + DAPL_EP * ep, DAPL_COOKIE * cookie, boolean_t is_signaled) { int ib_status; struct ib_mw_bind mw_bind_prop; @@ -682,7 +682,7 @@ dapl_ib_mw_unbind(DAPL_RMR * rmr, mw_bind_prop.length = 0; mw_bind_prop.mw_access_flags = 0; mw_bind_prop.send_flags = - (is_signaled == DAT_TRUE) ? IB_SEND_SIGNALED : 0; + (is_signaled == TRUE) ? IB_SEND_SIGNALED : 0; mw_bind_prop.mr = NULL; mw_bind_prop.wr_id = (u64) (uintptr_t) cookie; ib_status = ib_bind_mw(ep->qp_handle, rmr->mw_handle, &mw_bind_prop); @@ -1051,7 +1051,7 @@ static struct ib_cm_event_map { */ DAT_EVENT_NUMBER -dapl_ib_get_dat_event(const ib_cm_events_t ib_cm_event, DAT_BOOLEAN active) +dapl_ib_get_dat_event(const ib_cm_events_t ib_cm_event, boolean_t active) { DAT_EVENT_NUMBER dat_event_num; int index; Index: linux-kernel/dat-provider/dapl_ep_modify.c =================================================================== --- linux-kernel/dat-provider/dapl_ep_modify.c (revision 2295) +++ linux-kernel/dat-provider/dapl_ep_modify.c (working copy) @@ -322,18 +322,18 @@ DAT_RETURN dapl_ep_modify(DAT_EP_HANDLE DAT_RETURN dat_status; /* Flag indicating we've allocated a new one of these. */ - DAT_BOOLEAN qp_allocated = DAT_FALSE; - DAT_BOOLEAN rqst_cb_allocated = DAT_FALSE; - DAT_BOOLEAN recv_cb_allocated = DAT_FALSE; - DAT_BOOLEAN rqst_iov_allocated = DAT_FALSE; - DAT_BOOLEAN recv_iov_allocated = DAT_FALSE; + boolean_t qp_allocated = FALSE; + boolean_t rqst_cb_allocated = FALSE; + boolean_t recv_cb_allocated = FALSE; + boolean_t rqst_iov_allocated = FALSE; + boolean_t recv_iov_allocated = FALSE; /* Flag indicating we've used (assigned to QP) a new one of these. */ - DAT_BOOLEAN qp_used = DAT_FALSE; - DAT_BOOLEAN rqst_cb_used = DAT_FALSE; - DAT_BOOLEAN recv_cb_used = DAT_FALSE; - DAT_BOOLEAN rqst_iov_used = DAT_FALSE; - DAT_BOOLEAN recv_iov_used = DAT_FALSE; + boolean_t qp_used = FALSE; + boolean_t rqst_cb_used = FALSE; + boolean_t recv_cb_used = FALSE; + boolean_t rqst_iov_used = FALSE; + boolean_t recv_iov_used = FALSE; dat_status = dapl_ep_modify_validate_parameters(ep_handle, mask, @@ -381,14 +381,14 @@ DAT_RETURN dapl_ep_modify(DAT_EP_HANDLE if (DAT_SUCCESS != dat_status) { goto bail; } - rqst_cb_allocated = DAT_TRUE; + rqst_cb_allocated = TRUE; dat_status = dapl_cb_create(&alloc_ep.recv_buffer, ep1, /* For pointer in buffer bool. */ ep_attr1.max_recv_dtos); if (DAT_SUCCESS != dat_status) { goto bail; } - recv_cb_allocated = DAT_TRUE; + recv_cb_allocated = TRUE; alloc_ep.send_iov_num = ep_attr1.max_request_iov; alloc_ep.send_iov = kmalloc(ep_attr1.max_request_iov @@ -399,7 +399,7 @@ DAT_RETURN dapl_ep_modify(DAT_EP_HANDLE DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto bail; } - recv_iov_allocated = DAT_TRUE; + recv_iov_allocated = TRUE; alloc_ep.recv_iov_num = ep_attr1.max_recv_iov; alloc_ep.recv_iov = kmalloc(ep_attr1.max_recv_iov @@ -410,13 +410,13 @@ DAT_RETURN dapl_ep_modify(DAT_EP_HANDLE DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto bail; } - recv_iov_allocated = DAT_TRUE; + recv_iov_allocated = TRUE; dat_status = dapl_ib_qp_alloc(ia, &alloc_ep, ep1); if (dat_status != DAT_SUCCESS) { goto bail; } - qp_allocated = DAT_TRUE; + qp_allocated = TRUE; /* * Now we atomically modify the EP, under lock @@ -473,26 +473,26 @@ DAT_RETURN dapl_ep_modify(DAT_EP_HANDLE && (ep_param->ep_attr.max_recv_dtos != ep2->param.ep_attr.max_recv_dtos)) { new_ep.recv_buffer = alloc_ep.recv_buffer; - recv_cb_used = DAT_TRUE; + recv_cb_used = TRUE; } if (mask & DAT_EP_FIELD_EP_ATTR_MAX_REQUEST_DTOS && (ep_param->ep_attr.max_request_dtos != ep2->param.ep_attr.max_request_dtos)) { new_ep.req_buffer = alloc_ep.req_buffer; - rqst_cb_used = DAT_TRUE; + rqst_cb_used = TRUE; } if (mask & DAT_EP_FIELD_EP_ATTR_MAX_RECV_IOV && new_ep.recv_iov_num != ep2->recv_iov_num) { new_ep.recv_iov = alloc_ep.recv_iov; - recv_iov_used = DAT_TRUE; + recv_iov_used = TRUE; } if (mask & DAT_EP_FIELD_EP_ATTR_MAX_REQUEST_IOV && new_ep.send_iov_num != ep2->send_iov_num) { new_ep.send_iov = alloc_ep.send_iov; - recv_iov_used = DAT_TRUE; + recv_iov_used = TRUE; } /* Index: linux-kernel/dat-provider/dapl_cno_util.c =================================================================== --- linux-kernel/dat-provider/dapl_cno_util.c (revision 2295) +++ linux-kernel/dat-provider/dapl_cno_util.c (working copy) @@ -131,7 +131,7 @@ dapl_cno_trigger ( cno_ptr->cno_upcall.upcall_func ( cno_ptr->cno_upcall.instance_data, &event, - DAT_FALSE); + FALSE); } dat_status = dapl_evd_dequeue ( (DAT_EVD_HANDLE)evd_ptr, &event); } Index: linux-kernel/dat-provider/dapl_timer_util.c =================================================================== --- linux-kernel/dat-provider/dapl_timer_util.c (revision 2295) +++ linux-kernel/dat-provider/dapl_timer_util.c (working copy) @@ -97,7 +97,7 @@ dapl_timer_set(DAPL_OS_TIMER * timer, { DAPL_OS_TIMER *list_ptr; DAPL_OS_TIMEVAL cur_time; - DAT_BOOLEAN wakeup_tmo_thread; + boolean_t wakeup_tmo_thread; /* * Start the timer thread the first time we need a timer @@ -109,7 +109,7 @@ dapl_timer_set(DAPL_OS_TIMER * timer, } dapl_llist_init_entry(&timer->list_entry); - wakeup_tmo_thread = DAT_FALSE; + wakeup_tmo_thread = FALSE; dapl_os_get_time(&cur_time); timer->expires = cur_time + expires; /* calculate future time */ timer->function = func; @@ -132,7 +132,7 @@ dapl_timer_set(DAPL_OS_TIMER * timer, dapl_llist_add_head(&g_daplTimerHead.timer_list_head, (DAPL_LLIST_ENTRY *) & timer->list_entry, timer); - wakeup_tmo_thread = DAT_TRUE; + wakeup_tmo_thread = TRUE; } else { list_ptr = (DAPL_OS_TIMER *) dapl_llist_peek_head(&g_daplTimerHead.timer_list_head); @@ -142,7 +142,7 @@ dapl_timer_set(DAPL_OS_TIMER * timer, dapl_llist_add_head(&g_daplTimerHead.timer_list_head, (DAPL_LLIST_ENTRY *) & timer-> list_entry, timer); - wakeup_tmo_thread = DAT_TRUE; + wakeup_tmo_thread = TRUE; } else { /* Case 3: figure out where entry goes in sorted list */ list_ptr = @@ -182,7 +182,7 @@ dapl_timer_set(DAPL_OS_TIMER * timer, } spin_unlock(&g_daplTimerHead.lock); - if (wakeup_tmo_thread == DAT_TRUE) { + if (wakeup_tmo_thread == TRUE) { dapl_os_wait_object_wakeup(&g_daplTimerHead.wait_object); } Index: linux-kernel/dat-provider/dapl_rsp_create.c =================================================================== --- linux-kernel/dat-provider/dapl_rsp_create.c (revision 2295) +++ linux-kernel/dat-provider/dapl_rsp_create.c (working copy) @@ -78,7 +78,7 @@ dapl_rsp_create(DAT_IA_HANDLE ia_handle, DAPL_SP *sp_ptr; DAPL_EVD *evd_ptr; DAPL_EP *ep_ptr; - DAT_BOOLEAN sp_found; + boolean_t sp_found; DAT_RETURN dat_status; dat_status = DAT_SUCCESS; @@ -123,13 +123,13 @@ dapl_rsp_create(DAT_IA_HANDLE ia_handle, goto bail; } - sp_ptr = dapl_ia_sp_search(ia_ptr, conn_qual, DAT_FALSE); - sp_found = DAT_TRUE; + sp_ptr = dapl_ia_sp_search(ia_ptr, conn_qual, FALSE); + sp_found = TRUE; if (sp_ptr == NULL) { - sp_found = DAT_FALSE; + sp_found = FALSE; /* Allocate RSP */ - sp_ptr = dapl_sp_alloc(ia_ptr, DAT_FALSE); + sp_ptr = dapl_sp_alloc(ia_ptr, FALSE); if (sp_ptr == NULL) { dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, @@ -162,9 +162,9 @@ dapl_rsp_create(DAT_IA_HANDLE ia_handle, * even before this call returns! */ sp_ptr->state = DAPL_SP_STATE_RSP_LISTENING; - sp_ptr->listening = DAT_TRUE; + sp_ptr->listening = TRUE; - if (sp_found == DAT_FALSE) { + if (sp_found == FALSE) { /* Link it onto the IA */ dapl_ia_link_rsp(ia_ptr, sp_ptr); Index: linux-kernel/dat-provider/dapl_ep_util.c =================================================================== --- linux-kernel/dat-provider/dapl_ep_util.c (revision 2295) +++ linux-kernel/dat-provider/dapl_ep_util.c (working copy) @@ -359,7 +359,7 @@ void dapl_ep_timeout(uintptr_t arg) *cm event number. */ ib_cm_event = dapl_ib_get_cm_event(DAT_CONNECTION_EVENT_DISCONNECTED); - dapl_ib_disconnect_clean(ep_ptr, DAT_TRUE, ib_cm_event); + dapl_ib_disconnect_clean(ep_ptr, TRUE, ib_cm_event); (void)dapl_evd_post_connection_event((DAPL_EVD *) ep_ptr->param. connect_evd_handle, Index: linux-kernel/dat-provider/dapl_rmr_util.h =================================================================== --- linux-kernel/dat-provider/dapl_rmr_util.h (revision 2295) +++ linux-kernel/dat-provider/dapl_rmr_util.h (working copy) @@ -49,7 +49,7 @@ extern DAPL_RMR *dapl_rmr_alloc(DAPL_PZ extern void dapl_rmr_dealloc(DAPL_RMR * rmr); -static __inline__ DAT_BOOLEAN +static __inline__ boolean_t dapl_rmr_validate_completion_flag(DAT_COMPLETION_FLAGS mask, DAT_COMPLETION_FLAGS allow, DAT_COMPLETION_FLAGS request); @@ -60,15 +60,15 @@ dapl_rmr_validate_completion_flag(DAT_CO * * *********************************************************************/ -static __inline__ DAT_BOOLEAN +static __inline__ boolean_t dapl_rmr_validate_completion_flag(DAT_COMPLETION_FLAGS mask, DAT_COMPLETION_FLAGS allow, DAT_COMPLETION_FLAGS request) { if ((mask & request) && !(mask & allow)) { - return DAT_FALSE; + return FALSE; } else { - return DAT_TRUE; + return TRUE; } } Index: linux-kernel/dat-provider/dapl_rsp_free.c =================================================================== --- linux-kernel/dat-provider/dapl_rsp_free.c (revision 2295) +++ linux-kernel/dat-provider/dapl_rsp_free.c (working copy) @@ -112,13 +112,13 @@ DAT_RETURN dapl_rsp_free(DAT_RSP_HANDLE * else the last disconnect on this RSP will free it up. The RSP * is used to contain CR records for each connection, which * contain information necessary to disconnect. - * sp_ptr->listening will be DAT_TRUE if there has never been a - * connection event, and DAT_FALSE if a connection attempt resulted + * sp_ptr->listening will be TRUE if there has never been a + * connection event, and FALSE if a connection attempt resulted * in a reject. */ if (sp_ptr->cr_list_count == 0) { /* This RSP has never been used. Clean it up */ - sp_ptr->listening = DAT_FALSE; + sp_ptr->listening = FALSE; sp_ptr->state = DAPL_SP_STATE_FREE; spin_unlock(&sp_ptr->header.lock); Index: linux-kernel/dat-provider/dapl_ia_open.c =================================================================== --- linux-kernel/dat-provider/dapl_ia_open.c (revision 2295) +++ linux-kernel/dat-provider/dapl_ia_open.c (working copy) @@ -47,7 +47,7 @@ * LOCAL PROTOTYPES */ -void dapl_hca_cleanup(DAPL_HCA * hca_ptr, DAT_BOOLEAN dec_ref); +void dapl_hca_cleanup(DAPL_HCA * hca_ptr, boolean_t dec_ref); /* * Open a provider and return a handle. The handle enables the user @@ -124,7 +124,7 @@ dapl_ia_open(const DAT_NAME_PTR name, &hca_ptr->ia_attr, NULL, &hca_ptr->hca_address); if (dat_status != DAT_SUCCESS) { - dapl_hca_cleanup(hca_ptr, DAT_FALSE); + dapl_hca_cleanup(hca_ptr, FALSE); spin_unlock(&hca_ptr->lock); goto bail; } @@ -137,7 +137,7 @@ dapl_ia_open(const DAT_NAME_PTR name, ia_ptr = dapl_ia_alloc(provider, hca_ptr); if (!ia_ptr) { spin_lock(&hca_ptr->lock); - dapl_hca_cleanup(hca_ptr, DAT_TRUE); + dapl_hca_cleanup(hca_ptr, TRUE); spin_unlock(&hca_ptr->lock); dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); @@ -173,7 +173,7 @@ dapl_ia_open(const DAT_NAME_PTR name, goto bail; } - ia_ptr->cleanup_async_error_evd = DAT_FALSE; + ia_ptr->cleanup_async_error_evd = FALSE; ia_ptr->async_error_evd = evd_ptr; } else { /* Verify we have >0 length, and let the provider check the size */ @@ -204,13 +204,13 @@ dapl_ia_open(const DAT_NAME_PTR name, dat_status = dapl_ia_setup_callbacks(ia_ptr, evd_ptr); if (dat_status != DAT_SUCCESS) { /* Assign the EVD so it gets cleaned up */ - ia_ptr->cleanup_async_error_evd = DAT_TRUE; + ia_ptr->cleanup_async_error_evd = TRUE; ia_ptr->async_error_evd = evd_ptr; goto bail; } } - ia_ptr->cleanup_async_error_evd = DAT_TRUE; + ia_ptr->cleanup_async_error_evd = TRUE; ia_ptr->async_error_evd = evd_ptr; } @@ -236,10 +236,10 @@ dapl_ia_open(const DAT_NAME_PTR name, * Clean up partially allocated HCA stuff. Strictly to make cleanup * simple. */ -void dapl_hca_cleanup(DAPL_HCA * hca_ptr, DAT_BOOLEAN dec_ref) +void dapl_hca_cleanup(DAPL_HCA * hca_ptr, boolean_t dec_ref) { dapl_ib_close_hca(hca_ptr); - if (dec_ref == DAT_TRUE) { + if (dec_ref == TRUE) { atomic_dec(&hca_ptr->handle_ref_count); } } Index: linux-kernel/dat-provider/dapl_evd_util.c =================================================================== --- linux-kernel/dat-provider/dapl_evd_util.c (revision 2295) +++ linux-kernel/dat-provider/dapl_evd_util.c (working copy) @@ -163,7 +163,7 @@ DAPL_EVD *dapl_evd_alloc(DAPL_IA * ia_pt evd_ptr->evd_producer_locking_needed = 1; /* Conservative value. */ evd_ptr->ib_cq_handle = IB_INVALID_HANDLE; atomic_set(&evd_ptr->evd_ref_count, 0); - evd_ptr->catastrophic_overflow = DAT_FALSE; + evd_ptr->catastrophic_overflow = FALSE; evd_ptr->qlen = qlen; evd_ptr->cno_ptr = cno_ptr; @@ -476,7 +476,7 @@ dapl_evd_post_overflow_event(DAPL_EVD * * In that case we've got a catastrophic overflow. */ if (async_evd_ptr == overflow_evd_ptr) { - async_evd_ptr->catastrophic_overflow = DAT_TRUE; + async_evd_ptr->catastrophic_overflow = TRUE; async_evd_ptr->evd_state = DAPL_EVD_STATE_DEAD; return; } @@ -484,7 +484,7 @@ dapl_evd_post_overflow_event(DAPL_EVD * overflow_event = dapl_evd_get_event(overflow_evd_ptr); if (!overflow_event) { /* this is not good */ - overflow_evd_ptr->catastrophic_overflow = DAT_TRUE; + overflow_evd_ptr->catastrophic_overflow = TRUE; overflow_evd_ptr->evd_state = DAPL_EVD_STATE_DEAD; return; } Index: linux-kernel/dat-provider/dapl_sp_util.c =================================================================== --- linux-kernel/dat-provider/dapl_sp_util.c (revision 2295) +++ linux-kernel/dat-provider/dapl_sp_util.c (working copy) @@ -59,7 +59,7 @@ * pointer to sp info struct * */ -DAPL_SP *dapl_sp_alloc(DAPL_IA * ia_ptr, DAT_BOOLEAN is_psp) +DAPL_SP *dapl_sp_alloc(DAPL_IA * ia_ptr, boolean_t is_psp) { DAPL_SP *sp_ptr; Index: linux-kernel/dat-provider/dapl_hash.c =================================================================== --- linux-kernel/dat-provider/dapl_hash.c (revision 2295) +++ linux-kernel/dat-provider/dapl_hash.c (working copy) @@ -153,25 +153,24 @@ dapl_hash_rehash(DAPL_HASH_ELEM * elemen * * Inputs: * table, key and datum to be added - * allow_dup - DAT_TRUE if dups are allowed + * allow_dup - TRUE if dups are allowed * Outputs: * report_dup - should you care to know * Returns: - * DAT_TRUE on success + * TRUE on success */ -static DAT_BOOLEAN -dapl_hash_add(DAPL_HASH_TABLEP p_table, - DAPL_HASH_KEY key, - void *datum, DAT_BOOLEAN allow_dup, DAT_BOOLEAN * report_dup) +static boolean_t +dapl_hash_add(DAPL_HASH_TABLEP p_table, DAPL_HASH_KEY key, void *datum, + boolean_t allow_dup, boolean_t *report_dup) { void *olddatum; DAPL_HASH_KEY hashValue, save_key = key; DAPL_HASH_ELEM *found; - DAT_BOOLEAN status = DAT_FALSE; + boolean_t status = FALSE; unsigned int chain_len = 0; if (report_dup) { - (*report_dup) = DAT_FALSE; + (*report_dup) = FALSE; } if (NO_DATUM(datum)) { @@ -181,7 +180,7 @@ dapl_hash_add(DAPL_HASH_TABLEP p_table, dapl_dbg_log(DAPL_DBG_TYPE_ERR, "dapl_hash_add () called with magic NO_DATA value (%p) " "used as datum!\n", datum); - return DAT_FALSE; + return FALSE; } DAPL_HASHLOOKUP(p_table, key, olddatum, found); @@ -190,14 +189,14 @@ dapl_hash_add(DAPL_HASH_TABLEP p_table, * key exists already */ if (report_dup) { - *report_dup = DAT_TRUE; + *report_dup = TRUE; } if (!allow_dup) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, "dapl_hash_add () called with duplicate key (" F64x ")\n", key); - return DAT_FALSE; + return FALSE; } } @@ -211,7 +210,7 @@ dapl_hash_add(DAPL_HASH_TABLEP p_table, p_table->table[hashValue].datum = datum; p_table->table[hashValue].next_element = NULL; p_table->num_entries++; - status = DAT_TRUE; + status = TRUE; } else { DAPL_HASH_ELEM *newelement = (DAPL_HASH_ELEM *) kmalloc(sizeof(DAPL_HASH_ELEM), GFP_ATOMIC); @@ -231,10 +230,10 @@ dapl_hash_add(DAPL_HASH_TABLEP p_table, } lastelement->next_element = newelement; p_table->num_entries++; - status = DAT_TRUE; + status = TRUE; } else { /* allocation failed - should not happen */ - status = DAT_FALSE; + status = FALSE; } } @@ -262,9 +261,9 @@ dapl_hash_add(DAPL_HASH_TABLEP p_table, * Inputs: * element, key to be deleted * Returns: - * DAT_TRUE on success + * TRUE on success */ -static DAT_BOOLEAN +static boolean_t dapl_hash_delete_element(DAPL_HASH_ELEM * element, DAPL_HASH_KEY key, DAPL_HASH_DATA * p_datum) { @@ -309,7 +308,7 @@ dapl_hash_delete_element(DAPL_HASH_ELEM } } - return (curelement != NULL ? DAT_TRUE : DAT_FALSE); + return (curelement != NULL ? TRUE : FALSE); } /********************************************************************* @@ -405,7 +404,7 @@ dapl_hash_insert(DAPL_HASH_TABLE * p_tab dat_status = DAT_SUCCESS; spin_lock(&p_table->lock); - if (!dapl_hash_add(p_table, key, data, DAT_FALSE, NULL)) { + if (!dapl_hash_add(p_table, key, data, FALSE, NULL)) { dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); } Index: linux-kernel/dat-provider/dapl_sp_util.h =================================================================== --- linux-kernel/dat-provider/dapl_sp_util.h (revision 2295) +++ linux-kernel/dat-provider/dapl_sp_util.h (working copy) @@ -37,7 +37,7 @@ #ifndef DAPL_PSP_UTIL_H #define DAPL_PSP_UTIL_H -DAPL_SP *dapl_sp_alloc(DAPL_IA * ia_ptr, DAT_BOOLEAN is_psp); +DAPL_SP *dapl_sp_alloc(DAPL_IA * ia_ptr, boolean_t is_psp); void dapl_sp_free_sp(DAPL_SP * sp_ptr); Index: linux-kernel/dat-provider/dapl.h =================================================================== --- linux-kernel/dat-provider/dapl.h (revision 2295) +++ linux-kernel/dat-provider/dapl.h (working copy) @@ -240,7 +240,7 @@ struct dapl_ia { DAPL_HEADER header; DAPL_HCA *hca_ptr; DAPL_EVD *async_error_evd; - DAT_BOOLEAN cleanup_async_error_evd; + boolean_t cleanup_async_error_evd; DAPL_LLIST_ENTRY hca_ia_list_entry; /* HCAs list of IAs */ DAPL_LLIST_HEAD ep_list_head; /* EP queue */ @@ -269,7 +269,7 @@ struct dapl_evd DAT_EVD_FLAGS evd_flags; /* Derived from evd_flags; see dapls_evd_internal_create. */ - DAT_BOOLEAN evd_producer_locking_needed; + boolean_t evd_producer_locking_needed; /* Every EVD has a CQ unless it is a SOFTWARE_EVENT only EVD */ ib_cq_handle_t ib_cq_handle; @@ -280,7 +280,7 @@ struct dapl_evd atomic_t evd_ref_count; /* Set if there has been a catastrophic overflow */ - DAT_BOOLEAN catastrophic_overflow; + boolean_t catastrophic_overflow; /* the actual events */ DAT_COUNT qlen; @@ -395,7 +395,7 @@ struct dapl_sp { DAT_EP_HANDLE ep_handle; /* maintenence fields */ - DAT_BOOLEAN listening; /* PSP is registered & active */ + boolean_t listening; /* PSP is registered & active */ ib_cm_srvc_handle_t cm_srvc_handle; /* Used by CM */ DAPL_LLIST_HEAD cr_list_head; /* CR pending queue */ DAT_COUNT cr_list_count; /* count of CRs on queue */ @@ -617,8 +617,8 @@ extern DAT_RETURN dapl_ep_post_rdma_writ extern DAT_RETURN dapl_ep_get_status(DAT_EP_HANDLE, /* ep_handle */ DAT_EP_STATE *, /* ep_state */ - DAT_BOOLEAN *, /* in_dto_idle */ - DAT_BOOLEAN *); /* out_dto_idle */ + boolean_t *, /* in_dto_idle */ + boolean_t *); /* out_dto_idle */ extern DAT_RETURN dapl_ep_free(DAT_EP_HANDLE); /* ep_handle */ @@ -757,7 +757,7 @@ extern void dapl_llist_init_head(DAPL_LL extern void dapl_llist_init_entry(DAPL_LLIST_ENTRY * entry); -extern DAT_BOOLEAN dapl_llist_is_empty(DAPL_LLIST_HEAD * head); +extern boolean_t dapl_llist_is_empty(DAPL_LLIST_HEAD * head); extern void dapl_llist_add_head(DAPL_LLIST_HEAD * head, DAPL_LLIST_ENTRY * entry, void *data); Index: linux-kernel/dat-provider/dapl_psp_query.c =================================================================== --- linux-kernel/dat-provider/dapl_psp_query.c (revision 2295) +++ linux-kernel/dat-provider/dapl_psp_query.c (working copy) @@ -45,7 +45,7 @@ DAT_RETURN dapl_psp_query(DAT_PSP_HANDLE DAT_RETURN status; if (DAPL_BAD_HANDLE(psp_handle, DAPL_MAGIC_PSP) || - ((DAPL_SP *) psp_handle)->listening != DAT_TRUE) { + ((DAPL_SP *) psp_handle)->listening != TRUE) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PSP); goto bail; Index: linux-kernel/dat-provider/dapl_llist.c =================================================================== --- linux-kernel/dat-provider/dapl_llist.c (revision 2295) +++ linux-kernel/dat-provider/dapl_llist.c (working copy) @@ -80,7 +80,7 @@ void dapl_llist_init_entry(DAPL_LLIST_EN * * Purpose: returns TRUE if the linked list is empty */ -DAT_BOOLEAN dapl_llist_is_empty(DAPL_LLIST_HEAD * head) +boolean_t dapl_llist_is_empty(DAPL_LLIST_HEAD * head) { return (*head == NULL); } Index: linux-kernel/dat/dictionary.c =================================================================== --- linux-kernel/dat/dictionary.c (revision 2295) +++ linux-kernel/dat/dictionary.c (working copy) @@ -62,7 +62,7 @@ static DAT_RETURN dat_dictionary_key_dup(const struct dat_provider_info *old_key, struct dat_provider_info *new_key); -static DAT_BOOLEAN +static boolean_t dat_dictionary_key_is_equal(const struct dat_provider_info *key_a, const struct dat_provider_info *key_b); @@ -270,7 +270,7 @@ DAT_RETURN dat_dictionary_search(DAT_DIC for (cur_node = p_dictionary->head->next; p_dictionary->tail != cur_node; cur_node = cur_node->next) { - if (DAT_TRUE == dat_dictionary_key_is_equal(&cur_node->key, + if (TRUE == dat_dictionary_key_is_equal(&cur_node->key, key)) { if (NULL != p_data) *p_data = cur_node->data; @@ -335,7 +335,7 @@ DAT_RETURN dat_dictionary_remove(DAT_DIC for (cur_node = p_dictionary->head->next; p_dictionary->tail != cur_node; cur_node = cur_node->next) { - if (DAT_TRUE == dat_dictionary_key_is_equal(&cur_node->key, + if (TRUE == dat_dictionary_key_is_equal(&cur_node->key, key)) { if (NULL != p_data) *p_data = cur_node->data; @@ -387,7 +387,7 @@ DAT_RETURN dat_dictionary_key_dup(const * Function: dat_dictionary_key_is_equal ***********************************************************************/ -DAT_BOOLEAN dat_dictionary_key_is_equal(const struct dat_provider_info *key_a, +boolean_t dat_dictionary_key_is_equal(const struct dat_provider_info *key_a, const struct dat_provider_info *key_b) { BUG_ON(NULL == key_a); @@ -398,8 +398,8 @@ DAT_BOOLEAN dat_dictionary_key_is_equal( && (key_a->dat_version_major == key_b->dat_version_major) && (key_a->dat_version_minor == key_b->dat_version_minor) && (key_a->is_thread_safe == key_b->is_thread_safe)) { - return DAT_TRUE; + return TRUE; } else { - return DAT_FALSE; + return FALSE; } } Index: linux-kernel/dat/core.c =================================================================== --- linux-kernel/dat/core.c (revision 2295) +++ linux-kernel/dat/core.c (working copy) @@ -133,19 +133,19 @@ module_exit(dat_fini); * of handling DAT API calls and false otherwise. */ -DAT_BOOLEAN dat_check_state(void) +boolean_t dat_check_state(void) { - DAT_BOOLEAN status; + boolean_t status; if ( (DAT_MODULE_STATE_UNINITIALIZED == g_state) || (DAT_MODULE_STATE_INITIALIZING == g_state) ) { dat_init(); - status = DAT_TRUE; + status = TRUE; } else if ( (DAT_MODULE_STATE_DEINITIALIZED == g_state) || (DAT_MODULE_STATE_DEINITIALIZING == g_state) ) { - status = DAT_FALSE; + status = FALSE; } else { - status = DAT_TRUE; + status = TRUE; } return status; Index: linux-kernel/dat/api.c =================================================================== --- linux-kernel/dat/api.c (revision 2295) +++ linux-kernel/dat/api.c (working copy) @@ -580,7 +580,7 @@ DAT_RETURN dat_ia_openv(const DAT_NAME_P DAT_EVD_HANDLE *async_event_handle, DAT_IA_HANDLE *ia_handle, u32 dat_major, u32 dat_minor, - DAT_BOOLEAN thread_safety) + boolean_t thread_safety) { DAT_IA_OPEN_FUNC ia_open_func; struct dat_provider_info info; @@ -602,7 +602,7 @@ DAT_RETURN dat_ia_openv(const DAT_NAME_P return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG1); } - if (DAT_FALSE == dat_check_state()) { + if (FALSE == dat_check_state()) { return DAT_ERROR(DAT_INVALID_STATE, 0); } @@ -642,7 +642,7 @@ DAT_RETURN dat_ia_close(DAT_IA_HANDLE ia dat_dbg_print(DAT_DBG_CLASS_CONSUMER_API, "%s(ia:%x, flags:%x)\n", __func__, ia_handle, flags); - if (DAT_FALSE == dat_check_state()) { + if (FALSE == dat_check_state()) { return DAT_ERROR(DAT_INVALID_STATE, 0); } @@ -739,7 +739,7 @@ DAT_RETURN dat_registry_add_provider(con if (DAT_IS_BAD_POINTER(provider_info)) return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); - if (DAT_FALSE == dat_check_state()) + if (FALSE == dat_check_state()) return DAT_ERROR(DAT_INVALID_STATE, 0); return dat_dr_insert(provider_info, provider->ia_open_func); @@ -761,7 +761,7 @@ DAT_RETURN dat_registry_remove_provider( if (DAT_IS_BAD_POINTER(provider)) return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG1); - if (DAT_FALSE == dat_check_state()) + if (FALSE == dat_check_state()) return DAT_ERROR(DAT_INVALID_STATE, 0); return dat_dr_remove(provider_info); @@ -785,7 +785,7 @@ DAT_RETURN dat_registry_list_providers(D __func__, max_to_return, entries_returned, dat_provider_list); - if (DAT_FALSE == dat_check_state()) + if (FALSE == dat_check_state()) return DAT_ERROR(DAT_INVALID_STATE, 0); if ((DAT_IS_BAD_POINTER(entries_returned))) Index: linux-kernel/dat/dat.h =================================================================== --- linux-kernel/dat/dat.h (revision 2295) +++ linux-kernel/dat/dat.h (working copy) @@ -35,7 +35,7 @@ #define DAT_VERSION_MAJOR 1 #define DAT_VERSION_MINOR 2 -#define DAT_THREADSAFE DAT_FALSE +#define DAT_THREADSAFE FALSE /* unsigned longest native to compiler */ typedef unsigned long long DAT_UVERYLONG; @@ -364,10 +364,12 @@ struct dat_named_attr { const char *value; }; -typedef enum dat_boolean { - DAT_FALSE = 0, - DAT_TRUE = 1 -} DAT_BOOLEAN; +typedef enum { +#undef FALSE + FALSE = 0, +#undef TRUE + TRUE = 1 +} boolean_t; typedef union dat_context { DAT_PVOID as_ptr; @@ -930,7 +932,7 @@ typedef enum dat_upcall_policy { } DAT_UPCALL_POLICY; typedef void (*DAT_UPCALL_FUNC) (DAT_PVOID, const struct dat_event *, - DAT_BOOLEAN); + boolean_t); struct dat_upcall_object { DAT_PVOID instance_data; @@ -950,7 +952,7 @@ struct dat_provider_info { char ia_name[DAT_NAME_MAX_LENGTH]; u32 dat_version_major; u32 dat_version_minor; - DAT_BOOLEAN is_thread_safe; + boolean_t is_thread_safe; }; struct dat_evd_param { @@ -1033,8 +1035,8 @@ struct dat_ia_attr { DAT_COUNT max_iov_segments_per_rdma_write; DAT_COUNT max_rdma_read_in; DAT_COUNT max_rdma_read_out; - DAT_BOOLEAN max_rdma_read_per_ep_in_guaranteed; - DAT_BOOLEAN max_rdma_read_per_ep_out_guaranteed; + boolean_t max_rdma_read_per_ep_in_guaranteed; + boolean_t max_rdma_read_per_ep_out_guaranteed; DAT_COUNT num_transport_attr; struct dat_named_attr *transport_attr; DAT_COUNT num_vendor_attr; @@ -1079,21 +1081,21 @@ struct dat_provider_attr { DAT_IOV_OWNERSHIP iov_ownership_on_return; DAT_QOS dat_qos_supported; DAT_COMPLETION_FLAGS completion_flags_supported; - DAT_BOOLEAN is_thread_safe; + boolean_t is_thread_safe; DAT_COUNT max_private_data_size; - DAT_BOOLEAN supports_multipath; + boolean_t supports_multipath; DAT_EP_CREATOR_FOR_PSP ep_creator; DAT_UPCALL_POLICY upcall_policy; u32 optimal_buffer_alignment; - const DAT_BOOLEAN evd_stream_merging_supported[6][6]; - DAT_BOOLEAN srq_supported; + const boolean_t evd_stream_merging_supported[6][6]; + boolean_t srq_supported; DAT_COUNT srq_watermarks_supported; - DAT_BOOLEAN srq_ep_pz_difference_supported; + boolean_t srq_ep_pz_difference_supported; DAT_COUNT srq_info_supported; DAT_COUNT ep_recv_info_supported; - DAT_BOOLEAN lmr_sync_req; - DAT_BOOLEAN dto_async_return_guaranteed; - DAT_BOOLEAN rdma_write_for_rdma_read_req; + boolean_t lmr_sync_req; + boolean_t dto_async_return_guaranteed; + boolean_t rdma_write_for_rdma_read_req; DAT_COUNT num_provider_specific_attr; struct dat_named_attr *provider_specific_attr; }; @@ -1110,7 +1112,7 @@ typedef DAT_RETURN (*DAT_IA_OPEN_FUNC) ( typedef DAT_RETURN (*DAT_IA_OPENV_FUNC) ( const DAT_NAME_PTR, DAT_COUNT, DAT_EVD_HANDLE *, DAT_IA_HANDLE *, - u32, u32, DAT_BOOLEAN); + u32, u32, boolean_t); typedef DAT_RETURN (*DAT_IA_CLOSE_FUNC) (DAT_IA_HANDLE, DAT_CLOSE_FLAGS); @@ -1185,7 +1187,7 @@ typedef DAT_RETURN (*DAT_EP_POST_RDMA_WR const struct dat_rmr_triplet *, DAT_COMPLETION_FLAGS); typedef DAT_RETURN (*DAT_EP_GET_STATUS_FUNC) ( - DAT_EP_HANDLE, DAT_EP_STATE *, DAT_BOOLEAN *, DAT_BOOLEAN *); + DAT_EP_HANDLE, DAT_EP_STATE *, boolean_t *, boolean_t *); typedef DAT_RETURN (*DAT_EP_FREE_FUNC) (DAT_EP_HANDLE); @@ -1381,7 +1383,7 @@ extern DAT_RETURN dat_registry_remove_pr */ extern DAT_RETURN dat_ia_openv(const DAT_NAME_PTR, DAT_COUNT, DAT_EVD_HANDLE *, DAT_IA_HANDLE *, - u32, u32, DAT_BOOLEAN); + u32, u32, boolean_t); #define dat_ia_open(name, qlen, async_evd, ia) \ dat_ia_openv((name), (qlen), (async_evd), (ia), \ @@ -1581,8 +1583,8 @@ static inline DAT_RETURN dat_ep_free(DAT static inline DAT_RETURN dat_ep_get_status(DAT_EP_HANDLE ep, DAT_EP_STATE * state, - DAT_BOOLEAN * recv_idle, - DAT_BOOLEAN * req_idle) + boolean_t *recv_idle, + boolean_t *req_idle) { return DAT_CALL_PROVIDER_FUNC( ep_get_status_func, ep, state, recv_idle, req_idle); Index: linux-kernel/dat/core.h =================================================================== --- linux-kernel/dat/core.h (revision 2295) +++ linux-kernel/dat/core.h (working copy) @@ -65,6 +65,6 @@ typedef int DAT_DBG_CLASS; extern void dat_dbg_print(DAT_DBG_CLASS val, const char *fmt, ...); -extern DAT_BOOLEAN dat_check_state(void); +extern boolean_t dat_check_state(void); #endif /* CORE_H */ From halr at voltaire.com Tue May 10 10:54:55 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 10 May 2005 13:54:55 -0400 Subject: [openib-general] HCA Firmware In-Reply-To: <52acn3xagn.fsf@topspin.com> References: <1AC79F16F5C5284499BB9591B33D6F00045BCC37@orsmsx408> <52acn3xagn.fsf@topspin.com> Message-ID: <1115747549.4493.1024.camel@localhost.localdomain> On Tue, 2005-05-10 at 13:22, Roland Dreier wrote: > Just for completeness, for PCI-X HCAs: > . 3.3.2 is the first firmware release with working MSI-X > . No released versions of firmware will work on systems with 64K pages. > (This is a possible config for ia64 kernels at least) Time for a separate OpenIB firmware readme ? -- Hal From iod00d at hp.com Tue May 10 11:05:10 2005 From: iod00d at hp.com (Grant Grundler) Date: Tue, 10 May 2005 11:05:10 -0700 Subject: [openib-general] HCA Firmware In-Reply-To: <52acn3xagn.fsf@topspin.com> References: <1AC79F16F5C5284499BB9591B33D6F00045BCC37@orsmsx408> <52acn3xagn.fsf@topspin.com> Message-ID: <20050510180510.GK5495@esmail.cup.hp.com> On Tue, May 10, 2005 at 10:22:16AM -0700, Roland Dreier wrote: > Just for completeness, for PCI-X HCAs: > . 3.3.2 is the first firmware release with working MSI-X > . No released versions of firmware will work on systems with 64K pages. > (This is a possible config for ia64 kernels at least) Yes, but default is 16K - so most folks will be using that. AFAIK, only SGI/SuSE kernels use 64K by default. grant From roland at topspin.com Tue May 10 10:58:18 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 10 May 2005 10:58:18 -0700 Subject: [openib-general] HCA Firmware In-Reply-To: <1115747549.4493.1024.camel@localhost.localdomain> (Hal Rosenstock's message of "10 May 2005 13:54:55 -0400") References: <1AC79F16F5C5284499BB9591B33D6F00045BCC37@orsmsx408> <52acn3xagn.fsf@topspin.com> <1115747549.4493.1024.camel@localhost.localdomain> Message-ID: <52u0lbvu85.fsf@topspin.com> Hal> Time for a separate OpenIB firmware readme ? Probably time for an openib.org wiki. - R. From iod00d at hp.com Tue May 10 11:20:58 2005 From: iod00d at hp.com (Grant Grundler) Date: Tue, 10 May 2005 11:20:58 -0700 Subject: [openib-general] [KDAPL] module initialization order (was Re: [PATCH] kDAPL: remove typedef DAT_PROVIDER) In-Reply-To: References: <1115424198.19661.11.camel@duffman> <1115662195.9691.7.camel@duffman> Message-ID: <20050510182058.GL5495@esmail.cup.hp.com> On Tue, May 10, 2005 at 01:20:49PM -0400, James Lentini wrote: > > After reading through the kernel sources, I believe that builtin > module initialization functions are called by the do_initcalls > function in init/main.c. correct > I haven't conclusively determined how the initialization order is > specified. From what I've gleamed from mailing list archives, the > module initialization order was determined by link order. There are several "classes" of init modules that dictate a coarse level of ordering inside the list of init routines. See the list of *_initcall() macros in include/linux/init.h. > As a result > the relative position of the obj-$(CONFIG_XXX) lines in Makefiles > determined the order of module initialization. Often when I found > discussion of this topic, it was in the context of changing this > scheme to make the initialization dependencies explicit. However, it > does not appear that this was ever done. > > One final piece of information that I came across is that the order of > initialization functions in the System.map file is the order that they > will be called in. yup - but it's very arch specific and compile option specific as to what the final order will be. If there is a strong dependency within a class, one can fix this by directly calling the init routine of the module is needed by another module. A (bad) example might be suckyio^Wsuperio support on parisc platforms. > Can anyone confirm any of the information above? hth, grant From roland at topspin.com Tue May 10 11:17:32 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 10 May 2005 11:17:32 -0700 Subject: [openib-general] HCA Firmware In-Reply-To: <52u0lbvu85.fsf@topspin.com> (Roland Dreier's message of "Tue, 10 May 2005 10:58:18 -0700") References: <1AC79F16F5C5284499BB9591B33D6F00045BCC37@orsmsx408> <52acn3xagn.fsf@topspin.com> <1115747549.4493.1024.camel@localhost.localdomain> <52u0lbvu85.fsf@topspin.com> Message-ID: <52psvzvtc3.fsf@topspin.com> Roland> Probably time for an openib.org wiki. OK, I started a couple minimal pages, including https://openib.org/tiki/tiki-index.php?page=MellanoxHcaFirmware - R. From halr at voltaire.com Tue May 10 11:40:12 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 10 May 2005 14:40:12 -0400 Subject: [openib-general] HCA Firmware In-Reply-To: <52psvzvtc3.fsf@topspin.com> References: <1AC79F16F5C5284499BB9591B33D6F00045BCC37@orsmsx408> <52acn3xagn.fsf@topspin.com> <1115747549.4493.1024.camel@localhost.localdomain> <52u0lbvu85.fsf@topspin.com> <52psvzvtc3.fsf@topspin.com> Message-ID: <1115750113.4493.1067.camel@localhost.localdomain> On Tue, 2005-05-10 at 14:17, Roland Dreier wrote: > Roland> Probably time for an openib.org wiki. > > OK, I started a couple minimal pages, including > https://openib.org/tiki/tiki-index.php?page=MellanoxHcaFirmware Nice. I'll update the IPoIB FAQ to point to this or perhaps that should tuned in a wiki page and eliminated. -- Hal From roland at topspin.com Tue May 10 11:57:26 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 10 May 2005 11:57:26 -0700 Subject: [openib-general] Re: [OOPS] user-mode verbs on IPF In-Reply-To: <1AC79F16F5C5284499BB9591B33D6F00045BCC98@orsmsx408> (Robert J. Woodruff's message of "Tue, 10 May 2005 10:31:48 -0700") References: <1AC79F16F5C5284499BB9591B33D6F00045BCC98@orsmsx408> Message-ID: <52d5ryx621.fsf@topspin.com> I'm pretty sure this is yet another use-after-free bug. I finally got around to turning on CONFIG_DEBUG_SLAB as I promised, and that results in an instant crash in ib_uverbs_event_release() derefencing a pointer value of 0x6b6b6b6b6b6b6b6b (the slab use-after-free poison value). I think I understand the bug, I just need to figure out the right way to fix it. - R. From itamar at mellanox.co.il Tue May 10 12:44:24 2005 From: itamar at mellanox.co.il (Itamar) Date: Tue, 10 May 2005 22:44:24 +0300 Subject: [openib-general] [PATCH][kdapl] clean up async evd flow and init hca flow Message-ID: <20050510194424.GA28388@mellanox.co.il> clean up async evd flow. clean up init hca flow. Signed-off-by: Itamar Rabenstein Index: dapl_module.c =================================================================== --- dapl_module.c (revision 2296) +++ dapl_module.c (working copy) @@ -147,6 +147,28 @@ goto bail; } + dat_status = dapl_ib_open_hca(hca_ptr->name, hca_ptr); + + if (dat_status != DAT_SUCCESS) { + dapl_dbg_log(DAPL_DBG_TYPE_ERR, + "dapl_ib_open_hca failed %x\n", + dat_status); + goto bail; + } + /* + * Obtain IA attributes from the HCA to limit certain + * operations. + */ + dat_status = dapl_ib_query_hca(hca_ptr, + &hca_ptr->ia_attr, + NULL, &hca_ptr->hca_address); + if (dat_status != DAT_SUCCESS) { + dapl_dbg_log(DAPL_DBG_TYPE_ERR, + "dapl_ib_query_hca failed %x\n", + dat_status); + goto bail; + } + provider->extension = hca_ptr; /* register providers with dat_registry */ Index: dapl_ia_util.c =================================================================== --- dapl_ia_util.c (revision 2296) +++ dapl_ia_util.c (working copy) @@ -468,13 +468,6 @@ { spin_lock(&hca_ptr->lock); atomic_dec(&hca_ptr->handle_ref_count); - if (atomic_read(&hca_ptr->handle_ref_count) == 0) { - dapl_ib_close_hca(hca_ptr); -#if 0 - hca_ptr->ib_hca_handle = IB_INVALID_HANDLE; -#endif - hca_ptr->async_evd = NULL; - } spin_unlock(&hca_ptr->lock); } Index: dapl_ia_open.c =================================================================== --- dapl_ia_open.c (revision 2296) +++ dapl_ia_open.c (working copy) @@ -44,12 +44,6 @@ #include "dapl_adapter_util.h" /* - * LOCAL PROTOTYPES - */ - -void dapl_hca_cleanup(DAPL_HCA * hca_ptr, DAT_BOOLEAN dec_ref); - -/* * Open a provider and return a handle. The handle enables the user * to invoke operations on this provider. * @@ -59,9 +53,9 @@ */ DAT_RETURN dapl_ia_open(const DAT_NAME_PTR name, - DAT_COUNT async_evd_qlen, - DAT_EVD_HANDLE * async_evd_handle_ptr, - DAT_IA_HANDLE * ia_handle_ptr) + DAT_COUNT async_evd_qlen, + DAT_EVD_HANDLE * async_evd_handle_ptr, + DAT_IA_HANDLE * ia_handle_ptr) { DAT_RETURN dat_status; struct dat_provider *provider; @@ -74,8 +68,8 @@ ia_ptr = NULL; dapl_dbg_log(DAPL_DBG_TYPE_API, - "dapl_ia_open (%s, %d, %p, %p)\n", - name, async_evd_qlen, async_evd_handle_ptr, ia_handle_ptr); + "dapl_ia_open (%s, %d, %p, %p)\n", + name, async_evd_qlen, async_evd_handle_ptr, ia_handle_ptr); dat_status = dapl_provider_list_search(name, &provider); if (DAT_SUCCESS != dat_status) { @@ -99,37 +93,9 @@ /* get the hca_ptr */ hca_ptr = (DAPL_HCA *) provider->extension; - /* - * Open the HCA if it has not been done before. - */ - spin_lock(&hca_ptr->lock); - /* register with the HW */ - dat_status = dapl_ib_open_hca(hca_ptr->name, hca_ptr); - if (dat_status != DAT_SUCCESS) { - dapl_dbg_log(DAPL_DBG_TYPE_ERR, - "dapl_ib_open_hca failed %x\n", - dat_status); - spin_unlock(&hca_ptr->lock); - goto bail; - } - - /* - * Obtain IA attributes from the HCA to limit certain - * operations. - * If using DAPL_ATS naming, ib_query_hca will also set the ip - * address. - */ - dat_status = dapl_ib_query_hca(hca_ptr, - &hca_ptr->ia_attr, - NULL, &hca_ptr->hca_address); - if (dat_status != DAT_SUCCESS) { - dapl_hca_cleanup(hca_ptr, DAT_FALSE); - spin_unlock(&hca_ptr->lock); - goto bail; - } - /* Take a reference on the hca_handle */ + spin_lock(&hca_ptr->lock); atomic_inc(&hca_ptr->handle_ref_count); spin_unlock(&hca_ptr->lock); @@ -137,10 +103,10 @@ ia_ptr = dapl_ia_alloc(provider, hca_ptr); if (!ia_ptr) { spin_lock(&hca_ptr->lock); - dapl_hca_cleanup(hca_ptr, DAT_TRUE); + atomic_dec(&hca_ptr->handle_ref_count); spin_unlock(&hca_ptr->lock); dat_status = - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); + DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto bail; } @@ -151,74 +117,52 @@ evd_ptr = (DAPL_EVD *) * async_evd_handle_ptr; if (evd_ptr) { if (DAPL_BAD_HANDLE(evd_ptr, DAPL_MAGIC_EVD) || - !(evd_ptr->evd_flags & DAT_EVD_ASYNC_FLAG)) { + !(evd_ptr->evd_flags & DAT_EVD_ASYNC_FLAG)) { dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, - DAT_INVALID_HANDLE_EVD_ASYNC); + DAT_ERROR(DAT_INVALID_HANDLE, + DAT_INVALID_HANDLE_EVD_ASYNC); goto bail; } - - /* InfiniBand allows only 1 asychronous event handler per HCA */ - /* (see InfiniBand Spec, release 1.1, vol I, section 11.5.2, */ - /* page 559). */ - /* */ - /* We only need to make sure that this EVD's CQ belongs to */ - /* the same HCA as is being opened. */ - if (evd_ptr->header.owner_ia->hca_ptr->ib_hca_handle != - hca_ptr->ib_hca_handle) { + hca_ptr->ib_hca_handle) { dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, - DAT_INVALID_HANDLE_EVD_ASYNC); + DAT_ERROR(DAT_INVALID_HANDLE, + DAT_INVALID_HANDLE_EVD_ASYNC); goto bail; } - ia_ptr->cleanup_async_error_evd = DAT_FALSE; ia_ptr->async_error_evd = evd_ptr; } else { /* Verify we have >0 length, and let the provider check the size */ if (async_evd_qlen <= 0) { dat_status = - DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); + DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); goto bail; } dat_status = dapl_evd_internal_create(ia_ptr, - NULL, /* CNO ptr */ - async_evd_qlen, - DAT_EVD_ASYNC_FLAG, - &evd_ptr); + NULL, /* CNO ptr */ + async_evd_qlen, + DAT_EVD_ASYNC_FLAG, + &evd_ptr); if (dat_status != DAT_SUCCESS) { goto bail; } - atomic_inc(&evd_ptr->evd_ref_count); - - spin_lock(&hca_ptr->lock); - if (hca_ptr->async_evd != (DAPL_EVD *) 0) { - spin_unlock(&hca_ptr->lock); - } else { - hca_ptr->async_evd = evd_ptr; - spin_unlock(&hca_ptr->lock); - - /* Register the handlers associated with the async EVD. */ - dat_status = dapl_ia_setup_callbacks(ia_ptr, evd_ptr); - if (dat_status != DAT_SUCCESS) { - /* Assign the EVD so it gets cleaned up */ - ia_ptr->cleanup_async_error_evd = DAT_TRUE; - ia_ptr->async_error_evd = evd_ptr; - goto bail; - } - } - + /* Register the handlers associated with the async EVD. */ + dat_status = dapl_ia_setup_callbacks(ia_ptr, evd_ptr); + /* Assign the EVD so it gets cleaned up */ ia_ptr->cleanup_async_error_evd = DAT_TRUE; ia_ptr->async_error_evd = evd_ptr; + if (dat_status != DAT_SUCCESS) { + goto bail; + } } dat_status = DAT_SUCCESS; *ia_handle_ptr = ia_ptr; *async_evd_handle_ptr = evd_ptr; - bail: + bail: if (dat_status != DAT_SUCCESS) { if (ia_ptr) { /* This will release the async EVD if needed. */ @@ -227,19 +171,8 @@ } dapl_dbg_log(DAPL_DBG_TYPE_RTN, - "dapl_ia_open () returns 0x%x\n", dat_status); + "dapl_ia_open () returns 0x%x\n", dat_status); return dat_status; } -/* - * Clean up partially allocated HCA stuff. Strictly to make cleanup - * simple. - */ -void dapl_hca_cleanup(DAPL_HCA * hca_ptr, DAT_BOOLEAN dec_ref) -{ - dapl_ib_close_hca(hca_ptr); - if (dec_ref == DAT_TRUE) { - atomic_dec(&hca_ptr->handle_ref_count); - } -} Index: dapl.h =================================================================== --- dapl.h (revision 2296) +++ dapl.h (working copy) @@ -211,8 +211,6 @@ spinlock_t lock; DAPL_LLIST_HEAD ia_list_head; /* list of all open IAs */ atomic_t handle_ref_count; /* count of ia_opens on handle */ - DAPL_EVD *async_evd; - DAPL_EVD *async_error_evd; struct sockaddr_in6 hca_address; /* local address of HCA */ char *name; /* provider name */ ib_hca_handle_t ib_hca_handle; -- Itamar From bjordan.ics at gmail.com Tue May 10 12:57:50 2005 From: bjordan.ics at gmail.com (William Jordan) Date: Tue, 10 May 2005 15:57:50 -0400 Subject: [openib-general] [PATCH] CM: add invalid service ID handling Message-ID: <78d18e205051012572aec71c9@mail.gmail.com> Patch to respond to invalid REQs (no listener) with a REJ message. Signed-off-by: Bill Jordan Index: cm.c =================================================================== --- cm.c (revision 2293) +++ cm.c (working copy) @@ -70,6 +70,7 @@ static struct ib_cm { struct rb_root remote_sidr_table; struct idr local_id_table; struct workqueue_struct *wq; + struct ib_cm_id *reject_handler; } cm; struct cm_port { @@ -1049,10 +1050,7 @@ static int cm_req_handler(struct cm_work /* Find matching listen request. */ listen_cm_id_priv = cm_find_listen(req_msg->service_id); if (!listen_cm_id_priv) { - spin_unlock_irqrestore(&cm.lock, flags); - /* todo: reject with no match */ - ret = -EINVAL; - goto error1; + listen_cm_id_priv = container_of(cm.reject_handler, struct cm_id_private, id); } atomic_inc(&listen_cm_id_priv->refcount); atomic_inc(&cm_id_priv->refcount); @@ -3024,6 +3022,12 @@ static void cm_remove_one(struct ib_devi kfree(cm_dev); } +static int cm_reject_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event) +{ + ib_send_cm_rej(cm_id, IB_CM_REJ_INVALID_SERVICE_ID, NULL, 0, NULL, 0); + return -1; +} + static int __init ib_cm_init(void) { int ret; @@ -3043,14 +3047,32 @@ static int __init ib_cm_init(void) if (!cm.wq) return -ENOMEM; + cm.reject_handler = ib_create_cm_id(cm_reject_handler, NULL); + if (IS_ERR(cm.reject_handler)) { + ret = PTR_ERR(cm.reject_handler); + goto error2; + } + cm.reject_handler->state = IB_CM_LISTEN; + cm.reject_handler->service_id = 0; + cm.reject_handler->service_mask = ~0ULL; + ret = ib_register_client(&cm_client); - if (ret) - destroy_workqueue(cm.wq); + if (ret) goto error1; + + return 0; +error1: + cm.reject_handler->state = IB_CM_IDLE; + ib_destroy_cm_id(cm.reject_handler); +error2: + destroy_workqueue(cm.wq); return ret; } static void __exit ib_cm_cleanup(void) { + cm.reject_handler->state = IB_CM_IDLE; + ib_destroy_cm_id(cm.reject_handler); + flush_workqueue(cm.wq); destroy_workqueue(cm.wq); ib_unregister_client(&cm_client); From tduffy at sun.com Tue May 10 13:15:46 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 10 May 2005 13:15:46 -0700 Subject: [openib-general] [PATCH] CM: add invalid service ID handling In-Reply-To: <78d18e205051012572aec71c9@mail.gmail.com> References: <78d18e205051012572aec71c9@mail.gmail.com> Message-ID: <1115756146.12700.0.camel@duffman> On Tue, 2005-05-10 at 15:57 -0400, William Jordan wrote: > Patch to respond to invalid REQs (no listener) with a REJ message. [tduffy at duffman core]$ patch -p0 < /tmp/cm-rej.patch patching file cm.c patch: **** malformed patch at line 94: cm_id_private, id); Your mailer wrapped lines. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From roland at topspin.com Tue May 10 13:26:01 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 10 May 2005 13:26:01 -0700 Subject: [openib-general] Re: [OOPS] user-mode verbs on IPF In-Reply-To: <1AC79F16F5C5284499BB9591B33D6F00045BCC98@orsmsx408> (Robert J. Woodruff's message of "Tue, 10 May 2005 10:31:48 -0700") References: <1AC79F16F5C5284499BB9591B33D6F00045BCC98@orsmsx408> Message-ID: <52oebivndy.fsf@topspin.com> OK, I think this should be fixed in the latest svn. The crash was 100% reproducible (happened every time) with CONFIG_DEBUG_SLAB turned on, and it's now fixed on my setup. It might be interesting to run your MPI tests with CONFIG_DEBUG_SLAB=y to see if it shakes out any other problems. - R. From robert.j.woodruff at intel.com Tue May 10 13:28:56 2005 From: robert.j.woodruff at intel.com (Woodruff, Robert J) Date: Tue, 10 May 2005 13:28:56 -0700 Subject: [openib-general] RE: [OOPS] user-mode verbs on IPF Message-ID: <1AC79F16F5C5284499BB9591B33D6F00045BD087@orsmsx408> Roland wrote >OK, I think this should be fixed in the latest svn. The crash was >100% reproducible (happened every time) with CONFIG_DEBUG_SLAB turned >on, and it's now fixed on my setup. >It might be interesting to run your MPI tests with CONFIG_DEBUG_SLAB=y >to see if it shakes out any other problems. > - R. Ok, thanks for the quick response. I will do a fresh pull of SVN and will set CONFIG_DEBUG_SLAB=y to see if it shakes out any other issues. woody From jlentini at netapp.com Tue May 10 13:42:48 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 10 May 2005 16:42:48 -0400 (EDT) Subject: [openib-general] [KDAPL] module initialization order (was Re: [PATCH] kDAPL: remove typedef DAT_PROVIDER) In-Reply-To: <20050510182058.GL5495@esmail.cup.hp.com> References: <1115424198.19661.11.camel@duffman> <1115662195.9691.7.camel@duffman> <20050510182058.GL5495@esmail.cup.hp.com> Message-ID: On Tue, 10 May 2005, Grant Grundler wrote: > On Tue, May 10, 2005 at 01:20:49PM -0400, James Lentini wrote: >> >> After reading through the kernel sources, I believe that builtin >> module initialization functions are called by the do_initcalls >> function in init/main.c. > > correct > >> I haven't conclusively determined how the initialization order is >> specified. From what I've gleamed from mailing list archives, the >> module initialization order was determined by link order. > > There are several "classes" of init modules that dictate > a coarse level of ordering inside the list of init routines. > See the list of *_initcall() macros in include/linux/init.h. Are the *_initcall() macros intended for code that can be compiled both as part of the kernel or as a loadable module? I see module_init being used in these instances. >> As a result the relative position of the obj-$(CONFIG_XXX) lines in >> Makefiles determined the order of module initialization. Often when >> I found discussion of this topic, it was in the context of changing >> this scheme to make the initialization dependencies explicit. >> However, it does not appear that this was ever done. >> >> One final piece of information that I came across is that the order of >> initialization functions in the System.map file is the order that they >> will be called in. > > yup - but it's very arch specific and compile option specific > as to what the final order will be. If there is a strong dependency > within a class, one can fix this by directly calling the init routine > of the module is needed by another module. A (bad) example might be > suckyio^Wsuperio support on parisc platforms. > >> Can anyone confirm any of the information above? > > hth, > grant > From tduffy at sun.com Tue May 10 13:49:20 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 10 May 2005 13:49:20 -0700 Subject: [openib-general] [PATCH][kdapl] clean up async evd flow and init hca flow In-Reply-To: <20050510194424.GA28388@mellanox.co.il> References: <20050510194424.GA28388@mellanox.co.il> Message-ID: <1115758160.16713.8.camel@duffman> This patch has a lot of wonky whitespace issues. On Tue, 2005-05-10 at 22:44 +0300, Itamar wrote: > clean up async evd flow. > clean up init hca flow. > Index: dapl_ia_open.c > =================================================================== > --- dapl_ia_open.c (revision 2296) > +++ dapl_ia_open.c (working copy) > @@ -44,12 +44,6 @@ > #include "dapl_adapter_util.h" > > /* > - * LOCAL PROTOTYPES > - */ > - > -void dapl_hca_cleanup(DAPL_HCA * hca_ptr, DAT_BOOLEAN dec_ref); > - > -/* > * Open a provider and return a handle. The handle enables the user > * to invoke operations on this provider. > * > @@ -59,9 +53,9 @@ > */ > DAT_RETURN > dapl_ia_open(const DAT_NAME_PTR name, > - DAT_COUNT async_evd_qlen, > - DAT_EVD_HANDLE * async_evd_handle_ptr, > - DAT_IA_HANDLE * ia_handle_ptr) > + DAT_COUNT async_evd_qlen, > + DAT_EVD_HANDLE * async_evd_handle_ptr, > + DAT_IA_HANDLE * ia_handle_ptr) Bogus whitespace cleanup. > DAT_RETURN dat_status; > struct dat_provider *provider; > @@ -74,8 +68,8 @@ > ia_ptr = NULL; > > dapl_dbg_log(DAPL_DBG_TYPE_API, > - "dapl_ia_open (%s, %d, %p, %p)\n", > - name, async_evd_qlen, async_evd_handle_ptr, ia_handle_ptr); > + "dapl_ia_open (%s, %d, %p, %p)\n", > + name, async_evd_qlen, async_evd_handle_ptr, ia_handle_ptr); same here > dat_status = dapl_provider_list_search(name, &provider); > if (DAT_SUCCESS != dat_status) { > @@ -99,37 +93,9 @@ > /* get the hca_ptr */ > hca_ptr = (DAPL_HCA *) provider->extension; > > - /* > - * Open the HCA if it has not been done before. > - */ > - spin_lock(&hca_ptr->lock); > - /* register with the HW */ > - dat_status = dapl_ib_open_hca(hca_ptr->name, hca_ptr); > > - if (dat_status != DAT_SUCCESS) { > - dapl_dbg_log(DAPL_DBG_TYPE_ERR, > - "dapl_ib_open_hca failed %x\n", > - dat_status); > - spin_unlock(&hca_ptr->lock); > - goto bail; > - } > - > - /* > - * Obtain IA attributes from the HCA to limit certain > - * operations. > - * If using DAPL_ATS naming, ib_query_hca will also set the ip > - * address. > - */ > - dat_status = dapl_ib_query_hca(hca_ptr, > - &hca_ptr->ia_attr, > - NULL, &hca_ptr->hca_address); > - if (dat_status != DAT_SUCCESS) { > - dapl_hca_cleanup(hca_ptr, DAT_FALSE); > - spin_unlock(&hca_ptr->lock); > - goto bail; > - } > - > /* Take a reference on the hca_handle */ > + spin_lock(&hca_ptr->lock); > atomic_inc(&hca_ptr->handle_ref_count); > spin_unlock(&hca_ptr->lock); > > @@ -137,10 +103,10 @@ > ia_ptr = dapl_ia_alloc(provider, hca_ptr); > if (!ia_ptr) { > spin_lock(&hca_ptr->lock); > - dapl_hca_cleanup(hca_ptr, DAT_TRUE); > + atomic_dec(&hca_ptr->handle_ref_count); > spin_unlock(&hca_ptr->lock); > dat_status = > - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); > + DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); again > } > > @@ -151,74 +117,52 @@ > evd_ptr = (DAPL_EVD *) * async_evd_handle_ptr; > if (evd_ptr) { > if (DAPL_BAD_HANDLE(evd_ptr, DAPL_MAGIC_EVD) || > - !(evd_ptr->evd_flags & DAT_EVD_ASYNC_FLAG)) { > + !(evd_ptr->evd_flags & DAT_EVD_ASYNC_FLAG)) { > dat_status = > - DAT_ERROR(DAT_INVALID_HANDLE, > - DAT_INVALID_HANDLE_EVD_ASYNC); > + DAT_ERROR(DAT_INVALID_HANDLE, > + DAT_INVALID_HANDLE_EVD_ASYNC); > goto bail; > } > - > - /* InfiniBand allows only 1 asychronous event handler per HCA */ > - /* (see InfiniBand Spec, release 1.1, vol I, section 11.5.2, */ > - /* page 559). */ > - /* */ > - /* We only need to make sure that this EVD's CQ belongs to */ > - /* the same HCA as is being opened. */ > - > if (evd_ptr->header.owner_ia->hca_ptr->ib_hca_handle != > - hca_ptr->ib_hca_handle) { > + hca_ptr->ib_hca_handle) { again > dat_status = > - DAT_ERROR(DAT_INVALID_HANDLE, > - DAT_INVALID_HANDLE_EVD_ASYNC); > + DAT_ERROR(DAT_INVALID_HANDLE, > + DAT_INVALID_HANDLE_EVD_ASYNC); and more > goto bail; > } > - > ia_ptr->cleanup_async_error_evd = DAT_FALSE; > ia_ptr->async_error_evd = evd_ptr; > } else { > /* Verify we have >0 length, and let the provider check the size */ > if (async_evd_qlen <= 0) { > dat_status = > - DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); > + DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); more > goto bail; > } > dat_status = dapl_evd_internal_create(ia_ptr, > - NULL, /* CNO ptr */ > - async_evd_qlen, > - DAT_EVD_ASYNC_FLAG, > - &evd_ptr); > + NULL, /* CNO ptr */ > + async_evd_qlen, > + DAT_EVD_ASYNC_FLAG, > + &evd_ptr); whoa nelly! > if (dat_status != DAT_SUCCESS) { > goto bail; > } > - > atomic_inc(&evd_ptr->evd_ref_count); > - > - spin_lock(&hca_ptr->lock); > - if (hca_ptr->async_evd != (DAPL_EVD *) 0) { > - spin_unlock(&hca_ptr->lock); > - } else { > - hca_ptr->async_evd = evd_ptr; > - spin_unlock(&hca_ptr->lock); > - > - /* Register the handlers associated with the async EVD. */ > - dat_status = dapl_ia_setup_callbacks(ia_ptr, evd_ptr); > - if (dat_status != DAT_SUCCESS) { > - /* Assign the EVD so it gets cleaned up */ > - ia_ptr->cleanup_async_error_evd = DAT_TRUE; > - ia_ptr->async_error_evd = evd_ptr; > - goto bail; > - } > - } > - > + /* Register the handlers associated with the async EVD. */ > + dat_status = dapl_ia_setup_callbacks(ia_ptr, evd_ptr); > + /* Assign the EVD so it gets cleaned up */ > ia_ptr->cleanup_async_error_evd = DAT_TRUE; > ia_ptr->async_error_evd = evd_ptr; > + if (dat_status != DAT_SUCCESS) { > + goto bail; > + } > } > > dat_status = DAT_SUCCESS; > *ia_handle_ptr = ia_ptr; > *async_evd_handle_ptr = evd_ptr; > > - bail: > + bail: ... > if (dat_status != DAT_SUCCESS) { > if (ia_ptr) { > /* This will release the async EVD if needed. */ > @@ -227,19 +171,8 @@ > } > > dapl_dbg_log(DAPL_DBG_TYPE_RTN, > - "dapl_ia_open () returns 0x%x\n", dat_status); > + "dapl_ia_open () returns 0x%x\n", dat_status); you get the idea. > > return dat_status; > } > > -/* > - * Clean up partially allocated HCA stuff. Strictly to make cleanup > - * simple. > - */ > -void dapl_hca_cleanup(DAPL_HCA * hca_ptr, DAT_BOOLEAN dec_ref) > -{ > - dapl_ib_close_hca(hca_ptr); > - if (dec_ref == DAT_TRUE) { > - atomic_dec(&hca_ptr->handle_ref_count); > - } > -} > Index: dapl.h > =================================================================== > --- dapl.h (revision 2296) > +++ dapl.h (working copy) > @@ -211,8 +211,6 @@ > spinlock_t lock; > DAPL_LLIST_HEAD ia_list_head; /* list of all open IAs */ > atomic_t handle_ref_count; /* count of ia_opens on handle */ > - DAPL_EVD *async_evd; > - DAPL_EVD *async_error_evd; > struct sockaddr_in6 hca_address; /* local address of HCA */ > char *name; /* provider name */ > ib_hca_handle_t ib_hca_handle; -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From jlentini at netapp.com Tue May 10 13:53:59 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 10 May 2005 16:53:59 -0400 (EDT) Subject: [openib-general] [KDAPL] module initialization order In-Reply-To: <523bsvxab4.fsf@topspin.com> References: <1115424198.19661.11.camel@duffman> <1115662195.9691.7.camel@duffman> <523bsvxab4.fsf@topspin.com> Message-ID: roland> Yes, link order is the order that initialization functions will be roland> called in. Do you know if adjusting the order of a Makefile's obj-$(CONFIG_XXX) lines is the recommended was to specify the link order? roland> However (and I'm not sure if this really affects you) devices roland> can be added to or removed from the system at any time because roland> of PCI hotplug etc., which can cause device driver roland> init/cleanup functions to be called at arbitrary times. I don't foresee any problems in that case. I am only concerned about the initialization order when the code is built into the kernel image. From iod00d at hp.com Tue May 10 14:00:21 2005 From: iod00d at hp.com (Grant Grundler) Date: Tue, 10 May 2005 14:00:21 -0700 Subject: [openib-general] fixup to use timex.h Message-ID: <20050510210021.GS5495@esmail.cup.hp.com> Michael, userspace/perftests patch to add use of asm/timex.h to get_clock.h Andi Kleen agrees this the best way to pick up get_cycles(). PPC64 has a clean timex.h. PPC does not. thanks, grant Signed-off-by: Grant Grundler Index: get_clock.h =================================================================== --- get_clock.h (revision 2296) +++ get_clock.h (working copy) @@ -31,8 +31,10 @@ * * $Id$ */ + #include #include + #if defined (__x86_64__) || defined(__i386__) typedef unsigned long long cycles_t; static inline cycles_t get_clock() @@ -44,7 +46,12 @@ static inline cycles_t get_clock() val = (val << 32) | low; return val; } -#elif defined(__PPC64__) + +#elif defined(__PPC__) +/* PPC (32-bit) only provides a kernel version of get_cycles(). + * Which implies some PPC (32-bit) do not support mftb. + * Use at your own risk. + */ typedef unsigned long long cycles_t; static inline cycles_t get_cycles() { @@ -53,17 +60,12 @@ static inline cycles_t get_cycles() asm volatile ("mftb %0" : "=r" (ret) : ); return ret; } -#elif defined(__ia64__) -typedef unsigned long long cycles_t; -static inline cycles_t get_cycles() -{ - cycles_t ret; - asm volatile ("mov %0=ar.itc" : "=r"(ret)); - return ret; -} #else -#warning get_clock not implemented for this architecture + +/* Everyone else has nice timex.h */ +#include + #endif double get_cpu_mhz() From jlentini at netapp.com Tue May 10 14:05:11 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 10 May 2005 17:05:11 -0400 (EDT) Subject: [openib-general] [KDAPL] module initialization order (was Re: [PATCH] kDAPL: remove typedef DAT_PROVIDER) In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35AB6@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35AB6@mtvex01.mtv.mtl.com> Message-ID: itamar> I still see problem when NFSoRDMA or any other dat consumer itamar> and dat will be build statically but the dat_provider itamar> (ib,iwarp,...) will be build as a module. That situation could be viewed as a configuration error. I think we can use the kconfig language to prevent this. itamar> who will protect the dat consumer ? how will the dat consumer itamar> know that there is no dat_provider available when dat_provider itamar> modules are not loaded? Currently when a consumer (either one built into the kernel or a loadable module) attempts to open an IA that does not exist it will receive a "not found" error. The DAT specification's "static registry" provided a mechanism for loading a provider on demand. This part of the spec hasn't been implemented for the kernel yet, but I think it would solve this problem. itamar> itamar> Itamar itamar> itamar> > -----Original Message----- itamar> > From: James Lentini [mailto:jlentini at netapp.com] itamar> > Sent: Tuesday, May 10, 2005 8:21 PM itamar> > To: openib-general at openib.org itamar> > Cc: Tom Duffy itamar> > Subject: [openib-general] [KDAPL] module initialization order (was Re: itamar> > [PATCH] kDAPL: remove typedef DAT_PROVIDER) itamar> > itamar> > itamar> > itamar> > After reading through the kernel sources, I believe that builtin itamar> > module initialization functions are called by the do_initcalls itamar> > function in init/main.c. itamar> > itamar> > I haven't conclusively determined how the initialization order is itamar> > specified. From what I've gleamed from mailing list archives, the itamar> > module initialization order was determined by link order. As a result itamar> > the relative position of the obj-$(CONFIG_XXX) lines in Makefiles itamar> > determined the order of module initialization. Often when I found itamar> > discussion of this topic, it was in the context of changing this itamar> > scheme to make the initialization dependencies explicit. However, it itamar> > does not appear that this was ever done. itamar> > itamar> > One final piece of information that I came across is that the itamar> > order of itamar> > initialization functions in the System.map file is the order itamar> > that they itamar> > will be called in. itamar> > itamar> > Can anyone confirm any of the information above? itamar> > itamar> > james itamar> > itamar> > On Mon, 9 May 2005, James Lentini wrote: itamar> > itamar> > > itamar> > > Tom, itamar> > > itamar> > > If you are interested, there is one more thing you could look itamar> > > into/give me advice on: itamar> > > itamar> > > The dat module currently keeps track of its "state" (see the itamar> > > DAT_REGISTRY_STATE enumeration). The registry uses this itamar> > information to itamar> > > detect the case when a provider or consumer calls a dat registry itamar> > > function before the registry's init function (dat_init) has run. itamar> > > itamar> > > Do we need to protect against this in the kernel? itamar> > > itamar> > > This situation could occur in usersapce when the registry, itamar> > providers, itamar> > > and consumers could be shared libraries and the library itamar> > initialization itamar> > > functions were invoked in an arbitrary order. itamar> > > itamar> > > I suspect that we can remove the code, but I wanted to make sure. If itamar> > > the dat registry, dat provider, and a consumer (e.g. NFS-RDMA) were itamar> > > built statically as part of the kernel, would the initialization itamar> > > functions be automatically run in the correct order? itamar> > > itamar> > > james itamar> > > itamar> > > On Mon, 9 May 2005, Tom Duffy wrote: itamar> > > itamar> > > tduffy> On Mon, 2005-05-09 at 14:06 -0400, James Lentini wrote: itamar> > > tduffy> > Committed revision 2287. itamar> > > tduffy> > itamar> > > tduffy> > On Fri, 6 May 2005, Tom Duffy wrote: itamar> > > tduffy> > itamar> > > tduffy> > tduffy> James, thanks for applying my last patch. itamar> > > tduffy> > tduffy> itamar> > > tduffy> > tduffy> You know where I am going with this... itamar> > this is the first in what will be itamar> > > tduffy> > tduffy> a huge amount of patches. I am happy to itamar> > go through and send them all to itamar> > > tduffy> > tduffy> the list, but it might be quicker and itamar> > easier for you to do it directly itamar> > > tduffy> > tduffy> to your tree for all the structs and itamar> > enums. Sup to you. itamar> > > tduffy> > tduffy> itamar> > > tduffy> itamar> > > tduffy> So, did you want me to continue with sending these itamar> > type of patches? itamar> > > tduffy> itamar> > > tduffy> -tduffy itamar> > > tduffy> itamar> > _______________________________________________ itamar> > openib-general mailing list itamar> > openib-general at openib.org itamar> > http://openib.org/mailman/listinfo/openib-general itamar> > itamar> > To unsubscribe, please visit itamar> > http://openib.org/mailman/listinfo/openib-general itamar> > itamar> From roland at topspin.com Tue May 10 13:58:02 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 10 May 2005 13:58:02 -0700 Subject: [openib-general] [KDAPL] module initialization order In-Reply-To: (James Lentini's message of "Tue, 10 May 2005 16:53:59 -0400 (EDT)") References: <1115424198.19661.11.camel@duffman> <1115662195.9691.7.camel@duffman> <523bsvxab4.fsf@topspin.com> Message-ID: <52ekcevlwl.fsf@topspin.com> James> Do you know if adjusting the order of a Makefile's James> obj-$(CONFIG_XXX) lines is the recommended was to specify James> the link order? I believe so. See Documentation/kbuild/makefiles.txt, specifically section 3.2. - R. From jlentini at netapp.com Tue May 10 14:13:20 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 10 May 2005 17:13:20 -0400 (EDT) Subject: [openib-general] [KDAPL] module initialization order In-Reply-To: <52y8anvulz.fsf@topspin.com> References: <91DB792C7985D411BEC300B40080D29CC35AB6@mtvex01.mtv.mtl.com> <52y8anvulz.fsf@topspin.com> Message-ID: roland> Itamar> I still see problem when NFSoRDMA or any other dat roland> Itamar> consumer and dat will be build statically but the roland> Itamar> dat_provider (ib,iwarp,...) will be build as a module. roland> roland> Itamar> who will protect the dat consumer ? how will the dat roland> Itamar> consumer know that there is no dat_provider available when roland> Itamar> dat_provider modules are not loaded? roland> roland> It seems that the kDAPL design of having consumers call a function to roland> get a list of adapters does not fit very well with the Linux model of roland> adapters appearing asynchronously due to drivers being loaded or even roland> due to device hotplug. What would be the correct way to provide consumers this information in the Linux model? From roland at topspin.com Tue May 10 14:17:55 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 10 May 2005 14:17:55 -0700 Subject: [openib-general] [KDAPL] module initialization order In-Reply-To: (James Lentini's message of "Tue, 10 May 2005 17:13:20 -0400 (EDT)") References: <91DB792C7985D411BEC300B40080D29CC35AB6@mtvex01.mtv.mtl.com> <52y8anvulz.fsf@topspin.com> Message-ID: <521x8evkzg.fsf@topspin.com> roland> It seems that the kDAPL design of having consumers call a roland> function to get a list of adapters does not fit very well roland> with the Linux model of adapters appearing asynchronously roland> due to drivers being loaded or even due to device hotplug. James> What would be the correct way to provide consumers this James> information in the Linux model? The kernel currently uses a registration-callback model. For example a driver for PCI devices calls pci_register_driver() and then has its "probe" method called when a device appears (either from "coldplug" during PCI bus enumeration, or hotplug at an arbitrary time). This is parallelled by the IB core's ib_register_client() function, which allows consumers to get asynchronous add/remove callbacks. - R. From roland at topspin.com Tue May 10 14:13:47 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 10 May 2005 14:13:47 -0700 Subject: [openib-general] [KDAPL] module initialization order In-Reply-To: (James Lentini's message of "Tue, 10 May 2005 17:05:11 -0400 (EDT)") References: <91DB792C7985D411BEC300B40080D29CC35AB6@mtvex01.mtv.mtl.com> Message-ID: <5264xqvl6c.fsf@topspin.com> James> That situation could be viewed as a configuration error. I James> think we can use the kconfig language to prevent this. How do you deal with modprobe kdapl modprobe nfs_rdma modprobe mthca Or how about: modprobe mthca modprobe kdapl modprobe nfs_rdma - R. From sean.hefty at intel.com Tue May 10 14:47:07 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Tue, 10 May 2005 14:47:07 -0700 Subject: [openib-general] [PATCH] CM: add invalid service ID handling In-Reply-To: <78d18e205051012572aec71c9@mail.gmail.com> Message-ID: >Patch to respond to invalid REQs (no listener) with a REJ message. Thanks! See updates below. > static int __init ib_cm_init(void) . >+ cm.reject_handler = ib_create_cm_id(cm_reject_handler, NULL); . >+ cm.reject_handler->state = IB_CM_LISTEN; >+ cm.reject_handler->service_id = 0; >+ cm.reject_handler->service_mask = ~0ULL; I don't think that these assignments are necessary. >+error1: >+ cm.reject_handler->state = IB_CM_IDLE; >+ ib_destroy_cm_id(cm.reject_handler); >+error2: >+ destroy_workqueue(cm.wq); > return ret; I reversed the order of these labels to match other areas of the code. Modified patch below. - Sean Index: cm.c =================================================================== --- cm.c (revision 2258) +++ cm.c (working copy) @@ -70,6 +70,7 @@ struct rb_root remote_sidr_table; struct idr local_id_table; struct workqueue_struct *wq; + struct ib_cm_id *reject_cm_id; } cm; struct cm_port { @@ -1048,12 +1049,9 @@ } /* Find matching listen request. */ listen_cm_id_priv = cm_find_listen(req_msg->service_id); - if (!listen_cm_id_priv) { - spin_unlock_irqrestore(&cm.lock, flags); - /* todo: reject with no match */ - ret = -EINVAL; - goto error1; - } + if (!listen_cm_id_priv) + listen_cm_id_priv = container_of(cm.reject_cm_id, + struct cm_id_private, id); atomic_inc(&listen_cm_id_priv->refcount); atomic_inc(&cm_id_priv->refcount); cm_id_priv->id.state = IB_CM_REQ_RCVD; @@ -3024,6 +3022,12 @@ kfree(cm_dev); } +static int cm_reject_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event) +{ + ib_send_cm_rej(cm_id, IB_CM_REJ_INVALID_SERVICE_ID, NULL, 0, NULL, 0); + return -1; +} + static int __init ib_cm_init(void) { int ret; @@ -3043,14 +3047,28 @@ if (!cm.wq) return -ENOMEM; + cm.reject_cm_id = ib_create_cm_id(cm_reject_handler, NULL); + if (IS_ERR(cm.reject_cm_id)) { + ret = PTR_ERR(cm.reject_cm_id); + goto error1; + } + ret = ib_register_client(&cm_client); if (ret) - destroy_workqueue(cm.wq); + goto error2; + + return 0; +error2: + ib_destroy_cm_id(cm.reject_cm_id); +error1: + destroy_workqueue(cm.wq); return ret; } static void __exit ib_cm_cleanup(void) { + ib_destroy_cm_id(cm.reject_cm_id); + flush_workqueue(cm.wq); destroy_workqueue(cm.wq); ib_unregister_client(&cm_client); From halr at voltaire.com Tue May 10 14:46:25 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 10 May 2005 17:46:25 -0400 Subject: [openib-general] [KDAPL] module initialization order In-Reply-To: <52y8anvulz.fsf@topspin.com> References: <91DB792C7985D411BEC300B40080D29CC35AB6@mtvex01.mtv.mtl.com> <52y8anvulz.fsf@topspin.com> Message-ID: <1115761584.4522.11.camel@localhost.localdomain> On Tue, 2005-05-10 at 13:50, Roland Dreier wrote: > Itamar> I still see problem when NFSoRDMA or any other dat > Itamar> consumer and dat will be build statically but the > Itamar> dat_provider (ib,iwarp,...) will be build as a module. > > Itamar> who will protect the dat consumer ? how will the dat > Itamar> consumer know that there is no dat_provider available when > Itamar> dat_provider modules are not loaded? > > It seems that the kDAPL design of having consumers call a function to > get a list of adapters does not fit very well with the Linux model of > adapters appearing asynchronously due to drivers being loaded or even > due to device hotplug. Isn't that (dealing with dynamic devices) a somewhat orthogonal issue although the same IB core mechanism would be utilized (ib_register_client) ? -- Hal From mshefty at ichips.intel.com Tue May 10 14:50:12 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Tue, 10 May 2005 14:50:12 -0700 Subject: [openib-general] [PATCH] CM: add invalid service ID handling In-Reply-To: References: Message-ID: <42812C94.1090104@ichips.intel.com> Sean Hefty wrote: > Modified patch below. Ugh... my mailer wrapped the patch as well... - Sean From info at mynoodle.net Tue May 10 06:30:51 2005 From: info at mynoodle.net (info at mynoodle.net) Date: 10 May 2005 22:30:51 +0900 Subject: [openib-general] $BNY$N!{!{$G$9!&!&!&(B Message-ID: <20050510133051.31044.qmail@mail.mynoodle.net> $B!!!!!!!!!!!!"v!&!yL5NAEPO?%-%c%s%Z!<%sCf!y!&"v(B $B!!!!!!$d$C$Q$j=P0)$&$J$i$46a=j$G2q$($kAjR2pCW$7$^$9!#(B $B!!!!!!!!!!!!!!!!!!!!$^$:$O$*;n$7L5NAEPO?$+$i(B $B!!!!!!!!!!!!!!!!!!!!!!!!!!"-"-"-"-"-"-(B $B!!!!!!!!!!!!!!!!(Bhttp://www.jumpb2.net/?top $B!!!!!!!!!!!!!!!!(B $B!!!!!!!!!!!!!!!!!!(B $B%U%j!<%a!<%kBP1~$G$9!*(B $B5qH](B iranai at jumpb2.net From sean.hefty at intel.com Tue May 10 17:07:22 2005 From: sean.hefty at intel.com (Hefty, Sean) Date: Tue, 10 May 2005 17:07:22 -0700 Subject: [openib-general] proposed MAD layer features Message-ID: I'd like to propose adding the following features to the MAD layer. The purpose behind the request is to enable CM MRA support. - Expand the retries field in the send_wr.wr.ud to be usable by all MADs expecting a response (i.e. timeout_ms > 0). Currently, its use is restricted to RMPP. - Add a mechanism to modify a posted MAD request. Having the retries field is almost necessary for large RMPP messages, but a linear timeout/retry mechanism is useful to the CM as well. Clients that need an adjustable retry mechanism could set retries to zero and perform manual retries, similar to what is required today. The intent of modifying a posted MAD request is to allow the CM to change the timeout value for a posted REQ or REP upon receiving an MRA. RMPP actually uses a similar mechanism to adjust MAD timeouts after receiving an ACK, but the mechanism is not exposed outside of the MAD layer. A drawback is that I'm doubtful that other clients would find the ability to modify a posted MAD request useful. It could be if clients were notified of retry attempts, which would let them increase the timeout value. (This would also close a minor hole where a reply is received late, but in between retry attempts.) Comments? - Sean From libor at topspin.com Tue May 10 17:47:00 2005 From: libor at topspin.com (Libor Michalek) Date: Tue, 10 May 2005 17:47:00 -0700 Subject: [PATCH] Re: [openib-general] 0 op factor In-Reply-To: <20050507174718.GQ15688@aon.at>; from blist@aon.at on Sat, May 07, 2005 at 07:47:18PM +0200 References: <20050505155803.GS15688@aon.at> <20050505172122.B26428@topspin.com> <20050507174718.GQ15688@aon.at> Message-ID: <20050510174700.A5174@topspin.com> On Sat, May 07, 2005 at 07:47:18PM +0200, Bernhard Fischer wrote: > On Thu, May 05, 2005 at 05:21:22PM -0700, Libor Michalek wrote: > > No point other then the obvious, changing the sign of the variables > >value. It's just a style convention that I have no problem changing. > > - remove '0 operator factor' statements. > - a bit of whitespace removal. > - remove return at end of void functions. If you're removing the 'return' why not remove the blank line before it as well. > - remove expect from _sdp_cm_path_complete(). When I said that the SDP_EXPECT should eventually be removed, I meant that the functions which have their return values checked by SDP_EXPECT should either be turned into void return functions, or something intelligent should be done with the return value, such as error recovery or propogation. The functions which should be turned into void functions are the ones which will never return anything but success. -Libor From libor at topspin.com Tue May 10 17:53:42 2005 From: libor at topspin.com (Libor Michalek) Date: Tue, 10 May 2005 17:53:42 -0700 Subject: [openib-general] Re: A Couple of User CM Questions In-Reply-To: <1115667140.4490.244.camel@localhost.localdomain>; from halr@voltaire.com on Mon, May 09, 2005 at 03:32:20PM -0400 References: <1115659336.4490.91.camel@localhost.localdomain> <20050509113943.A2462@topspin.com> <1115667140.4490.244.camel@localhost.localdomain> Message-ID: <20050510175342.B5174@topspin.com> On Mon, May 09, 2005 at 03:32:20PM -0400, Hal Rosenstock wrote: > On Mon, 2005-05-09 at 14:39, Libor Michalek wrote: > > On Mon, May 09, 2005 at 01:22:17PM -0400, Hal Rosenstock wrote: > > > Hi Libor, > > > > > > I have a couple of questions pertaining to the user CM. > > > > > > 1. It appears that device (and port) are not supported either incoming > > > or outgoing. So if that is correct, I presume only the first device > > > (first port) is the only one which can be expected to work. Correct ? > > > If so, that should be added to README as a current limitation. > > So I am presuming this is a current limitation. No, since the REQ event will be delivered to the user CM listener regardless of the device/port on which it is received. > > > 2. Any idea on how device would be supported ? Would this just be with > > > an IB device number or a string like "mthca0" ? > > > > I was thinking that the destination GID in the req_event path record > > would be sufficient to identify the local device/port, but I had not > > taken a look at sidr_req_event. I think this is the correct GID to use > > for creating the QP, but I have not taken a look at how easy it is to > > turn a GID into a suitable device/portin user verbs. Thoughts? > > and source GID (in the REQ primary path) on the active side ? > > That seems like it would work but wouldn't you need to walk the GID > tables on each port on each IB device until you find the match ? Sounds > a little expensive but would only be done once per connection. Yes, the kernel CM already behaves this way, Sean made the change to support the user CM a littel while ago. It use to rely on the QP's device/port, but now it relies on the source GID and no QP is needed just the QPN. Sorry for the delayed response. The only question I had was how to bind the user QP to the device/port based on the source or destination GID based on the REQ send or REQ event respectively. I'm only just now looking at this... -Libor From libor at topspin.com Tue May 10 18:06:25 2005 From: libor at topspin.com (Libor Michalek) Date: Tue, 10 May 2005 18:06:25 -0700 Subject: [openib-general] [PATCH] SDP: add missing MODULE_PARM_DESC In-Reply-To: <1115483608.17241.2.camel@duffman>; from tduffy@sun.com on Sat, May 07, 2005 at 09:33:28AM -0700 References: <20050506003018.GD15688@aon.at> <1115483608.17241.2.camel@duffman> Message-ID: <20050510180625.C5174@topspin.com> On Sat, May 07, 2005 at 09:33:28AM -0700, Tom Duffy wrote: > On Fri, 2005-05-06 at 02:30 +0200, Bernhard Fischer wrote: > > in src/linux-kernel/infiniband/ulp/sdp/sdp_inet.c > > module_param seem to be missing MODULE_PARM_DESC > > Signed-off-by: Tom Duffy Thanks, applied and checked in. -Libor From iod00d at hp.com Tue May 10 18:21:48 2005 From: iod00d at hp.com (Grant Grundler) Date: Tue, 10 May 2005 18:21:48 -0700 Subject: [openib-general] Re: user-mode verbs on Itanium In-Reply-To: <20050510154030.GH2474@mellanox.co.il> References: <20050429162127.GA24871@esmail.cup.hp.com> <52ll71k09w.fsf@topspin.com> <20050429183846.GH24871@esmail.cup.hp.com> <20050504005534.GA13574@esmail.cup.hp.com> <20050504070910.GC9674@mellanox.co.il> <20050506190947.GA23958@esmail.cup.hp.com> <20050507053706.GN23958@esmail.cup.hp.com> <20050508142345.GQ9941@mellanox.co.il> <20050510152428.GA5495@esmail.cup.hp.com> <20050510154030.GH2474@mellanox.co.il> Message-ID: <20050511012148.GE5495@esmail.cup.hp.com> On Tue, May 10, 2005 at 06:40:30PM +0300, Michael S. Tsirkin wrote: > > Yes, but I think I want the first measurement *before* post_send > > since it seems to measure something different that's part of the > > code path too. > > I'd prefer all deltas to measure the same thing. Ok. Can we report the startup times as part of the summary? > > I suspect it's measuring the first "TLB" miss of HCA. > > I dont think so. ok. Any other theories on why it takes ~12x longer (4.7 vs 63usec)? > You still see this cache warm-up with the new code. I'll try running it again with code in SVN and see what I get. BTW, which cache are you referring to here? thanks, grant From iod00d at hp.com Tue May 10 18:33:58 2005 From: iod00d at hp.com (Grant Grundler) Date: Tue, 10 May 2005 18:33:58 -0700 Subject: [openib-general] perftest #2: replace last use of get_clock() Message-ID: <20050511013358.GF5495@esmail.cup.hp.com> Michael, rdma_lat won't link - still has reference to get_clock(). Patch below fixes. thanks, grant Signed-off-by: Grant Grundler Index: rdma_lat.c =================================================================== --- rdma_lat.c (revision 2296) +++ rdma_lat.c (working copy) @@ -724,7 +724,7 @@ int main(int argc, char *argv[]) scnt); return 1; } - tstamp[scnt - 1] = get_clock(); + tstamp[scnt - 1] = get_cycles(); } if (ccnt < iters) { From tduffy at sun.com Tue May 10 18:35:03 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 10 May 2005 18:35:03 -0700 Subject: [openib-general] [PATCH] SDP: add missing MODULE_PARM_DESC In-Reply-To: <20050510180625.C5174@topspin.com> References: <20050506003018.GD15688@aon.at> <1115483608.17241.2.camel@duffman> <20050510180625.C5174@topspin.com> Message-ID: <1115775303.5974.1.camel@duffman> On Tue, 2005-05-10 at 18:06 -0700, Libor Michalek wrote: > Thanks, applied and checked in. One minor nit: you changed total to maximuml in conn_size description, which is fine, but you misspelled maximum. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From iod00d at hp.com Tue May 10 18:40:14 2005 From: iod00d at hp.com (Grant Grundler) Date: Tue, 10 May 2005 18:40:14 -0700 Subject: [openib-general] perftest-03: put C code in .c file Message-ID: <20050511014014.GG5495@esmail.cup.hp.com> Hi Michael, Patch below fixes up the Makefile and getclock.h so C code is in C files like people expect. Unfortunately, I don't know how to make svn diff treat a new file as part of the diff. "cvs -upN" is how I would do it for cvs. svn just (correctly) whines it doesn't know about the new file. But it's easy to generate get_clock.c: cp get_clock.h get_clock.c edit get_clock.c and delete everything related to get_cycles(). (This preserves the copyright/license too) *THEN* apply the patch below. thanks, grant diff -u get_clock.h get_clock.h --- get_clock.h (working copy) +++ get_clock.h (working copy) @@ -32,9 +32,6 @@ * $Id$ */ -#include -#include - #if defined (__x86_64__) || defined(__i386__) typedef unsigned long long cycles_t; static inline cycles_t get_clock() @@ -71,29 +68,2 @@ -double get_cpu_mhz() -{ - FILE* f; - char buf[256]; - double mhz = 0.0; - - f = fopen("/proc/cpuinfo","r"); - if (!f) - return 0.0; - while(fgets(buf, sizeof(buf), f)) { - double m; - int rc; - rc = sscanf(buf, "cpu MHz : %lf", &m); - if (rc != 1) - continue; - if (mhz == 0.0) { - mhz = m; - continue; - } - if (mhz != m) { - fprintf(stderr,"Conflicting CPU frequency values " - " detected: %lf != %lf\n", - mhz, m); - return 0.0; - } - } - fclose(f); - return mhz; -} + +extern double get_cpu_mhz(void); only in patch2: unchanged: --- Makefile (revision 2296) +++ Makefile (working copy) @@ -1,6 +1,9 @@ CFLAGS = -Wall -O2 -g -D_GNU_SOURCE -rdma_lat: rdma_lat.c get_clock.h - gcc $(CFLAGS) -o rdma_lat rdma_lat.c -libverbs +RDMA_LAT_OBJ = rdma_lat.o get_clock.o + +rdma_lat: $(RDMA_LAT_OBJ) + gcc -o rdma_lat $(RDMA_LAT_OBJ) -libverbs + clean: - rm -f rdma_lat + rm -f rdma_lat $(RDMA_LAT_OBJ) From tduffy at sun.com Tue May 10 18:48:20 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 10 May 2005 18:48:20 -0700 Subject: [openib-general] perftest-03: put C code in .c file In-Reply-To: <20050511014014.GG5495@esmail.cup.hp.com> References: <20050511014014.GG5495@esmail.cup.hp.com> Message-ID: <1115776100.5974.6.camel@duffman> On Tue, 2005-05-10 at 18:40 -0700, Grant Grundler wrote: > Unfortunately, I don't know how to make svn diff treat a new file as > part of the diff. Do a "svn add" on the new file. You don't need check in privileges to do this. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From libor at topspin.com Tue May 10 19:46:16 2005 From: libor at topspin.com (Libor Michalek) Date: Tue, 10 May 2005 19:46:16 -0700 Subject: [openib-general] [PATCH] SDP: add missing MODULE_PARM_DESC In-Reply-To: <1115775303.5974.1.camel@duffman>; from tduffy@sun.com on Tue, May 10, 2005 at 06:35:03PM -0700 References: <20050506003018.GD15688@aon.at> <1115483608.17241.2.camel@duffman> <20050510180625.C5174@topspin.com> <1115775303.5974.1.camel@duffman> Message-ID: <20050510194616.A21981@topspin.com> On Tue, May 10, 2005 at 06:35:03PM -0700, Tom Duffy wrote: > On Tue, 2005-05-10 at 18:06 -0700, Libor Michalek wrote: > > Thanks, applied and checked in. > > One minor nit: you changed total to maximuml in conn_size description, > which is fine, but you misspelled maximum. If the compiler doesn't catch it, only a matter of time till I misspell it. :) -Libor From info at qsv04.com Tue May 10 07:20:02 2005 From: info at qsv04.com (info at qsv04.com) Date: 10 May 2005 23:20:02 +0900 Subject: [openib-general] $BC/$G$b(B5,000$B1_J,!D(B Message-ID: <20050510142002.8997.qmail@mail.qsv04.com> $B!!!!(.(,(/(.(,(/(.(,(/(.(,(/(.(,(/(.(,(/(.(,(/(.(,(/!!(B $B!!!!!!!!!!!!!!"#%a!<%kM'$*C5$7$O%3%3!*!*(B $B!!!!!!!!!!!!!!!!"-"-"-!!L5NA$*;n$7!!"-"-"-(B $B!!!!!!!!!!!!!!"#7k:'AjZ:Q!K!!!!("(B $B!!!!!!!!!!!!!!!!!!(&(!(!(!(!(!(!(!(!(%(B $BEv%5%$%HB~:#=P7l%-%c%s%Z!<%s $B!!!!!!!!!!!!"v!&!yL5NAEPO?%-%c%s%Z!<%sCf!y!&"v(B $B!!!!!!$d$C$Q$j=P0)$&$J$i$46a=j$G2q$($kAjR2pCW$7$^$9!#(B $B!!!!!!!!!!!!!!!!!!!!$^$:$O$*;n$7L5NAEPO?$+$i(B $B!!!!!!!!!!!!!!!!!!!!!!!!!!"-"-"-"-"-"-(B $B!!!!!!!!!!!!!!!!(Bhttp://www.jumpb2.net/?join $B!!!!!!!!!!!!!!!!(B $B!!!!!!!!!!!!!!!!!!(B $B%U%j!<%a!<%kBP1~$G$9!*(B $B5qH](B iranai at jumpb2.net From mst at mellanox.co.il Tue May 10 23:38:09 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 11 May 2005 09:38:09 +0300 Subject: [openib-general] Re: perftest-03: put C code in .c file In-Reply-To: <20050511014014.GG5495@esmail.cup.hp.com> References: <20050511014014.GG5495@esmail.cup.hp.com> Message-ID: <20050511063809.GA13349@mellanox.co.il> Quoting r. Grant Grundler : > Subject: perftest-03: put C code in .c file > > Hi Michael, > Patch below fixes up the Makefile and getclock.h so C code is in C files > like people expect. > > Unfortunately, I don't know how to make svn diff treat a new file as > part of the diff. "cvs -upN" is how I would do it for cvs. > svn just (correctly) whines it doesn't know about the new file. > > But it's easy to generate get_clock.c: > cp get_clock.h get_clock.c > edit get_clock.c and delete everything related to get_cycles(). > (This preserves the copyright/license too) > > *THEN* apply the patch below. > > thanks, > grant > You have to svn add before svn diff -x --diff-cmd=/usr/bin/diff -x -upN Grant, could you please resend, while also adding a Signed-off-by line? Quoting linux kernel's Documentation/SubmittingPatches: The sign-off is a simple line at the end of the explanation for the patch, which certifies that you wrote it or otherwise have the right to pass it on as a open-source patch. The rules are pretty simple: if you can certify the below: Developer's Certificate of Origin 1.0 By making a contribution to this project, I certify that: (a) The contribution was created in whole or in part by me and I have the right to submit it under the open source license indicated in the file; or (b) The contribution is based upon previous work that, to the best of my knowledge, is covered under an appropriate open source license and I have the right under that license to submit that work with modifications, whether created in whole or in part by me, under the same open source license (unless I am permitted to submit under a different license), as indicated in the file; or (c) The contribution was provided directly to me by some other person who certified (a), (b) or (c) and I have not modified it. then you just add a line saying Signed-off-by: Random J Developer -- MST - Michael S. Tsirkin From mst at mellanox.co.il Wed May 11 00:21:38 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 11 May 2005 10:21:38 +0300 Subject: [openib-general] Re: user-mode verbs on Itanium In-Reply-To: <20050511012148.GE5495@esmail.cup.hp.com> References: <52ll71k09w.fsf@topspin.com> <20050429183846.GH24871@esmail.cup.hp.com> <20050504005534.GA13574@esmail.cup.hp.com> <20050504070910.GC9674@mellanox.co.il> <20050506190947.GA23958@esmail.cup.hp.com> <20050507053706.GN23958@esmail.cup.hp.com> <20050508142345.GQ9941@mellanox.co.il> <20050510152428.GA5495@esmail.cup.hp.com> <20050510154030.GH2474@mellanox.co.il> <20050511012148.GE5495@esmail.cup.hp.com> Message-ID: <20050511072138.GB13349@mellanox.co.il> Quoting r. Grant Grundler : > Subject: Re: user-mode verbs on Itanium > > On Tue, May 10, 2005 at 06:40:30PM +0300, Michael S. Tsirkin wrote: > > > Yes, but I think I want the first measurement *before* post_send > > > since it seems to measure something different that's part of the > > > code path too. > > > > I'd prefer all deltas to measure the same thing. > > Ok. Can we report the startup times as part of the summary? You mean just sample the clock before the first post? That would be easy, but lets look at what does it measure: On the client, you just get the time it takes to perform the first post_send operation. If this is interesting to you, I agree, but lets take the timestamps around each post send operation to make it statistically relevant. Let me know - and need to think of a way to do this without affecting latency numbers. On the server, you get that, plus the latency, plus time it takes the OS to wake the client from a blocking socket operation. It's the last component that isnt IB-related and that makes it a non-interesting. By the way, what do you use for statistics? > > > > I suspect it's measuring the first "TLB" miss of HCA. > > > > I dont think so. > > ok. Any other theories on why it takes ~12x longer (4.7 vs 63usec)? > > > You still see this cache warm-up with the new code. > > I'll try running it again with code in SVN and see what I get. BTW, > which cache are you referring to here? > > thanks, > grant > Hard to tell for sure. That needs to be investigated. -- MST - Michael S. Tsirkin From itamar at mellanox.co.il Wed May 11 00:41:14 2005 From: itamar at mellanox.co.il (Itamar) Date: Wed, 11 May 2005 10:41:14 +0300 Subject: [openib-general] [PATCH][kdapl] resend patch of clean up async evd Message-ID: <20050511074114.GA32383@mellanox.co.il> Clean up async evd flow. Clean up init module. Signed-off-by: Itamar Rabenstein Index: dapl_module.c =================================================================== --- dapl_module.c (revision 2302) +++ dapl_module.c (working copy) @@ -147,6 +147,28 @@ goto bail; } + dat_status = dapl_ib_open_hca(hca_ptr->name, hca_ptr); + + if (dat_status != DAT_SUCCESS) { + dapl_dbg_log(DAPL_DBG_TYPE_ERR, + "dapl_ib_open_hca failed %x\n", + dat_status); + goto bail; + } + /* + * Obtain IA attributes from the HCA to limit certain + * operations. + */ + dat_status = dapl_ib_query_hca(hca_ptr, + &hca_ptr->ia_attr, + NULL, &hca_ptr->hca_address); + if (dat_status != DAT_SUCCESS) { + dapl_dbg_log(DAPL_DBG_TYPE_ERR, + "dapl_ib_query_hca failed %x\n", + dat_status); + goto bail; + } + provider->extension = hca_ptr; /* register providers with dat_registry */ Index: dapl_ia_util.c =================================================================== --- dapl_ia_util.c (revision 2302) +++ dapl_ia_util.c (working copy) @@ -468,13 +468,6 @@ { spin_lock(&hca_ptr->lock); atomic_dec(&hca_ptr->handle_ref_count); - if (atomic_read(&hca_ptr->handle_ref_count) == 0) { - dapl_ib_close_hca(hca_ptr); -#if 0 - hca_ptr->ib_hca_handle = IB_INVALID_HANDLE; -#endif - hca_ptr->async_evd = NULL; - } spin_unlock(&hca_ptr->lock); } Index: dapl_ia_open.c =================================================================== --- dapl_ia_open.c (revision 2302) +++ dapl_ia_open.c (working copy) @@ -44,12 +44,6 @@ #include "dapl_adapter_util.h" /* - * LOCAL PROTOTYPES - */ - -void dapl_hca_cleanup(DAPL_HCA * hca_ptr, DAT_BOOLEAN dec_ref); - -/* * Open a provider and return a handle. The handle enables the user * to invoke operations on this provider. * @@ -99,37 +93,7 @@ /* get the hca_ptr */ hca_ptr = (DAPL_HCA *) provider->extension; - /* - * Open the HCA if it has not been done before. - */ spin_lock(&hca_ptr->lock); - /* register with the HW */ - dat_status = dapl_ib_open_hca(hca_ptr->name, hca_ptr); - - if (dat_status != DAT_SUCCESS) { - dapl_dbg_log(DAPL_DBG_TYPE_ERR, - "dapl_ib_open_hca failed %x\n", - dat_status); - spin_unlock(&hca_ptr->lock); - goto bail; - } - - /* - * Obtain IA attributes from the HCA to limit certain - * operations. - * If using DAPL_ATS naming, ib_query_hca will also set the ip - * address. - */ - dat_status = dapl_ib_query_hca(hca_ptr, - &hca_ptr->ia_attr, - NULL, &hca_ptr->hca_address); - if (dat_status != DAT_SUCCESS) { - dapl_hca_cleanup(hca_ptr, DAT_FALSE); - spin_unlock(&hca_ptr->lock); - goto bail; - } - - /* Take a reference on the hca_handle */ atomic_inc(&hca_ptr->handle_ref_count); spin_unlock(&hca_ptr->lock); @@ -137,7 +101,7 @@ ia_ptr = dapl_ia_alloc(provider, hca_ptr); if (!ia_ptr) { spin_lock(&hca_ptr->lock); - dapl_hca_cleanup(hca_ptr, DAT_TRUE); + atomic_dec(&hca_ptr->handle_ref_count); spin_unlock(&hca_ptr->lock); dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); @@ -157,14 +121,6 @@ DAT_INVALID_HANDLE_EVD_ASYNC); goto bail; } - - /* InfiniBand allows only 1 asychronous event handler per HCA */ - /* (see InfiniBand Spec, release 1.1, vol I, section 11.5.2, */ - /* page 559). */ - /* */ - /* We only need to make sure that this EVD's CQ belongs to */ - /* the same HCA as is being opened. */ - if (evd_ptr->header.owner_ia->hca_ptr->ib_hca_handle != hca_ptr->ib_hca_handle) { dat_status = @@ -172,7 +128,6 @@ DAT_INVALID_HANDLE_EVD_ASYNC); goto bail; } - ia_ptr->cleanup_async_error_evd = DAT_FALSE; ia_ptr->async_error_evd = evd_ptr; } else { @@ -187,31 +142,17 @@ async_evd_qlen, DAT_EVD_ASYNC_FLAG, &evd_ptr); - if (dat_status != DAT_SUCCESS) { + if (dat_status != DAT_SUCCESS) goto bail; - } atomic_inc(&evd_ptr->evd_ref_count); - - spin_lock(&hca_ptr->lock); - if (hca_ptr->async_evd != (DAPL_EVD *) 0) { - spin_unlock(&hca_ptr->lock); - } else { - hca_ptr->async_evd = evd_ptr; - spin_unlock(&hca_ptr->lock); - - /* Register the handlers associated with the async EVD. */ - dat_status = dapl_ia_setup_callbacks(ia_ptr, evd_ptr); - if (dat_status != DAT_SUCCESS) { - /* Assign the EVD so it gets cleaned up */ - ia_ptr->cleanup_async_error_evd = DAT_TRUE; - ia_ptr->async_error_evd = evd_ptr; - goto bail; - } - } - + /* Register the handlers associated with the async EVD. */ + dat_status = dapl_ia_setup_callbacks(ia_ptr, evd_ptr); + /* Assign the EVD so it gets cleaned up */ ia_ptr->cleanup_async_error_evd = DAT_TRUE; ia_ptr->async_error_evd = evd_ptr; + if (dat_status != DAT_SUCCESS) + goto bail; } dat_status = DAT_SUCCESS; @@ -220,10 +161,8 @@ bail: if (dat_status != DAT_SUCCESS) { - if (ia_ptr) { - /* This will release the async EVD if needed. */ + if (ia_ptr) dapl_ia_close(ia_ptr, DAT_CLOSE_ABRUPT_FLAG); - } } dapl_dbg_log(DAPL_DBG_TYPE_RTN, @@ -231,15 +170,3 @@ return dat_status; } - -/* - * Clean up partially allocated HCA stuff. Strictly to make cleanup - * simple. - */ -void dapl_hca_cleanup(DAPL_HCA * hca_ptr, DAT_BOOLEAN dec_ref) -{ - dapl_ib_close_hca(hca_ptr); - if (dec_ref == DAT_TRUE) { - atomic_dec(&hca_ptr->handle_ref_count); - } -} Index: dapl.h =================================================================== --- dapl.h (revision 2302) +++ dapl.h (working copy) @@ -211,8 +211,6 @@ spinlock_t lock; DAPL_LLIST_HEAD ia_list_head; /* list of all open IAs */ atomic_t handle_ref_count; /* count of ia_opens on handle */ - DAPL_EVD *async_evd; - DAPL_EVD *async_error_evd; struct sockaddr_in6 hca_address; /* local address of HCA */ char *name; /* provider name */ ib_hca_handle_t ib_hca_handle; -- Itamar From mst at mellanox.co.il Wed May 11 01:05:47 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 11 May 2005 11:05:47 +0300 Subject: [openib-general] Re: [PATCH] kDAPL: remove DAT_BOOLEAN In-Reply-To: <1115747465.8222.15.camel@duffman> References: <1115747465.8222.15.camel@duffman> Message-ID: <20050511080547.GC13349@mellanox.co.il> Quoting r. Tom Duffy : > Subject: [PATCH] kDAPL: remove DAT_BOOLEAN > > I am sending this as a separate patch because I am not sure what the > "right" thing to do is. Looking around the kernel, it seems booleans > are handled in different ways. Some places, they are enums, others just > #defines. > > In e1000, ixgb, compaq fc, DAT960, and xfs, they are enums. In jfs and > arch parisc, a bool is an int and TRUE and FALSE are #define'd. TRUE > and FALSE are defined so many times in the kernel, it seems rather > ridiculous. Maybe the right thing to do is fix the whole kernel in a > consistent way and submit an upstream patch, but this will require quite > a bit of janitorial work. Further, include/linux/synclink.h has #define BOOLEAN int and include/linux/arcdevice.h has #define bool int > I don't mind using either way in kDAPL (or some third -- getting rid of > bool's all together?),but this patch uses the enum method. At least we > need to get rid of the DAT_ prefix. Is this type ever packed and passed on the wire? If yes that would be a valid reason for something DAT_BOOLEAN, to make sure it has a specific size. > Singed-off-by: Tom Duffy Surprisingly, some DAT code just uses "bool" (below). > Index: linux-kernel/test/dapltest/include/dapl_proto.h > =================================================================== > --- linux-kernel/test/dapltest/include/dapl_proto.h (revision 2295) > +++ linux-kernel/test/dapltest/include/dapl_proto.h (working copy) > @@ -88,8 +88,8 @@ Bpool * DT_BpoolAlloc (Per_Test_ > DAT_COUNT seg_size, > DAT_COUNT num_segs, > DAT_COUNT alignment, > - DAT_BOOLEAN enable_rdma_write, > - DAT_BOOLEAN enable_rdma_read); > + boolean_t enable_rdma_write, > + boolean_t enable_rdma_read); > > bool DT_Bpool_Destroy (Per_Test_Data_t * pt_ptr, > DT_Tdep_Print_Head *phead, > @@ -267,13 +267,13 @@ bool DT_Performance_Test_Serv > bool DT_Performance_Test_Create (Per_Test_Data_t * pt_ptr, > DAT_IA_HANDLE * ia_handle, > DAT_IA_ADDRESS_PTR remote_ia_addr, > - DAT_BOOLEAN is_server, > - DAT_BOOLEAN is_remote_little_endian, > + boolean_t is_server, > + boolean_t is_remote_little_endian, > Performance_Test_t **perf_test); > -- MST - Michael S. Tsirkin From blist at aon.at Wed May 11 05:41:57 2005 From: blist at aon.at (Bernhard Fischer) Date: Wed, 11 May 2005 14:41:57 +0200 Subject: [openib-general] [KDAPL] module initialization order In-Reply-To: <5264xqvl6c.fsf@topspin.com> References: <91DB792C7985D411BEC300B40080D29CC35AB6@mtvex01.mtv.mtl.com> <5264xqvl6c.fsf@topspin.com> Message-ID: <20050511124157.GA7621@aon.at> On Tue, May 10, 2005 at 02:13:47PM -0700, Roland Dreier wrote: > James> That situation could be viewed as a configuration error. I > James> think we can use the kconfig language to prevent this. You could select y the parent-provider of a module, yes. This doesn't take care of removal when all parts are modules, though. > >How do you deal with > > modprobe kdapl > modprobe nfs_rdma > modprobe mthca > >Or how about: > > modprobe mthca > modprobe kdapl > modprobe nfs_rdma > If i understand the question correctly, isn't hotplug (i.e. userspace) taking care of this? locally for e.g. ipoib, i have something along the lines of: tst11:~ # modprobe -v ib0 install /sbin/modprobe -v ib_mthca && /sbin/modprobe -v ib_ipoib insmod /lib/modules/2.6.12-rc4/kernel/drivers/infiniband/core/ib_core.ko insmod /lib/modules/2.6.12-rc4/kernel/drivers/infiniband/core/ib_mad.ko insmod /lib/modules/2.6.12-rc4/kernel/drivers/infiniband/hw/mthca/ib_mthca.ko insmod /lib/modules/2.6.12-rc4/kernel/drivers/infiniband/core/ib_sa.ko insmod /lib/modules/2.6.12-rc4/kernel/drivers/infiniband/ulp/ipoib/ib_ipoib.ko tst11:~ # now hotplug would call the 'remove' line for e.g. ipoib et al^A^K^J tst11:~ # modprobe -v -r ib_ipoib remove /sbin/rmmod ib_ipoib ; /sbin/rmmod ib_sa ; /sbin/rmmod ib_mthca ; /sbin/rmmod ib_mad ; /sbin/rmmod ib_core The pci hotplug should have taken care to decrement the use count, so if you pulled the last user, userspace hotplug would unload the unused (if configured to do so) parts of the driver chain. NFS users in the above example would just sit there and block, i think. From halr at voltaire.com Wed May 11 05:42:45 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 11 May 2005 08:42:45 -0400 Subject: [openib-general] [PATCH] kdapl: Make CM event rather than state based Message-ID: <1115815364.4479.751.camel@localhost.localdomain> Change kDAPL CM to be event rather than state based Also, fix connection and CM ID destruction relative to DAPL threading and CM callbacks Add REJ handling Some other more minor bug fixes and rewrite Kudos to Sean for making this happen so quickly. Signed-off-by: Sean Hefty Signed-off-by: Hal Rosenstock Index: dapl_openib_cm.h =================================================================== --- dapl_openib_cm.h (revision 2302) +++ dapl_openib_cm.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Voltaire Inc. All rights reserved. + * Copyright (c) 2005 Intel Corporation. All rights reserved. * Copyright (c) 2004-2005, Mellanox Technologies, Inc. All rights reserved. * Copyright (c) 2003 Topspin Corporation. All rights reserved. * @@ -56,39 +57,30 @@ IB_CME_BROKEN } ib_cm_events_t; -#define IB_CM_LOCAL_FAILURE IB_CME_LOCAL_FAILURE -#define IB_CM_HANDLE_FMT "%x" - -/* Currently needed for DAPL */ -typedef struct ib_cm_id *ib_cm_handle_t; -typedef struct ib_cm_id *ib_cm_srvc_handle_t; - -#define IB_MAX_REQ_PDATA_SIZE 92 -#define IB_MAX_REP_PDATA_SIZE 196 -#define IB_MAX_REJ_PDATA_SIZE 148 -#define IB_MAX_DREQ_PDATA_SIZE 220 -#define IB_MAX_DREP_PDATA_SIZE 224 - -#define OPENIB_REQ_MRA_TIMEOUT 27 /* a little over 9 minutes */ - -#define OPENIB_MAX_AT_RETRY 3 - -struct dapl_ib_destroy_cm_id { - struct work_struct work; - struct ib_cm_id *cm_id; -}; - -struct dapl_ib_connection { +struct dapl_cm_id { int retries; struct ib_at_ib_route dapl_rt; struct ib_sa_path_rec dapl_path; - struct dapl_ep *ep_ptr; + struct dapl_ep *ep; struct sockaddr *remote_ia_address; struct ib_cm_req_param param; u64 service_id; struct ib_at_completion dapl_comp; + + struct ib_cm_id *cm_id; + struct dapl_sp *sp; + spinlock_t lock; + int in_callback; + int destroy; }; +typedef struct dapl_cm_id *ib_cm_handle_t; +typedef struct ib_cm_id *ib_cm_srvc_handle_t; + +#define OPENIB_REQ_MRA_TIMEOUT 27 /* a little over 9 minutes */ + +#define OPENIB_MAX_AT_RETRY 3 + /************************ * Function declarations * ************************/ Index: dapl_openib_cm.c =================================================================== --- dapl_openib_cm.c (revision 2302) +++ dapl_openib_cm.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2005 Voltaire Inc. All rights reserved. + * Copyright (c) 2005 Intel Corporation. All rights reserved. * Copyright (c) 2004-2005, Mellanox Technologies, Inc. All rights reserved. * Copyright (c) 2003 Topspin Corporation. All rights reserved. * @@ -35,234 +36,344 @@ #include "dapl.h" #include "dapl_adapter_util.h" #include "dapl_evd_util.h" -#include "dapl_sp_util.h" #include "dapl_cr_util.h" -static void dapl_rt_comp_handler(u64 req_id, void *context, int rec_num); -static void dapl_path_comp_handler(u64 req_id, void *context, int rec_num); - -static struct workqueue_struct *dapl_cm_wq; - /* Should these be queried ? */ #define IB_TARGET_MAX 4 /* responder resources (max_qp_ous_rd_atom) */ #define IB_INITIATOR_DEPTH 4 /* initiator depth (max_qp_init_rd_atom) */ -static void do_active_idle(DAPL_EP * ep_ptr) +static void dapl_destroy_cm_id(struct dapl_cm_id *conn) { - ib_cm_events_t cm_event; + unsigned long flags; + int in_callback; - if (ep_ptr->cm_handle == IB_INVALID_HANDLE) - return; + spin_lock_irqsave(&conn->lock, flags); + conn->destroy = 1; + in_callback = conn->in_callback; + spin_unlock_irqrestore(&conn->lock, flags); - if (ep_ptr->param.connect_evd_handle) { - cm_event = IB_CME_DISCONNECTED; - dapl_evd_connection_callback(ep_ptr->cm_handle, - cm_event, NULL, (void *)ep_ptr); + if (!in_callback) { + ib_destroy_cm_id(conn->cm_id); + if (conn->ep) + conn->ep->cm_handle = IB_INVALID_HANDLE; + kfree(conn); } - } -static void do_active_rep_recv(DAPL_EP * ep_ptr, - struct ib_cm_id *comm_id, - struct ib_cm_event *rep_recv_param) +static void do_rep_recv(struct dapl_cm_id *conn, + struct ib_cm_event *rep_recv_param) { int status; - if (ep_ptr->cm_handle == IB_INVALID_HANDLE) - return; - - if (ep_ptr->qp_handle == IB_INVALID_HANDLE) { - dapl_dbg_log(DAPL_DBG_TYPE_ERR, " do_active_rep_recv: invalid qp handle\n"); - return; + if (conn->ep->qp_handle == IB_INVALID_HANDLE) { + dapl_dbg_log(DAPL_DBG_TYPE_ERR, " do_rep_recv: invalid qp " + "handle\n"); + goto reject; } /* First, transition QP to RTR */ - status = - dapl_modify_qp_state_to_rtr(ep_ptr->cm_handle, ep_ptr->qp_handle); - if (status) - dapl_dbg_log(DAPL_DBG_TYPE_ERR, - " do_active_rep_recv: could not modify QP state to RTR status %d\n", - status); - else { - /* Now, transition QP to RTS */ - status = - dapl_modify_qp_state_to_rts(ep_ptr->cm_handle, - ep_ptr->qp_handle); - if (status) - dapl_dbg_log(DAPL_DBG_TYPE_ERR, - " do_active_rep_recv: could not modify QP state to RTS status %d\n", - status); + status = dapl_modify_qp_state_to_rtr(conn->cm_id, conn->ep->qp_handle); + if (status) { + dapl_dbg_log(DAPL_DBG_TYPE_ERR, " do_rep_recv: could not " + "modify QP state to RTR status %d\n", status); + goto reject; } - status = ib_send_cm_rtu(comm_id, NULL, 0); - if (status) - dapl_dbg_log(DAPL_DBG_TYPE_ERR, - " do_active_rep_recv: ib_send_cm_rtu failed: %d\n", - status); + /* Now, transition QP to RTS */ + status = dapl_modify_qp_state_to_rts(conn->cm_id, conn->ep->qp_handle); + if (status) { + dapl_dbg_log(DAPL_DBG_TYPE_ERR, " do_rep_recv: could not " + "modify QP state to RTS status %d\n", status); + goto reject; + } - if (ep_ptr->param.connect_evd_handle) - dapl_evd_connection_callback(ep_ptr->cm_handle, - IB_CME_CONNECTED, - rep_recv_param->private_data, - (void *)ep_ptr); + status = ib_send_cm_rtu(conn->cm_id, NULL, 0); + if (status) { + dapl_dbg_log(DAPL_DBG_TYPE_ERR, " do_rep_recv: ib_send_cm_rtu " + "failed: %d\n", status); + goto reject; + } -} + dapl_evd_connection_callback(conn, IB_CME_CONNECTED, + rep_recv_param->private_data, conn->ep); + return; -static void do_active_disc(DAPL_EP * ep_ptr, struct ib_cm_id *comm_id) -{ - dapl_evd_connection_callback(comm_id, - IB_CME_DISCONNECTED, NULL, (void *)ep_ptr); +reject: + ib_send_cm_rej(conn->cm_id, IB_CM_REJ_CONSUMER_DEFINED, NULL, 0, + NULL, 0); + dapl_evd_connection_callback(conn, IB_CME_LOCAL_FAILURE, NULL, + conn->ep); } int dapl_cm_active_cb_handler(struct ib_cm_id *comm_id, struct ib_cm_event *event) { - DAPL_EP *ep_ptr; + struct dapl_cm_id *conn = comm_id->context; + int ret; + unsigned long flags; - ep_ptr = (DAPL_EP *) comm_id->context; - /* Is this check needed ? */ - if (ep_ptr == NULL) { - dapl_dbg_log(DAPL_DBG_TYPE_ERR, - " dapl_cm_active_cb_handler: NULL CM ID\n"); + spin_lock_irqsave(&conn->lock, flags); + if (conn->destroy) { + spin_unlock_irqrestore(&conn->lock, flags); return 0; } + conn->in_callback = 1; + spin_unlock_irqrestore(&conn->lock, flags); - switch (comm_id->state) { - case IB_CM_IDLE: - do_active_idle(ep_ptr); + switch (event->event) { + case IB_CM_REQ_ERROR: + dapl_evd_connection_callback(conn, + IB_CME_DESTINATION_UNREACHABLE, + NULL, conn->ep); break; - case IB_CM_REP_RCVD: - do_active_rep_recv(ep_ptr, comm_id, event); + case IB_CM_REJ_RECEIVED: + dapl_evd_connection_callback(conn, IB_CME_DESTINATION_REJECT, + NULL, conn->ep); break; - case IB_CM_ESTABLISHED: - /* Nothing to do */ + case IB_CM_REP_RECEIVED: + do_rep_recv(conn, event); break; - case IB_CM_DREQ_RCVD: - do_active_disc(ep_ptr, comm_id); + case IB_CM_DREQ_RECEIVED: + ib_send_cm_drep(comm_id, NULL, 0); break; + case IB_CM_DREQ_ERROR: + case IB_CM_DREP_RECEIVED: + /* Wait to exit timewait. */ + break; + case IB_CM_TIMEWAIT_EXIT: + dapl_evd_connection_callback(conn, IB_CME_DISCONNECTED, + NULL, conn->ep); + break; default: dapl_dbg_log(DAPL_DBG_TYPE_ERR, - " dapl_cm_active_cb_handler: Unexpected CM state %d on ID 0x%p event %d\n", - comm_id->state, comm_id, event->event); + " dapl_cm_active_cb_handler: Unexpected CM " + "event %d on ID 0x%p\n", event->event, comm_id); break; } - return 0; + spin_lock_irqsave(&conn->lock, flags); + ret = conn->destroy; + conn->in_callback = conn->destroy; + spin_unlock_irqrestore(&conn->lock, flags); + if (ret) { + if (conn->ep) + conn->ep->cm_handle = IB_INVALID_HANDLE; + kfree(conn); + } + return ret; } -static void do_passive_idle(struct ib_cm_id *comm_id) +static int do_rtu_recv(struct dapl_cm_id *conn) { - ib_cm_events_t cm_event; + int status; - cm_event = IB_CME_DISCONNECTED; + status = dapl_modify_qp_state_to_rts(conn->cm_id, conn->ep->qp_handle); + if (status) { + dapl_dbg_log(DAPL_DBG_TYPE_ERR, " do_rtu_recv: could not " + "modify QP state to RTS status %d\n", status); + goto reject; + } + dapl_cr_callback(conn, IB_CME_CONNECTED, NULL, conn->sp); + return 0; - dapl_cr_callback(comm_id, cm_event, NULL, comm_id->context); - +reject: + ib_send_cm_rej(conn->cm_id, IB_CM_REJ_CONSUMER_DEFINED, NULL, 0, + NULL, 0); + dapl_cr_callback(conn, IB_CME_LOCAL_FAILURE, NULL, conn->sp); + return 1; } -static void do_passive_req_recv(struct ib_cm_id *comm_id, - struct ib_cm_event *req_recv_param) +static struct dapl_cm_id * dapl_get_conn(struct ib_cm_id *comm_id, + struct ib_cm_event *event) { - dapl_cr_callback(comm_id, IB_CME_CONNECTION_REQUEST_PENDING, - req_recv_param->private_data, - comm_id->context); + struct dapl_cm_id *conn; + int status; -} + if (event->event != IB_CM_REQ_RECEIVED) + return comm_id->context; -static void do_passive_established(struct ib_cm_id *comm_id) -{ - DAPL_SP *sp_ptr; - DAPL_CR *cr_ptr; - DAPL_EP *ep_ptr; - int status; - - sp_ptr = comm_id->context; - cr_ptr = dapl_sp_search_cr(sp_ptr, comm_id); - if (cr_ptr == NULL) { - printk(KERN_ERR "do_passive_established: No CR found for CM ID 0x%p\n", comm_id); - return; + conn = kmalloc(sizeof *conn, GFP_KERNEL); + if (!conn) { + status = ib_send_cm_rej(comm_id, + IB_CM_REJ_CONSUMER_DEFINED, + NULL, 0, NULL, 0); + return NULL; } - ep_ptr = (DAPL_EP *)cr_ptr->param.local_ep_handle; - /* Quick check to ensure our EP is still valid */ - if ((DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP))) - ep_ptr = NULL; - if (ep_ptr) { - if (ep_ptr->qp_handle != IB_INVALID_HANDLE) { - /* Transition QP to RTS */ - status = - dapl_modify_qp_state_to_rts(comm_id, - ep_ptr->qp_handle); - if (status) - dapl_dbg_log(DAPL_DBG_TYPE_ERR, - " do_passive_established: could not modify QP state to RTS status %d\n", - status); - } else - printk(KERN_ERR "do_passive_established: invalid QP handle\n"); - } else - printk(KERN_ERR "do_passive_established: No EP found for CM ID 0x%p\n", comm_id); - dapl_cr_callback(comm_id, - IB_CME_CONNECTED, - NULL, (void *)comm_id->context); - + memset(conn, 0, sizeof *conn); + conn->cm_id = comm_id; + conn->sp = comm_id->context; + comm_id->context = conn; + spin_lock_init(&conn->lock); + return conn; } -static void do_passive_disc(struct ib_cm_id *comm_id) -{ - dapl_cr_callback(comm_id, - IB_CME_DISCONNECTED, - NULL, (void *)comm_id->context); -} - int dapl_cm_passive_cb_handler(struct ib_cm_id *comm_id, struct ib_cm_event *event) { - switch (comm_id->state) { - case IB_CM_IDLE: - do_passive_idle(comm_id); + struct dapl_cm_id *conn; + int ret; + unsigned long flags; + + conn = dapl_get_conn(comm_id, event); + if (!conn) { + printk(KERN_ERR "dapl_cm_passive_cb_handler: dapl_get_conn failed\n"); + return 1; + } + + spin_lock_irqsave(&conn->lock, flags); + if (conn->destroy) { + spin_unlock_irqrestore(&conn->lock, flags); + return 0; + } + conn->in_callback = 1; + spin_unlock_irqrestore(&conn->lock, flags); + + switch (event->event) { + case IB_CM_REQ_RECEIVED: + dapl_cr_callback(conn, IB_CME_CONNECTION_REQUEST_PENDING, + event->private_data, conn->sp); break; - case IB_CM_REQ_RCVD: - do_passive_req_recv(comm_id, event); + case IB_CM_REP_ERROR: + dapl_cr_callback(conn, IB_CME_DESTINATION_UNREACHABLE, + NULL, conn->sp); break; - case IB_CM_ESTABLISHED: - do_passive_established(comm_id); + case IB_CM_REJ_RECEIVED: + dapl_cr_callback(conn, IB_CME_DESTINATION_REJECT, NULL, + conn->sp); break; - case IB_CM_DREQ_RCVD: - do_passive_disc(comm_id); + case IB_CM_RTU_RECEIVED: + do_rtu_recv(conn); break; + /* TODO: case IB_CM_USER_ESTABLISHED: - when initiated by DAPL */ + case IB_CM_DREQ_RECEIVED: + ib_send_cm_drep(comm_id, NULL, 0); + break; + case IB_CM_DREQ_ERROR: + case IB_CM_DREP_RECEIVED: + /* Wait to exit timewait. */ + break; + case IB_CM_TIMEWAIT_EXIT: + dapl_cr_callback(conn, IB_CME_DISCONNECTED, NULL, conn->sp); + break; default: - dapl_dbg_log(DAPL_DBG_TYPE_ERR, - " dapl_cm_passive_cb_handler: Unexpected CM state %d on ID 0x%p event %d\n", - comm_id->state, comm_id, event->event); + dapl_dbg_log(DAPL_DBG_TYPE_ERR, " dapl_cm_passive_cb_handler: " + "Unexpected CM event %d on ID 0x%p\n", + event->event, comm_id); break; } - return 0; + spin_lock_irqsave(&conn->lock, flags); + ret = conn->destroy; + conn->in_callback = conn->destroy; + spin_unlock_irqrestore(&conn->lock, flags); + if (ret) { + if (conn->ep) + conn->ep->cm_handle = IB_INVALID_HANDLE; + kfree(conn); + } + return ret; } -static void dapl_ib_destroy_cm_id_work(void *data) +static void dapl_path_comp_handler(u64 req_id, void *context, int rec_num) { - struct dapl_ib_destroy_cm_id *work = (struct dapl_ib_destroy_cm_id *)data; - struct ib_cm_id *cm_id = work->cm_id; + struct dapl_cm_id *conn = context; + int status; + ib_cm_events_t event; - ib_destroy_cm_id(cm_id); /* invokes disconnect sequence if not already done */ - kfree(work); + if (rec_num <= 0) { + printk(KERN_ERR "dapl_path_comp_handler: path resolution " + "failed (%d) retries %d!!!\n", rec_num, conn->retries); + if (++conn->retries > OPENIB_MAX_AT_RETRY) { + printk(KERN_ERR "dapl_path_comp_handler: ep_ptr 0x%p\n", + conn->ep); + event = IB_CME_DESTINATION_UNREACHABLE; + goto error; + } + + status = ib_at_paths_by_route(&conn->dapl_rt, 0, + &conn->dapl_path, 1, + &conn->dapl_comp); + if (status) { + printk(KERN_ERR "dapl_path_comp_handler: " + "ib_at_paths_by_route returned %d id %lld\n", + status, conn->dapl_comp.req_id); + event = IB_CME_LOCAL_FAILURE; + goto error; + } + return; + } + + /* Mellanox performance workaround - best performance is MTU of 1024 */ + if (conn->dapl_path.mtu > IB_MTU_1024) + conn->dapl_path.mtu = IB_MTU_1024; + + conn->param.service_id = be64_to_cpu(conn->service_id); + conn->param.primary_path = &conn->dapl_path; + conn->param.alternate_path = NULL; + + status = ib_send_cm_req(conn->cm_id, &conn->param); + if (status) { + dapl_dbg_log(DAPL_DBG_TYPE_ERR, " ib_send_cm_req failed: %d\n", + status); + event = IB_CME_LOCAL_FAILURE; + goto error; + } + return; + +error: + //*** should we pass conn here? it will be destroyed... + dapl_evd_connection_callback(conn, event, NULL, conn->ep); + dapl_destroy_cm_id(conn); } -static int dapl_ib_queue_destroy_cm_id(struct ib_cm_id *cm_id) +static void dapl_rt_comp_handler(u64 req_id, void *context, int rec_num) { - struct dapl_ib_destroy_cm_id *work; + struct dapl_cm_id *conn = context; + int status; + ib_cm_events_t event; - work = kmalloc(sizeof *work, GFP_ATOMIC); - if (!work) { - printk(KERN_ERR "dapl_ib_queue_destroy_cm_id: No memory for dapl_ib_destroy_cm_id\n"); - return DAT_INSUFFICIENT_RESOURCES; + if (rec_num <= 0) { + printk(KERN_ERR "dapl_rt_comp_handler: rec_num %d retry %d\n", + rec_num, conn->retries); + if (++conn->retries > OPENIB_MAX_AT_RETRY) { + event = IB_CME_DESTINATION_UNREACHABLE; + goto error; + } + + status = ib_at_route_by_ip( ((struct sockaddr_in *) + conn->remote_ia_address)->sin_addr.s_addr, + 0, 0, 0, &conn->dapl_rt, &conn->dapl_comp); + if (status < 0) { + dapl_dbg_log(DAPL_DBG_TYPE_ERR, "dapl_rt_comp_handler: " + "ib_at_route_by_ip failed with status %d\n", + status); + event = IB_CME_DESTINATION_UNREACHABLE; + goto error; + } + + if (status == 1) + dapl_rt_comp_handler(conn->dapl_comp.req_id, conn, 1); + return; } - work->cm_id = cm_id; - INIT_WORK(&work->work, dapl_ib_destroy_cm_id_work, work); - queue_work(dapl_cm_wq, &work->work); - return 0; + conn->dapl_comp.fn = &dapl_path_comp_handler; + conn->dapl_comp.context = conn; + conn->retries = 0; + status = ib_at_paths_by_route(&conn->dapl_rt, 0, &conn->dapl_path, 1, + &conn->dapl_comp); + if (status) { + printk(KERN_ERR "dapl_rt_comp_handler: ib_at_paths_by_route " + "returned %d id %lld\n", status, conn->dapl_comp.req_id); + event = IB_CME_LOCAL_FAILURE; + goto error; + } + return; + +error: + //*** should we pass conn here? it will be destroyed... + dapl_evd_connection_callback(conn, event, NULL, conn->ep); + dapl_destroy_cm_id(conn); } /* @@ -293,7 +404,7 @@ { DAPL_IA *ia_ptr; DAPL_EP *ep_ptr; - struct dapl_ib_connection *conn; + struct dapl_cm_id *conn; int status; ep_ptr = (DAPL_EP *)ep_handle; @@ -302,30 +413,30 @@ return DAT_INTERNAL_ERROR; } - /* Is this always called out of task context ? */ - /* If so, change to GFP_KERNEL */ - conn = kmalloc(sizeof *conn, GFP_ATOMIC); + conn = kmalloc(sizeof *conn, GFP_KERNEL); if (!conn) { printk(KERN_ERR - "dapl_ib_connect: no memory for dapl_ib_connection struct\n"); + "dapl_ib_connect: no memory for dapl_cm_id struct\n"); return DAT_INSUFFICIENT_RESOURCES; } + memset(conn, 0, sizeof *conn); - conn->ep_ptr = ep_ptr; - conn->ep_ptr->cm_handle = ib_create_cm_id(dapl_cm_active_cb_handler, - ep_handle); - if (!conn->ep_ptr->cm_handle) { + spin_lock_init(&conn->lock); + conn->ep = ep_ptr; + conn->cm_id = ib_create_cm_id(dapl_cm_active_cb_handler, conn); + if (IS_ERR(conn->cm_id)) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, " CM ID creation failed\n"); kfree(conn); return DAT_INTERNAL_ERROR; } + conn->ep->cm_handle = conn; - ia_ptr = conn->ep_ptr->header.owner_ia; + ia_ptr = conn->ep->header.owner_ia; /* Setup QP/CM parameters */ memset(&conn->param, 0, sizeof(conn->param)); conn->param.qp_type = IB_QPT_RC; - conn->param.qp_num = conn->ep_ptr->qp_handle->qp_num; + conn->param.qp_num = conn->ep->qp_handle->qp_num; conn->param.private_data = private_data; conn->param.private_data_len = private_data_size; conn->param.responder_resources = IB_TARGET_MAX; @@ -348,126 +459,18 @@ sin_addr.s_addr, 0, 0, 0, &conn->dapl_rt, &conn->dapl_comp); if (status < 0) { - dapl_dbg_log(DAPL_DBG_TYPE_ERR, " ib_at_route_by_ip failed with status %d\n", status); - kfree(conn); - /* What happens with CM ID in this case ? */ + dapl_dbg_log(DAPL_DBG_TYPE_ERR, " ib_at_route_by_ip failed " + "with status %d\n", status); + dapl_destroy_cm_id(conn); return DAT_INTERNAL_ERROR; } if (status == 1) - dapl_rt_comp_handler(conn->dapl_comp.req_id, - conn->dapl_comp.context, 1); + dapl_rt_comp_handler(conn->dapl_comp.req_id, conn, 1); return DAT_SUCCESS; } -static void dapl_rt_comp_handler(u64 req_id, void *context, int rec_num) -{ - struct dapl_ib_connection *conn = context; - int status; - - if (rec_num <= 0) { - printk(KERN_ERR "dapl_rt_comp_handler: rec_num %d retry %d\n", rec_num, conn->retries); - if (++conn->retries > OPENIB_MAX_AT_RETRY) { - dapl_evd_connection_callback(conn->ep_ptr->cm_handle, - IB_CME_DESTINATION_UNREACHABLE, - NULL, - conn->ep_ptr); - kfree(conn); - return; - } else { - status = - ib_at_route_by_ip( - ((struct sockaddr_in *)conn-> - remote_ia_address)->sin_addr.s_addr, - 0, 0, 0, &conn->dapl_rt, - &conn->dapl_comp); - if (status < 0) { - dapl_dbg_log(DAPL_DBG_TYPE_ERR, " dapl_rt_comp_handler: ib_at_route_by_ip failed with status %d\n", status); - dapl_evd_connection_callback(conn->ep_ptr->cm_handle, - IB_CME_DESTINATION_UNREACHABLE, - NULL, - conn->ep_ptr); - kfree(conn); - return; - } - if (status == 1) - dapl_rt_comp_handler(conn->dapl_comp.req_id, - conn->dapl_comp.context, 1); - return; - } - } - - conn->dapl_comp.fn = &dapl_path_comp_handler; - conn->dapl_comp.context = conn; - conn->retries = 0; - status = ib_at_paths_by_route(&conn->dapl_rt, 0, &conn->dapl_path, - 1, &conn->dapl_comp); - if (status) { - printk(KERN_ERR - "dapl_rt_comp_handler: ib_at_paths_by_route returned %d id %lld\n", - status, conn->dapl_comp.req_id); - dapl_evd_connection_callback(conn->ep_ptr->cm_handle, - IB_CME_LOCAL_FAILURE, - NULL, - conn->ep_ptr); - kfree(conn); - } -} - -static void dapl_path_comp_handler(u64 req_id, void *context, int rec_num) -{ - struct dapl_ib_connection *conn = context; - int status; - - if (rec_num <= 0) { - printk(KERN_ERR "dapl_path_comp_handler: path resolution failed (%d) retries %d!!!\n", rec_num, conn->retries); - if (++conn->retries > OPENIB_MAX_AT_RETRY) { -printk(KERN_ERR "dapl_path_comp_handler: ep_ptr 0x%p\n", conn->ep_ptr); - dapl_evd_connection_callback(conn->ep_ptr->cm_handle, - IB_CME_DESTINATION_UNREACHABLE, - NULL, - conn->ep_ptr); - kfree(conn); - return; - } else { - status = ib_at_paths_by_route(&conn->dapl_rt, 0, - &conn->dapl_path, - 1, &conn->dapl_comp); - if (status) { - printk(KERN_ERR - "dapl_path_comp_handler: ib_at_paths_by_route returned %d id %lld\n", - status, conn->dapl_comp.req_id); - dapl_evd_connection_callback(conn->ep_ptr->cm_handle, - IB_CME_LOCAL_FAILURE, - NULL, - conn->ep_ptr); - kfree(conn); - } - return; - } - } - - /* Mellanox performance workaround */ - if (conn->dapl_path.mtu > IB_MTU_1024) - conn->dapl_path.mtu = IB_MTU_1024; /* Best performance is obtained at this MTU */ - - conn->param.service_id = be64_to_cpu(conn->service_id); - conn->param.primary_path = &conn->dapl_path; - conn->param.alternate_path = NULL; - - status = ib_send_cm_req(conn->ep_ptr->cm_handle, &conn->param); - if (status) { - dapl_dbg_log(DAPL_DBG_TYPE_ERR, " ib_send_cm_req failed: %d\n", - status); - dapl_evd_connection_callback(conn->ep_ptr->cm_handle, - IB_CME_LOCAL_FAILURE, NULL, - conn->ep_ptr); - } - - kfree(conn); -} - /* * dapl_ib_disconnect * @@ -488,6 +491,25 @@ */ DAT_RETURN dapl_ib_disconnect(DAPL_EP * ep_ptr, DAT_CLOSE_FLAGS close_flags) { + struct dapl_cm_id *conn = ep_ptr->cm_handle; + int status; + + dapl_dbg_log(DAPL_DBG_TYPE_CM, + " >>> dapl_ib_disconnect: EP: %p conn %p close flags %d\n", + ep_ptr, conn, close_flags); + + if (conn == IB_INVALID_HANDLE) + return DAT_SUCCESS; + + if (close_flags == DAT_CLOSE_ABRUPT_FLAG) + dapl_destroy_cm_id(conn); + else { + status = ib_send_cm_dreq(conn->cm_id, NULL, 0); + if (status) + printk(KERN_ERR "dapl_ib_disconnect: CM ID 0x%p " + "status %d\n", ep_ptr->cm_handle, status); + } + return DAT_SUCCESS; } @@ -516,27 +538,24 @@ { int status; - if (ep_ptr->cm_handle == IB_INVALID_HANDLE) - return; + dapl_dbg_log(DAPL_DBG_TYPE_CM, + " >>> dapl_ib_disconnect_clean: EP: %p active %d\n", + ep_ptr, active); - dapl_dbg_log(DAPL_DBG_TYPE_CM, - " >>> dapl_ib_disconnect_clean: CM handle: %p\n", - ep_ptr->cm_handle); - /* * Clean up outstanding connection state */ + dapl_ib_disconnect(ep_ptr, DAT_CLOSE_ABRUPT_FLAG); + if (ep_ptr->qp_handle != IB_INVALID_HANDLE) { /* Transition QP to ERROR */ status = dapl_modify_qp_state_to_error(ep_ptr->qp_handle); if (status) - printk(KERN_ERR "dapl_ib_disconnect_clean: status %d on transition to error\n", status); + printk(KERN_ERR "dapl_ib_disconnect_clean: status %d " + "on transition to error\n", status); } else - dapl_dbg_log(DAPL_DBG_TYPE_ERR, "dapl_ib_disconnect_clean: invalid qp handle\n"); - - dapl_ib_queue_destroy_cm_id(ep_ptr->cm_handle); - - ep_ptr->cm_handle = IB_INVALID_HANDLE; + dapl_dbg_log(DAPL_DBG_TYPE_ERR, "dapl_ib_disconnect_clean: " + "invalid qp handle\n"); } /* @@ -561,6 +580,13 @@ ia_ptr = ep_ptr->header.owner_ia; + //*** What if QP is connected? Do we need to destroy the cm_id + //*** here as well, send a DREQ, REJ? What cm_id state are we in? + //*** we may not care if we just destroy the cm_id + //*** ib_destroy_cm_id(ep_ptr->cm_handle); + //*** supporting this call may require setting the cm_handle to + //*** something like IB_INVALID_HANDLE wherever it is destroyed + ib_status = dapl_modify_qp_state_to_reset(ep_ptr->qp_handle); if (ib_status < 0) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, @@ -604,7 +630,7 @@ sp_ptr->cm_srvc_handle = ib_create_cm_id(dapl_cm_passive_cb_handler, sp_ptr); - if (sp_ptr->cm_srvc_handle == NULL) { + if (IS_ERR(sp_ptr->cm_srvc_handle)) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, " CM ID creation failed\n"); return DAT_INTERNAL_ERROR; } @@ -612,7 +638,7 @@ status = ib_cm_listen(sp_ptr->cm_srvc_handle, be64_to_cpu(ServiceID), 0); if (status) { - /* Need to also destroy CM ID ? */ + ib_destroy_cm_id(sp_ptr->cm_srvc_handle); sp_ptr->cm_srvc_handle = IB_INVALID_HANDLE; if (status == -EBUSY) @@ -643,19 +669,12 @@ */ DAT_RETURN dapl_ib_remove_conn_listener(DAPL_IA * ia_ptr, DAPL_SP * sp_ptr) { - int status; - - if (sp_ptr->cm_srvc_handle == IB_INVALID_HANDLE) - return DAT_SUCCESS; - - status = dapl_ib_queue_destroy_cm_id(sp_ptr->cm_srvc_handle); - sp_ptr->cm_srvc_handle = IB_INVALID_HANDLE; - if (status) { - dapl_dbg_log(DAPL_DBG_TYPE_ERR, - " dapl_ib_remove_conn_listener: ib_destroy_cm_id failed: %d\n", status); - return DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, 0); + //*** This will hang if called from CM thread context... + //*** Move back to using WQ... + if (sp_ptr->cm_srvc_handle != IB_INVALID_HANDLE) { + ib_destroy_cm_id(sp_ptr->cm_srvc_handle); + sp_ptr->cm_srvc_handle = IB_INVALID_HANDLE; } - return DAT_SUCCESS; } @@ -677,26 +696,25 @@ * */ DAT_RETURN -dapl_ib_reject_connection(struct ib_cm_id * ib_cm_handle, int reject_reason) +dapl_ib_reject_connection(struct dapl_cm_id *cm_handle, int reject_reason) { int status; - if (ib_cm_handle == IB_INVALID_HANDLE) { + if (cm_handle == IB_INVALID_HANDLE) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, " dapl_ib_reject_connection: invalid CM handle: reject reason %d\n", reject_reason); return DAT_SUCCESS; } - status = ib_send_cm_rej(ib_cm_handle, IB_CM_REJ_CONSUMER_DEFINED, + status = ib_send_cm_rej(cm_handle->cm_id, IB_CM_REJ_CONSUMER_DEFINED, NULL, 0, NULL, 0); if (status) { - dapl_dbg_log(DAPL_DBG_TYPE_ERR, - " dapl_ib_reject_connection: ib_send_cm_rej failed: %d\n", - status); + dapl_dbg_log(DAPL_DBG_TYPE_ERR, " dapl_ib_reject_connection: " + "ib_send_cm_rej failed: %d\n", status); return DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, 0); } - + dapl_destroy_cm_id(cm_handle); return DAT_SUCCESS; } @@ -731,14 +749,18 @@ DAT_RETURN dat_status; int status; struct ib_cm_rep_param passive_params; + struct dapl_cm_id *conn; cr_ptr = (DAPL_CR *) cr_handle; ep_ptr = (DAPL_EP *) ep_handle; ia_ptr = ep_ptr->header.owner_ia; + conn = cr_ptr->ib_cm_handle; /* Obtain size of private data structure & contents */ - if (private_data_size > IB_MAX_REP_PDATA_SIZE) - return DAT_ERROR(DAT_LENGTH_ERROR, DAT_NO_SUBTYPE); + if (private_data_size > IB_MAX_REP_PDATA_SIZE) { + dat_status = DAT_ERROR(DAT_LENGTH_ERROR, DAT_NO_SUBTYPE); + goto reject; + } if (ep_ptr->qp_state == DAPL_QP_STATE_UNATTACHED) { /* @@ -749,14 +771,16 @@ dat_status = dapl_ib_qp_alloc(ia_ptr, ep_ptr, NULL); if (dat_status != DAT_SUCCESS) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, - " dapl_ib_accept_connection: dapl_ib_qp_alloc failed: %d\n", + " dapl_ib_accept_connection: " + "dapl_ib_qp_alloc failed: %d\n", dat_status); - return dat_status; + goto reject; } } cr_ptr->param.local_ep_handle = ep_handle; - ep_ptr->cm_handle = cr_ptr->ib_cm_handle; + ep_ptr->cm_handle = conn; + conn->ep = ep_ptr; memset(&passive_params, 0, sizeof(passive_params)); passive_params.private_data = prd_ptr; @@ -767,28 +791,35 @@ passive_params.rnr_retry_count = DAPL_OPENIB_RNR_RETRY_COUNT; /* Transition QP to RTR */ - status = - dapl_modify_qp_state_to_rtr(ep_ptr->cm_handle, ep_ptr->qp_handle); - if (status) - dapl_dbg_log(DAPL_DBG_TYPE_ERR, - " dapl_ib_accept_connection: could not modify QP state to RTR status %d\n", + status = dapl_modify_qp_state_to_rtr(conn->cm_id, ep_ptr->qp_handle); + if (status) { + dapl_dbg_log(DAPL_DBG_TYPE_ERR, " dapl_ib_accept_connection: " + "could not modify QP state to RTR status %d\n", status); + dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, 0); + goto reject; + } - status = ib_send_cm_rep(cr_ptr->ib_cm_handle, &passive_params); + status = ib_send_cm_rep(conn->cm_id, &passive_params); if (status) { - dapl_dbg_log(DAPL_DBG_TYPE_ERR, - " dapl_ib_accept_connection: ib_send_cm_rep failed: %d\n", - status); - return DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, 0); + dapl_dbg_log(DAPL_DBG_TYPE_ERR, " dapl_ib_accept_connection: " + "ib_send_cm_rep failed: %d\n", status); + dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, 0); + goto reject; } + return DAT_SUCCESS; - return DAT_SUCCESS; +reject: + ib_send_cm_rej(conn->cm_id, IB_CM_REJ_CONSUMER_DEFINED, NULL, 0, + NULL, 0); + dapl_destroy_cm_id(conn); + return dat_status; } /* * ib_cm_get_remote_gid */ -static int ib_cm_get_remote_gid(struct ib_cm_id *cm_handle, u8 *remote_gid) +static int ib_cm_get_remote_gid(struct dapl_cm_id *cm_handle, u8 *remote_gid) { return DAT_INTERNAL_ERROR; /* for now!!! */ } @@ -837,19 +868,9 @@ int dapl_cm_init(void) { - dapl_cm_wq = create_workqueue("dapl_cm_wq"); - if (!dapl_cm_wq) { - printk(KERN_ERR "dapl_cm_init: failed to allocate work queue for DAPL CM\n"); - return DAT_INSUFFICIENT_RESOURCES; - } return 0; } void dapl_cm_cleanup(void) { - /* Need to handle in progress connections */ - if (dapl_cm_wq) { - flush_workqueue(dapl_cm_wq); - destroy_workqueue(dapl_cm_wq); - } } From blist at aon.at Wed May 11 06:00:04 2005 From: blist at aon.at (Bernhard Fischer) Date: Wed, 11 May 2005 15:00:04 +0200 Subject: [openib-general] perftest-03: put C code in .c file In-Reply-To: <20050511014014.GG5495@esmail.cup.hp.com> References: <20050511014014.GG5495@esmail.cup.hp.com> Message-ID: <20050511130004.GB7621@aon.at> Hi, On Tue, May 10, 2005 at 06:40:14PM -0700, Grant Grundler wrote: >Hi Michael, >Patch below fixes up the Makefile and getclock.h so C code is in C files >--- Makefile (revision 2296) >+++ Makefile (working copy) >@@ -1,6 +1,9 @@ > CFLAGS = -Wall -O2 -g -D_GNU_SOURCE + CC = gcc > >-rdma_lat: rdma_lat.c get_clock.h >- gcc $(CFLAGS) -o rdma_lat rdma_lat.c -libverbs please replace that gcc with $(CC) >+RDMA_LAT_OBJ = rdma_lat.o get_clock.o >+ >+rdma_lat: $(RDMA_LAT_OBJ) >+ gcc -o rdma_lat $(RDMA_LAT_OBJ) -libverbs ditto >+ > clean: >- rm -f rdma_lat >+ rm -f rdma_lat $(RDMA_LAT_OBJ) thank you From jlentini at netapp.com Wed May 11 07:49:53 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 11 May 2005 10:49:53 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: remove DAT_BOOLEAN In-Reply-To: <20050511080547.GC13349@mellanox.co.il> References: <1115747465.8222.15.camel@duffman> <20050511080547.GC13349@mellanox.co.il> Message-ID: > Quoting r. Tom Duffy : >> Subject: [PATCH] kDAPL: remove DAT_BOOLEAN >> >> I am sending this as a separate patch because I am not sure what the >> "right" thing to do is. Looking around the kernel, it seems booleans >> are handled in different ways. Some places, they are enums, others just >> #defines. >> >> In e1000, ixgb, compaq fc, DAT960, and xfs, they are enums. In jfs and >> arch parisc, a bool is an int and TRUE and FALSE are #define'd. TRUE >> and FALSE are defined so many times in the kernel, it seems rather >> ridiculous. Maybe the right thing to do is fix the whole kernel in a >> consistent way and submit an upstream patch, but this will require quite >> a bit of janitorial work. > > Further, include/linux/synclink.h has > #define BOOLEAN int > > and include/linux/arcdevice.h has > #define bool int > >> I don't mind using either way in kDAPL (or some third -- getting rid of >> bool's all together?),but this patch uses the enum method. At least we >> need to get rid of the DAT_ prefix. > > Is this type ever packed and passed on the wire? If yes that would > be a valid reason for something DAT_BOOLEAN, to make sure it has > a specific size. The DAT API does not mandate any wire protocols so this should not be an issue. >> Singed-off-by: Tom Duffy > > Surprisingly, some DAT code just uses "bool" (below). Good point. This is test code, but we should still change it to be consistent. > >> Index: linux-kernel/test/dapltest/include/dapl_proto.h >> =================================================================== >> --- linux-kernel/test/dapltest/include/dapl_proto.h (revision 2295) >> +++ linux-kernel/test/dapltest/include/dapl_proto.h (working copy) >> @@ -88,8 +88,8 @@ Bpool * DT_BpoolAlloc (Per_Test_ >> DAT_COUNT seg_size, >> DAT_COUNT num_segs, >> DAT_COUNT alignment, >> - DAT_BOOLEAN enable_rdma_write, >> - DAT_BOOLEAN enable_rdma_read); >> + boolean_t enable_rdma_write, >> + boolean_t enable_rdma_read); >> >> bool DT_Bpool_Destroy (Per_Test_Data_t * pt_ptr, >> DT_Tdep_Print_Head *phead, >> @@ -267,13 +267,13 @@ bool DT_Performance_Test_Serv >> bool DT_Performance_Test_Create (Per_Test_Data_t * pt_ptr, >> DAT_IA_HANDLE * ia_handle, >> DAT_IA_ADDRESS_PTR remote_ia_addr, >> - DAT_BOOLEAN is_server, >> - DAT_BOOLEAN is_remote_little_endian, >> + boolean_t is_server, >> + boolean_t is_remote_little_endian, >> Performance_Test_t **perf_test); >> > > -- > MST - Michael S. Tsirkin > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From webmaster at thaipd.com Wed May 11 06:52:53 2005 From: webmaster at thaipd.com (webmaster at thaipd.com) Date: 11 May 2005 13:52:53 -0000 Subject: [openib-general] =?iso-8859-1?q?=C5=A7=E2=A6=C9=B3=D2=BF=C3=D5_!?= Message-ID: <20050511135253.23289.qmail@ns1.mtsyber.com> ขอความกรุณา ร่วมแสดงความคิดเห็นด้วยครับ http://www.thaipd.com =========== e-mail นี้ส่งเพื่อต้องการข้อคิดเห็นของท่าน หากเป็นการรบกวน ต้องขออภัยมา ณ. ที่นี้ด้วย ขอบคุณมากครับ ========== From iod00d at hp.com Wed May 11 07:58:50 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 11 May 2005 07:58:50 -0700 Subject: [openib-general] perftest-03: put C code in .c file In-Reply-To: <1115776100.5974.6.camel@duffman> References: <20050511014014.GG5495@esmail.cup.hp.com> <1115776100.5974.6.camel@duffman> Message-ID: <20050511145850.GA10070@esmail.cup.hp.com> On Tue, May 10, 2005 at 06:48:20PM -0700, Tom Duffy wrote: > Do a "svn add" on the new file. You don't need check in privileges to > do this. ah thanks! Here's a new perftest-03 patch (with get_clock.c). grant diff -u get_clock.h get_clock.h --- get_clock.h (working copy) +++ get_clock.h (working copy) @@ -32,9 +32,6 @@ * $Id$ */ -#include -#include - #if defined (__x86_64__) || defined(__i386__) typedef unsigned long long cycles_t; static inline cycles_t get_clock() @@ -71,29 +68,2 @@ -double get_cpu_mhz() -{ - FILE* f; - char buf[256]; - double mhz = 0.0; - - f = fopen("/proc/cpuinfo","r"); - if (!f) - return 0.0; - while(fgets(buf, sizeof(buf), f)) { - double m; - int rc; - rc = sscanf(buf, "cpu MHz : %lf", &m); - if (rc != 1) - continue; - if (mhz == 0.0) { - mhz = m; - continue; - } - if (mhz != m) { - fprintf(stderr,"Conflicting CPU frequency values " - " detected: %lf != %lf\n", - mhz, m); - return 0.0; - } - } - fclose(f); - return mhz; -} + +extern double get_cpu_mhz(void); only in patch2: unchanged: --- Makefile (revision 2302) +++ Makefile (working copy) @@ -1,6 +1,9 @@ CFLAGS = -Wall -O2 -g -D_GNU_SOURCE -rdma_lat: rdma_lat.c get_clock.h - gcc $(CFLAGS) -o rdma_lat rdma_lat.c -libverbs +RDMA_LAT_OBJ = rdma_lat.o get_clock.o + +rdma_lat: $(RDMA_LAT_OBJ) + gcc -o rdma_lat $(RDMA_LAT_OBJ) -libverbs + clean: - rm -f rdma_lat + rm -f rdma_lat $(RDMA_LAT_OBJ) only in patch2: unchanged: --- get_clock.c (revision 0) +++ get_clock.c (revision 0) @@ -0,0 +1,66 @@ +/* + * Copyright (c) 2005 Mellanox Technologies Ltd. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + * + * $Id$ + */ + +#include +#include + +double get_cpu_mhz() +{ + FILE* f; + char buf[256]; + double mhz = 0.0; + + f = fopen("/proc/cpuinfo","r"); + if (!f) + return 0.0; + while(fgets(buf, sizeof(buf), f)) { + double m; + int rc; + rc = sscanf(buf, "cpu MHz : %lf", &m); + if (rc != 1) + continue; + if (mhz == 0.0) { + mhz = m; + continue; + } + if (mhz != m) { + fprintf(stderr,"Conflicting CPU frequency values " + " detected: %lf != %lf\n", + mhz, m); + return 0.0; + } + } + fclose(f); + return mhz; +} From iod00d at hp.com Wed May 11 08:05:43 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 11 May 2005 08:05:43 -0700 Subject: [openib-general] Re: perftest-03: put C code in .c file In-Reply-To: <20050511063809.GA13349@mellanox.co.il> References: <20050511014014.GG5495@esmail.cup.hp.com> <20050511063809.GA13349@mellanox.co.il> Message-ID: <20050511150543.GB10070@esmail.cup.hp.com> On Wed, May 11, 2005 at 09:38:09AM +0300, Michael S. Tsirkin wrote: > You have to svn add before svn diff -x --diff-cmd=/usr/bin/diff -x -upN certainly - Appended below > Grant, could you please resend, while also adding a Signed-off-by line? I've been ommitting the S-o-B line since I realized this isn't kernel code. I'm happy to include them if you like. Can we follow Documentation/Codingstyle too? thanks, grant Signed-off-by: Grant Grundler diff -u get_clock.h get_clock.h --- get_clock.h (working copy) +++ get_clock.h (working copy) @@ -32,9 +32,6 @@ * $Id$ */ -#include -#include - #if defined (__x86_64__) || defined(__i386__) typedef unsigned long long cycles_t; static inline cycles_t get_clock() @@ -71,29 +68,2 @@ -double get_cpu_mhz() -{ - FILE* f; - char buf[256]; - double mhz = 0.0; - - f = fopen("/proc/cpuinfo","r"); - if (!f) - return 0.0; - while(fgets(buf, sizeof(buf), f)) { - double m; - int rc; - rc = sscanf(buf, "cpu MHz : %lf", &m); - if (rc != 1) - continue; - if (mhz == 0.0) { - mhz = m; - continue; - } - if (mhz != m) { - fprintf(stderr,"Conflicting CPU frequency values " - " detected: %lf != %lf\n", - mhz, m); - return 0.0; - } - } - fclose(f); - return mhz; -} + +extern double get_cpu_mhz(void); only in patch2: unchanged: --- Makefile (revision 2302) +++ Makefile (working copy) @@ -1,6 +1,9 @@ CFLAGS = -Wall -O2 -g -D_GNU_SOURCE -rdma_lat: rdma_lat.c get_clock.h - gcc $(CFLAGS) -o rdma_lat rdma_lat.c -libverbs +RDMA_LAT_OBJ = rdma_lat.o get_clock.o + +rdma_lat: $(RDMA_LAT_OBJ) + gcc -o rdma_lat $(RDMA_LAT_OBJ) -libverbs + clean: - rm -f rdma_lat + rm -f rdma_lat $(RDMA_LAT_OBJ) --- get_clock.c (revision 0) +++ get_clock.c (revision 0) @@ -0,0 +1,66 @@ +/* + * Copyright (c) 2005 Mellanox Technologies Ltd. All rights reserved. + * + * This software is available to you under a choice of one of two + * licenses. You may choose to be licensed under the terms of the GNU + * General Public License (GPL) Version 2, available from the file + * COPYING in the main directory of this source tree, or the + * OpenIB.org BSD license below: + * + * Redistribution and use in source and binary forms, with or + * without modification, are permitted provided that the following + * conditions are met: + * + * - Redistributions of source code must retain the above + * copyright notice, this list of conditions and the following + * disclaimer. + * + * - Redistributions in binary form must reproduce the above + * copyright notice, this list of conditions and the following + * disclaimer in the documentation and/or other materials + * provided with the distribution. + * + * THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, + * EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF + * MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND + * NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS + * BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN + * ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN + * CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE + * SOFTWARE. + * + * $Id$ + */ + +#include +#include + +double get_cpu_mhz() +{ + FILE* f; + char buf[256]; + double mhz = 0.0; + + f = fopen("/proc/cpuinfo","r"); + if (!f) + return 0.0; + while(fgets(buf, sizeof(buf), f)) { + double m; + int rc; + rc = sscanf(buf, "cpu MHz : %lf", &m); + if (rc != 1) + continue; + if (mhz == 0.0) { + mhz = m; + continue; + } + if (mhz != m) { + fprintf(stderr,"Conflicting CPU frequency values " + " detected: %lf != %lf\n", + mhz, m); + return 0.0; + } + } + fclose(f); + return mhz; +} From iod00d at hp.com Wed May 11 08:11:05 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 11 May 2005 08:11:05 -0700 Subject: [openib-general] perftest-03: put C code in .c file In-Reply-To: <20050511130004.GB7621@aon.at> References: <20050511014014.GG5495@esmail.cup.hp.com> <20050511130004.GB7621@aon.at> Message-ID: <20050511151105.GC10070@esmail.cup.hp.com> On Wed, May 11, 2005 at 03:00:04PM +0200, Bernhard Fischer wrote: > >- gcc $(CFLAGS) -o rdma_lat rdma_lat.c -libverbs > > please replace that gcc with $(CC) yes, good idea. Patch below (perftest-04) removes the line. Applies on top of perftest-03. thanks, grant Signed-off-by: Grant Grundler diff -u Makefile Makefile --- Makefile (working copy) +++ Makefile (working copy) @@ -1,9 +1,9 @@ CFLAGS = -Wall -O2 -g -D_GNU_SOURCE +LDFLAGS = -libverbs RDMA_LAT_OBJ = rdma_lat.o get_clock.o rdma_lat: $(RDMA_LAT_OBJ) - gcc -o rdma_lat $(RDMA_LAT_OBJ) -libverbs clean: rm -f rdma_lat $(RDMA_LAT_OBJ) From jlentini at netapp.com Wed May 11 08:21:45 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 11 May 2005 11:21:45 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: remove DAT_BOOLEAN In-Reply-To: <1115747465.8222.15.camel@duffman> References: <1115747465.8222.15.camel@duffman> Message-ID: tduffy> I am sending this as a separate patch because I am not sure what the tduffy> "right" thing to do is. Looking around the kernel, it seems booleans tduffy> are handled in different ways. Some places, they are enums, others just tduffy> #defines. tduffy> tduffy> In e1000, ixgb, compaq fc, DAT960, and xfs, they are enums. In jfs and tduffy> arch parisc, a bool is an int and TRUE and FALSE are #define'd. TRUE tduffy> and FALSE are defined so many times in the kernel, it seems rather tduffy> ridiculous. I agree. tduffy> Maybe the right thing to do is fix the whole kernel in a tduffy> consistent way and submit an upstream patch, but this will tduffy> require quite a bit of janitorial work. That would be a noble act. tduffy> I don't mind using either way in kDAPL (or some third -- getting rid of tduffy> bool's all together?), but this patch uses the enum method. At least we tduffy> need to get rid of the DAT_ prefix. I like enums. I've commited this in revision 2303. tduffy> tduffy> Singed-off-by: Tom Duffy tduffy> tduffy> Index: linux-kernel/test/dapltest/test/dapl_performance_util.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_performance_util.c (revision 2295) tduffy> +++ linux-kernel/test/dapltest/test/dapl_performance_util.c (working copy) tduffy> @@ -35,8 +35,8 @@ DT_Performance_Test_Create ( tduffy> Per_Test_Data_t *pt_ptr, tduffy> DAT_IA_HANDLE *ia_handle, tduffy> DAT_IA_ADDRESS_PTR remote_ia_addr, tduffy> - DAT_BOOLEAN is_server, tduffy> - DAT_BOOLEAN is_remote_little_endian, tduffy> + boolean_t is_server, tduffy> + boolean_t is_remote_little_endian, tduffy> Performance_Test_t **perf_test) tduffy> { tduffy> Performance_Test_t *test_ptr; tduffy> @@ -266,7 +266,7 @@ int tduffy> DT_Performance_Test_Destroy ( tduffy> Per_Test_Data_t *pt_ptr, tduffy> Performance_Test_t *test_ptr, tduffy> - DAT_BOOLEAN is_server) tduffy> + boolean_t is_server) tduffy> { tduffy> DAT_RETURN ret; tduffy> DAT_EP_HANDLE ep_handle; tduffy> Index: linux-kernel/test/dapltest/test/dapl_cnxn.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_cnxn.c (revision 2295) tduffy> +++ linux-kernel/test/dapltest/test/dapl_cnxn.c (working copy) tduffy> @@ -32,8 +32,8 @@ int tduffy> get_ep_connection_state (DT_Tdep_Print_Head *phead, DAT_EP_HANDLE ep_handle) tduffy> { tduffy> DAT_EP_STATE ep_state; tduffy> - DAT_BOOLEAN in_dto_idle; tduffy> - DAT_BOOLEAN out_dto_idle; tduffy> + boolean_t in_dto_idle; tduffy> + boolean_t out_dto_idle; tduffy> DAT_RETURN ret; tduffy> char *recv_status = "Idle"; tduffy> char *req_status = "Idle"; tduffy> Index: linux-kernel/test/dapltest/test/dapl_transaction_test.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_transaction_test.c (revision 2295) tduffy> +++ linux-kernel/test/dapltest/test/dapl_transaction_test.c (working copy) tduffy> @@ -159,9 +159,9 @@ DT_Transaction_Test_Server (void *params tduffy> bool tduffy> DT_Transaction_Create_Test (Per_Test_Data_t * pt_ptr, tduffy> DAT_IA_HANDLE *ia_handle, tduffy> - DAT_BOOLEAN is_server, tduffy> + boolean_t is_server, tduffy> unsigned int port_num, tduffy> - DAT_BOOLEAN remote_is_little_endian, tduffy> + boolean_t remote_is_little_endian, tduffy> DAT_IA_ADDRESS_PTR remote_ia_addr) tduffy> { tduffy> Transaction_Test_t *test_ptr; tduffy> @@ -820,7 +820,7 @@ retry: tduffy> */ tduffy> for (j = 0; j < test_ptr->cmd->num_ops; j++) tduffy> { tduffy> - DAT_BOOLEAN us; tduffy> + boolean_t us; tduffy> tduffy> us = (pt_ptr->local_is_server && tduffy> test_ptr->ep_context[i].op[j].server_initiated) || tduffy> @@ -1039,7 +1039,7 @@ retry: tduffy> RMI_RECV_BUFFER_ID); tduffy> for (j = 0; j < test_ptr->cmd->num_ops; j++) tduffy> { tduffy> - DAT_BOOLEAN us; tduffy> + boolean_t us; tduffy> tduffy> us = (pt_ptr->local_is_server && tduffy> test_ptr->ep_context[i].op[j].server_initiated) || tduffy> Index: linux-kernel/test/dapltest/test/dapl_bpool.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_bpool.c (revision 2295) tduffy> +++ linux-kernel/test/dapltest/test/dapl_bpool.c (working copy) tduffy> @@ -93,8 +93,8 @@ DT_BpoolAlloc ( tduffy> DAT_COUNT seg_size, tduffy> DAT_COUNT num_segs, tduffy> DAT_COUNT alignment, tduffy> - DAT_BOOLEAN enable_rdma_write, tduffy> - DAT_BOOLEAN enable_rdma_read) tduffy> + boolean_t enable_rdma_write, tduffy> + boolean_t enable_rdma_read) tduffy> { tduffy> unsigned char *module = "DT_BpoolAlloc"; tduffy> unsigned char *alloc_ptr = NULL; tduffy> Index: linux-kernel/test/dapltest/test/dapl_test_util.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_test_util.c (revision 2295) tduffy> +++ linux-kernel/test/dapltest/test/dapl_test_util.c (working copy) tduffy> @@ -30,7 +30,7 @@ tduffy> /* ----------------------------------------------------------- tduffy> * Gather info about default attributes tduffy> */ tduffy> -DAT_BOOLEAN tduffy> +boolean_t tduffy> DT_query ( Per_Test_Data_t *pt_ptr, tduffy> DAT_IA_HANDLE ia_handle, tduffy> DAT_EP_HANDLE ep_handle) tduffy> @@ -158,7 +158,7 @@ DT_query ( Per_Test_Data_t *pt_ptr, tduffy> /* ----------------------------------------------------------- tduffy> * Post a recv buffer tduffy> */ tduffy> -DAT_BOOLEAN tduffy> +boolean_t tduffy> DT_post_recv_buffer (DT_Tdep_Print_Head *phead, tduffy> DAT_EP_HANDLE ep_handle, tduffy> Bpool * bp, tduffy> @@ -203,7 +203,7 @@ DT_post_recv_buffer (DT_Tdep_Print_Head tduffy> /* ----------------------------------------------------------- tduffy> * Post a send buffer tduffy> */ tduffy> -DAT_BOOLEAN tduffy> +boolean_t tduffy> DT_post_send_buffer (DT_Tdep_Print_Head *phead, tduffy> DAT_EP_HANDLE ep_handle, tduffy> Bpool * bp, tduffy> Index: linux-kernel/test/dapltest/include/dapl_performance_test.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/include/dapl_performance_test.h (revision 2295) tduffy> +++ linux-kernel/test/dapltest/include/dapl_performance_test.h (working copy) tduffy> @@ -67,7 +67,7 @@ typedef struct tduffy> Per_Test_Data_t *pt_ptr; tduffy> Performance_Cmd_t *cmd; tduffy> DAT_IA_ADDRESS_PTR remote_ia_addr; tduffy> - DAT_BOOLEAN is_remote_little_endian; tduffy> + boolean_t is_remote_little_endian; tduffy> DAT_CONN_QUAL base_port; tduffy> struct dat_ia_attr ia_attr; tduffy> DAT_IA_HANDLE ia_handle; tduffy> Index: linux-kernel/test/dapltest/include/dapl_proto.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/include/dapl_proto.h (revision 2295) tduffy> +++ linux-kernel/test/dapltest/include/dapl_proto.h (working copy) tduffy> @@ -88,8 +88,8 @@ Bpool * DT_BpoolAlloc (Per_Test_ tduffy> DAT_COUNT seg_size, tduffy> DAT_COUNT num_segs, tduffy> DAT_COUNT alignment, tduffy> - DAT_BOOLEAN enable_rdma_write, tduffy> - DAT_BOOLEAN enable_rdma_read); tduffy> + boolean_t enable_rdma_write, tduffy> + boolean_t enable_rdma_read); tduffy> tduffy> bool DT_Bpool_Destroy (Per_Test_Data_t * pt_ptr, tduffy> DT_Tdep_Print_Head *phead, tduffy> @@ -267,13 +267,13 @@ bool DT_Performance_Test_Serv tduffy> bool DT_Performance_Test_Create (Per_Test_Data_t * pt_ptr, tduffy> DAT_IA_HANDLE * ia_handle, tduffy> DAT_IA_ADDRESS_PTR remote_ia_addr, tduffy> - DAT_BOOLEAN is_server, tduffy> - DAT_BOOLEAN is_remote_little_endian, tduffy> + boolean_t is_server, tduffy> + boolean_t is_remote_little_endian, tduffy> Performance_Test_t **perf_test); tduffy> tduffy> int DT_Performance_Test_Destroy (Per_Test_Data_t * pt_ptr, tduffy> Performance_Test_t *test_ptr, tduffy> - DAT_BOOLEAN is_server); tduffy> + boolean_t is_server); tduffy> tduffy> bool DT_performance_post_rdma_op (Performance_Ep_Context_t *ep_context, tduffy> DAT_EVD_HANDLE reqt_evd_hdl, tduffy> @@ -349,17 +349,17 @@ Per_Test_Data_t *DT_Alloc_Per_Test_Data tduffy> void DT_Free_Per_Test_Data (Per_Test_Data_t * pt_ptr); tduffy> tduffy> /* dapl_test_util.c */ tduffy> -DAT_BOOLEAN DT_query (Per_Test_Data_t *pt_ptr, tduffy> +boolean_t DT_query (Per_Test_Data_t *pt_ptr, tduffy> DAT_IA_HANDLE ia_handle, tduffy> DAT_EP_HANDLE ep_handle); tduffy> tduffy> -DAT_BOOLEAN DT_post_recv_buffer (DT_Tdep_Print_Head* phead, tduffy> +boolean_t DT_post_recv_buffer (DT_Tdep_Print_Head* phead, tduffy> DAT_EP_HANDLE ep_handle, tduffy> Bpool * bp, tduffy> int index, tduffy> int size); tduffy> tduffy> -DAT_BOOLEAN DT_post_send_buffer (DT_Tdep_Print_Head* phead, tduffy> +boolean_t DT_post_send_buffer (DT_Tdep_Print_Head* phead, tduffy> DAT_EP_HANDLE ep_handle, tduffy> Bpool * bp, tduffy> int index, tduffy> @@ -471,9 +471,9 @@ void DT_Transaction_Test_Serv tduffy> tduffy> bool DT_Transaction_Create_Test (Per_Test_Data_t * pt_ptr, tduffy> DAT_IA_HANDLE * ia_handle, tduffy> - DAT_BOOLEAN is_server, tduffy> + boolean_t is_server, tduffy> unsigned int port_num, tduffy> - DAT_BOOLEAN remote_is_little_endian, tduffy> + boolean_t remote_is_little_endian, tduffy> DAT_IA_ADDRESS_PTR remote_ia_addr); tduffy> tduffy> void DT_Transaction_Main (void *param); tduffy> Index: linux-kernel/test/dapltest/include/dapl_bpool.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/include/dapl_bpool.h (revision 2295) tduffy> +++ linux-kernel/test/dapltest/include/dapl_bpool.h (working copy) tduffy> @@ -49,8 +49,8 @@ struct Bpool_tag { tduffy> DAT_LMR_HANDLE lmr_handle; /* local access */ tduffy> DAT_LMR_CONTEXT lmr_context; tduffy> struct dat_lmr_triplet *tripl_start; /* local IOV */ tduffy> - DAT_BOOLEAN enable_rdma_write; /* remote access */ tduffy> - DAT_BOOLEAN enable_rdma_read; tduffy> + boolean_t enable_rdma_write; /* remote access */ tduffy> + boolean_t enable_rdma_read; tduffy> DAT_RMR_HANDLE rmr_handle; tduffy> DAT_RMR_CONTEXT rmr_context; tduffy> DAT_EVD_HANDLE rmr_evd_handle; tduffy> Index: linux-kernel/test/dapltest/include/dapl_transaction_test.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/include/dapl_transaction_test.h (revision 2295) tduffy> +++ linux-kernel/test/dapltest/include/dapl_transaction_test.h (working copy) tduffy> @@ -35,11 +35,11 @@ tduffy> tduffy> typedef struct tduffy> { tduffy> - DAT_BOOLEAN server_initiated; tduffy> + boolean_t server_initiated; tduffy> DT_Transfer_Type transfer_type; tduffy> uint32_t num_segs; tduffy> uint32_t seg_size; tduffy> - DAT_BOOLEAN reap_send_on_recv; tduffy> + boolean_t reap_send_on_recv; tduffy> Bpool *bp; tduffy> tduffy> /* RDMA info */ tduffy> @@ -72,8 +72,8 @@ typedef struct tduffy> typedef struct tduffy> { tduffy> /* This group set up by DT_Transaction_Create_Test() */ tduffy> - DAT_BOOLEAN is_server; tduffy> - DAT_BOOLEAN remote_is_little_endian; tduffy> + boolean_t is_server; tduffy> + boolean_t remote_is_little_endian; tduffy> Per_Test_Data_t *pt_ptr; tduffy> DAT_IA_HANDLE ia_handle; tduffy> Transaction_Cmd_t *cmd; tduffy> Index: linux-kernel/test/dapltest/cmd/dapl_netaddr.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/cmd/dapl_netaddr.c (revision 2295) tduffy> +++ linux-kernel/test/dapltest/cmd/dapl_netaddr.c (working copy) tduffy> @@ -125,7 +125,7 @@ DT_NetAddrLookupHostAddress (DAT_IA_ADDR tduffy> tduffy> DT_Mdep_printf ("getaddrinfo (%s) failed (%s)\n", tduffy> hostname, whatzit); tduffy> - return DAT_FALSE; tduffy> + return FALSE; tduffy> } tduffy> tduffy> /* Pull out IP address and print it as a sanity check */ tduffy> @@ -139,6 +139,6 @@ DT_NetAddrLookupHostAddress (DAT_IA_ADDR tduffy> tduffy> *to_netaddr = * ((DAT_IA_ADDRESS_PTR) target->ai_addr); tduffy> tduffy> - return ( DAT_TRUE ); tduffy> + return ( TRUE ); tduffy> } tduffy> tduffy> Index: linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.c (revision 2295) tduffy> +++ linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.c (working copy) tduffy> @@ -431,7 +431,7 @@ DT_Mdep_wait_object_init ( tduffy> DT_WAIT_OBJECT *wait_obj) tduffy> { tduffy> tduffy> - wait_obj->signaled = DAT_FALSE; tduffy> + wait_obj->signaled = FALSE; tduffy> if ( 0 != pthread_cond_init ( &wait_obj->cv, NULL ) ) tduffy> { tduffy> return (-1); tduffy> @@ -492,7 +492,7 @@ DT_Mdep_wait_object_wait ( tduffy> future.tv_nsec = now.tv_usec * 1000; tduffy> tduffy> pthread_mutex_lock (&wait_obj->lock); tduffy> - while ( wait_obj->signaled == DAT_FALSE && pthread_status == 0) tduffy> + while ( wait_obj->signaled == FALSE && pthread_status == 0) tduffy> { tduffy> pthread_status = pthread_cond_timedwait ( tduffy> &wait_obj->cv, &wait_obj->lock, &future ); tduffy> @@ -512,7 +512,7 @@ DT_Mdep_wait_object_wait ( tduffy> else tduffy> { tduffy> pthread_mutex_lock (&wait_obj->lock); tduffy> - while ( wait_obj->signaled == DAT_FALSE && pthread_status == 0) tduffy> + while ( wait_obj->signaled == FALSE && pthread_status == 0) tduffy> { tduffy> pthread_status = pthread_cond_wait ( tduffy> &wait_obj->cv, &wait_obj->lock ); tduffy> Index: linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c (revision 2295) tduffy> +++ linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c (working copy) tduffy> @@ -47,7 +47,7 @@ typedef struct Tdep_Evd_Tag tduffy> static void DT_Tdep_Event_Callback ( tduffy> DAT_PVOID instance_data, tduffy> const struct dat_event *dat_event, tduffy> - DAT_BOOLEAN bool_arg); tduffy> + boolean_t bool_arg); tduffy> tduffy> /* static data */ tduffy> tduffy> @@ -324,7 +324,7 @@ DT_Tdep_evd_free (DAT_EVD_HANDLE evd_han tduffy> static void DT_Tdep_Event_Callback ( tduffy> DAT_PVOID instance_data, tduffy> const struct dat_event * dat_event, tduffy> - DAT_BOOLEAN bool_arg) tduffy> + boolean_t bool_arg) tduffy> { tduffy> Tdep_Event *event; tduffy> Tdep_Evd *evd_ptr; tduffy> Index: linux-kernel/dat-provider/dapl_cr_callback.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_cr_callback.c (revision 2295) tduffy> +++ linux-kernel/dat-provider/dapl_cr_callback.c (working copy) tduffy> @@ -87,7 +87,7 @@ void dapl_cr_callback(ib_cm_handle_t ib_ tduffy> sp_ptr->header.magic == DAPL_MAGIC_RSP); tduffy> tduffy> /* Obtain the event number from the provider layer */ tduffy> - dat_event_num = dapl_ib_get_dat_event(ib_cm_event, DAT_FALSE); tduffy> + dat_event_num = dapl_ib_get_dat_event(ib_cm_event, FALSE); tduffy> tduffy> /* tduffy> * CONNECT_REQUEST events create an event on the PSP tduffy> @@ -128,7 +128,7 @@ void dapl_cr_callback(ib_cm_handle_t ib_ tduffy> */ tduffy> tduffy> spin_lock(&sp_ptr->header.lock); tduffy> - if (sp_ptr->listening == DAT_FALSE) { tduffy> + if (sp_ptr->listening == FALSE) { tduffy> spin_unlock(&sp_ptr->header.lock); tduffy> dapl_dbg_log(DAPL_DBG_TYPE_CM, tduffy> "---> dapl_cr_callback: conn event on down SP\n"); tduffy> @@ -143,7 +143,7 @@ void dapl_cr_callback(ib_cm_handle_t ib_ tduffy> * RSP connections only allow a single connection. Close tduffy> * it down NOW so we reject any further connections. tduffy> */ tduffy> - sp_ptr->listening = DAT_FALSE; tduffy> + sp_ptr->listening = FALSE; tduffy> } tduffy> spin_unlock(&sp_ptr->header.lock); tduffy> tduffy> @@ -205,7 +205,7 @@ void dapl_cr_callback(ib_cm_handle_t ib_ tduffy> return; tduffy> } tduffy> ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; tduffy> - dapl_ib_disconnect_clean(ep_ptr, DAT_FALSE, tduffy> + dapl_ib_disconnect_clean(ep_ptr, FALSE, tduffy> ib_cm_event); tduffy> spin_unlock(&ep_ptr->header.lock); tduffy> tduffy> @@ -221,7 +221,7 @@ void dapl_cr_callback(ib_cm_handle_t ib_ tduffy> */ tduffy> spin_lock(&ep_ptr->header.lock); tduffy> ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; tduffy> - dapl_ib_disconnect_clean(ep_ptr, DAT_FALSE, tduffy> + dapl_ib_disconnect_clean(ep_ptr, FALSE, tduffy> ib_cm_event); tduffy> spin_unlock(&ep_ptr->header.lock); tduffy> tduffy> @@ -231,7 +231,7 @@ void dapl_cr_callback(ib_cm_handle_t ib_ tduffy> { tduffy> spin_lock(&ep_ptr->header.lock); tduffy> ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; tduffy> - dapl_ib_disconnect_clean(ep_ptr, DAT_FALSE, tduffy> + dapl_ib_disconnect_clean(ep_ptr, FALSE, tduffy> ib_cm_event); tduffy> spin_unlock(&ep_ptr->header.lock); tduffy> tduffy> @@ -433,7 +433,7 @@ DAPL_EP *dapl_get_sp_ep(ib_cm_handle_t i tduffy> * If this SP has been removed from service, free it tduffy> * up after the last CR is removed tduffy> */ tduffy> - if (sp_ptr->listening != DAT_TRUE && sp_ptr->cr_list_count == 0 tduffy> + if (sp_ptr->listening != TRUE && sp_ptr->cr_list_count == 0 tduffy> && sp_ptr->state != DAPL_SP_STATE_FREE) { tduffy> dapl_dbg_log(DAPL_DBG_TYPE_CM, tduffy> "--> dapl_get_sp_ep! disconnect dump sp: %p \n", tduffy> Index: linux-kernel/dat-provider/dapl_module.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_module.c (revision 2295) tduffy> +++ linux-kernel/dat-provider/dapl_module.c (working copy) tduffy> @@ -236,7 +236,7 @@ void dapl_add_one(struct ib_device *devi tduffy> dev_name_suffix_table[i + 1]); tduffy> provider_info.dat_version_major = DAT_VERSION_MAJOR; tduffy> provider_info.dat_version_minor = DAT_VERSION_MINOR; tduffy> - provider_info.is_thread_safe = DAT_FALSE; tduffy> + provider_info.is_thread_safe = FALSE; tduffy> DAT_PROVIDER_INIT_FUNC_NAME(&provider_info, device, i); tduffy> } tduffy> return; tduffy> @@ -258,7 +258,7 @@ void dapl_remove_one(struct ib_device *d tduffy> dev_name_suffix_table[i + 1]); tduffy> provider_info.dat_version_major = DAT_VERSION_MAJOR; tduffy> provider_info.dat_version_minor = DAT_VERSION_MINOR; tduffy> - provider_info.is_thread_safe = DAT_FALSE; tduffy> + provider_info.is_thread_safe = FALSE; tduffy> DAT_PROVIDER_FINI_FUNC_NAME(&provider_info); tduffy> } tduffy> return; tduffy> Index: linux-kernel/dat-provider/dapl_psp_create.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_psp_create.c (revision 2295) tduffy> +++ linux-kernel/dat-provider/dapl_psp_create.c (working copy) tduffy> @@ -76,7 +76,7 @@ dapl_psp_create(DAT_IA_HANDLE ia_handle, tduffy> DAPL_IA *ia_ptr; tduffy> DAPL_SP *sp_ptr; tduffy> DAPL_EVD *evd_ptr; tduffy> - DAT_BOOLEAN sp_found; tduffy> + boolean_t sp_found; tduffy> DAT_RETURN dat_status; tduffy> tduffy> ia_ptr = (DAPL_IA *) ia_handle; tduffy> @@ -115,19 +115,19 @@ dapl_psp_create(DAT_IA_HANDLE ia_handle, tduffy> * See if we have a quiescent listener to use for this PSP, else tduffy> * create one and set it listening tduffy> */ tduffy> - sp_ptr = dapl_ia_sp_search(ia_ptr, conn_qual, DAT_TRUE); tduffy> - sp_found = DAT_TRUE; tduffy> + sp_ptr = dapl_ia_sp_search(ia_ptr, conn_qual, TRUE); tduffy> + sp_found = TRUE; tduffy> if (sp_ptr == NULL) { tduffy> /* Allocate PSP */ tduffy> - sp_found = DAT_FALSE; tduffy> - sp_ptr = dapl_sp_alloc(ia_ptr, DAT_TRUE); tduffy> + sp_found = FALSE; tduffy> + sp_ptr = dapl_sp_alloc(ia_ptr, TRUE); tduffy> if (sp_ptr == NULL) { tduffy> dat_status = tduffy> DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, tduffy> DAT_RESOURCE_MEMORY); tduffy> goto bail; tduffy> } tduffy> - } else if (sp_ptr->listening == DAT_TRUE) { tduffy> + } else if (sp_ptr->listening == TRUE) { tduffy> dat_status = DAT_ERROR(DAT_CONN_QUAL_IN_USE, 0); tduffy> goto bail; tduffy> } tduffy> @@ -151,13 +151,13 @@ dapl_psp_create(DAT_IA_HANDLE ia_handle, tduffy> * even before this call returns! tduffy> */ tduffy> sp_ptr->state = DAPL_SP_STATE_PSP_LISTENING; tduffy> - sp_ptr->listening = DAT_TRUE; tduffy> + sp_ptr->listening = TRUE; tduffy> tduffy> /* tduffy> * If this is a new sp we need to add it to the IA queue, and set up tduffy> * a conn_listener. tduffy> */ tduffy> - if (sp_found == DAT_FALSE) { tduffy> + if (sp_found == FALSE) { tduffy> /* Link it onto the IA before enabling it to receive conn tduffy> * requests tduffy> */ tduffy> Index: linux-kernel/dat-provider/dapl_evd_connection_callb.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_evd_connection_callb.c (revision 2295) tduffy> +++ linux-kernel/dat-provider/dapl_evd_connection_callb.c (working copy) tduffy> @@ -109,7 +109,7 @@ dapl_evd_connection_callback(ib_cm_handl tduffy> } tduffy> tduffy> /* Obtain the event number from the provider layer */ tduffy> - dat_event_num = dapl_ib_get_dat_event(ib_cm_event, DAT_FALSE); tduffy> + dat_event_num = dapl_ib_get_dat_event(ib_cm_event, FALSE); tduffy> tduffy> switch (dat_event_num) { tduffy> case DAT_CONNECTION_EVENT_ESTABLISHED: tduffy> @@ -148,7 +148,7 @@ dapl_evd_connection_callback(ib_cm_handl tduffy> * another connection tduffy> */ tduffy> ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; tduffy> - dapl_ib_disconnect_clean(ep_ptr, DAT_TRUE, ib_cm_event); tduffy> + dapl_ib_disconnect_clean(ep_ptr, TRUE, ib_cm_event); tduffy> spin_unlock(&ep_ptr->header.lock); tduffy> tduffy> break; tduffy> @@ -156,7 +156,7 @@ dapl_evd_connection_callback(ib_cm_handl tduffy> case DAT_CONNECTION_EVENT_PEER_REJECTED: tduffy> { tduffy> ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; tduffy> - dapl_ib_disconnect_clean(ep_ptr, DAT_TRUE, ib_cm_event); tduffy> + dapl_ib_disconnect_clean(ep_ptr, TRUE, ib_cm_event); tduffy> spin_unlock(&ep_ptr->header.lock); tduffy> tduffy> break; tduffy> @@ -164,7 +164,7 @@ dapl_evd_connection_callback(ib_cm_handl tduffy> case DAT_CONNECTION_EVENT_UNREACHABLE: tduffy> { tduffy> ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; tduffy> - dapl_ib_disconnect_clean(ep_ptr, DAT_TRUE, ib_cm_event); tduffy> + dapl_ib_disconnect_clean(ep_ptr, TRUE, ib_cm_event); tduffy> spin_unlock(&ep_ptr->header.lock); tduffy> tduffy> break; tduffy> @@ -172,7 +172,7 @@ dapl_evd_connection_callback(ib_cm_handl tduffy> case DAT_CONNECTION_EVENT_NON_PEER_REJECTED: tduffy> { tduffy> ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; tduffy> - dapl_ib_disconnect_clean(ep_ptr, DAT_TRUE, ib_cm_event); tduffy> + dapl_ib_disconnect_clean(ep_ptr, TRUE, ib_cm_event); tduffy> spin_unlock(&ep_ptr->header.lock); tduffy> tduffy> break; tduffy> @@ -180,7 +180,7 @@ dapl_evd_connection_callback(ib_cm_handl tduffy> case DAT_CONNECTION_EVENT_BROKEN: tduffy> { tduffy> ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; tduffy> - dapl_ib_disconnect_clean(ep_ptr, DAT_FALSE, tduffy> + dapl_ib_disconnect_clean(ep_ptr, FALSE, tduffy> ib_cm_event); tduffy> spin_unlock(&ep_ptr->header.lock); tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_psp_create_any.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_psp_create_any.c (revision 2295) tduffy> +++ linux-kernel/dat-provider/dapl_psp_create_any.c (working copy) tduffy> @@ -121,7 +121,7 @@ dapl_psp_create_any(DAT_IA_HANDLE ia_han tduffy> } tduffy> tduffy> /* Allocate PSP */ tduffy> - sp_ptr = dapl_sp_alloc(ia_ptr, DAT_TRUE); tduffy> + sp_ptr = dapl_sp_alloc(ia_ptr, TRUE); tduffy> if (sp_ptr == NULL) { tduffy> dat_status = tduffy> DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> @@ -149,7 +149,7 @@ dapl_psp_create_any(DAT_IA_HANDLE ia_han tduffy> * even before this call returns! tduffy> */ tduffy> sp_ptr->state = DAPL_SP_STATE_PSP_LISTENING; tduffy> - sp_ptr->listening = DAT_TRUE; tduffy> + sp_ptr->listening = TRUE; tduffy> tduffy> limit_conn_qual = 0; tduffy> lcl_conn_qual = hint_conn_qual; tduffy> Index: linux-kernel/dat-provider/dapl_psp_free.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_psp_free.c (revision 2295) tduffy> +++ linux-kernel/dat-provider/dapl_psp_free.c (working copy) tduffy> @@ -89,7 +89,7 @@ DAT_RETURN dapl_psp_free(DAT_PSP_HANDLE tduffy> */ tduffy> spin_lock(&sp_ptr->header.lock); tduffy> tduffy> - sp_ptr->listening = DAT_FALSE; tduffy> + sp_ptr->listening = FALSE; tduffy> tduffy> /* Release reference on EVD. If an error was encountered in a previous tduffy> * free the evd_handle will be NULL tduffy> Index: linux-kernel/dat-provider/dapl_ep_get_status.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_ep_get_status.c (revision 2295) tduffy> +++ linux-kernel/dat-provider/dapl_ep_get_status.c (working copy) tduffy> @@ -62,7 +62,7 @@ tduffy> DAT_RETURN tduffy> dapl_ep_get_status(DAT_EP_HANDLE ep_handle, tduffy> DAT_EP_STATE * ep_state, tduffy> - DAT_BOOLEAN * in_dto_idle, DAT_BOOLEAN * out_dto_idle) tduffy> + boolean_t *in_dto_idle, boolean_t *out_dto_idle) tduffy> { tduffy> DAPL_EP *ep_ptr; tduffy> DAT_RETURN dat_status; tduffy> @@ -92,14 +92,12 @@ dapl_ep_get_status(DAT_EP_HANDLE ep_hand tduffy> tduffy> if (in_dto_idle != NULL) { tduffy> *in_dto_idle = tduffy> - (atomic_read(&ep_ptr->recv_count)) ? DAT_FALSE : tduffy> - DAT_TRUE; tduffy> + (atomic_read(&ep_ptr->recv_count)) ? FALSE : TRUE; tduffy> } tduffy> tduffy> if (out_dto_idle != NULL) { tduffy> *out_dto_idle = tduffy> - (atomic_read(&ep_ptr->req_count)) ? DAT_FALSE : tduffy> - DAT_TRUE; tduffy> + (atomic_read(&ep_ptr->req_count)) ? FALSE : TRUE; tduffy> } tduffy> tduffy> bail: tduffy> Index: linux-kernel/dat-provider/dapl_ia_query.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_ia_query.c (revision 2295) tduffy> +++ linux-kernel/dat-provider/dapl_ia_query.c (working copy) tduffy> @@ -68,9 +68,9 @@ dapl_ia_query(DAT_IA_HANDLE ia_handle, tduffy> DAPL_IA *ia_ptr; tduffy> DAT_RETURN dat_status; tduffy> struct evd_merge_type { tduffy> - DAT_BOOLEAN array[6][6]; tduffy> + boolean_t array[6][6]; tduffy> } *evd_merge; tduffy> - DAT_BOOLEAN val; tduffy> + boolean_t val; tduffy> int i; tduffy> int j; tduffy> tduffy> @@ -120,9 +120,9 @@ dapl_ia_query(DAT_IA_HANDLE ia_handle, tduffy> provider_attr->dat_qos_supported = DAT_QOS_BEST_EFFORT; tduffy> provider_attr->completion_flags_supported = tduffy> DAT_COMPLETION_DEFAULT_FLAG; tduffy> - provider_attr->is_thread_safe = DAT_FALSE; tduffy> + provider_attr->is_thread_safe = FALSE; tduffy> provider_attr->max_private_data_size = IB_MAX_REQ_PDATA_SIZE; tduffy> - provider_attr->supports_multipath = DAT_FALSE; tduffy> + provider_attr->supports_multipath = FALSE; tduffy> provider_attr->ep_creator = DAT_PSP_CREATES_EP_NEVER; tduffy> provider_attr->optimal_buffer_alignment = DAPL_OPTIMAL_ALIGNMENT; tduffy> /* The value of pz_support may vary by transport */ tduffy> @@ -149,17 +149,17 @@ dapl_ia_query(DAT_IA_HANDLE ia_handle, tduffy> evd_merge = tduffy> (struct evd_merge_type *)&provider_attr-> tduffy> evd_stream_merging_supported[0][0]; tduffy> - val = DAT_TRUE; tduffy> + val = TRUE; tduffy> for (i = 0; i < 6; i++) { tduffy> if (i > 4) { tduffy> /* ASYNC EVD is 5, so entire row will be 0 */ tduffy> - val = DAT_FALSE; tduffy> + val = FALSE; tduffy> } tduffy> for (j = 0; j < 5; j++) { tduffy> evd_merge->array[i][j] = val; tduffy> } tduffy> /* Set the ASYNC_EVD column to FALSE */ tduffy> - evd_merge->array[i][5] = DAT_FALSE; tduffy> + evd_merge->array[i][5] = FALSE; tduffy> } tduffy> tduffy> #ifndef DAPL_MERGE_CM_DTO tduffy> @@ -169,9 +169,9 @@ dapl_ia_query(DAT_IA_HANDLE ia_handle, tduffy> * skip the following code tduffy> */ tduffy> /* DAT_EVD_DTO_FLAG | DAT_EVD_CONNECTION_FLAG */ tduffy> - evd_merge->array[2][3] = DAT_FALSE; tduffy> + evd_merge->array[2][3] = FALSE; tduffy> /* DAT_EVD_CONNECTION_FLAG | DAT_EVD_DTO_FLAG */ tduffy> - evd_merge->array[3][2] = DAT_FALSE; tduffy> + evd_merge->array[3][2] = FALSE; tduffy> #endif /* DAPL_MERGE_CM_DTO */ tduffy> } tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_mr_util.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_mr_util.h (revision 2295) tduffy> +++ linux-kernel/dat-provider/dapl_mr_util.h (working copy) tduffy> @@ -49,7 +49,7 @@ tduffy> extern DAT_VADDR tduffy> dapl_mr_get_address(DAT_REGION_DESCRIPTION desc, DAT_MEM_TYPE type); tduffy> tduffy> -static __inline__ DAT_BOOLEAN tduffy> +static __inline__ boolean_t tduffy> dapl_mr_bounds_check(DAT_VADDR addr_a, tduffy> DAT_VLEN length_a, DAT_VADDR addr_b, DAT_VLEN length_b); tduffy> tduffy> @@ -67,14 +67,14 @@ dapl_mr_bounds_check(DAT_VADDR addr_a, tduffy> * tduffy> */ tduffy> tduffy> -static __inline__ DAT_BOOLEAN tduffy> +static __inline__ boolean_t tduffy> dapl_mr_bounds_check(DAT_VADDR addr_a, tduffy> DAT_VLEN length_a, DAT_VADDR addr_b, DAT_VLEN length_b) tduffy> { tduffy> if ((addr_a <= addr_b) && (addr_b + length_b) <= (addr_a + length_a)) { tduffy> - return DAT_TRUE; tduffy> + return TRUE; tduffy> } else { tduffy> - return DAT_FALSE; tduffy> + return FALSE; tduffy> } tduffy> } tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_adapter_util.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_adapter_util.h (revision 2295) tduffy> +++ linux-kernel/dat-provider/dapl_adapter_util.h (working copy) tduffy> @@ -139,11 +139,11 @@ DAT_RETURN dapl_ib_mw_bind(DAPL_RMR * rm tduffy> DAT_VADDR virtual_address, tduffy> DAT_VLEN length, tduffy> DAT_MEM_PRIV_FLAGS mem_priv, tduffy> - DAT_BOOLEAN is_signaled); tduffy> + boolean_t is_signaled); tduffy> tduffy> DAT_RETURN dapl_ib_mw_unbind(DAPL_RMR * rmr, tduffy> DAPL_EP * ep, tduffy> - DAPL_COOKIE * cookie, DAT_BOOLEAN is_signaled); tduffy> + DAPL_COOKIE * cookie, boolean_t is_signaled); tduffy> tduffy> DAT_RETURN dapl_ib_query_hca(DAPL_HCA * hca_ptr, tduffy> struct dat_ia_attr *ia_attr, tduffy> @@ -163,14 +163,14 @@ DAT_DTO_COMPLETION_STATUS dapl_ib_get_dt tduffy> void dapl_ib_reinit_ep(DAPL_EP * ep_ptr); tduffy> tduffy> void dapl_ib_disconnect_clean(DAPL_EP * ep_ptr, tduffy> - DAT_BOOLEAN passive, tduffy> + boolean_t passive, tduffy> const ib_cm_events_t ib_cm_event); tduffy> tduffy> DAT_RETURN dapl_ib_get_async_event(ib_error_record_t * cause_ptr, tduffy> DAT_EVENT_NUMBER * async_event); tduffy> tduffy> DAT_EVENT_NUMBER dapl_ib_get_dat_event(const ib_cm_events_t ib_cm_event, tduffy> - DAT_BOOLEAN active); tduffy> + boolean_t active); tduffy> tduffy> ib_cm_events_t dapl_ib_get_cm_event(DAT_EVENT_NUMBER dat_event_num); tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_ia_util.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_ia_util.c (revision 2295) tduffy> +++ linux-kernel/dat-provider/dapl_ia_util.c (working copy) tduffy> @@ -91,7 +91,7 @@ DAPL_IA *dapl_ia_alloc(struct dat_provid tduffy> */ tduffy> ia_ptr->hca_ptr = hca_ptr; tduffy> ia_ptr->async_error_evd = NULL; tduffy> - ia_ptr->cleanup_async_error_evd = DAT_FALSE; tduffy> + ia_ptr->cleanup_async_error_evd = FALSE; tduffy> dapl_llist_init_entry(&ia_ptr->hca_ia_list_entry); tduffy> dapl_llist_init_head(&ia_ptr->ep_list_head); tduffy> dapl_llist_init_head(&ia_ptr->lmr_list_head); tduffy> @@ -316,7 +316,7 @@ DAT_RETURN dapl_ia_abrupt_close(DAPL_IA tduffy> * Free the async EVD, shutting down callbacks from the HCA. tduffy> */ tduffy> if (ia_ptr->async_error_evd && tduffy> - (DAT_TRUE == ia_ptr->cleanup_async_error_evd)) { tduffy> + (TRUE == ia_ptr->cleanup_async_error_evd)) { tduffy> dat_status = dapl_ia_teardown_callbacks(ia_ptr); tduffy> tduffy> atomic_dec(&ia_ptr->async_error_evd->evd_ref_count); tduffy> @@ -382,7 +382,7 @@ DAT_RETURN dapl_ia_graceful_close(DAPL_I tduffy> /* if the async evd does not need to be cleaned up */ tduffy> /* (ie. it was not created by dapl_ia_open) */ tduffy> /* then the evd list should be empty */ tduffy> - if (DAT_FALSE == ia_ptr->cleanup_async_error_evd) { tduffy> + if (FALSE == ia_ptr->cleanup_async_error_evd) { tduffy> if (!dapl_llist_is_empty(&ia_ptr->evd_list_head)) { tduffy> dat_status = tduffy> DAT_ERROR(DAT_INVALID_STATE, tduffy> @@ -437,7 +437,7 @@ DAT_RETURN dapl_ia_graceful_close(DAPL_I tduffy> tduffy> /* Tear down the async EVD if needed, first shutting down callbacks. */ tduffy> if (ia_ptr->async_error_evd && tduffy> - (DAT_TRUE == ia_ptr->cleanup_async_error_evd)) { tduffy> + (TRUE == ia_ptr->cleanup_async_error_evd)) { tduffy> cur_dat_status = dapl_ia_teardown_callbacks(ia_ptr); tduffy> if (DAT_SUCCESS != cur_dat_status) { tduffy> dat_status = cur_dat_status; tduffy> @@ -683,7 +683,7 @@ void dapl_ia_unlink_sp(DAPL_IA * ia_ptr, tduffy> * tduffy> */ tduffy> DAPL_SP *dapl_ia_sp_search(DAPL_IA * ia_ptr, tduffy> - DAT_CONN_QUAL conn_qual, DAT_BOOLEAN is_psp) tduffy> + DAT_CONN_QUAL conn_qual, boolean_t is_psp) tduffy> { tduffy> DAPL_SP *sp_ptr; tduffy> DAPL_LLIST_HEAD *list_head; tduffy> Index: linux-kernel/dat-provider/dapl_provider.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_provider.c (revision 2295) tduffy> +++ linux-kernel/dat-provider/dapl_provider.c (working copy) tduffy> @@ -135,7 +135,7 @@ struct dat_provider g_dapl_provider_temp tduffy> * * tduffy> *********************************************************************/ tduffy> tduffy> -static DAT_BOOLEAN tduffy> +static boolean_t tduffy> dapl_provider_list_key_cmp(const char *name_a, const char *name_b); tduffy> tduffy> /********************************************************************* tduffy> @@ -319,17 +319,17 @@ DAT_RETURN dapl_provider_list_remove(con tduffy> return status; tduffy> } tduffy> tduffy> -DAT_BOOLEAN dapl_provider_list_key_cmp(const char *name_a, const char *name_b) tduffy> +boolean_t dapl_provider_list_key_cmp(const char *name_a, const char *name_b) tduffy> { tduffy> unsigned int len; tduffy> tduffy> len = strlen(name_a); tduffy> tduffy> if (strlen(name_b) != len) { tduffy> - return DAT_FALSE; tduffy> + return FALSE; tduffy> } else if (memcmp(name_a, name_b, len)) { tduffy> - return DAT_FALSE; tduffy> + return FALSE; tduffy> } else { tduffy> - return DAT_TRUE; tduffy> + return TRUE; tduffy> } tduffy> } tduffy> Index: linux-kernel/dat-provider/dapl_openib_cm.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_openib_cm.c (revision 2295) tduffy> +++ linux-kernel/dat-provider/dapl_openib_cm.c (working copy) tduffy> @@ -511,7 +511,7 @@ DAT_RETURN dapl_ib_disconnect(DAPL_EP * tduffy> */ tduffy> void tduffy> dapl_ib_disconnect_clean(DAPL_EP * ep_ptr, tduffy> - DAT_BOOLEAN active, tduffy> + boolean_t active, tduffy> const ib_cm_events_t ib_cm_event) tduffy> { tduffy> int status; tduffy> Index: linux-kernel/dat-provider/dapl_ia_util.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_ia_util.h (revision 2295) tduffy> +++ linux-kernel/dat-provider/dapl_ia_util.h (working copy) tduffy> @@ -78,7 +78,7 @@ void dapl_ia_unlink_sp(DAPL_IA * ia_ptr, tduffy> void dapl_ia_link_rsp(DAPL_IA * ia_ptr, DAPL_SP * sp_info); tduffy> tduffy> DAPL_SP *dapl_ia_sp_search(DAPL_IA * ia_ptr, tduffy> - DAT_CONN_QUAL conn_qual, DAT_BOOLEAN is_psp); tduffy> + DAT_CONN_QUAL conn_qual, boolean_t is_psp); tduffy> tduffy> DAT_RETURN dapl_ia_setup_callbacks(DAPL_IA * ia_ptr, DAPL_EVD * async_evd_ptr); tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_rmr_bind.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_rmr_bind.c (revision 2295) tduffy> +++ linux-kernel/dat-provider/dapl_rmr_bind.c (working copy) tduffy> @@ -79,7 +79,7 @@ dapl_rmr_bind_fuse(DAPL_RMR * rmr, tduffy> DAPL_LMR *lmr; tduffy> DAPL_COOKIE *cookie; tduffy> DAT_RETURN dat_status; tduffy> - DAT_BOOLEAN is_signaled; tduffy> + boolean_t is_signaled; tduffy> tduffy> dat_status = tduffy> dapl_hash_search(rmr->header.owner_ia->hca_ptr->lmr_hash_table, tduffy> @@ -104,7 +104,7 @@ dapl_rmr_bind_fuse(DAPL_RMR * rmr, tduffy> goto bail; tduffy> } tduffy> tduffy> - if (DAT_FALSE == tduffy> + if (FALSE == tduffy> dapl_mr_bounds_check(dapl_mr_get_address tduffy> (lmr->param.region_desc, lmr->param.mem_type), tduffy> lmr->param.length, tduffy> @@ -146,8 +146,8 @@ dapl_rmr_bind_fuse(DAPL_RMR * rmr, tduffy> } tduffy> tduffy> is_signaled = tduffy> - (completion_flags & DAT_COMPLETION_SUPPRESS_FLAG) ? DAT_FALSE : tduffy> - DAT_TRUE; tduffy> + (completion_flags & DAT_COMPLETION_SUPPRESS_FLAG) ? FALSE : tduffy> + TRUE; tduffy> tduffy> /* tduffy> * Take reference before posting to avoid race conditions with tduffy> @@ -196,7 +196,7 @@ dapl_rmr_bind_unfuse(DAPL_RMR * rmr, tduffy> { tduffy> DAPL_COOKIE *cookie; tduffy> DAT_RETURN dat_status; tduffy> - DAT_BOOLEAN is_signaled; tduffy> + boolean_t is_signaled; tduffy> tduffy> dat_status = DAT_SUCCESS; tduffy> /* tduffy> @@ -246,8 +246,8 @@ dapl_rmr_bind_unfuse(DAPL_RMR * rmr, tduffy> } tduffy> tduffy> is_signaled = tduffy> - (completion_flags & DAT_COMPLETION_UNSIGNALLED_FLAG) ? DAT_FALSE : tduffy> - DAT_TRUE; tduffy> + (completion_flags & DAT_COMPLETION_UNSIGNALLED_FLAG) ? FALSE : tduffy> + TRUE; tduffy> tduffy> /* tduffy> * Take reference before posting to avoid race conditions with tduffy> Index: linux-kernel/dat-provider/dapl_openib_util.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_openib_util.c (revision 2295) tduffy> +++ linux-kernel/dat-provider/dapl_openib_util.c (working copy) tduffy> @@ -623,7 +623,7 @@ dapl_ib_mw_bind(DAPL_RMR * rmr, tduffy> DAPL_COOKIE * cookie, tduffy> DAT_VADDR virtual_address, tduffy> DAT_VLEN length, tduffy> - DAT_MEM_PRIV_FLAGS mem_priv, DAT_BOOLEAN is_signaled) tduffy> + DAT_MEM_PRIV_FLAGS mem_priv, boolean_t is_signaled) tduffy> { tduffy> int ib_status; tduffy> struct ib_mw_bind mw_bind_prop; tduffy> @@ -632,7 +632,7 @@ dapl_ib_mw_bind(DAPL_RMR * rmr, tduffy> mw_bind_prop.length = length; tduffy> mw_bind_prop.mw_access_flags = dapl_ib_convert_mem_privileges(mem_priv); tduffy> mw_bind_prop.send_flags = tduffy> - (is_signaled == DAT_TRUE) ? IB_SEND_SIGNALED : 0; tduffy> + (is_signaled == TRUE) ? IB_SEND_SIGNALED : 0; tduffy> mw_bind_prop.mr = lmr->mr_handle; tduffy> mw_bind_prop.wr_id = (u64) (uintptr_t) cookie; tduffy> dapl_dbg_log(DAPL_DBG_TYPE_UTIL, tduffy> @@ -673,7 +673,7 @@ dapl_ib_mw_bind(DAPL_RMR * rmr, tduffy> */ tduffy> DAT_RETURN tduffy> dapl_ib_mw_unbind(DAPL_RMR * rmr, tduffy> - DAPL_EP * ep, DAPL_COOKIE * cookie, DAT_BOOLEAN is_signaled) tduffy> + DAPL_EP * ep, DAPL_COOKIE * cookie, boolean_t is_signaled) tduffy> { tduffy> int ib_status; tduffy> struct ib_mw_bind mw_bind_prop; tduffy> @@ -682,7 +682,7 @@ dapl_ib_mw_unbind(DAPL_RMR * rmr, tduffy> mw_bind_prop.length = 0; tduffy> mw_bind_prop.mw_access_flags = 0; tduffy> mw_bind_prop.send_flags = tduffy> - (is_signaled == DAT_TRUE) ? IB_SEND_SIGNALED : 0; tduffy> + (is_signaled == TRUE) ? IB_SEND_SIGNALED : 0; tduffy> mw_bind_prop.mr = NULL; tduffy> mw_bind_prop.wr_id = (u64) (uintptr_t) cookie; tduffy> ib_status = ib_bind_mw(ep->qp_handle, rmr->mw_handle, &mw_bind_prop); tduffy> @@ -1051,7 +1051,7 @@ static struct ib_cm_event_map { tduffy> */ tduffy> tduffy> DAT_EVENT_NUMBER tduffy> -dapl_ib_get_dat_event(const ib_cm_events_t ib_cm_event, DAT_BOOLEAN active) tduffy> +dapl_ib_get_dat_event(const ib_cm_events_t ib_cm_event, boolean_t active) tduffy> { tduffy> DAT_EVENT_NUMBER dat_event_num; tduffy> int index; tduffy> Index: linux-kernel/dat-provider/dapl_ep_modify.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_ep_modify.c (revision 2295) tduffy> +++ linux-kernel/dat-provider/dapl_ep_modify.c (working copy) tduffy> @@ -322,18 +322,18 @@ DAT_RETURN dapl_ep_modify(DAT_EP_HANDLE tduffy> DAT_RETURN dat_status; tduffy> tduffy> /* Flag indicating we've allocated a new one of these. */ tduffy> - DAT_BOOLEAN qp_allocated = DAT_FALSE; tduffy> - DAT_BOOLEAN rqst_cb_allocated = DAT_FALSE; tduffy> - DAT_BOOLEAN recv_cb_allocated = DAT_FALSE; tduffy> - DAT_BOOLEAN rqst_iov_allocated = DAT_FALSE; tduffy> - DAT_BOOLEAN recv_iov_allocated = DAT_FALSE; tduffy> + boolean_t qp_allocated = FALSE; tduffy> + boolean_t rqst_cb_allocated = FALSE; tduffy> + boolean_t recv_cb_allocated = FALSE; tduffy> + boolean_t rqst_iov_allocated = FALSE; tduffy> + boolean_t recv_iov_allocated = FALSE; tduffy> tduffy> /* Flag indicating we've used (assigned to QP) a new one of these. */ tduffy> - DAT_BOOLEAN qp_used = DAT_FALSE; tduffy> - DAT_BOOLEAN rqst_cb_used = DAT_FALSE; tduffy> - DAT_BOOLEAN recv_cb_used = DAT_FALSE; tduffy> - DAT_BOOLEAN rqst_iov_used = DAT_FALSE; tduffy> - DAT_BOOLEAN recv_iov_used = DAT_FALSE; tduffy> + boolean_t qp_used = FALSE; tduffy> + boolean_t rqst_cb_used = FALSE; tduffy> + boolean_t recv_cb_used = FALSE; tduffy> + boolean_t rqst_iov_used = FALSE; tduffy> + boolean_t recv_iov_used = FALSE; tduffy> tduffy> dat_status = dapl_ep_modify_validate_parameters(ep_handle, tduffy> mask, tduffy> @@ -381,14 +381,14 @@ DAT_RETURN dapl_ep_modify(DAT_EP_HANDLE tduffy> if (DAT_SUCCESS != dat_status) { tduffy> goto bail; tduffy> } tduffy> - rqst_cb_allocated = DAT_TRUE; tduffy> + rqst_cb_allocated = TRUE; tduffy> tduffy> dat_status = dapl_cb_create(&alloc_ep.recv_buffer, ep1, /* For pointer in buffer bool. */ tduffy> ep_attr1.max_recv_dtos); tduffy> if (DAT_SUCCESS != dat_status) { tduffy> goto bail; tduffy> } tduffy> - recv_cb_allocated = DAT_TRUE; tduffy> + recv_cb_allocated = TRUE; tduffy> tduffy> alloc_ep.send_iov_num = ep_attr1.max_request_iov; tduffy> alloc_ep.send_iov = kmalloc(ep_attr1.max_request_iov tduffy> @@ -399,7 +399,7 @@ DAT_RETURN dapl_ep_modify(DAT_EP_HANDLE tduffy> DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> goto bail; tduffy> } tduffy> - recv_iov_allocated = DAT_TRUE; tduffy> + recv_iov_allocated = TRUE; tduffy> tduffy> alloc_ep.recv_iov_num = ep_attr1.max_recv_iov; tduffy> alloc_ep.recv_iov = kmalloc(ep_attr1.max_recv_iov tduffy> @@ -410,13 +410,13 @@ DAT_RETURN dapl_ep_modify(DAT_EP_HANDLE tduffy> DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> goto bail; tduffy> } tduffy> - recv_iov_allocated = DAT_TRUE; tduffy> + recv_iov_allocated = TRUE; tduffy> tduffy> dat_status = dapl_ib_qp_alloc(ia, &alloc_ep, ep1); tduffy> if (dat_status != DAT_SUCCESS) { tduffy> goto bail; tduffy> } tduffy> - qp_allocated = DAT_TRUE; tduffy> + qp_allocated = TRUE; tduffy> tduffy> /* tduffy> * Now we atomically modify the EP, under lock tduffy> @@ -473,26 +473,26 @@ DAT_RETURN dapl_ep_modify(DAT_EP_HANDLE tduffy> && (ep_param->ep_attr.max_recv_dtos tduffy> != ep2->param.ep_attr.max_recv_dtos)) { tduffy> new_ep.recv_buffer = alloc_ep.recv_buffer; tduffy> - recv_cb_used = DAT_TRUE; tduffy> + recv_cb_used = TRUE; tduffy> } tduffy> tduffy> if (mask & DAT_EP_FIELD_EP_ATTR_MAX_REQUEST_DTOS tduffy> && (ep_param->ep_attr.max_request_dtos tduffy> != ep2->param.ep_attr.max_request_dtos)) { tduffy> new_ep.req_buffer = alloc_ep.req_buffer; tduffy> - rqst_cb_used = DAT_TRUE; tduffy> + rqst_cb_used = TRUE; tduffy> } tduffy> tduffy> if (mask & DAT_EP_FIELD_EP_ATTR_MAX_RECV_IOV tduffy> && new_ep.recv_iov_num != ep2->recv_iov_num) { tduffy> new_ep.recv_iov = alloc_ep.recv_iov; tduffy> - recv_iov_used = DAT_TRUE; tduffy> + recv_iov_used = TRUE; tduffy> } tduffy> tduffy> if (mask & DAT_EP_FIELD_EP_ATTR_MAX_REQUEST_IOV tduffy> && new_ep.send_iov_num != ep2->send_iov_num) { tduffy> new_ep.send_iov = alloc_ep.send_iov; tduffy> - recv_iov_used = DAT_TRUE; tduffy> + recv_iov_used = TRUE; tduffy> } tduffy> tduffy> /* tduffy> Index: linux-kernel/dat-provider/dapl_cno_util.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_cno_util.c (revision 2295) tduffy> +++ linux-kernel/dat-provider/dapl_cno_util.c (working copy) tduffy> @@ -131,7 +131,7 @@ dapl_cno_trigger ( tduffy> cno_ptr->cno_upcall.upcall_func ( tduffy> cno_ptr->cno_upcall.instance_data, tduffy> &event, tduffy> - DAT_FALSE); tduffy> + FALSE); tduffy> } tduffy> dat_status = dapl_evd_dequeue ( (DAT_EVD_HANDLE)evd_ptr, &event); tduffy> } tduffy> Index: linux-kernel/dat-provider/dapl_timer_util.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_timer_util.c (revision 2295) tduffy> +++ linux-kernel/dat-provider/dapl_timer_util.c (working copy) tduffy> @@ -97,7 +97,7 @@ dapl_timer_set(DAPL_OS_TIMER * timer, tduffy> { tduffy> DAPL_OS_TIMER *list_ptr; tduffy> DAPL_OS_TIMEVAL cur_time; tduffy> - DAT_BOOLEAN wakeup_tmo_thread; tduffy> + boolean_t wakeup_tmo_thread; tduffy> tduffy> /* tduffy> * Start the timer thread the first time we need a timer tduffy> @@ -109,7 +109,7 @@ dapl_timer_set(DAPL_OS_TIMER * timer, tduffy> } tduffy> tduffy> dapl_llist_init_entry(&timer->list_entry); tduffy> - wakeup_tmo_thread = DAT_FALSE; tduffy> + wakeup_tmo_thread = FALSE; tduffy> dapl_os_get_time(&cur_time); tduffy> timer->expires = cur_time + expires; /* calculate future time */ tduffy> timer->function = func; tduffy> @@ -132,7 +132,7 @@ dapl_timer_set(DAPL_OS_TIMER * timer, tduffy> dapl_llist_add_head(&g_daplTimerHead.timer_list_head, tduffy> (DAPL_LLIST_ENTRY *) & timer->list_entry, tduffy> timer); tduffy> - wakeup_tmo_thread = DAT_TRUE; tduffy> + wakeup_tmo_thread = TRUE; tduffy> } else { tduffy> list_ptr = (DAPL_OS_TIMER *) tduffy> dapl_llist_peek_head(&g_daplTimerHead.timer_list_head); tduffy> @@ -142,7 +142,7 @@ dapl_timer_set(DAPL_OS_TIMER * timer, tduffy> dapl_llist_add_head(&g_daplTimerHead.timer_list_head, tduffy> (DAPL_LLIST_ENTRY *) & timer-> tduffy> list_entry, timer); tduffy> - wakeup_tmo_thread = DAT_TRUE; tduffy> + wakeup_tmo_thread = TRUE; tduffy> } else { tduffy> /* Case 3: figure out where entry goes in sorted list */ tduffy> list_ptr = tduffy> @@ -182,7 +182,7 @@ dapl_timer_set(DAPL_OS_TIMER * timer, tduffy> } tduffy> spin_unlock(&g_daplTimerHead.lock); tduffy> tduffy> - if (wakeup_tmo_thread == DAT_TRUE) { tduffy> + if (wakeup_tmo_thread == TRUE) { tduffy> dapl_os_wait_object_wakeup(&g_daplTimerHead.wait_object); tduffy> } tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_rsp_create.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_rsp_create.c (revision 2295) tduffy> +++ linux-kernel/dat-provider/dapl_rsp_create.c (working copy) tduffy> @@ -78,7 +78,7 @@ dapl_rsp_create(DAT_IA_HANDLE ia_handle, tduffy> DAPL_SP *sp_ptr; tduffy> DAPL_EVD *evd_ptr; tduffy> DAPL_EP *ep_ptr; tduffy> - DAT_BOOLEAN sp_found; tduffy> + boolean_t sp_found; tduffy> DAT_RETURN dat_status; tduffy> tduffy> dat_status = DAT_SUCCESS; tduffy> @@ -123,13 +123,13 @@ dapl_rsp_create(DAT_IA_HANDLE ia_handle, tduffy> goto bail; tduffy> } tduffy> tduffy> - sp_ptr = dapl_ia_sp_search(ia_ptr, conn_qual, DAT_FALSE); tduffy> - sp_found = DAT_TRUE; tduffy> + sp_ptr = dapl_ia_sp_search(ia_ptr, conn_qual, FALSE); tduffy> + sp_found = TRUE; tduffy> if (sp_ptr == NULL) { tduffy> - sp_found = DAT_FALSE; tduffy> + sp_found = FALSE; tduffy> tduffy> /* Allocate RSP */ tduffy> - sp_ptr = dapl_sp_alloc(ia_ptr, DAT_FALSE); tduffy> + sp_ptr = dapl_sp_alloc(ia_ptr, FALSE); tduffy> if (sp_ptr == NULL) { tduffy> dat_status = tduffy> DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, tduffy> @@ -162,9 +162,9 @@ dapl_rsp_create(DAT_IA_HANDLE ia_handle, tduffy> * even before this call returns! tduffy> */ tduffy> sp_ptr->state = DAPL_SP_STATE_RSP_LISTENING; tduffy> - sp_ptr->listening = DAT_TRUE; tduffy> + sp_ptr->listening = TRUE; tduffy> tduffy> - if (sp_found == DAT_FALSE) { tduffy> + if (sp_found == FALSE) { tduffy> /* Link it onto the IA */ tduffy> dapl_ia_link_rsp(ia_ptr, sp_ptr); tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_ep_util.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_ep_util.c (revision 2295) tduffy> +++ linux-kernel/dat-provider/dapl_ep_util.c (working copy) tduffy> @@ -359,7 +359,7 @@ void dapl_ep_timeout(uintptr_t arg) tduffy> *cm event number. tduffy> */ tduffy> ib_cm_event = dapl_ib_get_cm_event(DAT_CONNECTION_EVENT_DISCONNECTED); tduffy> - dapl_ib_disconnect_clean(ep_ptr, DAT_TRUE, ib_cm_event); tduffy> + dapl_ib_disconnect_clean(ep_ptr, TRUE, ib_cm_event); tduffy> tduffy> (void)dapl_evd_post_connection_event((DAPL_EVD *) ep_ptr->param. tduffy> connect_evd_handle, tduffy> Index: linux-kernel/dat-provider/dapl_rmr_util.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_rmr_util.h (revision 2295) tduffy> +++ linux-kernel/dat-provider/dapl_rmr_util.h (working copy) tduffy> @@ -49,7 +49,7 @@ extern DAPL_RMR *dapl_rmr_alloc(DAPL_PZ tduffy> tduffy> extern void dapl_rmr_dealloc(DAPL_RMR * rmr); tduffy> tduffy> -static __inline__ DAT_BOOLEAN tduffy> +static __inline__ boolean_t tduffy> dapl_rmr_validate_completion_flag(DAT_COMPLETION_FLAGS mask, tduffy> DAT_COMPLETION_FLAGS allow, tduffy> DAT_COMPLETION_FLAGS request); tduffy> @@ -60,15 +60,15 @@ dapl_rmr_validate_completion_flag(DAT_CO tduffy> * * tduffy> *********************************************************************/ tduffy> tduffy> -static __inline__ DAT_BOOLEAN tduffy> +static __inline__ boolean_t tduffy> dapl_rmr_validate_completion_flag(DAT_COMPLETION_FLAGS mask, tduffy> DAT_COMPLETION_FLAGS allow, tduffy> DAT_COMPLETION_FLAGS request) tduffy> { tduffy> if ((mask & request) && !(mask & allow)) { tduffy> - return DAT_FALSE; tduffy> + return FALSE; tduffy> } else { tduffy> - return DAT_TRUE; tduffy> + return TRUE; tduffy> } tduffy> } tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_rsp_free.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_rsp_free.c (revision 2295) tduffy> +++ linux-kernel/dat-provider/dapl_rsp_free.c (working copy) tduffy> @@ -112,13 +112,13 @@ DAT_RETURN dapl_rsp_free(DAT_RSP_HANDLE tduffy> * else the last disconnect on this RSP will free it up. The RSP tduffy> * is used to contain CR records for each connection, which tduffy> * contain information necessary to disconnect. tduffy> - * sp_ptr->listening will be DAT_TRUE if there has never been a tduffy> - * connection event, and DAT_FALSE if a connection attempt resulted tduffy> + * sp_ptr->listening will be TRUE if there has never been a tduffy> + * connection event, and FALSE if a connection attempt resulted tduffy> * in a reject. tduffy> */ tduffy> if (sp_ptr->cr_list_count == 0) { tduffy> /* This RSP has never been used. Clean it up */ tduffy> - sp_ptr->listening = DAT_FALSE; tduffy> + sp_ptr->listening = FALSE; tduffy> sp_ptr->state = DAPL_SP_STATE_FREE; tduffy> spin_unlock(&sp_ptr->header.lock); tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_ia_open.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_ia_open.c (revision 2295) tduffy> +++ linux-kernel/dat-provider/dapl_ia_open.c (working copy) tduffy> @@ -47,7 +47,7 @@ tduffy> * LOCAL PROTOTYPES tduffy> */ tduffy> tduffy> -void dapl_hca_cleanup(DAPL_HCA * hca_ptr, DAT_BOOLEAN dec_ref); tduffy> +void dapl_hca_cleanup(DAPL_HCA * hca_ptr, boolean_t dec_ref); tduffy> tduffy> /* tduffy> * Open a provider and return a handle. The handle enables the user tduffy> @@ -124,7 +124,7 @@ dapl_ia_open(const DAT_NAME_PTR name, tduffy> &hca_ptr->ia_attr, tduffy> NULL, &hca_ptr->hca_address); tduffy> if (dat_status != DAT_SUCCESS) { tduffy> - dapl_hca_cleanup(hca_ptr, DAT_FALSE); tduffy> + dapl_hca_cleanup(hca_ptr, FALSE); tduffy> spin_unlock(&hca_ptr->lock); tduffy> goto bail; tduffy> } tduffy> @@ -137,7 +137,7 @@ dapl_ia_open(const DAT_NAME_PTR name, tduffy> ia_ptr = dapl_ia_alloc(provider, hca_ptr); tduffy> if (!ia_ptr) { tduffy> spin_lock(&hca_ptr->lock); tduffy> - dapl_hca_cleanup(hca_ptr, DAT_TRUE); tduffy> + dapl_hca_cleanup(hca_ptr, TRUE); tduffy> spin_unlock(&hca_ptr->lock); tduffy> dat_status = tduffy> DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> @@ -173,7 +173,7 @@ dapl_ia_open(const DAT_NAME_PTR name, tduffy> goto bail; tduffy> } tduffy> tduffy> - ia_ptr->cleanup_async_error_evd = DAT_FALSE; tduffy> + ia_ptr->cleanup_async_error_evd = FALSE; tduffy> ia_ptr->async_error_evd = evd_ptr; tduffy> } else { tduffy> /* Verify we have >0 length, and let the provider check the size */ tduffy> @@ -204,13 +204,13 @@ dapl_ia_open(const DAT_NAME_PTR name, tduffy> dat_status = dapl_ia_setup_callbacks(ia_ptr, evd_ptr); tduffy> if (dat_status != DAT_SUCCESS) { tduffy> /* Assign the EVD so it gets cleaned up */ tduffy> - ia_ptr->cleanup_async_error_evd = DAT_TRUE; tduffy> + ia_ptr->cleanup_async_error_evd = TRUE; tduffy> ia_ptr->async_error_evd = evd_ptr; tduffy> goto bail; tduffy> } tduffy> } tduffy> tduffy> - ia_ptr->cleanup_async_error_evd = DAT_TRUE; tduffy> + ia_ptr->cleanup_async_error_evd = TRUE; tduffy> ia_ptr->async_error_evd = evd_ptr; tduffy> } tduffy> tduffy> @@ -236,10 +236,10 @@ dapl_ia_open(const DAT_NAME_PTR name, tduffy> * Clean up partially allocated HCA stuff. Strictly to make cleanup tduffy> * simple. tduffy> */ tduffy> -void dapl_hca_cleanup(DAPL_HCA * hca_ptr, DAT_BOOLEAN dec_ref) tduffy> +void dapl_hca_cleanup(DAPL_HCA * hca_ptr, boolean_t dec_ref) tduffy> { tduffy> dapl_ib_close_hca(hca_ptr); tduffy> - if (dec_ref == DAT_TRUE) { tduffy> + if (dec_ref == TRUE) { tduffy> atomic_dec(&hca_ptr->handle_ref_count); tduffy> } tduffy> } tduffy> Index: linux-kernel/dat-provider/dapl_evd_util.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_evd_util.c (revision 2295) tduffy> +++ linux-kernel/dat-provider/dapl_evd_util.c (working copy) tduffy> @@ -163,7 +163,7 @@ DAPL_EVD *dapl_evd_alloc(DAPL_IA * ia_pt tduffy> evd_ptr->evd_producer_locking_needed = 1; /* Conservative value. */ tduffy> evd_ptr->ib_cq_handle = IB_INVALID_HANDLE; tduffy> atomic_set(&evd_ptr->evd_ref_count, 0); tduffy> - evd_ptr->catastrophic_overflow = DAT_FALSE; tduffy> + evd_ptr->catastrophic_overflow = FALSE; tduffy> evd_ptr->qlen = qlen; tduffy> evd_ptr->cno_ptr = cno_ptr; tduffy> tduffy> @@ -476,7 +476,7 @@ dapl_evd_post_overflow_event(DAPL_EVD * tduffy> * In that case we've got a catastrophic overflow. tduffy> */ tduffy> if (async_evd_ptr == overflow_evd_ptr) { tduffy> - async_evd_ptr->catastrophic_overflow = DAT_TRUE; tduffy> + async_evd_ptr->catastrophic_overflow = TRUE; tduffy> async_evd_ptr->evd_state = DAPL_EVD_STATE_DEAD; tduffy> return; tduffy> } tduffy> @@ -484,7 +484,7 @@ dapl_evd_post_overflow_event(DAPL_EVD * tduffy> overflow_event = dapl_evd_get_event(overflow_evd_ptr); tduffy> if (!overflow_event) { tduffy> /* this is not good */ tduffy> - overflow_evd_ptr->catastrophic_overflow = DAT_TRUE; tduffy> + overflow_evd_ptr->catastrophic_overflow = TRUE; tduffy> overflow_evd_ptr->evd_state = DAPL_EVD_STATE_DEAD; tduffy> return; tduffy> } tduffy> Index: linux-kernel/dat-provider/dapl_sp_util.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_sp_util.c (revision 2295) tduffy> +++ linux-kernel/dat-provider/dapl_sp_util.c (working copy) tduffy> @@ -59,7 +59,7 @@ tduffy> * pointer to sp info struct tduffy> * tduffy> */ tduffy> -DAPL_SP *dapl_sp_alloc(DAPL_IA * ia_ptr, DAT_BOOLEAN is_psp) tduffy> +DAPL_SP *dapl_sp_alloc(DAPL_IA * ia_ptr, boolean_t is_psp) tduffy> { tduffy> DAPL_SP *sp_ptr; tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_hash.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_hash.c (revision 2295) tduffy> +++ linux-kernel/dat-provider/dapl_hash.c (working copy) tduffy> @@ -153,25 +153,24 @@ dapl_hash_rehash(DAPL_HASH_ELEM * elemen tduffy> * tduffy> * Inputs: tduffy> * table, key and datum to be added tduffy> - * allow_dup - DAT_TRUE if dups are allowed tduffy> + * allow_dup - TRUE if dups are allowed tduffy> * Outputs: tduffy> * report_dup - should you care to know tduffy> * Returns: tduffy> - * DAT_TRUE on success tduffy> + * TRUE on success tduffy> */ tduffy> -static DAT_BOOLEAN tduffy> -dapl_hash_add(DAPL_HASH_TABLEP p_table, tduffy> - DAPL_HASH_KEY key, tduffy> - void *datum, DAT_BOOLEAN allow_dup, DAT_BOOLEAN * report_dup) tduffy> +static boolean_t tduffy> +dapl_hash_add(DAPL_HASH_TABLEP p_table, DAPL_HASH_KEY key, void *datum, tduffy> + boolean_t allow_dup, boolean_t *report_dup) tduffy> { tduffy> void *olddatum; tduffy> DAPL_HASH_KEY hashValue, save_key = key; tduffy> DAPL_HASH_ELEM *found; tduffy> - DAT_BOOLEAN status = DAT_FALSE; tduffy> + boolean_t status = FALSE; tduffy> unsigned int chain_len = 0; tduffy> tduffy> if (report_dup) { tduffy> - (*report_dup) = DAT_FALSE; tduffy> + (*report_dup) = FALSE; tduffy> } tduffy> tduffy> if (NO_DATUM(datum)) { tduffy> @@ -181,7 +180,7 @@ dapl_hash_add(DAPL_HASH_TABLEP p_table, tduffy> dapl_dbg_log(DAPL_DBG_TYPE_ERR, tduffy> "dapl_hash_add () called with magic NO_DATA value (%p) " tduffy> "used as datum!\n", datum); tduffy> - return DAT_FALSE; tduffy> + return FALSE; tduffy> } tduffy> tduffy> DAPL_HASHLOOKUP(p_table, key, olddatum, found); tduffy> @@ -190,14 +189,14 @@ dapl_hash_add(DAPL_HASH_TABLEP p_table, tduffy> * key exists already tduffy> */ tduffy> if (report_dup) { tduffy> - *report_dup = DAT_TRUE; tduffy> + *report_dup = TRUE; tduffy> } tduffy> tduffy> if (!allow_dup) { tduffy> dapl_dbg_log(DAPL_DBG_TYPE_ERR, tduffy> "dapl_hash_add () called with duplicate key (" tduffy> F64x ")\n", key); tduffy> - return DAT_FALSE; tduffy> + return FALSE; tduffy> } tduffy> } tduffy> tduffy> @@ -211,7 +210,7 @@ dapl_hash_add(DAPL_HASH_TABLEP p_table, tduffy> p_table->table[hashValue].datum = datum; tduffy> p_table->table[hashValue].next_element = NULL; tduffy> p_table->num_entries++; tduffy> - status = DAT_TRUE; tduffy> + status = TRUE; tduffy> } else { tduffy> DAPL_HASH_ELEM *newelement = (DAPL_HASH_ELEM *) tduffy> kmalloc(sizeof(DAPL_HASH_ELEM), GFP_ATOMIC); tduffy> @@ -231,10 +230,10 @@ dapl_hash_add(DAPL_HASH_TABLEP p_table, tduffy> } tduffy> lastelement->next_element = newelement; tduffy> p_table->num_entries++; tduffy> - status = DAT_TRUE; tduffy> + status = TRUE; tduffy> } else { tduffy> /* allocation failed - should not happen */ tduffy> - status = DAT_FALSE; tduffy> + status = FALSE; tduffy> } tduffy> } tduffy> tduffy> @@ -262,9 +261,9 @@ dapl_hash_add(DAPL_HASH_TABLEP p_table, tduffy> * Inputs: tduffy> * element, key to be deleted tduffy> * Returns: tduffy> - * DAT_TRUE on success tduffy> + * TRUE on success tduffy> */ tduffy> -static DAT_BOOLEAN tduffy> +static boolean_t tduffy> dapl_hash_delete_element(DAPL_HASH_ELEM * element, tduffy> DAPL_HASH_KEY key, DAPL_HASH_DATA * p_datum) tduffy> { tduffy> @@ -309,7 +308,7 @@ dapl_hash_delete_element(DAPL_HASH_ELEM tduffy> } tduffy> } tduffy> tduffy> - return (curelement != NULL ? DAT_TRUE : DAT_FALSE); tduffy> + return (curelement != NULL ? TRUE : FALSE); tduffy> } tduffy> tduffy> /********************************************************************* tduffy> @@ -405,7 +404,7 @@ dapl_hash_insert(DAPL_HASH_TABLE * p_tab tduffy> dat_status = DAT_SUCCESS; tduffy> tduffy> spin_lock(&p_table->lock); tduffy> - if (!dapl_hash_add(p_table, key, data, DAT_FALSE, NULL)) { tduffy> + if (!dapl_hash_add(p_table, key, data, FALSE, NULL)) { tduffy> dat_status = tduffy> DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> } tduffy> Index: linux-kernel/dat-provider/dapl_sp_util.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_sp_util.h (revision 2295) tduffy> +++ linux-kernel/dat-provider/dapl_sp_util.h (working copy) tduffy> @@ -37,7 +37,7 @@ tduffy> #ifndef DAPL_PSP_UTIL_H tduffy> #define DAPL_PSP_UTIL_H tduffy> tduffy> -DAPL_SP *dapl_sp_alloc(DAPL_IA * ia_ptr, DAT_BOOLEAN is_psp); tduffy> +DAPL_SP *dapl_sp_alloc(DAPL_IA * ia_ptr, boolean_t is_psp); tduffy> tduffy> void dapl_sp_free_sp(DAPL_SP * sp_ptr); tduffy> tduffy> Index: linux-kernel/dat-provider/dapl.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl.h (revision 2295) tduffy> +++ linux-kernel/dat-provider/dapl.h (working copy) tduffy> @@ -240,7 +240,7 @@ struct dapl_ia { tduffy> DAPL_HEADER header; tduffy> DAPL_HCA *hca_ptr; tduffy> DAPL_EVD *async_error_evd; tduffy> - DAT_BOOLEAN cleanup_async_error_evd; tduffy> + boolean_t cleanup_async_error_evd; tduffy> tduffy> DAPL_LLIST_ENTRY hca_ia_list_entry; /* HCAs list of IAs */ tduffy> DAPL_LLIST_HEAD ep_list_head; /* EP queue */ tduffy> @@ -269,7 +269,7 @@ struct dapl_evd tduffy> DAT_EVD_FLAGS evd_flags; tduffy> tduffy> /* Derived from evd_flags; see dapls_evd_internal_create. */ tduffy> - DAT_BOOLEAN evd_producer_locking_needed; tduffy> + boolean_t evd_producer_locking_needed; tduffy> tduffy> /* Every EVD has a CQ unless it is a SOFTWARE_EVENT only EVD */ tduffy> ib_cq_handle_t ib_cq_handle; tduffy> @@ -280,7 +280,7 @@ struct dapl_evd tduffy> atomic_t evd_ref_count; tduffy> tduffy> /* Set if there has been a catastrophic overflow */ tduffy> - DAT_BOOLEAN catastrophic_overflow; tduffy> + boolean_t catastrophic_overflow; tduffy> tduffy> /* the actual events */ tduffy> DAT_COUNT qlen; tduffy> @@ -395,7 +395,7 @@ struct dapl_sp { tduffy> DAT_EP_HANDLE ep_handle; tduffy> tduffy> /* maintenence fields */ tduffy> - DAT_BOOLEAN listening; /* PSP is registered & active */ tduffy> + boolean_t listening; /* PSP is registered & active */ tduffy> ib_cm_srvc_handle_t cm_srvc_handle; /* Used by CM */ tduffy> DAPL_LLIST_HEAD cr_list_head; /* CR pending queue */ tduffy> DAT_COUNT cr_list_count; /* count of CRs on queue */ tduffy> @@ -617,8 +617,8 @@ extern DAT_RETURN dapl_ep_post_rdma_writ tduffy> tduffy> extern DAT_RETURN dapl_ep_get_status(DAT_EP_HANDLE, /* ep_handle */ tduffy> DAT_EP_STATE *, /* ep_state */ tduffy> - DAT_BOOLEAN *, /* in_dto_idle */ tduffy> - DAT_BOOLEAN *); /* out_dto_idle */ tduffy> + boolean_t *, /* in_dto_idle */ tduffy> + boolean_t *); /* out_dto_idle */ tduffy> tduffy> extern DAT_RETURN dapl_ep_free(DAT_EP_HANDLE); /* ep_handle */ tduffy> tduffy> @@ -757,7 +757,7 @@ extern void dapl_llist_init_head(DAPL_LL tduffy> tduffy> extern void dapl_llist_init_entry(DAPL_LLIST_ENTRY * entry); tduffy> tduffy> -extern DAT_BOOLEAN dapl_llist_is_empty(DAPL_LLIST_HEAD * head); tduffy> +extern boolean_t dapl_llist_is_empty(DAPL_LLIST_HEAD * head); tduffy> tduffy> extern void dapl_llist_add_head(DAPL_LLIST_HEAD * head, tduffy> DAPL_LLIST_ENTRY * entry, void *data); tduffy> Index: linux-kernel/dat-provider/dapl_psp_query.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_psp_query.c (revision 2295) tduffy> +++ linux-kernel/dat-provider/dapl_psp_query.c (working copy) tduffy> @@ -45,7 +45,7 @@ DAT_RETURN dapl_psp_query(DAT_PSP_HANDLE tduffy> DAT_RETURN status; tduffy> tduffy> if (DAPL_BAD_HANDLE(psp_handle, DAPL_MAGIC_PSP) || tduffy> - ((DAPL_SP *) psp_handle)->listening != DAT_TRUE) { tduffy> + ((DAPL_SP *) psp_handle)->listening != TRUE) { tduffy> status = tduffy> DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PSP); tduffy> goto bail; tduffy> Index: linux-kernel/dat-provider/dapl_llist.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_llist.c (revision 2295) tduffy> +++ linux-kernel/dat-provider/dapl_llist.c (working copy) tduffy> @@ -80,7 +80,7 @@ void dapl_llist_init_entry(DAPL_LLIST_EN tduffy> * tduffy> * Purpose: returns TRUE if the linked list is empty tduffy> */ tduffy> -DAT_BOOLEAN dapl_llist_is_empty(DAPL_LLIST_HEAD * head) tduffy> +boolean_t dapl_llist_is_empty(DAPL_LLIST_HEAD * head) tduffy> { tduffy> return (*head == NULL); tduffy> } tduffy> Index: linux-kernel/dat/dictionary.c tduffy> =================================================================== tduffy> --- linux-kernel/dat/dictionary.c (revision 2295) tduffy> +++ linux-kernel/dat/dictionary.c (working copy) tduffy> @@ -62,7 +62,7 @@ static DAT_RETURN tduffy> dat_dictionary_key_dup(const struct dat_provider_info *old_key, tduffy> struct dat_provider_info *new_key); tduffy> tduffy> -static DAT_BOOLEAN tduffy> +static boolean_t tduffy> dat_dictionary_key_is_equal(const struct dat_provider_info *key_a, tduffy> const struct dat_provider_info *key_b); tduffy> tduffy> @@ -270,7 +270,7 @@ DAT_RETURN dat_dictionary_search(DAT_DIC tduffy> tduffy> for (cur_node = p_dictionary->head->next; tduffy> p_dictionary->tail != cur_node; cur_node = cur_node->next) { tduffy> - if (DAT_TRUE == dat_dictionary_key_is_equal(&cur_node->key, tduffy> + if (TRUE == dat_dictionary_key_is_equal(&cur_node->key, tduffy> key)) { tduffy> if (NULL != p_data) tduffy> *p_data = cur_node->data; tduffy> @@ -335,7 +335,7 @@ DAT_RETURN dat_dictionary_remove(DAT_DIC tduffy> tduffy> for (cur_node = p_dictionary->head->next; tduffy> p_dictionary->tail != cur_node; cur_node = cur_node->next) { tduffy> - if (DAT_TRUE == dat_dictionary_key_is_equal(&cur_node->key, tduffy> + if (TRUE == dat_dictionary_key_is_equal(&cur_node->key, tduffy> key)) { tduffy> if (NULL != p_data) tduffy> *p_data = cur_node->data; tduffy> @@ -387,7 +387,7 @@ DAT_RETURN dat_dictionary_key_dup(const tduffy> * Function: dat_dictionary_key_is_equal tduffy> ***********************************************************************/ tduffy> tduffy> -DAT_BOOLEAN dat_dictionary_key_is_equal(const struct dat_provider_info *key_a, tduffy> +boolean_t dat_dictionary_key_is_equal(const struct dat_provider_info *key_a, tduffy> const struct dat_provider_info *key_b) tduffy> { tduffy> BUG_ON(NULL == key_a); tduffy> @@ -398,8 +398,8 @@ DAT_BOOLEAN dat_dictionary_key_is_equal( tduffy> && (key_a->dat_version_major == key_b->dat_version_major) tduffy> && (key_a->dat_version_minor == key_b->dat_version_minor) tduffy> && (key_a->is_thread_safe == key_b->is_thread_safe)) { tduffy> - return DAT_TRUE; tduffy> + return TRUE; tduffy> } else { tduffy> - return DAT_FALSE; tduffy> + return FALSE; tduffy> } tduffy> } tduffy> Index: linux-kernel/dat/core.c tduffy> =================================================================== tduffy> --- linux-kernel/dat/core.c (revision 2295) tduffy> +++ linux-kernel/dat/core.c (working copy) tduffy> @@ -133,19 +133,19 @@ module_exit(dat_fini); tduffy> * of handling DAT API calls and false otherwise. tduffy> */ tduffy> tduffy> -DAT_BOOLEAN dat_check_state(void) tduffy> +boolean_t dat_check_state(void) tduffy> { tduffy> - DAT_BOOLEAN status; tduffy> + boolean_t status; tduffy> tduffy> if ( (DAT_MODULE_STATE_UNINITIALIZED == g_state) || tduffy> (DAT_MODULE_STATE_INITIALIZING == g_state) ) { tduffy> dat_init(); tduffy> - status = DAT_TRUE; tduffy> + status = TRUE; tduffy> } else if ( (DAT_MODULE_STATE_DEINITIALIZED == g_state) || tduffy> (DAT_MODULE_STATE_DEINITIALIZING == g_state) ) { tduffy> - status = DAT_FALSE; tduffy> + status = FALSE; tduffy> } else { tduffy> - status = DAT_TRUE; tduffy> + status = TRUE; tduffy> } tduffy> tduffy> return status; tduffy> Index: linux-kernel/dat/api.c tduffy> =================================================================== tduffy> --- linux-kernel/dat/api.c (revision 2295) tduffy> +++ linux-kernel/dat/api.c (working copy) tduffy> @@ -580,7 +580,7 @@ DAT_RETURN dat_ia_openv(const DAT_NAME_P tduffy> DAT_EVD_HANDLE *async_event_handle, tduffy> DAT_IA_HANDLE *ia_handle, tduffy> u32 dat_major, u32 dat_minor, tduffy> - DAT_BOOLEAN thread_safety) tduffy> + boolean_t thread_safety) tduffy> { tduffy> DAT_IA_OPEN_FUNC ia_open_func; tduffy> struct dat_provider_info info; tduffy> @@ -602,7 +602,7 @@ DAT_RETURN dat_ia_openv(const DAT_NAME_P tduffy> return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG1); tduffy> } tduffy> tduffy> - if (DAT_FALSE == dat_check_state()) { tduffy> + if (FALSE == dat_check_state()) { tduffy> return DAT_ERROR(DAT_INVALID_STATE, 0); tduffy> } tduffy> tduffy> @@ -642,7 +642,7 @@ DAT_RETURN dat_ia_close(DAT_IA_HANDLE ia tduffy> dat_dbg_print(DAT_DBG_CLASS_CONSUMER_API, tduffy> "%s(ia:%x, flags:%x)\n", __func__, ia_handle, flags); tduffy> tduffy> - if (DAT_FALSE == dat_check_state()) { tduffy> + if (FALSE == dat_check_state()) { tduffy> return DAT_ERROR(DAT_INVALID_STATE, 0); tduffy> } tduffy> tduffy> @@ -739,7 +739,7 @@ DAT_RETURN dat_registry_add_provider(con tduffy> if (DAT_IS_BAD_POINTER(provider_info)) tduffy> return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); tduffy> tduffy> - if (DAT_FALSE == dat_check_state()) tduffy> + if (FALSE == dat_check_state()) tduffy> return DAT_ERROR(DAT_INVALID_STATE, 0); tduffy> tduffy> return dat_dr_insert(provider_info, provider->ia_open_func); tduffy> @@ -761,7 +761,7 @@ DAT_RETURN dat_registry_remove_provider( tduffy> if (DAT_IS_BAD_POINTER(provider)) tduffy> return DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG1); tduffy> tduffy> - if (DAT_FALSE == dat_check_state()) tduffy> + if (FALSE == dat_check_state()) tduffy> return DAT_ERROR(DAT_INVALID_STATE, 0); tduffy> tduffy> return dat_dr_remove(provider_info); tduffy> @@ -785,7 +785,7 @@ DAT_RETURN dat_registry_list_providers(D tduffy> __func__, max_to_return, entries_returned, tduffy> dat_provider_list); tduffy> tduffy> - if (DAT_FALSE == dat_check_state()) tduffy> + if (FALSE == dat_check_state()) tduffy> return DAT_ERROR(DAT_INVALID_STATE, 0); tduffy> tduffy> if ((DAT_IS_BAD_POINTER(entries_returned))) tduffy> Index: linux-kernel/dat/dat.h tduffy> =================================================================== tduffy> --- linux-kernel/dat/dat.h (revision 2295) tduffy> +++ linux-kernel/dat/dat.h (working copy) tduffy> @@ -35,7 +35,7 @@ tduffy> tduffy> #define DAT_VERSION_MAJOR 1 tduffy> #define DAT_VERSION_MINOR 2 tduffy> -#define DAT_THREADSAFE DAT_FALSE tduffy> +#define DAT_THREADSAFE FALSE tduffy> tduffy> /* unsigned longest native to compiler */ tduffy> typedef unsigned long long DAT_UVERYLONG; tduffy> @@ -364,10 +364,12 @@ struct dat_named_attr { tduffy> const char *value; tduffy> }; tduffy> tduffy> -typedef enum dat_boolean { tduffy> - DAT_FALSE = 0, tduffy> - DAT_TRUE = 1 tduffy> -} DAT_BOOLEAN; tduffy> +typedef enum { tduffy> +#undef FALSE tduffy> + FALSE = 0, tduffy> +#undef TRUE tduffy> + TRUE = 1 tduffy> +} boolean_t; tduffy> tduffy> typedef union dat_context { tduffy> DAT_PVOID as_ptr; tduffy> @@ -930,7 +932,7 @@ typedef enum dat_upcall_policy { tduffy> } DAT_UPCALL_POLICY; tduffy> tduffy> typedef void (*DAT_UPCALL_FUNC) (DAT_PVOID, const struct dat_event *, tduffy> - DAT_BOOLEAN); tduffy> + boolean_t); tduffy> tduffy> struct dat_upcall_object { tduffy> DAT_PVOID instance_data; tduffy> @@ -950,7 +952,7 @@ struct dat_provider_info { tduffy> char ia_name[DAT_NAME_MAX_LENGTH]; tduffy> u32 dat_version_major; tduffy> u32 dat_version_minor; tduffy> - DAT_BOOLEAN is_thread_safe; tduffy> + boolean_t is_thread_safe; tduffy> }; tduffy> tduffy> struct dat_evd_param { tduffy> @@ -1033,8 +1035,8 @@ struct dat_ia_attr { tduffy> DAT_COUNT max_iov_segments_per_rdma_write; tduffy> DAT_COUNT max_rdma_read_in; tduffy> DAT_COUNT max_rdma_read_out; tduffy> - DAT_BOOLEAN max_rdma_read_per_ep_in_guaranteed; tduffy> - DAT_BOOLEAN max_rdma_read_per_ep_out_guaranteed; tduffy> + boolean_t max_rdma_read_per_ep_in_guaranteed; tduffy> + boolean_t max_rdma_read_per_ep_out_guaranteed; tduffy> DAT_COUNT num_transport_attr; tduffy> struct dat_named_attr *transport_attr; tduffy> DAT_COUNT num_vendor_attr; tduffy> @@ -1079,21 +1081,21 @@ struct dat_provider_attr { tduffy> DAT_IOV_OWNERSHIP iov_ownership_on_return; tduffy> DAT_QOS dat_qos_supported; tduffy> DAT_COMPLETION_FLAGS completion_flags_supported; tduffy> - DAT_BOOLEAN is_thread_safe; tduffy> + boolean_t is_thread_safe; tduffy> DAT_COUNT max_private_data_size; tduffy> - DAT_BOOLEAN supports_multipath; tduffy> + boolean_t supports_multipath; tduffy> DAT_EP_CREATOR_FOR_PSP ep_creator; tduffy> DAT_UPCALL_POLICY upcall_policy; tduffy> u32 optimal_buffer_alignment; tduffy> - const DAT_BOOLEAN evd_stream_merging_supported[6][6]; tduffy> - DAT_BOOLEAN srq_supported; tduffy> + const boolean_t evd_stream_merging_supported[6][6]; tduffy> + boolean_t srq_supported; tduffy> DAT_COUNT srq_watermarks_supported; tduffy> - DAT_BOOLEAN srq_ep_pz_difference_supported; tduffy> + boolean_t srq_ep_pz_difference_supported; tduffy> DAT_COUNT srq_info_supported; tduffy> DAT_COUNT ep_recv_info_supported; tduffy> - DAT_BOOLEAN lmr_sync_req; tduffy> - DAT_BOOLEAN dto_async_return_guaranteed; tduffy> - DAT_BOOLEAN rdma_write_for_rdma_read_req; tduffy> + boolean_t lmr_sync_req; tduffy> + boolean_t dto_async_return_guaranteed; tduffy> + boolean_t rdma_write_for_rdma_read_req; tduffy> DAT_COUNT num_provider_specific_attr; tduffy> struct dat_named_attr *provider_specific_attr; tduffy> }; tduffy> @@ -1110,7 +1112,7 @@ typedef DAT_RETURN (*DAT_IA_OPEN_FUNC) ( tduffy> tduffy> typedef DAT_RETURN (*DAT_IA_OPENV_FUNC) ( tduffy> const DAT_NAME_PTR, DAT_COUNT, DAT_EVD_HANDLE *, DAT_IA_HANDLE *, tduffy> - u32, u32, DAT_BOOLEAN); tduffy> + u32, u32, boolean_t); tduffy> tduffy> typedef DAT_RETURN (*DAT_IA_CLOSE_FUNC) (DAT_IA_HANDLE, DAT_CLOSE_FLAGS); tduffy> tduffy> @@ -1185,7 +1187,7 @@ typedef DAT_RETURN (*DAT_EP_POST_RDMA_WR tduffy> const struct dat_rmr_triplet *, DAT_COMPLETION_FLAGS); tduffy> tduffy> typedef DAT_RETURN (*DAT_EP_GET_STATUS_FUNC) ( tduffy> - DAT_EP_HANDLE, DAT_EP_STATE *, DAT_BOOLEAN *, DAT_BOOLEAN *); tduffy> + DAT_EP_HANDLE, DAT_EP_STATE *, boolean_t *, boolean_t *); tduffy> tduffy> typedef DAT_RETURN (*DAT_EP_FREE_FUNC) (DAT_EP_HANDLE); tduffy> tduffy> @@ -1381,7 +1383,7 @@ extern DAT_RETURN dat_registry_remove_pr tduffy> */ tduffy> extern DAT_RETURN dat_ia_openv(const DAT_NAME_PTR, DAT_COUNT, tduffy> DAT_EVD_HANDLE *, DAT_IA_HANDLE *, tduffy> - u32, u32, DAT_BOOLEAN); tduffy> + u32, u32, boolean_t); tduffy> tduffy> #define dat_ia_open(name, qlen, async_evd, ia) \ tduffy> dat_ia_openv((name), (qlen), (async_evd), (ia), \ tduffy> @@ -1581,8 +1583,8 @@ static inline DAT_RETURN dat_ep_free(DAT tduffy> tduffy> static inline DAT_RETURN dat_ep_get_status(DAT_EP_HANDLE ep, tduffy> DAT_EP_STATE * state, tduffy> - DAT_BOOLEAN * recv_idle, tduffy> - DAT_BOOLEAN * req_idle) tduffy> + boolean_t *recv_idle, tduffy> + boolean_t *req_idle) tduffy> { tduffy> return DAT_CALL_PROVIDER_FUNC( tduffy> ep_get_status_func, ep, state, recv_idle, req_idle); tduffy> Index: linux-kernel/dat/core.h tduffy> =================================================================== tduffy> --- linux-kernel/dat/core.h (revision 2295) tduffy> +++ linux-kernel/dat/core.h (working copy) tduffy> @@ -65,6 +65,6 @@ typedef int DAT_DBG_CLASS; tduffy> tduffy> extern void dat_dbg_print(DAT_DBG_CLASS val, const char *fmt, ...); tduffy> tduffy> -extern DAT_BOOLEAN dat_check_state(void); tduffy> +extern boolean_t dat_check_state(void); tduffy> tduffy> #endif /* CORE_H */ tduffy> From blist at aon.at Wed May 11 09:25:07 2005 From: blist at aon.at (Bernhard Fischer) Date: Wed, 11 May 2005 18:25:07 +0200 Subject: [openib-general] perftest-03: put C code in .c file In-Reply-To: <20050511151105.GC10070@esmail.cup.hp.com> References: <20050511014014.GG5495@esmail.cup.hp.com> <20050511130004.GB7621@aon.at> <20050511151105.GC10070@esmail.cup.hp.com> Message-ID: <20050511162506.GD7621@aon.at> On Wed, May 11, 2005 at 08:11:05AM -0700, Grant Grundler wrote: >On Wed, May 11, 2005 at 03:00:04PM +0200, Bernhard Fischer wrote: >> >- gcc $(CFLAGS) -o rdma_lat rdma_lat.c -libverbs >> >> please replace that gcc with $(CC) > >yes, good idea. Patch below (perftest-04) removes the line. >Applies on top of perftest-03. With the attached (let's call it perftest-05 as it's on top of -04) it links again on i386 (get_clock vs. get_cycles), furthermore some sparse warning about non-ANSI fn decls, and 0 vs. NULL. Also make pp_client_exch_dest and pp_server_connect static. Please consider. Signed-off-by: Bernhard Fischer > >thanks, >grant -------------- next part -------------- diff -rup perftest-04.oorig/get_clock.h perftest/get_clock.h --- perftest-04.oorig/get_clock.h 2005-05-11 16:05:48.000000000 +0000 +++ perftest/get_clock.h 2005-05-11 16:09:39.000000000 +0000 @@ -34,7 +34,7 @@ #if defined (__x86_64__) || defined(__i386__) typedef unsigned long long cycles_t; -static inline cycles_t get_clock() +static inline cycles_t get_cycles(void) { unsigned low, high; unsigned long long val; @@ -45,7 +45,7 @@ static inline cycles_t get_clock() } #elif defined(__PPC64__) typedef unsigned long long cycles_t; -static inline cycles_t get_cycles() +static inline cycles_t get_cycles(void) { cycles_t ret; @@ -54,7 +54,7 @@ static inline cycles_t get_cycles() } #elif defined(__ia64__) typedef unsigned long long cycles_t; -static inline cycles_t get_cycles() +static inline cycles_t get_cycles(void) { cycles_t ret; diff -rup perftest-04.oorig/rdma_lat.c perftest/rdma_lat.c --- perftest-04.oorig/rdma_lat.c 2005-05-11 10:59:37.000000000 +0000 +++ perftest/rdma_lat.c 2005-05-11 16:10:42.000000000 +0000 @@ -163,7 +163,7 @@ static int pp_client_connect(const char return sockfd; } -struct pingpong_dest * pp_client_exch_dest(int sockfd, +static struct pingpong_dest * pp_client_exch_dest(int sockfd, const struct pingpong_dest *my_dest) { struct pingpong_dest *rem_dest = NULL; @@ -202,7 +202,7 @@ out: return rem_dest; } -int pp_server_connect(int port) +static int pp_server_connect(int port) { struct addrinfo *res, *t; struct addrinfo hints = { @@ -244,7 +244,7 @@ int pp_server_connect(int port) } listen(sockfd, 1); - connfd = accept(sockfd, NULL, 0); + connfd = accept(sockfd, NULL, NULL); if (connfd < 0) { perror("server accept"); fprintf(stderr, "accept() failed\n"); From blist at aon.at Wed May 11 09:34:00 2005 From: blist at aon.at (Bernhard Fischer) Date: Wed, 11 May 2005 18:34:00 +0200 Subject: [openib-general] perftest-03: put C code in .c file In-Reply-To: <20050511162506.GD7621@aon.at> References: <20050511014014.GG5495@esmail.cup.hp.com> <20050511130004.GB7621@aon.at> <20050511151105.GC10070@esmail.cup.hp.com> <20050511162506.GD7621@aon.at> Message-ID: <20050511163400.GE7621@aon.at> On Wed, May 11, 2005 at 06:25:07PM +0200, Bernhard Fischer wrote: >On Wed, May 11, 2005 at 08:11:05AM -0700, Grant Grundler wrote: >>On Wed, May 11, 2005 at 03:00:04PM +0200, Bernhard Fischer wrote: >>> >- gcc $(CFLAGS) -o rdma_lat rdma_lat.c -libverbs >>> >>> please replace that gcc with $(CC) >> >>yes, good idea. Patch below (perftest-04) removes the line. >>Applies on top of perftest-03. > >With the attached (let's call it perftest-05 as it's on top of -04) it >links again on i386 (get_clock vs. get_cycles), >furthermore some sparse warning about non-ANSI fn decls, and 0 vs. NULL. >Also make pp_client_exch_dest and pp_server_connect static. > >Please consider. > >Signed-off-by: Bernhard Fischer s/\.com/\.at/ whatever. From blist at aon.at Wed May 11 10:21:16 2005 From: blist at aon.at (Bernhard Fischer) Date: Wed, 11 May 2005 19:21:16 +0200 Subject: [openib-general] [PATCH] dapltest typo in documentation Message-ID: <20050511172116.GF7621@aon.at> Hi James, - typo in documentation Signed-off-by: Bernhard Fischer -------------- next part -------------- diff -rup test.oorig/dapltest/README test/dapltest/README --- test.oorig/dapltest/README 2005-05-04 12:19:58.000000000 +0000 +++ test/dapltest/README 2005-05-11 17:05:42.000000000 +0000 @@ -175,7 +175,7 @@ USAGE - Performance test client of data transfers will be made over each endpoint. Default: 1000 - [ -p pipeline ] Specify the pipline length, valid arguments are in + [ -p pipeline ] Specify the pipeline length, valid arguments are in the range [0,MAX_SEND_DTOS]. If a value greater than MAX_SEND_DTOS is requested the value will be adjusted down to MAX_SEND_DTOS. diff -rup test.oorig/dapltest/cmd/dapl_performance_cmd.c test/dapltest/cmd/dapl_performance_cmd.c --- test.oorig/dapltest/cmd/dapl_performance_cmd.c 2005-05-09 09:34:06.000000000 +0000 +++ test/dapltest/cmd/dapl_performance_cmd.c 2005-05-11 17:05:17.000000000 +0000 @@ -37,7 +37,7 @@ DT_Performance_Cmd_Usage (void) DT_Mdep_printf ("USAGE: [-D ]\n"); DT_Mdep_printf ("USAGE: [-d] : debug (zero)\n"); DT_Mdep_printf ("USAGE: [-i ] : (1, 000)\n"); - DT_Mdep_printf ("USAGE: [-p ]\n"); + DT_Mdep_printf ("USAGE: [-p ]\n"); DT_Mdep_printf ("USAGE: [-R ]\n"); DT_Mdep_printf ("USAGE: (BE == QOS_BEST_EFFORT - Default)\n"); DT_Mdep_printf ("USAGE: (HT == QOS_HIGH_THROUGHPUT)\n"); @@ -213,12 +213,12 @@ DT_Performance_Cmd_Parse ( cmd->num_iterations = atol (opts->optarg); break; } - case 'p': /* pipline size */ + case 'p': /* pipeline size */ { len = strspn (opts->optarg, "0123456789"); if (len == 0 || len != strlen (opts->optarg)) { - DT_Mdep_printf ("Syntax Error -p option\n"); + DT_Mdep_printf ("Syntax Error -p option\n"); DT_Performance_Cmd_Usage (); return (false); } From tduffy at sun.com Wed May 11 10:29:38 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 11 May 2005 10:29:38 -0700 Subject: [openib-general] [PATCH] kdapltest: remove bool, use dat's boolean_t In-Reply-To: References: <1115747465.8222.15.camel@duffman> <20050511080547.GC13349@mellanox.co.il> Message-ID: <1115832578.11344.5.camel@duffman> On Wed, 2005-05-11 at 10:49 -0400, James Lentini wrote: > > Surprisingly, some DAT code just uses "bool" (below). > > Good point. This is test code, but we should still change it to be > consistent. Signed-off-by: Tom Duffy Index: linux-kernel/test/dapltest/test/dapl_performance_util.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_performance_util.c (revision 2306) +++ linux-kernel/test/dapltest/test/dapl_performance_util.c (working copy) @@ -30,7 +30,7 @@ #define DT_Mdep_GetContextSwitchNum() 0 /* FIXME */ /****************************************************************************/ -bool +boolean_t DT_Performance_Test_Create ( Per_Test_Data_t *pt_ptr, DAT_IA_HANDLE *ia_handle, @@ -52,7 +52,7 @@ DT_Performance_Test_Create ( sizeof (Performance_Test_t)); if ( NULL == test_ptr ) { - return false; + return FALSE; } *perf_test = test_ptr; @@ -72,7 +72,7 @@ DT_Performance_Test_Create ( { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ia_query error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - return false; + return FALSE; } pipeline_len = DT_min ( @@ -101,7 +101,7 @@ DT_Performance_Test_Create ( DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_pz_create error: %s\n", test_ptr->base_port, DT_RetToString (ret)); test_ptr->pz_handle = DAT_HANDLE_NULL; - return false; + return FALSE; } /* create 4 EVDs - recv, request+RMR, conn-request, connect */ @@ -115,7 +115,7 @@ DT_Performance_Test_Create ( DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (recv) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); test_ptr->recv_evd_hdl = DAT_HANDLE_NULL; - return false; + return FALSE; } ret = DT_Tdep_evd_create (test_ptr->ia_handle, @@ -128,7 +128,7 @@ DT_Performance_Test_Create ( DT_Tdep_PT_Printf (phead, "Test[" F64x "]: 0x%x dat_evd_create (request) error: %s\n", test_ptr->reqt_evd_length,test_ptr->base_port, DT_RetToString (ret)); test_ptr->reqt_evd_hdl = DAT_HANDLE_NULL; - return false; + return FALSE; } @@ -145,7 +145,7 @@ DT_Performance_Test_Create ( DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (cr) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); test_ptr->creq_evd_hdl = DAT_HANDLE_NULL; - return false; + return FALSE; } } @@ -159,7 +159,7 @@ DT_Performance_Test_Create ( DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_evd_create (conn) error: %s\n", test_ptr->base_port, DT_RetToString (ret)); test_ptr->conn_evd_hdl = DAT_HANDLE_NULL; - return false; + return FALSE; } /* @@ -188,7 +188,7 @@ DT_Performance_Test_Create ( DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ep_create error: %s\n", test_ptr->base_port, DT_RetToString (ret)); test_ptr->ep_context.ep_handle = DAT_HANDLE_NULL; - return false; + return FALSE; } /* @@ -204,13 +204,13 @@ DT_Performance_Test_Create ( DT_PERF_SYNC_BUFF_SIZE, 2, /* 2 RMIs */ 255, /* FIXME should query for this */ - false, - false); + FALSE, + FALSE); if ( !test_ptr->ep_context.bp ) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: no memory for remote memory buffers\n", test_ptr->base_port); - return false; + return FALSE; } DT_Tdep_PT_Debug (3, (phead, @@ -232,7 +232,7 @@ DT_Performance_Test_Create ( DT_PERF_SYNC_BUFF_SIZE) ) { /* error message printed by DT_post_recv_buffer */ - return false; + return FALSE; } /* @@ -257,7 +257,7 @@ DT_Performance_Test_Create ( /* update pipeline_len */ pt_ptr->Params.Client_Stats_P.pipeline_len = pipeline_len; - return true; + return TRUE; } @@ -414,7 +414,7 @@ DT_Performance_Test_Destroy ( /****************************************************************************/ -bool +boolean_t DT_performance_post_rdma_op ( Performance_Ep_Context_t *ep_context, DAT_EVD_HANDLE reqt_evd_hdl, @@ -488,14 +488,14 @@ DT_performance_post_rdma_op ( if ( DAT_SUCCESS != ret ) { - return false; + return FALSE; } DT_performance_stats_record_post (stats, post_ctxt_num - pre_ctxt_num, post_ts - pre_ts); - return true; + return TRUE; } /****************************************************************************/ Index: linux-kernel/test/dapltest/test/dapl_performance_client.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_performance_client.c (revision 2306) +++ linux-kernel/test/dapltest/test/dapl_performance_client.c (working copy) @@ -49,7 +49,7 @@ DT_Performance_Test_Client ( if ( !DT_Performance_Test_Create (pt_ptr, ia_handle, remote_ia_addr, - false, + FALSE, pt_ptr->Server_Info.is_little_endian, &test_ptr) ) { @@ -78,7 +78,7 @@ DT_Performance_Test_Client ( * because there was never a connection. */ - status = DT_Performance_Test_Destroy (pt_ptr, test_ptr, false); + status = DT_Performance_Test_Destroy (pt_ptr, test_ptr, FALSE); DT_Tdep_PT_Debug (1,(phead,"Client: Finished performance test\n")); return status; @@ -86,7 +86,7 @@ DT_Performance_Test_Client ( /****************************************************************************/ -bool +boolean_t DT_Performance_Test_Client_Connect ( DT_Tdep_Print_Head *phead, Performance_Test_t *test_ptr) @@ -114,7 +114,7 @@ retry: { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_ep_connect error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - return false; + return FALSE; } /* wait for DAT_CONNECTION_EVENT_ESTABLISHED */ @@ -135,17 +135,17 @@ retry: } } /* error message printed by DT_cr_event_wait */ - return false; + return FALSE; } DT_Tdep_PT_Debug (1,(phead,"Client[" F64x "]: Got Connection\n", test_ptr->base_port)); - return true; + return TRUE; } /****************************************************************************/ -static bool +static boolean_t DT_Performance_Test_Client_Phase1 ( DT_Tdep_Print_Head *phead, Performance_Test_t *test_ptr, @@ -164,7 +164,7 @@ DT_Performance_Test_Client_Phase1 ( if ( !DT_Mdep_GetCpuStat (&pre_cpu_stat) ) { - return false; + return FALSE; } pre_ts = DT_Mdep_GetTimeStamp (); @@ -182,7 +182,7 @@ DT_Performance_Test_Client_Phase1 ( DT_Tdep_PT_Debug (1,(phead,"Test[" F64x "]: Post %i failed\n", test_ptr->base_port, post_cnt)); - return false; + return FALSE; } } @@ -206,7 +206,7 @@ DT_Performance_Test_Client_Phase1 ( DT_Tdep_PT_Debug (1,(phead,"Test[" F64x "]: Poll %i failed\n", test_ptr->base_port, reap_cnt)); - return false; + return FALSE; } /* repost */ @@ -222,7 +222,7 @@ DT_Performance_Test_Client_Phase1 ( DT_Tdep_PT_Debug (1,(phead,"Test[" F64x "]: Post %i failed\n", test_ptr->base_port, post_cnt)); - return false; + return FALSE; } } @@ -237,16 +237,16 @@ DT_Performance_Test_Client_Phase1 ( if ( !DT_Mdep_GetCpuStat (&post_cpu_stat) ) { - return false; + return FALSE; } stats->post_cpu_stat = post_cpu_stat; stats->pre_cpu_stat = pre_cpu_stat; - return true; + return TRUE; } /****************************************************************************/ -static bool +static boolean_t DT_Performance_Test_Client_Phase2 ( DT_Tdep_Print_Head *phead, Performance_Test_t *test_ptr, @@ -318,7 +318,7 @@ DT_Performance_Test_Client_Phase2 ( if ( DAT_SUCCESS != ret ) { - return false; + return FALSE; } for (;;) @@ -337,7 +337,7 @@ DT_Performance_Test_Client_Phase2 ( { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_dequeue failed: %s\n", DT_RetToString (ret)); - return false; + return FALSE; } else if (event.event_number == DAT_DTO_COMPLETION_EVENT) { @@ -351,17 +351,17 @@ DT_Performance_Test_Client_Phase2 ( "Warning: dapl_performance_wait swallowing %s event\n", DT_EventToSTr (event.event_number)); - return false; + return FALSE; } } } - return true; + return TRUE; } /****************************************************************************/ -bool +boolean_t DT_Performance_Test_Client_Exchange ( Params_t *params_ptr, DT_Tdep_Print_Head *phead, @@ -382,14 +382,14 @@ DT_Performance_Test_Client_Exchange ( test_ptr->ep_context.op.seg_size, test_ptr->ep_context.op.num_segs, 256, /* FIXME should query for this */ - false, - false); + FALSE, + FALSE); if ( !test_ptr->ep_context.op.bp ) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: no memory for buffers (RDMA/RD)\n", test_ptr->base_port); - return false; + return FALSE; } /* @@ -411,7 +411,7 @@ DT_Performance_Test_Client_Exchange ( dto_cookie, "Recieve Sync_Msg") ) { - return false; + return FALSE; } /* @@ -450,12 +450,12 @@ DT_Performance_Test_Client_Exchange ( if ( !DT_Performance_Test_Client_Phase1 (phead, test_ptr, &stats) ) { - return false; + return FALSE; } if ( !DT_Performance_Test_Client_Phase2 (phead, test_ptr, &stats) ) { - return false; + return FALSE; } DT_Tdep_PT_Debug (1,(phead,"Test[" F64x "]: Sending Sync Msg\n", test_ptr->base_port)); @@ -467,7 +467,7 @@ DT_Performance_Test_Client_Exchange ( DT_PERF_SYNC_BUFF_SIZE)) { /* error message printed by DT_post_send_buffer */ - return false; + return FALSE; } dto_cookie.as_64 = LZERO; @@ -483,9 +483,9 @@ DT_Performance_Test_Client_Exchange ( dto_cookie, "Client_Sync_Send")) { - return false; + return FALSE; } stats.pipeline_len = params_ptr->Client_Stats_P.pipeline_len; params_ptr->Client_Stats_P = stats; - return true; + return TRUE; } Index: linux-kernel/test/dapltest/test/dapl_transaction_stats.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_transaction_stats.c (revision 2306) +++ linux-kernel/test/dapltest/test/dapl_transaction_stats.c (working copy) @@ -53,7 +53,7 @@ DT_transaction_stats_set_ready (DT_Tdep_ DT_Mdep_Unlock (&transaction_stats->lock); } -bool +boolean_t DT_transaction_stats_wait_for_all (DT_Tdep_Print_Head *phead, Transaction_Stats_t * transaction_stats) { @@ -68,9 +68,9 @@ DT_transaction_stats_wait_for_all (DT_Td { DT_Tdep_PT_Printf (phead, "FAIL: %d Server test connections did not report ready.\n", transaction_stats->wait_count); - return false; + return FALSE; } - return true; + return TRUE; } Index: linux-kernel/test/dapltest/test/dapl_server.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_server.c (revision 2306) +++ linux-kernel/test/dapltest/test/dapl_server.c (working copy) @@ -240,8 +240,8 @@ DT_cs_Server (Params_t * params_ptr) DT_RoundSize (sizeof (Transaction_Cmd_t), 8192), 3, /* num_buffers */ 256, /* FIXME query for this value */ - false, - false); + FALSE, + FALSE); if (ps_ptr->bpool == 0) { DT_Tdep_PT_Printf (phead, @@ -290,7 +290,7 @@ DT_cs_Server (Params_t * params_ptr) } DT_MemListInit (pt_ptr); DT_Thread_Init (pt_ptr); - pt_ptr->local_is_server = true; + pt_ptr->local_is_server = TRUE; pt_ptr->ps_ptr = ps_ptr; memcpy ( (void *) (uintptr_t) &pt_ptr->Params, (const void *) params_ptr, @@ -436,7 +436,7 @@ DT_cs_Server (Params_t * params_ptr) { Transaction_Cmd = (Transaction_Cmd_t *) DT_Bpool_GetBuffer (ps_ptr->bpool, 1); - DT_Transaction_Cmd_Endian (Transaction_Cmd, false); + DT_Transaction_Cmd_Endian (Transaction_Cmd, FALSE); memcpy ( (void *) (uintptr_t)&pt_ptr->Params.u.Transaction_Cmd, (const void *)Transaction_Cmd, sizeof (Transaction_Cmd_t)); @@ -446,7 +446,7 @@ DT_cs_Server (Params_t * params_ptr) case QUIT_TEST: { Quit_Cmd = (Quit_Cmd_t*)DT_Bpool_GetBuffer (ps_ptr->bpool, 1); - DT_Quit_Cmd_Endian (Quit_Cmd, false); + DT_Quit_Cmd_Endian (Quit_Cmd, FALSE); memcpy ( (void *) (uintptr_t)&pt_ptr->Params.u.Quit_Cmd, (const void *)Quit_Cmd, sizeof (Quit_Cmd_t)); @@ -555,7 +555,7 @@ DT_cs_Server (Params_t * params_ptr) /* Start the new test thread */ DT_Tdep_PT_Debug (1,(phead,"%s: Starting Test Thread\n", module)); - if (DT_Thread_Start (pt_ptr->thread) == false) + if (DT_Thread_Start (pt_ptr->thread) == FALSE) { DT_Tdep_PT_Debug (1,(phead,"failed to start test thread\n")); status = 1; Index: linux-kernel/test/dapltest/test/dapl_thread.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_thread.c (revision 2306) +++ linux-kernel/test/dapltest/test/dapl_thread.c (working copy) @@ -124,7 +124,7 @@ DT_Thread_Destroy (Thread * thread_ptr, * to starting it whereas DT_Thread_Create just sets up some data structures. * (this is counter-intuitive) */ -bool +boolean_t DT_Thread_Start (Thread * thread_ptr) { return DT_Mdep_Thread_Start (thread_ptr); Index: linux-kernel/test/dapltest/test/dapl_fft_util.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_util.c (revision 2306) +++ linux-kernel/test/dapltest/test/dapl_fft_util.c (working copy) @@ -137,7 +137,7 @@ void DT_fft_init_conn_struct (FFT_Connec conn->remote_netaddr = NULL; conn->bpool = NULL; conn->pt_ptr = NULL; - conn->connected = false; + conn->connected = FALSE; } /* helper function that simplifies many dat calls for the initiialization of a @@ -298,7 +298,7 @@ void DT_fft_init_server (Params_t *param /* allocate memory for buffers */ conn->bpool = DT_BpoolAlloc (NULL, phead, conn->ia_handle, conn->pz_handle, NULL, NULL, 8192, 2, 256 /* FIXME query */, - false, false); + FALSE, FALSE); DT_assert (phead, conn->bpool); cleanup: return; @@ -330,8 +330,8 @@ void DT_fft_listen (Params_t *params_ptr DT_conn_event_wait (phead, conn->ep_handle, conn->conn_evd, - &conn->event_num) == true); - conn->connected = true; + &conn->event_num) == TRUE); + conn->connected = TRUE; cleanup: return; } @@ -361,7 +361,7 @@ int DT_fft_connect (Params_t *params_ptr DT_conn_event_wait (phead, conn->ep_handle, conn->conn_evd, - &conn->event_num) == true); + &conn->event_num) == TRUE); /* make sure we weren't rejected by the peer */ if (conn->event_num == DAT_CONNECTION_EVENT_PEER_REJECTED) @@ -373,7 +373,7 @@ int DT_fft_connect (Params_t *params_ptr cleanup: if (conn->event_num == DAT_CONNECTION_EVENT_ESTABLISHED) { - conn->connected = true; + conn->connected = TRUE; } /* returns true if connected, false otherwise */ return (conn->connected); Index: linux-kernel/test/dapltest/test/dapl_limit.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_limit.c (revision 2306) +++ linux-kernel/test/dapltest/test/dapl_limit.c (working copy) @@ -31,7 +31,7 @@ * Increase the size of an array of handles */ -static bool +static boolean_t more_handles (DT_Tdep_Print_Head *phead, DAT_HANDLE **old_ptrptr, /* pointer to current pointer */ unsigned int *old_count, /* number pointed to */ @@ -44,14 +44,14 @@ more_handles (DT_Tdep_Print_Head *phead, if (!handle_tmp) { DT_Tdep_PT_Printf (phead, "Out of memory for more DAT_HANDLEs\n"); - return (false); + return (FALSE); } memcpy (handle_tmp, old_handles, count*size); DT_Mdep_Free (old_handles); *old_ptrptr = handle_tmp; *old_count = count * 2; - return (true); + return (TRUE); } @@ -69,7 +69,7 @@ more_handles (DT_Tdep_Print_Head *phead, * the constructor for the object indicated by 'depth', providing for * increased (or at least different) stress on the DAPL. */ -static bool +static boolean_t limit_test ( DT_Tdep_Print_Head *phead, Limit_Cmd_t *cmd, Limit_Index depth) @@ -91,7 +91,7 @@ limit_test ( DT_Tdep_Print_Head *phead, } Obj_Set; Obj_Set *hdl_sets = (Obj_Set *) NULL; - bool retval = false; + boolean_t retval = FALSE; char *module = "LimitTest"; #if defined (WIN32) @@ -183,7 +183,7 @@ limit_test ( DT_Tdep_Print_Head *phead, sizeof (*hdlptr))) { DT_Tdep_PT_Printf (phead, "%s: IAs opened: %d\n", module, w); - retval = true; + retval = TRUE; break; } /* Specify that we want to get back an async EVD. */ @@ -197,13 +197,13 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: dat_ia_open (%s) #%d fails: %s\n", module, cmd->device_name, w+1, DT_RetToString (ret)); - retval = true; + retval = TRUE; break; } } DT_Tdep_PT_Printf (phead, "%s: IAs opened: %d\n", module, w); - retval = true; + retval = TRUE; /* IA Cleanup loop */ for (tmp = 0; tmp < w; tmp++) @@ -215,7 +215,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Tdep_PT_Printf (phead, "%s: dat_ia_close (graceful) fails: %s\n", module, DT_RetToString (ret)); - retval = false; + retval = FALSE; ret = dat_ia_close (hdlptr[tmp].ia_handle, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) @@ -284,7 +284,7 @@ limit_test ( DT_Tdep_Print_Head *phead, sizeof (*hdlptr))) { DT_Tdep_PT_Printf (phead, "%s: PZs created: %d\n", module, w); - retval = true; + retval = TRUE; break; } ret = dat_pz_create (hdl_sets[w % cmd->width].ia_handle, @@ -293,13 +293,13 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Tdep_PT_Printf (phead, "%s: dat_pz_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); - retval = true; + retval = TRUE; break; } } DT_Tdep_PT_Printf (phead, "%s: PZs created: %d\n", module, w); - retval = true; + retval = TRUE; /* PZ Cleanup loop */ for (tmp = 0; tmp < w; tmp++) @@ -310,7 +310,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Tdep_PT_Printf (phead, "%s: dat_pz_free fails: %s\n", module, DT_RetToString (ret)); - retval = false; + retval = FALSE; } } DT_Mdep_Free (hdlptr); @@ -382,7 +382,7 @@ limit_test ( DT_Tdep_Print_Head *phead, sizeof (*hdlptr))) { DT_Tdep_PT_Printf (phead, "%s: CNOs created: %d\n", module, w); - retval = true; + retval = TRUE; break; } ret = dat_cno_create (hdl_sets[w % cmd->width].ia_handle, @@ -392,20 +392,20 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Tdep_PT_Printf (phead, "%s: dat_cno_create unimplemented\n", module); - retval = true; + retval = TRUE; break; } else if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "%s: dat_cno_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); - retval = true; + retval = TRUE; break; } } DT_Tdep_PT_Printf (phead, "%s: CNOs created: %d\n", module, w); - retval = true; + retval = TRUE; /* CNO Cleanup loop */ for (tmp = 0; tmp < w; tmp++) @@ -416,7 +416,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Tdep_PT_Printf (phead, "%s: dat_cno_free fails: %s\n", module, DT_RetToString (ret)); - retval = false; + retval = FALSE; } } DT_Mdep_Free (hdlptr); @@ -521,7 +521,7 @@ limit_test ( DT_Tdep_Print_Head *phead, sizeof (*hdlptr))) { DT_Tdep_PT_Printf (phead, "%s: EVDs created: %d\n", module, w); - retval = true; + retval = TRUE; break; } ret = DT_Tdep_evd_create (hdl_sets[w % cmd->width].ia_handle, @@ -533,13 +533,13 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Tdep_PT_Printf (phead, "%s: dat_evd_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); - retval = true; + retval = TRUE; break; } } DT_Tdep_PT_Printf (phead, "%s: EVDs created: %d\n", module, w); - retval = true; + retval = TRUE; /* EVD Cleanup loop */ if (conn_handle != DAT_HANDLE_NULL) @@ -555,7 +555,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Tdep_PT_Printf (phead, "%s: dat_evd_free fails: %s\n", module, DT_RetToString (ret)); - retval = false; + retval = FALSE; } } DT_Mdep_Free (hdlptr); @@ -620,7 +620,7 @@ limit_test ( DT_Tdep_Print_Head *phead, sizeof (*hdlptr))) { DT_Tdep_PT_Printf (phead, "%s: EPs created: %d\n", module, w); - retval = true; + retval = TRUE; break; } ret = dat_ep_create (hdl_sets[w % cmd->width].ia_handle, @@ -634,13 +634,13 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Tdep_PT_Printf (phead, "%s: dat_ep_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); - retval = true; + retval = TRUE; break; } } DT_Tdep_PT_Printf (phead, "%s: EPs created: %d\n", module, w); - retval = true; + retval = TRUE; /* EP Cleanup loop */ for (tmp = 0; tmp < w; tmp++) @@ -651,7 +651,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Tdep_PT_Printf (phead, "%s: dat_ep_free fails: %s\n", module, DT_RetToString (ret)); - retval = false; + retval = FALSE; } } DT_Mdep_Free (hdlptr); @@ -697,7 +697,7 @@ limit_test ( DT_Tdep_Print_Head *phead, sizeof (*hdlptr))) { DT_Tdep_PT_Printf (phead, "%s: RSPs created: %d\n", module, w); - retval = true; + retval = TRUE; break; } if (!more_handles (phead, (DAT_HANDLE **) &epptr, @@ -705,7 +705,7 @@ limit_test ( DT_Tdep_Print_Head *phead, sizeof (*epptr))) { DT_Tdep_PT_Printf (phead, "%s: RSPs created: %d\n", module, w); - retval = true; + retval = TRUE; break; } @@ -713,7 +713,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Tdep_PT_Printf (phead, "%s: Mismatch in allocation of handle arrays at point %d\n", module, w); - retval = true; + retval = TRUE; break; } @@ -734,7 +734,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Tdep_PT_Printf (phead, "%s: dat_ep_create #%d fails: %s testing RSPs\n", module, w+1, DT_RetToString (ret)); - retval = true; + retval = TRUE; break; } @@ -748,7 +748,7 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: dat_rsp_create unimplemented\n", module); /* ignore this error */ - retval = true; + retval = TRUE; break; } else if (ret != DAT_SUCCESS) @@ -762,13 +762,13 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: dat_ep_free (internal cleanup @ #%d) fails: %s\n", module, w+1, DT_RetToString (ret)); } - retval = true; + retval = TRUE; break; } } DT_Tdep_PT_Printf (phead, "%s: RSPs created: %d\n", module, w); - retval = true; + retval = TRUE; /* RSP Cleanup loop */ for (tmp = 0; tmp < w; tmp++) @@ -779,7 +779,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Tdep_PT_Printf (phead, "%s: dat_rsp_free fails: %s\n", module, DT_RetToString (ret)); - retval = false; + retval = FALSE; } /* Free EPs */ ret = dat_ep_free (epptr[tmp]); @@ -787,7 +787,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Tdep_PT_Printf (phead, "%s: dat_ep_free fails: %s for RSPs\n", module, DT_RetToString (ret)); - retval = false; + retval = FALSE; } } DT_Mdep_Free (hdlptr); @@ -827,7 +827,7 @@ limit_test ( DT_Tdep_Print_Head *phead, sizeof (*hdlptr))) { DT_Tdep_PT_Printf (phead, "%s: PSPs created: %d\n", module, w); - retval = true; + retval = TRUE; break; } ret = dat_psp_create (hdl_sets[w % cmd->width].ia_handle, @@ -839,14 +839,14 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Tdep_PT_Printf (phead, "%s: dat_psp_create #%d fails: %s\n", module, w+1, DT_RetToString (ret)); - retval = true; + retval = TRUE; hdlptr[w] = DAT_HANDLE_NULL; break; } } DT_Tdep_PT_Printf (phead, "%s: PSPs created: %d\n", module, w); - retval = true; + retval = TRUE; /* PSP Cleanup loop */ for (tmp = 0; tmp < w; tmp++) @@ -864,7 +864,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Tdep_PT_Printf (phead, "%s: dat_psp_free fails: %s\n", module, DT_RetToString (ret)); - retval = false; + retval = FALSE; } } DT_Mdep_Free (hdlptr); @@ -956,7 +956,7 @@ limit_test ( DT_Tdep_Print_Head *phead, DT_Tdep_PT_Printf (phead, "%s: no memory for LMR handles\n", module); DT_Tdep_PT_Printf (phead, "%s: LMRs created: %d\n", module, w); - retval = true; + retval = TRUE; break; } /* @@ -973,18 +973,18 @@ limit_test ( DT_Tdep_Print_Head *phead, DFLT_BUFFSZ, 1, 256, /* FIXME should query for this */ - false, - false); + FALSE, + FALSE); if (!hdlptr[w]) { DT_Tdep_PT_Printf (phead, "%s: LMRs created: %d\n", module, w); - retval = true; + retval = TRUE; break; } } DT_Tdep_PT_Printf (phead, "%s: LMRs created: %d\n", module, w); - retval = true; + retval = TRUE; /* LMR Cleanup loop */ for (tmp = 0; tmp <= w; tmp++) @@ -1037,7 +1037,7 @@ limit_test ( DT_Tdep_Print_Head *phead, module, w, w * cmd->width); - done = retval = true; + done = retval = TRUE; break; } for (i = 0; i < cmd->width; i++) @@ -1068,13 +1068,13 @@ limit_test ( DT_Tdep_Print_Head *phead, module, w, w * cmd->width + i); - done = retval = true; + done = retval = TRUE; break; } } /* end for each EP wide */ } /* end forever (!done) loop */ - retval = true; + retval = TRUE; DT_Tdep_PT_Printf (phead, "%s: recv buffers posted per EP: %d\n" "\t\t (total posted: %d)\n", module, @@ -1096,7 +1096,7 @@ limit_test ( DT_Tdep_Print_Head *phead, { DT_Tdep_PT_Printf (phead, "%s: dat_ep_disconnect (abrupt) fails: %s\n", module, DT_RetToString (ret)); - retval = false; + retval = FALSE; } else { @@ -1137,22 +1137,22 @@ limit_test ( DT_Tdep_Print_Head *phead, test_size, 1, 256, /* FIXME should query for this */ - false, - false); + FALSE, + FALSE); if (!test_bpool) { DT_Tdep_PT_Printf (phead, "%s: Largest LMR was 0x%x bytes\n" "\t (failed attempting 0x%x bytes)\n", module, last_size, test_size); - retval = true; + retval = TRUE; break; } else if (!DT_Bpool_Destroy ((Per_Test_Data_t *)0, phead, test_bpool)) { DT_Tdep_PT_Printf (phead, "%s: Largest LMR was 0x%x bytes\n", module, test_size); - retval = true; + retval = TRUE; break; } @@ -1163,7 +1163,7 @@ limit_test ( DT_Tdep_Print_Head *phead, /* could conceivably wrap on 32-bit architectures */ DT_Tdep_PT_Printf (phead, "%s: LMR of 0x%x bytes OK - %s\n", module, last_size, "stopping now."); - retval = true; + retval = TRUE; break; } } /* end forever loop */ @@ -1192,7 +1192,7 @@ clean_up_now: { DT_Tdep_PT_Printf (phead, "%s: dat_lmr_free fails: %s\n", module, DT_RetToString (ret)); - retval = false; + retval = FALSE; } } if ((void *) hdl_sets[w].lmr_buffer) @@ -1227,7 +1227,7 @@ clean_up_now: { DT_Tdep_PT_Printf (phead, "%s: dat_ep_free fails: %s\n", module, DT_RetToString (ret)); - retval = false; + retval = FALSE; } } } @@ -1251,7 +1251,7 @@ clean_up_now: { DT_Tdep_PT_Printf (phead, "%s: dat_evd_free fails: %s\n", module, DT_RetToString (ret)); - retval = false; + retval = FALSE; } } } @@ -1271,7 +1271,7 @@ clean_up_now: { DT_Tdep_PT_Printf (phead, "%s: dat_cno_free fails: %s\n", module, DT_RetToString (ret)); - retval = false; + retval = FALSE; } } } @@ -1291,7 +1291,7 @@ clean_up_now: { DT_Tdep_PT_Printf (phead, "%s: dat_pz_free fails: %s\n", module, DT_RetToString (ret)); - retval = false; + retval = FALSE; } } } @@ -1317,7 +1317,7 @@ clean_up_now: * this really is an error. But if we get here, * we may as well try the largest hammer we have. */ - retval = false; + retval = FALSE; ret = dat_ia_close (hdl_sets[w].ia_handle, DAT_CLOSE_ABRUPT_FLAG); if (ret != DAT_SUCCESS) Index: linux-kernel/test/dapltest/test/dapl_transaction_test.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_transaction_test.c (revision 2306) +++ linux-kernel/test/dapltest/test/dapl_transaction_test.c (working copy) @@ -69,7 +69,7 @@ DT_Transaction_Test_Client (Per_Test_Dat DT_Tdep_PT_Debug (1,(phead,"Client: Starting Client side of test\n")); if (!DT_Transaction_Create_Test (pt_ptr, ia_handle, - false, + FALSE, port_num, pt_ptr->Server_Info.is_little_endian, remote_ia_addr)) @@ -111,7 +111,7 @@ DT_Transaction_Test_Server (void *params if (!DT_Transaction_Create_Test (pt_ptr, pt_ptr->ps_ptr->ia_handle, - true, + TRUE, port_num, pt_ptr->Client_Info.is_little_endian, (DAT_IA_ADDRESS_PTR) 0)) @@ -156,7 +156,7 @@ DT_Transaction_Test_Server (void *params * work. */ -bool +boolean_t DT_Transaction_Create_Test (Per_Test_Data_t * pt_ptr, DAT_IA_HANDLE *ia_handle, boolean_t is_server, @@ -176,7 +176,7 @@ DT_Transaction_Create_Test (Per_Test_Dat if (!test_ptr) { DT_Tdep_PT_Printf (phead, "No Memory to create transaction test structure!\n"); - return false; + return FALSE; } /* Unused fields zeroed by allocator */ @@ -207,11 +207,11 @@ DT_Transaction_Create_Test (Per_Test_Dat { DT_Tdep_PT_Printf (phead, "No memory!\n"); DT_MemListFree (test_ptr->pt_ptr, test_ptr); - return false; + return FALSE; } DT_Thread_Start (test_ptr->thread); - return true; + return TRUE; } @@ -231,7 +231,7 @@ DT_Transaction_Main (void *param) Transaction_Test_t *test_ptr = (Transaction_Test_t *) param; DAT_RETURN ret; u32 i, j; - bool success = false; + boolean_t success = FALSE; Per_Test_Data_t *pt_ptr; Thread *thread; DAT_DTO_COOKIE dto_cookie; @@ -394,8 +394,8 @@ DT_Transaction_Main (void *param) buff_size, 4, pt_ptr->provider_attr.optimal_buffer_alignment, - false, - false); + FALSE, + FALSE); if (!test_ptr->ep_context[i].bp) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: no memory for remote memory buffers\n", @@ -844,8 +844,8 @@ retry: test_ptr->ep_context[i].op[j].seg_size, test_ptr->ep_context[i].op[j].num_segs, pt_ptr->provider_attr.optimal_buffer_alignment, - false, - !us ? true : false); + FALSE, + !us ? TRUE : FALSE); if (!test_ptr->ep_context[i].op[j].bp) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: no memory for buffers (RDMA/RD)\n", @@ -880,8 +880,8 @@ retry: test_ptr->ep_context[i].op[j].seg_size, test_ptr->ep_context[i].op[j].num_segs, pt_ptr->provider_attr.optimal_buffer_alignment, - !us ? true : false, - false); + !us ? TRUE : FALSE, + FALSE); if (!test_ptr->ep_context[i].op[j].bp) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: no memory for buffers (RDMA/WR)\n", @@ -915,8 +915,8 @@ retry: test_ptr->ep_context[i].op[j].seg_size, test_ptr->ep_context[i].op[j].num_segs, pt_ptr->provider_attr.optimal_buffer_alignment, - false, - false); + FALSE, + FALSE); if (!test_ptr->ep_context[i].op[j].bp) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: no memory for buffers (S/R)\n", @@ -1286,15 +1286,15 @@ test_failure: /* ----------------------------------------------------------------------- * The actual performance test */ -bool +boolean_t DT_Transaction_Run (DT_Tdep_Print_Head *phead, Transaction_Test_t * test_ptr) { unsigned int op; unsigned int iteration; int bytes; - bool ours; - bool success = false; - bool repost_recv; + boolean_t ours; + boolean_t success = FALSE; + boolean_t repost_recv; unsigned int i; /* pre-post all receive buffers */ @@ -1624,7 +1624,7 @@ DT_Transaction_Run (DT_Tdep_Print_Head * test_ptr->stats.stat_bytes_rdma_write); } DT_Tdep_PT_Debug (1,(phead,"Test[" F64x "]: End Successfully\n", test_ptr->base_port)); - success = true; + success = TRUE; bail: return ( success ); @@ -1637,7 +1637,7 @@ DT_Transaction_Validation_Fill ( DT_Tdep Transaction_Test_t * test_ptr, unsigned int iteration) { - bool ours; + boolean_t ours; unsigned int op; unsigned int i; unsigned int j; @@ -1746,13 +1746,13 @@ DT_Transaction_Validation_Fill ( DT_Tdep /*------------------------------------------------------------------------------ */ -bool +boolean_t DT_Transaction_Validation_Check (DT_Tdep_Print_Head *phead, Transaction_Test_t * test_ptr, int iteration) { - bool ours; - bool success = true; + boolean_t ours; + boolean_t success = TRUE; unsigned int op; unsigned int i; unsigned int j; @@ -1811,7 +1811,7 @@ DT_Transaction_Validation_Check (DT_Tdep (phead, "\twq=%d op=%d seg=%d byte=%d ptr=%p\n", ind, op, i, j, buff)); - success = false; + success = FALSE; break; } } @@ -1851,7 +1851,7 @@ DT_Transaction_Validation_Check (DT_Tdep (phead, "\twq=%d op=%d seg=%d byte=%d ptr=%p\n", ind, op, i, j, buff)); - success = false; + success = FALSE; break; } } @@ -1898,7 +1898,7 @@ DT_Transaction_Validation_Check (DT_Tdep (phead, "\twq=%d op=%d seg=%d byte=%d ptr=%p\n", ind, op, i, j, buff)); - success = false; + success = FALSE; break; } } Index: linux-kernel/test/dapltest/test/dapl_performance_server.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_performance_server.c (revision 2306) +++ linux-kernel/test/dapltest/test/dapl_performance_server.c (working copy) @@ -43,7 +43,7 @@ DT_Performance_Test_Server ( if ( !DT_Performance_Test_Create (pt_ptr, pt_ptr->ps_ptr->ia_handle, (DAT_IA_ADDRESS_PTR) 0, - true, + TRUE, pt_ptr->Client_Info.is_little_endian, &test_ptr) ) { @@ -68,7 +68,7 @@ DT_Performance_Test_Server ( } } - status = DT_Performance_Test_Destroy (pt_ptr, test_ptr, true); + status = DT_Performance_Test_Destroy (pt_ptr, test_ptr, TRUE); DT_Tdep_PT_Printf (phead, "Server: Finished performance test. Detaching.\n"); @@ -94,13 +94,13 @@ DT_Performance_Test_Server ( /****************************************************************************/ -bool +boolean_t DT_Performance_Test_Server_Connect ( DT_Tdep_Print_Head *phead, Performance_Test_t *test_ptr) { DAT_RETURN ret; - bool status; + boolean_t status; DAT_RSP_HANDLE rsp_handle; DAT_PSP_HANDLE psp_handle; @@ -126,7 +126,7 @@ DT_Performance_Test_Server_Connect ( { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_rsp_create error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - status = false; + status = FALSE; goto psp_free; } @@ -141,7 +141,7 @@ DT_Performance_Test_Server_Connect ( &cr_handle, "Server") ) { - status = false; + status = FALSE; goto psp_free; } @@ -154,7 +154,7 @@ DT_Performance_Test_Server_Connect ( DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_accept error: %s\n", test_ptr->base_port, DT_RetToString (ret)); /* cr_handle consumed on failure */ - status = false; + status = FALSE; goto psp_free; } @@ -164,7 +164,7 @@ DT_Performance_Test_Server_Connect ( &event_num)) { /* error message printed by DT_conn_event_wait */ - status = false; + status = FALSE; goto psp_free; } @@ -175,7 +175,7 @@ DT_Performance_Test_Server_Connect ( /* * Server - use a short-lived PSP instead of an RSP */ - status = true; + status = TRUE; ret = dat_psp_create (test_ptr->ia_handle, test_ptr->ep_context.port, @@ -186,7 +186,7 @@ DT_Performance_Test_Server_Connect ( { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_psp_create error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - status = false; + status = FALSE; psp_handle = DAT_HANDLE_NULL; return (status); } @@ -213,7 +213,7 @@ DT_Performance_Test_Server_Connect ( &cr_handle, "Server") ) { - status = false; + status = FALSE; goto psp_free; } @@ -227,7 +227,7 @@ DT_Performance_Test_Server_Connect ( DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_accept error: %s\n", test_ptr->base_port, DT_RetToString (ret)); /* cr_handle consumed on failure */ - status = false; + status = FALSE; goto psp_free; } @@ -238,7 +238,7 @@ DT_Performance_Test_Server_Connect ( &event_num ) ) { /* error message printed by DT_cr_event_wait */ - status = false; + status = FALSE; goto psp_free; } @@ -253,7 +253,7 @@ psp_free: { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_psp_free error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - status = false; + status = FALSE; } } if ( DAT_HANDLE_NULL != rsp_handle ) @@ -264,7 +264,7 @@ psp_free: { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_rsp_free error: %s\n", test_ptr->base_port, DT_RetToString (ret)); - status = false; + status = FALSE; } } /* end short-lived PSP */ @@ -274,7 +274,7 @@ psp_free: /****************************************************************************/ -bool +boolean_t DT_Performance_Test_Server_Exchange ( DT_Tdep_Print_Head *phead, Performance_Test_t *test_ptr) @@ -293,14 +293,14 @@ DT_Performance_Test_Server_Exchange ( test_ptr->ep_context.op.seg_size, test_ptr->ep_context.op.num_segs, 256, /* FIXME should query for this */ - true, - true); + TRUE, + TRUE); if ( !test_ptr->ep_context.op.bp ) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: no memory for buffers (RDMA/RD)\n", test_ptr->base_port); - return false; + return FALSE; } test_ptr->ep_context.op.Rdma_Context = @@ -350,7 +350,7 @@ DT_Performance_Test_Server_Exchange ( DT_PERF_SYNC_BUFF_SIZE)) { /* error message printed by DT_post_send_buffer */ - return false; + return FALSE; } /* reap the send and verify it */ @@ -367,7 +367,7 @@ DT_Performance_Test_Server_Exchange ( dto_cookie, "Send Sync_Msg") ) { - return false; + return FALSE; } /* @@ -389,12 +389,12 @@ DT_Performance_Test_Server_Exchange ( dto_cookie, "Recieve Sync_Msg") ) { - return false; + return FALSE; } DT_Tdep_PT_Debug (1,(phead,"Test[" F64x "]: Received Sync Msg\n", test_ptr->base_port)); - return true; + return TRUE; } Index: linux-kernel/test/dapltest/test/dapl_fft_dataxfer_client.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_dataxfer_client.c (revision 2306) +++ linux-kernel/test/dapltest/test/dapl_fft_dataxfer_client.c (working copy) @@ -65,8 +65,8 @@ int DT_dataxfer_client_generic (DT_Tdep_ 4096, 2, 256, /* FIXME should query for this */ - false, - false); + FALSE, + FALSE); DT_assert (phead, conn.bpool != 0); rc = DT_post_send_buffer (phead, conn.ep_handle, conn.bpool, 0, DT_Bpool_GetBuffSize (conn.bpool, 0)); Index: linux-kernel/test/dapltest/test/dapl_bpool.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_bpool.c (revision 2306) +++ linux-kernel/test/dapltest/test/dapl_bpool.c (working copy) @@ -312,13 +312,13 @@ err: } /*****************************************************************************/ -bool +boolean_t DT_Bpool_Destroy (Per_Test_Data_t * pt_ptr, DT_Tdep_Print_Head *phead, Bpool * bpool_ptr) { unsigned char *module = "DT_Bpool_Destroy"; - bool rval = true; + boolean_t rval = TRUE; if (bpool_ptr) { @@ -353,7 +353,7 @@ DT_Bpool_Destroy (Per_Test_Data_t * pt_p "%s: dat_rmr_free failed %s\n", module, DT_RetToString (ret)); - rval = false; + rval = FALSE; } } @@ -366,7 +366,7 @@ DT_Bpool_Destroy (Per_Test_Data_t * pt_p "%s: dat_lmr_free failed %s\n", module, DT_RetToString (ret)); - rval = false; + rval = FALSE; } } DT_MemListFree (pt_ptr, bpool_ptr->alloc_ptr); Index: linux-kernel/test/dapltest/test/dapl_test_util.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_test_util.c (revision 2306) +++ linux-kernel/test/dapltest/test/dapl_test_util.c (working copy) @@ -53,7 +53,7 @@ DT_query ( Per_Test_Data_t *pt_ptr, DT_Tdep_PT_Printf (phead, "%s: dat_ia_query error: %s\n", module, DT_RetToString (ret)); - return ( false ); + return ( FALSE ); } /* Query the EP */ @@ -63,7 +63,7 @@ DT_query ( Per_Test_Data_t *pt_ptr, DT_Tdep_PT_Printf (phead, "%s: dat_ep_query error: %s\n", module, DT_RetToString (ret)); - return ( false ); + return ( FALSE ); } pt_ptr->ep_attr = ep_params.ep_attr; @@ -151,7 +151,7 @@ DT_query ( Per_Test_Data_t *pt_ptr, DT_Tdep_PT_Printf (phead, "***** ***** ***** ***** ***** *****\n"); } - return ( true ); + return ( TRUE ); } @@ -194,9 +194,9 @@ DT_post_recv_buffer (DT_Tdep_Print_Head DT_Tdep_PT_Printf (phead, "Test Error: dat_ep_post_recv failed: %s\n", DT_RetToString (ret)); DT_Test_Error (); - return false; + return FALSE; } - return true; + return TRUE; } @@ -238,16 +238,16 @@ DT_post_send_buffer (DT_Tdep_Print_Head DT_Tdep_PT_Printf (phead, "Test Error: dat_ep_post_send failed: %s\n", DT_RetToString (ret)); DT_Test_Error (); - return false; + return FALSE; } - return true; + return TRUE; } /* ----------------------------------------------------------- * Wait for a CR event, returning false on error. */ -bool +boolean_t DT_cr_event_wait ( DT_Tdep_Print_Head *phead, DAT_EVD_HANDLE evd_handle, struct dat_cr_arrival_event_data *cr_stat_p) @@ -290,21 +290,21 @@ DT_cr_event_wait ( DT_Tdep_Print_Head *p { *cr_stat_p = event.event_data.cr_arrival_event_data; } - return (true); + return (TRUE); } DT_Tdep_PT_Printf (phead, "Warning: cr_event_wait swallowing %s event\n", DT_EventToSTr (event.event_number)); } - return (false); + return (FALSE); } /* ----------------------------------------------------------- * Wait for a connection event, returning false on error. */ -bool +boolean_t DT_conn_event_wait (DT_Tdep_Print_Head *phead, DAT_EP_HANDLE ep_handle, DAT_EVD_HANDLE evd_handle, @@ -345,21 +345,21 @@ DT_conn_event_wait (DT_Tdep_Print_Head * * event.event_data.connect_event_data.private_data_size * event.event_data.connect_event_data.private_data */ - return (true); + return (TRUE); } DT_Tdep_PT_Printf (phead, "Warning: conn_event_wait swallowing %s event\n", DT_EventToSTr (event.event_number)); } - return (false); + return (FALSE); } /* ----------------------------------------------------------- * Wait for a disconnection event, returning false on error. */ -bool +boolean_t DT_disco_event_wait ( DT_Tdep_Print_Head *phead, DAT_EVD_HANDLE evd_handle, DAT_EP_HANDLE *ep_handle ) @@ -396,24 +396,24 @@ DT_disco_event_wait ( DT_Tdep_Print_Head { *ep_handle = event.event_data.connect_event_data.ep_handle; } - return (true); + return (TRUE); } DT_Tdep_PT_Printf (phead, "Warning: conn_event_wait swallowing %s event\n", DT_EventToSTr (event.event_number)); } - return (false); + return (FALSE); } /* ----------------------------------------------------------- * Reap a DTO event using a wait or polling, returning false on error. */ -bool +boolean_t DT_dto_event_reap (DT_Tdep_Print_Head *phead, DAT_EVD_HANDLE evd_handle, - bool poll, + boolean_t poll, struct dat_dto_completion_event_data *dto_statusp) { if (poll) @@ -430,7 +430,7 @@ DT_dto_event_reap (DT_Tdep_Print_Head *p /* ----------------------------------------------------------- * Poll for a DTO event, returning false on error. */ -bool +boolean_t DT_dto_event_poll (DT_Tdep_Print_Head *phead, DAT_EVD_HANDLE evd_handle, struct dat_dto_completion_event_data *dto_statusp) @@ -468,21 +468,21 @@ DT_dto_event_poll (DT_Tdep_Print_Head *p *dto_statusp = event.event_data.dto_completion_event_data; } - return (true); + return (TRUE); } DT_Tdep_PT_Printf (phead, "Warning: dto_event_poll swallowing %s event\n", DT_EventToSTr (event.event_number)); } - return (false); + return (FALSE); } /* ----------------------------------------------------------- * Wait for a DTO event, returning false on error. */ -bool +boolean_t DT_dto_event_wait (DT_Tdep_Print_Head *phead, DAT_EVD_HANDLE evd_handle, struct dat_dto_completion_event_data *dto_statusp) @@ -512,21 +512,21 @@ DT_dto_event_wait (DT_Tdep_Print_Head *p { *dto_statusp = event.event_data.dto_completion_event_data; } - return (true); + return (TRUE); } DT_Tdep_PT_Printf (phead, "Warning: dto_event_wait swallowing %s event\n", DT_EventToSTr (event.event_number)); } - return (false); + return (FALSE); } /* ----------------------------------------------------------- * Wait for a RMR event, returning false on error. */ -bool +boolean_t DT_rmr_event_wait (DT_Tdep_Print_Head *phead, DAT_EVD_HANDLE evd_handle, struct dat_rmr_bind_completion_event_data *rmr_statusp) @@ -555,21 +555,21 @@ DT_rmr_event_wait (DT_Tdep_Print_Head *p { *rmr_statusp = event.event_data.rmr_completion_event_data; } - return (true); + return (TRUE); } DT_Tdep_PT_Printf (phead, "Warning: rmr_event_wait swallowing %s event\n", DT_EventToSTr (event.event_number)); } - return (false); + return (FALSE); } /* ----------------------------------------------------------- * Check a DTO and print some debug info if anything is amiss. */ -bool +boolean_t DT_dto_check ( DT_Tdep_Print_Head *phead, struct dat_dto_completion_event_data *dto_p, DAT_EP_HANDLE ep_expected, @@ -607,17 +607,17 @@ DT_dto_check ( DT_Tdep_Print_Head *phead dto_p->user_cookie.as_64, cookie_expected.as_64); } - return ( false ); + return ( FALSE ); } - return ( true ); + return ( TRUE ); } /* ----------------------------------------------------------- * Check an RMR Bind and print some debug info if anything is amiss. */ -bool +boolean_t DT_rmr_check ( DT_Tdep_Print_Head *phead, struct dat_rmr_bind_completion_event_data *rmr_p, DAT_RMR_HANDLE rmr_expected, @@ -646,17 +646,17 @@ DT_rmr_check ( DT_Tdep_Print_Head *phead rmr_p->user_cookie.as_ptr, cookie_expected); } - return ( false ); + return ( FALSE ); } - return ( true ); + return ( TRUE ); } /* ----------------------------------------------------------- * Check a CR and print some debug info if anything is amiss. */ -bool +boolean_t DT_cr_check ( DT_Tdep_Print_Head *phead, struct dat_cr_arrival_event_data *cr_stat_p, DAT_PSP_HANDLE psp_handle_expected, @@ -704,12 +704,12 @@ DT_cr_check ( DT_Tdep_Print_Head *phead DT_RetToString (ret)); } } - return ( false ); + return ( FALSE ); } if (cr_handlep) { *cr_handlep = cr_stat_p->cr_handle; } - return ( true ); + return ( TRUE ); } Index: linux-kernel/test/dapltest/test/dapl_client.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_client.c (revision 2306) +++ linux-kernel/test/dapltest/test/dapl_client.c (working copy) @@ -82,7 +82,7 @@ DT_cs_Client (Params_t * params_ptr, } DT_MemListInit (pt_ptr); /* init MemlistLock and memListHead */ DT_Thread_Init (pt_ptr); /* init ThreadLock and threadcount */ - pt_ptr->local_is_server = false; + pt_ptr->local_is_server = FALSE; pt_ptr->Client_Info.dapltest_version = DAPLTEST_VERSION; pt_ptr->Client_Info.is_little_endian = DT_local_is_little_endian; pt_ptr->Client_Info.test_type = params_ptr->test_type; @@ -213,8 +213,8 @@ DT_cs_Client (Params_t * params_ptr, DT_RoundSize (sizeof (Transaction_Cmd_t), 8192), 3, /* num_buffers */ pt_ptr->provider_attr.optimal_buffer_alignment, - false, - false); + FALSE, + FALSE); if (bpool == 0) { DT_Tdep_PT_Printf (phead, @@ -363,7 +363,7 @@ retry: memcpy ( (void *)buffp, (const void *)Transaction_Cmd, sizeof (Transaction_Cmd_t)); - DT_Transaction_Cmd_Endian ((Transaction_Cmd_t *)buffp, true); + DT_Transaction_Cmd_Endian ((Transaction_Cmd_t *)buffp, TRUE); break; } @@ -373,7 +373,7 @@ retry: memcpy ( (void *)buffp, (const void *)Quit_Cmd, sizeof (Quit_Cmd_t)); - DT_Quit_Cmd_Endian ((Quit_Cmd_t *) buffp, true); + DT_Quit_Cmd_Endian ((Quit_Cmd_t *) buffp, TRUE); break; } Index: linux-kernel/test/dapltest/test/dapl_fft_endpoint.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_endpoint.c (revision 2306) +++ linux-kernel/test/dapltest/test/dapl_fft_endpoint.c (working copy) @@ -33,7 +33,7 @@ int DT_endpoint_generic (Params_t *params_ptr, FFT_Cmd_t *cmd, - bool destroy_pz_early) + boolean_t destroy_pz_early) { char *dev_name; DAT_IA_HANDLE ia_handle; @@ -148,7 +148,7 @@ int DT_endpoint_case0 (Params_t *params_ Description: Test if we can normally create endpoint and destory it.\n"); DT_Tdep_PT_Printf (phead, "\ The endpoint is not associated with a CQ\n"); - return DT_endpoint_generic (params_ptr, cmd, false); /* destroy pz early */ + return DT_endpoint_generic (params_ptr, cmd, FALSE); /* destroy pz early */ } int DT_endpoint_case1 (Params_t *params_ptr, FFT_Cmd_t *cmd) @@ -158,7 +158,7 @@ int DT_endpoint_case1 (Params_t *params_ phead = params_ptr->phead; DT_Tdep_PT_Printf (phead, "\ Description: try to create endpoint with pz already destroyed\n"); - return DT_endpoint_generic (params_ptr, cmd, true); /* destroy pz early */ + return DT_endpoint_generic (params_ptr, cmd, TRUE); /* destroy pz early */ } int DT_endpoint_case2 (Params_t *params_ptr, FFT_Cmd_t *cmd) @@ -201,13 +201,13 @@ int DT_endpoint_case2 (Params_t *params_ &ep_handle); DT_assert_dat (phead, rc == DAT_SUCCESS); bpool = DT_BpoolAlloc (NULL, phead, ia_handle, pz_handle, NULL, NULL, 4096, - 1, 256 /* FIXME query */, false, false); + 1, 256 /* FIXME query */, FALSE, FALSE); DT_assert (phead, bpool != 0); DT_assert (phead, DT_post_recv_buffer (phead, ep_handle, bpool, 0, - 4096) == true); + 4096) == TRUE); if (ep_handle) { rc = dat_ep_free (ep_handle); @@ -228,7 +228,7 @@ cleanup: if (bpool) { rc = DT_Bpool_Destroy (NULL, phead, bpool); - DT_assert_clean (phead, rc != false); + DT_assert_clean (phead, rc != FALSE); } if (pz_handle) { Index: linux-kernel/test/dapltest/test/dapl_transaction_util.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_transaction_util.c (revision 2306) +++ linux-kernel/test/dapltest/test/dapl_transaction_util.c (working copy) @@ -30,7 +30,7 @@ /* ----------------------------------------------------------- * Post a recv buffer on each of this thread's EPs. */ -bool +boolean_t DT_handle_post_recv_buf (DT_Tdep_Print_Head *phead, Ep_Context_t * ep_context, unsigned int num_eps, @@ -71,24 +71,24 @@ DT_handle_post_recv_buf (DT_Tdep_Print_H "Test Error: dat_ep_post_recv failed: %s\n", DT_RetToString (ret)); DT_Test_Error (); - return false; + return FALSE; } } - return true; + return TRUE; } /* ----------------------------------------------------------- * Post a send buffer on each of this thread's EPs. */ -bool +boolean_t DT_handle_send_op (DT_Tdep_Print_Head *phead, Ep_Context_t * ep_context, DAT_EVD_HANDLE reqt_evd_hdl, unsigned int num_eps, int op_indx, - bool poll) + boolean_t poll) { unsigned int i, j; unsigned char *completion_reaped; @@ -97,7 +97,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p if (!completion_reaped) { - return false; + return FALSE; } for (i = 0; i < num_eps; i++) @@ -135,7 +135,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p DT_RetToString (ret)); DT_Test_Error (); DT_Mdep_Free (completion_reaped); - return false; + return FALSE; } } @@ -147,7 +147,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p { /* we will reap the send on the recv (Client SR) */ DT_Mdep_Free (completion_reaped); - return true; + return TRUE; } } @@ -164,7 +164,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p if (!DT_dto_event_reap (phead, reqt_evd_hdl, poll, &dto_stat)) { DT_Mdep_Free (completion_reaped); - return false; + return FALSE; } epnum = dto_stat.user_cookie.as_64 >> 32; @@ -177,7 +177,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p dto_stat.transfered_length); DT_Test_Error (); DT_Mdep_Free (completion_reaped); - return false; + return FALSE; } op = &ep_context[epnum].op[op_indx]; @@ -194,7 +194,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p "Send")) { DT_Mdep_Free (completion_reaped); - return false; + return FALSE; } if (completion_reaped[epnum]) @@ -204,7 +204,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p ep_context[epnum].ep_handle, epnum); DT_Test_Error (); DT_Mdep_Free (completion_reaped); - return ( false ); + return ( FALSE ); } completion_reaped[epnum] = 1; } @@ -218,12 +218,12 @@ DT_handle_send_op (DT_Tdep_Print_Head *p ep_context[i].ep_handle, i); DT_Test_Error (); DT_Mdep_Free (completion_reaped); - return ( false ); + return ( FALSE ); } } DT_Mdep_Free (completion_reaped); - return true; + return TRUE; } @@ -232,15 +232,15 @@ DT_handle_send_op (DT_Tdep_Print_Head *p * then if requested reap the corresponding send ops, * and re-post all of the recv buffers. */ -bool +boolean_t DT_handle_recv_op (DT_Tdep_Print_Head *phead, Ep_Context_t * ep_context, DAT_EVD_HANDLE recv_evd_hdl, DAT_EVD_HANDLE reqt_evd_hdl, unsigned int num_eps, int op_indx, - bool poll, - bool repost_recv) + boolean_t poll, + boolean_t repost_recv) { unsigned int i; unsigned char *recv_completion_reaped; @@ -249,14 +249,14 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p recv_completion_reaped = DT_Mdep_Malloc (num_eps); if (recv_completion_reaped == NULL) { - return false; + return FALSE; } send_completion_reaped = DT_Mdep_Malloc (num_eps); if (send_completion_reaped == NULL) { DT_Mdep_Free (recv_completion_reaped); - return false; + return FALSE; } /* Foreach EP, reap */ @@ -274,7 +274,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p { DT_Mdep_Free (recv_completion_reaped); DT_Mdep_Free (send_completion_reaped); - return false; + return FALSE; } epnum = dto_stat.user_cookie.as_64 >> 32; @@ -288,7 +288,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); DT_Mdep_Free (send_completion_reaped); - return false; + return FALSE; } op = &ep_context[epnum].op[op_indx]; @@ -308,7 +308,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); DT_Mdep_Free (send_completion_reaped); - return false; + return FALSE; } if (recv_completion_reaped[epnum]) @@ -319,7 +319,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); DT_Mdep_Free (send_completion_reaped); - return ( false ); + return ( FALSE ); } recv_completion_reaped[epnum] = 1; @@ -337,14 +337,14 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p " but current op == #%d\n", op_indx); DT_Mdep_Free (recv_completion_reaped); DT_Mdep_Free (send_completion_reaped); - return false; + return FALSE; } if (!DT_dto_event_reap (phead, reqt_evd_hdl, poll, &dto_stat)) { DT_Mdep_Free (recv_completion_reaped); DT_Mdep_Free (send_completion_reaped); - return false; + return FALSE; } epnum = dto_stat.user_cookie.as_64 >> 32; @@ -358,7 +358,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); DT_Mdep_Free (send_completion_reaped); - return false; + return FALSE; } /* @@ -388,7 +388,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); DT_Mdep_Free (send_completion_reaped); - return false; + return FALSE; } if (send_completion_reaped[epnum]) @@ -399,7 +399,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); DT_Mdep_Free (send_completion_reaped); - return ( false ); + return ( FALSE ); } send_completion_reaped[epnum] = 1; } @@ -415,7 +415,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); DT_Mdep_Free (send_completion_reaped); - return ( false ); + return ( FALSE ); } } @@ -432,7 +432,7 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); DT_Mdep_Free (send_completion_reaped); - return ( false ); + return ( FALSE ); } } } @@ -447,27 +447,27 @@ DT_handle_recv_op (DT_Tdep_Print_Head *p DT_Test_Error (); DT_Mdep_Free (recv_completion_reaped); DT_Mdep_Free (send_completion_reaped); - return false; + return FALSE; } } DT_Mdep_Free (recv_completion_reaped); DT_Mdep_Free (send_completion_reaped); - return true; + return TRUE; } /* ----------------------------------------------------------- * Initiate an RDMA op (synchronous) on each of this thread's EPs. */ -bool +boolean_t DT_handle_rdma_op (DT_Tdep_Print_Head *phead, Ep_Context_t * ep_context, DAT_EVD_HANDLE reqt_evd_hdl, unsigned int num_eps, DT_Transfer_Type opcode, int op_indx, - bool poll) + boolean_t poll) { unsigned int i, j; DAT_RETURN ret; @@ -477,7 +477,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p if (!completion_reaped) { - return false; + return FALSE; } /* Initiate the operation */ @@ -542,7 +542,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p DT_RetToString (ret)); DT_Test_Error (); DT_Mdep_Free (completion_reaped); - return ( false ); + return ( FALSE ); } else { @@ -565,7 +565,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p if (!DT_dto_event_reap (phead, reqt_evd_hdl, poll, &dto_stat)) { DT_Mdep_Free (completion_reaped); - return ( false ); + return ( FALSE ); } epnum = dto_stat.user_cookie.as_64 >> 32; @@ -578,7 +578,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p dto_stat.transfered_length); DT_Test_Error (); DT_Mdep_Free (completion_reaped); - return false; + return FALSE; } op = &ep_context[epnum].op[op_indx]; @@ -594,7 +594,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p (opcode == RDMA_WRITE ? "RDMA/WR" : "RDMA/RD"))) { DT_Mdep_Free (completion_reaped); - return ( false ); + return ( FALSE ); } if (completion_reaped[epnum]) @@ -604,7 +604,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p ep_context[epnum].ep_handle, epnum); DT_Test_Error (); DT_Mdep_Free (completion_reaped); - return ( false ); + return ( FALSE ); } completion_reaped[epnum] = 1; @@ -622,13 +622,13 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p ep_context[i].ep_handle, i); DT_Test_Error (); DT_Mdep_Free (completion_reaped); - return ( false ); + return ( FALSE ); } } DT_Mdep_Free (completion_reaped); - return ( true ); + return ( TRUE ); } @@ -636,7 +636,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p * Verify whether we (the client side) can support * the requested 'T' test. */ -bool +boolean_t DT_check_params (Per_Test_Data_t *pt_ptr, unsigned char *module) { @@ -646,7 +646,7 @@ DT_check_params (Per_Test_Data_t *pt_pt unsigned long num_rdma_wr = 0U; unsigned long max_size = 0U; unsigned long max_segs = 0U; - bool rval = true; + boolean_t rval = TRUE; unsigned int i; DT_Tdep_Print_Head *phead; @@ -715,13 +715,13 @@ DT_check_params (Per_Test_Data_t *pt_pt } case QUIT_TEST: { - return true; + return TRUE; } default: { DT_Tdep_PT_Printf (phead, "Unknown Test Type\n"); - return false; + return FALSE; } } @@ -748,7 +748,7 @@ DT_check_params (Per_Test_Data_t *pt_pt module, num_recvs > num_sends ? num_recvs : num_sends, pt_ptr->ia_attr.max_dto_per_ep); - rval = false; + rval = FALSE; } if (max_size > pt_ptr->ia_attr.max_lmr_block_size) { @@ -758,7 +758,7 @@ DT_check_params (Per_Test_Data_t *pt_pt module, max_size, pt_ptr->ia_attr.max_lmr_block_size); - rval = false; + rval = FALSE; } if (max_segs > pt_ptr->ep_attr.max_recv_iov || max_segs > pt_ptr->ep_attr.max_request_iov) @@ -775,7 +775,7 @@ DT_check_params (Per_Test_Data_t *pt_pt max_segs, pt_ptr->ep_attr.max_request_iov, pt_ptr->ep_attr.max_recv_iov ); - rval = false; + rval = FALSE; } return ( rval ); Index: linux-kernel/test/dapltest/test/dapl_fft_pz.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_fft_pz.c (revision 2306) +++ linux-kernel/test/dapltest/test/dapl_fft_pz.c (working copy) @@ -188,7 +188,7 @@ int DT_pz_case2 (Params_t *params_ptr, F /* allocate and register bpool */ bpool = DT_BpoolAlloc (NULL, phead, ia_handle, pz_handle, NULL, NULL, BUFFSIZE, 1, 256 /* FIXME query */, - false, false); + FALSE, FALSE); DT_assert (phead, bpool != 0); if (pz_handle) @@ -200,7 +200,7 @@ int DT_pz_case2 (Params_t *params_ptr, F cleanup: /* deregister and free bpool */ - if (DT_Bpool_Destroy (NULL, phead, bpool)==false) + if (DT_Bpool_Destroy (NULL, phead, bpool)==FALSE) { DT_Tdep_PT_Printf (phead, "Warning: Destroy bpool fails, reboot for cleanup\n"); return 0; Index: linux-kernel/test/dapltest/include/dapl_server_cmd.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_server_cmd.h (revision 2306) +++ linux-kernel/test/dapltest/include/dapl_server_cmd.h (working copy) @@ -32,7 +32,7 @@ typedef struct { - bool debug; /* -d */ + boolean_t debug; /* -d */ char dapl_name[256]; /* -D device name */ DAT_QOS ReliabilityLevel; /* -R */ DAT_MEM_TYPE DT_mem_type; /* -M */ Index: linux-kernel/test/dapltest/include/dapl_global.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_global.h (revision 2306) +++ linux-kernel/test/dapltest/include/dapl_global.h (working copy) @@ -31,6 +31,6 @@ #include "dapl_proto.h" extern DAT_COUNT DT_dapltest_debug; -extern bool DT_local_is_little_endian; +extern boolean_t DT_local_is_little_endian; #endif Index: linux-kernel/test/dapltest/include/dapl_fft_cmd.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_fft_cmd.h (revision 2306) +++ linux-kernel/test/dapltest/include/dapl_fft_cmd.h (working copy) @@ -58,7 +58,7 @@ typedef struct FFT_Type_e fft_type; char device_name[256]; /* -D */ char server_name[256]; - bool cases_flag[MAXCASES]; + boolean_t cases_flag[MAXCASES]; int size; int num_iter; /* -i */ int num_threads; /* -t */ Index: linux-kernel/test/dapltest/include/dapl_proto.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_proto.h (revision 2306) +++ linux-kernel/test/dapltest/include/dapl_proto.h (working copy) @@ -91,7 +91,7 @@ Bpool * DT_BpoolAlloc (Per_Test_ boolean_t enable_rdma_write, boolean_t enable_rdma_read); -bool DT_Bpool_Destroy (Per_Test_Data_t * pt_ptr, +boolean_t DT_Bpool_Destroy (Per_Test_Data_t * pt_ptr, DT_Tdep_Print_Head *phead, Bpool * bpool_ptr); @@ -126,7 +126,7 @@ void DT_init_transaction_stat void DT_transaction_stats_set_ready (DT_Tdep_Print_Head* phead, Transaction_Stats_t* transaction_stats); -bool DT_transaction_stats_wait_for_all (DT_Tdep_Print_Head* phead, +boolean_t DT_transaction_stats_wait_for_all (DT_Tdep_Print_Head* phead, Transaction_Stats_t* transaction_stats); void DT_update_transaction_stats (Transaction_Stats_t * transaction_stats, @@ -166,22 +166,22 @@ void Dapltest_Main_Usage (voi /* dapl_mdep.c */ void DT_Mdep_Init (void); void DT_Mdep_End (void); -bool DT_Mdep_GetDefaultDeviceName (char *dapl_name); +boolean_t DT_Mdep_GetDefaultDeviceName (char *dapl_name); void DT_Mdep_Sleep (int msec); void DT_Mdep_Schedule (void); -bool DT_Mdep_GetCpuStat (DT_CpuStat *sys_stat); +boolean_t DT_Mdep_GetCpuStat (DT_CpuStat *sys_stat); unsigned long DT_Mdep_GetTime (void); double DT_Mdep_GetCpuMhz (void); unsigned long DT_Mdep_GetContextSwitchNum (void); void *DT_Mdep_Malloc (size_t l_); void DT_Mdep_Free (void *a_); -bool DT_Mdep_LockInit (DT_Mdep_LockType * lock_ptr); +boolean_t DT_Mdep_LockInit (DT_Mdep_LockType * lock_ptr); void DT_Mdep_LockDestroy (DT_Mdep_LockType * lock_ptr); void DT_Mdep_Lock (DT_Mdep_LockType * lock_ptr); void DT_Mdep_Unlock (DT_Mdep_LockType * lock_ptr); void DT_Mdep_Thread_Init_Attributes (Thread * thread_ptr); void DT_Mdep_Thread_Destroy_Attributes (Thread * thread_ptr); -bool DT_Mdep_Thread_Start (Thread * thread_ptr); +boolean_t DT_Mdep_Thread_Start (Thread * thread_ptr); void DT_Mdep_Thread_Detach (DT_Mdep_ThreadHandleType thread_id); DT_Mdep_ThreadHandleType DT_Mdep_Thread_SELF ( void ); @@ -208,7 +208,7 @@ void DT_MemListFree (Per_Test void DT_PrintMemList (Per_Test_Data_t * pt_ptr); /* dapl_netaddr.c */ -bool DT_NetAddrLookupHostAddress (DAT_IA_ADDRESS_PTR to_netaddr, +boolean_t DT_NetAddrLookupHostAddress (DAT_IA_ADDRESS_PTR to_netaddr, char *hostname); DAT_IA_ADDRESS_PTR DT_NetAddrAlloc (Per_Test_Data_t * pt_ptr); @@ -217,16 +217,16 @@ void DT_NetAddrFree (Per_Test DAT_IA_ADDRESS_PTR netaddr); /* dapl_params.c */ -bool DT_Params_Parse (int argc, +boolean_t DT_Params_Parse (int argc, char *argv[], Params_t * params_ptr); /* dapl_performance_cmd.c */ const char * DT_PerformanceModeToString (Performance_Mode_Type mode); -bool DT_Performance_Cmd_Init (Performance_Cmd_t * cmd); +boolean_t DT_Performance_Cmd_Init (Performance_Cmd_t * cmd); -bool DT_Performance_Cmd_Parse (Performance_Cmd_t * cmd, +boolean_t DT_Performance_Cmd_Parse (Performance_Cmd_t * cmd, int my_argc, char **my_argv, mygetopt_t * opts); @@ -243,11 +243,11 @@ int DT_Performance_Test_Clie DAT_IA_HANDLE * ia_handle, DAT_IA_ADDRESS_PTR remote); -bool DT_Performance_Test_Client_Connect ( +boolean_t DT_Performance_Test_Client_Connect ( DT_Tdep_Print_Head *phead, Performance_Test_t * test_ptr); -bool DT_Performance_Test_Client_Exchange ( +boolean_t DT_Performance_Test_Client_Exchange ( Params_t *params_ptr, DT_Tdep_Print_Head *phead, Performance_Test_t *test_ptr); @@ -255,16 +255,16 @@ bool DT_Performance_Test_Clie /* dapl_performance_server.c */ void DT_Performance_Test_Server (void * pt_ptr); -bool DT_Performance_Test_Server_Connect ( +boolean_t DT_Performance_Test_Server_Connect ( DT_Tdep_Print_Head *phead, Performance_Test_t * test_ptr); -bool DT_Performance_Test_Server_Exchange ( +boolean_t DT_Performance_Test_Server_Exchange ( DT_Tdep_Print_Head *phead, Performance_Test_t *test_ptr); /* dapl_performance_util.c */ -bool DT_Performance_Test_Create (Per_Test_Data_t * pt_ptr, +boolean_t DT_Performance_Test_Create (Per_Test_Data_t * pt_ptr, DAT_IA_HANDLE * ia_handle, DAT_IA_ADDRESS_PTR remote_ia_addr, boolean_t is_server, @@ -275,7 +275,7 @@ int DT_Performance_Test_Dest Performance_Test_t *test_ptr, boolean_t is_server); -bool DT_performance_post_rdma_op (Performance_Ep_Context_t *ep_context, +boolean_t DT_performance_post_rdma_op (Performance_Ep_Context_t *ep_context, DAT_EVD_HANDLE reqt_evd_hdl, Performance_Stats_t *stats); @@ -326,7 +326,7 @@ int DT_cs_Server (Params_t * /* dapl_server_cmd.c */ void DT_Server_Cmd_Init (Server_Cmd_t * Server_Cmd); -bool DT_Server_Cmd_Parse (Server_Cmd_t * Server_Cmd, +boolean_t DT_Server_Cmd_Parse (Server_Cmd_t * Server_Cmd, int my_argc, char **my_argv, mygetopt_t * opts); @@ -365,50 +365,50 @@ boolean_t DT_post_send_buffer (DT_Td int index, int size); -bool DT_conn_event_wait (DT_Tdep_Print_Head* phead, +boolean_t DT_conn_event_wait (DT_Tdep_Print_Head* phead, DAT_EP_HANDLE ep_handle, DAT_EVD_HANDLE evd_handle, DAT_EVENT_NUMBER *event_number); -bool DT_disco_event_wait ( DT_Tdep_Print_Head* phead, +boolean_t DT_disco_event_wait ( DT_Tdep_Print_Head* phead, DAT_EVD_HANDLE evd_handle, DAT_EP_HANDLE *ep_handle ); -bool DT_cr_event_wait (DT_Tdep_Print_Head* phead, +boolean_t DT_cr_event_wait (DT_Tdep_Print_Head* phead, DAT_EVD_HANDLE evd_handle, struct dat_cr_arrival_event_data *cr_stat_p); -bool DT_dto_event_reap (DT_Tdep_Print_Head* phead, +boolean_t DT_dto_event_reap (DT_Tdep_Print_Head* phead, DAT_EVD_HANDLE evd_handle, - bool poll, + boolean_t poll, struct dat_dto_completion_event_data *dtop); -bool DT_dto_event_wait (DT_Tdep_Print_Head* phead, +boolean_t DT_dto_event_wait (DT_Tdep_Print_Head* phead, DAT_EVD_HANDLE evd_handle, struct dat_dto_completion_event_data *dtop); -bool DT_dto_event_poll (DT_Tdep_Print_Head* phead, +boolean_t DT_dto_event_poll (DT_Tdep_Print_Head* phead, DAT_EVD_HANDLE evd_handle, struct dat_dto_completion_event_data *dtop); -bool DT_rmr_event_wait (DT_Tdep_Print_Head* phead, +boolean_t DT_rmr_event_wait (DT_Tdep_Print_Head* phead, DAT_EVD_HANDLE evd_handle, struct dat_rmr_bind_completion_event_data *rmr_ptr); -bool DT_dto_check ( DT_Tdep_Print_Head* phead, +boolean_t DT_dto_check ( DT_Tdep_Print_Head* phead, struct dat_dto_completion_event_data *dto_p, DAT_EP_HANDLE ep_expected, DAT_COUNT len_expected, DAT_DTO_COOKIE cookie_expected, char *message); -bool DT_rmr_check ( DT_Tdep_Print_Head* phead, +boolean_t DT_rmr_check ( DT_Tdep_Print_Head* phead, struct dat_rmr_bind_completion_event_data *rmr_p, DAT_RMR_HANDLE rmr_expected, DAT_PVOID cookie_expected, char *message); -bool DT_cr_check (DT_Tdep_Print_Head* phead, +boolean_t DT_cr_check (DT_Tdep_Print_Head* phead, struct dat_cr_arrival_event_data *cr_stat_p, DAT_PSP_HANDLE psp_handle_expected, DAT_CONN_QUAL port_expected, @@ -428,20 +428,20 @@ Thread *DT_Thread_Create (Per_Te void DT_Thread_Destroy (Thread * thread_ptr, Per_Test_Data_t * pt_ptr); -bool DT_Thread_Start (Thread * thread_ptr); +boolean_t DT_Thread_Start (Thread * thread_ptr); /* dapl_quit_cmd.c */ void DT_Quit_Cmd_Init (Quit_Cmd_t * cmd); -bool DT_Quit_Cmd_Parse (Quit_Cmd_t * cmd, +boolean_t DT_Quit_Cmd_Parse (Quit_Cmd_t * cmd, int my_argc, char **my_argv, mygetopt_t * opts); -bool DT_Quit_Cmd_Validate (Quit_Cmd_t * cmd); +boolean_t DT_Quit_Cmd_Validate (Quit_Cmd_t * cmd); void DT_Quit_Cmd_Endian (Quit_Cmd_t * cmd, - bool to_wire); + boolean_t to_wire); void DT_Quit_Cmd_Print (Quit_Cmd_t * cmd); void DT_Quit_Cmd_PT_Print (DT_Tdep_Print_Head *phead, Quit_Cmd_t * cmd); @@ -451,7 +451,7 @@ void DT_Quit_Cmd_Usage (void) /* dapl_transaction_cmd.c */ void DT_Transaction_Cmd_Init (Transaction_Cmd_t * cmd); -bool DT_Transaction_Cmd_Parse (Transaction_Cmd_t * cmd, +boolean_t DT_Transaction_Cmd_Parse (Transaction_Cmd_t * cmd, int my_argc, char **my_argv, mygetopt_t * opts); @@ -461,7 +461,7 @@ void DT_Transaction_Cmd_PT_Pr Transaction_Cmd_t * cmd); void DT_Transaction_Cmd_Endian (Transaction_Cmd_t * cmd, - bool to_wire); + boolean_t to_wire); /* dapl_transaction_test.c */ int DT_Transaction_Test_Client (Per_Test_Data_t * pt_ptr, DAT_IA_HANDLE ia_handle, @@ -469,7 +469,7 @@ int DT_Transaction_Test_Clie void DT_Transaction_Test_Server (void *params); -bool DT_Transaction_Create_Test (Per_Test_Data_t * pt_ptr, +boolean_t DT_Transaction_Create_Test (Per_Test_Data_t * pt_ptr, DAT_IA_HANDLE * ia_handle, boolean_t is_server, unsigned int port_num, @@ -477,47 +477,47 @@ bool DT_Transaction_Create_Te DAT_IA_ADDRESS_PTR remote_ia_addr); void DT_Transaction_Main (void *param); -bool DT_Transaction_Run (DT_Tdep_Print_Head* phead, +boolean_t DT_Transaction_Run (DT_Tdep_Print_Head* phead, Transaction_Test_t * test_ptr); void DT_Transaction_Validation_Fill (DT_Tdep_Print_Head* phead, Transaction_Test_t * test_ptr, unsigned int iteration); -bool DT_Transaction_Validation_Check (DT_Tdep_Print_Head* phead, +boolean_t DT_Transaction_Validation_Check (DT_Tdep_Print_Head* phead, Transaction_Test_t * test_ptr, int iteration); void DT_Print_Transaction_Test (DT_Tdep_Print_Head* phead, Transaction_Test_t* test_ptr); /* dapl_transaction_util.c */ -bool DT_handle_post_recv_buf (DT_Tdep_Print_Head* phead, +boolean_t DT_handle_post_recv_buf (DT_Tdep_Print_Head* phead, Ep_Context_t * ep_context, unsigned int num_eps, int op_indx); -bool DT_handle_send_op (DT_Tdep_Print_Head* phead, +boolean_t DT_handle_send_op (DT_Tdep_Print_Head* phead, Ep_Context_t * ep_context, DAT_EVD_HANDLE reqt_evd_hdl, unsigned int num_eps, int op_indx, - bool poll); + boolean_t poll); -bool DT_handle_recv_op (DT_Tdep_Print_Head* phead, +boolean_t DT_handle_recv_op (DT_Tdep_Print_Head* phead, Ep_Context_t * ep_context, DAT_EVD_HANDLE recv_evd_hdl, DAT_EVD_HANDLE reqt_evd_hdl, unsigned int num_eps, int op_indx, - bool poll, - bool repost_recv); + boolean_t poll, + boolean_t repost_recv); -bool DT_handle_rdma_op (DT_Tdep_Print_Head* phead, +boolean_t DT_handle_rdma_op (DT_Tdep_Print_Head* phead, Ep_Context_t * ep_context, DAT_EVD_HANDLE reqt_evd_hdl, unsigned int num_eps, DT_Transfer_Type opcode, int op_indx, - bool poll); + boolean_t poll); -bool DT_check_params (Per_Test_Data_t *pt_ptr, +boolean_t DT_check_params (Per_Test_Data_t *pt_ptr, unsigned char *module); void DT_Test_Error (void); @@ -544,7 +544,7 @@ DAT_COUNT DT_RoundSize (DAT_COUNT /* dapl_limit_cmd.c */ void DT_Limit_Cmd_Init ( Limit_Cmd_t * cmd); -bool DT_Limit_Cmd_Parse ( Limit_Cmd_t * cmd, +boolean_t DT_Limit_Cmd_Parse ( Limit_Cmd_t * cmd, int my_argc, char **my_argv, mygetopt_t * opts); @@ -557,7 +557,7 @@ void DT_cs_Limit (Params_t *param /* dapl_fft_cmd.c */ void DT_FFT_Cmd_Init ( FFT_Cmd_t * cmd); -bool DT_FFT_Cmd_Parse ( FFT_Cmd_t * cmd, +boolean_t DT_FFT_Cmd_Parse ( FFT_Cmd_t * cmd, int my_argc, char **my_argv, mygetopt_t * opts); @@ -581,7 +581,7 @@ int DT_hwconn_case7 (Params_t *param /* dapl_fft_endpoint.c */ void DT_endpoint_test (Params_t *params_ptr, FFT_Cmd_t *cmd); int DT_endpoint_generic (Params_t *params_ptr, FFT_Cmd_t *cmd, - bool destroy_pz_early); + boolean_t destroy_pz_early); int DT_endpoint_case0 (Params_t *params_ptr, FFT_Cmd_t *cmd); int DT_endpoint_case1 (Params_t *params_ptr, FFT_Cmd_t *cmd); int DT_endpoint_case2 (Params_t *params_ptr, FFT_Cmd_t *cmd); Index: linux-kernel/test/dapltest/include/dapl_test_data.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_test_data.h (revision 2306) +++ linux-kernel/test/dapltest/include/dapl_test_data.h (working copy) @@ -76,7 +76,7 @@ typedef struct int Thread_counter; Thread *thread; - bool local_is_server; + boolean_t local_is_server; Server_Info_t Server_Info; Client_Info_t Client_Info; Params_t Params; Index: linux-kernel/test/dapltest/include/dapl_fft_util.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_fft_util.h (revision 2306) +++ linux-kernel/test/dapltest/include/dapl_fft_util.h (working copy) @@ -68,7 +68,7 @@ typedef struct DAT_EVENT_NUMBER event_num; DAT_IA_ADDRESS_PTR remote_netaddr; Per_Test_Data_t *pt_ptr; - bool connected; + boolean_t connected; } FFT_Connection_t; typedef enum Index: linux-kernel/test/dapltest/include/dapl_params.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_params.h (revision 2306) +++ linux-kernel/test/dapltest/include/dapl_params.h (working copy) @@ -64,8 +64,8 @@ typedef struct DAT_QOS ReliabilityLevel; struct sockaddr server_netaddr; void * phead; - bool local_is_little_endian; - bool debug; + boolean_t local_is_little_endian; + boolean_t debug; DAT_MEM_TYPE mem_type; Transaction_Stats2_t Client_Stats_T; Performance_Stats_t Client_Stats_P; Index: linux-kernel/test/dapltest/cmd/dapl_netaddr.c =================================================================== --- linux-kernel/test/dapltest/cmd/dapl_netaddr.c (revision 2306) +++ linux-kernel/test/dapltest/cmd/dapl_netaddr.c (working copy) @@ -48,7 +48,7 @@ DT_NetAddrFree (Per_Test_Data_t * pt_ptr } -bool +boolean_t DT_NetAddrLookupHostAddress (DAT_IA_ADDRESS_PTR to_netaddr, DAT_NAME_PTR hostname) { Index: linux-kernel/test/dapltest/cmd/dapl_limit_cmd.c =================================================================== --- linux-kernel/test/dapltest/cmd/dapl_limit_cmd.c (revision 2306) +++ linux-kernel/test/dapltest/cmd/dapl_limit_cmd.c (working copy) @@ -39,7 +39,7 @@ DT_Limit_Cmd_Init (Limit_Cmd_t * cmd) } /* --------------------------------------------------- */ -bool +boolean_t DT_Limit_Cmd_Parse ( Limit_Cmd_t * cmd, int my_argc, char **my_argv, @@ -70,7 +70,7 @@ DT_Limit_Cmd_Parse ( Limit_Cmd_t * cmd, case 'd': /* print debug messages */ { DT_dapltest_debug++; - cmd->debug = true; + cmd->debug = TRUE; break; } case 'm': /* maximum for exhaustion testing */ @@ -81,7 +81,7 @@ DT_Limit_Cmd_Parse ( Limit_Cmd_t * cmd, { DT_Mdep_printf ("Syntax Error -m option\n"); DT_Limit_Cmd_Usage (); - return (false); + return (FALSE); } cmd->maximum = atol (opts->optarg); break; @@ -94,7 +94,7 @@ DT_Limit_Cmd_Parse ( Limit_Cmd_t * cmd, { DT_Mdep_printf ("Syntax Error -w option\n"); DT_Limit_Cmd_Usage (); - return (false); + return (FALSE); } cmd->width = atol (opts->optarg); break; @@ -104,7 +104,7 @@ DT_Limit_Cmd_Parse ( Limit_Cmd_t * cmd, { DT_Mdep_printf ("Invalid Limit Test Parameter: %c\n", c); DT_Limit_Cmd_Usage (); - return (false); + return (FALSE); } } } @@ -114,7 +114,7 @@ DT_Limit_Cmd_Parse ( Limit_Cmd_t * cmd, { DT_Mdep_printf ("can't get default device name\n"); DT_Limit_Cmd_Usage (); - return (false); + return (FALSE); } } @@ -189,12 +189,12 @@ DT_Limit_Cmd_Parse ( Limit_Cmd_t * cmd, DT_Mdep_printf ("Cannot find this limit test: %s\n", my_argv[i]); DT_Limit_Cmd_Usage (); - return (false); + return (FALSE); } /* end foreach remaining argv */ } - return (true); + return (TRUE); } /* --------------------------------------------------- */ Index: linux-kernel/test/dapltest/cmd/dapl_fft_cmd.c =================================================================== --- linux-kernel/test/dapltest/cmd/dapl_fft_cmd.c (revision 2306) +++ linux-kernel/test/dapltest/cmd/dapl_fft_cmd.c (working copy) @@ -37,7 +37,7 @@ void DT_FFT_Cmd_Init (FFT_Cmd_t *cmd) cmd->server_name[0] = '\0'; for (i=0; icases_flag[i] = false; + cmd->cases_flag[i] = FALSE; } cmd->size = 0; cmd->num_iter = 1000; @@ -48,7 +48,7 @@ void DT_FFT_Cmd_Init (FFT_Cmd_t *cmd) } /*------------------------------------------------------------------------------*/ -bool DT_FFT_Cmd_Parse (FFT_Cmd_t *cmd, +boolean_t DT_FFT_Cmd_Parse (FFT_Cmd_t *cmd, int my_argc, char ** my_argv, mygetopt_t *opts) @@ -83,7 +83,7 @@ bool DT_FFT_Cmd_Parse (FFT_Cmd_t *cmd, { DT_Mdep_printf ("Syntax Error -i option\n"); DT_FFT_Cmd_Usage (); - return (false); + return (FALSE); } cmd->num_iter = atoi (opts->optarg); break; @@ -95,7 +95,7 @@ bool DT_FFT_Cmd_Parse (FFT_Cmd_t *cmd, { DT_Mdep_printf ("Syntax Error -t option\n"); DT_FFT_Cmd_Usage (); - return (false); + return (FALSE); } cmd->num_threads = atoi (opts->optarg); break; @@ -107,7 +107,7 @@ bool DT_FFT_Cmd_Parse (FFT_Cmd_t *cmd, { DT_Mdep_printf ("Syntax Error -v option\n"); DT_FFT_Cmd_Usage (); - return (false); + return (FALSE); } cmd->num_vis = atoi (opts->optarg); break; @@ -207,7 +207,7 @@ bool DT_FFT_Cmd_Parse (FFT_Cmd_t *cmd, DT_Mdep_printf ("don't know this function feature: %s\n", opts->optarg); DT_FFT_Cmd_Usage (); - return (false); + return (FALSE); } } case 'R': /* Service Reliability Level */ @@ -217,7 +217,7 @@ bool DT_FFT_Cmd_Parse (FFT_Cmd_t *cmd, { DT_Mdep_printf ("Invalid FFT Test Parameter: %c\n", c); DT_FFT_Cmd_Usage (); - return (false); + return (FALSE); } break; } @@ -227,7 +227,7 @@ bool DT_FFT_Cmd_Parse (FFT_Cmd_t *cmd, { DT_Mdep_printf ("Invalid FFT Test Parameter: %c\n", c); DT_FFT_Cmd_Usage (); - return (false); + return (FALSE); } } } @@ -237,7 +237,7 @@ bool DT_FFT_Cmd_Parse (FFT_Cmd_t *cmd, { DT_Mdep_printf ("can't get default device name\n"); DT_FFT_Cmd_Usage (); - return (false); + return (FALSE); } } @@ -245,7 +245,7 @@ bool DT_FFT_Cmd_Parse (FFT_Cmd_t *cmd, { DT_Mdep_printf ("must define the function feature with -f to test\n"); DT_FFT_Cmd_Usage (); - return (false); + return (FALSE); } if (cmd->server_name[0] =='\0' && (cmd->fft_type==CONNMGT_CLIENT || cmd->fft_type == DATAXFER_CLIENT || @@ -253,7 +253,7 @@ bool DT_FFT_Cmd_Parse (FFT_Cmd_t *cmd, { DT_Mdep_printf ("must define the server name with -s option\n"); DT_FFT_Cmd_Usage (); - return (false); + return (FALSE); } if (cmd->server_name[0] =='\0' && cmd->fft_type==NS ) @@ -261,7 +261,7 @@ bool DT_FFT_Cmd_Parse (FFT_Cmd_t *cmd, DT_Mdep_printf ("\ Must specify host name or host IP address with -s option to be tested\n"); DT_FFT_Cmd_Usage (); - return (false); + return (FALSE); } /* now parse the test cases */ @@ -269,9 +269,9 @@ bool DT_FFT_Cmd_Parse (FFT_Cmd_t *cmd, { for (i=0; isize; i++) { - cmd->cases_flag[i] = true; + cmd->cases_flag[i] = TRUE; } - return true; + return TRUE; } /* test specified cases */ @@ -282,14 +282,14 @@ bool DT_FFT_Cmd_Parse (FFT_Cmd_t *cmd, { DT_Mdep_printf ("test cases format is not correct: %s\n", my_argv[i]); DT_FFT_Cmd_Usage (); - return (false); + return (FALSE); } len = strspn (my_argv[i] + 4, "0123456789"); if (len==0 || len != strlen (my_argv[i] + 4) ) { DT_Mdep_printf ("must specify case number: %s\n", my_argv[i]); DT_FFT_Cmd_Usage (); - return (false); + return (FALSE); } caseNum = atoi (my_argv[i] + 4); if (caseNum<0 || caseNum>=cmd->size) @@ -297,12 +297,12 @@ bool DT_FFT_Cmd_Parse (FFT_Cmd_t *cmd, DT_Mdep_printf ("test case number must be within range : 0 -- %d\n", cmd->size-1); DT_FFT_Cmd_Usage (); - return (false); + return (FALSE); } - cmd->cases_flag[caseNum] = true; + cmd->cases_flag[caseNum] = TRUE; i++; } - return (true); + return (TRUE); } /*--------------------------------------------------------------*/ Index: linux-kernel/test/dapltest/cmd/dapl_params.c =================================================================== --- linux-kernel/test/dapltest/cmd/dapl_params.c (revision 2306) +++ linux-kernel/test/dapltest/cmd/dapl_params.c (working copy) @@ -32,7 +32,7 @@ /* Parse command line arguments */ -bool +boolean_t DT_Params_Parse (int argc, char *argv[], Params_t * params_ptr) { Server_Cmd_t *Server_Cmd; @@ -73,9 +73,9 @@ DT_Params_Parse (int argc, char *argv[], if (!DT_Mdep_GetDefaultDeviceName (Server_Cmd->dapl_name)) { DT_Mdep_printf ("can't get default device name\n"); - return false; + return FALSE; } - return true; + return TRUE; } /* check for a script file */ if (strncmp (argv[1], "-f", 2) == 0) @@ -273,7 +273,7 @@ DT_Params_Parse (int argc, char *argv[], params_ptr->mem_type = FFT_Cmd->DT_mem_type; DT_NetAddrLookupHostAddress (¶ms_ptr->server_netaddr, FFT_Cmd->server_name); - params_ptr->debug = false; + params_ptr->debug = FALSE; break; } @@ -292,7 +292,7 @@ DT_Params_Parse (int argc, char *argv[], } fclose (fd); } - return true; + return TRUE; main_usage: Dapltest_Main_Usage (); @@ -305,7 +305,7 @@ error_return: } fclose (fd); } - return false; + return FALSE; } DAT_MEM_TYPE Index: linux-kernel/test/dapltest/cmd/dapl_performance_cmd.c =================================================================== --- linux-kernel/test/dapltest/cmd/dapl_performance_cmd.c (revision 2306) +++ linux-kernel/test/dapltest/cmd/dapl_performance_cmd.c (working copy) @@ -58,7 +58,7 @@ DT_Performance_Cmd_Usage (void) DT_Mdep_printf ("USAGE: [seg_size [num_segs] ] : (4096, 1)\n"); } -static bool +static boolean_t DT_Performance_Cmd_Parse_Op ( Performance_Cmd_t * cmd, int index, @@ -74,7 +74,7 @@ DT_Performance_Cmd_Parse_Op ( if ( index == my_argc ) { DT_Mdep_printf ("Operation Missing Transfer Type\n"); - return (false); + return (FALSE); } for ( i = 0; index < my_argc; i++, index++ ) @@ -94,7 +94,7 @@ DT_Performance_Cmd_Parse_Op ( else { DT_Mdep_printf ("OP type must be \n"); - return (false); + return (FALSE); } break; } @@ -111,35 +111,35 @@ DT_Performance_Cmd_Parse_Op ( default: { DT_Mdep_printf ("Too many OP args\n"); - return (false); + return (FALSE); } } } - return (true); + return (TRUE); } -static bool +static boolean_t DT_Performance_Cmd_Validate ( Performance_Cmd_t *cmd) { if ( '\0' == cmd->server_name[0] ) { DT_Mdep_printf ("Must specify server_name in command line or scriptfile\n"); - return (false); + return (FALSE); } if ( '\0' == cmd->dapl_name[0] ) { DT_Mdep_printf ("Must specify device_name in command line or scriptfile\n"); - return (false); + return (FALSE); } if ( 0 == cmd->pipeline_len ) { DT_Mdep_printf ("Pipeline size must not be 0\n"); - return (false); + return (FALSE); } if ( cmd->debug ) @@ -147,10 +147,10 @@ DT_Performance_Cmd_Validate ( DT_Performance_Cmd_Print (cmd); } - return true; + return TRUE; } -bool +boolean_t DT_Performance_Cmd_Parse ( Performance_Cmd_t *cmd, int my_argc, @@ -179,7 +179,7 @@ DT_Performance_Cmd_Parse ( case 'd': /* print debug messages */ { DT_dapltest_debug++; - cmd->debug = true; + cmd->debug = TRUE; break; } case 'm': /* mode */ @@ -196,7 +196,7 @@ DT_Performance_Cmd_Parse ( { DT_Mdep_printf ("Syntax Error -m option\n"); DT_Performance_Cmd_Usage (); - return (false); + return (FALSE); } break; @@ -208,7 +208,7 @@ DT_Performance_Cmd_Parse ( { DT_Mdep_printf ("Syntax Error -i option\n"); DT_Performance_Cmd_Usage (); - return (false); + return (FALSE); } cmd->num_iterations = atol (opts->optarg); break; @@ -220,7 +220,7 @@ DT_Performance_Cmd_Parse ( { DT_Mdep_printf ("Syntax Error -p option\n"); DT_Performance_Cmd_Usage (); - return (false); + return (FALSE); } cmd->pipeline_len = atol (opts->optarg); break; @@ -243,7 +243,7 @@ DT_Performance_Cmd_Parse ( { DT_Mdep_printf ("must specify server name\n"); DT_Performance_Cmd_Usage (); - return (false); + return (FALSE); } strncpy (cmd->server_name, opts->optarg, NAME_SZ); @@ -253,7 +253,7 @@ DT_Performance_Cmd_Parse ( { DT_Mdep_printf ("Invalid Performance Test Parameter: %c\n", c); DT_Performance_Cmd_Usage (); - return (false); + return (FALSE); } } } @@ -264,27 +264,27 @@ DT_Performance_Cmd_Parse ( if ( !DT_Performance_Cmd_Parse_Op (cmd, opts->optind, my_argc, my_argv) ) { DT_Performance_Cmd_Usage (); - return (false); + return (FALSE); } if ( !DT_Performance_Cmd_Validate (cmd) ) { DT_Performance_Cmd_Usage (); - return (false); + return (FALSE); } - return (true); + return (TRUE); } -bool +boolean_t DT_Performance_Cmd_Init (Performance_Cmd_t * cmd) { memset (cmd, 0, sizeof (Performance_Cmd_t)); cmd->dapltest_version = DAPLTEST_VERSION; cmd->client_is_little_endian = DT_local_is_little_endian; cmd->qos = DAT_QOS_BEST_EFFORT; - cmd->debug = false; + cmd->debug = FALSE; cmd->num_iterations = 1000; cmd->pipeline_len = ~0; @@ -296,10 +296,10 @@ DT_Performance_Cmd_Init (Performance_Cmd if ( !DT_Mdep_GetDefaultDeviceName (cmd->dapl_name) ) { DT_Mdep_printf ("can't get default device name\n"); - return (false); + return (FALSE); } - return (true); + return (TRUE); } void Index: linux-kernel/test/dapltest/cmd/dapl_quit_cmd.c =================================================================== --- linux-kernel/test/dapltest/cmd/dapl_quit_cmd.c (revision 2306) +++ linux-kernel/test/dapltest/cmd/dapl_quit_cmd.c (working copy) @@ -37,7 +37,7 @@ DT_Quit_Cmd_Init (Quit_Cmd_t * cmd) } /*--------------------------------------------------------- */ -bool +boolean_t DT_Quit_Cmd_Parse (Quit_Cmd_t * cmd, int my_argc, char **my_argv, @@ -67,7 +67,7 @@ DT_Quit_Cmd_Parse (Quit_Cmd_t * cmd, case 'd': /* print debug messages */ { DT_dapltest_debug++; - cmd->debug = true; + cmd->debug = TRUE; break; } case 'R': /* Service Reliability Level */ @@ -85,7 +85,7 @@ DT_Quit_Cmd_Parse (Quit_Cmd_t * cmd, { DT_Mdep_printf ("Invalid Quit option: %c\n", opts->optopt); DT_Quit_Cmd_Usage (); - return (false); + return (FALSE); } } } @@ -95,27 +95,27 @@ DT_Quit_Cmd_Parse (Quit_Cmd_t * cmd, { DT_Mdep_printf ("can't get default device name\n"); DT_Quit_Cmd_Usage (); - return (false); + return (FALSE); } } if (!DT_Quit_Cmd_Validate (cmd)) { DT_Quit_Cmd_Usage (); - return (false); + return (FALSE); } - return (true); + return (TRUE); } /*------------------------------------------------------------------------------ */ -bool +boolean_t DT_Quit_Cmd_Validate (Quit_Cmd_t * cmd) { if (cmd->server_name[0] == '\0') { DT_Mdep_printf ("Must specify server_name in command line or scriptfile\n"); - return (false); + return (FALSE); } - return (true); + return (TRUE); } /*--------------------------------------------------------- */ Index: linux-kernel/test/dapltest/cmd/dapl_transaction_cmd.c =================================================================== --- linux-kernel/test/dapltest/cmd/dapl_transaction_cmd.c (revision 2306) +++ linux-kernel/test/dapltest/cmd/dapl_transaction_cmd.c (working copy) @@ -27,7 +27,7 @@ #include "dapl_proto.h" -static bool +static boolean_t DT_Transaction_Cmd_Parse_Op (Transaction_Cmd_t * cmd, char *arg) { char *c_ptr; @@ -43,7 +43,7 @@ DT_Transaction_Cmd_Parse_Op (Transaction /* set some defaults */ cmd->op[op].seg_size = 4096; cmd->op[op].num_segs = 1; - cmd->op[op].reap_send_on_recv = false; + cmd->op[op].reap_send_on_recv = FALSE; /* * packet format: [seg_size] [num_segs] @@ -57,13 +57,13 @@ DT_Transaction_Cmd_Parse_Op (Transaction /* first token is : */ if (strcmp (c_ptr, "server") == 0) { - cmd->op[op].server_initiated = true; + cmd->op[op].server_initiated = TRUE; } else { if (strcmp (c_ptr, "client") == 0) { - cmd->op[op].server_initiated = false; + cmd->op[op].server_initiated = FALSE; } else { @@ -119,7 +119,7 @@ DT_Transaction_Cmd_Parse_Op (Transaction } if (c_ptr && strcmp (c_ptr, "-f") == 0) { - cmd->op[op].reap_send_on_recv = true; + cmd->op[op].reap_send_on_recv = TRUE; if (cmd->op[op].transfer_type != SEND_RECV) { DT_Mdep_printf ("OP: -f only valid on SEND_RECV\n"); @@ -132,27 +132,27 @@ DT_Transaction_Cmd_Parse_Op (Transaction DT_Mdep_printf ("OP too many args \n"); goto error_return; } - return true; + return TRUE; error_return: - return false; + return FALSE; } -static bool +static boolean_t DT_Transaction_Cmd_Validate (Transaction_Cmd_t * cmd) { unsigned int i; - bool has_server_send; - bool has_client_send; + boolean_t has_server_send; + boolean_t has_client_send; unsigned int reap_count; - has_server_send = false; - has_client_send = false; + has_server_send = FALSE; + has_client_send = FALSE; reap_count = 0; if (cmd->server_name[0] == '\0') { DT_Mdep_printf ("Must specify server_name in command line or scriptfile\n"); - return (false); + return (FALSE); } for (i = 0; i < cmd->num_ops; i++) { @@ -162,11 +162,11 @@ DT_Transaction_Cmd_Validate (Transaction { if (cmd->op[i].server_initiated) { - has_server_send = true; + has_server_send = TRUE; } else { - has_client_send = true; + has_client_send = TRUE; } if (cmd->op[i].reap_send_on_recv) { @@ -185,7 +185,7 @@ DT_Transaction_Cmd_Validate (Transaction else { DT_Mdep_printf ("OP: Unbalanced -f options\n"); - return false; + return FALSE; } } } @@ -206,18 +206,18 @@ DT_Transaction_Cmd_Validate (Transaction { DT_Mdep_printf ("Error: Transaction test requires \n"); DT_Mdep_printf ("Error: At least one server SR and one client SR Operation\n"); - return false; + return FALSE; } if (reap_count != 0) { DT_Mdep_printf ("OP: Unbalanced -f options\n"); - return false; + return FALSE; } if (cmd->debug) { DT_Transaction_Cmd_Print (cmd); } - return true; + return TRUE; } @@ -275,15 +275,15 @@ DT_Transaction_Cmd_Init (Transaction_Cmd cmd->num_iterations = 1000; cmd->num_threads = 1; cmd->eps_per_thread = 1; - cmd->debug = false; - cmd->validate = false; + cmd->debug = FALSE; + cmd->validate = FALSE; cmd->ReliabilityLevel = DAT_QOS_BEST_EFFORT; - cmd->poll = false; + cmd->poll = FALSE; cmd->DT_mem_type = DAT_MEM_TYPE_PHYSICAL; } -bool +boolean_t DT_Transaction_Cmd_Parse (Transaction_Cmd_t * cmd, int my_argc, char **my_argv, @@ -309,7 +309,7 @@ DT_Transaction_Cmd_Parse (Transaction_Cm { DT_Mdep_printf ("must specify server name\n"); DT_Transaction_Cmd_Usage (); - return (false); + return (FALSE); } strcpy (cmd->server_name, opts->optarg); break; @@ -327,7 +327,7 @@ DT_Transaction_Cmd_Parse (Transaction_Cm { DT_Mdep_printf ("Syntax Error -i option\n"); DT_Transaction_Cmd_Usage (); - return (false); + return (FALSE); } cmd->num_iterations = atol (opts->optarg); @@ -340,7 +340,7 @@ DT_Transaction_Cmd_Parse (Transaction_Cm { DT_Mdep_printf ("Syntax Error -t option\n"); DT_Transaction_Cmd_Usage (); - return (false); + return (FALSE); } cmd->num_threads = atol (opts->optarg); break; @@ -352,7 +352,7 @@ DT_Transaction_Cmd_Parse (Transaction_Cm { DT_Mdep_printf ("Syntax Error -w option\n"); DT_Transaction_Cmd_Usage (); - return (false); + return (FALSE); } cmd->eps_per_thread = atol (opts->optarg); break; @@ -360,22 +360,22 @@ DT_Transaction_Cmd_Parse (Transaction_Cm case 'd': /* print debug messages */ { DT_dapltest_debug++; - cmd->debug = true; + cmd->debug = TRUE; break; } case 'r': /* use RSP instead of PSP */ { - cmd->use_rsp = true; + cmd->use_rsp = TRUE; break; } case 'V': /* validate data being sent/received */ { - cmd->validate = true; + cmd->validate = TRUE; break; } case 'P': /* use completion polling */ { - cmd->poll = true; + cmd->poll = TRUE; break; } case 'R': /* Service Reliability Level */ @@ -393,7 +393,7 @@ DT_Transaction_Cmd_Parse (Transaction_Cm { DT_Mdep_printf ("Invalid Transaction Test Parameter: %c\n", c); DT_Transaction_Cmd_Usage (); - return (false); + return (FALSE); } } } @@ -403,7 +403,7 @@ DT_Transaction_Cmd_Parse (Transaction_Cm { DT_Mdep_printf ("can't get default device name\n"); DT_Transaction_Cmd_Usage (); - return (false); + return (FALSE); } } /* @@ -428,7 +428,7 @@ DT_Transaction_Cmd_Parse (Transaction_Cm if (!DT_Transaction_Cmd_Parse_Op (cmd, op)) { DT_Transaction_Cmd_Usage (); - return (false); + return (FALSE); } } @@ -463,9 +463,9 @@ DT_Transaction_Cmd_Parse (Transaction_Cm if (!DT_Transaction_Cmd_Validate (cmd)) { DT_Transaction_Cmd_Usage (); - return (false); + return (FALSE); } - return (true); + return (TRUE); } void Index: linux-kernel/test/dapltest/cmd/dapl_server_cmd.c =================================================================== --- linux-kernel/test/dapltest/cmd/dapl_server_cmd.c (revision 2306) +++ linux-kernel/test/dapltest/cmd/dapl_server_cmd.c (working copy) @@ -31,14 +31,14 @@ void DT_Server_Cmd_Init (Server_Cmd_t * Server_Cmd) { DT_dapltest_debug = 0; - Server_Cmd->debug = false; + Server_Cmd->debug = FALSE; Server_Cmd->dapl_name[0] = '\0'; Server_Cmd->ReliabilityLevel = DAT_QOS_BEST_EFFORT; Server_Cmd->DT_mem_type = DAT_MEM_TYPE_PHYSICAL; } -bool +boolean_t DT_Server_Cmd_Parse (Server_Cmd_t * Server_Cmd, int my_argc, char **my_argv, @@ -62,7 +62,7 @@ DT_Server_Cmd_Parse (Server_Cmd_t * Serv case 'd': { DT_dapltest_debug++; - Server_Cmd->debug = true; + Server_Cmd->debug = TRUE; break; } case 'R': /* Service Reliability Level */ @@ -70,7 +70,7 @@ DT_Server_Cmd_Parse (Server_Cmd_t * Serv Server_Cmd->ReliabilityLevel = DT_ParseQoS (opts->optarg); if (0 == Server_Cmd->ReliabilityLevel) { - return (false); + return (FALSE); } break; } @@ -84,7 +84,7 @@ DT_Server_Cmd_Parse (Server_Cmd_t * Serv { DT_Mdep_printf ("Invalid Server option: %c\n", opts->optopt); DT_Server_Cmd_Usage (); - return (false); + return (FALSE); } } } @@ -94,10 +94,10 @@ DT_Server_Cmd_Parse (Server_Cmd_t * Serv { DT_Mdep_printf ("can't get default device name\n"); DT_Server_Cmd_Usage (); - return (false); + return (FALSE); } } - return (true); + return (TRUE); } void Index: linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.c =================================================================== --- linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.c (revision 2306) +++ linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.c (working copy) @@ -92,11 +92,11 @@ DT_Mdep_End (void) * Generate name of IB device */ -bool +boolean_t DT_Mdep_GetDefaultDeviceName (char *dapl_name) { strcpy (dapl_name, DT_MdepDeviceName); - return true; + return TRUE; } /* @@ -121,7 +121,7 @@ DT_Mdep_Schedule (void) * Get system statistics including uptime and idle time */ -bool +boolean_t DT_Mdep_GetCpuStat ( DT_CpuStat *cpu_stat ) { @@ -164,7 +164,7 @@ DT_Mdep_GetCpuStat ( rewind (Stat_Fp); - return true; + return TRUE; } /* @@ -302,10 +302,10 @@ DT_Mdep_Free (void *a_) * * Lock object constructor */ -bool +boolean_t DT_Mdep_LockInit (DT_Mdep_LockType * lock_ptr) { - return pthread_mutex_init (lock_ptr, 0) ? false : true; + return pthread_mutex_init (lock_ptr, 0) ? FALSE : TRUE; } /* @@ -361,7 +361,7 @@ DT_Mdep_Thread_Destroy_Attributes (Threa /* * Start the thread */ -bool +boolean_t DT_Mdep_Thread_Start (Thread * thread_ptr) { return pthread_create (&thread_ptr->thread_handle, @@ -505,7 +505,7 @@ DT_Mdep_wait_object_wait ( /* Reset the signaled status if we were woken up. */ if (pthread_status == 0) { - wait_obj->signaled = false; + wait_obj->signaled = FALSE; } pthread_mutex_unlock (&wait_obj->lock); } @@ -520,7 +520,7 @@ DT_Mdep_wait_object_wait ( /* Reset the signaled status if we were woken up. */ if (pthread_status == 0) { - wait_obj->signaled = false; + wait_obj->signaled = FALSE; } pthread_mutex_unlock (&wait_obj->lock); } @@ -555,7 +555,7 @@ DT_Mdep_wait_object_wakeup ( DT_WAIT_OBJECT *wait_obj ) { pthread_mutex_lock ( &wait_obj->lock ); - wait_obj->signaled = true; + wait_obj->signaled = TRUE; pthread_mutex_unlock ( &wait_obj->lock ); if ( 0 != pthread_cond_signal ( &wait_obj->cv ) ) { Index: linux-kernel/test/dapltest/mdep/linux/dapl_mdep_kernel.c =================================================================== --- linux-kernel/test/dapltest/mdep/linux/dapl_mdep_kernel.c (revision 2306) +++ linux-kernel/test/dapltest/mdep/linux/dapl_mdep_kernel.c (working copy) @@ -57,7 +57,7 @@ DT_Mdep_Schedule (void) * Get system statistics including uptime and idle time */ -bool +boolean_t DT_Mdep_GetCpuStat ( DT_CpuStat *cpu_stat ) { @@ -98,7 +98,7 @@ DT_Mdep_GetCpuStat ( } #endif cpu_stat->idle = (unsigned long)(jif * num_of_cpus - (cpu_stat->user + nice + cpu_stat->system)); - return true; + return TRUE; } /* @@ -161,11 +161,11 @@ DT_Mdep_Free (void *a_) * * Lock object constructor */ -bool +boolean_t DT_Mdep_LockInit (DT_Mdep_LockType * lock_ptr) { spin_lock_init (lock_ptr); - return true; + return TRUE; } /* @@ -217,7 +217,7 @@ DT_Mdep_Thread_Destroy_Attributes (Threa /* * Start the thread */ -bool +boolean_t DT_Mdep_Thread_Start (Thread * thread_ptr) { thread_ptr->thread_handle = kernel_thread ( Index: linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.h =================================================================== --- linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.h (revision 2306) +++ linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.h (working copy) @@ -54,12 +54,6 @@ /* Default Device Name */ #define DT_MdepDeviceName "ia0a" -/* Boolean */ -typedef int bool; - -#define true (1) -#define false (0) - #ifndef __BASE_FILE__ #define __BASE_FILE__ __FILE__ #endif @@ -78,7 +72,7 @@ typedef pthread_mutex_t DT_Mdep_LockType typedef struct { - bool signaled; + boolean_t signaled; pthread_cond_t cv; pthread_mutex_t lock; } DT_WAIT_OBJECT; Index: linux-kernel/test/dapltest/mdep/linux/dapl_mdep_kernel.h =================================================================== --- linux-kernel/test/dapltest/mdep/linux/dapl_mdep_kernel.h (revision 2306) +++ linux-kernel/test/dapltest/mdep/linux/dapl_mdep_kernel.h (working copy) @@ -46,12 +46,6 @@ /* Default Device Name */ #define DT_MdepDeviceName "ia0a" -/* Boolean */ -typedef int bool; - -#define true (1) -#define false (0) - #ifndef __BASE_FILE__ #define __BASE_FILE__ __FILE__ #endif Index: linux-kernel/test/dapltest/common/dapl_global.c =================================================================== --- linux-kernel/test/dapltest/common/dapl_global.c (revision 2306) +++ linux-kernel/test/dapltest/common/dapl_global.c (working copy) @@ -27,7 +27,7 @@ #include "dapl_proto.h" -bool DT_local_is_little_endian; +boolean_t DT_local_is_little_endian; DAT_COUNT DT_dapltest_debug = 0; Index: linux-kernel/test/dapltest/common/dapl_quit_cmd_util.c =================================================================== --- linux-kernel/test/dapltest/common/dapl_quit_cmd_util.c (revision 2306) +++ linux-kernel/test/dapltest/common/dapl_quit_cmd_util.c (working copy) @@ -30,7 +30,7 @@ /*--------------------------------------------------------- */ void DT_Quit_Cmd_Endian (Quit_Cmd_t * cmd, - bool to_wire) + boolean_t to_wire) { /* do nothing */ } Index: linux-kernel/test/dapltest/common/dapl_transaction_cmd_util.c =================================================================== --- linux-kernel/test/dapltest/common/dapl_transaction_cmd_util.c (revision 2306) +++ linux-kernel/test/dapltest/common/dapl_transaction_cmd_util.c (working copy) @@ -29,7 +29,7 @@ void -DT_Transaction_Cmd_Endian (Transaction_Cmd_t * cmd, bool to_wire) +DT_Transaction_Cmd_Endian (Transaction_Cmd_t * cmd, boolean_t to_wire) { unsigned int i; From tduffy at sun.com Wed May 11 10:32:42 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 11 May 2005 10:32:42 -0700 Subject: [openib-general] Re: kdapltest works again!, still having slab corruption In-Reply-To: <1115650013.4490.16.camel@localhost.localdomain> References: <1115411060.4476.346.camel@localhost.localdomain> <1115413920.4476.1.camel@localhost.localdomain> <1115425669.19661.14.camel@duffman> <1115445245.4733.26.camel@localhost.localdomain> <1115481438.5263.4.camel@duffman> <1115650013.4490.16.camel@localhost.localdomain> Message-ID: <1115832762.11344.8.camel@duffman> On Mon, 2005-05-09 at 10:46 -0400, Hal Rosenstock wrote: > On Mon, 2005-05-09 at 10:39, James Lentini wrote: > > The slab corruption message below seems to involve IBAT's > > req_comp_work function. Any ideas? > > I looked at this over the weekend and still don't have a working theory. > req_comp_work calls req_free which based on the pend->type frees the (in > this case) path request (calls free_path_req). It is the same as the one > which was allocated from the path request cache in > ib_at_paths_by_route. We are still looking at this. Can you reproduce this on your end? I am still seeing it every time I run the kdapltest program on x86_64 with slab debugging turned on. With the latest bits (v2305). -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From halr at voltaire.com Wed May 11 10:35:17 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 11 May 2005 13:35:17 -0400 Subject: [openib-general] Re: kdapltest works again!, still having slab corruption In-Reply-To: <1115832762.11344.8.camel@duffman> References: <1115411060.4476.346.camel@localhost.localdomain> <1115413920.4476.1.camel@localhost.localdomain> <1115425669.19661.14.camel@duffman> <1115445245.4733.26.camel@localhost.localdomain> <1115481438.5263.4.camel@duffman> <1115650013.4490.16.camel@localhost.localdomain> <1115832762.11344.8.camel@duffman> Message-ID: <1115832917.4490.128.camel@localhost.localdomain> On Wed, 2005-05-11 at 13:32, Tom Duffy wrote: > Can you reproduce this on your end? I am still seeing it every time I > run the kdapltest program on x86_64 with slab debugging turned on. With > the latest bits (v2305). Yes, I can reproduce it and did spend some time looking at it but do not have a workable theory on this right now. -- Hal From sean.hefty at intel.com Wed May 11 12:28:47 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Wed, 11 May 2005 12:28:47 -0700 Subject: [openib-general] [PATCH] [MAD] expand retry ability to all MADs Message-ID: The following patch expands the automatic retry ability in the MAD layer to be usable by all clients, not just RMPP. Retries are done using a linear timeout. Signed-off-by: Sean Hefty Index: core/user_mad.c =================================================================== --- core/user_mad.c (revision 2306) +++ core/user_mad.c (working copy) @@ -322,6 +322,7 @@ static ssize_t ib_umad_write(struct file wr.wr.ud.remote_qpn = be32_to_cpu(packet->mad.qpn); wr.wr.ud.remote_qkey = be32_to_cpu(packet->mad.qkey); wr.wr.ud.timeout_ms = packet->mad.timeout_ms; + wr.wr.ud.retries = 0; wr.wr_id = (unsigned long) packet; Index: core/mad_rmpp.c =================================================================== --- core/mad_rmpp.c (revision 2306) +++ core/mad_rmpp.c (working copy) @@ -594,11 +594,9 @@ static int send_next_seg(struct ib_mad_s mad_send_wr->pad); } - /* 5 seconds until we can find the packet lifetime */ + /* 5 seconds for an ACK until we can find the packet lifetime */ timeout = mad_send_wr->send_wr.wr.ud.timeout_ms; - if (timeout && timeout < 5000) - mad_send_wr->timeout = msecs_to_jiffies(timeout); - else + if (!timeout || timeout > 5000) mad_send_wr->timeout = msecs_to_jiffies(5000); mad_send_wr->seg_num++; return ib_send_mad(mad_send_wr); @@ -734,7 +732,6 @@ int ib_send_rmpp_mad(struct ib_mad_send_ (sizeof(struct ib_rmpp_mad) - mad_send_wr->data_offset); mad_send_wr->pad = total_len - offsetof(struct ib_rmpp_mad, data) - be32_to_cpu(rmpp_mad->rmpp_hdr.paylen_newwin); - mad_send_wr->retries = mad_send_wr->send_wr.wr.ud.retries; /* We need to wait for the final ACK even if there isn't a response */ mad_send_wr->refcount += (mad_send_wr->timeout == 0); @@ -787,7 +784,7 @@ int ib_process_rmpp_send_wc(struct ib_ma return IB_RMPP_RESULT_CONSUMED; } -int ib_timeout_rmpp(struct ib_mad_send_wr_private *mad_send_wr) +int ib_retry_rmpp(struct ib_mad_send_wr_private *mad_send_wr) { struct ib_rmpp_mad *rmpp_mad; int ret; @@ -797,8 +794,7 @@ int ib_timeout_rmpp(struct ib_mad_send_w IB_MGMT_RMPP_FLAG_ACTIVE)) return IB_RMPP_RESULT_UNHANDLED; /* RMPP not active */ - if (mad_send_wr->last_ack == mad_send_wr->total_seg || - !mad_send_wr->retries--) + if (mad_send_wr->last_ack == mad_send_wr->total_seg) return IB_RMPP_RESULT_PROCESSED; mad_send_wr->seg_num = mad_send_wr->last_ack + 1; @@ -806,9 +802,5 @@ int ib_timeout_rmpp(struct ib_mad_send_w if (ret) return IB_RMPP_RESULT_PROCESSED; - mad_send_wr->refcount++; - list_del(&mad_send_wr->agent_list); - list_add_tail(&mad_send_wr->agent_list, - &mad_send_wr->mad_agent_priv->send_list); return IB_RMPP_RESULT_CONSUMED; } Index: core/mad.c =================================================================== --- core/mad.c (revision 2306) +++ core/mad.c (working copy) @@ -971,6 +971,7 @@ int ib_post_send_mad(struct ib_mad_agent /* Timeout will be updated after send completes */ mad_send_wr->timeout = msecs_to_jiffies(send_wr->wr. ud.timeout_ms); + mad_send_wr->retries = mad_send_wr->send_wr.wr.ud.retries; /* One reference for each work request to QP + response */ mad_send_wr->refcount = 1 + (mad_send_wr->timeout > 0); mad_send_wr->status = IB_WC_SUCCESS; @@ -2233,6 +2234,41 @@ local_send_completion: spin_unlock_irqrestore(&mad_agent_priv->lock, flags); } +static int retry_send(struct ib_mad_send_wr_private *mad_send_wr) +{ + int ret; + + if (!mad_send_wr->retries--) + return -ETIMEDOUT; + + mad_send_wr->timeout = msecs_to_jiffies(mad_send_wr->send_wr. + wr.ud.timeout_ms); + + if (mad_send_wr->mad_agent_priv->agent.rmpp_version) { + ret = ib_retry_rmpp(mad_send_wr); + switch (ret) { + case IB_RMPP_RESULT_UNHANDLED: + ret = ib_send_mad(mad_send_wr); + break; + case IB_RMPP_RESULT_CONSUMED: + ret = 0; + break; + default: + ret = -ECOMM; + break; + } + } else + ret = ib_send_mad(mad_send_wr); + + if (!ret) { + mad_send_wr->refcount++; + list_del(&mad_send_wr->agent_list); + list_add_tail(&mad_send_wr->agent_list, + &mad_send_wr->mad_agent_priv->send_list); + } + return ret; +} + static void timeout_sends(void *data) { struct ib_mad_agent_private *mad_agent_priv; @@ -2261,9 +2297,8 @@ static void timeout_sends(void *data) break; } - if (mad_agent_priv->agent.rmpp_version && - ib_timeout_rmpp(mad_send_wr) == IB_RMPP_RESULT_CONSUMED) - continue; + if (!retry_send(mad_send_wr)) + continue; list_del(&mad_send_wr->agent_list); spin_unlock_irqrestore(&mad_agent_priv->lock, flags); Index: core/mad_rmpp.h =================================================================== --- core/mad_rmpp.h (revision 2306) +++ core/mad_rmpp.h (working copy) @@ -55,6 +55,6 @@ int ib_process_rmpp_send_wc(struct ib_ma void ib_cancel_rmpp_recvs(struct ib_mad_agent_private *agent); -int ib_timeout_rmpp(struct ib_mad_send_wr_private *mad_send_wr); +int ib_retry_rmpp(struct ib_mad_send_wr_private *mad_send_wr); #endif /* __MAD_RMPP_H__ */ Index: core/mad_priv.h =================================================================== --- core/mad_priv.h (revision 2306) +++ core/mad_priv.h (working copy) @@ -123,6 +123,7 @@ struct ib_mad_send_wr_private { u64 wr_id; /* client WR ID */ u64 tid; unsigned long timeout; + int retries; int retry; int refcount; enum ib_wc_status status; @@ -134,7 +135,6 @@ struct ib_mad_send_wr_private { int total_seg; int data_offset; int pad; - int retries; }; struct ib_mad_local_private { Index: core/sa_query.c =================================================================== --- core/sa_query.c (revision 2306) +++ core/sa_query.c (working copy) @@ -442,7 +442,8 @@ static int send_mad(struct ib_sa_query * .mad_hdr = &query->mad->mad_hdr, .remote_qpn = 1, .remote_qkey = IB_QP1_QKEY, - .timeout_ms = timeout_ms + .timeout_ms = timeout_ms, + .retries = 0 } } }; From bjordan.ics at gmail.com Wed May 11 13:12:41 2005 From: bjordan.ics at gmail.com (William Jordan) Date: Wed, 11 May 2005 16:12:41 -0400 Subject: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation In-Reply-To: References: <200544159.Ahk9l0puXy39U6u6@topspin.com> <20050411171347.7e05859f.akpm@osdl.org> <20050412180447.E6958@topspin.com> <20050425203110.A9729@topspin.com> <4279142A.8050501@ammasso.com> <427A6A7E.8000604@ammasso.com> <427BF8E1.2080006@ammasso.com> <427CD49E.6080300@ammasso.com> Message-ID: <78d18e2050511131246075b37@mail.gmail.com> On 5/7/05, Hugh Dickins wrote: > > My understanding is that mlock() could in theory allow the page to be moved, > > but that currently nothing in the kernel would actually move it. However, > > that could change in the future to allow hot-swapping of RAM. > > That's my understanding too, that nothing currently does so. Aside from > hot-swapping RAM, there's also a need to be able to migrate pages around > RAM, either to unfragment memory allowing higher-order allocations to > succeed more often, or to get around extreme dmamem/normal-mem/highmem > imbalances without dedicating huge reserves. Those would more often > succeed if uninhibited by mlock. Hugh, If I am reading you correctly, you are saying that mlock currently prevents pages from migrating around to unfragment memory, but get_user_pages does not prevent this? If this is the case, this could very easily be the problem Timur was experiencing. He is using get_user_pages to lock pages long term (for the life of the process, beyond the bounds of a single system call). If it is possible for a page to be migrated in physical memory during extreme virtual memory pressure while the reference count is held with get_user_pages, that would cause the problem where the hardware is no longer mapped to the same page as the application. BTW: In earlier kernels, I experienced the same issues in our IB drivers when trying to pin pages using only get_user_pages. -- Bill Jordan InfiniCon Systems From hugh at veritas.com Wed May 11 13:42:24 2005 From: hugh at veritas.com (Hugh Dickins) Date: Wed, 11 May 2005 21:42:24 +0100 (BST) Subject: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation In-Reply-To: <78d18e2050511131246075b37@mail.gmail.com> References: <200544159.Ahk9l0puXy39U6u6@topspin.com> <20050411171347.7e05859f.akpm@osdl.org> <20050412180447.E6958@topspin.com> <20050425203110.A9729@topspin.com> <4279142A.8050501@ammasso.com> <427A6A7E.8000604@ammasso.com> <427BF8E1.2080006@ammasso.com> <427CD49E.6080300@ammasso.com> <78d18e2050511131246075b37@mail.gmail.com> Message-ID: On Wed, 11 May 2005, William Jordan wrote: > On 5/7/05, Hugh Dickins wrote: > > > My understanding is that mlock() could in theory allow the page to be moved, > > > but that currently nothing in the kernel would actually move it. However, > > > that could change in the future to allow hot-swapping of RAM. > > > > That's my understanding too, that nothing currently does so. Aside from > > hot-swapping RAM, there's also a need to be able to migrate pages around > > RAM, either to unfragment memory allowing higher-order allocations to > > succeed more often, or to get around extreme dmamem/normal-mem/highmem > > imbalances without dedicating huge reserves. Those would more often > > succeed if uninhibited by mlock. > > If I am reading you correctly, you are saying that mlock currently > prevents pages from migrating around to unfragment memory, but > get_user_pages does not prevent this? No, not what I meant at all. I'm saying that currently (aside from proposed patches) there is no such migration of pages; that we'd prefer to implement migration in such a way that mlock does not inhibit it (though there might prove to be strong arguments defeating that); and that get_user_pages _must_ prevent migration (and if there were already such migration, I'd be saying it _does_ prevent it). Hugh From halr at voltaire.com Wed May 11 14:07:33 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 11 May 2005 17:07:33 -0400 Subject: [openib-general] [RFC] [PATCH] user_mad: Support RMPP on send side Message-ID: <1115845653.4490.188.camel@localhost.localdomain> user_mad: Support RMPP on send side Note that this change will need a coordinated change to OpenSM and some userspace/management libraries which will be done as soon as possible after this patch is accepted. Receive side support for RMPP will be added separately. A couple of notes on this patch: 1. send side copying There are currently 2 copies from user space done even in the non RMPP case. It would be nice to reduce that to one in the non RMPP case. This is an optimization to add at some point. 2. Also, I'm not sure about the effects of receiving RMPP on the read side. -- Hal Index: infiniband/include/ib_user_mad.h =================================================================== --- infiniband/include/ib_user_mad.h (revision 2265) +++ infiniband/include/ib_user_mad.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Voltaire, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -42,7 +43,7 @@ * Increment this value if any changes that break userspace ABI * compatibility are made. */ -#define IB_USER_MAD_ABI_VERSION 2 +#define IB_USER_MAD_ABI_VERSION 3 /* * Make sure that all structs defined in this file remain laid out so @@ -51,8 +52,7 @@ */ /** - * ib_user_mad - MAD packet - * @data - Contents of MAD + * ib_user_mad_hdr - MAD packet header * @id - ID of agent MAD received with/to be sent with * @status - 0 on successful receive, ETIMEDOUT if no response * received (transaction ID in data[] will be set to TID of original @@ -72,8 +72,7 @@ * * All multi-byte quantities are stored in network (big endian) byte order. */ -struct ib_user_mad { - __u8 data[256]; +struct ib_user_mad_hdr { __u32 id; __u32 status; __u32 timeout_ms; @@ -91,6 +90,17 @@ }; /** + * ib_user_mad - MAD packet + * @hdr - MAD packet header + * @data - Contents of MAD + * + */ +struct ib_user_mad { + struct ib_user_mad_hdr hdr; + __u8 data[0]; +}; + +/** * ib_user_mad_reg_req - MAD registration request * @id - Set by the kernel; used to identify agent in future requests. * @qpn - Queue pair number; must be 0 or 1. @@ -103,6 +113,8 @@ * management class to receive. * @oui: Indicates IEEE OUI when mgmt_class is a vendor class * in the range from 0x30 to 0x4f. Otherwise not used. + * @rmpp_version: If set, indicates the RMPP version used. + * */ struct ib_user_mad_reg_req { __u32 id; @@ -111,6 +123,7 @@ __u8 mgmt_class; __u8 mgmt_class_version; __u8 oui[3]; + __u8 rmpp_version; }; #define IB_IOCTL_MAGIC 0x1b Index: infiniband/core/user_mad.c =================================================================== --- infiniband/core/user_mad.c (revision 2265) +++ infiniband/core/user_mad.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Voltaire, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -94,10 +95,12 @@ }; struct ib_umad_packet { - struct ib_user_mad mad; struct ib_ah *ah; + struct ib_mad_send_buf *msg; struct list_head list; + int length; DECLARE_PCI_UNMAP_ADDR(mapping) + struct ib_user_mad mad; }; static const dev_t base_dev = MKDEV(IB_UMAD_MAJOR, IB_UMAD_MINOR_BASE); @@ -114,10 +117,10 @@ int ret = 1; down_read(&file->agent_mutex); - for (packet->mad.id = 0; - packet->mad.id < IB_UMAD_MAX_AGENTS; - packet->mad.id++) - if (agent == file->agent[packet->mad.id]) { + for (packet->mad.hdr.id = 0; + packet->mad.hdr.id < IB_UMAD_MAX_AGENTS; + packet->mad.hdr.id++) + if (agent == file->agent[packet->mad.hdr.id]) { spin_lock_irq(&file->recv_lock); list_add_tail(&packet->list, &file->recv_list); spin_unlock_irq(&file->recv_lock); @@ -138,14 +141,11 @@ struct ib_umad_packet *packet = (void *) (unsigned long) send_wc->wr_id; - dma_unmap_single(agent->device->dma_device, - pci_unmap_addr(packet, mapping), - sizeof packet->mad.data, - DMA_TO_DEVICE); - ib_destroy_ah(packet->ah); + ib_free_send_mad(packet->msg); + ib_destroy_ah(packet->msg->send_wr.wr.ud.ah); if (send_wc->status == IB_WC_RESP_TIMEOUT_ERR) { - packet->mad.status = ETIMEDOUT; + packet->mad.hdr.status = ETIMEDOUT; if (!queue_packet(file, agent, packet)) return; @@ -159,30 +159,34 @@ { struct ib_umad_file *file = agent->context; struct ib_umad_packet *packet; + int length; + if (mad_recv_wc->wc->status != IB_WC_SUCCESS) goto out; - packet = kmalloc(sizeof *packet, GFP_KERNEL); + length = 256; /* until RMPP is supported */ + packet = kmalloc(sizeof *packet + length, GFP_KERNEL); if (!packet) goto out; - memset(packet, 0, sizeof *packet); + memset(packet, 0, sizeof *packet + length); + packet->length = length; - memcpy(packet->mad.data, mad_recv_wc->recv_buf.mad, sizeof packet->mad.data); - packet->mad.status = 0; - packet->mad.qpn = cpu_to_be32(mad_recv_wc->wc->src_qp); - packet->mad.lid = cpu_to_be16(mad_recv_wc->wc->slid); - packet->mad.sl = mad_recv_wc->wc->sl; - packet->mad.path_bits = mad_recv_wc->wc->dlid_path_bits; - packet->mad.grh_present = !!(mad_recv_wc->wc->wc_flags & IB_WC_GRH); - if (packet->mad.grh_present) { + memcpy(packet->mad.data, mad_recv_wc->recv_buf.mad, length); + packet->mad.hdr.status = 0; + packet->mad.hdr.qpn = cpu_to_be32(mad_recv_wc->wc->src_qp); + packet->mad.hdr.lid = cpu_to_be16(mad_recv_wc->wc->slid); + packet->mad.hdr.sl = mad_recv_wc->wc->sl; + packet->mad.hdr.path_bits = mad_recv_wc->wc->dlid_path_bits; + packet->mad.hdr.grh_present = !!(mad_recv_wc->wc->wc_flags & IB_WC_GRH); + if (packet->mad.hdr.grh_present) { /* XXX parse GRH */ - packet->mad.gid_index = 0; - packet->mad.hop_limit = 0; - packet->mad.traffic_class = 0; - memset(packet->mad.gid, 0, 16); - packet->mad.flow_label = 0; + packet->mad.hdr.gid_index = 0; + packet->mad.hdr.hop_limit = 0; + packet->mad.hdr.traffic_class = 0; + memset(packet->mad.hdr.gid, 0, 16); + packet->mad.hdr.flow_label = 0; } if (queue_packet(file, agent, packet)) @@ -199,7 +203,7 @@ struct ib_umad_packet *packet; ssize_t ret; - if (count < sizeof (struct ib_user_mad)) + if (count < sizeof (struct ib_user_mad) + 256) /* until RMPP supported */ return -EINVAL; spin_lock_irq(&file->recv_lock); @@ -222,10 +226,12 @@ spin_unlock_irq(&file->recv_lock); - if (copy_to_user(buf, &packet->mad, sizeof packet->mad)) + if (copy_to_user(buf, &packet->mad, + min(count, packet->length + + sizeof (struct ib_user_mad)))) ret = -EFAULT; else - ret = sizeof packet->mad; + ret = count; kfree(packet); return ret; @@ -238,106 +244,155 @@ struct ib_umad_packet *packet; struct ib_mad_agent *agent; struct ib_ah_attr ah_attr; - struct ib_sge gather_list; - struct ib_send_wr *bad_wr, wr = { - .opcode = IB_WR_SEND, - .sg_list = &gather_list, - .num_sge = 1, - .send_flags = IB_SEND_SIGNALED, - }; + struct ib_send_wr *bad_wr; + struct ib_rmpp_mad *rmpp_mad; u8 method; u64 *tid; - int ret; + int ret, length, hdr_len, data_len, rmpp_hdr_size; + int rmpp_active = 0; if (count < sizeof (struct ib_user_mad)) return -EINVAL; - packet = kmalloc(sizeof *packet, GFP_KERNEL); + length = count - sizeof (struct ib_user_mad); + packet = kmalloc(sizeof *packet + sizeof(struct ib_mad_hdr) + + sizeof(struct ib_rmpp_hdr), GFP_KERNEL); if (!packet) return -ENOMEM; - if (copy_from_user(&packet->mad, buf, sizeof packet->mad)) { - kfree(packet); - return -EFAULT; + if (copy_from_user(&packet->mad, buf, + sizeof (struct ib_user_mad) + + sizeof(struct ib_mad_hdr) + + sizeof(struct ib_rmpp_hdr))) { + ret = -EFAULT; + goto err; } - if (packet->mad.id < 0 || packet->mad.id >= IB_UMAD_MAX_AGENTS) { + if (packet->mad.hdr.id < 0 || + packet->mad.hdr.id >= IB_UMAD_MAX_AGENTS) { ret = -EINVAL; goto err; } + packet->length = length; + down_read(&file->agent_mutex); - agent = file->agent[packet->mad.id]; + agent = file->agent[packet->mad.hdr.id]; if (!agent) { ret = -EINVAL; goto err_up; } + memset(&ah_attr, 0, sizeof ah_attr); + ah_attr.dlid = be16_to_cpu(packet->mad.hdr.lid); + ah_attr.sl = packet->mad.hdr.sl; + ah_attr.src_path_bits = packet->mad.hdr.path_bits; + ah_attr.port_num = file->port->port_num; + if (packet->mad.hdr.grh_present) { + ah_attr.ah_flags = IB_AH_GRH; + memcpy(ah_attr.grh.dgid.raw, packet->mad.hdr.gid, 16); + ah_attr.grh.flow_label = packet->mad.hdr.flow_label; + ah_attr.grh.hop_limit = packet->mad.hdr.hop_limit; + ah_attr.grh.traffic_class = packet->mad.hdr.traffic_class; + } + + packet->ah = ib_create_ah(agent->qp->pd, &ah_attr); + if (IS_ERR(packet->ah)) { + ret = PTR_ERR(packet->ah); + goto err_up; + } + + rmpp_mad = (struct ib_rmpp_mad *) packet->mad.data; + if (ib_get_rmpp_flags(&rmpp_mad->rmpp_hdr) & IB_MGMT_RMPP_FLAG_ACTIVE) { + /* RMPP active */ + if (!agent->rmpp_version) { + ret = -EINVAL; + goto err_ah; + } + /* Validate that management class can support RMPP */ + if (rmpp_mad->mad_hdr.mgmt_class == IB_MGMT_CLASS_SUBN_ADM) { + hdr_len = offsetof(struct ib_sa_mad, data); + data_len = length; + } else if ((rmpp_mad->mad_hdr.mgmt_class >= IB_MGMT_CLASS_VENDOR_RANGE2_START) && + (rmpp_mad->mad_hdr.mgmt_class <= IB_MGMT_CLASS_VENDOR_RANGE2_END)) { + hdr_len = offsetof(struct ib_vendor_mad, data); + data_len = length - hdr_len; + } else { + ret = -EINVAL; + goto err_ah; + } + rmpp_active = 1; + } else { + if (length != sizeof(struct ib_mad)) { + ret = -EINVAL; + goto err_ah; + } + hdr_len = offsetof(struct ib_mad, data); + data_len = length - hdr_len; + } + + packet->msg = ib_create_send_mad(agent, + be32_to_cpu(packet->mad.hdr.qpn), + 0, packet->ah, rmpp_active, + hdr_len, data_len, + GFP_KERNEL); + if (IS_ERR(packet->msg)) { + ret = PTR_ERR(packet->msg); + goto err_ah; + } + + packet->msg->send_wr.wr.ud.retries = 3; /* !!! */ + packet->msg->send_wr.wr.ud.timeout_ms = packet->mad.hdr.timeout_ms; + /* Override send WR WRID created by ib_create_send_mad */ + packet->msg->send_wr.wr_id = (unsigned long) packet; + + if (!rmpp_active) { + /* Copy message from user into send buffer */ + copy_from_user(packet->msg->mad, + buf + sizeof(struct ib_user_mad), length); + } else { + rmpp_hdr_size = sizeof(struct ib_mad_hdr) + + sizeof(struct ib_rmpp_hdr); + /* Only copy MAD headers (RMPP header in place) */ + memcpy(packet->msg->mad, packet->mad.data, + sizeof(struct ib_mad_hdr)); + /* Now, copy rest of message from user into send buffer */ + copy_from_user(((struct ib_rmpp_mad *)packet->msg->mad)->data, + buf + sizeof(struct ib_user_mad) + rmpp_hdr_size, + length - rmpp_hdr_size); + } + /* * If userspace is generating a request that will generate a * response, we need to make sure the high-order part of the * transaction ID matches the agent being used to send the * MAD. */ - method = ((struct ib_mad_hdr *) packet->mad.data)->method; + method = packet->msg->mad->mad_hdr.method; if (!(method & IB_MGMT_METHOD_RESP) && method != IB_MGMT_METHOD_TRAP_REPRESS && method != IB_MGMT_METHOD_SEND) { - tid = &((struct ib_mad_hdr *) packet->mad.data)->tid; + tid = &packet->msg->mad->mad_hdr.tid; *tid = cpu_to_be64(((u64) agent->hi_tid) << 32 | (be64_to_cpup(tid) & 0xffffffff)); } - memset(&ah_attr, 0, sizeof ah_attr); - ah_attr.dlid = be16_to_cpu(packet->mad.lid); - ah_attr.sl = packet->mad.sl; - ah_attr.src_path_bits = packet->mad.path_bits; - ah_attr.port_num = file->port->port_num; - if (packet->mad.grh_present) { - ah_attr.ah_flags = IB_AH_GRH; - memcpy(ah_attr.grh.dgid.raw, packet->mad.gid, 16); - ah_attr.grh.flow_label = packet->mad.flow_label; - ah_attr.grh.hop_limit = packet->mad.hop_limit; - ah_attr.grh.traffic_class = packet->mad.traffic_class; - } + ret = ib_post_send_mad(agent, &packet->msg->send_wr, &bad_wr); + if (ret) + goto err_msg; - packet->ah = ib_create_ah(agent->qp->pd, &ah_attr); - if (IS_ERR(packet->ah)) { - ret = PTR_ERR(packet->ah); - goto err_up; - } + up_read(&file->agent_mutex); - gather_list.addr = dma_map_single(agent->device->dma_device, - packet->mad.data, - sizeof packet->mad.data, - DMA_TO_DEVICE); - gather_list.length = sizeof packet->mad.data; - gather_list.lkey = file->mr[packet->mad.id]->lkey; - pci_unmap_addr_set(packet, mapping, gather_list.addr); + return sizeof (struct ib_user_mad_hdr) + packet->length; - wr.wr.ud.mad_hdr = (struct ib_mad_hdr *) packet->mad.data; - wr.wr.ud.ah = packet->ah; - wr.wr.ud.remote_qpn = be32_to_cpu(packet->mad.qpn); - wr.wr.ud.remote_qkey = be32_to_cpu(packet->mad.qkey); - wr.wr.ud.timeout_ms = packet->mad.timeout_ms; +err_msg: + ib_free_send_mad(packet->msg); - wr.wr_id = (unsigned long) packet; +err_ah: + ib_destroy_ah(packet->ah); - ret = ib_post_send_mad(agent, &wr, &bad_wr); - if (ret) { - dma_unmap_single(agent->device->dma_device, - pci_unmap_addr(packet, mapping), - sizeof packet->mad.data, - DMA_TO_DEVICE); - goto err_up; - } - - up_read(&file->agent_mutex); - - return sizeof packet->mad; - err_up: up_read(&file->agent_mutex); @@ -399,7 +454,8 @@ agent = ib_register_mad_agent(file->port->ib_dev, file->port->port_num, ureq.qpn ? IB_QPT_GSI : IB_QPT_SMI, ureq.mgmt_class ? &req : NULL, - 0, send_handler, recv_handler, file); + ureq.rmpp_version, + send_handler, recv_handler, file); if (IS_ERR(agent)) { ret = PTR_ERR(agent); goto out; Index: docs/user_mad.txt =================================================================== --- docs/user_mad.txt (revision 2265) +++ docs/user_mad.txt (working copy) @@ -28,11 +28,13 @@ Receiving MADs - MADs are received using read(). The buffer passed to read() must be - large enough to hold at least one struct ib_user_mad. For example: + MADs are received using read(). The receive side does not currently + support RMPP so the buffer passed to read() must be at least one + struct ib_user_mad + 256 bytes. For example: - struct ib_user_mad mad; - ret = read(fd, &mad, sizeof mad); + struct ib_user_mad *mad; + mad = malloc(sizeof *mad + 256); + ret = read(fd, mad, sizeof *mad + 256); if (ret != sizeof mad) perror("read"); @@ -50,18 +52,21 @@ MADs are sent using write(). The agent ID for sending should be filled into the id field of the MAD, the destination LID should be - filled into the lid field, and so on. For example: + filled into the lid field, and so on. The send side does support + RMPP so arbitrary length MAD can be sent. For example: - struct ib_user_mad mad; + struct ib_user_mad *mad; - /* fill in mad.data */ + mad = malloc(sizeof *mad + mad_length); - mad.id = my_agent; /* req.id from agent registration */ - mad.lid = my_dest; /* in network byte order... */ + /* fill in mad->data */ + + mad->hdr.id = my_agent; /* req.id from agent registration */ + mad->hdr.lid = my_dest; /* in network byte order... */ /* etc. */ - ret = write(fd, &mad, sizeof mad); - if (ret != sizeof mad) + ret = write(fd, &mad, sizeof *mad + mad_length); + if (ret != sizeof *mad + mad_length) perror("write"); Setting IsSM Capability Bit From tduffy at sun.com Wed May 11 14:19:48 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 11 May 2005 14:19:48 -0700 Subject: [openib-general] [PATCH] kDAPL: remove typedef DAT_RETURN Message-ID: <1115846388.11344.25.camel@duffman> This patch is a biggie. It removes the type DAT_RETURN and uses just u32 instead. At some point, we should go through and make sure all the functions actually need a return type or if they can just use int with standard Linux negative error codes. Signed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat/api.c linux-kernel2/dat/api.c --- linux-kernel/dat/api.c 2005-05-10 10:19:45.647003000 -0700 +++ linux-kernel2/dat/api.c 2005-05-11 11:23:22.776964000 -0700 @@ -44,7 +44,7 @@ * Function: dat_strerror_major ***********************************************************************/ -static DAT_RETURN dat_strerror_major(DAT_RETURN value, const char **message) +static u32 dat_strerror_major(u32 value, const char **message) { switch (DAT_GET_TYPE(value)) { case DAT_SUCCESS: @@ -163,7 +163,7 @@ static DAT_RETURN dat_strerror_major(DAT /*********************************************************************** * Function: dat_strerror_minor ***********************************************************************/ -static DAT_RETURN dat_strerror_minor(DAT_RETURN value, const char **message) +static u32 dat_strerror_minor(u32 value, const char **message) { switch (DAT_GET_SUBTYPE(value)) { @@ -575,16 +575,13 @@ static DAT_RETURN dat_strerror_minor(DAT * Function: dat_ia_openv ***********************************************************************/ -DAT_RETURN dat_ia_openv(const DAT_NAME_PTR name, - DAT_COUNT async_event_qlen, - DAT_EVD_HANDLE *async_event_handle, - DAT_IA_HANDLE *ia_handle, - u32 dat_major, u32 dat_minor, - boolean_t thread_safety) +u32 dat_ia_openv(const DAT_NAME_PTR name, DAT_COUNT async_event_qlen, + DAT_EVD_HANDLE *async_event_handle, DAT_IA_HANDLE *ia_handle, + u32 dat_major, u32 dat_minor, boolean_t thread_safety) { DAT_IA_OPEN_FUNC ia_open_func; struct dat_provider_info info; - DAT_RETURN status; + u32 status; size_t len; dat_dbg_print(DAT_DBG_CLASS_CONSUMER_API, @@ -632,11 +629,11 @@ EXPORT_SYMBOL(dat_ia_openv); * Function: dat_ia_close ***********************************************************************/ -DAT_RETURN dat_ia_close(DAT_IA_HANDLE ia_handle, DAT_CLOSE_FLAGS flags) +u32 dat_ia_close(DAT_IA_HANDLE ia_handle, DAT_CLOSE_FLAGS flags) { struct dat_provider *provider; struct dat_provider_attr provider_attr; - DAT_RETURN status; + u32 status; const char *ia_name; dat_dbg_print(DAT_DBG_CLASS_CONSUMER_API, @@ -690,8 +687,8 @@ EXPORT_SYMBOL(dat_ia_close); * Function: dat_strerror ***********************************************************************/ -DAT_RETURN dat_strerror(DAT_RETURN value, const char **major_message, - const char **minor_message) +u32 dat_strerror(u32 value, const char **major_message, + const char **minor_message) { dat_dbg_print(DAT_DBG_CLASS_CONSUMER_API, @@ -700,9 +697,9 @@ DAT_RETURN dat_strerror(DAT_RETURN value /* * The DAT specification contains a note to implementers - * suggesting that the consumer's DAT_RETURN value be used + * suggesting that the consumer's return value be used * as an index into a table of text strings. However, - * the DAT_RETURN values are not consecutive. Therefore this + * the return values are not consecutive. Therefore this * implementation does not follow the suggested implementation. */ @@ -723,8 +720,8 @@ EXPORT_SYMBOL(dat_strerror); * Function: dat_registry_add_provider ***********************************************************************/ -DAT_RETURN dat_registry_add_provider(const struct dat_provider *provider, - const struct dat_provider_info *provider_info) +u32 dat_registry_add_provider(const struct dat_provider *provider, + const struct dat_provider_info *provider_info) { dat_dbg_print(DAT_DBG_CLASS_PROVIDER_API, "%s(ia:%s, ver:%x.%x, thrd_safe:%x)\n", @@ -751,8 +748,8 @@ EXPORT_SYMBOL(dat_registry_remove_provid * Function: dat_registry_remove_provider ***********************************************************************/ -DAT_RETURN dat_registry_remove_provider(const struct dat_provider *provider, - const struct dat_provider_info *provider_info) +u32 dat_registry_remove_provider(const struct dat_provider *provider, + const struct dat_provider_info *provider_info) { dat_dbg_print(DAT_DBG_CLASS_PROVIDER_API, "%s(ia:%s)\n", @@ -773,13 +770,12 @@ EXPORT_SYMBOL(dat_registry_add_provider) * Function: dat_registry_list_providers ***********************************************************************/ -DAT_RETURN dat_registry_list_providers(DAT_COUNT max_to_return, - DAT_COUNT *entries_returned, - struct dat_provider_info *(dat_provider_list[])) +u32 dat_registry_list_providers(DAT_COUNT max_to_return, + DAT_COUNT *entries_returned, + struct dat_provider_info *(dat_provider_list[])) { - DAT_RETURN status; + u32 status = DAT_SUCCESS; - status = DAT_SUCCESS; dat_dbg_print(DAT_DBG_CLASS_CONSUMER_API, "%s(max_to_return:%i, entries_rtrn:%p, prvdr_list:%p)\n", __func__, max_to_return, entries_returned, diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat/consumer.c linux-kernel2/dat/consumer.c --- linux-kernel/dat/consumer.c 2005-05-10 10:19:45.654000000 -0700 +++ linux-kernel2/dat/consumer.c 2005-05-11 11:22:58.743960000 -0700 @@ -47,7 +47,7 @@ * Function: dat_strerror_major ***********************************************************************/ -static DAT_RETURN dat_strerror_major(DAT_RETURN value, const char **message) +static u32 dat_strerror_major(u32 value, const char **message) { switch (DAT_GET_TYPE(value)) { case DAT_SUCCESS: @@ -166,7 +166,7 @@ static DAT_RETURN dat_strerror_major(DAT /*********************************************************************** * Function: dat_strerror_minor ***********************************************************************/ -static DAT_RETURN dat_strerror_minor(DAT_RETURN value, const char **message) +static u32 dat_strerror_minor(u32 value, const char **message) { switch (DAT_GET_SUBTYPE(value)) { @@ -574,7 +574,7 @@ static DAT_RETURN dat_strerror_minor(DAT * * *********************************************************************/ -DAT_RETURN +u32 dat_ia_openv(const DAT_NAME_PTR name, DAT_COUNT async_event_qlen, DAT_EVD_HANDLE * async_event_handle, @@ -584,7 +584,7 @@ dat_ia_openv(const DAT_NAME_PTR name, { DAT_IA_OPEN_FUNC ia_open_func; DAT_PROVIDER_INFO info; - DAT_RETURN status; + u32 status; size_t len; dat_dbg_print(DAT_DBG_CLASS_CONSUMER_API, @@ -627,11 +627,11 @@ dat_ia_openv(const DAT_NAME_PTR name, EXPORT_SYMBOL(dat_ia_openv); -DAT_RETURN dat_ia_close(DAT_IA_HANDLE ia_handle, DAT_CLOSE_FLAGS flags) +u32 dat_ia_close(DAT_IA_HANDLE ia_handle, DAT_CLOSE_FLAGS flags) { struct dat_provider *provider; struct dat_provider_attr provider_attr; - DAT_RETURN status; + u32 status; const char *ia_name; dat_dbg_print(DAT_DBG_CLASS_CONSUMER_API, @@ -681,9 +681,8 @@ DAT_RETURN dat_ia_close(DAT_IA_HANDLE ia EXPORT_SYMBOL(dat_ia_close); -DAT_RETURN -dat_strerror(DAT_RETURN value, - const char **major_message, const char **minor_message) +u32 +dat_strerror(u32 value, const char **major_message, const char **minor_message) { dat_dbg_print(DAT_DBG_CLASS_CONSUMER_API, @@ -692,9 +691,9 @@ dat_strerror(DAT_RETURN value, /* * The DAT specification contains a note to implementers - * suggesting that the consumer's DAT_RETURN value be used + * suggesting that the consumer's return value be used * as an index into a table of text strings. However, - * the DAT_RETURN values are not consecutive. Therefore this + * the return values are not consecutive. Therefore this * implementation does not follow the suggested implementation. */ diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat/dat.h linux-kernel2/dat/dat.h --- linux-kernel/dat/dat.h 2005-05-10 10:32:56.912966000 -0700 +++ linux-kernel2/dat/dat.h 2005-05-11 13:07:34.341014000 -0700 @@ -47,25 +47,23 @@ typedef u64 DAT_PADDR; /* * All return codes are actually a 3-way tuple: * - * type: DAT_RETURN_CLASS DAT_RETURN_TYPE DAT_RETURN_SUBTYPE + * type: dat_return_class dat_return_type dat_return_subtype * bits: 31-30 29-16 15-0 * * 3 2 1 * 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - * | C | DAT_RETURN_TYPE | DAT_RETURN_SUBTYPE | + * | C | dat_return_type | dat_return_subtype | * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ */ -typedef u32 DAT_RETURN; - -typedef enum dat_return_class { +enum dat_return_class { DAT_CLASS_ERROR = 0x80000000, DAT_CLASS_WARNING = 0x40000000, DAT_CLASS_SUCCESS = 0x00000000 -} DAT_RETURN_CLASS; +}; -typedef enum dat_return_type { +enum dat_return_type { /* The operation was successful. */ DAT_SUCCESS = 0x000000, @@ -144,9 +142,9 @@ typedef enum dat_return_type { /* Provider does not support the operation yet. */ DAT_NOT_IMPLEMENTED = 0x3FFF0000 -} DAT_STATUS_TYPE; +}; -typedef enum dat_return_subtype { +enum dat_return_subtype { /* First element is no subtype */ DAT_NO_SUBTYPE, /* ABORT sub types */ @@ -321,7 +319,7 @@ typedef enum dat_return_subtype { DAT_MAJOR_NOT_FOUND, DAT_MINOR_NOT_FOUND, DAT_THREAD_SAFETY_NOT_FOUND -} DAT_RETURN_SUBTYPE; +}; /* * Error bits @@ -336,8 +334,8 @@ typedef enum dat_return_subtype { #define DAT_IS_ERROR(status) ((u32)(status) & DAT_CLASS_ERROR) #define DAT_IS_WARNING(status) ((u32)(status) & DAT_CLASS_WARNING) -#define DAT_ERROR(Type, SubType) ((DAT_RETURN)(DAT_CLASS_ERROR | Type | SubType)) -#define DAT_WARNING(Type, SubType) ((DAT_RETURN)(DAT_CLASS_WARNING | Type | SubType)) +#define DAT_ERROR(type, sub_type) ((u32)(DAT_CLASS_ERROR | type | sub_type)) +#define DAT_WARNING(type, sub_type) ((u32)(DAT_CLASS_WARNING | type | sub_type)) #define DAT_GET_TYPE(status) ((u32)(status) & DAT_TYPE_MASK) #define DAT_GET_SUBTYPE(status) ((u32)(status) & DAT_SUBTYPE_MASK) @@ -1107,178 +1105,178 @@ struct dat_provider_attr { * ****************************************************************/ -typedef DAT_RETURN (*DAT_IA_OPEN_FUNC) ( +typedef u32 (*DAT_IA_OPEN_FUNC) ( const DAT_NAME_PTR, DAT_COUNT, DAT_EVD_HANDLE *, DAT_IA_HANDLE *); -typedef DAT_RETURN (*DAT_IA_OPENV_FUNC) ( +typedef u32 (*DAT_IA_OPENV_FUNC) ( const DAT_NAME_PTR, DAT_COUNT, DAT_EVD_HANDLE *, DAT_IA_HANDLE *, u32, u32, boolean_t); -typedef DAT_RETURN (*DAT_IA_CLOSE_FUNC) (DAT_IA_HANDLE, DAT_CLOSE_FLAGS); +typedef u32 (*DAT_IA_CLOSE_FUNC) (DAT_IA_HANDLE, DAT_CLOSE_FLAGS); -typedef DAT_RETURN (*DAT_IA_QUERY_FUNC) (DAT_IA_HANDLE, DAT_EVD_HANDLE *, +typedef u32 (*DAT_IA_QUERY_FUNC) (DAT_IA_HANDLE, DAT_EVD_HANDLE *, struct dat_ia_attr *, struct dat_provider_attr *); -typedef DAT_RETURN (*DAT_SET_CONSUMER_CONTEXT_FUNC) (DAT_HANDLE, DAT_CONTEXT); +typedef u32 (*DAT_SET_CONSUMER_CONTEXT_FUNC) (DAT_HANDLE, DAT_CONTEXT); -typedef DAT_RETURN (*DAT_GET_CONSUMER_CONTEXT_FUNC) (DAT_HANDLE, DAT_CONTEXT *); +typedef u32 (*DAT_GET_CONSUMER_CONTEXT_FUNC) (DAT_HANDLE, DAT_CONTEXT *); -typedef DAT_RETURN (*DAT_GET_HANDLE_TYPE_FUNC) (DAT_HANDLE, DAT_HANDLE_TYPE *); +typedef u32 (*DAT_GET_HANDLE_TYPE_FUNC) (DAT_HANDLE, DAT_HANDLE_TYPE *); -typedef DAT_RETURN (*DAT_CR_QUERY_FUNC) (DAT_CR_HANDLE, struct dat_cr_param *); +typedef u32 (*DAT_CR_QUERY_FUNC) (DAT_CR_HANDLE, struct dat_cr_param *); -typedef DAT_RETURN (*DAT_CR_ACCEPT_FUNC) ( +typedef u32 (*DAT_CR_ACCEPT_FUNC) ( DAT_CR_HANDLE, DAT_EP_HANDLE, DAT_COUNT, const DAT_PVOID); -typedef DAT_RETURN (*DAT_CR_REJECT_FUNC) (DAT_CR_HANDLE); +typedef u32 (*DAT_CR_REJECT_FUNC) (DAT_CR_HANDLE); -typedef DAT_RETURN (*DAT_CR_HANDOFF_FUNC) (DAT_CR_HANDLE, DAT_CONN_QUAL); +typedef u32 (*DAT_CR_HANDOFF_FUNC) (DAT_CR_HANDLE, DAT_CONN_QUAL); -typedef DAT_RETURN (*DAT_EVD_RESIZE_FUNC) (DAT_EVD_HANDLE, DAT_COUNT); +typedef u32 (*DAT_EVD_RESIZE_FUNC) (DAT_EVD_HANDLE, DAT_COUNT); -typedef DAT_RETURN (*DAT_EVD_POST_SE_FUNC) (DAT_EVD_HANDLE, +typedef u32 (*DAT_EVD_POST_SE_FUNC) (DAT_EVD_HANDLE, const struct dat_event *); -typedef DAT_RETURN (*DAT_EVD_DEQUEUE_FUNC) (DAT_EVD_HANDLE, struct dat_event *); +typedef u32 (*DAT_EVD_DEQUEUE_FUNC) (DAT_EVD_HANDLE, struct dat_event *); -typedef DAT_RETURN (*DAT_EVD_FREE_FUNC) (DAT_EVD_HANDLE); +typedef u32 (*DAT_EVD_FREE_FUNC) (DAT_EVD_HANDLE); -typedef DAT_RETURN (*DAT_EVD_QUERY_FUNC) (DAT_EVD_HANDLE, +typedef u32 (*DAT_EVD_QUERY_FUNC) (DAT_EVD_HANDLE, struct dat_evd_param *); -typedef DAT_RETURN (*DAT_EP_CREATE_FUNC) ( +typedef u32 (*DAT_EP_CREATE_FUNC) ( DAT_IA_HANDLE, DAT_PZ_HANDLE, DAT_EVD_HANDLE, DAT_EVD_HANDLE, DAT_EVD_HANDLE, const struct dat_ep_attr *, DAT_EP_HANDLE *); -typedef DAT_RETURN (*DAT_EP_CREATE_WITH_SRQ_FUNC) ( +typedef u32 (*DAT_EP_CREATE_WITH_SRQ_FUNC) ( DAT_IA_HANDLE, DAT_PZ_HANDLE, DAT_EVD_HANDLE, DAT_EVD_HANDLE, DAT_EVD_HANDLE, DAT_SRQ_HANDLE, const struct dat_ep_attr *, DAT_EP_HANDLE *); -typedef DAT_RETURN (*DAT_EP_QUERY_FUNC) (DAT_EP_HANDLE, struct dat_ep_param *); +typedef u32 (*DAT_EP_QUERY_FUNC) (DAT_EP_HANDLE, struct dat_ep_param *); -typedef DAT_RETURN (*DAT_EP_MODIFY_FUNC) ( +typedef u32 (*DAT_EP_MODIFY_FUNC) ( DAT_EP_HANDLE, DAT_EP_PARAM_MASK, const struct dat_ep_param *); -typedef DAT_RETURN (*DAT_EP_CONNECT_FUNC) ( +typedef u32 (*DAT_EP_CONNECT_FUNC) ( DAT_EP_HANDLE, DAT_IA_ADDRESS_PTR, DAT_CONN_QUAL, DAT_TIMEOUT, DAT_COUNT, const DAT_PVOID, DAT_QOS, DAT_CONNECT_FLAGS); -typedef DAT_RETURN (*DAT_EP_DUP_CONNECT_FUNC) ( +typedef u32 (*DAT_EP_DUP_CONNECT_FUNC) ( DAT_EP_HANDLE, DAT_EP_HANDLE, DAT_TIMEOUT, DAT_COUNT, const DAT_PVOID, DAT_QOS); -typedef DAT_RETURN (*DAT_EP_DISCONNECT_FUNC) (DAT_EP_HANDLE, DAT_CLOSE_FLAGS); +typedef u32 (*DAT_EP_DISCONNECT_FUNC) (DAT_EP_HANDLE, DAT_CLOSE_FLAGS); -typedef DAT_RETURN (*DAT_EP_POST_SEND_FUNC) ( +typedef u32 (*DAT_EP_POST_SEND_FUNC) ( DAT_EP_HANDLE, DAT_COUNT, struct dat_lmr_triplet *, DAT_DTO_COOKIE, DAT_COMPLETION_FLAGS); -typedef DAT_RETURN (*DAT_EP_POST_RECV_FUNC) ( +typedef u32 (*DAT_EP_POST_RECV_FUNC) ( DAT_EP_HANDLE, DAT_COUNT, struct dat_lmr_triplet *, DAT_DTO_COOKIE, DAT_COMPLETION_FLAGS); -typedef DAT_RETURN (*DAT_EP_POST_RDMA_READ_FUNC) ( +typedef u32 (*DAT_EP_POST_RDMA_READ_FUNC) ( DAT_EP_HANDLE, DAT_COUNT, struct dat_lmr_triplet *, DAT_DTO_COOKIE, const struct dat_rmr_triplet *, DAT_COMPLETION_FLAGS); -typedef DAT_RETURN (*DAT_EP_POST_RDMA_WRITE_FUNC) ( +typedef u32 (*DAT_EP_POST_RDMA_WRITE_FUNC) ( DAT_EP_HANDLE, DAT_COUNT, struct dat_lmr_triplet *, DAT_DTO_COOKIE, const struct dat_rmr_triplet *, DAT_COMPLETION_FLAGS); -typedef DAT_RETURN (*DAT_EP_GET_STATUS_FUNC) ( +typedef u32 (*DAT_EP_GET_STATUS_FUNC) ( DAT_EP_HANDLE, DAT_EP_STATE *, boolean_t *, boolean_t *); -typedef DAT_RETURN (*DAT_EP_FREE_FUNC) (DAT_EP_HANDLE); +typedef u32 (*DAT_EP_FREE_FUNC) (DAT_EP_HANDLE); -typedef DAT_RETURN (*DAT_EP_RESET_FUNC) (DAT_EP_HANDLE); +typedef u32 (*DAT_EP_RESET_FUNC) (DAT_EP_HANDLE); -typedef DAT_RETURN (*DAT_EP_RECV_QUERY_FUNC) ( +typedef u32 (*DAT_EP_RECV_QUERY_FUNC) ( DAT_EP_HANDLE, DAT_COUNT *, DAT_COUNT *); -typedef DAT_RETURN (*DAT_EP_SET_WATERMARK_FUNC) ( +typedef u32 (*DAT_EP_SET_WATERMARK_FUNC) ( DAT_EP_HANDLE, DAT_COUNT, DAT_COUNT); -typedef DAT_RETURN (*DAT_LMR_FREE_FUNC) (DAT_LMR_HANDLE); +typedef u32 (*DAT_LMR_FREE_FUNC) (DAT_LMR_HANDLE); -typedef DAT_RETURN (*DAT_LMR_QUERY_FUNC) (DAT_LMR_HANDLE, +typedef u32 (*DAT_LMR_QUERY_FUNC) (DAT_LMR_HANDLE, struct dat_lmr_param *); -typedef DAT_RETURN (*DAT_LMR_SYNC_RDMA_READ_FUNC) ( +typedef u32 (*DAT_LMR_SYNC_RDMA_READ_FUNC) ( DAT_IA_HANDLE, const struct dat_lmr_triplet *, DAT_VLEN); -typedef DAT_RETURN (*DAT_LMR_SYNC_RDMA_WRITE_FUNC) ( +typedef u32 (*DAT_LMR_SYNC_RDMA_WRITE_FUNC) ( DAT_IA_HANDLE, const struct dat_lmr_triplet *, DAT_VLEN); -typedef DAT_RETURN (*DAT_RMR_CREATE_FUNC) (DAT_PZ_HANDLE, DAT_RMR_HANDLE *); +typedef u32 (*DAT_RMR_CREATE_FUNC) (DAT_PZ_HANDLE, DAT_RMR_HANDLE *); -typedef DAT_RETURN (*DAT_RMR_QUERY_FUNC) (DAT_RMR_HANDLE, struct dat_rmr_param *); +typedef u32 (*DAT_RMR_QUERY_FUNC) (DAT_RMR_HANDLE, struct dat_rmr_param *); -typedef DAT_RETURN (*DAT_RMR_BIND_FUNC) ( +typedef u32 (*DAT_RMR_BIND_FUNC) ( DAT_RMR_HANDLE, const struct dat_lmr_triplet *, DAT_MEM_PRIV_FLAGS, DAT_EP_HANDLE, DAT_RMR_COOKIE, DAT_COMPLETION_FLAGS, DAT_RMR_CONTEXT *); -typedef DAT_RETURN (*DAT_RMR_FREE_FUNC) (DAT_RMR_HANDLE); +typedef u32 (*DAT_RMR_FREE_FUNC) (DAT_RMR_HANDLE); -typedef DAT_RETURN (*DAT_PSP_CREATE_FUNC) ( +typedef u32 (*DAT_PSP_CREATE_FUNC) ( DAT_IA_HANDLE, DAT_CONN_QUAL, DAT_EVD_HANDLE, DAT_PSP_FLAGS, DAT_PSP_HANDLE *); -typedef DAT_RETURN (*DAT_PSP_CREATE_ANY_FUNC) ( +typedef u32 (*DAT_PSP_CREATE_ANY_FUNC) ( DAT_IA_HANDLE, DAT_CONN_QUAL *, DAT_EVD_HANDLE, DAT_PSP_FLAGS, DAT_PSP_HANDLE *); -typedef DAT_RETURN (*DAT_PSP_QUERY_FUNC) (DAT_PSP_HANDLE, +typedef u32 (*DAT_PSP_QUERY_FUNC) (DAT_PSP_HANDLE, struct dat_psp_param *); -typedef DAT_RETURN (*DAT_PSP_FREE_FUNC) (DAT_PSP_HANDLE); +typedef u32 (*DAT_PSP_FREE_FUNC) (DAT_PSP_HANDLE); -typedef DAT_RETURN (*DAT_RSP_CREATE_FUNC) ( +typedef u32 (*DAT_RSP_CREATE_FUNC) ( DAT_IA_HANDLE, DAT_CONN_QUAL, DAT_EP_HANDLE, DAT_EVD_HANDLE, DAT_RSP_HANDLE *); -typedef DAT_RETURN (*DAT_RSP_QUERY_FUNC) (DAT_RSP_HANDLE, +typedef u32 (*DAT_RSP_QUERY_FUNC) (DAT_RSP_HANDLE, struct dat_rsp_param *); -typedef DAT_RETURN (*DAT_RSP_FREE_FUNC) (DAT_RSP_HANDLE); +typedef u32 (*DAT_RSP_FREE_FUNC) (DAT_RSP_HANDLE); -typedef DAT_RETURN (*DAT_PZ_CREATE_FUNC) (DAT_IA_HANDLE, DAT_PZ_HANDLE *); +typedef u32 (*DAT_PZ_CREATE_FUNC) (DAT_IA_HANDLE, DAT_PZ_HANDLE *); -typedef DAT_RETURN (*DAT_PZ_QUERY_FUNC) (DAT_PZ_HANDLE, struct dat_pz_param *); +typedef u32 (*DAT_PZ_QUERY_FUNC) (DAT_PZ_HANDLE, struct dat_pz_param *); -typedef DAT_RETURN (*DAT_PZ_FREE_FUNC) (DAT_PZ_HANDLE); +typedef u32 (*DAT_PZ_FREE_FUNC) (DAT_PZ_HANDLE); -typedef DAT_RETURN (*DAT_SRQ_CREATE_FUNC) ( +typedef u32 (*DAT_SRQ_CREATE_FUNC) ( DAT_IA_HANDLE, DAT_PZ_HANDLE, struct dat_srq_attr *, DAT_SRQ_HANDLE *); -typedef DAT_RETURN (*DAT_SRQ_SET_LW_FUNC) (DAT_SRQ_HANDLE, DAT_COUNT); +typedef u32 (*DAT_SRQ_SET_LW_FUNC) (DAT_SRQ_HANDLE, DAT_COUNT); -typedef DAT_RETURN (*DAT_SRQ_FREE_FUNC) (DAT_SRQ_HANDLE); +typedef u32 (*DAT_SRQ_FREE_FUNC) (DAT_SRQ_HANDLE); -typedef DAT_RETURN (*DAT_SRQ_QUERY_FUNC) (DAT_SRQ_HANDLE, +typedef u32 (*DAT_SRQ_QUERY_FUNC) (DAT_SRQ_HANDLE, struct dat_srq_param *); -typedef DAT_RETURN (*DAT_SRQ_RESIZE_FUNC) (DAT_SRQ_HANDLE, DAT_COUNT); +typedef u32 (*DAT_SRQ_RESIZE_FUNC) (DAT_SRQ_HANDLE, DAT_COUNT); -typedef DAT_RETURN (*DAT_SRQ_POST_RECV_FUNC) ( +typedef u32 (*DAT_SRQ_POST_RECV_FUNC) ( DAT_SRQ_HANDLE, DAT_COUNT, struct dat_lmr_triplet *, DAT_DTO_COOKIE); -typedef DAT_RETURN (*DAT_LMR_KCREATE_FUNC) ( +typedef u32 (*DAT_LMR_KCREATE_FUNC) ( DAT_IA_HANDLE, DAT_MEM_TYPE, DAT_REGION_DESCRIPTION, DAT_VLEN, DAT_PZ_HANDLE, DAT_MEM_PRIV_FLAGS, DAT_MEM_OPTIMIZE_FLAGS, DAT_LMR_HANDLE *, DAT_LMR_CONTEXT *, DAT_RMR_CONTEXT *, DAT_VLEN *, DAT_VADDR *); -typedef DAT_RETURN (*DAT_IA_MEMTYPE_HINT_FUNC) ( +typedef u32 (*DAT_IA_MEMTYPE_HINT_FUNC) ( DAT_IA_HANDLE, DAT_MEM_TYPE, DAT_VLEN, DAT_MEM_OPTIMIZE_FLAGS, DAT_VLEN *, DAT_VADDR *); -typedef DAT_RETURN (*DAT_EVD_KCREATE_FUNC) ( +typedef u32 (*DAT_EVD_KCREATE_FUNC) ( DAT_IA_HANDLE, DAT_COUNT, DAT_UPCALL_POLICY, const struct dat_upcall_object *, DAT_EVD_FLAGS, DAT_EVD_HANDLE *); -typedef DAT_RETURN (*DAT_EVD_MODIFY_UPCALL_FUNC) ( +typedef u32 (*DAT_EVD_MODIFY_UPCALL_FUNC) ( DAT_EVD_HANDLE, DAT_UPCALL_POLICY, const struct dat_upcall_object *); struct dat_provider { @@ -1372,16 +1370,16 @@ struct dat_provider { * DAT registry functions for providers */ -extern DAT_RETURN dat_registry_add_provider( +extern u32 dat_registry_add_provider( const struct dat_provider *, const struct dat_provider_info *); -extern DAT_RETURN dat_registry_remove_provider( +extern u32 dat_registry_remove_provider( const struct dat_provider *, const struct dat_provider_info *); /* * DAT registry functions for consumers */ -extern DAT_RETURN dat_ia_openv(const DAT_NAME_PTR, DAT_COUNT, +extern u32 dat_ia_openv(const DAT_NAME_PTR, DAT_COUNT, DAT_EVD_HANDLE *, DAT_IA_HANDLE *, u32, u32, boolean_t); @@ -1390,11 +1388,11 @@ extern DAT_RETURN dat_ia_openv(const DAT DAT_VERSION_MAJOR, DAT_VERSION_MINOR, \ DAT_THREADSAFE) -extern DAT_RETURN dat_ia_close(DAT_IA_HANDLE, DAT_CLOSE_FLAGS); +extern u32 dat_ia_close(DAT_IA_HANDLE, DAT_CLOSE_FLAGS); -extern DAT_RETURN dat_strerror(DAT_RETURN, const char **, const char **); +extern u32 dat_strerror(u32, const char **, const char **); -extern DAT_RETURN dat_registry_list_providers( +extern u32 dat_registry_list_providers( DAT_COUNT, DAT_COUNT *, struct dat_provider_info *dat_provider_list[]); /* @@ -1405,27 +1403,27 @@ extern DAT_RETURN dat_registry_list_prov #define DAT_CALL_PROVIDER_FUNC(func, handle, ...) \ DAT_HANDLE_TO_PROVIDER(handle)->func(handle, ##__VA_ARGS__) -static inline DAT_RETURN dat_set_consumer_context(DAT_HANDLE handle, +static inline u32 dat_set_consumer_context(DAT_HANDLE handle, DAT_CONTEXT context) { return DAT_CALL_PROVIDER_FUNC( set_consumer_context_func, handle, context); } -static inline DAT_RETURN dat_get_consumer_context(DAT_HANDLE handle, +static inline u32 dat_get_consumer_context(DAT_HANDLE handle, DAT_CONTEXT *context) { return DAT_CALL_PROVIDER_FUNC( get_consumer_context_func, handle, context); } -static inline DAT_RETURN dat_get_handle_type(DAT_HANDLE handle, +static inline u32 dat_get_handle_type(DAT_HANDLE handle, DAT_HANDLE_TYPE *type) { return DAT_CALL_PROVIDER_FUNC(get_handle_type_func, handle, type); } -static inline DAT_RETURN dat_ia_memtype_hint(DAT_IA_HANDLE ia, +static inline u32 dat_ia_memtype_hint(DAT_IA_HANDLE ia, DAT_MEM_TYPE mem_type, DAT_VLEN length, DAT_MEM_OPTIMIZE_FLAGS mem_optimize, @@ -1437,7 +1435,7 @@ static inline DAT_RETURN dat_ia_memtype_ preferred_length, preferred_alignment); } -static inline DAT_RETURN dat_ia_query(DAT_IA_HANDLE ia, +static inline u32 dat_ia_query(DAT_IA_HANDLE ia, DAT_EVD_HANDLE * async_evd, struct dat_ia_attr *ia_attr, struct dat_provider_attr *provider_attr) @@ -1446,7 +1444,7 @@ static inline DAT_RETURN dat_ia_query(DA ia_query_func, ia, async_evd, ia_attr, provider_attr); } -static inline DAT_RETURN dat_cr_accept(DAT_CR_HANDLE cr, +static inline u32 dat_cr_accept(DAT_CR_HANDLE cr, DAT_EP_HANDLE ep, DAT_COUNT private_data_size, const DAT_PVOID private_data) @@ -1455,35 +1453,35 @@ static inline DAT_RETURN dat_cr_accept(D cr_accept_func, cr, ep, private_data_size, private_data); } -static inline DAT_RETURN dat_cr_handoff(DAT_CR_HANDLE cr, +static inline u32 dat_cr_handoff(DAT_CR_HANDLE cr, DAT_CONN_QUAL handoff) { return DAT_CALL_PROVIDER_FUNC(cr_handoff_func, cr, handoff); } -static inline DAT_RETURN dat_cr_query(DAT_CR_HANDLE cr, +static inline u32 dat_cr_query(DAT_CR_HANDLE cr, struct dat_cr_param *param) { return DAT_CALL_PROVIDER_FUNC(cr_query_func, cr, param); } -static inline DAT_RETURN dat_cr_reject(DAT_CR_HANDLE cr) +static inline u32 dat_cr_reject(DAT_CR_HANDLE cr) { return DAT_CALL_PROVIDER_FUNC(cr_reject_func, cr); } -static inline DAT_RETURN dat_evd_dequeue(DAT_EVD_HANDLE evd, +static inline u32 dat_evd_dequeue(DAT_EVD_HANDLE evd, struct dat_event *event) { return DAT_CALL_PROVIDER_FUNC(evd_dequeue_func, evd, event); } -static inline DAT_RETURN dat_evd_free(DAT_EVD_HANDLE evd) +static inline u32 dat_evd_free(DAT_EVD_HANDLE evd) { return DAT_CALL_PROVIDER_FUNC(evd_free_func, evd); } -static inline DAT_RETURN dat_evd_kcreate(DAT_IA_HANDLE ia, +static inline u32 dat_evd_kcreate(DAT_IA_HANDLE ia, DAT_COUNT qlen, DAT_UPCALL_POLICY policy, const struct dat_upcall_object *upcall, @@ -1494,7 +1492,7 @@ static inline DAT_RETURN dat_evd_kcreate evd_kcreate_func, ia, qlen, policy, upcall, flags, evd); } -static inline DAT_RETURN dat_evd_modify_upcall(DAT_EVD_HANDLE evd, +static inline u32 dat_evd_modify_upcall(DAT_EVD_HANDLE evd, DAT_UPCALL_POLICY policy, const struct dat_upcall_object *upcall) { @@ -1502,24 +1500,24 @@ static inline DAT_RETURN dat_evd_modify_ evd_modify_upcall_func, evd, policy, upcall); } -static inline DAT_RETURN dat_evd_post_se(DAT_EVD_HANDLE evd, +static inline u32 dat_evd_post_se(DAT_EVD_HANDLE evd, const struct dat_event *event) { return DAT_CALL_PROVIDER_FUNC(evd_post_se_func, evd, event); } -static inline DAT_RETURN dat_evd_query(DAT_EVD_HANDLE evd, +static inline u32 dat_evd_query(DAT_EVD_HANDLE evd, struct dat_evd_param *param) { return DAT_CALL_PROVIDER_FUNC(evd_query_func, evd, param); } -static inline DAT_RETURN dat_evd_resize(DAT_EVD_HANDLE evd, DAT_COUNT qlen) +static inline u32 dat_evd_resize(DAT_EVD_HANDLE evd, DAT_COUNT qlen) { return DAT_CALL_PROVIDER_FUNC(evd_resize_func, evd, qlen); } -static inline DAT_RETURN dat_ep_connect(DAT_EP_HANDLE ep, +static inline u32 dat_ep_connect(DAT_EP_HANDLE ep, DAT_IA_ADDRESS_PTR ia_addr, DAT_CONN_QUAL conn_qual, DAT_TIMEOUT timeout, @@ -1533,7 +1531,7 @@ static inline DAT_RETURN dat_ep_connect( private_data_size, private_data, qos, flags); } -static inline DAT_RETURN dat_ep_create(DAT_IA_HANDLE ia, +static inline u32 dat_ep_create(DAT_IA_HANDLE ia, DAT_PZ_HANDLE pz, DAT_EVD_HANDLE in_evd, DAT_EVD_HANDLE out_evd, @@ -1546,7 +1544,7 @@ static inline DAT_RETURN dat_ep_create(D } -static inline DAT_RETURN dat_ep_create_with_srq(DAT_IA_HANDLE ia, +static inline u32 dat_ep_create_with_srq(DAT_IA_HANDLE ia, DAT_PZ_HANDLE pz, DAT_EVD_HANDLE in_evd, DAT_EVD_HANDLE out_evd, @@ -1559,13 +1557,13 @@ static inline DAT_RETURN dat_ep_create_w out_evd, connect_evd, srq, attr, ep); } -static inline DAT_RETURN dat_ep_disconnect(DAT_EP_HANDLE ep, +static inline u32 dat_ep_disconnect(DAT_EP_HANDLE ep, DAT_CLOSE_FLAGS flags) { return DAT_CALL_PROVIDER_FUNC(ep_disconnect_func, ep, flags); } -static inline DAT_RETURN dat_ep_dup_connect(DAT_EP_HANDLE ep, +static inline u32 dat_ep_dup_connect(DAT_EP_HANDLE ep, DAT_EP_HANDLE dup_ep, DAT_TIMEOUT timeout, DAT_COUNT private_data_size, @@ -1576,12 +1574,12 @@ static inline DAT_RETURN dat_ep_dup_conn private_data_size, private_data, qos); } -static inline DAT_RETURN dat_ep_free(DAT_EP_HANDLE ep) +static inline u32 dat_ep_free(DAT_EP_HANDLE ep) { return DAT_CALL_PROVIDER_FUNC(ep_free_func, ep); } -static inline DAT_RETURN dat_ep_get_status(DAT_EP_HANDLE ep, +static inline u32 dat_ep_get_status(DAT_EP_HANDLE ep, DAT_EP_STATE * state, boolean_t *recv_idle, boolean_t *req_idle) @@ -1590,14 +1588,14 @@ static inline DAT_RETURN dat_ep_get_stat ep_get_status_func, ep, state, recv_idle, req_idle); } -static inline DAT_RETURN dat_ep_modify(DAT_EP_HANDLE ep, +static inline u32 dat_ep_modify(DAT_EP_HANDLE ep, DAT_EP_PARAM_MASK mask, const struct dat_ep_param *param) { return DAT_CALL_PROVIDER_FUNC(ep_modify_func, ep, mask, param); } -static inline DAT_RETURN dat_ep_post_rdma_read(DAT_EP_HANDLE ep, +static inline u32 dat_ep_post_rdma_read(DAT_EP_HANDLE ep, DAT_COUNT size, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, @@ -1608,7 +1606,7 @@ static inline DAT_RETURN dat_ep_post_rdm local_iov, cookie, remote_iov, flags); } -static inline DAT_RETURN dat_ep_post_rdma_write(DAT_EP_HANDLE ep, +static inline u32 dat_ep_post_rdma_write(DAT_EP_HANDLE ep, DAT_COUNT size, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, @@ -1619,7 +1617,7 @@ static inline DAT_RETURN dat_ep_post_rdm local_iov, cookie, remote_iov, flags); } -static inline DAT_RETURN dat_ep_post_recv(DAT_EP_HANDLE ep, +static inline u32 dat_ep_post_recv(DAT_EP_HANDLE ep, DAT_COUNT size, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, @@ -1629,7 +1627,7 @@ static inline DAT_RETURN dat_ep_post_rec ep_post_recv_func, ep, size, local_iov, cookie, flags); } -static inline DAT_RETURN dat_ep_post_send(DAT_EP_HANDLE ep, +static inline u32 dat_ep_post_send(DAT_EP_HANDLE ep, DAT_COUNT size, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, @@ -1639,12 +1637,12 @@ static inline DAT_RETURN dat_ep_post_sen ep_post_send_func, ep, size, local_iov, cookie, flags); } -static inline DAT_RETURN dat_ep_query(DAT_EP_HANDLE ep, struct dat_ep_param *param) +static inline u32 dat_ep_query(DAT_EP_HANDLE ep, struct dat_ep_param *param) { return DAT_CALL_PROVIDER_FUNC(ep_query_func, ep, param); } -static inline DAT_RETURN dat_ep_recv_query(DAT_EP_HANDLE ep, +static inline u32 dat_ep_recv_query(DAT_EP_HANDLE ep, DAT_COUNT * bufs_alloc, DAT_COUNT * bufs_avail) { @@ -1652,12 +1650,12 @@ static inline DAT_RETURN dat_ep_recv_que ep_recv_query_func, ep, bufs_alloc, bufs_avail); } -static inline DAT_RETURN dat_ep_reset(DAT_EP_HANDLE ep) +static inline u32 dat_ep_reset(DAT_EP_HANDLE ep) { return DAT_CALL_PROVIDER_FUNC(ep_reset_func, ep); } -static inline DAT_RETURN dat_ep_set_watermark(DAT_EP_HANDLE ep, +static inline u32 dat_ep_set_watermark(DAT_EP_HANDLE ep, DAT_COUNT soft_high_watermark, DAT_COUNT hard_high_watermark) { @@ -1665,18 +1663,18 @@ static inline DAT_RETURN dat_ep_set_wate soft_high_watermark, hard_high_watermark); } -static inline DAT_RETURN dat_lmr_query(DAT_LMR_HANDLE lmr, +static inline u32 dat_lmr_query(DAT_LMR_HANDLE lmr, struct dat_lmr_param *param) { return DAT_CALL_PROVIDER_FUNC(lmr_query_func, lmr, param); } -static inline DAT_RETURN dat_lmr_free(DAT_LMR_HANDLE lmr) +static inline u32 dat_lmr_free(DAT_LMR_HANDLE lmr) { return DAT_CALL_PROVIDER_FUNC(lmr_free_func, lmr); } -static inline DAT_RETURN dat_lmr_sync_rdma_read(DAT_IA_HANDLE ia, +static inline u32 dat_lmr_sync_rdma_read(DAT_IA_HANDLE ia, const struct dat_lmr_triplet *iovs, DAT_VLEN num_iovs) { @@ -1684,7 +1682,7 @@ static inline DAT_RETURN dat_lmr_sync_rd lmr_sync_rdma_read_func, ia, iovs, num_iovs); } -static inline DAT_RETURN dat_lmr_sync_rdma_write(DAT_IA_HANDLE ia, +static inline u32 dat_lmr_sync_rdma_write(DAT_IA_HANDLE ia, const struct dat_lmr_triplet *iovs, DAT_VLEN num_iovs) { @@ -1692,17 +1690,17 @@ static inline DAT_RETURN dat_lmr_sync_rd lmr_sync_rdma_write_func, ia, iovs, num_iovs); } -static inline DAT_RETURN dat_rmr_create(DAT_PZ_HANDLE pz, DAT_RMR_HANDLE * rmr) +static inline u32 dat_rmr_create(DAT_PZ_HANDLE pz, DAT_RMR_HANDLE * rmr) { return DAT_CALL_PROVIDER_FUNC(rmr_create_func, pz, rmr); } -static inline DAT_RETURN dat_rmr_query(DAT_RMR_HANDLE rmr, struct dat_rmr_param *param) +static inline u32 dat_rmr_query(DAT_RMR_HANDLE rmr, struct dat_rmr_param *param) { return DAT_CALL_PROVIDER_FUNC(rmr_query_func, rmr, param); } -static inline DAT_RETURN dat_rmr_bind(DAT_RMR_HANDLE rmr, +static inline u32 dat_rmr_bind(DAT_RMR_HANDLE rmr, const struct dat_lmr_triplet *iov, DAT_MEM_PRIV_FLAGS mem_flags, DAT_EP_HANDLE ep, @@ -1714,12 +1712,12 @@ static inline DAT_RETURN dat_rmr_bind(DA cookie, comp_flags, context); } -static inline DAT_RETURN dat_rmr_free(DAT_RMR_HANDLE rmr) +static inline u32 dat_rmr_free(DAT_RMR_HANDLE rmr) { return DAT_CALL_PROVIDER_FUNC(rmr_free_func, rmr); } -static inline DAT_RETURN dat_psp_create(DAT_IA_HANDLE ia, +static inline u32 dat_psp_create(DAT_IA_HANDLE ia, DAT_CONN_QUAL conn_qual, DAT_EVD_HANDLE evd, DAT_PSP_FLAGS flags, @@ -1729,7 +1727,7 @@ static inline DAT_RETURN dat_psp_create( psp_create_func, ia, conn_qual, evd, flags, psp); } -static inline DAT_RETURN dat_psp_create_any(DAT_IA_HANDLE ia, +static inline u32 dat_psp_create_any(DAT_IA_HANDLE ia, DAT_CONN_QUAL * conn_qual, DAT_EVD_HANDLE evd, DAT_PSP_FLAGS flags, @@ -1739,18 +1737,18 @@ static inline DAT_RETURN dat_psp_create_ psp_create_any_func, ia, conn_qual, evd, flags, psp); } -static inline DAT_RETURN dat_psp_query(DAT_PSP_HANDLE psp, +static inline u32 dat_psp_query(DAT_PSP_HANDLE psp, struct dat_psp_param *param) { return DAT_CALL_PROVIDER_FUNC(psp_query_func, psp, param); } -static inline DAT_RETURN dat_psp_free(DAT_PSP_HANDLE psp) +static inline u32 dat_psp_free(DAT_PSP_HANDLE psp) { return DAT_CALL_PROVIDER_FUNC(psp_free_func, psp); } -static inline DAT_RETURN dat_rsp_create(DAT_IA_HANDLE ia, +static inline u32 dat_rsp_create(DAT_IA_HANDLE ia, DAT_CONN_QUAL conn_qual, DAT_EP_HANDLE ep, DAT_EVD_HANDLE evd, @@ -1760,34 +1758,34 @@ static inline DAT_RETURN dat_rsp_create( rsp_create_func, ia, conn_qual, ep, evd, rsp); } -static inline DAT_RETURN dat_rsp_query(DAT_RSP_HANDLE rsp, +static inline u32 dat_rsp_query(DAT_RSP_HANDLE rsp, struct dat_rsp_param *param) { return DAT_CALL_PROVIDER_FUNC(rsp_query_func, rsp, param); } -static inline DAT_RETURN dat_rsp_free(DAT_RSP_HANDLE rsp) +static inline u32 dat_rsp_free(DAT_RSP_HANDLE rsp) { return DAT_CALL_PROVIDER_FUNC(rsp_free_func, rsp); } -static inline DAT_RETURN dat_pz_create(DAT_IA_HANDLE ia, DAT_PZ_HANDLE * pz) +static inline u32 dat_pz_create(DAT_IA_HANDLE ia, DAT_PZ_HANDLE * pz) { return DAT_CALL_PROVIDER_FUNC(pz_create_func, ia, pz); } -static inline DAT_RETURN dat_pz_query(DAT_PZ_HANDLE pz, +static inline u32 dat_pz_query(DAT_PZ_HANDLE pz, struct dat_pz_param *param) { return DAT_CALL_PROVIDER_FUNC(pz_query_func, pz, param); } -static inline DAT_RETURN dat_pz_free(DAT_PZ_HANDLE pz) +static inline u32 dat_pz_free(DAT_PZ_HANDLE pz) { return DAT_CALL_PROVIDER_FUNC(pz_free_func, pz); } -static inline DAT_RETURN dat_srq_create(DAT_IA_HANDLE ia, +static inline u32 dat_srq_create(DAT_IA_HANDLE ia, DAT_PZ_HANDLE pz, struct dat_srq_attr *attr, DAT_SRQ_HANDLE * srq) @@ -1795,12 +1793,12 @@ static inline DAT_RETURN dat_srq_create( return DAT_CALL_PROVIDER_FUNC(srq_create_func, ia, pz, attr, srq); } -static inline DAT_RETURN dat_srq_free(DAT_SRQ_HANDLE srq) +static inline u32 dat_srq_free(DAT_SRQ_HANDLE srq) { return DAT_CALL_PROVIDER_FUNC(srq_free_func, srq); } -static inline DAT_RETURN dat_srq_post_recv(DAT_SRQ_HANDLE srq, +static inline u32 dat_srq_post_recv(DAT_SRQ_HANDLE srq, DAT_COUNT num_iovs, struct dat_lmr_triplet *iovs, DAT_DTO_COOKIE cookie) @@ -1809,25 +1807,25 @@ static inline DAT_RETURN dat_srq_post_re srq_post_recv_func, srq, num_iovs, iovs, cookie); } -static inline DAT_RETURN dat_srq_query(DAT_SRQ_HANDLE srq, +static inline u32 dat_srq_query(DAT_SRQ_HANDLE srq, struct dat_srq_param *param) { return DAT_CALL_PROVIDER_FUNC(srq_query_func, srq, param); } -static inline DAT_RETURN dat_srq_resize(DAT_SRQ_HANDLE srq, +static inline u32 dat_srq_resize(DAT_SRQ_HANDLE srq, DAT_COUNT max_recv_dtos) { return DAT_CALL_PROVIDER_FUNC(srq_resize_func, srq, max_recv_dtos); } -static inline DAT_RETURN dat_srq_set_lw(DAT_SRQ_HANDLE srq, +static inline u32 dat_srq_set_lw(DAT_SRQ_HANDLE srq, DAT_COUNT low_watermark) { return DAT_CALL_PROVIDER_FUNC(srq_set_lw_func, srq, low_watermark); } -static inline DAT_RETURN dat_lmr_kcreate(DAT_IA_HANDLE ia, +static inline u32 dat_lmr_kcreate(DAT_IA_HANDLE ia, DAT_MEM_TYPE type, DAT_REGION_DESCRIPTION region, DAT_VLEN len, diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat/dictionary.c linux-kernel2/dat/dictionary.c --- linux-kernel/dat/dictionary.c 2005-05-11 08:28:13.736004000 -0700 +++ linux-kernel2/dat/dictionary.c 2005-05-11 11:26:48.545963000 -0700 @@ -58,9 +58,8 @@ struct DAT_DICTIONARY { * * *********************************************************************/ -static DAT_RETURN -dat_dictionary_key_dup(const struct dat_provider_info *old_key, - struct dat_provider_info *new_key); +static u32 dat_dictionary_key_dup(const struct dat_provider_info *old_key, + struct dat_provider_info *new_key); static boolean_t dat_dictionary_key_is_equal(const struct dat_provider_info *key_a, @@ -76,15 +75,13 @@ dat_dictionary_key_is_equal(const struct * Function: dat_dictionary_create ***********************************************************************/ -DAT_RETURN dat_dictionary_create(DAT_DICTIONARY ** pp_dictionary) +u32 dat_dictionary_create(DAT_DICTIONARY **pp_dictionary) { DAT_DICTIONARY *p_dictionary; - DAT_RETURN status; + u32 status = DAT_SUCCESS; BUG_ON(NULL == pp_dictionary); - status = DAT_SUCCESS; - /* create the dictionary */ p_dictionary = kmalloc(sizeof(DAT_DICTIONARY), GFP_ATOMIC); if (NULL == p_dictionary) { @@ -140,7 +137,7 @@ DAT_RETURN dat_dictionary_create(DAT_DIC * Function: dat_dictionary_destroy ***********************************************************************/ -DAT_RETURN dat_dictionary_destroy(DAT_DICTIONARY * p_dictionary) +u32 dat_dictionary_destroy(DAT_DICTIONARY *p_dictionary) { DAT_DICTIONARY_NODE *cur_node; @@ -162,8 +159,7 @@ DAT_RETURN dat_dictionary_destroy(DAT_DI * Function: dat_dictionary_size ***********************************************************************/ -DAT_RETURN -dat_dictionary_size(DAT_DICTIONARY * p_dictionary, DAT_COUNT * p_size) +u32 dat_dictionary_size(DAT_DICTIONARY *p_dictionary, DAT_COUNT *p_size) { BUG_ON(NULL == p_dictionary); BUG_ON(NULL == p_size); @@ -177,15 +173,13 @@ dat_dictionary_size(DAT_DICTIONARY * p_d * Function: dat_dictionary_entry_create ***********************************************************************/ -DAT_RETURN dat_dictionary_entry_create(DAT_DICTIONARY_ENTRY * p_entry) +u32 dat_dictionary_entry_create(DAT_DICTIONARY_ENTRY *p_entry) { DAT_DICTIONARY_NODE *node; - DAT_RETURN status; + u32 status = DAT_SUCCESS; BUG_ON(NULL == p_entry); - status = DAT_SUCCESS; - node = kmalloc(sizeof(DAT_DICTIONARY_NODE), GFP_ATOMIC); if (NULL == node) { status = @@ -203,7 +197,7 @@ DAT_RETURN dat_dictionary_entry_create(D * Function: dat_dictionary_entry_destroy ***********************************************************************/ -DAT_RETURN dat_dictionary_entry_destroy(const DAT_DICTIONARY_ENTRY entry) +u32 dat_dictionary_entry_destroy(const DAT_DICTIONARY_ENTRY entry) { kfree(entry); return DAT_SUCCESS; @@ -213,12 +207,12 @@ DAT_RETURN dat_dictionary_entry_destroy( * Function: dat_dictionary_insert ***********************************************************************/ -DAT_RETURN dat_dictionary_insert(DAT_DICTIONARY * p_dictionary, - DAT_DICTIONARY_ENTRY entry, - const struct dat_provider_info *key, - DAT_DICTIONARY_DATA data) +u32 dat_dictionary_insert(DAT_DICTIONARY *p_dictionary, + DAT_DICTIONARY_ENTRY entry, + const struct dat_provider_info *key, + DAT_DICTIONARY_DATA data) { - DAT_RETURN status; + u32 status; DAT_DICTIONARY_NODE *cur_node, *prev_node, *next_node; BUG_ON(NULL == p_dictionary); @@ -256,12 +250,12 @@ DAT_RETURN dat_dictionary_insert(DAT_DIC * Function: dat_dictionary_search ***********************************************************************/ -DAT_RETURN dat_dictionary_search(DAT_DICTIONARY * p_dictionary, - const struct dat_provider_info *key, - DAT_DICTIONARY_DATA * p_data) +u32 dat_dictionary_search(DAT_DICTIONARY *p_dictionary, + const struct dat_provider_info *key, + DAT_DICTIONARY_DATA *p_data) { DAT_DICTIONARY_NODE *cur_node; - DAT_RETURN status; + u32 status; BUG_ON(NULL == p_dictionary); BUG_ON(NULL == key); @@ -288,13 +282,13 @@ DAT_RETURN dat_dictionary_search(DAT_DIC * Function: dat_dictionary_enumerate ***********************************************************************/ -DAT_RETURN dat_dictionary_enumerate(DAT_DICTIONARY * p_dictionary, - DAT_DICTIONARY_DATA array[], - DAT_COUNT array_size) +u32 dat_dictionary_enumerate(DAT_DICTIONARY *p_dictionary, + DAT_DICTIONARY_DATA array[], + DAT_COUNT array_size) { DAT_DICTIONARY_NODE *cur_node; DAT_COUNT i; - DAT_RETURN status; + u32 status; BUG_ON(NULL == p_dictionary); BUG_ON(NULL == array); @@ -319,13 +313,13 @@ DAT_RETURN dat_dictionary_enumerate(DAT_ * Function: dat_dictionary_remove ***********************************************************************/ -DAT_RETURN dat_dictionary_remove(DAT_DICTIONARY * p_dictionary, - DAT_DICTIONARY_ENTRY * p_entry, - const struct dat_provider_info *key, - DAT_DICTIONARY_DATA * p_data) +u32 dat_dictionary_remove(DAT_DICTIONARY *p_dictionary, + DAT_DICTIONARY_ENTRY *p_entry, + const struct dat_provider_info *key, + DAT_DICTIONARY_DATA *p_data) { DAT_DICTIONARY_NODE *cur_node, *prev_node, *next_node; - DAT_RETURN status; + u32 status; BUG_ON(NULL == p_dictionary); BUG_ON(NULL == p_entry); @@ -369,8 +363,8 @@ DAT_RETURN dat_dictionary_remove(DAT_DIC * Function: dat_dictionary_key_create ***********************************************************************/ -DAT_RETURN dat_dictionary_key_dup(const struct dat_provider_info *old_key, - struct dat_provider_info *new_key) +u32 dat_dictionary_key_dup(const struct dat_provider_info *old_key, + struct dat_provider_info *new_key) { BUG_ON(NULL == old_key); BUG_ON(NULL == new_key); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat/dictionary.h linux-kernel2/dat/dictionary.h --- linux-kernel/dat/dictionary.h 2005-05-10 08:13:29.739017000 -0700 +++ linux-kernel2/dat/dictionary.h 2005-05-11 11:28:11.413961000 -0700 @@ -52,33 +52,32 @@ typedef void *DAT_DICTIONARY_DATA; * * *********************************************************************/ -extern DAT_RETURN dat_dictionary_create(DAT_DICTIONARY ** pp_dictionary); +extern u32 dat_dictionary_create(DAT_DICTIONARY **pp_dictionary); -extern DAT_RETURN dat_dictionary_destroy(DAT_DICTIONARY * p_dictionary); +extern u32 dat_dictionary_destroy(DAT_DICTIONARY *p_dictionary); -extern DAT_RETURN dat_dictionary_size(DAT_DICTIONARY * p_dictionary, - DAT_COUNT * p_size); +extern u32 dat_dictionary_size(DAT_DICTIONARY *p_dictionary, DAT_COUNT *p_size); -extern DAT_RETURN dat_dictionary_entry_create(DAT_DICTIONARY_ENTRY * p_entry); +extern u32 dat_dictionary_entry_create(DAT_DICTIONARY_ENTRY *p_entry); -extern DAT_RETURN dat_dictionary_entry_destroy(const DAT_DICTIONARY_ENTRY entry); +extern u32 dat_dictionary_entry_destroy(const DAT_DICTIONARY_ENTRY entry); -extern DAT_RETURN dat_dictionary_insert(DAT_DICTIONARY * p_dictionary, - DAT_DICTIONARY_ENTRY entry, - const struct dat_provider_info *key, - DAT_DICTIONARY_DATA data); +extern u32 dat_dictionary_insert(DAT_DICTIONARY *p_dictionary, + DAT_DICTIONARY_ENTRY entry, + const struct dat_provider_info *key, + DAT_DICTIONARY_DATA data); -extern DAT_RETURN dat_dictionary_search(DAT_DICTIONARY * p_dictionary, - const struct dat_provider_info *key, - DAT_DICTIONARY_DATA * p_data); +extern u32 dat_dictionary_search(DAT_DICTIONARY *p_dictionary, + const struct dat_provider_info *key, + DAT_DICTIONARY_DATA *p_data); -extern DAT_RETURN dat_dictionary_enumerate(DAT_DICTIONARY * p_dictionary, - DAT_DICTIONARY_DATA array[], - DAT_COUNT array_size); +extern u32 dat_dictionary_enumerate(DAT_DICTIONARY *p_dictionary, + DAT_DICTIONARY_DATA array[], + DAT_COUNT array_size); -extern DAT_RETURN dat_dictionary_remove(DAT_DICTIONARY * p_dictionary, - DAT_DICTIONARY_ENTRY * p_entry, - const struct dat_provider_info *key, - DAT_DICTIONARY_DATA * p_data); +extern u32 dat_dictionary_remove(DAT_DICTIONARY *p_dictionary, + DAT_DICTIONARY_ENTRY *p_entry, + const struct dat_provider_info *key, + DAT_DICTIONARY_DATA *p_data); #endif /* DICTIONARY_H */ diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat/dr.c linux-kernel2/dat/dr.c --- linux-kernel/dat/dr.c 2005-05-10 08:13:29.644001000 -0700 +++ linux-kernel2/dat/dr.c 2005-05-11 11:29:23.917967000 -0700 @@ -68,9 +68,9 @@ static DAT_DICTIONARY *g_dr_dictionary = * Function: dat_dr_init ***********************************************************************/ -DAT_RETURN dat_dr_init(void) +u32 dat_dr_init(void) { - DAT_RETURN status; + u32 status; spin_lock_init(&g_dr_lock); @@ -83,9 +83,9 @@ DAT_RETURN dat_dr_init(void) * Function: dat_dr_fini ************************************************************************/ -DAT_RETURN dat_dr_fini(void) +u32 dat_dr_fini(void) { - DAT_RETURN status; + u32 status; /* if g_dr_lock needed to be de-allocated we would do it here */ @@ -98,10 +98,10 @@ DAT_RETURN dat_dr_fini(void) * Function: dat_dr_insert ************************************************************************/ -DAT_RETURN dat_dr_insert(const struct dat_provider_info *info, - const DAT_IA_OPEN_FUNC ia_open_func) +u32 dat_dr_insert(const struct dat_provider_info *info, + const DAT_IA_OPEN_FUNC ia_open_func) { - DAT_RETURN status; + u32 status; DAT_DICTIONARY_ENTRY dict_entry; DAT_DR_ENTRY *data; @@ -144,11 +144,11 @@ DAT_RETURN dat_dr_insert(const struct da * Function: dat_dr_remove ************************************************************************/ -DAT_RETURN dat_dr_remove(const struct dat_provider_info *info) +u32 dat_dr_remove(const struct dat_provider_info *info) { DAT_DR_ENTRY *data; DAT_DICTIONARY_ENTRY dict_entry; - DAT_RETURN status; + u32 status; dict_entry = NULL; spin_lock(&g_dr_lock); @@ -184,10 +184,10 @@ DAT_RETURN dat_dr_remove(const struct da * Function: dat_dr_provider_open ************************************************************************/ -DAT_RETURN dat_dr_provider_open(const struct dat_provider_info *info, +u32 dat_dr_provider_open(const struct dat_provider_info *info, DAT_IA_OPEN_FUNC *p_ia_open_func) { - DAT_RETURN status; + u32 status; DAT_DR_ENTRY *data; spin_lock(&g_dr_lock); @@ -209,9 +209,9 @@ DAT_RETURN dat_dr_provider_open(const st * Function: dat_dr_provider_close ************************************************************************/ -DAT_RETURN dat_dr_provider_close(const struct dat_provider_info *info) +u32 dat_dr_provider_close(const struct dat_provider_info *info) { - DAT_RETURN status; + u32 status; DAT_DR_ENTRY *data; spin_lock(&g_dr_lock); @@ -231,7 +231,7 @@ DAT_RETURN dat_dr_provider_close(const s * Function: dat_dr_size ************************************************************************/ -DAT_RETURN dat_dr_size(DAT_COUNT *size) +u32 dat_dr_size(DAT_COUNT *size) { return dat_dictionary_size(g_dr_dictionary, size); } @@ -240,17 +240,16 @@ DAT_RETURN dat_dr_size(DAT_COUNT *size) * Function: dat_dr_list ***********************************************************************/ -DAT_RETURN dat_dr_list(DAT_COUNT max_to_return, - DAT_COUNT *entries_returned, - struct dat_provider_info *(dat_provider_list[])) +u32 dat_dr_list(DAT_COUNT max_to_return, + DAT_COUNT *entries_returned, + struct dat_provider_info *(dat_provider_list[])) { DAT_DR_ENTRY **array; DAT_COUNT array_size; DAT_COUNT i; - DAT_RETURN status; + u32 status = DAT_SUCCESS; array = NULL; - status = DAT_SUCCESS; /* The dictionary size may increase between the call to */ /* dat_dictionary_size() and dat_dictionary_enumerate(). */ diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat/dr.h linux-kernel2/dat/dr.h --- linux-kernel/dat/dr.h 2005-05-10 08:13:29.758011000 -0700 +++ linux-kernel2/dat/dr.h 2005-05-11 11:29:53.343962000 -0700 @@ -42,24 +42,24 @@ * * *********************************************************************/ -extern DAT_RETURN dat_dr_init(void); +extern u32 dat_dr_init(void); -extern DAT_RETURN dat_dr_fini(void); +extern u32 dat_dr_fini(void); -extern DAT_RETURN dat_dr_insert(const struct dat_provider_info *info, - const DAT_IA_OPEN_FUNC ia_open_func); +extern u32 dat_dr_insert(const struct dat_provider_info *info, + const DAT_IA_OPEN_FUNC ia_open_func); -extern DAT_RETURN dat_dr_remove(const struct dat_provider_info *info); +extern u32 dat_dr_remove(const struct dat_provider_info *info); -extern DAT_RETURN dat_dr_provider_open(const struct dat_provider_info *info, - DAT_IA_OPEN_FUNC *p_ia_open_func); +extern u32 dat_dr_provider_open(const struct dat_provider_info *info, + DAT_IA_OPEN_FUNC *p_ia_open_func); -extern DAT_RETURN dat_dr_provider_close(const struct dat_provider_info *info); +extern u32 dat_dr_provider_close(const struct dat_provider_info *info); -extern DAT_RETURN dat_dr_size(DAT_COUNT *size); +extern u32 dat_dr_size(DAT_COUNT *size); -extern DAT_RETURN dat_dr_list(DAT_COUNT max_to_return, - DAT_COUNT *entries_returned, - struct dat_provider_info *(dat_provider_list[])); +extern u32 dat_dr_list(DAT_COUNT max_to_return, + DAT_COUNT *entries_returned, + struct dat_provider_info *(dat_provider_list[])); #endif /* DR_H */ diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_adapter_util.h linux-kernel2/dat-provider/dapl_adapter_util.h --- linux-kernel/dat-provider/dapl_adapter_util.h 2005-05-11 10:09:59.053024000 -0700 +++ linux-kernel2/dat-provider/dapl_adapter_util.h 2005-05-11 11:37:05.389975000 -0700 @@ -49,112 +49,93 @@ typedef enum async_handler_type { */ #define IB_CM_REJ_REASON_CONSUMER_REJ 0x001C -DAT_RETURN dapl_ib_enum_hcas(const char *vendor, - DAPL_HCA_NAME ** hca_names, - DAT_COUNT * total_hca_count); +u32 dapl_ib_enum_hcas(const char *vendor, + DAPL_HCA_NAME **hca_names, + DAT_COUNT *total_hca_count); -DAT_RETURN dapl_ib_get_instance_data(DAPL_HCA_NAME hca_name, char *instance); +u32 dapl_ib_get_instance_data(DAPL_HCA_NAME hca_name, char *instance); -DAT_RETURN dapl_ib_open_hca(char *namestr, DAPL_HCA * hca_ptr); +u32 dapl_ib_open_hca(char *namestr, DAPL_HCA *hca_ptr); -DAT_RETURN dapl_ib_close_hca(DAPL_HCA * hca_ptr); +u32 dapl_ib_close_hca(DAPL_HCA *hca_ptr); -DAT_RETURN dapl_ib_qp_alloc(DAPL_IA * ia_ptr, - DAPL_EP * ep_ptr, DAPL_EP * ep_ctx_ptr); +u32 dapl_ib_qp_alloc(DAPL_IA *ia_ptr, DAPL_EP *ep_ptr, DAPL_EP *ep_ctx_ptr); -DAT_RETURN dapl_ib_qp_free(DAPL_IA * ia_ptr, DAPL_EP * ep_ptr); +u32 dapl_ib_qp_free(DAPL_IA *ia_ptr, DAPL_EP *ep_ptr); -DAT_RETURN dapl_ib_qp_modify(DAPL_IA * ia_ptr, - DAPL_EP * ep_ptr, struct dat_ep_attr *ep_attr); +u32 dapl_ib_qp_modify(DAPL_IA *ia_ptr, DAPL_EP *ep_ptr, + struct dat_ep_attr *ep_attr); -DAT_RETURN dapl_ib_connect(DAT_EP_HANDLE ep_handle, - DAT_IA_ADDRESS_PTR remote_ia_address, - DAT_CONN_QUAL remote_conn_qual, - DAT_COUNT private_data_size, DAT_PVOID private_data); +u32 dapl_ib_connect(DAT_EP_HANDLE ep_handle, + DAT_IA_ADDRESS_PTR remote_ia_address, + DAT_CONN_QUAL remote_conn_qual, + DAT_COUNT private_data_size, DAT_PVOID private_data); -DAT_RETURN dapl_ib_disconnect(DAPL_EP * ep_ptr, DAT_CLOSE_FLAGS close_flags); +u32 dapl_ib_disconnect(DAPL_EP *ep_ptr, DAT_CLOSE_FLAGS close_flags); -DAT_RETURN dapl_ib_setup_conn_listener(DAPL_IA * ia_ptr, - u64 ServiceID, DAPL_SP * sp_ptr); +u32 dapl_ib_setup_conn_listener(DAPL_IA *ia_ptr, u64 ServiceID, + DAPL_SP *sp_ptr); -DAT_RETURN dapl_ib_remove_conn_listener(DAPL_IA * ia_ptr, DAPL_SP * sp_ptr); +u32 dapl_ib_remove_conn_listener(DAPL_IA *ia_ptr, DAPL_SP *sp_ptr); -DAT_RETURN dapl_ib_accept_connection(DAT_CR_HANDLE cr_handle, - DAT_EP_HANDLE ep_handle, - DAT_COUNT private_data_size, - const DAT_PVOID private_data); +u32 dapl_ib_accept_connection(DAT_CR_HANDLE cr_handle, + DAT_EP_HANDLE ep_handle, + DAT_COUNT private_data_size, + const DAT_PVOID private_data); -DAT_RETURN dapl_ib_reject_connection(ib_cm_handle_t cm_handle, - int reject_reason); +u32 dapl_ib_reject_connection(ib_cm_handle_t cm_handle, int reject_reason); -DAT_RETURN dapl_ib_setup_async_callback(DAPL_IA * ia_ptr, - DAPL_ASYNC_HANDLER_TYPE handler_type, - ib_async_handler_t callback, - void *context); +u32 dapl_ib_setup_async_callback(DAPL_IA *ia_ptr, + DAPL_ASYNC_HANDLER_TYPE handler_type, + ib_async_handler_t callback, void *context); -DAT_RETURN dapl_ib_cq_alloc(DAPL_IA * ia_ptr, - DAPL_EVD * evd_ptr, - DAT_COUNT * cqlen, ib_comp_handle_t callback); +u32 dapl_ib_cq_alloc(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr, DAT_COUNT *cqlen, + ib_comp_handle_t callback); -DAT_RETURN dapl_ib_cq_free(DAPL_IA * ia_ptr, DAPL_EVD * evd_ptr); +u32 dapl_ib_cq_free(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr); -DAT_RETURN dapl_set_cq_notify(DAPL_IA * ia_ptr, DAPL_EVD * evd_ptr); +u32 dapl_set_cq_notify(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr); -DAT_RETURN dapl_ib_cq_resize(DAPL_IA * ia_ptr, - DAPL_EVD * evd_ptr, DAT_COUNT * cqlen); +u32 dapl_ib_cq_resize(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr, DAT_COUNT *cqlen); -DAT_RETURN dapl_ib_pd_alloc(DAPL_IA * ia_ptr, DAPL_PZ * pz); +u32 dapl_ib_pd_alloc(DAPL_IA *ia_ptr, DAPL_PZ *pz); -DAT_RETURN dapl_ib_pd_free(DAPL_PZ * pz); +u32 dapl_ib_pd_free(DAPL_PZ *pz); -DAT_RETURN dapl_ib_mr_register(DAPL_IA * ia_ptr, - DAPL_LMR * lmr, - DAT_PVOID virt_addr, - DAT_VLEN length, DAT_MEM_PRIV_FLAGS privileges); +u32 dapl_ib_mr_register(DAPL_IA *ia_ptr, DAPL_LMR *lmr, DAT_PVOID virt_addr, + DAT_VLEN length, DAT_MEM_PRIV_FLAGS privileges); -DAT_RETURN dapl_ib_mr_register_physical(DAPL_IA * ia_ptr, - DAPL_LMR * lmr, - DAT_PVOID phys_addr, - DAT_VLEN length, - DAT_MEM_PRIV_FLAGS privileges); +u32 dapl_ib_mr_register_physical(DAPL_IA *ia_ptr, DAPL_LMR *lmr, + DAT_PVOID phys_addr, DAT_VLEN length, + DAT_MEM_PRIV_FLAGS privileges); -DAT_RETURN dapl_ib_mr_deregister(DAPL_LMR * lmr); +u32 dapl_ib_mr_deregister(DAPL_LMR *lmr); -DAT_RETURN dapl_ib_mr_register_shared(DAPL_IA * ia_ptr, - DAPL_LMR * lmr, - DAT_MEM_PRIV_FLAGS privileges); +u32 dapl_ib_mr_register_shared(DAPL_IA *ia_ptr, DAPL_LMR *lmr, + DAT_MEM_PRIV_FLAGS privileges); -DAT_RETURN dapl_ib_mw_alloc(DAPL_RMR * rmr); +u32 dapl_ib_mw_alloc(DAPL_RMR *rmr); -DAT_RETURN dapl_ib_mw_free(DAPL_RMR * rmr); +u32 dapl_ib_mw_free(DAPL_RMR *rmr); -DAT_RETURN dapl_ib_mw_bind(DAPL_RMR * rmr, - DAPL_LMR * lmr, - DAPL_EP * ep, - DAPL_COOKIE * cookie, - DAT_VADDR virtual_address, - DAT_VLEN length, - DAT_MEM_PRIV_FLAGS mem_priv, - boolean_t is_signaled); +u32 dapl_ib_mw_bind(DAPL_RMR *rmr, DAPL_LMR *lmr, DAPL_EP *ep, + DAPL_COOKIE *cookie, DAT_VADDR virtual_address, + DAT_VLEN length, DAT_MEM_PRIV_FLAGS mem_priv, + boolean_t is_signaled); -DAT_RETURN dapl_ib_mw_unbind(DAPL_RMR * rmr, - DAPL_EP * ep, - DAPL_COOKIE * cookie, boolean_t is_signaled); +u32 dapl_ib_mw_unbind(DAPL_RMR *rmr, DAPL_EP *ep, DAPL_COOKIE *cookie, + boolean_t is_signaled); -DAT_RETURN dapl_ib_query_hca(DAPL_HCA * hca_ptr, - struct dat_ia_attr *ia_attr, - struct dat_ep_attr *ep_attr, - struct sockaddr_in6 *ip_addr); +u32 dapl_ib_query_hca(DAPL_HCA *hca_ptr, struct dat_ia_attr *ia_attr, + struct dat_ep_attr *ep_attr, + struct sockaddr_in6 *ip_addr); -DAT_RETURN dapl_ib_completion_poll(DAPL_HCA * hca_ptr, - DAPL_EVD * evd_ptr, - ib_work_completion_t * cqe_ptr); +u32 dapl_ib_completion_poll(DAPL_HCA *hca_ptr, DAPL_EVD *evd_ptr, + ib_work_completion_t *cqe_ptr); -DAT_RETURN dapl_ib_completion_notify(DAPL_EVD * evd_ptr, - ib_notification_type_t type); +u32 dapl_ib_completion_notify(DAPL_EVD *evd_ptr, ib_notification_type_t type); -DAT_DTO_COMPLETION_STATUS dapl_ib_get_dto_status(ib_work_completion_t * - cqe_ptr); +DAT_DTO_COMPLETION_STATUS dapl_ib_get_dto_status(ib_work_completion_t *cqe_ptr); void dapl_ib_reinit_ep(DAPL_EP * ep_ptr); @@ -162,17 +143,16 @@ void dapl_ib_disconnect_clean(DAPL_EP * boolean_t passive, const ib_cm_events_t ib_cm_event); -DAT_RETURN dapl_ib_get_async_event(ib_error_record_t * cause_ptr, - DAT_EVENT_NUMBER * async_event); +u32 dapl_ib_get_async_event(ib_error_record_t *cause_ptr, + DAT_EVENT_NUMBER *async_event); DAT_EVENT_NUMBER dapl_ib_get_dat_event(const ib_cm_events_t ib_cm_event, boolean_t active); ib_cm_events_t dapl_ib_get_cm_event(DAT_EVENT_NUMBER dat_event_num); -DAT_RETURN dapl_ib_cm_remote_addr(DAT_HANDLE dat_handle, - struct sockaddr_in6 *remote_ia_address); - +u32 dapl_ib_cm_remote_addr(DAT_HANDLE dat_handle, + struct sockaddr_in6 *remote_ia_address); /* * Values for provider struct dat_named_attr diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_cno_util.c linux-kernel2/dat-provider/dapl_cno_util.c --- linux-kernel/dat-provider/dapl_cno_util.c 2005-05-11 08:28:12.472019000 -0700 +++ linux-kernel2/dat-provider/dapl_cno_util.c 2005-05-11 11:37:30.214963000 -0700 @@ -111,11 +111,9 @@ dapl_cno_trigger ( DAPL_CNO *cno_ptr, DAPL_EVD *evd_ptr) { - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; struct dat_event event; - dat_status = DAT_SUCCESS; - dapl_os_assert (cno_ptr->header.magic == DAPL_MAGIC_CNO); /* The spec allows NULL EVDs. kDAPL doesn't have CNOs, they * are strictly used behind the scenes diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_cookie.c linux-kernel2/dat-provider/dapl_cookie.c --- linux-kernel/dat-provider/dapl_cookie.c 2005-04-27 11:38:34.700014000 -0700 +++ linux-kernel2/dat-provider/dapl_cookie.c 2005-05-11 11:41:23.413960000 -0700 @@ -84,9 +84,9 @@ * * *********************************************************************/ -DAT_RETURN dapl_cb_get(DAPL_COOKIE_BUFFER * buffer, DAPL_COOKIE ** cookie_ptr); +u32 dapl_cb_get(DAPL_COOKIE_BUFFER *buffer, DAPL_COOKIE **cookie_ptr); -DAT_RETURN dapl_cb_put(DAPL_COOKIE_BUFFER * buffer, DAPL_COOKIE * cookie); +u32 dapl_cb_put(DAPL_COOKIE_BUFFER *buffer, DAPL_COOKIE *cookie); /********************************************************************* * * @@ -113,8 +113,7 @@ DAT_RETURN dapl_cb_put(DAPL_COOKIE_BUFFE * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN -dapl_cb_create(DAPL_COOKIE_BUFFER * buffer, DAPL_EP * ep, DAT_COUNT size) +u32 dapl_cb_create(DAPL_COOKIE_BUFFER *buffer, DAPL_EP *ep, DAT_COUNT size) { DAT_COUNT i; @@ -179,9 +178,9 @@ void dapl_cb_free(DAPL_COOKIE_BUFFER * b * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN dapl_cb_get(DAPL_COOKIE_BUFFER * buffer, DAPL_COOKIE ** cookie_ptr) +u32 dapl_cb_get(DAPL_COOKIE_BUFFER *buffer, DAPL_COOKIE **cookie_ptr) { - DAT_RETURN dat_status; + u32 dat_status; DAT_COUNT new_head; dapl_os_assert(NULL != cookie_ptr); @@ -218,7 +217,7 @@ DAT_RETURN dapl_cb_get(DAPL_COOKIE_BUFFE * DAT_INSUFFICIENT_EMPTY * */ -DAT_RETURN dapl_cb_put(DAPL_COOKIE_BUFFER * buffer, DAPL_COOKIE * cookie) +u32 dapl_cb_put(DAPL_COOKIE_BUFFER *buffer, DAPL_COOKIE *cookie) { atomic_set(&buffer->tail, cookie->index); @@ -243,13 +242,11 @@ DAT_RETURN dapl_cb_put(DAPL_COOKIE_BUFFE * DAT_INSUFFICIENT_EMPTY * */ -DAT_RETURN -dapl_rmr_cookie_alloc(DAPL_COOKIE_BUFFER * buffer, - DAPL_RMR * rmr, - DAT_RMR_COOKIE user_cookie, DAPL_COOKIE ** cookie_ptr) +u32 dapl_rmr_cookie_alloc(DAPL_COOKIE_BUFFER *buffer, DAPL_RMR *rmr, + DAT_RMR_COOKIE user_cookie, DAPL_COOKIE **cookie_ptr) { DAPL_COOKIE *cookie; - DAT_RETURN dat_status; + u32 dat_status; if (DAT_SUCCESS != dapl_cb_get(buffer, &cookie)) { *cookie_ptr = NULL; @@ -287,10 +284,8 @@ dapl_rmr_cookie_alloc(DAPL_COOKIE_BUFFER * DAT_INSUFFICIENT_EMPTY * */ -DAT_RETURN -dapl_dto_cookie_alloc(DAPL_COOKIE_BUFFER * buffer, - DAPL_DTO_TYPE type, - DAT_DTO_COOKIE user_cookie, DAPL_COOKIE ** cookie_ptr) +u32 dapl_dto_cookie_alloc(DAPL_COOKIE_BUFFER *buffer, DAPL_DTO_TYPE type, + DAT_DTO_COOKIE user_cookie, DAPL_COOKIE **cookie_ptr) { DAPL_COOKIE *cookie; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_cookie.h linux-kernel2/dat-provider/dapl_cookie.h --- linux-kernel/dat-provider/dapl_cookie.h 2005-04-20 12:42:06.513011000 -0700 +++ linux-kernel2/dat-provider/dapl_cookie.h 2005-05-11 11:42:25.658973000 -0700 @@ -36,20 +36,18 @@ #include "dapl.h" -extern DAT_RETURN -dapl_cb_create(DAPL_COOKIE_BUFFER * buffer, DAPL_EP * ep, DAT_COUNT size); +extern u32 dapl_cb_create(DAPL_COOKIE_BUFFER *buffer, DAPL_EP *ep, + DAT_COUNT size); -extern void dapl_cb_free(DAPL_COOKIE_BUFFER * buffer); +extern void dapl_cb_free(DAPL_COOKIE_BUFFER *buffer); -extern DAT_RETURN -dapl_rmr_cookie_alloc(DAPL_COOKIE_BUFFER * buffer, - DAPL_RMR * rmr, - DAT_RMR_COOKIE user_cookie, DAPL_COOKIE ** cookie_ptr); - -extern DAT_RETURN -dapl_dto_cookie_alloc(DAPL_COOKIE_BUFFER * buffer, - DAPL_DTO_TYPE type, - DAT_DTO_COOKIE user_cookie, DAPL_COOKIE ** cookie_ptr); +extern u32 dapl_rmr_cookie_alloc(DAPL_COOKIE_BUFFER *buffer, DAPL_RMR *rmr, + DAT_RMR_COOKIE user_cookie, + DAPL_COOKIE **cookie_ptr); + +extern u32 dapl_dto_cookie_alloc(DAPL_COOKIE_BUFFER *buffer, + DAPL_DTO_TYPE type, DAT_DTO_COOKIE user_cookie, + DAPL_COOKIE **cookie_ptr); extern void dapl_cookie_dealloc(DAPL_COOKIE_BUFFER * buffer, DAPL_COOKIE * cookie); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_cr_accept.c linux-kernel2/dat-provider/dapl_cr_accept.c --- linux-kernel/dat-provider/dapl_cr_accept.c 2005-05-02 08:48:17.581023000 -0700 +++ linux-kernel2/dat-provider/dapl_cr_accept.c 2005-05-11 11:42:47.385963000 -0700 @@ -39,13 +39,11 @@ * Establish a connection between active side (remote endpoint) * and passive side (local endpoint). */ -DAT_RETURN -dapl_cr_accept(DAT_CR_HANDLE cr_handle, - DAT_EP_HANDLE ep_handle, - DAT_COUNT private_data_size, const DAT_PVOID private_data) +u32 dapl_cr_accept(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, + DAT_COUNT private_data_size, const DAT_PVOID private_data) { DAPL_EP *ep_ptr; - DAT_RETURN dat_status; + u32 dat_status; DAPL_CR *cr_ptr; DAT_EP_STATE entry_ep_state; DAT_EP_HANDLE entry_ep_handle; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_cr_callback.c linux-kernel2/dat-provider/dapl_cr_callback.c --- linux-kernel/dat-provider/dapl_cr_callback.c 2005-05-11 08:28:11.511012000 -0700 +++ linux-kernel2/dat-provider/dapl_cr_callback.c 2005-05-11 11:43:26.176964000 -0700 @@ -45,9 +45,8 @@ /* * Prototypes */ -DAT_RETURN dapl_connection_request(ib_cm_handle_t ib_cm_handle, - DAPL_SP * sp_ptr, - const void *prd_ptr, DAPL_EVD * evd_ptr); +u32 dapl_connection_request(ib_cm_handle_t ib_cm_handle, DAPL_SP *sp_ptr, + const void *prd_ptr, DAPL_EVD *evd_ptr); DAPL_EP *dapl_get_sp_ep(ib_cm_handle_t ib_cm_handle, DAPL_SP * sp_ptr, DAT_EVENT_NUMBER dat_event_num); @@ -65,7 +64,7 @@ void dapl_cr_callback(ib_cm_handle_t ib_ DAPL_EVD *evd_ptr; DAPL_SP *sp_ptr; DAT_EVENT_NUMBER dat_event_num; - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_CM | DAPL_DBG_TYPE_CALLBACK, "--> dapl_cr_callback! context: %p event: %x cm_handle %p\n", @@ -266,12 +265,10 @@ void dapl_cr_callback(ib_cm_handle_t ib_ * Create a CR record and link it on to the SP so we can update it * and free it later. Create an EP if specified by the PSP flags. */ -DAT_RETURN -dapl_connection_request(ib_cm_handle_t ib_cm_handle, - DAPL_SP * sp_ptr, - const void *prd_ptr, DAPL_EVD * evd_ptr) +u32 dapl_connection_request(ib_cm_handle_t ib_cm_handle, DAPL_SP *sp_ptr, + const void *prd_ptr, DAPL_EVD *evd_ptr) { - DAT_RETURN dat_status; + u32 dat_status; DAPL_CR *cr_ptr; DAPL_EP *ep_ptr; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_cr_handoff.c linux-kernel2/dat-provider/dapl_cr_handoff.c --- linux-kernel/dat-provider/dapl_cr_handoff.c 2005-04-20 12:42:07.958024000 -0700 +++ linux-kernel2/dat-provider/dapl_cr_handoff.c 2005-05-11 11:16:06.189964000 -0700 @@ -37,7 +37,7 @@ * Hand the connection request to another Sevice pont specified by the * Connectin Qualifier. */ -DAT_RETURN dapl_cr_handoff(DAT_CR_HANDLE cr_handle, DAT_CONN_QUAL cr_handoff) +u32 dapl_cr_handoff(DAT_CR_HANDLE cr_handle, DAT_CONN_QUAL cr_handoff) { /* handoff */ return DAT_ERROR(DAT_NOT_IMPLEMENTED, 0); } diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_cr_query.c linux-kernel2/dat-provider/dapl_cr_query.c --- linux-kernel/dat-provider/dapl_cr_query.c 2005-05-10 08:13:29.281019000 -0700 +++ linux-kernel2/dat-provider/dapl_cr_query.c 2005-05-11 11:16:06.193960000 -0700 @@ -34,10 +34,10 @@ #include "dapl.h" #include "dapl_adapter_util.h" -DAT_RETURN dapl_cr_query(DAT_CR_HANDLE cr_handle, struct dat_cr_param *cr_param) +u32 dapl_cr_query(DAT_CR_HANDLE cr_handle, struct dat_cr_param *cr_param) { DAPL_CR *cr_ptr; - DAT_RETURN status; + u32 status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_cr_query (%p, %x, %p)\n", diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_cr_reject.c linux-kernel2/dat-provider/dapl_cr_reject.c --- linux-kernel/dat-provider/dapl_cr_reject.c 2005-04-20 12:42:07.641003000 -0700 +++ linux-kernel2/dat-provider/dapl_cr_reject.c 2005-05-11 11:16:06.196963000 -0700 @@ -40,14 +40,14 @@ * Reject a connection request from the active remote side requesting * an Endpoint. */ -DAT_RETURN dapl_cr_reject(DAT_CR_HANDLE cr_handle) +u32 dapl_cr_reject(DAT_CR_HANDLE cr_handle) { DAPL_CR *cr_ptr; DAPL_EP *ep_ptr; DAT_EP_STATE entry_ep_state; DAT_EP_HANDLE entry_ep_handle; DAPL_SP *sp_ptr; - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_cr_reject (%p)\n", cr_handle); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_connect.c linux-kernel2/dat-provider/dapl_ep_connect.c --- linux-kernel/dat-provider/dapl_ep_connect.c 2005-05-10 08:13:28.285058000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_connect.c 2005-05-11 11:58:33.206963000 -0700 @@ -42,18 +42,15 @@ * and a remote Endpoint. This operation is used by the active/client * side of a connection */ -DAT_RETURN -dapl_ep_connect(DAT_EP_HANDLE ep_handle, - DAT_IA_ADDRESS_PTR remote_ia_address, - DAT_CONN_QUAL remote_conn_qual, - DAT_TIMEOUT timeout, - DAT_COUNT private_data_size, - const DAT_PVOID private_data, - DAT_QOS qos, DAT_CONNECT_FLAGS connect_flags) +u32 dapl_ep_connect(DAT_EP_HANDLE ep_handle, + DAT_IA_ADDRESS_PTR remote_ia_address, + DAT_CONN_QUAL remote_conn_qual, DAT_TIMEOUT timeout, + DAT_COUNT private_data_size, const DAT_PVOID private_data, + DAT_QOS qos, DAT_CONNECT_FLAGS connect_flags) { DAPL_EP *ep_ptr; DAPL_EP alloc_ep; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API | DAPL_DBG_TYPE_CM, "dapl_ep_connect (%p, {%u.%u.%u.%u}, %X, %d, %d, %p, %x, %x)\n", @@ -66,7 +63,6 @@ dapl_ep_connect(DAT_EP_HANDLE ep_handle, timeout, private_data_size, private_data, qos, connect_flags); - dat_status = DAT_SUCCESS; ep_ptr = (DAPL_EP *) ep_handle; /* diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_create.c linux-kernel2/dat-provider/dapl_ep_create.c --- linux-kernel/dat-provider/dapl_ep_create.c 2005-05-10 08:13:28.361017000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_create.c 2005-05-11 11:59:08.236964000 -0700 @@ -63,18 +63,16 @@ * DAT_INVALID_ATTRIBUTE * DAT_MODEL_NOT_SUPPORTED */ -DAT_RETURN -dapl_ep_create(DAT_IA_HANDLE ia_handle, - DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE recv_evd_handle, - DAT_EVD_HANDLE request_evd_handle, - DAT_EVD_HANDLE connect_evd_handle, - const struct dat_ep_attr *ep_attr, DAT_EP_HANDLE * ep_handle) +u32 dapl_ep_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, + DAT_EVD_HANDLE recv_evd_handle, + DAT_EVD_HANDLE request_evd_handle, + DAT_EVD_HANDLE connect_evd_handle, + const struct dat_ep_attr *ep_attr, DAT_EP_HANDLE *ep_handle) { DAPL_IA *ia_ptr; DAPL_EP *ep_ptr; struct dat_ep_attr ep_attr_limit; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_create (%p, %p, %p, %p, %p, %p, %p)\n", @@ -85,7 +83,6 @@ dapl_ep_create(DAT_IA_HANDLE ia_handle, connect_evd_handle, ep_attr, ep_handle); ia_ptr = (DAPL_IA *) ia_handle; - dat_status = DAT_SUCCESS; /* * Verify parameters diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_create_with_srq.c linux-kernel2/dat-provider/dapl_ep_create_with_srq.c --- linux-kernel/dat-provider/dapl_ep_create_with_srq.c 2005-05-10 08:13:29.078008000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_create_with_srq.c 2005-05-11 12:00:02.787969000 -0700 @@ -69,22 +69,19 @@ * DAT_INVALID_ATTRIBUTE * DAT_MODEL_NOT_SUPPORTED */ -DAT_RETURN -dapl_ep_create_with_srq(DAT_IA_HANDLE ia_handle, - DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE recv_evd_handle, - DAT_EVD_HANDLE request_evd_handle, - DAT_EVD_HANDLE connect_evd_handle, - DAT_SRQ_HANDLE srq_handle, - const struct dat_ep_attr *ep_attr, - DAT_EP_HANDLE * ep_handle) +u32 dapl_ep_create_with_srq(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, + DAT_EVD_HANDLE recv_evd_handle, + DAT_EVD_HANDLE request_evd_handle, + DAT_EVD_HANDLE connect_evd_handle, + DAT_SRQ_HANDLE srq_handle, + const struct dat_ep_attr *ep_attr, + DAT_EP_HANDLE *ep_handle) { DAPL_IA *ia_ptr; DAPL_EP *ep_ptr; struct dat_ep_attr ep_attr_limit; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; - dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_create_with_srq (%p, %p, %p, %p, %p, %p, %p %p)\n", ia_handle, diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_disconnect.c linux-kernel2/dat-provider/dapl_ep_disconnect.c --- linux-kernel/dat-provider/dapl_ep_disconnect.c 2005-05-02 08:48:17.787970000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_disconnect.c 2005-05-11 12:00:23.524963000 -0700 @@ -63,14 +63,14 @@ * DAT_INSUFFICIENT_RESOURCES * DAT_INVALID_PARAMETER */ -DAT_RETURN -dapl_ep_disconnect(DAT_EP_HANDLE ep_handle, DAT_CLOSE_FLAGS disconnect_flags) +u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_handle, + DAT_CLOSE_FLAGS disconnect_flags) { DAPL_EP *ep_ptr; DAPL_EVD *evd_ptr; DAPL_CR *cr_ptr; ib_cm_events_t ib_cm_event; - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API | DAPL_DBG_TYPE_CM, "dapl_ep_disconnect (%p, %x)\n", diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_dup_connect.c linux-kernel2/dat-provider/dapl_ep_dup_connect.c --- linux-kernel/dat-provider/dapl_ep_dup_connect.c 2005-04-20 12:42:08.547017000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_dup_connect.c 2005-05-11 12:00:47.734963000 -0700 @@ -69,15 +69,12 @@ * DAT_INVALID_STATE * DAT_MODEL_NOT_SUPPORTED */ -DAT_RETURN -dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, - DAT_EP_HANDLE ep_dup_handle, - DAT_TIMEOUT timeout, - DAT_COUNT private_data_size, - const DAT_PVOID private_data, DAT_QOS qos) +u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, DAT_EP_HANDLE ep_dup_handle, + DAT_TIMEOUT timeout, DAT_COUNT private_data_size, + const DAT_PVOID private_data, DAT_QOS qos) { DAPL_EP *ep_dup_ptr; - DAT_RETURN dat_status; + u32 dat_status; DAT_IA_ADDRESS_PTR remote_ia_address_ptr; DAT_CONN_QUAL remote_conn_qual; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_free.c linux-kernel2/dat-provider/dapl_ep_free.c --- linux-kernel/dat-provider/dapl_ep_free.c 2005-05-10 08:13:28.517011000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_free.c 2005-05-11 11:16:06.224962000 -0700 @@ -61,13 +61,13 @@ * DAT_INVALID_PARAMETER * DAT_INVALID_STATE */ -DAT_RETURN dapl_ep_free(DAT_EP_HANDLE ep_handle) +u32 dapl_ep_free(DAT_EP_HANDLE ep_handle) { DAPL_EP *ep_ptr; DAPL_IA *ia_ptr; struct dat_ep_param *param; ib_qp_state_t save_qp_state; - DAT_RETURN dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_free (%p)\n", ep_handle); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_get_status.c linux-kernel2/dat-provider/dapl_ep_get_status.c --- linux-kernel/dat-provider/dapl_ep_get_status.c 2005-05-11 08:28:11.861017000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_get_status.c 2005-05-11 12:02:16.057971000 -0700 @@ -59,20 +59,17 @@ * DAT_SUCCESS * DAT_INVALID_PARAMETER */ -DAT_RETURN -dapl_ep_get_status(DAT_EP_HANDLE ep_handle, - DAT_EP_STATE * ep_state, - boolean_t *in_dto_idle, boolean_t *out_dto_idle) +u32 dapl_ep_get_status(DAT_EP_HANDLE ep_handle, DAT_EP_STATE *ep_state, + boolean_t *in_dto_idle, boolean_t *out_dto_idle) { DAPL_EP *ep_ptr; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_get_status (%p, %p, %p, %p)\n", ep_handle, ep_state, in_dto_idle, out_dto_idle); ep_ptr = (DAPL_EP *) ep_handle; - dat_status = DAT_SUCCESS; /* * Verify parameter & state diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_modify.c linux-kernel2/dat-provider/dapl_ep_modify.c --- linux-kernel/dat-provider/dapl_ep_modify.c 2005-05-11 08:28:12.424000000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_modify.c 2005-05-11 12:04:01.023960000 -0700 @@ -52,7 +52,7 @@ * */ -static inline DAT_RETURN +static inline u32 dapl_ep_modify_validate_parameters(DAT_EP_HANDLE ep_handle, DAT_EP_PARAM_MASK mask, const struct dat_ep_param *ep_param, @@ -65,11 +65,10 @@ dapl_ep_modify_validate_parameters(DAT_E struct dat_ep_attr ep_attr; struct dat_ep_attr ep_attr_limit; struct dat_ep_attr ep_attr_request; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; *ia_ptr = NULL; *ep_ptr = NULL; - dat_status = DAT_SUCCESS; if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) { dat_status = @@ -309,8 +308,8 @@ dapl_ep_modify_validate_parameters(DAT_E * DAT_INVALID_ATTRIBUTE * DAT_INVALID_STATE */ -DAT_RETURN dapl_ep_modify(DAT_EP_HANDLE ep_handle, DAT_EP_PARAM_MASK mask, - const struct dat_ep_param *ep_param) +u32 dapl_ep_modify(DAT_EP_HANDLE ep_handle, DAT_EP_PARAM_MASK mask, + const struct dat_ep_param *ep_param) { DAPL_IA *ia; DAPL_EP *ep1, *ep2; @@ -319,7 +318,7 @@ DAT_RETURN dapl_ep_modify(DAT_EP_HANDLE DAPL_EP alloc_ep; /* Holder for resources. */ DAPL_PZ *tmp_pz; DAPL_EVD *tmp_evd; - DAT_RETURN dat_status; + u32 dat_status; /* Flag indicating we've allocated a new one of these. */ boolean_t qp_allocated = FALSE; @@ -585,7 +584,7 @@ DAT_RETURN dapl_ep_modify(DAT_EP_HANDLE bail: if (qp_allocated) { - DAT_RETURN local_dat_status; + u32 local_dat_status; if (dat_status != DAT_SUCCESS || !qp_used) { local_dat_status = dapl_ib_qp_free(ia, &alloc_ep); } else { diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_post_rdma_read.c linux-kernel2/dat-provider/dapl_ep_post_rdma_read.c --- linux-kernel/dat-provider/dapl_ep_post_rdma_read.c 2005-05-10 08:13:29.096006000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_post_rdma_read.c 2005-05-11 12:04:41.106971000 -0700 @@ -66,16 +66,14 @@ * DAT_PROTECTION_VIOLATION * DAT_PRIVILEGES_VIOLATION */ -DAT_RETURN -dapl_ep_post_rdma_read(DAT_EP_HANDLE ep_handle, - DAT_COUNT num_segments, - struct dat_lmr_triplet *local_iov, - DAT_DTO_COOKIE user_cookie, - const struct dat_rmr_triplet *remote_iov, - DAT_COMPLETION_FLAGS completion_flags) +u32 dapl_ep_post_rdma_read(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, + struct dat_lmr_triplet *local_iov, + DAT_DTO_COOKIE user_cookie, + const struct dat_rmr_triplet *remote_iov, + DAT_COMPLETION_FLAGS completion_flags) { - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_post_rdma_read (%p, %d, %p, %p, %p, %x)\n", diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_post_rdma_write.c linux-kernel2/dat-provider/dapl_ep_post_rdma_write.c --- linux-kernel/dat-provider/dapl_ep_post_rdma_write.c 2005-05-10 08:13:28.342024000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_post_rdma_write.c 2005-05-11 12:05:08.228968000 -0700 @@ -66,15 +66,13 @@ * DAT_PROTECTION_VIOLATION * DAT_PRIVILEGES_VIOLATION */ -DAT_RETURN -dapl_ep_post_rdma_write(DAT_EP_HANDLE ep_handle, - DAT_COUNT num_segments, - struct dat_lmr_triplet *local_iov, - DAT_DTO_COOKIE user_cookie, - const struct dat_rmr_triplet *remote_iov, - DAT_COMPLETION_FLAGS completion_flags) +u32 dapl_ep_post_rdma_write(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, + struct dat_lmr_triplet *local_iov, + DAT_DTO_COOKIE user_cookie, + const struct dat_rmr_triplet *remote_iov, + DAT_COMPLETION_FLAGS completion_flags) { - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_post_rdma_write (%p, %d, %p, %P, %p, %x)\n", diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_post_recv.c linux-kernel2/dat-provider/dapl_ep_post_recv.c --- linux-kernel/dat-provider/dapl_ep_post_recv.c 2005-05-10 08:13:29.263024000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_post_recv.c 2005-05-11 12:05:45.066971000 -0700 @@ -66,16 +66,14 @@ * DAT_PROTECTION_VIOLATION * DAT_PROVILEGES_VIOLATION */ -DAT_RETURN -dapl_ep_post_recv(DAT_EP_HANDLE ep_handle, - DAT_COUNT num_segments, - struct dat_lmr_triplet *local_iov, - DAT_DTO_COOKIE user_cookie, - DAT_COMPLETION_FLAGS completion_flags) +u32 dapl_ep_post_recv(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, + struct dat_lmr_triplet *local_iov, + DAT_DTO_COOKIE user_cookie, + DAT_COMPLETION_FLAGS completion_flags) { DAPL_EP *ep_ptr; DAPL_COOKIE *cookie; - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_post_recv (%p, %d, %p, %P, %x)\n", diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_post_send.c linux-kernel2/dat-provider/dapl_ep_post_send.c --- linux-kernel/dat-provider/dapl_ep_post_send.c 2005-05-10 08:13:28.702022000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_post_send.c 2005-05-11 12:06:13.573960000 -0700 @@ -63,15 +63,13 @@ * DAT_PROTECTION_VIOLATION * DAT_PRIVILEGES_VIOLATION */ -DAT_RETURN -dapl_ep_post_send(DAT_EP_HANDLE ep_handle, - DAT_COUNT num_segments, - struct dat_lmr_triplet *local_iov, - DAT_DTO_COOKIE user_cookie, - DAT_COMPLETION_FLAGS completion_flags) +u32 dapl_ep_post_send(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, + struct dat_lmr_triplet *local_iov, + DAT_DTO_COOKIE user_cookie, + DAT_COMPLETION_FLAGS completion_flags) { struct dat_rmr_triplet remote_iov = { 0, 0, 0, 0 }; - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_post_send (%p, %d, %p, %P, %x)\n", diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_query.c linux-kernel2/dat-provider/dapl_ep_query.c --- linux-kernel/dat-provider/dapl_ep_query.c 2005-05-10 08:13:28.778009000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_query.c 2005-05-11 12:06:40.494963000 -0700 @@ -57,17 +57,15 @@ * DAT_SUCCESS * DAT_INVALID_PARAMETER */ -DAT_RETURN -dapl_ep_query(DAT_EP_HANDLE ep_handle, struct dat_ep_param *ep_param) +u32 dapl_ep_query(DAT_EP_HANDLE ep_handle, struct dat_ep_param *ep_param) { DAPL_EP *ep_ptr; - DAT_RETURN status; + u32 status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_query (%p, %p)\n", ep_handle, ep_param); - status = DAT_SUCCESS; ep_ptr = (DAPL_EP *) ep_handle; /* diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_recv_query.c linux-kernel2/dat-provider/dapl_ep_recv_query.c --- linux-kernel/dat-provider/dapl_ep_recv_query.c 2005-05-04 08:19:32.717975000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_recv_query.c 2005-05-11 12:07:41.941768000 -0700 @@ -54,14 +54,11 @@ * DAT_INVALID_HANDLE * DAT_MODEL_NOT_SUPPORTED */ -DAT_RETURN -dapl_ep_recv_query(DAT_EP_HANDLE ep_handle, - DAT_COUNT * nbufs_allocate, DAT_COUNT * bufs_alloc_span) +u32 dapl_ep_recv_query(DAT_EP_HANDLE ep_handle, + DAT_COUNT *nbufs_allocate, DAT_COUNT *bufs_alloc_span) { DAPL_EP *ep_ptr; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_recv_query (%p, %p, %p)\n", ep_handle, nbufs_allocate, bufs_alloc_span); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_reset.c linux-kernel2/dat-provider/dapl_ep_reset.c --- linux-kernel/dat-provider/dapl_ep_reset.c 2005-04-20 12:42:07.660015000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_reset.c 2005-05-11 12:08:05.331962000 -0700 @@ -61,12 +61,10 @@ * DAT_INVALID_PARAMETER * DAT_INVALID_STATE */ -DAT_RETURN dapl_ep_reset(DAT_EP_HANDLE ep_handle) +u32 dapl_ep_reset(DAT_EP_HANDLE ep_handle) { DAPL_EP *ep_ptr; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; ep_ptr = (DAPL_EP *) ep_handle; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_set_watermark.c linux-kernel2/dat-provider/dapl_ep_set_watermark.c --- linux-kernel/dat-provider/dapl_ep_set_watermark.c 2005-04-20 12:42:08.865015000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_set_watermark.c 2005-05-11 12:08:28.540010000 -0700 @@ -59,15 +59,12 @@ * DAT_INVALID_HANDLE * DAT_MODEL_NOT_SUPPORTED */ -DAT_RETURN -dapl_ep_set_watermark(DAT_EP_HANDLE ep_handle, - DAT_COUNT soft_high_watermark, - DAT_COUNT hard_high_watermark) +u32 dapl_ep_set_watermark(DAT_EP_HANDLE ep_handle, + DAT_COUNT soft_high_watermark, + DAT_COUNT hard_high_watermark) { DAPL_EP *ep_ptr; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_set_watermark (%p, %d, %d)\n", ep_handle, soft_high_watermark, hard_high_watermark); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_util.c linux-kernel2/dat-provider/dapl_ep_util.c --- linux-kernel/dat-provider/dapl_ep_util.c 2005-05-11 08:28:12.629009000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_util.c 2005-05-11 12:09:16.971011000 -0700 @@ -257,7 +257,7 @@ void dapl_ep_default_attrs(DAPL_EP * ep_ return; } -DAT_RETURN dapl_ep_check_recv_completion_flags(DAT_COMPLETION_FLAGS flags) +u32 dapl_ep_check_recv_completion_flags(DAT_COMPLETION_FLAGS flags) { /* @@ -276,23 +276,21 @@ DAT_RETURN dapl_ep_check_recv_completion return DAT_SUCCESS; } -DAT_RETURN dapl_ep_check_request_completion_flags(DAT_COMPLETION_FLAGS flags) +u32 dapl_ep_check_request_completion_flags(DAT_COMPLETION_FLAGS flags) { return DAT_SUCCESS; } -DAT_RETURN -dapl_ep_post_send_req(DAT_EP_HANDLE ep_handle, - DAT_COUNT num_segments, - struct dat_lmr_triplet *local_iov, - DAT_DTO_COOKIE user_cookie, - const struct dat_rmr_triplet *remote_iov, - DAT_COMPLETION_FLAGS completion_flags, - DAPL_DTO_TYPE dto_type, int op_type) +u32 dapl_ep_post_send_req(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, + struct dat_lmr_triplet *local_iov, + DAT_DTO_COOKIE user_cookie, + const struct dat_rmr_triplet *remote_iov, + DAT_COMPLETION_FLAGS completion_flags, + DAPL_DTO_TYPE dto_type, int op_type) { DAPL_EP *ep_ptr; DAPL_COOKIE *cookie; - DAT_RETURN dat_status; + u32 dat_status; if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) { dat_status = @@ -373,9 +371,9 @@ void dapl_ep_timeout(uintptr_t arg) * Return the INVALID_STATE connection subtype associated with an * INVALID_STATE on an EP. Strictly for error reporting. */ -DAT_RETURN_SUBTYPE dapl_ep_state_subtype(DAPL_EP * ep_ptr) +enum dat_return_subtype dapl_ep_state_subtype(DAPL_EP * ep_ptr) { - DAT_RETURN_SUBTYPE dat_status; + enum dat_return_subtype dat_status; switch (ep_ptr->param.ep_state) { case DAT_EP_STATE_UNCONNECTED: diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_util.h linux-kernel2/dat-provider/dapl_ep_util.h --- linux-kernel/dat-provider/dapl_ep_util.h 2005-05-10 08:13:29.059014000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_util.h 2005-05-11 12:09:54.401014000 -0700 @@ -46,23 +46,20 @@ extern DAPL_EP *dapl_ep_alloc(DAPL_IA * extern void dapl_ep_dealloc(DAPL_EP * ep_ptr); -extern DAT_RETURN -dapl_ep_check_recv_completion_flags(DAT_COMPLETION_FLAGS flags); +extern u32 dapl_ep_check_recv_completion_flags(DAT_COMPLETION_FLAGS flags); -extern DAT_RETURN -dapl_ep_check_request_completion_flags(DAT_COMPLETION_FLAGS flags); +extern u32 dapl_ep_check_request_completion_flags(DAT_COMPLETION_FLAGS flags); -extern DAT_RETURN -dapl_ep_post_send_req(DAT_EP_HANDLE ep_handle, - DAT_COUNT num_segments, - struct dat_lmr_triplet *local_iov, - DAT_DTO_COOKIE user_cookie, - const struct dat_rmr_triplet *remote_iov, - DAT_COMPLETION_FLAGS completion_flags, - DAPL_DTO_TYPE dto_type, int op_type); +extern u32 dapl_ep_post_send_req(DAT_EP_HANDLE ep_handle, + DAT_COUNT num_segments, + struct dat_lmr_triplet *local_iov, + DAT_DTO_COOKIE user_cookie, + const struct dat_rmr_triplet *remote_iov, + DAT_COMPLETION_FLAGS completion_flags, + DAPL_DTO_TYPE dto_type, int op_type); void dapl_ep_timeout(uintptr_t arg); -DAT_RETURN_SUBTYPE dapl_ep_state_subtype(DAPL_EP * ep_ptr); +enum dat_return_subtype dapl_ep_state_subtype(DAPL_EP * ep_ptr); #endif /* DAPL_EP_UTIL_H */ diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_evd_connection_callb.c linux-kernel2/dat-provider/dapl_evd_connection_callb.c --- linux-kernel/dat-provider/dapl_evd_connection_callb.c 2005-05-11 08:28:11.694002000 -0700 +++ linux-kernel2/dat-provider/dapl_evd_connection_callb.c 2005-05-11 11:16:06.285963000 -0700 @@ -69,7 +69,7 @@ dapl_evd_connection_callback(ib_cm_handl DAPL_EP *ep_ptr; DAPL_EVD *evd_ptr; DAT_EVENT_NUMBER dat_event_num; - DAT_RETURN dat_status; + u32 dat_status; int private_data_size; dapl_dbg_log(DAPL_DBG_TYPE_CM | DAPL_DBG_TYPE_CALLBACK, diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_evd_cq_async_error_callb.c linux-kernel2/dat-provider/dapl_evd_cq_async_error_callb.c --- linux-kernel/dat-provider/dapl_evd_cq_async_error_callb.c 2005-04-20 12:42:06.412004000 -0700 +++ linux-kernel2/dat-provider/dapl_evd_cq_async_error_callb.c 2005-05-11 11:16:06.290966000 -0700 @@ -58,7 +58,7 @@ dapl_evd_cq_async_error_callback(ib_erro { DAPL_EVD *async_evd; DAPL_EVD *evd; - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK | DAPL_DBG_TYPE_EXCEPTION, "dapl_evd_cq_async_error_callback (%p, %p)\n", diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_evd_dequeue.c linux-kernel2/dat-provider/dapl_evd_dequeue.c --- linux-kernel/dat-provider/dapl_evd_dequeue.c 2005-05-10 08:13:28.796004000 -0700 +++ linux-kernel2/dat-provider/dapl_evd_dequeue.c 2005-05-11 12:11:09.256002000 -0700 @@ -62,17 +62,16 @@ * DAT_QUEUE_EMPTY */ -DAT_RETURN dapl_evd_dequeue(DAT_EVD_HANDLE evd_handle, struct dat_event *event) +u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_handle, struct dat_event *event) { DAPL_EVD *evd_ptr; struct dat_event *local_event; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_evd_dequeue (%p, %p)\n", evd_handle, event); evd_ptr = (DAPL_EVD *) evd_handle; - dat_status = DAT_SUCCESS; if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, 0); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_evd_dto_callb.c linux-kernel2/dat-provider/dapl_evd_dto_callb.c --- linux-kernel/dat-provider/dapl_evd_dto_callb.c 2005-05-09 11:07:01.636005000 -0700 +++ linux-kernel2/dat-provider/dapl_evd_dto_callb.c 2005-05-11 11:16:06.298964000 -0700 @@ -71,7 +71,7 @@ void dapl_evd_dto_callback(ib_cq_handle_t cq_handle, void *user_context) { DAPL_EVD *evd_ptr; - DAT_RETURN dat_status; + u32 dat_status; DAPL_EVD_STATE state; dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK, diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_evd_free.c linux-kernel2/dat-provider/dapl_evd_free.c --- linux-kernel/dat-provider/dapl_evd_free.c 2005-05-04 13:38:41.149012000 -0700 +++ linux-kernel2/dat-provider/dapl_evd_free.c 2005-05-11 12:12:01.796013000 -0700 @@ -39,15 +39,14 @@ /* * Destroy a specific instance of the Event Dispatcher */ -DAT_RETURN dapl_evd_free(DAT_EVD_HANDLE evd_handle) +u32 dapl_evd_free(DAT_EVD_HANDLE evd_handle) { DAPL_EVD *evd_ptr; DAPL_CNO *cno_ptr; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_evd_free (%p)\n", evd_handle); - dat_status = DAT_SUCCESS; evd_ptr = (DAPL_EVD *) evd_handle; if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_evd_kcreate.c linux-kernel2/dat-provider/dapl_evd_kcreate.c --- linux-kernel/dat-provider/dapl_evd_kcreate.c 2005-05-10 08:13:28.913024000 -0700 +++ linux-kernel2/dat-provider/dapl_evd_kcreate.c 2005-05-11 12:12:41.439041000 -0700 @@ -48,16 +48,15 @@ * even if it is not required. However, it will not be armed. */ -DAT_RETURN dapl_evd_kcreate(DAT_IA_HANDLE ia_handle, DAT_COUNT evd_min_qlen, - DAT_UPCALL_POLICY upcall_policy, /* ignore */ - const struct dat_upcall_object *upcall, - DAT_EVD_FLAGS evd_flags, - DAT_EVD_HANDLE * evd_handle) +u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_handle, DAT_COUNT evd_min_qlen, + DAT_UPCALL_POLICY upcall_policy, /* ignore */ + const struct dat_upcall_object *upcall, + DAT_EVD_FLAGS evd_flags, DAT_EVD_HANDLE *evd_handle) { DAPL_IA *ia_ptr; DAPL_EVD *evd_ptr; DAPL_CNO *cno_ptr; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; DAT_EVD_HANDLE evdhandle = (DAT_EVD_HANDLE) NULL; dapl_dbg_log(DAPL_DBG_TYPE_API, @@ -69,7 +68,6 @@ DAT_RETURN dapl_evd_kcreate(DAT_IA_HANDL cno_ptr = NULL; evd_ptr = NULL; *evd_handle = NULL; - dat_status = DAT_SUCCESS; if (DAPL_BAD_HANDLE(ia_handle, DAPL_MAGIC_IA)) { dat_status = diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_evd_kquery.c linux-kernel2/dat-provider/dapl_evd_kquery.c --- linux-kernel/dat-provider/dapl_evd_kquery.c 2005-05-10 08:13:28.949014000 -0700 +++ linux-kernel2/dat-provider/dapl_evd_kquery.c 2005-05-11 12:13:16.420015000 -0700 @@ -38,13 +38,10 @@ #include "dapl.h" -DAT_RETURN dapl_evd_kquery(DAT_EVD_HANDLE evd_handle, - struct dat_evd_param *evd_param) +u32 dapl_evd_kquery(DAT_EVD_HANDLE evd_handle, struct dat_evd_param *evd_param) { DAPL_EVD *evd_ptr; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; if (NULL == evd_param) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_evd_modify_upcall.c linux-kernel2/dat-provider/dapl_evd_modify_upcall.c --- linux-kernel/dat-provider/dapl_evd_modify_upcall.c 2005-05-10 08:13:28.739072000 -0700 +++ linux-kernel2/dat-provider/dapl_evd_modify_upcall.c 2005-05-11 12:13:46.318006000 -0700 @@ -43,19 +43,17 @@ * Modify the upcall instance on an EVD */ -DAT_RETURN dapl_evd_modify_upcall(DAT_EVD_HANDLE evd_handle, - DAT_UPCALL_POLICY upcall_policy, /* ignore */ - const struct dat_upcall_object *upcall) +u32 dapl_evd_modify_upcall(DAT_EVD_HANDLE evd_handle, + DAT_UPCALL_POLICY upcall_policy, /* ignore */ + const struct dat_upcall_object *upcall) { DAPL_EVD *evd_ptr; DAPL_CNO *cno_ptr; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_modify_upcall (%p)\n", evd_handle); - dat_status = DAT_SUCCESS; - evd_ptr = (DAPL_EVD *) evd_handle; cno_ptr = evd_ptr->cno_ptr; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_evd_post_se.c linux-kernel2/dat-provider/dapl_evd_post_se.c --- linux-kernel/dat-provider/dapl_evd_post_se.c 2005-05-10 08:13:28.553029000 -0700 +++ linux-kernel2/dat-provider/dapl_evd_post_se.c 2005-05-11 12:14:04.553039000 -0700 @@ -59,14 +59,12 @@ * DAT_INVALID_PARAMETER */ -DAT_RETURN dapl_evd_post_se(DAT_EVD_HANDLE evd_handle, - const struct dat_event *event) +u32 dapl_evd_post_se(DAT_EVD_HANDLE evd_handle, const struct dat_event *event) { DAPL_EVD *evd_ptr; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; evd_ptr = (DAPL_EVD *) evd_handle; - dat_status = DAT_SUCCESS; if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, 0); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_evd_qp_async_error_callb.c linux-kernel2/dat-provider/dapl_evd_qp_async_error_callb.c --- linux-kernel/dat-provider/dapl_evd_qp_async_error_callb.c 2005-05-04 08:19:32.774965000 -0700 +++ linux-kernel2/dat-provider/dapl_evd_qp_async_error_callb.c 2005-05-11 11:16:06.330963000 -0700 @@ -60,7 +60,7 @@ dapl_evd_qp_async_error_callback(ib_erro DAPL_EP *ep_ptr; DAPL_EVD *async_evd; DAT_EVENT_NUMBER async_event; - DAT_RETURN dat_status; + u32 dat_status; ep_ptr = (DAPL_EP *) context; async_evd = (DAPL_EVD *) ep_ptr->header.owner_ia->async_error_evd; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_evd_resize.c linux-kernel2/dat-provider/dapl_evd_resize.c --- linux-kernel/dat-provider/dapl_evd_resize.c 2005-05-09 11:07:01.737008000 -0700 +++ linux-kernel2/dat-provider/dapl_evd_resize.c 2005-05-11 11:16:06.333962000 -0700 @@ -63,12 +63,12 @@ * DAT_INVALID_STATE */ -DAT_RETURN dapl_evd_resize(DAT_EVD_HANDLE evd_handle, DAT_COUNT evd_qlen) +u32 dapl_evd_resize(DAT_EVD_HANDLE evd_handle, DAT_COUNT evd_qlen) { DAPL_IA *ia_ptr; DAPL_EVD *evd_ptr; DAT_COUNT pend_cnt; - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_evd_resize (%p, %d)\n", evd_handle, evd_qlen); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_evd_un_async_error_callb.c linux-kernel2/dat-provider/dapl_evd_un_async_error_callb.c --- linux-kernel/dat-provider/dapl_evd_un_async_error_callb.c 2005-04-20 12:42:07.893006000 -0700 +++ linux-kernel2/dat-provider/dapl_evd_un_async_error_callb.c 2005-05-11 11:16:06.338964000 -0700 @@ -59,7 +59,7 @@ dapl_evd_un_async_error_callback(ib_erro { DAPL_EVD *async_evd; DAT_EVENT_NUMBER async_event; - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK | DAPL_DBG_TYPE_EXCEPTION, "dapl_evd_un_async_error_callback (%p, %p)\n", diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_evd_util.c linux-kernel2/dat-provider/dapl_evd_util.c --- linux-kernel/dat-provider/dapl_evd_util.c 2005-05-11 08:28:12.854000000 -0700 +++ linux-kernel2/dat-provider/dapl_evd_util.c 2005-05-11 12:18:40.507003000 -0700 @@ -41,7 +41,7 @@ static __inline__ void dapl_evd_eh_print_cqe(ib_work_completion_t * cqe); -DAT_RETURN dapl_evd_event_alloc(DAPL_EVD * evd_ptr, DAT_COUNT qlen); +u32 dapl_evd_event_alloc(DAPL_EVD *evd_ptr, DAT_COUNT qlen); /* * actually create the evd. this is called after all parameter checking @@ -49,17 +49,14 @@ DAT_RETURN dapl_evd_event_alloc(DAPL_EVD * to create the default async evd. */ -DAT_RETURN -dapl_evd_internal_create(DAPL_IA * ia_ptr, - DAPL_CNO *cno_ptr, - DAT_COUNT min_qlen, - DAT_EVD_FLAGS evd_flags, DAPL_EVD ** evd_ptr_ptr) +u32 dapl_evd_internal_create(DAPL_IA *ia_ptr, DAPL_CNO *cno_ptr, + DAT_COUNT min_qlen, DAT_EVD_FLAGS evd_flags, + DAPL_EVD **evd_ptr_ptr) { DAPL_EVD *evd_ptr; DAT_COUNT cq_len; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; - dat_status = DAT_SUCCESS; *evd_ptr_ptr = NULL; cq_len = min_qlen; @@ -174,13 +171,11 @@ DAPL_EVD *dapl_evd_alloc(DAPL_IA * ia_pt /* * alloc events into an EVD. */ -DAT_RETURN dapl_evd_event_alloc(DAPL_EVD * evd_ptr, DAT_COUNT qlen) +u32 dapl_evd_event_alloc(DAPL_EVD *evd_ptr, DAT_COUNT qlen) { struct dat_event *event_ptr; DAT_COUNT i; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; /* Allocate EVENTs */ event_ptr = @@ -218,13 +213,13 @@ DAT_RETURN dapl_evd_event_alloc(DAPL_EVD /* * realloc events into an EVD. */ -DAT_RETURN dapl_evd_event_realloc(DAPL_EVD * evd_ptr, DAT_COUNT qlen) +u32 dapl_evd_event_realloc(DAPL_EVD *evd_ptr, DAT_COUNT qlen) { struct dat_event *events; DAT_COUNT old_qlen; DAT_COUNT i; intptr_t diff; - DAT_RETURN dat_status; + u32 dat_status; /* Allocate EVENTs */ events = (struct dat_event *)dapl_os_realloc(evd_ptr->events, @@ -274,13 +269,11 @@ bail: * will clean up all of the internal data structures and report the * error. */ -DAT_RETURN dapl_evd_dealloc(DAPL_EVD * evd_ptr) +u32 dapl_evd_dealloc(DAPL_EVD *evd_ptr) { - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; DAPL_IA *ia_ptr; - dat_status = DAT_SUCCESS; - dapl_os_assert(evd_ptr->header.magic == DAPL_MAGIC_EVD); dapl_os_assert(atomic_read(&evd_ptr->evd_ref_count) == 0); @@ -419,9 +412,10 @@ static struct dat_event *dapl_evd_get_ev * entry to this function. */ -static void dapl_evd_post_event(DAPL_EVD * evd_ptr, const struct dat_event * event_ptr) +static void dapl_evd_post_event(DAPL_EVD *evd_ptr, + const struct dat_event *event_ptr) { - DAT_RETURN dat_status; + u32 dat_status; DAPL_CNO * cno_to_trigger = NULL; dapl_dbg_log(DAPL_DBG_TYPE_EVD, @@ -511,12 +505,12 @@ static struct dat_event *dapl_evd_get_an return event_ptr; } -DAT_RETURN -dapl_evd_post_cr_arrival_event(DAPL_EVD * evd_ptr, - DAT_EVENT_NUMBER event_number, - DAT_SP_HANDLE sp_handle, - DAT_IA_ADDRESS_PTR ia_address_ptr, - DAT_CONN_QUAL conn_qual, DAT_CR_HANDLE cr_handle) +u32 dapl_evd_post_cr_arrival_event(DAPL_EVD *evd_ptr, + DAT_EVENT_NUMBER event_number, + DAT_SP_HANDLE sp_handle, + DAT_IA_ADDRESS_PTR ia_address_ptr, + DAT_CONN_QUAL conn_qual, + DAT_CR_HANDLE cr_handle) { struct dat_event *event_ptr; event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number); @@ -542,12 +536,11 @@ dapl_evd_post_cr_arrival_event(DAPL_EVD return DAT_SUCCESS; } -DAT_RETURN -dapl_evd_post_connection_event(DAPL_EVD * evd_ptr, - DAT_EVENT_NUMBER event_number, - DAT_EP_HANDLE ep_handle, - DAT_COUNT private_data_size, - DAT_PVOID private_data) +u32 dapl_evd_post_connection_event(DAPL_EVD *evd_ptr, + DAT_EVENT_NUMBER event_number, + DAT_EP_HANDLE ep_handle, + DAT_COUNT private_data_size, + DAT_PVOID private_data) { struct dat_event *event_ptr; event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number); @@ -572,10 +565,9 @@ dapl_evd_post_connection_event(DAPL_EVD return DAT_SUCCESS; } -DAT_RETURN -dapl_evd_post_async_error_event(DAPL_EVD * evd_ptr, - DAT_EVENT_NUMBER event_number, - DAT_IA_HANDLE ia_handle) +u32 dapl_evd_post_async_error_event(DAPL_EVD *evd_ptr, + DAT_EVENT_NUMBER event_number, + DAT_IA_HANDLE ia_handle) { struct dat_event *event_ptr; event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number); @@ -598,9 +590,9 @@ dapl_evd_post_async_error_event(DAPL_EVD return DAT_SUCCESS; } -DAT_RETURN -dapl_evd_post_software_event(DAPL_EVD * evd_ptr, - DAT_EVENT_NUMBER event_number, DAT_PVOID pointer) +u32 dapl_evd_post_software_event(DAPL_EVD *evd_ptr, + DAT_EVENT_NUMBER event_number, + DAT_PVOID pointer) { struct dat_event *event_ptr; event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number); @@ -775,10 +767,9 @@ dapl_evd_cqe_to_event(DAPL_EVD * evd_ptr * Status of operation * */ -DAT_RETURN dapl_evd_cq_poll_to_event(DAPL_EVD * evd_ptr, - struct dat_event *event) +u32 dapl_evd_cq_poll_to_event(DAPL_EVD *evd_ptr, struct dat_event *event) { - DAT_RETURN dat_status; + u32 dat_status; ib_work_completion_t cur_cqe; dat_status = dapl_ib_completion_poll(evd_ptr->header.owner_ia->hca_ptr, diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_evd_util.h linux-kernel2/dat-provider/dapl_evd_util.h --- linux-kernel/dat-provider/dapl_evd_util.h 2005-05-10 08:13:29.134055000 -0700 +++ linux-kernel2/dat-provider/dapl_evd_util.h 2005-05-11 12:21:06.564007000 -0700 @@ -40,17 +40,16 @@ #include "dapl.h" -DAT_RETURN -dapl_evd_internal_create(DAPL_IA * ia_ptr, DAPL_CNO * cno_ptr, - DAT_COUNT min_qlen, - DAT_EVD_FLAGS evd_flags, DAPL_EVD ** evd_ptr_ptr); +u32 dapl_evd_internal_create(DAPL_IA *ia_ptr, DAPL_CNO *cno_ptr, + DAT_COUNT min_qlen, DAT_EVD_FLAGS evd_flags, + DAPL_EVD **evd_ptr_ptr); DAPL_EVD *dapl_evd_alloc(DAPL_IA * ia_ptr, DAPL_CNO * cno_ptr, DAT_EVD_FLAGS evd_flags, DAT_COUNT qlen); -DAT_RETURN dapl_evd_dealloc(DAPL_EVD * evd_ptr); +u32 dapl_evd_dealloc(DAPL_EVD *evd_ptr); -DAT_RETURN dapl_evd_event_realloc(DAPL_EVD * evd_ptr, DAT_COUNT qlen); +u32 dapl_evd_event_realloc(DAPL_EVD *evd_ptr, DAT_COUNT qlen); /* * Each of these functions will retrieve a free event from @@ -63,30 +62,26 @@ DAT_RETURN dapl_evd_event_realloc(DAPL_E * DAT_SUCCESS otherwise. */ -DAT_RETURN -dapl_evd_post_cr_arrival_event(DAPL_EVD * evd_ptr, - DAT_EVENT_NUMBER event_number, - DAT_SP_HANDLE sp_handle, - DAT_IA_ADDRESS_PTR ia_address_ptr, - DAT_CONN_QUAL conn_qual, - DAT_CR_HANDLE cr_handle); - -DAT_RETURN -dapl_evd_post_connection_event(DAPL_EVD * evd_ptr, - DAT_EVENT_NUMBER event_number, - DAT_EP_HANDLE ep_handle, - DAT_COUNT private_data_size, - DAT_PVOID private_data); - -DAT_RETURN -dapl_evd_post_async_error_event(DAPL_EVD * evd_ptr, - DAT_EVENT_NUMBER event_number, - DAT_IA_HANDLE ia_handle); - -DAT_RETURN -dapl_evd_post_software_event(DAPL_EVD * evd_ptr, - DAT_EVENT_NUMBER event_number, DAT_PVOID pointer); - +u32 dapl_evd_post_cr_arrival_event(DAPL_EVD *evd_ptr, + DAT_EVENT_NUMBER event_number, + DAT_SP_HANDLE sp_handle, + DAT_IA_ADDRESS_PTR ia_address_ptr, + DAT_CONN_QUAL conn_qual, + DAT_CR_HANDLE cr_handle); + +u32 dapl_evd_post_connection_event(DAPL_EVD *evd_ptr, + DAT_EVENT_NUMBER event_number, + DAT_EP_HANDLE ep_handle, + DAT_COUNT private_data_size, + DAT_PVOID private_data); + +u32 dapl_evd_post_async_error_event(DAPL_EVD *evd_ptr, + DAT_EVENT_NUMBER event_number, + DAT_IA_HANDLE ia_handle); + +u32 dapl_evd_post_software_event(DAPL_EVD *evd_ptr, + DAT_EVENT_NUMBER event_number, + DAT_PVOID pointer); /************************************* * dapl internal callbacks functions * @@ -111,7 +106,7 @@ extern void dapl_evd_cq_async_error_call extern void dapl_evd_qp_async_error_callback(ib_error_record_t * cause_ptr, void *context); -extern DAT_RETURN dapl_evd_cq_poll_to_event(DAPL_EVD * evd_ptr, - struct dat_event *event); +extern u32 dapl_evd_cq_poll_to_event(DAPL_EVD *evd_ptr, + struct dat_event *event); #endif diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_get_consumer_context.c linux-kernel2/dat-provider/dapl_get_consumer_context.c --- linux-kernel/dat-provider/dapl_get_consumer_context.c 2005-04-20 12:42:06.961003000 -0700 +++ linux-kernel2/dat-provider/dapl_get_consumer_context.c 2005-05-11 12:21:23.159014000 -0700 @@ -55,14 +55,11 @@ * DAT_SUCCESS * DAT_INVALID_PARAMETER */ -DAT_RETURN -dapl_get_consumer_context(DAT_HANDLE dat_handle, DAT_CONTEXT * context) +u32 dapl_get_consumer_context(DAT_HANDLE dat_handle, DAT_CONTEXT *context) { - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; DAPL_HEADER *header; - dat_status = DAT_SUCCESS; - header = (DAPL_HEADER *) dat_handle; if (((header) == NULL) || ((unsigned long)(header) & 3) || diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_get_handle_type.c linux-kernel2/dat-provider/dapl_get_handle_type.c --- linux-kernel/dat-provider/dapl_get_handle_type.c 2005-04-20 12:42:08.013010000 -0700 +++ linux-kernel2/dat-provider/dapl_get_handle_type.c 2005-05-11 12:21:50.885012000 -0700 @@ -56,14 +56,11 @@ * DAT_INVALID_PARAMETER */ -DAT_RETURN -dapl_get_handle_type(DAT_HANDLE dat_handle, DAT_HANDLE_TYPE * handle_type) +u32 dapl_get_handle_type(DAT_HANDLE dat_handle, DAT_HANDLE_TYPE *handle_type) { - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; DAPL_HEADER *header; - dat_status = DAT_SUCCESS; - header = (DAPL_HEADER *) dat_handle; if (((header) == NULL) || ((unsigned long)(header) & 3) || diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl.h linux-kernel2/dat-provider/dapl.h --- linux-kernel/dat-provider/dapl.h 2005-05-11 11:08:04.453960000 -0700 +++ linux-kernel2/dat-provider/dapl.h 2005-05-11 13:23:31.584000000 -0700 @@ -138,8 +138,6 @@ typedef enum dapl_qp_state { #define DEBUG_IS_BAD_HANDLE(h, magicNum) (0) #endif -#define DAT_ERROR(Type, SubType) ((DAT_RETURN)(DAT_CLASS_ERROR | Type | SubType)) - /********************************************************************* * * * Typedefs * @@ -477,275 +475,245 @@ typedef char DAPL_HCA_NAME[DAPL_HCA_NAME * DAT API functions */ -extern DAT_RETURN dapl_ia_open(const DAT_NAME_PTR, DAT_COUNT, DAT_EVD_HANDLE *, - DAT_IA_HANDLE *); +extern u32 dapl_ia_open(const DAT_NAME_PTR, DAT_COUNT, DAT_EVD_HANDLE *, + DAT_IA_HANDLE *); -extern DAT_RETURN dapl_ia_close(DAT_IA_HANDLE, DAT_CLOSE_FLAGS); +extern u32 dapl_ia_close(DAT_IA_HANDLE, DAT_CLOSE_FLAGS); -extern DAT_RETURN dapl_ia_query(DAT_IA_HANDLE, - DAT_EVD_HANDLE *, - struct dat_ia_attr *, - struct dat_provider_attr *); +extern u32 dapl_ia_query(DAT_IA_HANDLE, DAT_EVD_HANDLE *, struct dat_ia_attr *, + struct dat_provider_attr *); /* helper functions */ -extern DAT_RETURN dapl_set_consumer_context(DAT_HANDLE, /* dat handle */ - DAT_CONTEXT); /* context */ +extern u32 dapl_set_consumer_context(DAT_HANDLE handle, DAT_CONTEXT context); -extern DAT_RETURN dapl_get_consumer_context(DAT_HANDLE, /* dat handle */ - DAT_CONTEXT *); /* context */ +extern u32 dapl_get_consumer_context(DAT_HANDLE handle, DAT_CONTEXT *context); -extern DAT_RETURN dapl_get_handle_type(DAT_HANDLE, DAT_HANDLE_TYPE *); +extern u32 dapl_get_handle_type(DAT_HANDLE handle, DAT_HANDLE_TYPE *context); /* CR Functions */ -extern DAT_RETURN dapl_cr_query(DAT_CR_HANDLE, struct dat_cr_param *); +extern u32 dapl_cr_query(DAT_CR_HANDLE, struct dat_cr_param *); -extern DAT_RETURN dapl_cr_accept(DAT_CR_HANDLE, /* cr_handle */ - DAT_EP_HANDLE, /* ep_handle */ - DAT_COUNT, /* private_data_size */ - const DAT_PVOID); /* private_data */ +extern u32 dapl_cr_accept(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, + DAT_COUNT private_data_size, + const DAT_PVOID private_data); -extern DAT_RETURN dapl_cr_reject(DAT_CR_HANDLE); +extern u32 dapl_cr_reject(DAT_CR_HANDLE cr_handle); -extern DAT_RETURN dapl_cr_handoff(DAT_CR_HANDLE, /* cr_handle */ - DAT_CONN_QUAL); /* handoff */ +extern u32 dapl_cr_handoff(DAT_CR_HANDLE cr_handle, DAT_CONN_QUAL handoff); /* EVD Functions */ -extern DAT_RETURN dapl_ia_memtype_hint(DAT_IA_HANDLE, /* ia_handle */ - DAT_MEM_TYPE, /* mem_type */ - DAT_VLEN, /* length */ - DAT_MEM_OPT, /* mem_optimization */ - DAT_VLEN *, /* suggested_length */ - DAT_VADDR *); /* suggested_alignment */ +extern u32 dapl_ia_memtype_hint(DAT_IA_HANDLE ia_handle, DAT_MEM_TYPE mem_type, + DAT_VLEN length, DAT_MEM_OPT mem_optimization, + DAT_VLEN *suggested_length, + DAT_VADDR *suggested_alignment); -extern DAT_RETURN dapl_evd_kcreate(DAT_IA_HANDLE, /* ia_handle */ - DAT_COUNT, /* evd_min_qlen */ - DAT_UPCALL_POLICY, /* upcall_policy */ - const struct dat_upcall_object *,/* upcall */ - DAT_EVD_FLAGS, /* evd_flags */ - DAT_EVD_HANDLE *); /* evd_handle */ +extern u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_handle, DAT_COUNT evd_min_qlen, + DAT_UPCALL_POLICY upcall_policy, + const struct dat_upcall_object * upcall, + DAT_EVD_FLAGS evd_flags, + DAT_EVD_HANDLE * evd_handle); -extern DAT_RETURN dapl_evd_kquery(DAT_EVD_HANDLE, /* evd_handle */ - struct dat_evd_param *); /* evd_args */ +extern u32 dapl_evd_kquery(DAT_EVD_HANDLE evd_handle, + struct dat_evd_param *evd_args); -extern DAT_RETURN dapl_evd_modify_upcall(DAT_EVD_HANDLE, /* evd_handle */ - DAT_UPCALL_POLICY, /* upcall_policy */ - const struct dat_upcall_object *); /* upcall */ +extern u32 dapl_evd_modify_upcall(DAT_EVD_HANDLE evd_handle, + DAT_UPCALL_POLICY upcall_policy, + const struct dat_upcall_object * upcall); -extern DAT_RETURN dapl_evd_enable(DAT_EVD_HANDLE); /* evd_handle */ +extern u32 dapl_evd_enable(DAT_EVD_HANDLE evd_handle); -extern DAT_RETURN dapl_evd_disable(DAT_EVD_HANDLE); /* evd_handle */ +extern u32 dapl_evd_disable(DAT_EVD_HANDLE evd_handle); -extern DAT_RETURN dapl_evd_resize(DAT_EVD_HANDLE, /* evd_handle */ - DAT_COUNT); /* evd_qlen */ +extern u32 dapl_evd_resize(DAT_EVD_HANDLE evd_handle, DAT_COUNT evd_qlen); -extern DAT_RETURN dapl_evd_post_se(DAT_EVD_HANDLE, /* evd_handle */ - const struct dat_event *); /* event */ +extern u32 dapl_evd_post_se(DAT_EVD_HANDLE evd_handle, + const struct dat_event *event); -extern DAT_RETURN dapl_evd_dequeue(DAT_EVD_HANDLE, /* evd_handle */ - struct dat_event *); /* event */ +extern u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_handle, struct dat_event *event); -extern DAT_RETURN dapl_evd_free(DAT_EVD_HANDLE); +extern u32 dapl_evd_free(DAT_EVD_HANDLE evd_handle); -extern DAT_RETURN dapl_evd_set_unwaitable(DAT_EVD_HANDLE evd_handle); +extern u32 dapl_evd_set_unwaitable(DAT_EVD_HANDLE evd_handle); -extern DAT_RETURN dapl_evd_clear_unwaitable(DAT_EVD_HANDLE evd_handle); +extern u32 dapl_evd_clear_unwaitable(DAT_EVD_HANDLE evd_handle); /* EP functions */ -extern DAT_RETURN dapl_ep_create(DAT_IA_HANDLE, /* ia_handle */ - DAT_PZ_HANDLE, /* pz_handle */ - DAT_EVD_HANDLE, /* in_dto_completion_evd_handle */ - DAT_EVD_HANDLE, /* out_dto_completion_evd_handle */ - DAT_EVD_HANDLE, /* connect_evd_handle */ - const struct dat_ep_attr *, /* ep_parameters */ - DAT_EP_HANDLE *); /* ep_handle */ - -extern DAT_RETURN dapl_ep_query(DAT_EP_HANDLE, struct dat_ep_param *); - -extern DAT_RETURN dapl_ep_modify(DAT_EP_HANDLE, /* ep_handle */ - DAT_EP_PARAM_MASK, /* ep_args_mask */ - const struct dat_ep_param *); /* ep_args */ - -extern DAT_RETURN dapl_ep_connect(DAT_EP_HANDLE, /* ep_handle */ - DAT_IA_ADDRESS_PTR, /* remote_ia_address */ - DAT_CONN_QUAL, /* remote_conn_qual */ - DAT_TIMEOUT, /* timeout */ - DAT_COUNT, /* private_data_size */ - const DAT_PVOID, /* private_data */ - DAT_QOS, /* quality_of_service */ - DAT_CONNECT_FLAGS); /* connect_flags */ - -extern DAT_RETURN dapl_ep_dup_connect(DAT_EP_HANDLE, /* ep_handle */ - DAT_EP_HANDLE, /* ep_dup_handle */ - DAT_TIMEOUT, /* timeout */ - DAT_COUNT, /* private_data_size */ - const DAT_PVOID, /* private_data */ - DAT_QOS); /* quality_of_service */ - -extern DAT_RETURN dapl_ep_disconnect(DAT_EP_HANDLE, /* ep_handle */ - DAT_CLOSE_FLAGS); /* close_flags */ - -extern DAT_RETURN dapl_ep_post_send(DAT_EP_HANDLE, /* ep_handle */ - DAT_COUNT, /* num_segments */ - struct dat_lmr_triplet *, /* local_iov */ - DAT_DTO_COOKIE, /* user_cookie */ - DAT_COMPLETION_FLAGS); /* completion_flags */ - -extern DAT_RETURN dapl_ep_post_recv(DAT_EP_HANDLE, /* ep_handle */ - DAT_COUNT, /* num_segments */ - struct dat_lmr_triplet *, /* local_iov */ - DAT_DTO_COOKIE, /* user_cookie */ - DAT_COMPLETION_FLAGS); /* completion_flags */ - -extern DAT_RETURN dapl_ep_post_rdma_read(DAT_EP_HANDLE, /* ep_handle */ - DAT_COUNT, /* num_segments */ - struct dat_lmr_triplet *, /* local_iov */ - DAT_DTO_COOKIE, /* user_cookie */ - const struct dat_rmr_triplet *, /* remote_iov */ - DAT_COMPLETION_FLAGS); /* completion_flags */ - -extern DAT_RETURN dapl_ep_post_rdma_write(DAT_EP_HANDLE, /* ep_handle */ - DAT_COUNT, /* num_segments */ - struct dat_lmr_triplet *, /* local_iov */ - DAT_DTO_COOKIE, /* user_cookie */ - const struct dat_rmr_triplet *, /* remote_iov */ - DAT_COMPLETION_FLAGS); /* completion_flags */ - -extern DAT_RETURN dapl_ep_get_status(DAT_EP_HANDLE, /* ep_handle */ - DAT_EP_STATE *, /* ep_state */ - boolean_t *, /* in_dto_idle */ - boolean_t *); /* out_dto_idle */ - -extern DAT_RETURN dapl_ep_free(DAT_EP_HANDLE); /* ep_handle */ - -extern DAT_RETURN dapl_ep_reset(DAT_EP_HANDLE); /* ep_handle */ - -extern DAT_RETURN dapl_ep_create_with_srq(DAT_IA_HANDLE, /* ia_handle */ - DAT_PZ_HANDLE, /* pz_handle */ - DAT_EVD_HANDLE, /* recv_evd_handle */ - DAT_EVD_HANDLE, /* request_evd_handle */ - DAT_EVD_HANDLE, /* connect_evd_handle */ - DAT_SRQ_HANDLE, /* srq_handle */ - const struct dat_ep_attr *, /* ep_attributes */ - DAT_EP_HANDLE *); /* ep_handle */ - -extern DAT_RETURN dapl_ep_recv_query(DAT_EP_HANDLE, /* ep_handle */ - DAT_COUNT *, /* nbufs_allocated */ - DAT_COUNT *); /* bufs_alloc_span */ - -extern DAT_RETURN dapl_ep_set_watermark(DAT_EP_HANDLE, /* ep_handle */ - DAT_COUNT, /* soft_high_watermark */ - DAT_COUNT); /* hard_high_watermark */ +extern u32 dapl_ep_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, + DAT_EVD_HANDLE in_dto_completion_evd_handle, + DAT_EVD_HANDLE out_dto_completion_evd_handle, + DAT_EVD_HANDLE connect_evd_handle, + const struct dat_ep_attr *ep_parameters, + DAT_EP_HANDLE *ep_handle); + +extern u32 dapl_ep_query(DAT_EP_HANDLE, struct dat_ep_param *); + +extern u32 dapl_ep_modify(DAT_EP_HANDLE ep_handle, + DAT_EP_PARAM_MASK ep_args_mask, + const struct dat_ep_param *ep_args); + +extern u32 dapl_ep_connect(DAT_EP_HANDLE ep_handle, + DAT_IA_ADDRESS_PTR remote_ia_address, + DAT_CONN_QUAL remote_conn_qual, + DAT_TIMEOUT timeout, + DAT_COUNT private_data_size, + const DAT_PVOID private_data, + DAT_QOS quality_of_service, + DAT_CONNECT_FLAGS connect_flags); + +extern u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, + DAT_EP_HANDLE ep_dup_handle, DAT_TIMEOUT timeout, + DAT_COUNT private_data_size, + const DAT_PVOID private_data, + DAT_QOS quality_of_service); + +extern u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_handle, + DAT_CLOSE_FLAGS close_flags); + +extern u32 dapl_ep_post_send(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, + struct dat_lmr_triplet *local_iov, + DAT_DTO_COOKIE user_cookie, + DAT_COMPLETION_FLAGS completion_flags); + +extern u32 dapl_ep_post_recv(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, + struct dat_lmr_triplet *local_iov, + DAT_DTO_COOKIE user_cookie, + DAT_COMPLETION_FLAGS completion_flags); + +extern u32 dapl_ep_post_rdma_read(DAT_EP_HANDLE ep_handle, + DAT_COUNT num_segments, + struct dat_lmr_triplet *local_iov, + DAT_DTO_COOKIE user_cookie, + const struct dat_rmr_triplet *remote_iov, + DAT_COMPLETION_FLAGS completion_flags); + +extern u32 dapl_ep_post_rdma_write(DAT_EP_HANDLE ep_handle, + DAT_COUNT num_segments, + struct dat_lmr_triplet *local_iov, + DAT_DTO_COOKIE user_cookie, + const struct dat_rmr_triplet *remote_iov, + DAT_COMPLETION_FLAGS completion_flags); + +extern u32 dapl_ep_get_status(DAT_EP_HANDLE ep_handle, DAT_EP_STATE *ep_state, + boolean_t *in_dto_idle, boolean_t *out_dto_idle); + +extern u32 dapl_ep_free(DAT_EP_HANDLE ep_handle); + +extern u32 dapl_ep_reset(DAT_EP_HANDLE ep_handle); + +extern u32 dapl_ep_create_with_srq(DAT_IA_HANDLE ia_handle, + DAT_PZ_HANDLE pz_handle, + DAT_EVD_HANDLE recv_evd_handle, + DAT_EVD_HANDLE request_evd_handle, + DAT_EVD_HANDLE connect_evd_handle, + DAT_SRQ_HANDLE srq_handle, + const struct dat_ep_attr *ep_attributes, + DAT_EP_HANDLE *ep_handle); + +extern u32 dapl_ep_recv_query(DAT_EP_HANDLE ep_handle, + DAT_COUNT *nbufs_allocated, + DAT_COUNT *bufs_alloc_span); + +extern u32 dapl_ep_set_watermark(DAT_EP_HANDLE ep_handle, + DAT_COUNT soft_high_watermark, + DAT_COUNT hard_high_watermark); /* LMR functions */ -extern DAT_RETURN dapl_lmr_kcreate(DAT_IA_HANDLE, /* ia_handle */ - DAT_MEM_TYPE, /* mem_type */ - DAT_REGION_DESCRIPTION, /* region_description */ - DAT_VLEN, /* length */ - DAT_PZ_HANDLE, /* pz_handle */ - DAT_MEM_PRIV_FLAGS, /* privileges */ - DAT_MEM_OPT, /* optimization */ - DAT_LMR_HANDLE *, /* lmr_handle */ - DAT_LMR_CONTEXT *, /* lmr_context */ - DAT_RMR_CONTEXT *, /* rmr_context */ - DAT_VLEN *, /* registered_length */ - DAT_VADDR *); /* registered_address */ - -extern DAT_RETURN dapl_lmr_query(DAT_LMR_HANDLE, struct dat_lmr_param *); - -extern DAT_RETURN dapl_lmr_free(DAT_LMR_HANDLE); - -extern DAT_RETURN dapl_lmr_sync_rdma_read(DAT_IA_HANDLE, /* ia_handle */ - const struct dat_lmr_triplet *, /* local_segments */ - DAT_VLEN); /* num_segments */ - -extern DAT_RETURN dapl_lmr_sync_rdma_write(DAT_IA_HANDLE, /* ia_handle */ - const struct dat_lmr_triplet *, /* local_segments */ - DAT_VLEN); /* num_segments */ +extern u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_handle, DAT_MEM_TYPE mem_type, + DAT_REGION_DESCRIPTION region_description, + DAT_VLEN length, DAT_PZ_HANDLE pz_handle, + DAT_MEM_PRIV_FLAGS privileges, + DAT_MEM_OPT optimization, + DAT_LMR_HANDLE *lmr_handle, + DAT_LMR_CONTEXT *lmr_context, + DAT_RMR_CONTEXT *rmr_context, + DAT_VLEN *registered_length, + DAT_VADDR *registered_address); + +extern u32 dapl_lmr_query(DAT_LMR_HANDLE, struct dat_lmr_param *); + +extern u32 dapl_lmr_free(DAT_LMR_HANDLE); + +extern u32 dapl_lmr_sync_rdma_read(DAT_IA_HANDLE ia_handle, + const struct dat_lmr_triplet *local_segments, + DAT_VLEN num_segments); + +extern u32 dapl_lmr_sync_rdma_write(DAT_IA_HANDLE ia_handle, + const struct dat_lmr_triplet *local_segments, + DAT_VLEN num_segments); /* RMR Functions */ -extern DAT_RETURN dapl_rmr_create(DAT_PZ_HANDLE, /* pz_handle */ - DAT_RMR_HANDLE *); /* rmr_handle */ +extern u32 dapl_rmr_create(DAT_PZ_HANDLE pz_handle, DAT_RMR_HANDLE *rmr_handle); -extern DAT_RETURN dapl_rmr_query(DAT_RMR_HANDLE, /* rmr_handle */ - struct dat_rmr_param *); /* rmr_args */ +extern u32 dapl_rmr_query(DAT_RMR_HANDLE rmr_handle, + struct dat_rmr_param *rmr_args); -extern DAT_RETURN dapl_rmr_bind(DAT_RMR_HANDLE, /* rmr_handle */ - const struct dat_lmr_triplet *, /* lmr_triplet */ - DAT_MEM_PRIV_FLAGS, /* mem_priv */ - DAT_EP_HANDLE, /* ep_handle */ - DAT_RMR_COOKIE, /* user_cookie */ - DAT_COMPLETION_FLAGS, /* completion_flags */ - DAT_RMR_CONTEXT *); /* context */ +extern u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_handle, + const struct dat_lmr_triplet *lmr_triplet, + DAT_MEM_PRIV_FLAGS mem_priv, DAT_EP_HANDLE ep_handle, + DAT_RMR_COOKIE user_cookie, + DAT_COMPLETION_FLAGS completion_flags, + DAT_RMR_CONTEXT *context); -extern DAT_RETURN dapl_rmr_free(DAT_RMR_HANDLE); +extern u32 dapl_rmr_free(DAT_RMR_HANDLE); /* PSP Functions */ -extern DAT_RETURN dapl_psp_create(DAT_IA_HANDLE, /* ia_handle */ - DAT_CONN_QUAL, /* conn_qual */ - DAT_EVD_HANDLE, /* evd_handle */ - DAT_PSP_FLAGS, /* psp_flags */ - DAT_PSP_HANDLE *); /* psp_handle */ - -extern DAT_RETURN dapl_psp_create_any(DAT_IA_HANDLE, /* ia_handle */ - DAT_CONN_QUAL *, /* conn_qual */ - DAT_EVD_HANDLE, /* evd_handle */ - DAT_PSP_FLAGS, /* psp_flags */ - DAT_PSP_HANDLE *); /* psp_handle */ +extern u32 dapl_psp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, + DAT_EVD_HANDLE evd_handle, DAT_PSP_FLAGS psp_flags, + DAT_PSP_HANDLE *psp_handle); + +extern u32 dapl_psp_create_any(DAT_IA_HANDLE ia_handle, + DAT_CONN_QUAL *conn_qual, + DAT_EVD_HANDLE evd_handle, + DAT_PSP_FLAGS psp_flags, + DAT_PSP_HANDLE *psp_handle); -extern DAT_RETURN dapl_psp_query(DAT_PSP_HANDLE, struct dat_psp_param *); +extern u32 dapl_psp_query(DAT_PSP_HANDLE, struct dat_psp_param *); -extern DAT_RETURN dapl_psp_free(DAT_PSP_HANDLE); /* psp_handle */ +extern u32 dapl_psp_free(DAT_PSP_HANDLE psp_handle); /* RSP Functions */ -extern DAT_RETURN dapl_rsp_create(DAT_IA_HANDLE, /* ia_handle */ - DAT_CONN_QUAL, /* conn_qual */ - DAT_EP_HANDLE, /* ep_handle */ - DAT_EVD_HANDLE, /* evd_handle */ - DAT_RSP_HANDLE *); /* rsp_handle */ +extern u32 dapl_rsp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, + DAT_EP_HANDLE ep_handle, DAT_EVD_HANDLE evd_handle, + DAT_RSP_HANDLE *rsp_handle); -extern DAT_RETURN dapl_rsp_query(DAT_RSP_HANDLE, struct dat_rsp_param *); +extern u32 dapl_rsp_query(DAT_RSP_HANDLE, struct dat_rsp_param *); -extern DAT_RETURN dapl_rsp_free(DAT_RSP_HANDLE); /* rsp_handle */ +extern u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_handle); /* PZ Functions */ -extern DAT_RETURN dapl_pz_create(DAT_IA_HANDLE, /* ia_handle */ - DAT_PZ_HANDLE *); /* pz_handle */ +extern u32 dapl_pz_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE *pz_handle); -extern DAT_RETURN dapl_pz_query(DAT_PZ_HANDLE, /* pz_handle */ - struct dat_pz_param *); /* pz_args */ +extern u32 dapl_pz_query(DAT_PZ_HANDLE pz_handle, struct dat_pz_param *pz_args); -extern DAT_RETURN dapl_pz_free(DAT_PZ_HANDLE); /* pz_handle */ +extern u32 dapl_pz_free(DAT_PZ_HANDLE pz_handle); /* SRQ functions */ -extern DAT_RETURN dapl_srq_create(DAT_IA_HANDLE, /* ia_handle */ - DAT_PZ_HANDLE, /* pz_handle */ - struct dat_srq_attr *, /* srq_attr */ - DAT_SRQ_HANDLE *); /* srq_handle */ - -extern DAT_RETURN dapl_srq_free(DAT_SRQ_HANDLE); /* srq_handle */ - -extern DAT_RETURN dapl_srq_post_recv(DAT_SRQ_HANDLE, /* srq_handle */ - DAT_COUNT, /* num_segments */ - struct dat_lmr_triplet *, /* local_iov */ - DAT_DTO_COOKIE); /* user_cookie */ +extern u32 dapl_srq_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, + struct dat_srq_attr *srq_attr, + DAT_SRQ_HANDLE *srq_handle); + +extern u32 dapl_srq_free(DAT_SRQ_HANDLE srq_handle); + +extern u32 dapl_srq_post_recv(DAT_SRQ_HANDLE srq_handle, DAT_COUNT num_segments, + struct dat_lmr_triplet *local_iov, + DAT_DTO_COOKIE user_cookie); -extern DAT_RETURN dapl_srq_query(DAT_SRQ_HANDLE, /* srq_handle */ - struct dat_srq_param *); /* srq_param */ +extern u32 dapl_srq_query(DAT_SRQ_HANDLE srq_handle, + struct dat_srq_param *srq_param); -extern DAT_RETURN dapl_srq_resize(DAT_SRQ_HANDLE, /* srq_handle */ - DAT_COUNT); /* srq_max_recv_dto */ +extern u32 dapl_srq_resize(DAT_SRQ_HANDLE srq_handle, + DAT_COUNT srq_max_recv_dto); -extern DAT_RETURN dapl_srq_set_lw(DAT_SRQ_HANDLE, /* srq_handle */ - DAT_COUNT); /* low_watermark */ +extern u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, DAT_COUNT low_watermark); /* * DAPL internal utility function prototpyes diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_hash.c linux-kernel2/dat-provider/dapl_hash.c --- linux-kernel/dat-provider/dapl_hash.c 2005-05-11 08:28:12.964000000 -0700 +++ linux-kernel2/dat-provider/dapl_hash.c 2005-05-11 13:24:53.952031000 -0700 @@ -320,15 +320,14 @@ dapl_hash_delete_element(DAPL_HASH_ELEM /* * Create a new hash table with at least 'table_size' hash buckets. */ -DAT_RETURN dapl_hash_create(DAT_COUNT table_size, DAPL_HASH_TABLE ** pp_table) +u32 dapl_hash_create(DAT_COUNT table_size, DAPL_HASH_TABLE **pp_table) { DAPL_HASH_TABLE *p_table; DAT_COUNT table_length = table_size * sizeof(DAPL_HASH_ELEM); - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; DAT_COUNT i; dapl_os_assert(pp_table); - dat_status = DAT_SUCCESS; /* Allocate hash table */ p_table = kmalloc(sizeof(DAPL_HASH_TABLE), GFP_ATOMIC); @@ -365,7 +364,7 @@ DAT_RETURN dapl_hash_create(DAT_COUNT ta /* * Destroy a hash table */ -DAT_RETURN dapl_hash_free(DAPL_HASH_TABLE * p_table) +u32 dapl_hash_free(DAPL_HASH_TABLE *p_table) { dapl_os_assert(p_table && p_table->table); @@ -380,7 +379,7 @@ DAT_RETURN dapl_hash_free(DAPL_HASH_TABL * Returns the number of elements stored in the table */ -DAT_RETURN dapl_hash_size(DAPL_HASH_TABLE * p_table, DAT_COUNT * p_size) +u32 dapl_hash_size(DAPL_HASH_TABLE *p_table, DAT_COUNT *p_size) { dapl_os_assert(p_table && p_size); @@ -394,14 +393,12 @@ DAT_RETURN dapl_hash_size(DAPL_HASH_TABL * Duplicates are not expected, and return in error, having done nothing. */ -DAT_RETURN -dapl_hash_insert(DAPL_HASH_TABLE * p_table, - DAPL_HASH_KEY key, DAPL_HASH_DATA data) +u32 dapl_hash_insert(DAPL_HASH_TABLE *p_table, DAPL_HASH_KEY key, + DAPL_HASH_DATA data) { - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; dapl_os_assert(p_table); - dat_status = DAT_SUCCESS; spin_lock(&p_table->lock); if (!dapl_hash_add(p_table, key, data, FALSE, NULL)) { @@ -419,11 +416,10 @@ dapl_hash_insert(DAPL_HASH_TABLE * p_tab * data is returned in the DAPL_HASH_DATA * pointer if that pointer is not NULL. */ -DAT_RETURN -dapl_hash_search(DAPL_HASH_TABLE * p_table, - DAPL_HASH_KEY key, DAPL_HASH_DATA * p_data) +u32 dapl_hash_search(DAPL_HASH_TABLE *p_table, DAPL_HASH_KEY key, + DAPL_HASH_DATA *p_data) { - DAT_RETURN dat_status; + u32 dat_status; void *olddatum; DAPL_HASH_ELEM *found; @@ -444,11 +440,10 @@ dapl_hash_search(DAPL_HASH_TABLE * p_tab return dat_status; } -DAT_RETURN -dapl_hash_remove(DAPL_HASH_TABLE * p_table, - DAPL_HASH_KEY key, DAPL_HASH_DATA * p_data) +u32 dapl_hash_remove(DAPL_HASH_TABLE *p_table, DAPL_HASH_KEY key, + DAPL_HASH_DATA *p_data) { - DAT_RETURN dat_status; + u32 dat_status; DAPL_HASH_KEY hashValue, save_key = key; dapl_os_assert(p_table); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_hash.h linux-kernel2/dat-provider/dapl_hash.h --- linux-kernel/dat-provider/dapl_hash.h 2005-04-20 12:42:08.701003000 -0700 +++ linux-kernel2/dat-provider/dapl_hash.h 2005-05-11 13:25:44.049020000 -0700 @@ -67,23 +67,19 @@ * * *********************************************************************/ -extern DAT_RETURN -dapl_hash_create(DAT_COUNT capacity, DAPL_HASH_TABLE ** pp_table); +extern u32 dapl_hash_create(DAT_COUNT capacity, DAPL_HASH_TABLE **pp_table); -extern DAT_RETURN dapl_hash_free(DAPL_HASH_TABLE * p_table); +extern u32 dapl_hash_free(DAPL_HASH_TABLE *p_table); -extern DAT_RETURN dapl_hash_size(DAPL_HASH_TABLE * p_table, DAT_COUNT * p_size); +extern u32 dapl_hash_size(DAPL_HASH_TABLE *p_table, DAT_COUNT *p_size); -extern DAT_RETURN -dapl_hash_insert(DAPL_HASH_TABLE * p_table, - DAPL_HASH_KEY key, DAPL_HASH_DATA data); +extern u32 dapl_hash_insert(DAPL_HASH_TABLE *p_table, DAPL_HASH_KEY key, + DAPL_HASH_DATA data); -extern DAT_RETURN -dapl_hash_search(DAPL_HASH_TABLE * p_table, - DAPL_HASH_KEY key, DAPL_HASH_DATA * p_data); +extern u32 dapl_hash_search(DAPL_HASH_TABLE *p_table, DAPL_HASH_KEY key, + DAPL_HASH_DATA *p_data); -extern DAT_RETURN -dapl_hash_remove(DAPL_HASH_TABLE * p_table, - DAPL_HASH_KEY key, DAPL_HASH_DATA * p_data); +extern u32 dapl_hash_remove(DAPL_HASH_TABLE *p_table, DAPL_HASH_KEY key, + DAPL_HASH_DATA *p_data); #endif /* DAPL_HASH_H */ diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ia_close.c linux-kernel2/dat-provider/dapl_ia_close.c --- linux-kernel/dat-provider/dapl_ia_close.c 2005-04-20 12:42:07.789026000 -0700 +++ linux-kernel2/dat-provider/dapl_ia_close.c 2005-05-11 11:16:06.367966000 -0700 @@ -57,10 +57,10 @@ * DAT_INSUFFICIENT_RESOURCES * DAT_INVALID_PARAMETER */ -DAT_RETURN dapl_ia_close(DAT_IA_HANDLE ia_handle, DAT_CLOSE_FLAGS ia_flags) +u32 dapl_ia_close(DAT_IA_HANDLE ia_handle, DAT_CLOSE_FLAGS ia_flags) { DAPL_IA *ia_ptr; - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ia_close (%p, %d)\n", ia_handle, ia_flags); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ia_memtype_hint.c linux-kernel2/dat-provider/dapl_ia_memtype_hint.c --- linux-kernel/dat-provider/dapl_ia_memtype_hint.c 2005-04-20 12:42:06.476020000 -0700 +++ linux-kernel2/dat-provider/dapl_ia_memtype_hint.c 2005-05-11 13:26:28.672013000 -0700 @@ -63,22 +63,18 @@ * DAT_INVALID_HANDLE * DAT_MODEL_NOT_SUPPORTED */ -DAT_RETURN -dapl_ia_memtype_hint(DAT_IA_HANDLE ia_handle, - DAT_MEM_TYPE mem_type, - DAT_VLEN length, - DAT_MEM_OPT mem_optimization, - DAT_VLEN * suggested_length, - DAT_VADDR * suggested_alignment) +u32 dapl_ia_memtype_hint(DAT_IA_HANDLE ia_handle, DAT_MEM_TYPE mem_type, + DAT_VLEN length, DAT_MEM_OPT mem_optimization, + DAT_VLEN *suggested_length, + DAT_VADDR *suggested_alignment) { DAPL_IA *ia_ptr; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ia_memtype_hint (%p)\n", ia_handle); ia_ptr = (DAPL_IA *) ia_handle; - dat_status = DAT_SUCCESS; if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { dat_status = diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ia_open.c linux-kernel2/dat-provider/dapl_ia_open.c --- linux-kernel/dat-provider/dapl_ia_open.c 2005-05-11 11:08:04.435965000 -0700 +++ linux-kernel2/dat-provider/dapl_ia_open.c 2005-05-11 13:26:48.054000000 -0700 @@ -51,19 +51,16 @@ * That function maps the DAT_NAME parameter of dat_ia_open to a DAT_PROVIDER, * and calls this function. */ -DAT_RETURN -dapl_ia_open(const DAT_NAME_PTR name, - DAT_COUNT async_evd_qlen, - DAT_EVD_HANDLE * async_evd_handle_ptr, - DAT_IA_HANDLE * ia_handle_ptr) +u32 dapl_ia_open(const DAT_NAME_PTR name, DAT_COUNT async_evd_qlen, + DAT_EVD_HANDLE *async_evd_handle_ptr, + DAT_IA_HANDLE *ia_handle_ptr) { - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; struct dat_provider *provider; DAPL_HCA *hca_ptr; DAPL_IA *ia_ptr; DAPL_EVD *evd_ptr; - dat_status = DAT_SUCCESS; hca_ptr = NULL; ia_ptr = NULL; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ia_query.c linux-kernel2/dat-provider/dapl_ia_query.c --- linux-kernel/dat-provider/dapl_ia_query.c 2005-05-11 08:28:11.913022000 -0700 +++ linux-kernel2/dat-provider/dapl_ia_query.c 2005-05-11 13:27:31.889018000 -0700 @@ -59,14 +59,12 @@ * DAT_SUCCESS * DAT_INVALID_PARAMETER */ -DAT_RETURN -dapl_ia_query(DAT_IA_HANDLE ia_handle, - DAT_EVD_HANDLE * async_evd_handle, - struct dat_ia_attr *ia_attr, - struct dat_provider_attr *provider_attr) +u32 dapl_ia_query(DAT_IA_HANDLE ia_handle, DAT_EVD_HANDLE *async_evd_handle, + struct dat_ia_attr *ia_attr, + struct dat_provider_attr *provider_attr) { DAPL_IA *ia_ptr; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; struct evd_merge_type { boolean_t array[6][6]; } *evd_merge; @@ -82,7 +80,6 @@ dapl_ia_query(DAT_IA_HANDLE ia_handle, provider_attr); ia_ptr = (DAPL_IA *) ia_handle; - dat_status = DAT_SUCCESS; if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { dat_status = diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ia_util.c linux-kernel2/dat-provider/dapl_ia_util.c --- linux-kernel/dat-provider/dapl_ia_util.c 2005-05-11 11:08:04.416964000 -0700 +++ linux-kernel2/dat-provider/dapl_ia_util.c 2005-05-11 13:28:41.996001000 -0700 @@ -107,9 +107,9 @@ DAPL_IA *dapl_ia_alloc(struct dat_provid * Performs an abrupt close of the IA */ -DAT_RETURN dapl_ia_abrupt_close(DAPL_IA * ia_ptr) +u32 dapl_ia_abrupt_close(DAPL_IA *ia_ptr) { - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; DAPL_EP *ep_ptr, *next_ep_ptr; DAPL_LMR *lmr_ptr, *next_lmr_ptr; DAPL_RMR *rmr_ptr, *next_rmr_ptr; @@ -119,8 +119,6 @@ DAT_RETURN dapl_ia_abrupt_close(DAPL_IA DAPL_CR *cr_ptr, *next_cr_ptr; DAPL_HCA *hca_ptr; - dat_status = DAT_SUCCESS; - /* * clear all the data structures associated with the IA. * this must be done in order (rmr,rsp) before (ep lmr psp) before @@ -351,16 +349,14 @@ DAT_RETURN dapl_ia_abrupt_close(DAPL_IA * */ -DAT_RETURN dapl_ia_graceful_close(DAPL_IA * ia_ptr) +u32 dapl_ia_graceful_close(DAPL_IA *ia_ptr) { - DAT_RETURN dat_status; - DAT_RETURN cur_dat_status; + u32 dat_status = DAT_SUCCESS; + u32 cur_dat_status; DAPL_EVD *evd_ptr; DAPL_LLIST_ENTRY *entry; DAPL_HCA *hca_ptr; - dat_status = DAT_SUCCESS; - if (!dapl_llist_is_empty(&ia_ptr->rmr_list_head) || !dapl_llist_is_empty(&ia_ptr->rsp_list_head) || !dapl_llist_is_empty(&ia_ptr->ep_list_head) || @@ -710,9 +706,9 @@ void dapl_ia_link_rsp(DAPL_IA * ia_ptr, spin_unlock(&ia_ptr->header.lock); } -DAT_RETURN dapl_ia_setup_callbacks(DAPL_IA * ia_ptr, DAPL_EVD * async_evd_ptr) +u32 dapl_ia_setup_callbacks(DAPL_IA * ia_ptr, DAPL_EVD * async_evd_ptr) { - DAT_RETURN dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; /* unaffiliated handler */ dat_status = @@ -733,9 +729,9 @@ DAT_RETURN dapl_ia_setup_callbacks(DAPL_ return dat_status; } -DAT_RETURN dapl_ia_teardown_callbacks(DAPL_IA * ia_ptr) +u32 dapl_ia_teardown_callbacks(DAPL_IA * ia_ptr) { - DAT_RETURN dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; /* unaffiliated handler */ dat_status = diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ia_util.h linux-kernel2/dat-provider/dapl_ia_util.h --- linux-kernel/dat-provider/dapl_ia_util.h 2005-05-11 08:28:12.246007000 -0700 +++ linux-kernel2/dat-provider/dapl_ia_util.h 2005-05-11 13:29:00.006004000 -0700 @@ -41,9 +41,9 @@ DAPL_IA *dapl_ia_alloc(struct dat_provider *provider, DAPL_HCA * hca_ptr); -DAT_RETURN dapl_ia_abrupt_close(DAPL_IA * ia_ptr); +u32 dapl_ia_abrupt_close(DAPL_IA *ia_ptr); -DAT_RETURN dapl_ia_graceful_close(DAPL_IA * ia_ptr); +u32 dapl_ia_graceful_close(DAPL_IA *ia_ptr); void dapl_ia_free(DAPL_IA * ia_ptr); @@ -80,8 +80,8 @@ void dapl_ia_link_rsp(DAPL_IA * ia_ptr, DAPL_SP *dapl_ia_sp_search(DAPL_IA * ia_ptr, DAT_CONN_QUAL conn_qual, boolean_t is_psp); -DAT_RETURN dapl_ia_setup_callbacks(DAPL_IA * ia_ptr, DAPL_EVD * async_evd_ptr); +u32 dapl_ia_setup_callbacks(DAPL_IA *ia_ptr, DAPL_EVD *async_evd_ptr); -DAT_RETURN dapl_ia_teardown_callbacks(DAPL_IA * ia_ptr); +u32 dapl_ia_teardown_callbacks(DAPL_IA *ia_ptr); #endif diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_lmr_free.c linux-kernel2/dat-provider/dapl_lmr_free.c --- linux-kernel/dat-provider/dapl_lmr_free.c 2005-04-27 11:38:35.485003000 -0700 +++ linux-kernel2/dat-provider/dapl_lmr_free.c 2005-05-11 11:16:06.394963000 -0700 @@ -57,10 +57,10 @@ * DAT_INVALID_STATE */ -DAT_RETURN dapl_lmr_free(DAT_LMR_HANDLE lmr_handle) +u32 dapl_lmr_free(DAT_LMR_HANDLE lmr_handle) { DAPL_LMR *lmr; - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_free (%p)\n", lmr_handle); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_lmr_kcreate.c linux-kernel2/dat-provider/dapl_lmr_kcreate.c --- linux-kernel/dat-provider/dapl_lmr_kcreate.c 2005-05-06 16:16:19.180012000 -0700 +++ linux-kernel2/dat-provider/dapl_lmr_kcreate.c 2005-05-11 13:37:00.876008000 -0700 @@ -37,71 +37,20 @@ #include "dapl_lmr_util.h" #include "dapl_adapter_util.h" -/********************************************************************* - * * - * Function Prototypes * - * * - *********************************************************************/ - -static __inline__ DAT_RETURN -dapl_lmr_create_virtual(DAPL_IA * ia, - DAT_PVOID virt_addr, - DAT_VLEN length, - DAPL_PZ * pz, - DAT_MEM_PRIV_FLAGS privileges, - DAT_LMR_HANDLE * lmr_handle, - DAT_LMR_CONTEXT * lmr_context, - DAT_RMR_CONTEXT * rmr_context, - DAT_VLEN * registered_length, - DAT_VADDR * registered_address); - -static __inline__ DAT_RETURN -dapl_lmr_create_physical(DAPL_IA * ia, - DAT_REGION_DESCRIPTION phys_addr, - DAT_VLEN length, - DAPL_PZ * pz, - DAT_MEM_PRIV_FLAGS privileges, - DAT_LMR_HANDLE * lmr_handle, - DAT_LMR_CONTEXT * lmr_context, - DAT_RMR_CONTEXT * rmr_context, - DAT_VLEN * registered_length, - DAT_VADDR * registered_address); - -static __inline__ DAT_RETURN -dapl_lmr_create_lmr(DAPL_IA * ia, - DAPL_LMR * original_lmr, - DAPL_PZ * pz, - DAT_MEM_PRIV_FLAGS privileges, - DAT_LMR_HANDLE * lmr_handle, - DAT_LMR_CONTEXT * lmr_context, - DAT_RMR_CONTEXT * rmr_context, - DAT_VLEN * registered_length, - DAT_VADDR * registered_address); - -/********************************************************************* - * * - * Function Definitions * - * * - *********************************************************************/ - -DAT_RETURN -dapl_lmr_create_virtual(DAPL_IA * ia, - DAT_PVOID virt_addr, - DAT_VLEN length, - DAPL_PZ * pz, - DAT_MEM_PRIV_FLAGS privileges, - DAT_LMR_HANDLE * lmr_handle, - DAT_LMR_CONTEXT * lmr_context, - DAT_RMR_CONTEXT * rmr_context, - DAT_VLEN * registered_length, - DAT_VADDR * registered_address) +static inline u32 dapl_lmr_create_virtual(DAPL_IA *ia, DAT_PVOID virt_addr, + DAT_VLEN length, DAPL_PZ *pz, + DAT_MEM_PRIV_FLAGS privileges, + DAT_LMR_HANDLE *lmr_handle, + DAT_LMR_CONTEXT *lmr_context, + DAT_RMR_CONTEXT *rmr_context, + DAT_VLEN *registered_length, + DAT_VADDR *registered_address) { DAPL_LMR *lmr; DAT_REGION_DESCRIPTION reg_desc; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; reg_desc.for_va = virt_addr; - dat_status = DAT_SUCCESS; lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_VIRTUAL, @@ -170,25 +119,21 @@ dapl_lmr_create_virtual(DAPL_IA * ia, } -DAT_RETURN -dapl_lmr_create_physical(DAPL_IA * ia, - DAT_REGION_DESCRIPTION phys_addr, - DAT_VLEN page_count, - DAPL_PZ * pz, - DAT_MEM_PRIV_FLAGS privileges, - DAT_LMR_HANDLE * lmr_handle, - DAT_LMR_CONTEXT * lmr_context, - DAT_RMR_CONTEXT * rmr_context, - DAT_VLEN * registered_length, - DAT_VADDR * registered_address) +static inline u32 dapl_lmr_create_physical(DAPL_IA *ia, + DAT_REGION_DESCRIPTION phys_addr, + DAT_VLEN page_count, DAPL_PZ *pz, + DAT_MEM_PRIV_FLAGS privileges, + DAT_LMR_HANDLE *lmr_handle, + DAT_LMR_CONTEXT *lmr_context, + DAT_RMR_CONTEXT *rmr_context, + DAT_VLEN *registered_length, + DAT_VADDR *registered_address) { DAPL_LMR *lmr; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; u64 *array; array = (u64 *) phys_addr.for_array; - dat_status = DAT_SUCCESS; - lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_PHYSICAL, phys_addr, @@ -258,20 +203,18 @@ dapl_lmr_create_physical(DAPL_IA * ia, return dat_status; } -DAT_RETURN -dapl_lmr_create_lmr(DAPL_IA * ia, - DAPL_LMR * original_lmr, - DAPL_PZ * pz, - DAT_MEM_PRIV_FLAGS privileges, - DAT_LMR_HANDLE * lmr_handle, - DAT_LMR_CONTEXT * lmr_context, - DAT_RMR_CONTEXT * rmr_context, - DAT_VLEN * registered_length, - DAT_VADDR * registered_address) +static inline u32 dapl_lmr_create_lmr(DAPL_IA *ia, DAPL_LMR *original_lmr, + DAPL_PZ *pz, + DAT_MEM_PRIV_FLAGS privileges, + DAT_LMR_HANDLE *lmr_handle, + DAT_LMR_CONTEXT *lmr_context, + DAT_RMR_CONTEXT *rmr_context, + DAT_VLEN *registered_length, + DAT_VADDR *registered_address) { DAPL_LMR *lmr; DAT_REGION_DESCRIPTION reg_desc; - DAT_RETURN dat_status; + u32 dat_status; dat_status = dapl_hash_search(ia->hca_ptr->lmr_hash_table, original_lmr->param.lmr_context, @@ -378,22 +321,15 @@ dapl_lmr_create_lmr(DAPL_IA * ia, * DAT_MODEL_NOT_SUPPORTED * */ -DAT_RETURN -dapl_lmr_kcreate(DAT_IA_HANDLE ia_handle, - DAT_MEM_TYPE mem_type, - DAT_REGION_DESCRIPTION region_description, - DAT_VLEN length, - DAT_PZ_HANDLE pz_handle, - DAT_MEM_PRIV_FLAGS privileges, - DAT_MEM_OPT optimization, - DAT_LMR_HANDLE * lmr_handle, - DAT_LMR_CONTEXT * lmr_context, - DAT_RMR_CONTEXT * rmr_context, - DAT_VLEN * registered_length, DAT_VADDR * registered_address) +u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_handle, DAT_MEM_TYPE mem_type, + DAT_REGION_DESCRIPTION region_description, DAT_VLEN length, + DAT_PZ_HANDLE pz_handle, DAT_MEM_PRIV_FLAGS privileges, + DAT_MEM_OPT optimization, DAT_LMR_HANDLE *lmr_handle, + DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, DAT_VLEN *registered_length, DAT_VADDR *registered_address) { DAPL_IA *ia; DAPL_PZ *pz; - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_kcreate(ia:%p, mem_type:%x, ...)\n", diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_lmr_query.c linux-kernel2/dat-provider/dapl_lmr_query.c --- linux-kernel/dat-provider/dapl_lmr_query.c 2005-05-10 08:13:28.665004000 -0700 +++ linux-kernel2/dat-provider/dapl_lmr_query.c 2005-05-11 13:37:47.609016000 -0700 @@ -36,11 +36,10 @@ #include "dapl.h" -DAT_RETURN dapl_lmr_query(DAT_LMR_HANDLE lmr_handle, - struct dat_lmr_param *lmr_param) +u32 dapl_lmr_query(DAT_LMR_HANDLE lmr_handle, struct dat_lmr_param *lmr_param) { DAPL_LMR *lmr; - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_query (%p, %p)\n", diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_lmr_sync_rdma_read.c linux-kernel2/dat-provider/dapl_lmr_sync_rdma_read.c --- linux-kernel/dat-provider/dapl_lmr_sync_rdma_read.c 2005-05-10 08:13:28.304000000 -0700 +++ linux-kernel2/dat-provider/dapl_lmr_sync_rdma_read.c 2005-05-11 13:38:06.653039000 -0700 @@ -56,15 +56,12 @@ * DAT_INVALID_HANDLE * DAT_INVALID_PARAMETER */ -DAT_RETURN -dapl_lmr_sync_rdma_read(DAT_IA_HANDLE ia_handle, - const struct dat_lmr_triplet *local_segments, - DAT_VLEN num_segments) +u32 dapl_lmr_sync_rdma_read(DAT_IA_HANDLE ia_handle, + const struct dat_lmr_triplet *local_segments, + DAT_VLEN num_segments) { DAPL_IA *ia_ptr; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dat_lmr_sync_rdma_read (%p, %p, %ld)\n", diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_lmr_sync_rdma_write.c linux-kernel2/dat-provider/dapl_lmr_sync_rdma_write.c --- linux-kernel/dat-provider/dapl_lmr_sync_rdma_write.c 2005-05-10 08:13:29.023025000 -0700 +++ linux-kernel2/dat-provider/dapl_lmr_sync_rdma_write.c 2005-05-11 13:38:29.184000000 -0700 @@ -41,15 +41,12 @@ * Ensure a region of memory is consistent by locally flushing * non-coherent cache */ -DAT_RETURN -dapl_lmr_sync_rdma_write(DAT_IA_HANDLE ia_handle, - const struct dat_lmr_triplet *local_segments, - DAT_VLEN num_segments) +u32 dapl_lmr_sync_rdma_write(DAT_IA_HANDLE ia_handle, + const struct dat_lmr_triplet *local_segments, + DAT_VLEN num_segments) { DAPL_IA *ia_ptr; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dat_lmr_sync_rdma_write (%p, %p, %ld)\n", diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_module.c linux-kernel2/dat-provider/dapl_module.c --- linux-kernel/dat-provider/dapl_module.c 2005-05-11 11:08:04.396968000 -0700 +++ linux-kernel2/dat-provider/dapl_module.c 2005-05-11 11:16:06.423960000 -0700 @@ -58,7 +58,7 @@ static void dapl_unregister_providers(vo */ static int __init dapl_init(void) { - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_UTIL, "DAPL: Started (dapl_init)\n"); @@ -96,7 +96,7 @@ module_init(dapl_init); */ static void __exit dapl_fini(void) { - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_UTIL, "DAPL: Stopped (dapl_fini)\n"); @@ -121,7 +121,7 @@ void DAT_PROVIDER_INIT_FUNC_NAME(const s { struct dat_provider *provider; DAPL_HCA *hca_ptr; - DAT_RETURN dat_status; + u32 dat_status; provider = NULL; hca_ptr = NULL; @@ -194,7 +194,7 @@ void DAT_PROVIDER_INIT_FUNC_NAME(const s void DAT_PROVIDER_FINI_FUNC_NAME(const struct dat_provider_info *provider_info) { struct dat_provider *provider; - DAT_RETURN dat_status; + u32 dat_status; dat_status = dapl_provider_list_search(provider_info->ia_name, &provider); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_openib_cm.c linux-kernel2/dat-provider/dapl_openib_cm.c --- linux-kernel/dat-provider/dapl_openib_cm.c 2005-05-11 10:09:59.074056000 -0700 +++ linux-kernel2/dat-provider/dapl_openib_cm.c 2005-05-11 13:40:22.442023000 -0700 @@ -397,10 +397,10 @@ error: * DAT_INVALID_PARAMETER * */ -DAT_RETURN dapl_ib_connect(DAT_EP_HANDLE ep_handle, - DAT_IA_ADDRESS_PTR remote_ia_address, - DAT_CONN_QUAL remote_conn_qual, - DAT_COUNT private_data_size, DAT_PVOID private_data) +u32 dapl_ib_connect(DAT_EP_HANDLE ep_handle, + DAT_IA_ADDRESS_PTR remote_ia_address, + DAT_CONN_QUAL remote_conn_qual, + DAT_COUNT private_data_size, DAT_PVOID private_data) { DAPL_IA *ia_ptr; DAPL_EP *ep_ptr; @@ -489,7 +489,7 @@ DAT_RETURN dapl_ib_connect(DAT_EP_HANDLE * DAT_INVALID_PARAMETER * */ -DAT_RETURN dapl_ib_disconnect(DAPL_EP * ep_ptr, DAT_CLOSE_FLAGS close_flags) +u32 dapl_ib_disconnect(DAPL_EP *ep_ptr, DAT_CLOSE_FLAGS close_flags) { struct dapl_cm_id *conn = ep_ptr->cm_handle; int status; @@ -622,9 +622,7 @@ void dapl_ib_reinit_ep(DAPL_EP * ep_ptr) * DAT_INVALID_PARAMETER * */ -DAT_RETURN -dapl_ib_setup_conn_listener(DAPL_IA * ia_ptr, - u64 ServiceID, DAPL_SP * sp_ptr) +u32 dapl_ib_setup_conn_listener(DAPL_IA *ia_ptr, u64 ServiceID, DAPL_SP *sp_ptr) { int status; @@ -667,7 +665,7 @@ dapl_ib_setup_conn_listener(DAPL_IA * ia * DAT_INVALID_PARAMETER * */ -DAT_RETURN dapl_ib_remove_conn_listener(DAPL_IA * ia_ptr, DAPL_SP * sp_ptr) +u32 dapl_ib_remove_conn_listener(DAPL_IA *ia_ptr, DAPL_SP *sp_ptr) { //*** This will hang if called from CM thread context... //*** Move back to using WQ... @@ -695,8 +693,7 @@ DAT_RETURN dapl_ib_remove_conn_listener( * DAT_INVALID_PARAMETER * */ -DAT_RETURN -dapl_ib_reject_connection(struct dapl_cm_id *cm_handle, int reject_reason) +u32 dapl_ib_reject_connection(struct dapl_cm_id *cm_handle, int reject_reason) { int status; @@ -738,15 +735,14 @@ dapl_ib_reject_connection(struct dapl_cm * DAT_INVALID_PARAMETER * */ -DAT_RETURN dapl_ib_accept_connection(DAT_CR_HANDLE cr_handle, - DAT_EP_HANDLE ep_handle, - DAT_COUNT private_data_size, - const DAT_PVOID prd_ptr) +u32 dapl_ib_accept_connection(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, + DAT_COUNT private_data_size, + const DAT_PVOID prd_ptr) { DAPL_CR *cr_ptr; DAPL_EP *ep_ptr; DAPL_IA *ia_ptr; - DAT_RETURN dat_status; + u32 dat_status; int status; struct ib_cm_rep_param passive_params; struct dapl_cm_id *conn; @@ -842,8 +838,8 @@ static int ib_cm_get_remote_gid(struct d * DAT_INVALID_PARAMETER * */ -DAT_RETURN dapl_ib_cm_remote_addr(DAT_HANDLE dat_handle, - struct sockaddr_in6 *remote_ia_address) +u32 dapl_ib_cm_remote_addr(DAT_HANDLE dat_handle, + struct sockaddr_in6 *remote_ia_address) { DAPL_IA *ia_ptr; int rc; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_openib_dto.h linux-kernel2/dat-provider/dapl_openib_dto.h --- linux-kernel/dat-provider/dapl_openib_dto.h 2005-05-10 08:13:28.379014000 -0700 +++ linux-kernel2/dat-provider/dapl_openib_dto.h 2005-05-11 13:41:39.620043000 -0700 @@ -50,10 +50,9 @@ * * Provider specific Post RECV function */ -static __inline__ DAT_RETURN -dapl_ib_post_recv(DAPL_EP * ep_ptr, - DAPL_COOKIE * cookie, - DAT_COUNT num_segments, struct dat_lmr_triplet *local_iov) +static __inline__ u32 dapl_ib_post_recv(DAPL_EP *ep_ptr, DAPL_COOKIE *cookie, + DAT_COUNT num_segments, + struct dat_lmr_triplet *local_iov) { struct ib_recv_wr rr_desc = { NULL }; struct ib_recv_wr *rr_desc_fail; @@ -100,14 +99,12 @@ dapl_ib_post_recv(DAPL_EP * ep_ptr, * * Provider specific Post SEND function */ -static __inline__ DAT_RETURN -dapl_ib_post_send(DAPL_EP * ep_ptr, - int op_type, - DAPL_COOKIE * cookie, - DAT_COUNT num_segments, - struct dat_lmr_triplet *local_iov, - const struct dat_rmr_triplet *remote_iov, - DAT_COMPLETION_FLAGS completion_flags) +static __inline__ u32 dapl_ib_post_send(DAPL_EP *ep_ptr, int op_type, + DAPL_COOKIE *cookie, + DAT_COUNT num_segments, + struct dat_lmr_triplet *local_iov, + const struct dat_rmr_triplet *remote_iov, + DAT_COMPLETION_FLAGS completion_flags) { struct ib_send_wr send_desc = { NULL }; struct ib_send_wr *send_desc_fail; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_openib_qp.c linux-kernel2/dat-provider/dapl_openib_qp.c --- linux-kernel/dat-provider/dapl_openib_qp.c 2005-05-10 08:13:28.894000000 -0700 +++ linux-kernel2/dat-provider/dapl_openib_qp.c 2005-05-11 13:42:06.454000000 -0700 @@ -67,8 +67,7 @@ ib_cq_handle_t dapl_get_dto_cq(DAPL_IA * * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN -dapl_ib_qp_alloc(DAPL_IA * ia_ptr, DAPL_EP * ep_ptr, DAPL_EP * ep_ctx_ptr) +u32 dapl_ib_qp_alloc(DAPL_IA *ia_ptr, DAPL_EP *ep_ptr, DAPL_EP *ep_ctx_ptr) { struct dat_ep_attr *attr; int ib_status; @@ -151,7 +150,7 @@ dapl_ib_qp_alloc(DAPL_IA * ia_ptr, DAPL_ * Status of destroy operation. * */ -DAT_RETURN dapl_ib_qp_free(DAPL_IA * ia_ptr, DAPL_EP * ep_ptr) +u32 dapl_ib_qp_free(DAPL_IA *ia_ptr, DAPL_EP *ep_ptr) { int ib_status; @@ -231,9 +230,8 @@ ib_cq_handle_t dapl_get_dto_cq(DAPL_IA * * DAT_INVALID_PARAMETER * */ -DAT_RETURN -dapl_ib_qp_modify(DAPL_IA * ia_ptr, DAPL_EP * ep_ptr, - struct dat_ep_attr *ep_attr) +u32 dapl_ib_qp_modify(DAPL_IA *ia_ptr, DAPL_EP *ep_ptr, + struct dat_ep_attr *ep_attr) { struct ib_qp_attr qp_attr; struct ib_qp *qp_handle; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_openib_util.c linux-kernel2/dat-provider/dapl_openib_util.c --- linux-kernel/dat-provider/dapl_openib_util.c 2005-05-11 10:09:59.144000000 -0700 +++ linux-kernel2/dat-provider/dapl_openib_util.c 2005-05-11 13:48:41.352007000 -0700 @@ -47,9 +47,8 @@ #include "dapl_vendor.h" #include "dapl_openib_cm.h" -DAT_RETURN -dapl_ib_get_hca_ids(ib_hca_handle_t hca, - u8 port, union ib_gid * gid, u16 * lid); +u32 dapl_ib_get_hca_ids(ib_hca_handle_t hca, u8 port, union ib_gid *gid, + u16 *lid); /* * Map all IB DTO completion codes to their DAT equivalent. @@ -162,9 +161,9 @@ dapl_ib_convert_mem_privileges(DAT_MEM_P * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN dapl_ib_open_hca(char *name, DAPL_HCA * hca_ptr) +u32 dapl_ib_open_hca(char *name, DAPL_HCA * hca_ptr) { - DAT_RETURN dat_ret; + u32 dat_ret; memset(&hca_ptr->ib_trans.gid.global, 0, sizeof(union ib_gid)); hca_ptr->ib_trans.lid = 0; @@ -199,7 +198,7 @@ DAT_RETURN dapl_ib_open_hca(char *name, * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN dapl_ib_close_hca(DAPL_HCA * hca_ptr) +u32 dapl_ib_close_hca(DAPL_HCA *hca_ptr) { return DAT_SUCCESS; } @@ -222,10 +221,8 @@ DAT_RETURN dapl_ib_close_hca(DAPL_HCA * * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN -dapl_ib_cq_alloc(DAPL_IA * ia_ptr, - DAPL_EVD * evd_ptr, - DAT_COUNT * cqlen, ib_comp_handle_t callback) +u32 dapl_ib_cq_alloc(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr, DAT_COUNT *cqlen, + ib_comp_handle_t callback) { int ib_status; @@ -258,7 +255,7 @@ dapl_ib_cq_alloc(DAPL_IA * ia_ptr, * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN dapl_ib_cq_free(DAPL_IA * ia_ptr, DAPL_EVD * evd_ptr) +u32 dapl_ib_cq_free(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr) { int ib_status; @@ -266,8 +263,7 @@ DAT_RETURN dapl_ib_cq_free(DAPL_IA * ia_ return dapl_ib_status_convert(ib_status); } -DAT_RETURN -dapl_ib_cq_resize(DAPL_IA * ia_ptr, DAPL_EVD * evd_ptr, DAT_COUNT * cqlen) +u32 dapl_ib_cq_resize(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr, DAT_COUNT *cqlen) { int ib_status; @@ -299,7 +295,7 @@ dapl_ib_cq_resize(DAPL_IA * ia_ptr, DAPL * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN dapl_set_cq_notify(DAPL_IA * ia_ptr, DAPL_EVD * evd_ptr) +u32 dapl_set_cq_notify(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr) { int ib_status; @@ -324,7 +320,7 @@ DAT_RETURN dapl_set_cq_notify(DAPL_IA * * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN dapl_ib_pd_alloc(DAPL_IA * ia, DAPL_PZ * pz) +u32 dapl_ib_pd_alloc(DAPL_IA *ia, DAPL_PZ *pz) { struct ib_pd *pd; int ib_status; @@ -359,7 +355,7 @@ DAT_RETURN dapl_ib_pd_alloc(DAPL_IA * ia * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN dapl_ib_pd_free(DAPL_PZ * pz) +u32 dapl_ib_pd_free(DAPL_PZ *pz) { int ib_status; @@ -394,21 +390,16 @@ DAT_RETURN dapl_ib_pd_free(DAPL_PZ * pz) * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN -dapl_ib_mr_register(DAPL_IA * ia, - DAPL_LMR * lmr, - DAT_PVOID virt_addr, - DAT_VLEN length, DAT_MEM_PRIV_FLAGS privileges) +u32 dapl_ib_mr_register(DAPL_IA *ia, DAPL_LMR *lmr, DAT_PVOID virt_addr, + DAT_VLEN length, DAT_MEM_PRIV_FLAGS privileges) { /* ITBD virtual memory registration ??? */ return (DAT_SUCCESS); } -DAT_RETURN dapl_ib_mr_register_physical(DAPL_IA * ia_ptr, - DAPL_LMR * lmr, - DAT_PVOID phys_addr, - DAT_VLEN length, - DAT_MEM_PRIV_FLAGS privileges) +u32 dapl_ib_mr_register_physical(DAPL_IA *ia_ptr, DAPL_LMR *lmr, + DAT_PVOID phys_addr, DAT_VLEN length, + DAT_MEM_PRIV_FLAGS privileges) { int ib_status; int i; @@ -481,7 +472,7 @@ DAT_RETURN dapl_ib_mr_register_physical( * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN dapl_ib_mr_deregister(DAPL_LMR * lmr) +u32 dapl_ib_mr_deregister(DAPL_LMR *lmr) { int ib_status; @@ -515,7 +506,7 @@ DAT_RETURN dapl_ib_mr_deregister(DAPL_LM * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN dapl_ib_mw_alloc(DAPL_RMR * rmr) +u32 dapl_ib_mw_alloc(DAPL_RMR *rmr) { int ib_status; struct ib_mw *mw; @@ -552,7 +543,7 @@ DAT_RETURN dapl_ib_mw_alloc(DAPL_RMR * r * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN dapl_ib_mw_free(DAPL_RMR * rmr) +u32 dapl_ib_mw_free(DAPL_RMR *rmr) { int ib_status; @@ -585,14 +576,10 @@ DAT_RETURN dapl_ib_mw_free(DAPL_RMR * rm * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN -dapl_ib_mw_bind(DAPL_RMR * rmr, - DAPL_LMR * lmr, - DAPL_EP * ep, - DAPL_COOKIE * cookie, - DAT_VADDR virtual_address, - DAT_VLEN length, - DAT_MEM_PRIV_FLAGS mem_priv, boolean_t is_signaled) +u32 dapl_ib_mw_bind(DAPL_RMR *rmr, DAPL_LMR *lmr, DAPL_EP *ep, + DAPL_COOKIE *cookie, DAT_VADDR virtual_address, + DAT_VLEN length, DAT_MEM_PRIV_FLAGS mem_priv, + boolean_t is_signaled) { int ib_status; struct ib_mw_bind mw_bind_prop; @@ -640,9 +627,8 @@ dapl_ib_mw_bind(DAPL_RMR * rmr, * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN -dapl_ib_mw_unbind(DAPL_RMR * rmr, - DAPL_EP * ep, DAPL_COOKIE * cookie, boolean_t is_signaled) +u32 dapl_ib_mw_unbind(DAPL_RMR *rmr, DAPL_EP *ep, DAPL_COOKIE *cookie, + boolean_t is_signaled) { int ib_status; struct ib_mw_bind mw_bind_prop; @@ -688,10 +674,9 @@ dapl_ib_mw_unbind(DAPL_RMR * rmr, * DAT_INVALID_PARAMETER * */ -DAT_RETURN -dapl_ib_setup_async_callback(DAPL_IA * ia_ptr, - DAPL_ASYNC_HANDLER_TYPE handler_type, - ib_async_handler_t callback, void *context) +u32 dapl_ib_setup_async_callback(DAPL_IA *ia_ptr, + DAPL_ASYNC_HANDLER_TYPE handler_type, + ib_async_handler_t callback, void *context) { int ib_status = 0; @@ -714,10 +699,8 @@ dapl_ib_setup_async_callback(DAPL_IA * i * DAT_SUCCESS * DAT_INVALID_PARAMETER */ -DAT_RETURN dapl_ib_query_hca(DAPL_HCA * hca_ptr, - struct dat_ia_attr *ia_attr, - struct dat_ep_attr *ep_attr, - struct sockaddr_in6 *ip_addr) +u32 dapl_ib_query_hca(DAPL_HCA *hca_ptr, struct dat_ia_attr *ia_attr, + struct dat_ep_attr *ep_attr, struct sockaddr_in6 *ip_addr) { int ib_status; struct ib_device_attr device_attr; @@ -795,9 +778,8 @@ DAT_RETURN dapl_ib_query_hca(DAPL_HCA * return DAT_SUCCESS; } -DAT_RETURN -dapl_ib_completion_poll(DAPL_HCA * hca_ptr, - DAPL_EVD * evd_ptr, ib_work_completion_t * cqe_ptr) +u32 dapl_ib_completion_poll(DAPL_HCA *hca_ptr, DAPL_EVD *evd_ptr, + ib_work_completion_t *cqe_ptr) { int ib_status; @@ -805,12 +787,11 @@ dapl_ib_completion_poll(DAPL_HCA * hca_p if (ib_status == 1) return DAT_SUCCESS; if (ib_status == 0) - return DAT_ERRORP(DAT_QUEUE_EMPTY, DAT_NO_SUBTYPE); + return DAT_ERROR(DAT_QUEUE_EMPTY, DAT_NO_SUBTYPE); return dapl_ib_status_convert(ib_status); } -DAT_RETURN -dapl_ib_completion_notify(DAPL_EVD * evd_ptr, ib_notification_type_t type) +u32 dapl_ib_completion_notify(DAPL_EVD *evd_ptr, ib_notification_type_t type) { int ib_status; @@ -877,14 +858,12 @@ DAT_DTO_COMPLETION_STATUS dapl_ib_get_dt * DAT_NOT_IMPLEMENTED Caller is not interested this event */ -DAT_RETURN -dapl_ib_get_async_event(ib_error_record_t * cause_ptr, - DAT_EVENT_NUMBER * async_event) +u32 dapl_ib_get_async_event(ib_error_record_t *cause_ptr, + DAT_EVENT_NUMBER *async_event) { ib_async_event_type event_id; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; - dat_status = DAT_SUCCESS; event_id = cause_ptr->event; switch (event_id) { @@ -921,8 +900,8 @@ dapl_ib_get_async_event(ib_error_record_ return dat_status; } -DAT_RETURN -dapl_ib_get_hca_ids(ib_hca_handle_t hca, u8 port, union ib_gid * gid, u16 * lid) +u32 dapl_ib_get_hca_ids(ib_hca_handle_t hca, u8 port, union ib_gid *gid, + u16 *lid) { int status; struct ib_port_attr port_attr; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_openib_util.h linux-kernel2/dat-provider/dapl_openib_util.h --- linux-kernel/dat-provider/dapl_openib_util.h 2005-05-11 10:09:59.163023000 -0700 +++ linux-kernel2/dat-provider/dapl_openib_util.h 2005-05-11 13:48:41.355003000 -0700 @@ -47,12 +47,7 @@ void dapl_add_one(struct ib_device *device); void dapl_remove_one(struct ib_device *device); -static __inline__ DAT_RETURN dapl_ib_status_convert(int32_t ib_status); - -/* A version of DAT_ERROR is redefined here as the order of include - * files makes it unavailable at this point - */ -#define DAT_ERRORP(Type, SubType) ((DAT_RETURN)(DAT_CLASS_ERROR | Type | SubType)) +static __inline__ u32 dapl_ib_status_convert(int32_t ib_status); /* * Typedefs to map OpenIB types to more generic 'ib' types @@ -126,28 +121,28 @@ typedef struct ib_hca_transport { * Do not return invalid Handles, the user is not able * to deal with them. */ -static __inline__ DAT_RETURN dapl_ib_status_convert(int32_t ib_status) +static __inline__ u32 dapl_ib_status_convert(int32_t ib_status) { switch (ib_status) { case 0: return DAT_SUCCESS; case -EAGAIN: - return DAT_ERRORP(DAT_QUEUE_EMPTY, DAT_NO_SUBTYPE); + return DAT_ERROR(DAT_QUEUE_EMPTY, DAT_NO_SUBTYPE); case -EBUSY: - return DAT_ERRORP(DAT_INTERNAL_ERROR, DAT_NO_SUBTYPE); + return DAT_ERROR(DAT_INTERNAL_ERROR, DAT_NO_SUBTYPE); case -ENOMEM: - return DAT_ERRORP(DAT_INSUFFICIENT_RESOURCES, DAT_NO_SUBTYPE); + return DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_NO_SUBTYPE); case -EINVAL: - return DAT_ERRORP(DAT_INVALID_HANDLE, DAT_NO_SUBTYPE); + return DAT_ERROR(DAT_INVALID_HANDLE, DAT_NO_SUBTYPE); /* * Udapl only, not sure what will be the return value from the IB * case ??????: - * return DAT_ERRORP(DAT_TIMEOUT_EXPIRED, DAT_NO_SUBTYPE); + * return DAT_ERROR(DAT_TIMEOUT_EXPIRED, DAT_NO_SUBTYPE); */ case -EINTR: - return DAT_ERRORP(DAT_INTERRUPTED_CALL, DAT_NO_SUBTYPE); + return DAT_ERROR(DAT_INTERRUPTED_CALL, DAT_NO_SUBTYPE); default: - return DAT_ERRORP(DAT_INTERNAL_ERROR, DAT_NO_SUBTYPE); + return DAT_ERROR(DAT_INTERNAL_ERROR, DAT_NO_SUBTYPE); } } diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_provider.c linux-kernel2/dat-provider/dapl_provider.c --- linux-kernel/dat-provider/dapl_provider.c 2005-05-11 08:28:12.130010000 -0700 +++ linux-kernel2/dat-provider/dapl_provider.c 2005-05-11 13:50:39.276005000 -0700 @@ -37,7 +37,7 @@ #include "dapl_provider.h" -extern DAT_RETURN dapl_not_implemented(void); +extern u32 dapl_not_implemented(void); /********************************************************************* * * @@ -144,11 +144,9 @@ dapl_provider_list_key_cmp(const char *n * * *********************************************************************/ -DAT_RETURN dapl_provider_list_create(void) +u32 dapl_provider_list_create(void) { - DAT_RETURN status; - - status = DAT_SUCCESS; + u32 status = DAT_SUCCESS; /* create the head node */ g_dapl_provider_list.head = @@ -192,7 +190,7 @@ DAT_RETURN dapl_provider_list_create(voi return status; } -DAT_RETURN dapl_provider_list_destroy(void) +u32 dapl_provider_list_destroy(void) { DAPL_PROVIDER_LIST_NODE *cur_node; @@ -211,15 +209,12 @@ DAT_COUNT dapl_provider_list_size(void) return g_dapl_provider_list.size; } -DAT_RETURN dapl_provider_list_insert(const char *name, - struct dat_provider **p_data) +u32 dapl_provider_list_insert(const char *name, struct dat_provider **p_data) { DAPL_PROVIDER_LIST_NODE *cur_node, *prev_node, *next_node; - DAT_RETURN status; + u32 status = DAT_SUCCESS; unsigned int len; - status = DAT_SUCCESS; - cur_node = kmalloc(sizeof(DAPL_PROVIDER_LIST_NODE), GFP_ATOMIC); if (NULL == cur_node) { @@ -266,13 +261,10 @@ DAT_RETURN dapl_provider_list_insert(con return status; } -DAT_RETURN dapl_provider_list_search(const char *name, - struct dat_provider **p_data) +u32 dapl_provider_list_search(const char *name, struct dat_provider **p_data) { DAPL_PROVIDER_LIST_NODE *cur_node; - DAT_RETURN status; - - status = DAT_ERROR(DAT_PROVIDER_NOT_FOUND, DAT_NAME_NOT_REGISTERED); + u32 status = DAT_ERROR(DAT_PROVIDER_NOT_FOUND, DAT_NAME_NOT_REGISTERED); for (cur_node = g_dapl_provider_list.head->next; g_dapl_provider_list.tail != cur_node; cur_node = cur_node->next) { @@ -290,12 +282,10 @@ DAT_RETURN dapl_provider_list_search(con return status; } -DAT_RETURN dapl_provider_list_remove(const char *name) +u32 dapl_provider_list_remove(const char *name) { DAPL_PROVIDER_LIST_NODE *cur_node, *prev_node, *next_node; - DAT_RETURN status; - - status = DAT_ERROR(DAT_PROVIDER_NOT_FOUND, DAT_NAME_NOT_REGISTERED); + u32 status = DAT_ERROR(DAT_PROVIDER_NOT_FOUND, DAT_NAME_NOT_REGISTERED); for (cur_node = g_dapl_provider_list.head->next; g_dapl_provider_list.tail != cur_node; cur_node = cur_node->next) { diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_provider.h linux-kernel2/dat-provider/dapl_provider.h --- linux-kernel/dat-provider/dapl_provider.h 2005-05-09 11:07:01.934000000 -0700 +++ linux-kernel2/dat-provider/dapl_provider.h 2005-05-11 13:51:03.534001000 -0700 @@ -74,18 +74,18 @@ extern struct dat_provider g_dapl_provid * * *********************************************************************/ -extern DAT_RETURN dapl_provider_list_create(void); +extern u32 dapl_provider_list_create(void); -extern DAT_RETURN dapl_provider_list_destroy(void); +extern u32 dapl_provider_list_destroy(void); extern DAT_COUNT dapl_provider_list_size(void); -extern DAT_RETURN -dapl_provider_list_insert(const char *name, struct dat_provider **p_data); +extern u32 dapl_provider_list_insert(const char *name, + struct dat_provider **p_data); -extern DAT_RETURN -dapl_provider_list_search(const char *name, struct dat_provider **p_data); +extern u32 dapl_provider_list_search(const char *name, + struct dat_provider **p_data); -extern DAT_RETURN dapl_provider_list_remove(const char *name); +extern u32 dapl_provider_list_remove(const char *name); #endif /* DAPL_PROVIDER_H */ diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_psp_create_any.c linux-kernel2/dat-provider/dapl_psp_create_any.c --- linux-kernel/dat-provider/dapl_psp_create_any.c 2005-05-11 08:28:11.754000000 -0700 +++ linux-kernel2/dat-provider/dapl_psp_create_any.c 2005-05-11 13:51:42.543040000 -0700 @@ -70,22 +70,19 @@ * DAT_CONN_QUAL_IN_USE * DAT_MODEL_NOT_SUPPORTED */ -DAT_RETURN -dapl_psp_create_any(DAT_IA_HANDLE ia_handle, - DAT_CONN_QUAL * conn_qual, - DAT_EVD_HANDLE evd_handle, - DAT_PSP_FLAGS psp_flags, DAT_PSP_HANDLE * psp_handle) +u32 dapl_psp_create_any(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL *conn_qual, + DAT_EVD_HANDLE evd_handle, DAT_PSP_FLAGS psp_flags, + DAT_PSP_HANDLE *psp_handle) { DAPL_IA *ia_ptr; DAPL_SP *sp_ptr; DAPL_EVD *evd_ptr; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; static DAT_CONN_QUAL hint_conn_qual = 1000; /* seed value */ DAT_CONN_QUAL lcl_conn_qual; DAT_CONN_QUAL limit_conn_qual; ia_ptr = (DAPL_IA *) ia_handle; - dat_status = DAT_SUCCESS; if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { dat_status = diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_psp_create.c linux-kernel2/dat-provider/dapl_psp_create.c --- linux-kernel/dat-provider/dapl_psp_create.c 2005-05-11 08:28:11.632017000 -0700 +++ linux-kernel2/dat-provider/dapl_psp_create.c 2005-05-11 13:52:10.176006000 -0700 @@ -67,20 +67,17 @@ * DAT_CONN_QUAL_IN_USE * DAT_MODEL_NOT_SUPPORTED */ -DAT_RETURN -dapl_psp_create(DAT_IA_HANDLE ia_handle, - DAT_CONN_QUAL conn_qual, - DAT_EVD_HANDLE evd_handle, - DAT_PSP_FLAGS psp_flags, DAT_PSP_HANDLE * psp_handle) +u32 dapl_psp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, + DAT_EVD_HANDLE evd_handle, DAT_PSP_FLAGS psp_flags, + DAT_PSP_HANDLE *psp_handle) { DAPL_IA *ia_ptr; DAPL_SP *sp_ptr; DAPL_EVD *evd_ptr; boolean_t sp_found; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; ia_ptr = (DAPL_IA *) ia_handle; - dat_status = DAT_SUCCESS; if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { dat_status = diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_psp_free.c linux-kernel2/dat-provider/dapl_psp_free.c --- linux-kernel/dat-provider/dapl_psp_free.c 2005-05-11 08:28:11.814001000 -0700 +++ linux-kernel2/dat-provider/dapl_psp_free.c 2005-05-11 13:52:46.992015000 -0700 @@ -58,15 +58,14 @@ * DAT_SUCCESS * DAT_INVALID_PARAMETER */ -DAT_RETURN dapl_psp_free(DAT_PSP_HANDLE psp_handle) +u32 dapl_psp_free(DAT_PSP_HANDLE psp_handle) { DAPL_IA *ia_ptr; DAPL_SP *sp_ptr; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; DAPL_SP_STATE save_state; sp_ptr = (DAPL_SP *) psp_handle; - dat_status = DAT_SUCCESS; /* * Verify handle */ diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_psp_query.c linux-kernel2/dat-provider/dapl_psp_query.c --- linux-kernel/dat-provider/dapl_psp_query.c 2005-05-11 08:28:13.131019000 -0700 +++ linux-kernel2/dat-provider/dapl_psp_query.c 2005-05-11 13:53:16.824005000 -0700 @@ -38,11 +38,10 @@ #include "dapl.h" -DAT_RETURN dapl_psp_query(DAT_PSP_HANDLE psp_handle, - struct dat_psp_param *psp_param) +u32 dapl_psp_query(DAT_PSP_HANDLE psp_handle, struct dat_psp_param *psp_param) { DAPL_SP *sp_ptr; - DAT_RETURN status; + u32 status; if (DAPL_BAD_HANDLE(psp_handle, DAPL_MAGIC_PSP) || ((DAPL_SP *) psp_handle)->listening != TRUE) { diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_pz_create.c linux-kernel2/dat-provider/dapl_pz_create.c --- linux-kernel/dat-provider/dapl_pz_create.c 2005-04-20 12:42:07.269030000 -0700 +++ linux-kernel2/dat-provider/dapl_pz_create.c 2005-05-11 13:53:40.203026000 -0700 @@ -55,16 +55,15 @@ * DAT_INVALID_PARAMETER * DAT_INVLAID_HANDLE */ -DAT_RETURN dapl_pz_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE * pz_handle) +u32 dapl_pz_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE *pz_handle) { DAPL_IA *ia; DAPL_PZ *pz; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_create (%p, %p)\n", ia_handle, pz_handle); - dat_status = DAT_SUCCESS; if (DAPL_BAD_HANDLE(ia_handle, DAPL_MAGIC_IA)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_pz_free.c linux-kernel2/dat-provider/dapl_pz_free.c --- linux-kernel/dat-provider/dapl_pz_free.c 2005-04-27 11:38:35.840017000 -0700 +++ linux-kernel2/dat-provider/dapl_pz_free.c 2005-05-11 13:53:55.799011000 -0700 @@ -54,14 +54,13 @@ * DAT_INVALID_STATE * DAT_INVALID_HANDLE */ -DAT_RETURN dapl_pz_free(DAT_PZ_HANDLE pz_handle) +u32 dapl_pz_free(DAT_PZ_HANDLE pz_handle) { DAPL_PZ *pz; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_free (%p)\n", pz_handle); - dat_status = DAT_SUCCESS; if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_pz_query.c linux-kernel2/dat-provider/dapl_pz_query.c --- linux-kernel/dat-provider/dapl_pz_query.c 2005-05-10 08:13:28.647010000 -0700 +++ linux-kernel2/dat-provider/dapl_pz_query.c 2005-05-11 11:16:06.498965000 -0700 @@ -36,10 +36,10 @@ #include "dapl.h" -DAT_RETURN dapl_pz_query(DAT_PZ_HANDLE pz_handle, struct dat_pz_param *pz_param) +u32 dapl_pz_query(DAT_PZ_HANDLE pz_handle, struct dat_pz_param *pz_param) { DAPL_PZ *pz; - DAT_RETURN status; + u32 status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_query (%p, %x, %p)\n", pz_handle, pz_param); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ring_buffer_util.c linux-kernel2/dat-provider/dapl_ring_buffer_util.c --- linux-kernel/dat-provider/dapl_ring_buffer_util.c 2005-04-27 11:38:34.816004000 -0700 +++ linux-kernel2/dat-provider/dapl_ring_buffer_util.c 2005-05-11 13:54:51.243023000 -0700 @@ -56,7 +56,7 @@ * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN dapl_rbuf_alloc(DAPL_RING_BUFFER * rbuf, DAT_COUNT size) +u32 dapl_rbuf_alloc(DAPL_RING_BUFFER *rbuf, DAT_COUNT size) { unsigned int rsize; /* real size */ @@ -104,13 +104,11 @@ DAT_RETURN dapl_rbuf_alloc(DAPL_RING_BUF * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN dapl_rbuf_realloc(DAPL_RING_BUFFER * rbuf, DAT_COUNT size) +u32 dapl_rbuf_realloc(DAPL_RING_BUFFER *rbuf, DAT_COUNT size) { DAPL_RING_BUFFER new_rbuf; void *entry; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; /* decreasing the size or retaining the old size is not allowed */ if (size <= rbuf->lim + 1) @@ -191,7 +189,7 @@ void dapl_rbuf_destroy(DAPL_RING_BUFFER * DAT_INSUFFICIENT_RESOURCES (queue full) * */ -DAT_RETURN dapl_rbuf_add(DAPL_RING_BUFFER * rbuf, void *entry) +u32 dapl_rbuf_add(DAPL_RING_BUFFER *rbuf, void *entry) { int pos; int val; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ring_buffer_util.h linux-kernel2/dat-provider/dapl_ring_buffer_util.h --- linux-kernel/dat-provider/dapl_ring_buffer_util.h 2005-04-27 11:38:34.911019000 -0700 +++ linux-kernel2/dat-provider/dapl_ring_buffer_util.h 2005-05-11 13:55:12.302088000 -0700 @@ -42,13 +42,13 @@ /* * Prototypes */ -DAT_RETURN dapl_rbuf_alloc(DAPL_RING_BUFFER * rbuf, DAT_COUNT size); +u32 dapl_rbuf_alloc(DAPL_RING_BUFFER *rbuf, DAT_COUNT size); -DAT_RETURN dapl_rbuf_realloc(DAPL_RING_BUFFER * rbuf, DAT_COUNT size); +u32 dapl_rbuf_realloc(DAPL_RING_BUFFER *rbuf, DAT_COUNT size); void dapl_rbuf_destroy(DAPL_RING_BUFFER * rbuf); -DAT_RETURN dapl_rbuf_add(DAPL_RING_BUFFER * rbuf, void *entry); +u32 dapl_rbuf_add(DAPL_RING_BUFFER *rbuf, void *entry); void *dapl_rbuf_remove(DAPL_RING_BUFFER * rbuf); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_rmr_bind.c linux-kernel2/dat-provider/dapl_rmr_bind.c --- linux-kernel/dat-provider/dapl_rmr_bind.c 2005-05-11 10:09:59.093023000 -0700 +++ linux-kernel2/dat-provider/dapl_rmr_bind.c 2005-05-11 13:56:47.864000000 -0700 @@ -46,18 +46,17 @@ * * *********************************************************************/ -static inline DAT_RETURN -dapl_rmr_bind_fuse(DAPL_RMR * rmr, - const struct dat_lmr_triplet *lmr_triplet, - DAT_MEM_PRIV_FLAGS mem_priv, - DAPL_EP * ep_ptr, - DAT_RMR_COOKIE user_cookie, - DAT_COMPLETION_FLAGS completion_flags, - DAT_RMR_CONTEXT * rmr_context) +static inline u32 dapl_rmr_bind_fuse(DAPL_RMR *rmr, + const struct dat_lmr_triplet *lmr_triplet, + DAT_MEM_PRIV_FLAGS mem_priv, + DAPL_EP *ep_ptr, + DAT_RMR_COOKIE user_cookie, + DAT_COMPLETION_FLAGS completion_flags, + DAT_RMR_CONTEXT *rmr_context) { DAPL_LMR *lmr; DAPL_COOKIE *cookie; - DAT_RETURN status; + u32 status; boolean_t is_signaled; status = dapl_hash_search(rmr->header.owner_ia->hca_ptr->lmr_hash_table, @@ -157,17 +156,14 @@ dapl_rmr_bind_fuse(DAPL_RMR * rmr, return status; } -static inline DAT_RETURN -dapl_rmr_bind_unfuse(DAPL_RMR * rmr, - DAPL_EP * ep_ptr, - DAT_RMR_COOKIE user_cookie, - DAT_COMPLETION_FLAGS completion_flags) +static inline u32 dapl_rmr_bind_unfuse(DAPL_RMR *rmr, DAPL_EP *ep_ptr, + DAT_RMR_COOKIE user_cookie, + DAT_COMPLETION_FLAGS completion_flags) { DAPL_COOKIE *cookie; - DAT_RETURN status; + u32 status = DAT_SUCCESS; boolean_t is_signaled; - status = DAT_SUCCESS; /* * if the ep in unconnected return an error. IB requires that the * QP be connected to change a memory window binding since: @@ -253,13 +249,12 @@ dapl_rmr_bind_unfuse(DAPL_RMR * rmr, * Input: * Output: */ -DAT_RETURN dapl_rmr_bind(DAT_RMR_HANDLE rmr_handle, - const struct dat_lmr_triplet *lmr_triplet, - DAT_MEM_PRIV_FLAGS mem_priv, - DAT_EP_HANDLE ep_handle, - DAT_RMR_COOKIE user_cookie, - DAT_COMPLETION_FLAGS completion_flags, - DAT_RMR_CONTEXT * rmr_context) +u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_handle, + const struct dat_lmr_triplet *lmr_triplet, + DAT_MEM_PRIV_FLAGS mem_priv, DAT_EP_HANDLE ep_handle, + DAT_RMR_COOKIE user_cookie, + DAT_COMPLETION_FLAGS completion_flags, + DAT_RMR_CONTEXT *rmr_context) { DAPL_RMR *rmr; DAPL_EP *ep_ptr; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_rmr_create.c linux-kernel2/dat-provider/dapl_rmr_create.c --- linux-kernel/dat-provider/dapl_rmr_create.c 2005-04-27 11:38:34.778010000 -0700 +++ linux-kernel2/dat-provider/dapl_rmr_create.c 2005-05-11 13:57:10.428101000 -0700 @@ -53,13 +53,11 @@ * DAT_INSUFFICIENT_RESOURCES * DAT_INVALID_PARAMETER */ -DAT_RETURN dapl_rmr_create(DAT_PZ_HANDLE pz_handle, DAT_RMR_HANDLE * rmr_handle) +u32 dapl_rmr_create(DAT_PZ_HANDLE pz_handle, DAT_RMR_HANDLE *rmr_handle) { DAPL_PZ *pz; DAPL_RMR *rmr; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { dat_status = diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_rmr_free.c linux-kernel2/dat-provider/dapl_rmr_free.c --- linux-kernel/dat-provider/dapl_rmr_free.c 2005-04-27 11:38:35.027007000 -0700 +++ linux-kernel2/dat-provider/dapl_rmr_free.c 2005-05-11 13:57:34.719027000 -0700 @@ -53,12 +53,10 @@ * DAT_SUCCESS * DAT_INVALID_PARAMETER */ -DAT_RETURN dapl_rmr_free(DAT_RMR_HANDLE rmr_handle) +u32 dapl_rmr_free(DAT_RMR_HANDLE rmr_handle) { DAPL_RMR *rmr; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) { dat_status = diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_rmr_query.c linux-kernel2/dat-provider/dapl_rmr_query.c --- linux-kernel/dat-provider/dapl_rmr_query.c 2005-05-10 08:13:28.721018000 -0700 +++ linux-kernel2/dat-provider/dapl_rmr_query.c 2005-05-11 13:57:51.347004000 -0700 @@ -38,13 +38,10 @@ #include "dapl.h" -DAT_RETURN dapl_rmr_query(DAT_RMR_HANDLE rmr_handle, - struct dat_rmr_param *rmr_param) +u32 dapl_rmr_query(DAT_RMR_HANDLE rmr_handle, struct dat_rmr_param *rmr_param) { DAPL_RMR *rmr; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) { dat_status = diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_rsp_create.c linux-kernel2/dat-provider/dapl_rsp_create.c --- linux-kernel/dat-provider/dapl_rsp_create.c 2005-05-11 08:28:12.578010000 -0700 +++ linux-kernel2/dat-provider/dapl_rsp_create.c 2005-05-11 13:58:29.482113000 -0700 @@ -68,20 +68,17 @@ * DAT_INVALID_STATE * DAT_CONN_QUAL_IN_USE */ -DAT_RETURN -dapl_rsp_create(DAT_IA_HANDLE ia_handle, - DAT_CONN_QUAL conn_qual, - DAT_EP_HANDLE ep_handle, - DAT_EVD_HANDLE evd_handle, DAT_RSP_HANDLE * rsp_handle) +u32 dapl_rsp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, + DAT_EP_HANDLE ep_handle, DAT_EVD_HANDLE evd_handle, + DAT_RSP_HANDLE *rsp_handle) { DAPL_IA *ia_ptr; DAPL_SP *sp_ptr; DAPL_EVD *evd_ptr; DAPL_EP *ep_ptr; boolean_t sp_found; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; - dat_status = DAT_SUCCESS; ia_ptr = (DAPL_IA *) ia_handle; dapl_dbg_log(DAPL_DBG_TYPE_CM, diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_rsp_free.c linux-kernel2/dat-provider/dapl_rsp_free.c --- linux-kernel/dat-provider/dapl_rsp_free.c 2005-05-11 08:28:12.737008000 -0700 +++ linux-kernel2/dat-provider/dapl_rsp_free.c 2005-05-11 13:58:44.765006000 -0700 @@ -58,14 +58,12 @@ * DAT_SUCCESS * DAT_INVALID_HANDLE */ -DAT_RETURN dapl_rsp_free(DAT_RSP_HANDLE rsp_handle) +u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_handle) { DAPL_IA *ia_ptr; DAPL_SP *sp_ptr; DAPL_EP *ep_ptr; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; sp_ptr = (DAPL_SP *) rsp_handle; /* diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_rsp_query.c linux-kernel2/dat-provider/dapl_rsp_query.c --- linux-kernel/dat-provider/dapl_rsp_query.c 2005-05-10 08:13:29.299014000 -0700 +++ linux-kernel2/dat-provider/dapl_rsp_query.c 2005-05-11 13:58:53.725005000 -0700 @@ -38,11 +38,10 @@ #include "dapl.h" -DAT_RETURN dapl_rsp_query(DAT_RSP_HANDLE rsp_handle, - struct dat_rsp_param *rsp_param) +u32 dapl_rsp_query(DAT_RSP_HANDLE rsp_handle, struct dat_rsp_param *rsp_param) { DAPL_SP *sp_ptr; - DAT_RETURN status; + u32 status; if (DAPL_BAD_HANDLE(rsp_handle, DAPL_MAGIC_RSP)) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RSP); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_set_consumer_context.c linux-kernel2/dat-provider/dapl_set_consumer_context.c --- linux-kernel/dat-provider/dapl_set_consumer_context.c 2005-04-20 12:42:07.233010000 -0700 +++ linux-kernel2/dat-provider/dapl_set_consumer_context.c 2005-05-11 13:59:07.265009000 -0700 @@ -56,13 +56,11 @@ * DAT_SUCCESS * DAT_INVALID_HANDLE */ -DAT_RETURN dapl_set_consumer_context(DAT_HANDLE dat_handle, DAT_CONTEXT context) +u32 dapl_set_consumer_context(DAT_HANDLE dat_handle, DAT_CONTEXT context) { - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; DAPL_HEADER *header; - dat_status = DAT_SUCCESS; - header = (DAPL_HEADER *) dat_handle; if (((header) == NULL) || ((unsigned long)(header) & 3) || diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_srq_create.c linux-kernel2/dat-provider/dapl_srq_create.c --- linux-kernel/dat-provider/dapl_srq_create.c 2005-05-10 08:13:28.397009000 -0700 +++ linux-kernel2/dat-provider/dapl_srq_create.c 2005-05-11 13:59:33.909018000 -0700 @@ -65,16 +65,12 @@ * ?DAT_INVALID_ATTRIBUTE?? * DAT_MODEL_NOT_SUPPORTED */ -DAT_RETURN -dapl_srq_create(DAT_IA_HANDLE ia_handle, - DAT_PZ_HANDLE pz_handle, - struct dat_srq_attr *srq_attr, DAT_SRQ_HANDLE * srq_handle) +u32 dapl_srq_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, + struct dat_srq_attr *srq_attr, DAT_SRQ_HANDLE *srq_handle) { DAPL_IA *ia_ptr; DAPL_SRQ *srq_ptr; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_create (%p, %p, %p, %p)\n", diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_srq_free.c linux-kernel2/dat-provider/dapl_srq_free.c --- linux-kernel/dat-provider/dapl_srq_free.c 2005-05-10 08:13:29.041072000 -0700 +++ linux-kernel2/dat-provider/dapl_srq_free.c 2005-05-11 13:59:46.496011000 -0700 @@ -59,14 +59,12 @@ * DAT_INVALID_PARAMETER * DAT_INVALID_STATE */ -DAT_RETURN dapl_srq_free(DAT_SRQ_HANDLE srq_handle) +u32 dapl_srq_free(DAT_SRQ_HANDLE srq_handle) { DAPL_SRQ *srq_ptr; DAPL_IA *ia_ptr; struct dat_srq_param *param; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_free (%p)\n", srq_handle); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_srq_post_recv.c linux-kernel2/dat-provider/dapl_srq_post_recv.c --- linux-kernel/dat-provider/dapl_srq_post_recv.c 2005-05-10 08:13:28.985006000 -0700 +++ linux-kernel2/dat-provider/dapl_srq_post_recv.c 2005-05-11 14:00:05.629017000 -0700 @@ -67,15 +67,13 @@ * DAT_PROTECTION_VIOLATION * DAT_PROVILEGES_VIOLATION */ -DAT_RETURN -dapl_srq_post_recv(DAT_SRQ_HANDLE srq_handle, - DAT_COUNT num_segments, - struct dat_lmr_triplet *local_iov, - DAT_DTO_COOKIE user_cookie) +u32 dapl_srq_post_recv(DAT_SRQ_HANDLE srq_handle, DAT_COUNT num_segments, + struct dat_lmr_triplet *local_iov, + DAT_DTO_COOKIE user_cookie) { DAPL_SRQ *srq_ptr; DAPL_COOKIE *cookie; - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_post_recv (%p, %d, %p, %P)\n", diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_srq_query.c linux-kernel2/dat-provider/dapl_srq_query.c --- linux-kernel/dat-provider/dapl_srq_query.c 2005-05-10 08:13:28.434065000 -0700 +++ linux-kernel2/dat-provider/dapl_srq_query.c 2005-05-11 14:00:20.113028000 -0700 @@ -39,13 +39,10 @@ #include "dapl.h" -DAT_RETURN dapl_srq_query(DAT_SRQ_HANDLE srq_handle, - struct dat_srq_param *srq_param) +u32 dapl_srq_query(DAT_SRQ_HANDLE srq_handle, struct dat_srq_param *srq_param) { DAPL_SRQ *srq_ptr; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_query (%p, %x, %p)\n", diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_srq_resize.c linux-kernel2/dat-provider/dapl_srq_resize.c --- linux-kernel/dat-provider/dapl_srq_resize.c 2005-04-20 12:42:06.684011000 -0700 +++ linux-kernel2/dat-provider/dapl_srq_resize.c 2005-05-11 14:00:32.574007000 -0700 @@ -63,14 +63,11 @@ * DAT_INVALID_STATE */ -DAT_RETURN -dapl_srq_resize(DAT_SRQ_HANDLE srq_handle, DAT_COUNT srq_max_recv_dto) +u32 dapl_srq_resize(DAT_SRQ_HANDLE srq_handle, DAT_COUNT srq_max_recv_dto) { DAPL_IA *ia_ptr; DAPL_SRQ *srq_ptr; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_resize (%p, %d)\n", srq_handle, srq_max_recv_dto); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_srq_set_lw.c linux-kernel2/dat-provider/dapl_srq_set_lw.c --- linux-kernel/dat-provider/dapl_srq_set_lw.c 2005-04-20 12:42:07.855013000 -0700 +++ linux-kernel2/dat-provider/dapl_srq_set_lw.c 2005-05-11 14:00:43.964001000 -0700 @@ -63,12 +63,10 @@ * DAT_MODEL_NOT_SUPPORTED */ -DAT_RETURN dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, DAT_COUNT low_watermark) +u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, DAT_COUNT low_watermark) { DAPL_SRQ *srq_ptr; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_set_lw (%p, %d)\n", srq_handle, low_watermark); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_timer_util.c linux-kernel2/dat-provider/dapl_timer_util.c --- linux-kernel/dat-provider/dapl_timer_util.c 2005-05-11 08:28:12.521015000 -0700 +++ linux-kernel2/dat-provider/dapl_timer_util.c 2005-05-11 14:01:14.200017000 -0700 @@ -91,9 +91,8 @@ void dapl_timer_init(void) * no return value * */ -DAT_RETURN -dapl_timer_set(DAPL_OS_TIMER * timer, - void (*func) (uintptr_t), void *data, DAPL_OS_TIMEVAL expires) +u32 dapl_timer_set(DAPL_OS_TIMER *timer, void (*func) (uintptr_t), + void *data, DAPL_OS_TIMEVAL expires) { DAPL_OS_TIMER *list_ptr; DAPL_OS_TIMEVAL cur_time; @@ -240,7 +239,7 @@ void dapl_timer_thread(void *arg) { DAPL_OS_TIMER *list_ptr; DAPL_OS_TIMEVAL cur_time; - DAT_RETURN dat_status; + u32 dat_status; DAPL_TIMER_HEAD *timer_head; timer_head = arg; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_timer_util.h linux-kernel2/dat-provider/dapl_timer_util.h --- linux-kernel/dat-provider/dapl_timer_util.h 2005-04-20 12:42:07.873010000 -0700 +++ linux-kernel2/dat-provider/dapl_timer_util.h 2005-05-11 14:01:34.201022000 -0700 @@ -37,8 +37,7 @@ void dapl_timer_init(void); -DAT_RETURN dapl_timer_set(DAPL_OS_TIMER * timer, - void (*func) (uintptr_t), - void *data, DAPL_OS_TIMEVAL expires); +u32 dapl_timer_set(DAPL_OS_TIMER *timer, void (*func) (uintptr_t), + void *data, DAPL_OS_TIMEVAL expires); void dapl_timer_cancel(DAPL_OS_TIMER * timer); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_util.c linux-kernel2/dat-provider/dapl_util.c --- linux-kernel/dat-provider/dapl_util.c 2005-05-06 16:22:03.058010000 -0700 +++ linux-kernel2/dat-provider/dapl_util.c 2005-05-11 14:02:26.035004000 -0700 @@ -55,7 +55,7 @@ * DAT_SUCCESS */ -DAT_RETURN dapl_os_get_time(DAPL_OS_TIMEVAL * loc) +u32 dapl_os_get_time(DAPL_OS_TIMEVAL *loc) { struct timeval tv; @@ -81,7 +81,7 @@ DAT_RETURN dapl_os_get_time(DAPL_OS_TIME * DAT_SUCCESS * DAT_INTERNAL_ERROR */ -DAT_RETURN dapl_os_wait_object_init(DAPL_OS_WAIT_OBJECT * wait_obj) +u32 dapl_os_wait_object_init(DAPL_OS_WAIT_OBJECT *wait_obj) { init_waitqueue_head(&wait_obj->wait_queue); @@ -101,13 +101,10 @@ DAT_RETURN dapl_os_wait_object_init(DAPL * DAT_TIMEOUT -- the specified time limit was reached. */ -DAT_RETURN -dapl_os_wait_object_wait(DAPL_OS_WAIT_OBJECT * wait_obj, - DAT_TIMEOUT timeout_val) +u32 dapl_os_wait_object_wait(DAPL_OS_WAIT_OBJECT *wait_obj, + DAT_TIMEOUT timeout_val) { - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; if (DAT_TIMEOUT_INFINITE == timeout_val) { interruptible_sleep_on(&wait_obj->wait_queue); @@ -137,7 +134,7 @@ dapl_os_wait_object_wait(DAPL_OS_WAIT_OB * DAT_SUCCESS * DAT_INTERNAL_ERROR */ -DAT_RETURN dapl_os_wait_object_wakeup(DAPL_OS_WAIT_OBJECT * wait_obj) +u32 dapl_os_wait_object_wakeup(DAPL_OS_WAIT_OBJECT *wait_obj) { wake_up_interruptible(&wait_obj->wait_queue); @@ -156,7 +153,7 @@ DAT_RETURN dapl_os_wait_object_wakeup(DA * DAT_SUCCESS * DAT_INTERNAL_ERROR */ -DAT_RETURN dapl_os_wait_object_destroy(DAPL_OS_WAIT_OBJECT * wait_obj) +u32 dapl_os_wait_object_destroy(DAPL_OS_WAIT_OBJECT *wait_obj) { return DAT_SUCCESS; @@ -188,9 +185,8 @@ void dapl_thread_init(struct thread_draf * Returns: * DAT_SUCCESS */ -DAT_RETURN -dapl_os_thread_create(void (*func) (void *), - void *data, DAPL_OS_THREAD * thread_id) +u32 dapl_os_thread_create(void (*func) (void *), void *data, + DAPL_OS_THREAD * thread_id) { struct thread_draft *thread_draft; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_util.h linux-kernel2/dat-provider/dapl_util.h --- linux-kernel/dat-provider/dapl_util.h 2005-05-06 16:22:03.118012000 -0700 +++ linux-kernel2/dat-provider/dapl_util.h 2005-05-11 14:03:17.730035000 -0700 @@ -125,9 +125,8 @@ dapl_os_atomic_assign(atomic_t * v, DAT_ */ typedef int DAPL_OS_THREAD; -DAT_RETURN -dapl_os_thread_create(void (*func) (void *), - void *data, DAPL_OS_THREAD * thread_id); +u32 dapl_os_thread_create(void (*func) (void *), void *data, + DAPL_OS_THREAD *thread_id); /* * Wait Objects @@ -144,15 +143,14 @@ typedef struct _dapl_os_wait_object_ { } DAPL_OS_WAIT_OBJECT; /* function prototypes */ -DAT_RETURN dapl_os_wait_object_init(DAPL_OS_WAIT_OBJECT * wait_obj); +u32 dapl_os_wait_object_init(DAPL_OS_WAIT_OBJECT *wait_obj); -DAT_RETURN -dapl_os_wait_object_wait(DAPL_OS_WAIT_OBJECT * wait_obj, - DAT_TIMEOUT timeout_val); +u32 dapl_os_wait_object_wait(DAPL_OS_WAIT_OBJECT *wait_obj, + DAT_TIMEOUT timeout_val); -DAT_RETURN dapl_os_wait_object_wakeup(DAPL_OS_WAIT_OBJECT * wait_obj); +u32 dapl_os_wait_object_wakeup(DAPL_OS_WAIT_OBJECT *wait_obj); -DAT_RETURN dapl_os_wait_object_destroy(DAPL_OS_WAIT_OBJECT * wait_obj); +u32 dapl_os_wait_object_destroy(DAPL_OS_WAIT_OBJECT *wait_obj); /* * Memory Functions @@ -193,7 +191,7 @@ typedef struct dapl_timer_entry DAPL_OS_ typedef unsigned long long int DAPL_OS_TICKS; /* timer function prototype */ -DAT_RETURN dapl_os_get_time(DAPL_OS_TIMEVAL *); +u32 dapl_os_get_time(DAPL_OS_TIMEVAL *); /* * *printf format helpers. We use the C string constant concatenation @@ -224,12 +222,11 @@ static __inline__ long dapl_os_strtol(co /* * OS inter module functions to obtain and release verbs list */ -DAT_RETURN dapl_os_get_verbs_vector(void **verbs_list, - char *list_name, char *driver_name); +u32 dapl_os_get_verbs_vector(void **verbs_list, char *list_name, + char *driver_name); -DAT_RETURN -dapl_os_release_verbs_vector(void **verbs_list, - char *list_name, char *driver_name); +u32 dapl_os_release_verbs_vector(void **verbs_list, char *list_name, + char *driver_name); /* * helper routines diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/include/dapl_proto.h linux-kernel2/test/dapltest/include/dapl_proto.h --- linux-kernel/test/dapltest/include/dapl_proto.h 2005-05-11 08:52:07.424000000 -0700 +++ linux-kernel2/test/dapltest/include/dapl_proto.h 2005-05-11 11:16:06.658962000 -0700 @@ -523,7 +523,7 @@ boolean_t DT_check_params (Pe void DT_Test_Error (void); /* dapl_util.c */ -const char *DT_RetToString (DAT_RETURN ret_value); +const char *DT_RetToString (u32 ret_value); const char *DT_TransferTypeToString (DT_Transfer_Type type); @@ -640,7 +640,7 @@ void DT_mem_test (Params_t *params_p /* dapl_fft_queryinfo.c */ int DT_queryinfo_basic (Params_t *params_ptr, FFT_Cmd_t *cmd, FFT_query_enum object_to_query, - DAT_RETURN result_wanted); + u32 result_wanted); int DT_queryinfo_case0 (Params_t *params_ptr, FFT_Cmd_t *cmd); int DT_queryinfo_case1 (Params_t *params_ptr, FFT_Cmd_t *cmd); int DT_queryinfo_case2 (Params_t *params_ptr, FFT_Cmd_t *cmd); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/include/dapl_tdep.h linux-kernel2/test/dapltest/include/dapl_tdep.h --- linux-kernel/test/dapltest/include/dapl_tdep.h 2005-05-10 08:13:26.796006000 -0700 +++ linux-kernel2/test/dapltest/include/dapl_tdep.h 2005-05-11 11:16:06.664963000 -0700 @@ -44,21 +44,21 @@ DT_Tdep_End ( void ) ; int DT_Tdep_Execute_Test ( Params_t *params_ptr ) ; -DAT_RETURN +u32 DT_Tdep_evd_create (DAT_IA_HANDLE ia_handle, DAT_COUNT evd_min_qlen, DAT_CNO_HANDLE cno_handle, DAT_EVD_FLAGS evd_flags, DAT_EVD_HANDLE *evd_handle_ptr); -DAT_RETURN +u32 DT_Tdep_evd_free (DAT_EVD_HANDLE evd_handle); -DAT_RETURN +u32 DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_handle, DAT_TIMEOUT timeout, struct dat_event *event); -DAT_RETURN +u32 DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_handle, struct dat_event *event); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c linux-kernel2/test/dapltest/kdapl/kdapl_tdep_evd.c --- linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c 2005-05-10 10:19:18.034000000 -0700 +++ linux-kernel2/test/dapltest/kdapl/kdapl_tdep_evd.c 2005-05-11 11:16:06.670966000 -0700 @@ -94,7 +94,7 @@ KDT_Evd_Destroy(void) DT_Mdep_LockDestroy (&DT_Evd_Lock); } -DAT_RETURN +u32 DT_Tdep_evd_create (DAT_IA_HANDLE ia_handle, DAT_COUNT evd_min_qlen, DAT_CNO_HANDLE cno_handle, @@ -102,7 +102,7 @@ DT_Tdep_evd_create (DAT_IA_HANDLE ia_ha DAT_EVD_HANDLE *evd_handle_ptr) { - DAT_RETURN dat_status; + u32 dat_status; struct dat_upcall_object upcall; Tdep_Evd *evd_ptr; @@ -154,11 +154,11 @@ error: } return (dat_status); } -DAT_RETURN +u32 DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_handle, struct dat_event *dat_event) { - DAT_RETURN dat_status; + u32 dat_status; Tdep_Evd *evd_ptr; Tdep_Event *event; @@ -212,12 +212,12 @@ DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_ return dat_status; } -DAT_RETURN +u32 DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_handle, DAT_TIMEOUT timeout, struct dat_event *dat_event) { - DAT_RETURN dat_status; + u32 dat_status; Tdep_Evd *evd_ptr; Tdep_Event *event; int rc; @@ -290,7 +290,7 @@ DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_han return dat_status; } -DAT_RETURN +u32 DT_Tdep_evd_free (DAT_EVD_HANDLE evd_handle) { Tdep_Evd *next; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_bpool.c linux-kernel2/test/dapltest/test/dapl_bpool.c --- linux-kernel/test/dapltest/test/dapl_bpool.c 2005-05-11 09:01:40.530003000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_bpool.c 2005-05-11 11:16:06.675963000 -0700 @@ -101,7 +101,7 @@ DT_BpoolAlloc ( Bpool *bpool_ptr = NULL; DAT_COUNT alloc_size, bpool_size; DAT_REGION_DESCRIPTION region; - DAT_RETURN ret = DAT_SUCCESS; + u32 ret = DAT_SUCCESS; DAT_VLEN bp_len; DAT_MEM_TYPE DT_mem_type; u64 buf_region[10]; @@ -328,7 +328,7 @@ DT_Bpool_Destroy (Per_Test_Data_t * pt_p { struct dat_lmr_triplet iov; DAT_RMR_COOKIE cookie; - DAT_RETURN ret; + u32 ret; iov.virtual_address = bpool_ptr->reg_addr; iov.segment_length = 0; /* un-bind */ @@ -359,7 +359,7 @@ DT_Bpool_Destroy (Per_Test_Data_t * pt_p if (bpool_ptr->lmr_handle) { - DAT_RETURN ret = dat_lmr_free (bpool_ptr->lmr_handle); + u32 ret = dat_lmr_free (bpool_ptr->lmr_handle); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_client.c linux-kernel2/test/dapltest/test/dapl_client.c --- linux-kernel/test/dapltest/test/dapl_client.c 2005-05-11 08:59:09.049005000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_client.c 2005-05-11 11:16:06.680966000 -0700 @@ -63,7 +63,7 @@ DT_cs_Client (Params_t * params_ptr, struct dat_dto_completion_event_data dto_stat; DAT_EVENT_NUMBER event_num; unsigned char * buffp; - DAT_RETURN ret; + u32 ret; DT_Tdep_Print_Head *phead; int status = 0; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_cnxn.c linux-kernel2/test/dapltest/test/dapl_cnxn.c --- linux-kernel/test/dapltest/test/dapl_cnxn.c 2005-05-10 10:19:18.044000000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_cnxn.c 2005-05-11 11:16:06.687963000 -0700 @@ -34,7 +34,7 @@ get_ep_connection_state (DT_Tdep_Print_H DAT_EP_STATE ep_state; boolean_t in_dto_idle; boolean_t out_dto_idle; - DAT_RETURN ret; + u32 ret; char *recv_status = "Idle"; char *req_status = "Idle"; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_fft_connmgt.c linux-kernel2/test/dapltest/test/dapl_fft_connmgt.c --- linux-kernel/test/dapltest/test/dapl_fft_connmgt.c 2005-04-06 22:02:35.416990000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_fft_connmgt.c 2005-05-11 11:16:06.693960000 -0700 @@ -31,7 +31,7 @@ int DT_connmgt_case0 (Params_t *params_p { FFT_Connection_t conn; int res = 1; - DAT_RETURN rc = 0; + u32 rc = 0; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; @@ -54,7 +54,7 @@ int DT_connmgt_case1 (Params_t *params_p { FFT_Connection_t conn; int res = 1; - DAT_RETURN rc; + u32 rc; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_fft_dataxfer.c linux-kernel2/test/dapltest/test/dapl_fft_dataxfer.c --- linux-kernel/test/dapltest/test/dapl_fft_dataxfer.c 2005-04-06 22:02:35.866991000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_fft_dataxfer.c 2005-05-11 11:16:06.696963000 -0700 @@ -31,7 +31,7 @@ int DT_dataxfer_generic ( DT_Tdep_Print_Head * phead, FFT_Cmd_t *cmd, int test_case) { FFT_Connection_t conn; - DAT_RETURN rc=0; + u32 rc=0; int res=1; DT_fft_init_server (phead, cmd, &conn); DT_assert (phead, NULL != conn.ia_handle); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_fft_dataxfer_client.c linux-kernel2/test/dapltest/test/dapl_fft_dataxfer_client.c --- linux-kernel/test/dapltest/test/dapl_fft_dataxfer_client.c 2005-05-11 08:59:09.052008000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_fft_dataxfer_client.c 2005-05-11 11:16:06.700967000 -0700 @@ -37,7 +37,7 @@ int DT_dataxfer_client_generic (DT_Tdep_ { int res=1; FFT_Connection_t conn; - DAT_RETURN rc=0; + u32 rc=0; DT_fft_init_client (phead, cmd, &conn); DT_assert_dat (phead, conn.ia_handle != NULL) diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_fft_endpoint.c linux-kernel2/test/dapltest/test/dapl_fft_endpoint.c --- linux-kernel/test/dapltest/test/dapl_fft_endpoint.c 2005-05-11 08:59:09.055001000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_fft_endpoint.c 2005-05-11 11:16:06.705962000 -0700 @@ -43,7 +43,7 @@ int DT_endpoint_generic (Params_t *param DAT_EVD_HANDLE conn_evd_handle; DAT_EVD_HANDLE send_evd_handle; DAT_EVD_HANDLE recv_evd_handle; - DAT_RETURN rc, wanted; + u32 rc, wanted; int res; DT_Tdep_Print_Head *phead; @@ -171,7 +171,7 @@ int DT_endpoint_case2 (Params_t *params_ struct dat_event event; Bpool *bpool; int res; - DAT_RETURN rc; + u32 rc; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; DT_Tdep_PT_Printf (phead, "\ diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_fft_hwconn.c linux-kernel2/test/dapltest/test/dapl_fft_hwconn.c --- linux-kernel/test/dapltest/test/dapl_fft_hwconn.c 2005-05-05 13:35:54.484001000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_fft_hwconn.c 2005-05-11 11:16:06.710964000 -0700 @@ -33,7 +33,7 @@ int DT_hwconn_case0 ( Params_t *params_p char* dev_name; DAT_IA_HANDLE nic_handle; DAT_EVD_HANDLE evd_handle; - DAT_RETURN rc; + u32 rc; int res = 1; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; @@ -59,7 +59,7 @@ cleanup: int DT_hwconn_case1 ( Params_t *params_ptr, FFT_Cmd_t *cmd) { DAT_IA_HANDLE nic_handle; - DAT_RETURN rc; + u32 rc; DAT_EVD_HANDLE evd_handle; char dev_name[100]; int i; @@ -121,7 +121,7 @@ int DT_hwconn_case1 ( Params_t *params_p int DT_hwconn_case2 (Params_t *params_ptr, FFT_Cmd_t *cmd) { DAT_IA_HANDLE nic_handle; - DAT_RETURN rc; + u32 rc; int res=1; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; @@ -140,7 +140,7 @@ cleanup: int DT_hwconn_case3 (Params_t *params_ptr, FFT_Cmd_t *cmd) { FFT_Connection_t conn; - DAT_RETURN rc; + u32 rc; int res; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_fft_mem.c linux-kernel2/test/dapltest/test/dapl_fft_mem.c --- linux-kernel/test/dapltest/test/dapl_fft_mem.c 2005-05-02 12:52:13.731015000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_fft_mem.c 2005-05-11 11:16:06.715962000 -0700 @@ -33,7 +33,7 @@ /*--------------------------------------------------------*/ int DT_mem_generic (Params_t *params_ptr, FFT_Cmd_t *cmd, int flag) { - DAT_RETURN rc, expect; + u32 rc, expect; FFT_Connection_t conn; DAT_REGION_DESCRIPTION region; DAT_VLEN reg_size; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_fft_pz.c linux-kernel2/test/dapltest/test/dapl_fft_pz.c --- linux-kernel/test/dapltest/test/dapl_fft_pz.c 2005-05-11 08:59:09.058004000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_fft_pz.c 2005-05-11 11:16:06.722964000 -0700 @@ -38,7 +38,7 @@ int DT_pz_case0 ( Params_t *params_ptr, DAT_IA_HANDLE ia_handle; DAT_PZ_HANDLE pz_handle; DAT_EVD_HANDLE evd_handle; - DAT_RETURN rc; + u32 rc; int res; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; @@ -79,7 +79,7 @@ int DT_pz_case1 (Params_t *params_ptr, F DAT_PZ_HANDLE pz_handle; DAT_EP_HANDLE ep_handle; DAT_EVD_HANDLE conn_evd, send_evd, recv_evd, cr_evd; - DAT_RETURN rc; + u32 rc; int res; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; @@ -163,7 +163,7 @@ int DT_pz_case2 (Params_t *params_ptr, F DAT_IA_HANDLE ia_handle; DAT_PZ_HANDLE pz_handle; Bpool *bpool; - DAT_RETURN rc; + u32 rc; int res; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c linux-kernel2/test/dapltest/test/dapl_fft_queryinfo.c --- linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c 2005-05-10 08:16:11.583008000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_fft_queryinfo.c 2005-05-11 11:16:06.726963000 -0700 @@ -41,7 +41,7 @@ static DAT_OS_WAIT_PROXY_AGENT NULLPROXY int DT_queryinfo_basic (Params_t *params_ptr, FFT_Cmd_t *cmd, FFT_query_enum object_to_query, - DAT_RETURN result_wanted) + u32 result_wanted) { char *dev_name; DAT_IA_HANDLE ia_handle; @@ -77,7 +77,7 @@ int DT_queryinfo_basic (Params_t *params unsigned char *alloc_ptr; DT_Tdep_Print_Head *phead; - DAT_RETURN rc; + u32 rc; int res = 1; buffer_size = BUFFSIZE * sizeof (unsigned char); phead = params_ptr->phead; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_fft_util.c linux-kernel2/test/dapltest/test/dapl_fft_util.c --- linux-kernel/test/dapltest/test/dapl_fft_util.c 2005-05-11 08:59:09.062007000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_fft_util.c 2005-05-11 11:16:06.731964000 -0700 @@ -72,7 +72,7 @@ int DT_ep_create (Params_t *params_ptr, DAT_EVD_HANDLE *recv_evd, DAT_EP_HANDLE *ep_handle) { - DAT_RETURN status; + u32 status; DT_Tdep_Print_Head *phead; *conn_evd = NULL; *send_evd = NULL; @@ -146,7 +146,7 @@ void DT_fft_init_conn_struct (FFT_Connec void DT_fft_init_client (Params_t *params_ptr, FFT_Cmd_t *cmd, FFT_Connection_t *conn) { int res; - DAT_RETURN rc=0; + u32 rc=0; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; @@ -194,7 +194,7 @@ cleanup: int DT_fft_destroy_conn_struct (Params_t *params_ptr, FFT_Connection_t *conn) { - DAT_RETURN rc = DAT_SUCCESS; + u32 rc = DAT_SUCCESS; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; @@ -264,7 +264,7 @@ int DT_fft_destroy_conn_struct (Params_t void DT_fft_init_server (Params_t *params_ptr, FFT_Cmd_t *cmd, FFT_Connection_t *conn) { int res; - DAT_RETURN rc=0; + u32 rc=0; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; @@ -308,7 +308,7 @@ cleanup: void DT_fft_listen (Params_t *params_ptr, FFT_Connection_t *conn) { int res; - DAT_RETURN rc=0; + u32 rc=0; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; @@ -341,7 +341,7 @@ int DT_fft_connect (Params_t *params_ptr { int wait_count; int res; - DAT_RETURN rc=0; + u32 rc=0; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_limit.c linux-kernel2/test/dapltest/test/dapl_limit.c --- linux-kernel/test/dapltest/test/dapl_limit.c 2005-05-11 09:04:51.554002000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_limit.c 2005-05-11 11:16:06.742963000 -0700 @@ -105,7 +105,7 @@ limit_test ( DT_Tdep_Print_Head *phead, DAT_OS_WAIT_PROXY_AGENT DAT_OS_WAIT_PROXY_AGENT_NULL = {NULL, NULL}; #endif - DAT_RETURN ret; + u32 ret; # define DFLT_QLEN 10 /* a small event queue size */ # define START_COUNT 1024 /* initial # handles */ # define DFLT_BUFFSZ 4096 /* default size for buffer */ diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_performance_client.c linux-kernel2/test/dapltest/test/dapl_performance_client.c --- linux-kernel/test/dapltest/test/dapl_performance_client.c 2005-05-11 09:04:51.559004000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_performance_client.c 2005-05-11 11:16:06.745962000 -0700 @@ -91,7 +91,7 @@ DT_Performance_Test_Client_Connect ( DT_Tdep_Print_Head *phead, Performance_Test_t *test_ptr) { - DAT_RETURN ret; + u32 ret; DAT_EVENT_NUMBER event_num; unsigned int retry_cnt = 0; @@ -256,7 +256,7 @@ DT_Performance_Test_Client_Phase2 ( struct dat_rmr_triplet rmr_triplet; DAT_DTO_COOKIE cookie; struct dat_event event; - DAT_RETURN ret; + u32 ret; Performance_Ep_Context_t *ep_context; Performance_Test_Op_t *op; DT_Mdep_TimeStamp pre_ts; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_performance_server.c linux-kernel2/test/dapltest/test/dapl_performance_server.c --- linux-kernel/test/dapltest/test/dapl_performance_server.c 2005-05-11 09:04:51.566003000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_performance_server.c 2005-05-11 11:16:06.751964000 -0700 @@ -99,7 +99,7 @@ DT_Performance_Test_Server_Connect ( DT_Tdep_Print_Head *phead, Performance_Test_t *test_ptr) { - DAT_RETURN ret; + u32 ret; boolean_t status; DAT_RSP_HANDLE rsp_handle; DAT_PSP_HANDLE psp_handle; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_performance_util.c linux-kernel2/test/dapltest/test/dapl_performance_util.c --- linux-kernel/test/dapltest/test/dapl_performance_util.c 2005-05-11 09:04:51.571006000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_performance_util.c 2005-05-11 11:16:06.756962000 -0700 @@ -41,7 +41,7 @@ DT_Performance_Test_Create ( { Performance_Test_t *test_ptr; DAT_COUNT pipeline_len; - DAT_RETURN ret; + u32 ret; DT_Tdep_Print_Head *phead; phead = pt_ptr->Params.phead; @@ -268,7 +268,7 @@ DT_Performance_Test_Destroy ( Performance_Test_t *test_ptr, boolean_t is_server) { - DAT_RETURN ret; + u32 ret; DAT_EP_HANDLE ep_handle; DT_Tdep_Print_Head *phead; int status = 0; @@ -427,7 +427,7 @@ DT_performance_post_rdma_op ( DT_Mdep_TimeStamp pre_ts; DT_Mdep_TimeStamp post_ts; DAT_DTO_COOKIE cookie; - DAT_RETURN ret; + u32 ret; Performance_Test_Op_t *op = &ep_context->op; struct dat_lmr_triplet *iov = DT_Bpool_GetIOV(op->bp, 0); struct dat_rmr_triplet rmr_triplet; @@ -525,7 +525,7 @@ DT_performance_wait ( { DAT_COUNT i; DAT_COUNT queue_size; - DAT_RETURN ret; + u32 ret; struct dat_event event; unsigned long pre_ctxt_num; unsigned long post_ctxt_num; @@ -614,7 +614,7 @@ DT_performance_poll ( DAT_EVD_HANDLE evd_handle, Performance_Stats_t *stats) { - DAT_RETURN ret; + u32 ret; struct dat_event event; unsigned long pre_ctxt_num; unsigned long post_ctxt_num; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_server.c linux-kernel2/test/dapltest/test/dapl_server.c --- linux-kernel/test/dapltest/test/dapl_server.c 2005-05-11 08:59:09.094002000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_server.c 2005-05-11 11:16:06.763960000 -0700 @@ -52,7 +52,7 @@ DT_cs_Server (Params_t * params_ptr) DAT_DTO_COOKIE dto_cookie; struct dat_dto_completion_event_data dto_stat; - DAT_RETURN ret; + u32 ret; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_test_util.c linux-kernel2/test/dapltest/test/dapl_test_util.c --- linux-kernel/test/dapltest/test/dapl_test_util.c 2005-05-11 09:04:51.577002000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_test_util.c 2005-05-11 11:16:06.768963000 -0700 @@ -38,7 +38,7 @@ DT_query ( Per_Test_Data_t *pt_ptr, unsigned char *module = "DT_query"; DAT_EVD_HANDLE async_evd_hdl; /* not used */ struct dat_ep_param ep_params; - DAT_RETURN ret; + u32 ret; DT_Tdep_Print_Head *phead; phead = pt_ptr->Params.phead; @@ -169,7 +169,7 @@ DT_post_recv_buffer (DT_Tdep_Print_Head struct dat_lmr_triplet *iov = DT_Bpool_GetIOV(bp, index); DAT_LMR_CONTEXT lmr_c = DT_Bpool_GetLMR (bp, index); DAT_DTO_COOKIE cookie; - DAT_RETURN ret; + u32 ret; /* * Prep the inputs @@ -214,7 +214,7 @@ DT_post_send_buffer (DT_Tdep_Print_Head struct dat_lmr_triplet *iov = DT_Bpool_GetIOV(bp, index); DAT_LMR_CONTEXT lmr_c = DT_Bpool_GetLMR (bp, index); DAT_DTO_COOKIE cookie; - DAT_RETURN ret; + u32 ret; /* * Prep the inputs @@ -258,7 +258,7 @@ DT_cr_event_wait ( DT_Tdep_Print_Head *p for (;;) { - DAT_RETURN ret; + u32 ret; struct dat_event event; ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); @@ -313,7 +313,7 @@ DT_conn_event_wait (DT_Tdep_Print_Head * { for (;;) { - DAT_RETURN ret; + u32 ret; struct dat_event event; ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); @@ -366,7 +366,7 @@ DT_disco_event_wait ( DT_Tdep_Print_Head { for (;;) { - DAT_RETURN ret; + u32 ret; struct dat_event event; ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); @@ -437,7 +437,7 @@ DT_dto_event_poll (DT_Tdep_Print_Head *p { for (;;) { - DAT_RETURN ret; + u32 ret; struct dat_event event; ret = DT_Tdep_evd_dequeue ( evd_handle, @@ -489,7 +489,7 @@ DT_dto_event_wait (DT_Tdep_Print_Head *p { for (;;) { - DAT_RETURN ret; + u32 ret; struct dat_event event; ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); @@ -533,7 +533,7 @@ DT_rmr_event_wait (DT_Tdep_Print_Head *p { for (;;) { - DAT_RETURN ret; + u32 ret; struct dat_event event; ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); @@ -664,7 +664,7 @@ DT_cr_check ( DT_Tdep_Print_Head *phead DAT_CR_HANDLE *cr_handlep, char *message) { - DAT_RETURN ret; + u32 ret; if (cr_handlep) { diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_transaction_test.c linux-kernel2/test/dapltest/test/dapl_transaction_test.c --- linux-kernel/test/dapltest/test/dapl_transaction_test.c 2005-05-11 09:04:51.593002000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_transaction_test.c 2005-05-11 11:16:06.783962000 -0700 @@ -229,7 +229,7 @@ void DT_Transaction_Main (void *param) { Transaction_Test_t *test_ptr = (Transaction_Test_t *) param; - DAT_RETURN ret; + u32 ret; u32 i, j; boolean_t success = FALSE; Per_Test_Data_t *pt_ptr; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_transaction_util.c linux-kernel2/test/dapltest/test/dapl_transaction_util.c --- linux-kernel/test/dapltest/test/dapl_transaction_util.c 2005-05-11 09:04:51.599004000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_transaction_util.c 2005-05-11 11:16:06.789964000 -0700 @@ -43,7 +43,7 @@ DT_handle_post_recv_buf (DT_Tdep_Print_H Transaction_Test_Op_t *op = &ep_context[i].op[op_indx]; struct dat_lmr_triplet *iov = DT_Bpool_GetIOV(op->bp, 0); DAT_DTO_COOKIE cookie; - DAT_RETURN ret; + u32 ret; /* Prep the inputs */ for (j = 0; j < op->num_segs; j++) @@ -105,7 +105,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p Transaction_Test_Op_t *op = &ep_context[i].op[op_indx]; struct dat_lmr_triplet *iov = DT_Bpool_GetIOV(op->bp, 0); DAT_DTO_COOKIE cookie; - DAT_RETURN ret; + u32 ret; /* Prep the inputs */ for (j = 0; j < op->num_segs; j++) @@ -470,7 +470,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p boolean_t poll) { unsigned int i, j; - DAT_RETURN ret; + u32 ret; unsigned char *completion_reaped; completion_reaped = DT_Mdep_Malloc (num_eps * sizeof (unsigned char)); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_util.c linux-kernel2/test/dapltest/test/dapl_util.c --- linux-kernel/test/dapltest/test/dapl_util.c 2005-05-05 09:42:06.416005000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_util.c 2005-05-11 11:16:06.794962000 -0700 @@ -28,11 +28,11 @@ #include "dapl_proto.h" /* - * Map DAT_RETURN values to readable strings, + * Map u32 values to readable strings, * but don't assume the values are zero-based or contiguous. */ const char * -DT_RetToString (DAT_RETURN ret_value) +DT_RetToString (u32 ret_value) { const char *major_msg, *minor_msg; int sz; @@ -61,7 +61,7 @@ DT_AsyncErr2Str (DAT_EVENT_NUMBER error_ static struct { const char *name; - DAT_RETURN value; + u32 value; } dat_errors[] = { # define DATxx(x) { # x, x } @@ -106,7 +106,7 @@ DT_EventToSTr (DAT_EVENT_NUMBER event_co static struct { const char *name; - DAT_RETURN value; + u32 value; } dat_events[] = { @@ -154,7 +154,7 @@ DT_State2Str (DAT_EP_STATE state_code) static struct { const char *name; - DAT_RETURN value; + u32 value; } dat_state[] = { From mshefty at ichips.intel.com Wed May 11 14:37:17 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Wed, 11 May 2005 14:37:17 -0700 Subject: [openib-general] [RFC] [PATCH] user_mad: Support RMPP on send side In-Reply-To: <1115845653.4490.188.camel@localhost.localdomain> References: <1115845653.4490.188.camel@localhost.localdomain> Message-ID: <42827B0D.9050403@ichips.intel.com> Hal Rosenstock wrote: > A couple of notes on this patch: > 1. send side copying > There are currently 2 copies from user space done even in the non RMPP > case. It would be nice to reduce that to one in the non RMPP case. > This is an optimization to add at some point. Having a single data copy on both the send side and receive side should be possible without too much work. I know that the receive side copies twice now, but sends are one. Why not keep it this way, versus having to come back, modify the code again, and retest? - Sean From sean.hefty at intel.com Wed May 11 15:19:37 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Wed, 11 May 2005 15:19:37 -0700 Subject: [openib-general] [PATCH] [CM] use MAD retry functionality In-Reply-To: Message-ID: >The following patch expands the automatic retry ability in the MAD >layer to be usable by all clients, not just RMPP. Retries are done >using a linear timeout. This patch converts the CM to make use of the MAD retry functionality. Neither of these patches has been committed yet. Signed-off-by: Sean Hefty Index: core/cm.c =================================================================== --- core/cm.c (revision 2306) +++ core/cm.c (working copy) @@ -91,7 +91,6 @@ struct cm_msg { struct ib_send_wr send_wr; struct ib_sge sge; enum ib_cm_state sent_state; - u8 retry; DECLARE_PCI_UNMAP_ADDR(mapping) struct ib_mad mad; }; @@ -202,6 +201,7 @@ static int cm_alloc_msg(struct cm_id_pri m->send_wr.wr.ud.remote_qpn = 1; m->send_wr.wr.ud.remote_qkey = IB_QP1_QKEY; /* Timeout set by caller if response is expected. */ + m->send_wr.wr.ud.retries = cm_id_priv->max_cm_retries; m->send_wr.wr.ud.pkey_index = cm_id_priv->av.pkey_index; atomic_inc(&cm_id_priv->refcount); @@ -2564,33 +2564,6 @@ discard: cm_free_msg(msg); } -static void cm_process_send_timeout(struct cm_msg *msg) -{ - struct cm_id_private *cm_id_priv; - struct ib_send_wr *bad_send_wr; - unsigned long flags; - int ret; - - cm_id_priv = msg->cm_id_priv; - - /* Discard old sends. */ - spin_lock_irqsave(&cm_id_priv->lock, flags); - if (msg != cm_id_priv->msg || msg->sent_state != cm_id_priv->id.state) { - spin_unlock_irqrestore(&cm_id_priv->lock, flags); - cm_free_msg(msg); - return; - } - spin_unlock_irqrestore(&cm_id_priv->lock, flags); - - if (msg->retry++ < cm_id_priv->max_cm_retries) { - ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent, - &msg->send_wr, &bad_send_wr); - if (ret) - cm_process_send_error(msg, IB_WC_GENERAL_ERR); - } else - cm_process_send_error(msg, IB_WC_RESP_TIMEOUT_ERR); -} - static void cm_send_handler(struct ib_mad_agent *mad_agent, struct ib_mad_send_wc *mad_send_wc) { @@ -2603,9 +2576,6 @@ static void cm_send_handler(struct ib_ma case IB_WC_WR_FLUSH_ERR: cm_free_msg(msg); break; - case IB_WC_RESP_TIMEOUT_ERR: - cm_process_send_timeout(msg); - break; default: cm_process_send_error(msg, mad_send_wc->status); break; Index: include/ib_cm.h =================================================================== --- include/ib_cm.h (revision 2306) +++ include/ib_cm.h (working copy) @@ -285,8 +285,8 @@ struct ib_cm_id { void *context; u64 service_id; u64 service_mask; - enum ib_cm_state state; - enum ib_cm_lap_state lap_state; + enum ib_cm_state state; /* internal CM/debug use */ + enum ib_cm_lap_state lap_state; /* internal CM/debug use */ u32 local_id; u32 remote_id; }; From halr at voltaire.com Wed May 11 15:25:28 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 11 May 2005 18:25:28 -0400 Subject: [openib-general] [RFC] [PATCH] user_mad: Support RMPP on send side In-Reply-To: <42827B0D.9050403@ichips.intel.com> References: <1115845653.4490.188.camel@localhost.localdomain> <42827B0D.9050403@ichips.intel.com> Message-ID: <1115850328.4490.213.camel@localhost.localdomain> On Wed, 2005-05-11 at 17:37, Sean Hefty wrote: > Hal Rosenstock wrote: > > A couple of notes on this patch: > > 1. send side copying > > There are currently 2 copies from user space done even in the non RMPP > > case. It would be nice to reduce that to one in the non RMPP case. > > This is an optimization to add at some point. > > Having a single data copy on both the send side and receive side should be > possible without too much work. I know that the receive side copies twice > now, but sends are one. Why not keep it this way, versus having to come > back, modify the code again, and retest? The main issue I see on the send side is that without examining the header, one can't get all the parameters for ib_create_send_mad. Even with some variant of this call (perhaps broken up into 2 parts), I think the mad_agent is the key parameter needed. I don't see a way to get this ahead of time without 2 copies. Am I then back to not using this routine and doing it "by hand" ? -- Hal From mshefty at ichips.intel.com Wed May 11 15:38:25 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Wed, 11 May 2005 15:38:25 -0700 Subject: [openib-general] [RFC] [PATCH] user_mad: Support RMPP on send side In-Reply-To: <1115850328.4490.213.camel@localhost.localdomain> References: <1115845653.4490.188.camel@localhost.localdomain> <42827B0D.9050403@ichips.intel.com> <1115850328.4490.213.camel@localhost.localdomain> Message-ID: <42828961.8090701@ichips.intel.com> Hal Rosenstock wrote: > The main issue I see on the send side is that without examining the > header, one can't get all the parameters for ib_create_send_mad. Even > with some variant of this call (perhaps broken up into 2 parts), I think > the mad_agent is the key parameter needed. I don't see a way to get this > ahead of time without 2 copies. Am I then back to not using this routine > and doing it "by hand" ? Why not copy down the ib_user_mad, or any necessary header information, separately from the actual MAD data itself? I.e. read in the mad_agent and MAD length first, allocate the send MAD, and then copy the data. (I'll admit that I'm not overly familiar with this code, so if this is overly difficult just say so...) I'm concerned about having excessive copies on the opensm node, especially if it's sending a large number of RMPP packets. - Sean From andrea at suse.de Wed May 11 15:49:47 2005 From: andrea at suse.de (Andrea Arcangeli) Date: Thu, 12 May 2005 00:49:47 +0200 Subject: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation In-Reply-To: <78d18e2050511131246075b37@mail.gmail.com> References: <20050411171347.7e05859f.akpm@osdl.org> <20050412180447.E6958@topspin.com> <20050425203110.A9729@topspin.com> <4279142A.8050501@ammasso.com> <427A6A7E.8000604@ammasso.com> <427BF8E1.2080006@ammasso.com> <427CD49E.6080300@ammasso.com> <78d18e2050511131246075b37@mail.gmail.com> Message-ID: <20050511224947.GL6313@g5.random> On Wed, May 11, 2005 at 04:12:41PM -0400, William Jordan wrote: > If I am reading you correctly, you are saying that mlock currently > prevents pages from migrating around to unfragment memory, but > get_user_pages does not prevent this? If this is the case, this could This is not the case. Infact get_user_pages is a stronger pin than mlock. But if you call it by hand and you plan to write to the page, you have to use the "write=1" flag, this is fundamental if you want to write to the physical page from userland while it's being tracked by IB dma. In short you should not use mlock and you should use only get_user_pages(write=1). If the problem appears again even after the last fix for the COW I did last year, than it means we've another yet another bug to fix. Using mlock for this is unnecessary. mlock is a "virtual" pin and it provides weaker guarantees than what you need. You need _physical_ pin and get_user_pages(write=1) is the only one that will give it to you. write=0 is ok too if you're never ever going to write to the page with the cpu from userland. In the old days there was the concept that get_user_pages wasn't a "pte-pin", but that was infact broken in the way COW was working with threads, but this is fixed now that is really a "pte-pin" again (like in 2.2 which never had the corruption cow bug!) even though the pte may temporarily be set to swapcache or null. In current 2.6 you're guaranteed that despite the pte may be temporarly be set to not-present, the next minor fault will bring into memory the very same physical page that was there before. At least unless you map the thing writeprotect (i.e. write=0) and you write to it from userland.. ;). From andrea at suse.de Wed May 11 15:52:43 2005 From: andrea at suse.de (Andrea Arcangeli) Date: Thu, 12 May 2005 00:52:43 +0200 Subject: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation In-Reply-To: References: <20050412180447.E6958@topspin.com> <20050425203110.A9729@topspin.com> <4279142A.8050501@ammasso.com> <427A6A7E.8000604@ammasso.com> <427BF8E1.2080006@ammasso.com> <427CD49E.6080300@ammasso.com> <78d18e2050511131246075b37@mail.gmail.com> Message-ID: <20050511225243.GM6313@g5.random> On Wed, May 11, 2005 at 09:42:24PM +0100, Hugh Dickins wrote: > proposed patches) there is no such migration of pages; that we'd prefer > to implement migration in such a way that mlock does not inhibit it > (though there might prove to be strong arguments defeating that); > and that get_user_pages _must_ prevent migration (and if there > were already such migration, I'd be saying it _does_ prevent it). Indeed, mlock is a virtual pin and as such it won't be guaranteed to always prevent migration. While get_user_pages is a physical pin on the physical page so it has to prevent migration. I think for him the physical pin is better since I guess IB would break (at least unless you've some method to call to stop IB, adjust the IB dma tracking, and restart IB, that hotplug can call). For the short term using only get_user_pages sounds simpler IMHO. From timur.tabi at ammasso.com Wed May 11 15:53:36 2005 From: timur.tabi at ammasso.com (Timur Tabi) Date: Wed, 11 May 2005 17:53:36 -0500 Subject: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation In-Reply-To: <20050511224947.GL6313@g5.random> References: <20050411171347.7e05859f.akpm@osdl.org> <20050412180447.E6958@topspin.com> <20050425203110.A9729@topspin.com> <4279142A.8050501@ammasso.com> <427A6A7E.8000604@ammasso.com> <427BF8E1.2080006@ammasso.com> <427CD49E.6080300@ammasso.com> <78d18e2050511131246075b37@mail.gmail.com> <20050511224947.GL6313@g5.random> Message-ID: <42828CF0.2080400@ammasso.com> Andrea Arcangeli wrote: > If the problem appears again even after the last fix for the COW I did > last year, than it means we've another yet another bug to fix. All of my memory pinning test cases pass when I use get_user_pages() with kernels 2.6.7 and later. -- Timur Tabi Staff Software Engineer timur.tabi at ammasso.com One thing a Southern boy will never say is, "I don't think duct tape will fix it." -- Ed Smylie, NASA engineer for Apollo 13 From andrea at suse.de Wed May 11 16:05:23 2005 From: andrea at suse.de (Andrea Arcangeli) Date: Thu, 12 May 2005 01:05:23 +0200 Subject: [openib-general] Re: [PATCH][RFC][0/4] InfiniBand userspace verbs implementation In-Reply-To: <42828CF0.2080400@ammasso.com> References: <20050425203110.A9729@topspin.com> <4279142A.8050501@ammasso.com> <427A6A7E.8000604@ammasso.com> <427BF8E1.2080006@ammasso.com> <427CD49E.6080300@ammasso.com> <78d18e2050511131246075b37@mail.gmail.com> <20050511224947.GL6313@g5.random> <42828CF0.2080400@ammasso.com> Message-ID: <20050511230523.GO6313@g5.random> On Wed, May 11, 2005 at 05:53:36PM -0500, Timur Tabi wrote: > Andrea Arcangeli wrote: > > >If the problem appears again even after the last fix for the COW I did > >last year, than it means we've another yet another bug to fix. > > All of my memory pinning test cases pass when I use get_user_pages() with > kernels 2.6.7 and later. Well then your problem was the cow bug, that was corrupting userland with O_DIRECT too... From roland at topspin.com Wed May 11 16:04:38 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 11 May 2005 16:04:38 -0700 Subject: [openib-general] [RFC] [PATCH] user_mad: Support RMPP on send side In-Reply-To: <42828961.8090701@ichips.intel.com> (Sean Hefty's message of "Wed, 11 May 2005 15:38:25 -0700") References: <1115845653.4490.188.camel@localhost.localdomain> <42827B0D.9050403@ichips.intel.com> <1115850328.4490.213.camel@localhost.localdomain> <42828961.8090701@ichips.intel.com> Message-ID: <52zmv1uzy1.fsf@topspin.com> Sean> Why not copy down the ib_user_mad, or any necessary header Sean> information, separately from the actual MAD data itself? Sean> I.e. read in the mad_agent and MAD length first, allocate Sean> the send MAD, and then copy the data. (I'll admit that I'm Sean> not overly familiar with this code, so if this is overly Sean> difficult just say so...) Sean> I'm concerned about having excessive copies on the opensm Sean> node, especially if it's sending a large number of RMPP Sean> packets. I agree that it's better to avoid copies if it doesn't cost much in complexity, but we should also keep in mind that a modern CPU has a rather huge amount of memory bandwidth (gigabytes per second). It's hard to imagine that any remotely realistic RMPP workload could consume even 1/100 of that. In this case it does seem reasonable to copy the header part before proceeding with the rest of the data. - R. From halr at voltaire.com Wed May 11 16:07:17 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 11 May 2005 19:07:17 -0400 Subject: [openib-general] [RFC] [PATCH] user_mad: Support RMPP on send side In-Reply-To: <42828961.8090701@ichips.intel.com> References: <1115845653.4490.188.camel@localhost.localdomain> <42827B0D.9050403@ichips.intel.com> <1115850328.4490.213.camel@localhost.localdomain> <42828961.8090701@ichips.intel.com> Message-ID: <1115852837.4490.222.camel@localhost.localdomain> On Wed, 2005-05-11 at 18:38, Sean Hefty wrote: > Hal Rosenstock wrote: > > The main issue I see on the send side is that without examining the > > header, one can't get all the parameters for ib_create_send_mad. Even > > with some variant of this call (perhaps broken up into 2 parts), I think > > the mad_agent is the key parameter needed. I don't see a way to get this > > ahead of time without 2 copies. Am I then back to not using this routine > > and doing it "by hand" ? > > Why not copy down the ib_user_mad, or any necessary header information, > separately from the actual MAD data itself? I.e. read in the mad_agent and > MAD length first, allocate the send MAD, and then copy the data. (I'll > admit that I'm not overly familiar with this code, so if this is overly > difficult just say so...) That's exactly what is done. That's what I was calling 2 copies as copy_from_user is called twice. My wording was poor. The details are as follows: Initially the user_mad header, MAD header, and RMPP header are copied from user space. Later on, if it is a normal MAD, the MAD is copied from user space. If it is an RMPP MAD, the MAD header is mem copied from where it was put and the rest of the MAD is copied from user space. I think some improvements to this code may be possible (perhaps more with normal MADs). > I'm concerned about having excessive copies on the opensm node, especially > if it's sending a large number of RMPP packets. -- Hal From roland at topspin.com Wed May 11 16:15:31 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 11 May 2005 16:15:31 -0700 Subject: [openib-general] [RFC] [PATCH] user_mad: Support RMPP on send side In-Reply-To: <1115852837.4490.222.camel@localhost.localdomain> (Hal Rosenstock's message of "11 May 2005 19:07:17 -0400") References: <1115845653.4490.188.camel@localhost.localdomain> <42827B0D.9050403@ichips.intel.com> <1115850328.4490.213.camel@localhost.localdomain> <42828961.8090701@ichips.intel.com> <1115852837.4490.222.camel@localhost.localdomain> Message-ID: <52r7gduzfw.fsf@topspin.com> Hal> Initially the user_mad header, MAD header, and RMPP header Hal> are copied from user space. Later on, if it is a normal MAD, Hal> the MAD is copied from user space. If it is an RMPP MAD, the Hal> MAD header is mem copied from where it was put and the rest Hal> of the MAD is copied from user space. This seems fine to me. I doubt you could even measure the difference between this and only making one call to copy_from_user(). BTW I still haven't had a chance to really read the patch but I will comment soon. - R. From tduffy at sun.com Wed May 11 16:47:10 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 11 May 2005 16:47:10 -0700 Subject: [openib-general] [PATCH] kDAPL: remove typedef DAT_PVOID In-Reply-To: <1115846388.11344.25.camel@duffman> References: <1115846388.11344.25.camel@duffman> Message-ID: <1115855230.11339.5.camel@duffman> Remove DAT_PVOID. Use void * instead. Singed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/dat/dat.h linux-kernel3/dat/dat.h --- linux-kernel2/dat/dat.h 2005-05-11 13:07:34.341014000 -0700 +++ linux-kernel3/dat/dat.h 2005-05-11 16:40:16.555002000 -0700 @@ -40,7 +40,6 @@ /* unsigned longest native to compiler */ typedef unsigned long long DAT_UVERYLONG; -typedef void * DAT_PVOID; typedef int DAT_COUNT; typedef u64 DAT_PADDR; @@ -370,7 +369,7 @@ typedef enum { } boolean_t; typedef union dat_context { - DAT_PVOID as_ptr; + void *as_ptr; u64 as_64; DAT_UVERYLONG as_index; } DAT_CONTEXT; @@ -402,7 +401,7 @@ typedef u32 DAT_TIMEOUT; /* microseconds #define DAT_TIMEOUT_INFINITE ((DAT_TIMEOUT) ~0) /* dat handles */ -typedef DAT_PVOID DAT_HANDLE; +typedef void *DAT_HANDLE; typedef DAT_HANDLE DAT_CR_HANDLE; typedef DAT_HANDLE DAT_EP_HANDLE; typedef DAT_HANDLE DAT_EVD_HANDLE; @@ -469,7 +468,7 @@ typedef enum dat_psp_flags { * Memory Buffers * * Both LMR and RMR triplets specify 64-bit addresses in the local host's byte - * order, even when that exceeds the size of a DAT_PVOID for the host + * order, even when that exceeds the size of a void *for the host * architecture. */ @@ -729,7 +728,8 @@ struct dat_cr_param { /* Pointer to the Private Data passed by remote side in the Connection * Request. */ - DAT_PVOID private_data; + void *private_data; + /* The local Endpoint provided by the Service Point for the requested * connection. It is the only Endpoint that can accept a Connection * Request on this Service Point. The value DAT_HANDLE_NULL @@ -821,7 +821,7 @@ struct dat_cr_arrival_event_data { struct dat_connection_event_data { DAT_EP_HANDLE ep_handle; DAT_COUNT private_data_size; - DAT_PVOID private_data; + void *private_data; }; /* Async Error event data */ @@ -876,7 +876,7 @@ typedef enum pz_async_error_reason { /* Software event data */ struct dat_software_event_data { - DAT_PVOID pointer; + void *pointer; }; typedef enum dat_event_number { @@ -929,18 +929,17 @@ typedef enum dat_upcall_policy { /* UpCalls return */ } DAT_UPCALL_POLICY; -typedef void (*DAT_UPCALL_FUNC) (DAT_PVOID, const struct dat_event *, - boolean_t); +typedef void (*DAT_UPCALL_FUNC) (void *, const struct dat_event *, boolean_t); struct dat_upcall_object { - DAT_PVOID instance_data; + void *instance_data; DAT_UPCALL_FUNC upcall_func; }; /* Define NULL upcall */ #define DAT_UPCALL_NULL \ - ((struct dat_upcall_object) { (DAT_PVOID) NULL, (DAT_UPCALL_FUNC) NULL }) + ((struct dat_upcall_object) { (void *) NULL, (DAT_UPCALL_FUNC) NULL }) #define DAT_UPCALL_SAME ((struct dat_upcall_object *) NULL) @@ -981,7 +980,7 @@ typedef enum dat_mem_optimize_flags { } DAT_MEM_OPTIMIZE_FLAGS; typedef union dat_region_description { - DAT_PVOID for_va; + void *for_va; DAT_LMR_HANDLE for_lmr_handle; void *for_pointer; /* For kdapl only */ void *for_array; /* For kdapl only */ @@ -1126,8 +1125,8 @@ typedef u32 (*DAT_GET_HANDLE_TYPE_FUNC) typedef u32 (*DAT_CR_QUERY_FUNC) (DAT_CR_HANDLE, struct dat_cr_param *); -typedef u32 (*DAT_CR_ACCEPT_FUNC) ( - DAT_CR_HANDLE, DAT_EP_HANDLE, DAT_COUNT, const DAT_PVOID); +typedef u32 (*DAT_CR_ACCEPT_FUNC) (DAT_CR_HANDLE, DAT_EP_HANDLE, DAT_COUNT, + const void*); typedef u32 (*DAT_CR_REJECT_FUNC) (DAT_CR_HANDLE); @@ -1158,13 +1157,13 @@ typedef u32 (*DAT_EP_QUERY_FUNC) (DAT_EP typedef u32 (*DAT_EP_MODIFY_FUNC) ( DAT_EP_HANDLE, DAT_EP_PARAM_MASK, const struct dat_ep_param *); -typedef u32 (*DAT_EP_CONNECT_FUNC) ( - DAT_EP_HANDLE, DAT_IA_ADDRESS_PTR, DAT_CONN_QUAL, DAT_TIMEOUT, DAT_COUNT, - const DAT_PVOID, DAT_QOS, DAT_CONNECT_FLAGS); - -typedef u32 (*DAT_EP_DUP_CONNECT_FUNC) ( - DAT_EP_HANDLE, DAT_EP_HANDLE, DAT_TIMEOUT, DAT_COUNT, const DAT_PVOID, - DAT_QOS); +typedef u32 (*DAT_EP_CONNECT_FUNC) (DAT_EP_HANDLE, DAT_IA_ADDRESS_PTR, + DAT_CONN_QUAL, DAT_TIMEOUT, DAT_COUNT, + const void *, DAT_QOS, DAT_CONNECT_FLAGS); + +typedef u32 (*DAT_EP_DUP_CONNECT_FUNC) (DAT_EP_HANDLE, DAT_EP_HANDLE, + DAT_TIMEOUT, DAT_COUNT, const void *, + DAT_QOS); typedef u32 (*DAT_EP_DISCONNECT_FUNC) (DAT_EP_HANDLE, DAT_CLOSE_FLAGS); @@ -1281,7 +1280,7 @@ typedef u32 (*DAT_EVD_MODIFY_UPCALL_FUNC struct dat_provider { const char *device_name; - DAT_PVOID extension; + void *extension; DAT_IA_OPEN_FUNC ia_open_func; DAT_IA_QUERY_FUNC ia_query_func; @@ -1447,7 +1446,7 @@ static inline u32 dat_ia_query(DAT_IA_HA static inline u32 dat_cr_accept(DAT_CR_HANDLE cr, DAT_EP_HANDLE ep, DAT_COUNT private_data_size, - const DAT_PVOID private_data) + const void *private_data) { return DAT_CALL_PROVIDER_FUNC( cr_accept_func, cr, ep, private_data_size, private_data); @@ -1522,7 +1521,7 @@ static inline u32 dat_ep_connect(DAT_EP_ DAT_CONN_QUAL conn_qual, DAT_TIMEOUT timeout, DAT_COUNT private_data_size, - const DAT_PVOID private_data, + const void *private_data, DAT_QOS qos, DAT_CONNECT_FLAGS flags) { @@ -1567,7 +1566,7 @@ static inline u32 dat_ep_dup_connect(DAT DAT_EP_HANDLE dup_ep, DAT_TIMEOUT timeout, DAT_COUNT private_data_size, - const DAT_PVOID private_data, + const void *private_data, DAT_QOS qos) { return DAT_CALL_PROVIDER_FUNC(ep_dup_connect_func, ep, dup_ep, timeout, diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/dat-provider/dapl_adapter_util.h linux-kernel3/dat-provider/dapl_adapter_util.h --- linux-kernel2/dat-provider/dapl_adapter_util.h 2005-05-11 11:37:05.389975000 -0700 +++ linux-kernel3/dat-provider/dapl_adapter_util.h 2005-05-11 16:11:16.507003000 -0700 @@ -69,7 +69,7 @@ u32 dapl_ib_qp_modify(DAPL_IA *ia_ptr, D u32 dapl_ib_connect(DAT_EP_HANDLE ep_handle, DAT_IA_ADDRESS_PTR remote_ia_address, DAT_CONN_QUAL remote_conn_qual, - DAT_COUNT private_data_size, DAT_PVOID private_data); + DAT_COUNT private_data_size, void *private_data); u32 dapl_ib_disconnect(DAPL_EP *ep_ptr, DAT_CLOSE_FLAGS close_flags); @@ -81,7 +81,7 @@ u32 dapl_ib_remove_conn_listener(DAPL_IA u32 dapl_ib_accept_connection(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, DAT_COUNT private_data_size, - const DAT_PVOID private_data); + const void *private_data); u32 dapl_ib_reject_connection(ib_cm_handle_t cm_handle, int reject_reason); @@ -102,11 +102,11 @@ u32 dapl_ib_pd_alloc(DAPL_IA *ia_ptr, DA u32 dapl_ib_pd_free(DAPL_PZ *pz); -u32 dapl_ib_mr_register(DAPL_IA *ia_ptr, DAPL_LMR *lmr, DAT_PVOID virt_addr, +u32 dapl_ib_mr_register(DAPL_IA *ia_ptr, DAPL_LMR *lmr, void *virt_addr, DAT_VLEN length, DAT_MEM_PRIV_FLAGS privileges); u32 dapl_ib_mr_register_physical(DAPL_IA *ia_ptr, DAPL_LMR *lmr, - DAT_PVOID phys_addr, DAT_VLEN length, + void *phys_addr, DAT_VLEN length, DAT_MEM_PRIV_FLAGS privileges); u32 dapl_ib_mr_deregister(DAPL_LMR *lmr); diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/dat-provider/dapl_cr_accept.c linux-kernel3/dat-provider/dapl_cr_accept.c --- linux-kernel2/dat-provider/dapl_cr_accept.c 2005-05-11 11:42:47.385963000 -0700 +++ linux-kernel3/dat-provider/dapl_cr_accept.c 2005-05-11 16:11:16.512006000 -0700 @@ -40,7 +40,7 @@ * and passive side (local endpoint). */ u32 dapl_cr_accept(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, - DAT_COUNT private_data_size, const DAT_PVOID private_data) + DAT_COUNT private_data_size, const void *private_data) { DAPL_EP *ep_ptr; u32 dat_status; diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/dat-provider/dapl_ep_connect.c linux-kernel3/dat-provider/dapl_ep_connect.c --- linux-kernel2/dat-provider/dapl_ep_connect.c 2005-05-11 11:58:33.206963000 -0700 +++ linux-kernel3/dat-provider/dapl_ep_connect.c 2005-05-11 16:11:16.516003000 -0700 @@ -45,7 +45,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_handle, DAT_IA_ADDRESS_PTR remote_ia_address, DAT_CONN_QUAL remote_conn_qual, DAT_TIMEOUT timeout, - DAT_COUNT private_data_size, const DAT_PVOID private_data, + DAT_COUNT private_data_size, const void *private_data, DAT_QOS qos, DAT_CONNECT_FLAGS connect_flags) { DAPL_EP *ep_ptr; diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/dat-provider/dapl_ep_dup_connect.c linux-kernel3/dat-provider/dapl_ep_dup_connect.c --- linux-kernel2/dat-provider/dapl_ep_dup_connect.c 2005-05-11 12:00:47.734963000 -0700 +++ linux-kernel3/dat-provider/dapl_ep_dup_connect.c 2005-05-11 16:11:16.519006000 -0700 @@ -71,7 +71,7 @@ */ u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, DAT_EP_HANDLE ep_dup_handle, DAT_TIMEOUT timeout, DAT_COUNT private_data_size, - const DAT_PVOID private_data, DAT_QOS qos) + const void *private_data, DAT_QOS qos) { DAPL_EP *ep_dup_ptr; u32 dat_status; diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/dat-provider/dapl_evd_util.c linux-kernel3/dat-provider/dapl_evd_util.c --- linux-kernel2/dat-provider/dapl_evd_util.c 2005-05-11 12:18:40.507003000 -0700 +++ linux-kernel3/dat-provider/dapl_evd_util.c 2005-05-11 16:11:16.526002000 -0700 @@ -540,7 +540,7 @@ u32 dapl_evd_post_connection_event(DAPL_ DAT_EVENT_NUMBER event_number, DAT_EP_HANDLE ep_handle, DAT_COUNT private_data_size, - DAT_PVOID private_data) + void *private_data) { struct dat_event *event_ptr; event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number); @@ -592,7 +592,7 @@ u32 dapl_evd_post_async_error_event(DAPL u32 dapl_evd_post_software_event(DAPL_EVD *evd_ptr, DAT_EVENT_NUMBER event_number, - DAT_PVOID pointer) + void *pointer) { struct dat_event *event_ptr; event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number); diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/dat-provider/dapl_evd_util.h linux-kernel3/dat-provider/dapl_evd_util.h --- linux-kernel2/dat-provider/dapl_evd_util.h 2005-05-11 12:21:06.564007000 -0700 +++ linux-kernel3/dat-provider/dapl_evd_util.h 2005-05-11 16:11:16.530004000 -0700 @@ -73,7 +73,7 @@ u32 dapl_evd_post_connection_event(DAPL_ DAT_EVENT_NUMBER event_number, DAT_EP_HANDLE ep_handle, DAT_COUNT private_data_size, - DAT_PVOID private_data); + void *private_data); u32 dapl_evd_post_async_error_event(DAPL_EVD *evd_ptr, DAT_EVENT_NUMBER event_number, @@ -81,7 +81,7 @@ u32 dapl_evd_post_async_error_event(DAPL u32 dapl_evd_post_software_event(DAPL_EVD *evd_ptr, DAT_EVENT_NUMBER event_number, - DAT_PVOID pointer); + void *pointer); /************************************* * dapl internal callbacks functions * diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/dat-provider/dapl.h linux-kernel3/dat-provider/dapl.h --- linux-kernel2/dat-provider/dapl.h 2005-05-11 13:23:31.584000000 -0700 +++ linux-kernel3/dat-provider/dapl.h 2005-05-11 16:11:16.504000000 -0700 @@ -497,7 +497,7 @@ extern u32 dapl_cr_query(DAT_CR_HANDLE, extern u32 dapl_cr_accept(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, DAT_COUNT private_data_size, - const DAT_PVOID private_data); + const void *private_data); extern u32 dapl_cr_reject(DAT_CR_HANDLE cr_handle); @@ -559,14 +559,14 @@ extern u32 dapl_ep_connect(DAT_EP_HANDLE DAT_CONN_QUAL remote_conn_qual, DAT_TIMEOUT timeout, DAT_COUNT private_data_size, - const DAT_PVOID private_data, + const void *private_data, DAT_QOS quality_of_service, DAT_CONNECT_FLAGS connect_flags); extern u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, DAT_EP_HANDLE ep_dup_handle, DAT_TIMEOUT timeout, DAT_COUNT private_data_size, - const DAT_PVOID private_data, + const void *private_data, DAT_QOS quality_of_service); extern u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_handle, diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/dat-provider/dapl_lmr_kcreate.c linux-kernel3/dat-provider/dapl_lmr_kcreate.c --- linux-kernel2/dat-provider/dapl_lmr_kcreate.c 2005-05-11 13:37:00.876008000 -0700 +++ linux-kernel3/dat-provider/dapl_lmr_kcreate.c 2005-05-11 16:11:16.537002000 -0700 @@ -37,7 +37,7 @@ #include "dapl_lmr_util.h" #include "dapl_adapter_util.h" -static inline u32 dapl_lmr_create_virtual(DAPL_IA *ia, DAT_PVOID virt_addr, +static inline u32 dapl_lmr_create_virtual(DAPL_IA *ia, void *virt_addr, DAT_VLEN length, DAPL_PZ *pz, DAT_MEM_PRIV_FLAGS privileges, DAT_LMR_HANDLE *lmr_handle, diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/dat-provider/dapl_openib_cm.c linux-kernel3/dat-provider/dapl_openib_cm.c --- linux-kernel2/dat-provider/dapl_openib_cm.c 2005-05-11 13:40:22.442023000 -0700 +++ linux-kernel3/dat-provider/dapl_openib_cm.c 2005-05-11 16:11:16.545001000 -0700 @@ -400,7 +400,7 @@ error: u32 dapl_ib_connect(DAT_EP_HANDLE ep_handle, DAT_IA_ADDRESS_PTR remote_ia_address, DAT_CONN_QUAL remote_conn_qual, - DAT_COUNT private_data_size, DAT_PVOID private_data) + DAT_COUNT private_data_size, void *private_data) { DAPL_IA *ia_ptr; DAPL_EP *ep_ptr; @@ -737,7 +737,7 @@ u32 dapl_ib_reject_connection(struct dap */ u32 dapl_ib_accept_connection(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, DAT_COUNT private_data_size, - const DAT_PVOID prd_ptr) + const void *prd_ptr) { DAPL_CR *cr_ptr; DAPL_EP *ep_ptr; diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/dat-provider/dapl_openib_util.c linux-kernel3/dat-provider/dapl_openib_util.c --- linux-kernel2/dat-provider/dapl_openib_util.c 2005-05-11 13:48:41.352007000 -0700 +++ linux-kernel3/dat-provider/dapl_openib_util.c 2005-05-11 16:11:16.552002000 -0700 @@ -390,7 +390,7 @@ u32 dapl_ib_pd_free(DAPL_PZ *pz) * DAT_INSUFFICIENT_RESOURCES * */ -u32 dapl_ib_mr_register(DAPL_IA *ia, DAPL_LMR *lmr, DAT_PVOID virt_addr, +u32 dapl_ib_mr_register(DAPL_IA *ia, DAPL_LMR *lmr, void *virt_addr, DAT_VLEN length, DAT_MEM_PRIV_FLAGS privileges) { /* ITBD virtual memory registration ??? */ @@ -398,7 +398,7 @@ u32 dapl_ib_mr_register(DAPL_IA *ia, DAP } u32 dapl_ib_mr_register_physical(DAPL_IA *ia_ptr, DAPL_LMR *lmr, - DAT_PVOID phys_addr, DAT_VLEN length, + void *phys_addr, DAT_VLEN length, DAT_MEM_PRIV_FLAGS privileges) { int ib_status; diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/test/dapltest/include/dapl_proto.h linux-kernel3/test/dapltest/include/dapl_proto.h --- linux-kernel2/test/dapltest/include/dapl_proto.h 2005-05-11 11:16:06.658962000 -0700 +++ linux-kernel3/test/dapltest/include/dapl_proto.h 2005-05-11 16:11:16.569002000 -0700 @@ -405,7 +405,7 @@ boolean_t DT_dto_check ( DT_T boolean_t DT_rmr_check ( DT_Tdep_Print_Head* phead, struct dat_rmr_bind_completion_event_data *rmr_p, DAT_RMR_HANDLE rmr_expected, - DAT_PVOID cookie_expected, + void * cookie_expected, char *message); boolean_t DT_cr_check (DT_Tdep_Print_Head* phead, diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/test/dapltest/kdapl/kdapl_tdep_evd.c linux-kernel3/test/dapltest/kdapl/kdapl_tdep_evd.c --- linux-kernel2/test/dapltest/kdapl/kdapl_tdep_evd.c 2005-05-11 11:16:06.670966000 -0700 +++ linux-kernel3/test/dapltest/kdapl/kdapl_tdep_evd.c 2005-05-11 16:23:15.972014000 -0700 @@ -45,7 +45,7 @@ typedef struct Tdep_Evd_Tag /* static function prototypes */ static void DT_Tdep_Event_Callback ( - DAT_PVOID instance_data, + void *instance_data, const struct dat_event *dat_event, boolean_t bool_arg); @@ -322,7 +322,7 @@ DT_Tdep_evd_free (DAT_EVD_HANDLE evd_han } static void DT_Tdep_Event_Callback ( - DAT_PVOID instance_data, + void *instance_data, const struct dat_event * dat_event, boolean_t bool_arg) { diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/test/dapltest/test/dapl_bpool.c linux-kernel3/test/dapltest/test/dapl_bpool.c --- linux-kernel2/test/dapltest/test/dapl_bpool.c 2005-05-11 11:16:06.675963000 -0700 +++ linux-kernel3/test/dapltest/test/dapl_bpool.c 2005-05-11 16:15:37.116001000 -0700 @@ -227,7 +227,7 @@ DT_BpoolAlloc ( iov.lmr_context = bpool_ptr->lmr_context; cookie.as_64 = (u64)0UL; - cookie.as_ptr = (DAT_PVOID) (uintptr_t) bpool_ptr->reg_addr; + cookie.as_ptr = (void *) (uintptr_t) bpool_ptr->reg_addr; mflags = (enable_rdma_write && enable_rdma_read ? DAT_MEM_PRIV_ALL_FLAG : (enable_rdma_write ? DAT_MEM_PRIV_WRITE_FLAG : (enable_rdma_read ? DAT_MEM_PRIV_READ_FLAG : 0))); @@ -258,7 +258,7 @@ DT_BpoolAlloc ( !DT_rmr_check (phead, &rmr_stat, bpool_ptr->rmr_handle, - (DAT_PVOID) (uintptr_t) bpool_ptr->reg_addr, + (void *) (uintptr_t) bpool_ptr->reg_addr, "Bpool")) { goto err; @@ -334,7 +334,7 @@ DT_Bpool_Destroy (Per_Test_Data_t * pt_p iov.segment_length = 0; /* un-bind */ iov.lmr_context = bpool_ptr->lmr_context; cookie.as_64 = (u64)0UL; - cookie.as_ptr = (DAT_PVOID) (uintptr_t)bpool_ptr->reg_addr; + cookie.as_ptr = (void *) (uintptr_t)bpool_ptr->reg_addr; /* * Do not attempt to unbind here. The remote node diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/test/dapltest/test/dapl_client.c linux-kernel3/test/dapltest/test/dapl_client.c --- linux-kernel2/test/dapltest/test/dapl_client.c 2005-05-11 11:16:06.680966000 -0700 +++ linux-kernel3/test/dapltest/test/dapl_client.c 2005-05-11 16:15:37.124000000 -0700 @@ -226,13 +226,13 @@ DT_cs_Client (Params_t * params_ptr, DT_Tdep_PT_Debug (3, (phead, "RecvSrvInfo 0 %p\n", - (DAT_PVOID) DT_Bpool_GetBuffer (bpool, 0))); + (void *) DT_Bpool_GetBuffer (bpool, 0))); DT_Tdep_PT_Debug (3, (phead, "SndCliInfo 1 %p\n", - (DAT_PVOID) DT_Bpool_GetBuffer (bpool, 1))); + (void *) DT_Bpool_GetBuffer (bpool, 1))); DT_Tdep_PT_Debug (3, (phead, "SndCommand 2 %p\n", - (DAT_PVOID) DT_Bpool_GetBuffer (bpool, 2))); + (void *) DT_Bpool_GetBuffer (bpool, 2))); /* Post recv buffer for Server_Info (1st buffer in pool) */ DT_Tdep_PT_Debug (1,(phead, "%s: Posting 1 recv buffer\n", module)); @@ -257,7 +257,7 @@ retry: server_netaddr, SERVER_PORT_NUMBER, DAT_TIMEOUT_INFINITE, - 0, (DAT_PVOID) 0, /* no private data */ + 0, (void *) 0, /* no private data */ params_ptr->ReliabilityLevel, DAT_CONNECT_DEFAULT_FLAG); if (ret != DAT_SUCCESS) @@ -336,7 +336,7 @@ retry: goto client_exit; } /* reap the send and verify it */ - dto_cookie.as_ptr = (DAT_PVOID) (uintptr_t) DT_Bpool_GetBuffer ( bpool, 1); + dto_cookie.as_ptr = (void *) (uintptr_t) DT_Bpool_GetBuffer ( bpool, 1); DT_Tdep_PT_Debug (1,(phead, "%s: Sent Client_Info - awaiting completion\n", module)); @@ -407,7 +407,7 @@ retry: } /* reap the send and verify it */ dto_cookie.as_64 = LZERO; - dto_cookie.as_ptr = (DAT_PVOID) (uintptr_t) DT_Bpool_GetBuffer ( bpool, 2); + dto_cookie.as_ptr = (void *) (uintptr_t) DT_Bpool_GetBuffer ( bpool, 2); DT_Tdep_PT_Debug (1,(phead, "%s: Sent Command - awaiting completion\n", module)); @@ -425,7 +425,7 @@ retry: /************************************************************************/ dto_cookie.as_64 = LZERO; - dto_cookie.as_ptr = (DAT_PVOID) (uintptr_t) DT_Bpool_GetBuffer ( bpool, 0); + dto_cookie.as_ptr = (void *) (uintptr_t) DT_Bpool_GetBuffer ( bpool, 0); DT_Tdep_PT_Debug (1,(phead, "%s: Waiting for Server_Info\n", module)); if (!DT_dto_event_wait (phead, recv_evd_hdl, &dto_stat) || !DT_dto_check ( phead, diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/test/dapltest/test/dapl_fft_queryinfo.c linux-kernel3/test/dapltest/test/dapl_fft_queryinfo.c --- linux-kernel2/test/dapltest/test/dapl_fft_queryinfo.c 2005-05-11 11:16:06.726963000 -0700 +++ linux-kernel3/test/dapltest/test/dapl_fft_queryinfo.c 2005-05-11 16:12:51.454001000 -0700 @@ -31,13 +31,6 @@ #define BUFFSIZE 1024 #define DEFAULT_QUEUE_LEN 10 -#if defined(WIN32) -static DAT_OS_WAIT_PROXY_AGENT NULLPROXY = -{ - (DAT_PVOID) NULL, (DAT_AGENT_FUNC) NULL -}; -#endif - int DT_queryinfo_basic (Params_t *params_ptr, FFT_Cmd_t *cmd, FFT_query_enum object_to_query, diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/test/dapltest/test/dapl_fft_util.c linux-kernel3/test/dapltest/test/dapl_fft_util.c --- linux-kernel2/test/dapltest/test/dapl_fft_util.c 2005-05-11 11:16:06.731964000 -0700 +++ linux-kernel3/test/dapltest/test/dapl_fft_util.c 2005-05-11 16:15:37.130004000 -0700 @@ -322,7 +322,7 @@ void DT_fft_listen (Params_t *params_ptr "DT_fft_listen")); /* accept the connection */ - rc =dat_cr_accept (conn->cr_handle, conn->ep_handle, 0, (DAT_PVOID)0); + rc =dat_cr_accept (conn->cr_handle, conn->ep_handle, 0, (void *)0); DT_assert_dat (phead, rc == DAT_SUCCESS); /* wait on a conn event via the conn EVD */ @@ -352,7 +352,7 @@ int DT_fft_connect (Params_t *params_ptr /* attempt to connect, timeout = 10 secs */ rc = dat_ep_connect (conn->ep_handle, conn->remote_netaddr, - SERVER_PORT_NUMBER, 10*1000, 0, (DAT_PVOID)0, + SERVER_PORT_NUMBER, 10*1000, 0, (void *)0, DAT_QOS_BEST_EFFORT, DAT_CONNECT_DEFAULT_FLAG); DT_assert_dat (phead, rc == DAT_SUCCESS); diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/test/dapltest/test/dapl_limit.c linux-kernel3/test/dapltest/test/dapl_limit.c --- linux-kernel2/test/dapltest/test/dapl_limit.c 2005-05-11 11:16:06.742963000 -0700 +++ linux-kernel3/test/dapltest/test/dapl_limit.c 2005-05-11 16:15:37.142002000 -0700 @@ -1050,7 +1050,7 @@ limit_test ( DT_Tdep_Print_Head *phead, iovp->segment_length = DFLT_BUFFSZ; iovp->lmr_context = hdl_sets[i].lmr_context; cookie.as_64 = (u64)0UL; - cookie.as_ptr = (DAT_PVOID) hdl_sets[i].lmr_buffer; + cookie.as_ptr = (void *) hdl_sets[i].lmr_buffer; DT_Tdep_PT_Printf (phead, "%s: dat_ep_post_recv #%d\n", module, w * cmd->width + i + 1); diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/test/dapltest/test/dapl_performance_client.c linux-kernel3/test/dapltest/test/dapl_performance_client.c --- linux-kernel2/test/dapltest/test/dapl_performance_client.c 2005-05-11 11:16:06.745962000 -0700 +++ linux-kernel3/test/dapltest/test/dapl_performance_client.c 2005-05-11 16:15:37.148003000 -0700 @@ -107,7 +107,7 @@ retry: test_ptr->ep_context.port, DAT_TIMEOUT_INFINITE, 0, - (DAT_PVOID) 0, /* no private data */ + (void *) 0, /* no private data */ test_ptr->cmd->qos, DAT_CONNECT_DEFAULT_FLAG); if (ret != DAT_SUCCESS) @@ -400,7 +400,7 @@ DT_Performance_Test_Client_Exchange ( dto_cookie.as_64 = LZERO; dto_cookie.as_ptr = - (DAT_PVOID) DT_Bpool_GetBuffer ( + (void *) DT_Bpool_GetBuffer ( test_ptr->ep_context.bp, DT_PERF_SYNC_RECV_BUFFER_ID); if ( !DT_dto_event_wait (phead, test_ptr->recv_evd_hdl, &dto_stat) || @@ -472,7 +472,7 @@ DT_Performance_Test_Client_Exchange ( dto_cookie.as_64 = LZERO; dto_cookie.as_ptr = - (DAT_PVOID) DT_Bpool_GetBuffer ( + (void *) DT_Bpool_GetBuffer ( test_ptr->ep_context.bp, DT_PERF_SYNC_SEND_BUFFER_ID); if (!DT_dto_event_wait (phead, test_ptr->reqt_evd_hdl, &dto_stat) || diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/test/dapltest/test/dapl_performance_server.c linux-kernel3/test/dapltest/test/dapl_performance_server.c --- linux-kernel2/test/dapltest/test/dapl_performance_server.c 2005-05-11 11:16:06.751964000 -0700 +++ linux-kernel3/test/dapltest/test/dapl_performance_server.c 2005-05-11 16:15:37.154000000 -0700 @@ -148,7 +148,7 @@ DT_Performance_Test_Server_Connect ( /* what, me query? just try to accept the connection */ ret = dat_cr_accept (cr_handle, test_ptr->ep_context.ep_handle, - 0, (DAT_PVOID)0 /* no private data */ ); + 0, (void *)0 /* no private data */ ); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_accept error: %s\n", @@ -221,7 +221,7 @@ DT_Performance_Test_Server_Connect ( ret = dat_cr_accept (cr_handle, test_ptr->ep_context.ep_handle, 0, - (DAT_PVOID)0 /* no private data */ ); + (void *)0 /* no private data */ ); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_accept error: %s\n", @@ -356,7 +356,7 @@ DT_Performance_Test_Server_Exchange ( /* reap the send and verify it */ dto_cookie.as_64 = LZERO; dto_cookie.as_ptr = - (DAT_PVOID) DT_Bpool_GetBuffer ( + (void *) DT_Bpool_GetBuffer ( test_ptr->ep_context.bp, DT_PERF_SYNC_SEND_BUFFER_ID); if ( !DT_dto_event_wait (phead, test_ptr->reqt_evd_hdl, &dto_stat) || @@ -378,7 +378,7 @@ DT_Performance_Test_Server_Exchange ( dto_cookie.as_64 = LZERO; dto_cookie.as_ptr = - (DAT_PVOID) DT_Bpool_GetBuffer ( + (void *) DT_Bpool_GetBuffer ( test_ptr->ep_context.bp, DT_PERF_SYNC_RECV_BUFFER_ID); if ( !DT_dto_event_wait (phead, test_ptr->recv_evd_hdl, &dto_stat) || diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/test/dapltest/test/dapl_performance_util.c linux-kernel3/test/dapltest/test/dapl_performance_util.c --- linux-kernel2/test/dapltest/test/dapl_performance_util.c 2005-05-11 11:16:06.756962000 -0700 +++ linux-kernel3/test/dapltest/test/dapl_performance_util.c 2005-05-11 16:15:37.162003000 -0700 @@ -215,11 +215,11 @@ DT_Performance_Test_Create ( DT_Tdep_PT_Debug (3, (phead, "0: SYNC_SEND %p\n", - (DAT_PVOID) DT_Bpool_GetBuffer (test_ptr->ep_context.bp, + (void *) DT_Bpool_GetBuffer (test_ptr->ep_context.bp, DT_PERF_SYNC_SEND_BUFFER_ID))); DT_Tdep_PT_Debug (3, (phead, "1: SYNC_RECV %p\n", - (DAT_PVOID) DT_Bpool_GetBuffer (test_ptr->ep_context.bp, + (void *) DT_Bpool_GetBuffer (test_ptr->ep_context.bp, DT_PERF_SYNC_RECV_BUFFER_ID))); /* diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/test/dapltest/test/dapl_server.c linux-kernel3/test/dapltest/test/dapl_server.c --- linux-kernel2/test/dapltest/test/dapl_server.c 2005-05-11 11:16:06.763960000 -0700 +++ linux-kernel3/test/dapltest/test/dapl_server.c 2005-05-11 16:15:37.167002000 -0700 @@ -253,13 +253,13 @@ DT_cs_Server (Params_t * params_ptr) DT_Tdep_PT_Debug (3, (phead, "Recv 0 %p\n", - (DAT_PVOID) DT_Bpool_GetBuffer (ps_ptr->bpool, 0))); + (void *) DT_Bpool_GetBuffer (ps_ptr->bpool, 0))); DT_Tdep_PT_Debug (3, (phead, "Recv 1 %p\n", - (DAT_PVOID) DT_Bpool_GetBuffer (ps_ptr->bpool, 1))); + (void *) DT_Bpool_GetBuffer (ps_ptr->bpool, 1))); DT_Tdep_PT_Debug (3, (phead, "SrvInfo 2 %p\n", - (DAT_PVOID) DT_Bpool_GetBuffer (ps_ptr->bpool, 2))); + (void *) DT_Bpool_GetBuffer (ps_ptr->bpool, 2))); /* Initialize the performance test lock in case an incoming test * is a performance test, so we can allow only one at a time. @@ -357,7 +357,7 @@ DT_cs_Server (Params_t * params_ptr) } DT_Tdep_PT_Debug (1,(phead,"%s: Accepting Connection Request\n", module)); - ret = dat_cr_accept (cr_handle, ps_ptr->ep_handle, 0, (DAT_PVOID)0); + ret = dat_cr_accept (cr_handle, ps_ptr->ep_handle, 0, (void *)0); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, @@ -390,7 +390,7 @@ DT_cs_Server (Params_t * params_ptr) /* Wait for Client_Info */ dto_cookie.as_64 = LZERO; dto_cookie.as_ptr = - (DAT_PVOID) (uintptr_t) DT_Bpool_GetBuffer ( ps_ptr->bpool, 0); + (void *) (uintptr_t) DT_Bpool_GetBuffer ( ps_ptr->bpool, 0); DT_Tdep_PT_Debug (1,(phead,"%s: Waiting for Client_Info\n", module)); if (!DT_dto_event_wait (phead, ps_ptr->recv_evd_hdl, &dto_stat) || !DT_dto_check ( phead, @@ -415,7 +415,7 @@ DT_cs_Server (Params_t * params_ptr) /* Wait for client's command info */ dto_cookie.as_64 = LZERO; dto_cookie.as_ptr = - (DAT_PVOID) (uintptr_t) DT_Bpool_GetBuffer ( ps_ptr->bpool, 1); + (void *) (uintptr_t) DT_Bpool_GetBuffer ( ps_ptr->bpool, 1); DT_Tdep_PT_Debug (1,(phead,"%s: Waiting for Client_Cmd_Info\n", module)); if (!DT_dto_event_wait (phead, ps_ptr->recv_evd_hdl, &dto_stat) || !DT_dto_check ( phead, @@ -601,7 +601,7 @@ DT_cs_Server (Params_t * params_ptr) /* reap the send and verify it */ dto_cookie.as_64 = LZERO; dto_cookie.as_ptr = - (DAT_PVOID) (uintptr_t) DT_Bpool_GetBuffer ( ps_ptr->bpool, 2); + (void *) (uintptr_t) DT_Bpool_GetBuffer ( ps_ptr->bpool, 2); if (!DT_dto_event_wait (phead, ps_ptr->reqt_evd_hdl, &dto_stat) || !DT_dto_check ( phead, &dto_stat, @@ -889,7 +889,7 @@ send_control_data ( /* reap the send and verify it */ dto_cookie.as_64 = LZERO; dto_cookie.as_ptr = - (DAT_PVOID) (uintptr_t) DT_Bpool_GetBuffer ( ps_ptr->bpool, 2); + (void *) (uintptr_t) DT_Bpool_GetBuffer ( ps_ptr->bpool, 2); if (!DT_dto_event_wait (phead, ps_ptr->reqt_evd_hdl, &dto_stat) || !DT_dto_check (phead, &dto_stat, diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/test/dapltest/test/dapl_test_util.c linux-kernel3/test/dapltest/test/dapl_test_util.c --- linux-kernel2/test/dapltest/test/dapl_test_util.c 2005-05-11 11:16:06.768963000 -0700 +++ linux-kernel3/test/dapltest/test/dapl_test_util.c 2005-05-11 16:15:37.174000000 -0700 @@ -179,7 +179,7 @@ DT_post_recv_buffer (DT_Tdep_Print_Head iov->segment_length = size; iov->lmr_context = lmr_c; cookie.as_64 = (u64)0UL; - cookie.as_ptr = (DAT_PVOID) buff; + cookie.as_ptr = (void *) buff; DT_Tdep_PT_Debug (3, (phead, "Post-Recv #%d [%p, %x]\n", index, buff, size)); @@ -224,7 +224,7 @@ DT_post_send_buffer (DT_Tdep_Print_Head iov->segment_length = size; iov->lmr_context = lmr_c; cookie.as_64 = (u64)0UL; - cookie.as_ptr = (DAT_PVOID) buff; + cookie.as_ptr = (void *) buff; DT_Tdep_PT_Debug (3, (phead, "Post-Send #%d [%p, %x]\n", index, buff, size)); /* Post the recv buffer */ @@ -621,7 +621,7 @@ boolean_t DT_rmr_check ( DT_Tdep_Print_Head *phead, struct dat_rmr_bind_completion_event_data *rmr_p, DAT_RMR_HANDLE rmr_expected, - DAT_PVOID cookie_expected, + void * cookie_expected, char *message) { if ( rmr_p->rmr_handle != rmr_expected diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/test/dapltest/test/dapl_transaction_test.c linux-kernel3/test/dapltest/test/dapl_transaction_test.c --- linux-kernel2/test/dapltest/test/dapl_transaction_test.c 2005-05-11 11:16:06.783962000 -0700 +++ linux-kernel3/test/dapltest/test/dapl_transaction_test.c 2005-05-11 16:15:37.184002000 -0700 @@ -406,19 +406,19 @@ DT_Transaction_Main (void *param) DT_Tdep_PT_Debug (3, (phead, "0: RMI_SEND %p\n", - (DAT_PVOID) DT_Bpool_GetBuffer ( + (void *) DT_Bpool_GetBuffer ( test_ptr->ep_context[i].bp, 0))); DT_Tdep_PT_Debug (3, (phead, "1: RMI_RECV %p\n", - (DAT_PVOID) DT_Bpool_GetBuffer ( + (void *) DT_Bpool_GetBuffer ( test_ptr->ep_context[i].bp, 1))); DT_Tdep_PT_Debug (3, (phead, "2: SYNC_SEND %p\n", - (DAT_PVOID) DT_Bpool_GetBuffer ( + (void *) DT_Bpool_GetBuffer ( test_ptr->ep_context[i].bp, 2))); DT_Tdep_PT_Debug (3, (phead, "3: SYNC_RECV %p\n", - (DAT_PVOID) DT_Bpool_GetBuffer ( + (void *) DT_Bpool_GetBuffer ( test_ptr->ep_context[i].bp, 3))); /* @@ -575,7 +575,7 @@ DT_Transaction_Main (void *param) /* what, me query? just try to accept the connection */ ret = dat_cr_accept (cr_handle, NULL, /* NULL for RSP */ - 0, (DAT_PVOID)0 /* no private data */ ); + 0, (void *)0 /* no private data */ ); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_accept #%d error: %s\n", @@ -660,7 +660,7 @@ DT_Transaction_Main (void *param) /* what, me query? just try to accept the connection */ ret = dat_cr_accept (cr_handle, test_ptr->ep_context[i].ep_handle, - 0, (DAT_PVOID)0 /* no private data */ ); + 0, (void *)0 /* no private data */ ); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test[" F64x "]: dat_cr_accept #%d error: %s\n", @@ -926,7 +926,7 @@ retry: } DT_Tdep_PT_Debug (3, (phead, - "%d: S/R [ va=%p ]\n", j, (DAT_PVOID) + "%d: S/R [ va=%p ]\n", j, (void *) DT_Bpool_GetBuffer ( test_ptr->ep_context[i].op[j].bp, 0))); break; } @@ -989,7 +989,7 @@ retry: /* reap the send and verify it */ dto_cookie.as_64 = LZERO; dto_cookie.as_ptr = - (DAT_PVOID) DT_Bpool_GetBuffer ( + (void *) DT_Bpool_GetBuffer ( test_ptr->ep_context[i].bp, RMI_SEND_BUFFER_ID); if (!DT_dto_event_wait (phead, test_ptr->reqt_evd_hdl, &dto_stat) || @@ -1013,7 +1013,7 @@ retry: test_ptr->is_server ? "Client" : "Server")); dto_cookie.as_64 = LZERO; dto_cookie.as_ptr = - (DAT_PVOID) DT_Bpool_GetBuffer ( + (void *) DT_Bpool_GetBuffer ( test_ptr->ep_context[i].bp, RMI_RECV_BUFFER_ID); if (!DT_dto_event_wait (phead, test_ptr->recv_evd_hdl, &dto_stat) || From libor at topspin.com Wed May 11 18:01:08 2005 From: libor at topspin.com (Libor Michalek) Date: Wed, 11 May 2005 18:01:08 -0700 Subject: [openib-general] Re: [PATCH][SDP] AIO buffer corruption In-Reply-To: <20050504190035.GA32291@mellanox.co.il>; from mst@mellanox.co.il on Wed, May 04, 2005 at 10:00:35PM +0300 References: <20050427182254.A11429@topspin.com> <20050504190035.GA32291@mellanox.co.il> Message-ID: <20050511180108.D5174@topspin.com> On Wed, May 04, 2005 at 10:00:35PM +0300, Michael S. Tsirkin wrote: > Quoting r. Libor Michalek : > > Subject: [PATCH][SDP] AIO buffer corruption > > > > Patch to fix the problem a few people reported as ttcp.aio.c > > aborting with an error (-104) on longer AIO runs. > > > > The bug is in the calculation of an AIO buffers starting address. > > It would cause data to potentially be written past the end of the > > AIO buffer corrupting whatever happen to be there. In the case of > > ttcp.aio.c this happen to be the iocb array, which once corrupted > > would generate this error when passed to io_submit. > > Unfortunately I still see data corruptions sometimes with this patch applied. > The result for me is the server reporting verification error, closing the > socket, and client printing the 104 event. > I'm still debugging, but wanted to ask if someone else is seeing this too. Michael, I found a couple issues, the one above is with ttcp.aio.c itself. When running with verification the receive side is incorrectly calculating which value it should be seeing in the received buffer. It is effectively one buffer ahead because the num_bytes variable is being updated too early. The patch is below. I also found a buffer mismatch problem when using buffers below the zcopy threshold. Signed-off-by: Libor Michalek Index: ttcp.aio.c =================================================================== --- ttcp.aio.c (revision 2248) +++ ttcp.aio.c (working copy) @@ -538,9 +538,7 @@ for (ev = 0; ev < result && 0 < (long)events[ev].res; - ev++, ex++, cu--) { - - num_bytes += events[ev].res; + num_bytes += events[ev++].res, ex++, cu--) { if (!verify) continue; @@ -563,7 +561,7 @@ } if (0 > (long)events[ev].res) - fprintf(stderr, "ttcp%s: Event error <%ld> <%ld>\n", + fprintf(stderr, "ttcp%s: Event error <%ld> <%ld>\n", trans ? "-t" : "-r", (long)events[ev].res, (long)events[ev].data); From drevel at fadmail.com Wed May 11 20:11:11 2005 From: drevel at fadmail.com (Randell Hensley) Date: Thu, 12 May 2005 09:11:11 +0600 Subject: [openib-general] Lowest rates in 45 years Message-ID: <119741.2049.drevel@fadmail.com> Hello, We tried contacting you awhile ago about your low interest morta(ge rate. You have qualified for the lowest rate in years... You could get over $380,000 for as little as $500 a month! Ba(d credit? Doesn't matter, low rates are fixed no matter what! To get a free, no obli,gation consultation click below: http://www.trust1ng.net/sign.asp Best Regards, Morris Schwartz to be remov(ed: http://www.trust1ng.net/gone.asp this process takes one week, so please be patient. we do our best to take your email/s off but you have to fill out a rem/ove or else you will continue to recieve email/s. From info at kqsv13.com Wed May 11 06:59:54 2005 From: info at kqsv13.com (info at kqsv13.com) Date: 11 May 2005 22:59:54 +0900 Subject: [openib-general] $B?75,EPO?$NJ}$O(B10000$B1_J,L5NA(B Message-ID: <20050511135954.26991.qmail@mail.kqsv12.com> $B0lK|1_J,!*40A4L5NA$G;HMQ(BOK $B"#:#$J$i4V$K9g$&(BW$B%A%c%s%9!*L5NA%]%$%s%H%"%C%W3Z$7$5(B100$BG\!*"#(B $B"(EPO?=w at -2q0w$N%8%c%s%kJL?M5$%i%s%-%s%0(B $B0l0L!'5U1g!J(B189$BL>EPO?!K(B $BFs0L!'1g8r!J(B429$BL>EPO?!K(B $B;00L!'(BSM$B5U1g!J(B243$BL>EPO?!K(B $B;M0L!'%;%U%l!J(B1038$BL>EPO?!K(B $B8^0L!'ITNQ!J(B2421$BL>EPO?!K(B $BO;0L!'%F%l%(%C%A!J(B3463$BL>EPO?!K(B $B!&(B $B!&(B $B!&(B $B0lK|1_J,40A4L5NA$G$*;n$7$O(B $B"-"-"-(B http://awg.webchu.com/?summer13 $BB~:#!*?75,EPO?$7$?J}$K$OL5NA$G(B10,000$B1_J,$4MxMQ=PMh$^$9!#(B $B!ZL5NA%]%$%s%HFb$G==J,$K%Q!<%H%J!<$rC5$9;v$,=PMh$^$9!#![(B $B"((B18$B:P0J>e$NHkL)87l9g$O2<5-(BURL$B$KJV?.2<$5$$!#(B henshin_senyou at ok.kz From rothen at emailaccount.com Thu May 12 04:53:07 2005 From: rothen at emailaccount.com (Mack Hansen) Date: Thu, 12 May 2005 04:53:07 -0700 Subject: [openib-general] Need a low mortage rate? Message-ID: Hello, We tried contacting you awhile ago about your low interest morta(ge rate.= You have qualified for the lowest rate in years... You could get over $380,000 for as little as $500 a month! Ba(d credit? Doesn't matter, low rates are fixed no matter what! To get a free, no obli,gation consultation click below: http://www.ch00s3.com/sign.asp Best Regards, Sterling Newton to be remov(ed: http://www.ch00s3.com/gone.asp this process takes one week, so please be patient. we do our best to take your email/s off but you have to fill out a rem/ove or else you will continue to recieve email/s. From halr at voltaire.com Thu May 12 05:41:45 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 12 May 2005 08:41:45 -0400 Subject: [openib-general] DAPL CM Reject Questions Message-ID: <1115901591.4766.38.camel@localhost.localdomain> Hi Sean, I have a couple questions related to REJ and the CM based on the IBA spec and the CM. On the passive side, an RTU is received but the QP cannot be transition to RTS so a REJ is attempted as below: static int do_rtu_recv(struct dapl_cm_id *conn) { int status; status = dapl_modify_qp_state_to_rts(conn->cm_id, conn->ep->qp_handle); if (status) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, " do_rtu_recv: could not " "modify QP state to RTS status %d\n", status); goto reject; } ... reject: ib_send_cm_rej(conn->cm_id, IB_CM_REJ_CONSUMER_DEFINED, NULL, 0, NULL, 0); If RTU is received, the connection would be in ESTABLISHED state. Is sending REJ valid there ? What's odd to me is that the CM state machine in the IBA spec shows REJ being received in the established state on the active side (and moving to TIMEWAIT) but there is no action out of established on the passive side showing send REJ. If sending REJ is valid on the passive side in established state, does the current CM handle this ? It looks to me like it returns -EINVAL for this case. switch (cm_id->state) { case IB_CM_REQ_SENT: case IB_CM_MRA_REQ_RCVD: case IB_CM_REQ_RCVD: case IB_CM_MRA_REQ_SENT: case IB_CM_REP_RCVD: case IB_CM_MRA_REP_SENT: cm_reset_to_idle(cm_id_priv); break; case IB_CM_REP_SENT: case IB_CM_MRA_REP_RCVD: cm_enter_timewait(cm_id_priv); break; default: spin_unlock_irqrestore(&cm_id_priv->lock, flags); ret = -EINVAL; goto out; } -- Hal From halr at voltaire.com Thu May 12 05:44:58 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 12 May 2005 08:44:58 -0400 Subject: [openib-general] [PATCH] kdapl CM: Fix some double deletions in error paths on active side Message-ID: <1115901608.4766.40.camel@localhost.localdomain> kdapl CM: Fix some double deletions in error paths on active side Also, minor change to error message Signed-off-by: Hal Rosenstock Index: dapl_openib_cm.c =================================================================== -- dapl_openib_cm.c (revision 2309) +++ dapl_openib_cm.c (working copy) @@ -283,7 +283,7 @@ if (rec_num <= 0) { printk(KERN_ERR "dapl_path_comp_handler: path resolution " - "failed (%d) retries %d!!!\n", rec_num, conn->retries); + "failed %d retry %d!!!\n", rec_num, conn->retries); if (++conn->retries > OPENIB_MAX_AT_RETRY) { printk(KERN_ERR "dapl_path_comp_handler: ep_ptr 0x%p\n", conn->ep); @@ -322,9 +322,7 @@ return; error: - //*** should we pass conn here? it will be destroyed... dapl_evd_connection_callback(conn, event, NULL, conn->ep); - dapl_destroy_cm_id(conn); } static void dapl_rt_comp_handler(u64 req_id, void *context, int rec_num) @@ -334,7 +332,7 @@ ib_cm_events_t event; if (rec_num <= 0) { - printk(KERN_ERR "dapl_rt_comp_handler: rec_num %d retry %d\n", + printk(KERN_ERR "dapl_rt_comp_handler: rec num %d retry %d\n", rec_num, conn->retries); if (++conn->retries > OPENIB_MAX_AT_RETRY) { event = IB_CME_DESTINATION_UNREACHABLE; @@ -371,9 +369,7 @@ return; error: - //*** should we pass conn here? it will be destroyed... dapl_evd_connection_callback(conn, event, NULL, conn->ep); - dapl_destroy_cm_id(conn); } /* From tduffy at sun.com Thu May 12 08:01:10 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 12 May 2005 08:01:10 -0700 Subject: [openib-general] [PATCHv2] kDAPL: remove DAT_RETURN Message-ID: <1115910070.25530.2.camel@duffman> This is the same patch I sent out yesterday to remove DAT_RETURN, just rebased against v2313. Signed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat/api.c linux-kernel2/dat/api.c --- linux-kernel/dat/api.c 2005-05-12 07:46:08.886002000 -0700 +++ linux-kernel2/dat/api.c 2005-05-12 07:47:31.724000000 -0700 @@ -44,7 +44,7 @@ * Function: dat_strerror_major ***********************************************************************/ -static DAT_RETURN dat_strerror_major(DAT_RETURN value, const char **message) +static u32 dat_strerror_major(u32 value, const char **message) { switch (DAT_GET_TYPE(value)) { case DAT_SUCCESS: @@ -163,7 +163,7 @@ static DAT_RETURN dat_strerror_major(DAT /*********************************************************************** * Function: dat_strerror_minor ***********************************************************************/ -static DAT_RETURN dat_strerror_minor(DAT_RETURN value, const char **message) +static u32 dat_strerror_minor(u32 value, const char **message) { switch (DAT_GET_SUBTYPE(value)) { @@ -575,16 +575,13 @@ static DAT_RETURN dat_strerror_minor(DAT * Function: dat_ia_openv ***********************************************************************/ -DAT_RETURN dat_ia_openv(const DAT_NAME_PTR name, - DAT_COUNT async_event_qlen, - DAT_EVD_HANDLE *async_event_handle, - DAT_IA_HANDLE *ia_handle, - u32 dat_major, u32 dat_minor, - boolean_t thread_safety) +u32 dat_ia_openv(const DAT_NAME_PTR name, DAT_COUNT async_event_qlen, + DAT_EVD_HANDLE *async_event_handle, DAT_IA_HANDLE *ia_handle, + u32 dat_major, u32 dat_minor, boolean_t thread_safety) { DAT_IA_OPEN_FUNC ia_open_func; struct dat_provider_info info; - DAT_RETURN status; + u32 status; size_t len; dat_dbg_print(DAT_DBG_CLASS_CONSUMER_API, @@ -628,11 +625,11 @@ EXPORT_SYMBOL(dat_ia_openv); * Function: dat_ia_close ***********************************************************************/ -DAT_RETURN dat_ia_close(DAT_IA_HANDLE ia_handle, DAT_CLOSE_FLAGS flags) +u32 dat_ia_close(DAT_IA_HANDLE ia_handle, DAT_CLOSE_FLAGS flags) { struct dat_provider *provider; struct dat_provider_attr provider_attr; - DAT_RETURN status; + u32 status; const char *ia_name; dat_dbg_print(DAT_DBG_CLASS_CONSUMER_API, @@ -682,8 +679,8 @@ EXPORT_SYMBOL(dat_ia_close); * Function: dat_strerror ***********************************************************************/ -DAT_RETURN dat_strerror(DAT_RETURN value, const char **major_message, - const char **minor_message) +u32 dat_strerror(u32 value, const char **major_message, + const char **minor_message) { dat_dbg_print(DAT_DBG_CLASS_CONSUMER_API, @@ -692,9 +689,9 @@ DAT_RETURN dat_strerror(DAT_RETURN value /* * The DAT specification contains a note to implementers - * suggesting that the consumer's DAT_RETURN value be used + * suggesting that the consumer's return value be used * as an index into a table of text strings. However, - * the DAT_RETURN values are not consecutive. Therefore this + * the return values are not consecutive. Therefore this * implementation does not follow the suggested implementation. */ @@ -715,8 +712,8 @@ EXPORT_SYMBOL(dat_strerror); * Function: dat_registry_add_provider ***********************************************************************/ -DAT_RETURN dat_registry_add_provider(const struct dat_provider *provider, - const struct dat_provider_info *provider_info) +u32 dat_registry_add_provider(const struct dat_provider *provider, + const struct dat_provider_info *provider_info) { dat_dbg_print(DAT_DBG_CLASS_PROVIDER_API, "%s(ia:%s, ver:%x.%x, thrd_safe:%x)\n", @@ -740,8 +737,8 @@ EXPORT_SYMBOL(dat_registry_remove_provid * Function: dat_registry_remove_provider ***********************************************************************/ -DAT_RETURN dat_registry_remove_provider(const struct dat_provider *provider, - const struct dat_provider_info *provider_info) +u32 dat_registry_remove_provider(const struct dat_provider *provider, + const struct dat_provider_info *provider_info) { dat_dbg_print(DAT_DBG_CLASS_PROVIDER_API, "%s(ia:%s)\n", @@ -759,13 +756,12 @@ EXPORT_SYMBOL(dat_registry_add_provider) * Function: dat_registry_list_providers ***********************************************************************/ -DAT_RETURN dat_registry_list_providers(DAT_COUNT max_to_return, - DAT_COUNT *entries_returned, - struct dat_provider_info *(dat_provider_list[])) +u32 dat_registry_list_providers(DAT_COUNT max_to_return, + DAT_COUNT *entries_returned, + struct dat_provider_info *(dat_provider_list[])) { - DAT_RETURN status; + u32 status = DAT_SUCCESS; - status = DAT_SUCCESS; dat_dbg_print(DAT_DBG_CLASS_CONSUMER_API, "%s(max_to_return:%i, entries_rtrn:%p, prvdr_list:%p)\n", __func__, max_to_return, entries_returned, diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat/consumer.c linux-kernel2/dat/consumer.c --- linux-kernel/dat/consumer.c 2005-05-10 10:19:45.654000000 -0700 +++ linux-kernel2/dat/consumer.c 2005-05-11 11:22:58.743960000 -0700 @@ -47,7 +47,7 @@ * Function: dat_strerror_major ***********************************************************************/ -static DAT_RETURN dat_strerror_major(DAT_RETURN value, const char **message) +static u32 dat_strerror_major(u32 value, const char **message) { switch (DAT_GET_TYPE(value)) { case DAT_SUCCESS: @@ -166,7 +166,7 @@ static DAT_RETURN dat_strerror_major(DAT /*********************************************************************** * Function: dat_strerror_minor ***********************************************************************/ -static DAT_RETURN dat_strerror_minor(DAT_RETURN value, const char **message) +static u32 dat_strerror_minor(u32 value, const char **message) { switch (DAT_GET_SUBTYPE(value)) { @@ -574,7 +574,7 @@ static DAT_RETURN dat_strerror_minor(DAT * * *********************************************************************/ -DAT_RETURN +u32 dat_ia_openv(const DAT_NAME_PTR name, DAT_COUNT async_event_qlen, DAT_EVD_HANDLE * async_event_handle, @@ -584,7 +584,7 @@ dat_ia_openv(const DAT_NAME_PTR name, { DAT_IA_OPEN_FUNC ia_open_func; DAT_PROVIDER_INFO info; - DAT_RETURN status; + u32 status; size_t len; dat_dbg_print(DAT_DBG_CLASS_CONSUMER_API, @@ -627,11 +627,11 @@ dat_ia_openv(const DAT_NAME_PTR name, EXPORT_SYMBOL(dat_ia_openv); -DAT_RETURN dat_ia_close(DAT_IA_HANDLE ia_handle, DAT_CLOSE_FLAGS flags) +u32 dat_ia_close(DAT_IA_HANDLE ia_handle, DAT_CLOSE_FLAGS flags) { struct dat_provider *provider; struct dat_provider_attr provider_attr; - DAT_RETURN status; + u32 status; const char *ia_name; dat_dbg_print(DAT_DBG_CLASS_CONSUMER_API, @@ -681,9 +681,8 @@ DAT_RETURN dat_ia_close(DAT_IA_HANDLE ia EXPORT_SYMBOL(dat_ia_close); -DAT_RETURN -dat_strerror(DAT_RETURN value, - const char **major_message, const char **minor_message) +u32 +dat_strerror(u32 value, const char **major_message, const char **minor_message) { dat_dbg_print(DAT_DBG_CLASS_CONSUMER_API, @@ -692,9 +691,9 @@ dat_strerror(DAT_RETURN value, /* * The DAT specification contains a note to implementers - * suggesting that the consumer's DAT_RETURN value be used + * suggesting that the consumer's return value be used * as an index into a table of text strings. However, - * the DAT_RETURN values are not consecutive. Therefore this + * the return values are not consecutive. Therefore this * implementation does not follow the suggested implementation. */ diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat/dat.h linux-kernel2/dat/dat.h --- linux-kernel/dat/dat.h 2005-05-10 10:32:56.912966000 -0700 +++ linux-kernel2/dat/dat.h 2005-05-11 13:07:34.341014000 -0700 @@ -47,25 +47,23 @@ typedef u64 DAT_PADDR; /* * All return codes are actually a 3-way tuple: * - * type: DAT_RETURN_CLASS DAT_RETURN_TYPE DAT_RETURN_SUBTYPE + * type: dat_return_class dat_return_type dat_return_subtype * bits: 31-30 29-16 15-0 * * 3 2 1 * 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 9 8 7 6 5 4 3 2 1 0 * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ - * | C | DAT_RETURN_TYPE | DAT_RETURN_SUBTYPE | + * | C | dat_return_type | dat_return_subtype | * +-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+ */ -typedef u32 DAT_RETURN; - -typedef enum dat_return_class { +enum dat_return_class { DAT_CLASS_ERROR = 0x80000000, DAT_CLASS_WARNING = 0x40000000, DAT_CLASS_SUCCESS = 0x00000000 -} DAT_RETURN_CLASS; +}; -typedef enum dat_return_type { +enum dat_return_type { /* The operation was successful. */ DAT_SUCCESS = 0x000000, @@ -144,9 +142,9 @@ typedef enum dat_return_type { /* Provider does not support the operation yet. */ DAT_NOT_IMPLEMENTED = 0x3FFF0000 -} DAT_STATUS_TYPE; +}; -typedef enum dat_return_subtype { +enum dat_return_subtype { /* First element is no subtype */ DAT_NO_SUBTYPE, /* ABORT sub types */ @@ -321,7 +319,7 @@ typedef enum dat_return_subtype { DAT_MAJOR_NOT_FOUND, DAT_MINOR_NOT_FOUND, DAT_THREAD_SAFETY_NOT_FOUND -} DAT_RETURN_SUBTYPE; +}; /* * Error bits @@ -336,8 +334,8 @@ typedef enum dat_return_subtype { #define DAT_IS_ERROR(status) ((u32)(status) & DAT_CLASS_ERROR) #define DAT_IS_WARNING(status) ((u32)(status) & DAT_CLASS_WARNING) -#define DAT_ERROR(Type, SubType) ((DAT_RETURN)(DAT_CLASS_ERROR | Type | SubType)) -#define DAT_WARNING(Type, SubType) ((DAT_RETURN)(DAT_CLASS_WARNING | Type | SubType)) +#define DAT_ERROR(type, sub_type) ((u32)(DAT_CLASS_ERROR | type | sub_type)) +#define DAT_WARNING(type, sub_type) ((u32)(DAT_CLASS_WARNING | type | sub_type)) #define DAT_GET_TYPE(status) ((u32)(status) & DAT_TYPE_MASK) #define DAT_GET_SUBTYPE(status) ((u32)(status) & DAT_SUBTYPE_MASK) @@ -1107,178 +1105,178 @@ struct dat_provider_attr { * ****************************************************************/ -typedef DAT_RETURN (*DAT_IA_OPEN_FUNC) ( +typedef u32 (*DAT_IA_OPEN_FUNC) ( const DAT_NAME_PTR, DAT_COUNT, DAT_EVD_HANDLE *, DAT_IA_HANDLE *); -typedef DAT_RETURN (*DAT_IA_OPENV_FUNC) ( +typedef u32 (*DAT_IA_OPENV_FUNC) ( const DAT_NAME_PTR, DAT_COUNT, DAT_EVD_HANDLE *, DAT_IA_HANDLE *, u32, u32, boolean_t); -typedef DAT_RETURN (*DAT_IA_CLOSE_FUNC) (DAT_IA_HANDLE, DAT_CLOSE_FLAGS); +typedef u32 (*DAT_IA_CLOSE_FUNC) (DAT_IA_HANDLE, DAT_CLOSE_FLAGS); -typedef DAT_RETURN (*DAT_IA_QUERY_FUNC) (DAT_IA_HANDLE, DAT_EVD_HANDLE *, +typedef u32 (*DAT_IA_QUERY_FUNC) (DAT_IA_HANDLE, DAT_EVD_HANDLE *, struct dat_ia_attr *, struct dat_provider_attr *); -typedef DAT_RETURN (*DAT_SET_CONSUMER_CONTEXT_FUNC) (DAT_HANDLE, DAT_CONTEXT); +typedef u32 (*DAT_SET_CONSUMER_CONTEXT_FUNC) (DAT_HANDLE, DAT_CONTEXT); -typedef DAT_RETURN (*DAT_GET_CONSUMER_CONTEXT_FUNC) (DAT_HANDLE, DAT_CONTEXT *); +typedef u32 (*DAT_GET_CONSUMER_CONTEXT_FUNC) (DAT_HANDLE, DAT_CONTEXT *); -typedef DAT_RETURN (*DAT_GET_HANDLE_TYPE_FUNC) (DAT_HANDLE, DAT_HANDLE_TYPE *); +typedef u32 (*DAT_GET_HANDLE_TYPE_FUNC) (DAT_HANDLE, DAT_HANDLE_TYPE *); -typedef DAT_RETURN (*DAT_CR_QUERY_FUNC) (DAT_CR_HANDLE, struct dat_cr_param *); +typedef u32 (*DAT_CR_QUERY_FUNC) (DAT_CR_HANDLE, struct dat_cr_param *); -typedef DAT_RETURN (*DAT_CR_ACCEPT_FUNC) ( +typedef u32 (*DAT_CR_ACCEPT_FUNC) ( DAT_CR_HANDLE, DAT_EP_HANDLE, DAT_COUNT, const DAT_PVOID); -typedef DAT_RETURN (*DAT_CR_REJECT_FUNC) (DAT_CR_HANDLE); +typedef u32 (*DAT_CR_REJECT_FUNC) (DAT_CR_HANDLE); -typedef DAT_RETURN (*DAT_CR_HANDOFF_FUNC) (DAT_CR_HANDLE, DAT_CONN_QUAL); +typedef u32 (*DAT_CR_HANDOFF_FUNC) (DAT_CR_HANDLE, DAT_CONN_QUAL); -typedef DAT_RETURN (*DAT_EVD_RESIZE_FUNC) (DAT_EVD_HANDLE, DAT_COUNT); +typedef u32 (*DAT_EVD_RESIZE_FUNC) (DAT_EVD_HANDLE, DAT_COUNT); -typedef DAT_RETURN (*DAT_EVD_POST_SE_FUNC) (DAT_EVD_HANDLE, +typedef u32 (*DAT_EVD_POST_SE_FUNC) (DAT_EVD_HANDLE, const struct dat_event *); -typedef DAT_RETURN (*DAT_EVD_DEQUEUE_FUNC) (DAT_EVD_HANDLE, struct dat_event *); +typedef u32 (*DAT_EVD_DEQUEUE_FUNC) (DAT_EVD_HANDLE, struct dat_event *); -typedef DAT_RETURN (*DAT_EVD_FREE_FUNC) (DAT_EVD_HANDLE); +typedef u32 (*DAT_EVD_FREE_FUNC) (DAT_EVD_HANDLE); -typedef DAT_RETURN (*DAT_EVD_QUERY_FUNC) (DAT_EVD_HANDLE, +typedef u32 (*DAT_EVD_QUERY_FUNC) (DAT_EVD_HANDLE, struct dat_evd_param *); -typedef DAT_RETURN (*DAT_EP_CREATE_FUNC) ( +typedef u32 (*DAT_EP_CREATE_FUNC) ( DAT_IA_HANDLE, DAT_PZ_HANDLE, DAT_EVD_HANDLE, DAT_EVD_HANDLE, DAT_EVD_HANDLE, const struct dat_ep_attr *, DAT_EP_HANDLE *); -typedef DAT_RETURN (*DAT_EP_CREATE_WITH_SRQ_FUNC) ( +typedef u32 (*DAT_EP_CREATE_WITH_SRQ_FUNC) ( DAT_IA_HANDLE, DAT_PZ_HANDLE, DAT_EVD_HANDLE, DAT_EVD_HANDLE, DAT_EVD_HANDLE, DAT_SRQ_HANDLE, const struct dat_ep_attr *, DAT_EP_HANDLE *); -typedef DAT_RETURN (*DAT_EP_QUERY_FUNC) (DAT_EP_HANDLE, struct dat_ep_param *); +typedef u32 (*DAT_EP_QUERY_FUNC) (DAT_EP_HANDLE, struct dat_ep_param *); -typedef DAT_RETURN (*DAT_EP_MODIFY_FUNC) ( +typedef u32 (*DAT_EP_MODIFY_FUNC) ( DAT_EP_HANDLE, DAT_EP_PARAM_MASK, const struct dat_ep_param *); -typedef DAT_RETURN (*DAT_EP_CONNECT_FUNC) ( +typedef u32 (*DAT_EP_CONNECT_FUNC) ( DAT_EP_HANDLE, DAT_IA_ADDRESS_PTR, DAT_CONN_QUAL, DAT_TIMEOUT, DAT_COUNT, const DAT_PVOID, DAT_QOS, DAT_CONNECT_FLAGS); -typedef DAT_RETURN (*DAT_EP_DUP_CONNECT_FUNC) ( +typedef u32 (*DAT_EP_DUP_CONNECT_FUNC) ( DAT_EP_HANDLE, DAT_EP_HANDLE, DAT_TIMEOUT, DAT_COUNT, const DAT_PVOID, DAT_QOS); -typedef DAT_RETURN (*DAT_EP_DISCONNECT_FUNC) (DAT_EP_HANDLE, DAT_CLOSE_FLAGS); +typedef u32 (*DAT_EP_DISCONNECT_FUNC) (DAT_EP_HANDLE, DAT_CLOSE_FLAGS); -typedef DAT_RETURN (*DAT_EP_POST_SEND_FUNC) ( +typedef u32 (*DAT_EP_POST_SEND_FUNC) ( DAT_EP_HANDLE, DAT_COUNT, struct dat_lmr_triplet *, DAT_DTO_COOKIE, DAT_COMPLETION_FLAGS); -typedef DAT_RETURN (*DAT_EP_POST_RECV_FUNC) ( +typedef u32 (*DAT_EP_POST_RECV_FUNC) ( DAT_EP_HANDLE, DAT_COUNT, struct dat_lmr_triplet *, DAT_DTO_COOKIE, DAT_COMPLETION_FLAGS); -typedef DAT_RETURN (*DAT_EP_POST_RDMA_READ_FUNC) ( +typedef u32 (*DAT_EP_POST_RDMA_READ_FUNC) ( DAT_EP_HANDLE, DAT_COUNT, struct dat_lmr_triplet *, DAT_DTO_COOKIE, const struct dat_rmr_triplet *, DAT_COMPLETION_FLAGS); -typedef DAT_RETURN (*DAT_EP_POST_RDMA_WRITE_FUNC) ( +typedef u32 (*DAT_EP_POST_RDMA_WRITE_FUNC) ( DAT_EP_HANDLE, DAT_COUNT, struct dat_lmr_triplet *, DAT_DTO_COOKIE, const struct dat_rmr_triplet *, DAT_COMPLETION_FLAGS); -typedef DAT_RETURN (*DAT_EP_GET_STATUS_FUNC) ( +typedef u32 (*DAT_EP_GET_STATUS_FUNC) ( DAT_EP_HANDLE, DAT_EP_STATE *, boolean_t *, boolean_t *); -typedef DAT_RETURN (*DAT_EP_FREE_FUNC) (DAT_EP_HANDLE); +typedef u32 (*DAT_EP_FREE_FUNC) (DAT_EP_HANDLE); -typedef DAT_RETURN (*DAT_EP_RESET_FUNC) (DAT_EP_HANDLE); +typedef u32 (*DAT_EP_RESET_FUNC) (DAT_EP_HANDLE); -typedef DAT_RETURN (*DAT_EP_RECV_QUERY_FUNC) ( +typedef u32 (*DAT_EP_RECV_QUERY_FUNC) ( DAT_EP_HANDLE, DAT_COUNT *, DAT_COUNT *); -typedef DAT_RETURN (*DAT_EP_SET_WATERMARK_FUNC) ( +typedef u32 (*DAT_EP_SET_WATERMARK_FUNC) ( DAT_EP_HANDLE, DAT_COUNT, DAT_COUNT); -typedef DAT_RETURN (*DAT_LMR_FREE_FUNC) (DAT_LMR_HANDLE); +typedef u32 (*DAT_LMR_FREE_FUNC) (DAT_LMR_HANDLE); -typedef DAT_RETURN (*DAT_LMR_QUERY_FUNC) (DAT_LMR_HANDLE, +typedef u32 (*DAT_LMR_QUERY_FUNC) (DAT_LMR_HANDLE, struct dat_lmr_param *); -typedef DAT_RETURN (*DAT_LMR_SYNC_RDMA_READ_FUNC) ( +typedef u32 (*DAT_LMR_SYNC_RDMA_READ_FUNC) ( DAT_IA_HANDLE, const struct dat_lmr_triplet *, DAT_VLEN); -typedef DAT_RETURN (*DAT_LMR_SYNC_RDMA_WRITE_FUNC) ( +typedef u32 (*DAT_LMR_SYNC_RDMA_WRITE_FUNC) ( DAT_IA_HANDLE, const struct dat_lmr_triplet *, DAT_VLEN); -typedef DAT_RETURN (*DAT_RMR_CREATE_FUNC) (DAT_PZ_HANDLE, DAT_RMR_HANDLE *); +typedef u32 (*DAT_RMR_CREATE_FUNC) (DAT_PZ_HANDLE, DAT_RMR_HANDLE *); -typedef DAT_RETURN (*DAT_RMR_QUERY_FUNC) (DAT_RMR_HANDLE, struct dat_rmr_param *); +typedef u32 (*DAT_RMR_QUERY_FUNC) (DAT_RMR_HANDLE, struct dat_rmr_param *); -typedef DAT_RETURN (*DAT_RMR_BIND_FUNC) ( +typedef u32 (*DAT_RMR_BIND_FUNC) ( DAT_RMR_HANDLE, const struct dat_lmr_triplet *, DAT_MEM_PRIV_FLAGS, DAT_EP_HANDLE, DAT_RMR_COOKIE, DAT_COMPLETION_FLAGS, DAT_RMR_CONTEXT *); -typedef DAT_RETURN (*DAT_RMR_FREE_FUNC) (DAT_RMR_HANDLE); +typedef u32 (*DAT_RMR_FREE_FUNC) (DAT_RMR_HANDLE); -typedef DAT_RETURN (*DAT_PSP_CREATE_FUNC) ( +typedef u32 (*DAT_PSP_CREATE_FUNC) ( DAT_IA_HANDLE, DAT_CONN_QUAL, DAT_EVD_HANDLE, DAT_PSP_FLAGS, DAT_PSP_HANDLE *); -typedef DAT_RETURN (*DAT_PSP_CREATE_ANY_FUNC) ( +typedef u32 (*DAT_PSP_CREATE_ANY_FUNC) ( DAT_IA_HANDLE, DAT_CONN_QUAL *, DAT_EVD_HANDLE, DAT_PSP_FLAGS, DAT_PSP_HANDLE *); -typedef DAT_RETURN (*DAT_PSP_QUERY_FUNC) (DAT_PSP_HANDLE, +typedef u32 (*DAT_PSP_QUERY_FUNC) (DAT_PSP_HANDLE, struct dat_psp_param *); -typedef DAT_RETURN (*DAT_PSP_FREE_FUNC) (DAT_PSP_HANDLE); +typedef u32 (*DAT_PSP_FREE_FUNC) (DAT_PSP_HANDLE); -typedef DAT_RETURN (*DAT_RSP_CREATE_FUNC) ( +typedef u32 (*DAT_RSP_CREATE_FUNC) ( DAT_IA_HANDLE, DAT_CONN_QUAL, DAT_EP_HANDLE, DAT_EVD_HANDLE, DAT_RSP_HANDLE *); -typedef DAT_RETURN (*DAT_RSP_QUERY_FUNC) (DAT_RSP_HANDLE, +typedef u32 (*DAT_RSP_QUERY_FUNC) (DAT_RSP_HANDLE, struct dat_rsp_param *); -typedef DAT_RETURN (*DAT_RSP_FREE_FUNC) (DAT_RSP_HANDLE); +typedef u32 (*DAT_RSP_FREE_FUNC) (DAT_RSP_HANDLE); -typedef DAT_RETURN (*DAT_PZ_CREATE_FUNC) (DAT_IA_HANDLE, DAT_PZ_HANDLE *); +typedef u32 (*DAT_PZ_CREATE_FUNC) (DAT_IA_HANDLE, DAT_PZ_HANDLE *); -typedef DAT_RETURN (*DAT_PZ_QUERY_FUNC) (DAT_PZ_HANDLE, struct dat_pz_param *); +typedef u32 (*DAT_PZ_QUERY_FUNC) (DAT_PZ_HANDLE, struct dat_pz_param *); -typedef DAT_RETURN (*DAT_PZ_FREE_FUNC) (DAT_PZ_HANDLE); +typedef u32 (*DAT_PZ_FREE_FUNC) (DAT_PZ_HANDLE); -typedef DAT_RETURN (*DAT_SRQ_CREATE_FUNC) ( +typedef u32 (*DAT_SRQ_CREATE_FUNC) ( DAT_IA_HANDLE, DAT_PZ_HANDLE, struct dat_srq_attr *, DAT_SRQ_HANDLE *); -typedef DAT_RETURN (*DAT_SRQ_SET_LW_FUNC) (DAT_SRQ_HANDLE, DAT_COUNT); +typedef u32 (*DAT_SRQ_SET_LW_FUNC) (DAT_SRQ_HANDLE, DAT_COUNT); -typedef DAT_RETURN (*DAT_SRQ_FREE_FUNC) (DAT_SRQ_HANDLE); +typedef u32 (*DAT_SRQ_FREE_FUNC) (DAT_SRQ_HANDLE); -typedef DAT_RETURN (*DAT_SRQ_QUERY_FUNC) (DAT_SRQ_HANDLE, +typedef u32 (*DAT_SRQ_QUERY_FUNC) (DAT_SRQ_HANDLE, struct dat_srq_param *); -typedef DAT_RETURN (*DAT_SRQ_RESIZE_FUNC) (DAT_SRQ_HANDLE, DAT_COUNT); +typedef u32 (*DAT_SRQ_RESIZE_FUNC) (DAT_SRQ_HANDLE, DAT_COUNT); -typedef DAT_RETURN (*DAT_SRQ_POST_RECV_FUNC) ( +typedef u32 (*DAT_SRQ_POST_RECV_FUNC) ( DAT_SRQ_HANDLE, DAT_COUNT, struct dat_lmr_triplet *, DAT_DTO_COOKIE); -typedef DAT_RETURN (*DAT_LMR_KCREATE_FUNC) ( +typedef u32 (*DAT_LMR_KCREATE_FUNC) ( DAT_IA_HANDLE, DAT_MEM_TYPE, DAT_REGION_DESCRIPTION, DAT_VLEN, DAT_PZ_HANDLE, DAT_MEM_PRIV_FLAGS, DAT_MEM_OPTIMIZE_FLAGS, DAT_LMR_HANDLE *, DAT_LMR_CONTEXT *, DAT_RMR_CONTEXT *, DAT_VLEN *, DAT_VADDR *); -typedef DAT_RETURN (*DAT_IA_MEMTYPE_HINT_FUNC) ( +typedef u32 (*DAT_IA_MEMTYPE_HINT_FUNC) ( DAT_IA_HANDLE, DAT_MEM_TYPE, DAT_VLEN, DAT_MEM_OPTIMIZE_FLAGS, DAT_VLEN *, DAT_VADDR *); -typedef DAT_RETURN (*DAT_EVD_KCREATE_FUNC) ( +typedef u32 (*DAT_EVD_KCREATE_FUNC) ( DAT_IA_HANDLE, DAT_COUNT, DAT_UPCALL_POLICY, const struct dat_upcall_object *, DAT_EVD_FLAGS, DAT_EVD_HANDLE *); -typedef DAT_RETURN (*DAT_EVD_MODIFY_UPCALL_FUNC) ( +typedef u32 (*DAT_EVD_MODIFY_UPCALL_FUNC) ( DAT_EVD_HANDLE, DAT_UPCALL_POLICY, const struct dat_upcall_object *); struct dat_provider { @@ -1372,16 +1370,16 @@ struct dat_provider { * DAT registry functions for providers */ -extern DAT_RETURN dat_registry_add_provider( +extern u32 dat_registry_add_provider( const struct dat_provider *, const struct dat_provider_info *); -extern DAT_RETURN dat_registry_remove_provider( +extern u32 dat_registry_remove_provider( const struct dat_provider *, const struct dat_provider_info *); /* * DAT registry functions for consumers */ -extern DAT_RETURN dat_ia_openv(const DAT_NAME_PTR, DAT_COUNT, +extern u32 dat_ia_openv(const DAT_NAME_PTR, DAT_COUNT, DAT_EVD_HANDLE *, DAT_IA_HANDLE *, u32, u32, boolean_t); @@ -1390,11 +1388,11 @@ extern DAT_RETURN dat_ia_openv(const DAT DAT_VERSION_MAJOR, DAT_VERSION_MINOR, \ DAT_THREADSAFE) -extern DAT_RETURN dat_ia_close(DAT_IA_HANDLE, DAT_CLOSE_FLAGS); +extern u32 dat_ia_close(DAT_IA_HANDLE, DAT_CLOSE_FLAGS); -extern DAT_RETURN dat_strerror(DAT_RETURN, const char **, const char **); +extern u32 dat_strerror(u32, const char **, const char **); -extern DAT_RETURN dat_registry_list_providers( +extern u32 dat_registry_list_providers( DAT_COUNT, DAT_COUNT *, struct dat_provider_info *dat_provider_list[]); /* @@ -1405,27 +1403,27 @@ extern DAT_RETURN dat_registry_list_prov #define DAT_CALL_PROVIDER_FUNC(func, handle, ...) \ DAT_HANDLE_TO_PROVIDER(handle)->func(handle, ##__VA_ARGS__) -static inline DAT_RETURN dat_set_consumer_context(DAT_HANDLE handle, +static inline u32 dat_set_consumer_context(DAT_HANDLE handle, DAT_CONTEXT context) { return DAT_CALL_PROVIDER_FUNC( set_consumer_context_func, handle, context); } -static inline DAT_RETURN dat_get_consumer_context(DAT_HANDLE handle, +static inline u32 dat_get_consumer_context(DAT_HANDLE handle, DAT_CONTEXT *context) { return DAT_CALL_PROVIDER_FUNC( get_consumer_context_func, handle, context); } -static inline DAT_RETURN dat_get_handle_type(DAT_HANDLE handle, +static inline u32 dat_get_handle_type(DAT_HANDLE handle, DAT_HANDLE_TYPE *type) { return DAT_CALL_PROVIDER_FUNC(get_handle_type_func, handle, type); } -static inline DAT_RETURN dat_ia_memtype_hint(DAT_IA_HANDLE ia, +static inline u32 dat_ia_memtype_hint(DAT_IA_HANDLE ia, DAT_MEM_TYPE mem_type, DAT_VLEN length, DAT_MEM_OPTIMIZE_FLAGS mem_optimize, @@ -1437,7 +1435,7 @@ static inline DAT_RETURN dat_ia_memtype_ preferred_length, preferred_alignment); } -static inline DAT_RETURN dat_ia_query(DAT_IA_HANDLE ia, +static inline u32 dat_ia_query(DAT_IA_HANDLE ia, DAT_EVD_HANDLE * async_evd, struct dat_ia_attr *ia_attr, struct dat_provider_attr *provider_attr) @@ -1446,7 +1444,7 @@ static inline DAT_RETURN dat_ia_query(DA ia_query_func, ia, async_evd, ia_attr, provider_attr); } -static inline DAT_RETURN dat_cr_accept(DAT_CR_HANDLE cr, +static inline u32 dat_cr_accept(DAT_CR_HANDLE cr, DAT_EP_HANDLE ep, DAT_COUNT private_data_size, const DAT_PVOID private_data) @@ -1455,35 +1453,35 @@ static inline DAT_RETURN dat_cr_accept(D cr_accept_func, cr, ep, private_data_size, private_data); } -static inline DAT_RETURN dat_cr_handoff(DAT_CR_HANDLE cr, +static inline u32 dat_cr_handoff(DAT_CR_HANDLE cr, DAT_CONN_QUAL handoff) { return DAT_CALL_PROVIDER_FUNC(cr_handoff_func, cr, handoff); } -static inline DAT_RETURN dat_cr_query(DAT_CR_HANDLE cr, +static inline u32 dat_cr_query(DAT_CR_HANDLE cr, struct dat_cr_param *param) { return DAT_CALL_PROVIDER_FUNC(cr_query_func, cr, param); } -static inline DAT_RETURN dat_cr_reject(DAT_CR_HANDLE cr) +static inline u32 dat_cr_reject(DAT_CR_HANDLE cr) { return DAT_CALL_PROVIDER_FUNC(cr_reject_func, cr); } -static inline DAT_RETURN dat_evd_dequeue(DAT_EVD_HANDLE evd, +static inline u32 dat_evd_dequeue(DAT_EVD_HANDLE evd, struct dat_event *event) { return DAT_CALL_PROVIDER_FUNC(evd_dequeue_func, evd, event); } -static inline DAT_RETURN dat_evd_free(DAT_EVD_HANDLE evd) +static inline u32 dat_evd_free(DAT_EVD_HANDLE evd) { return DAT_CALL_PROVIDER_FUNC(evd_free_func, evd); } -static inline DAT_RETURN dat_evd_kcreate(DAT_IA_HANDLE ia, +static inline u32 dat_evd_kcreate(DAT_IA_HANDLE ia, DAT_COUNT qlen, DAT_UPCALL_POLICY policy, const struct dat_upcall_object *upcall, @@ -1494,7 +1492,7 @@ static inline DAT_RETURN dat_evd_kcreate evd_kcreate_func, ia, qlen, policy, upcall, flags, evd); } -static inline DAT_RETURN dat_evd_modify_upcall(DAT_EVD_HANDLE evd, +static inline u32 dat_evd_modify_upcall(DAT_EVD_HANDLE evd, DAT_UPCALL_POLICY policy, const struct dat_upcall_object *upcall) { @@ -1502,24 +1500,24 @@ static inline DAT_RETURN dat_evd_modify_ evd_modify_upcall_func, evd, policy, upcall); } -static inline DAT_RETURN dat_evd_post_se(DAT_EVD_HANDLE evd, +static inline u32 dat_evd_post_se(DAT_EVD_HANDLE evd, const struct dat_event *event) { return DAT_CALL_PROVIDER_FUNC(evd_post_se_func, evd, event); } -static inline DAT_RETURN dat_evd_query(DAT_EVD_HANDLE evd, +static inline u32 dat_evd_query(DAT_EVD_HANDLE evd, struct dat_evd_param *param) { return DAT_CALL_PROVIDER_FUNC(evd_query_func, evd, param); } -static inline DAT_RETURN dat_evd_resize(DAT_EVD_HANDLE evd, DAT_COUNT qlen) +static inline u32 dat_evd_resize(DAT_EVD_HANDLE evd, DAT_COUNT qlen) { return DAT_CALL_PROVIDER_FUNC(evd_resize_func, evd, qlen); } -static inline DAT_RETURN dat_ep_connect(DAT_EP_HANDLE ep, +static inline u32 dat_ep_connect(DAT_EP_HANDLE ep, DAT_IA_ADDRESS_PTR ia_addr, DAT_CONN_QUAL conn_qual, DAT_TIMEOUT timeout, @@ -1533,7 +1531,7 @@ static inline DAT_RETURN dat_ep_connect( private_data_size, private_data, qos, flags); } -static inline DAT_RETURN dat_ep_create(DAT_IA_HANDLE ia, +static inline u32 dat_ep_create(DAT_IA_HANDLE ia, DAT_PZ_HANDLE pz, DAT_EVD_HANDLE in_evd, DAT_EVD_HANDLE out_evd, @@ -1546,7 +1544,7 @@ static inline DAT_RETURN dat_ep_create(D } -static inline DAT_RETURN dat_ep_create_with_srq(DAT_IA_HANDLE ia, +static inline u32 dat_ep_create_with_srq(DAT_IA_HANDLE ia, DAT_PZ_HANDLE pz, DAT_EVD_HANDLE in_evd, DAT_EVD_HANDLE out_evd, @@ -1559,13 +1557,13 @@ static inline DAT_RETURN dat_ep_create_w out_evd, connect_evd, srq, attr, ep); } -static inline DAT_RETURN dat_ep_disconnect(DAT_EP_HANDLE ep, +static inline u32 dat_ep_disconnect(DAT_EP_HANDLE ep, DAT_CLOSE_FLAGS flags) { return DAT_CALL_PROVIDER_FUNC(ep_disconnect_func, ep, flags); } -static inline DAT_RETURN dat_ep_dup_connect(DAT_EP_HANDLE ep, +static inline u32 dat_ep_dup_connect(DAT_EP_HANDLE ep, DAT_EP_HANDLE dup_ep, DAT_TIMEOUT timeout, DAT_COUNT private_data_size, @@ -1576,12 +1574,12 @@ static inline DAT_RETURN dat_ep_dup_conn private_data_size, private_data, qos); } -static inline DAT_RETURN dat_ep_free(DAT_EP_HANDLE ep) +static inline u32 dat_ep_free(DAT_EP_HANDLE ep) { return DAT_CALL_PROVIDER_FUNC(ep_free_func, ep); } -static inline DAT_RETURN dat_ep_get_status(DAT_EP_HANDLE ep, +static inline u32 dat_ep_get_status(DAT_EP_HANDLE ep, DAT_EP_STATE * state, boolean_t *recv_idle, boolean_t *req_idle) @@ -1590,14 +1588,14 @@ static inline DAT_RETURN dat_ep_get_stat ep_get_status_func, ep, state, recv_idle, req_idle); } -static inline DAT_RETURN dat_ep_modify(DAT_EP_HANDLE ep, +static inline u32 dat_ep_modify(DAT_EP_HANDLE ep, DAT_EP_PARAM_MASK mask, const struct dat_ep_param *param) { return DAT_CALL_PROVIDER_FUNC(ep_modify_func, ep, mask, param); } -static inline DAT_RETURN dat_ep_post_rdma_read(DAT_EP_HANDLE ep, +static inline u32 dat_ep_post_rdma_read(DAT_EP_HANDLE ep, DAT_COUNT size, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, @@ -1608,7 +1606,7 @@ static inline DAT_RETURN dat_ep_post_rdm local_iov, cookie, remote_iov, flags); } -static inline DAT_RETURN dat_ep_post_rdma_write(DAT_EP_HANDLE ep, +static inline u32 dat_ep_post_rdma_write(DAT_EP_HANDLE ep, DAT_COUNT size, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, @@ -1619,7 +1617,7 @@ static inline DAT_RETURN dat_ep_post_rdm local_iov, cookie, remote_iov, flags); } -static inline DAT_RETURN dat_ep_post_recv(DAT_EP_HANDLE ep, +static inline u32 dat_ep_post_recv(DAT_EP_HANDLE ep, DAT_COUNT size, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, @@ -1629,7 +1627,7 @@ static inline DAT_RETURN dat_ep_post_rec ep_post_recv_func, ep, size, local_iov, cookie, flags); } -static inline DAT_RETURN dat_ep_post_send(DAT_EP_HANDLE ep, +static inline u32 dat_ep_post_send(DAT_EP_HANDLE ep, DAT_COUNT size, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, @@ -1639,12 +1637,12 @@ static inline DAT_RETURN dat_ep_post_sen ep_post_send_func, ep, size, local_iov, cookie, flags); } -static inline DAT_RETURN dat_ep_query(DAT_EP_HANDLE ep, struct dat_ep_param *param) +static inline u32 dat_ep_query(DAT_EP_HANDLE ep, struct dat_ep_param *param) { return DAT_CALL_PROVIDER_FUNC(ep_query_func, ep, param); } -static inline DAT_RETURN dat_ep_recv_query(DAT_EP_HANDLE ep, +static inline u32 dat_ep_recv_query(DAT_EP_HANDLE ep, DAT_COUNT * bufs_alloc, DAT_COUNT * bufs_avail) { @@ -1652,12 +1650,12 @@ static inline DAT_RETURN dat_ep_recv_que ep_recv_query_func, ep, bufs_alloc, bufs_avail); } -static inline DAT_RETURN dat_ep_reset(DAT_EP_HANDLE ep) +static inline u32 dat_ep_reset(DAT_EP_HANDLE ep) { return DAT_CALL_PROVIDER_FUNC(ep_reset_func, ep); } -static inline DAT_RETURN dat_ep_set_watermark(DAT_EP_HANDLE ep, +static inline u32 dat_ep_set_watermark(DAT_EP_HANDLE ep, DAT_COUNT soft_high_watermark, DAT_COUNT hard_high_watermark) { @@ -1665,18 +1663,18 @@ static inline DAT_RETURN dat_ep_set_wate soft_high_watermark, hard_high_watermark); } -static inline DAT_RETURN dat_lmr_query(DAT_LMR_HANDLE lmr, +static inline u32 dat_lmr_query(DAT_LMR_HANDLE lmr, struct dat_lmr_param *param) { return DAT_CALL_PROVIDER_FUNC(lmr_query_func, lmr, param); } -static inline DAT_RETURN dat_lmr_free(DAT_LMR_HANDLE lmr) +static inline u32 dat_lmr_free(DAT_LMR_HANDLE lmr) { return DAT_CALL_PROVIDER_FUNC(lmr_free_func, lmr); } -static inline DAT_RETURN dat_lmr_sync_rdma_read(DAT_IA_HANDLE ia, +static inline u32 dat_lmr_sync_rdma_read(DAT_IA_HANDLE ia, const struct dat_lmr_triplet *iovs, DAT_VLEN num_iovs) { @@ -1684,7 +1682,7 @@ static inline DAT_RETURN dat_lmr_sync_rd lmr_sync_rdma_read_func, ia, iovs, num_iovs); } -static inline DAT_RETURN dat_lmr_sync_rdma_write(DAT_IA_HANDLE ia, +static inline u32 dat_lmr_sync_rdma_write(DAT_IA_HANDLE ia, const struct dat_lmr_triplet *iovs, DAT_VLEN num_iovs) { @@ -1692,17 +1690,17 @@ static inline DAT_RETURN dat_lmr_sync_rd lmr_sync_rdma_write_func, ia, iovs, num_iovs); } -static inline DAT_RETURN dat_rmr_create(DAT_PZ_HANDLE pz, DAT_RMR_HANDLE * rmr) +static inline u32 dat_rmr_create(DAT_PZ_HANDLE pz, DAT_RMR_HANDLE * rmr) { return DAT_CALL_PROVIDER_FUNC(rmr_create_func, pz, rmr); } -static inline DAT_RETURN dat_rmr_query(DAT_RMR_HANDLE rmr, struct dat_rmr_param *param) +static inline u32 dat_rmr_query(DAT_RMR_HANDLE rmr, struct dat_rmr_param *param) { return DAT_CALL_PROVIDER_FUNC(rmr_query_func, rmr, param); } -static inline DAT_RETURN dat_rmr_bind(DAT_RMR_HANDLE rmr, +static inline u32 dat_rmr_bind(DAT_RMR_HANDLE rmr, const struct dat_lmr_triplet *iov, DAT_MEM_PRIV_FLAGS mem_flags, DAT_EP_HANDLE ep, @@ -1714,12 +1712,12 @@ static inline DAT_RETURN dat_rmr_bind(DA cookie, comp_flags, context); } -static inline DAT_RETURN dat_rmr_free(DAT_RMR_HANDLE rmr) +static inline u32 dat_rmr_free(DAT_RMR_HANDLE rmr) { return DAT_CALL_PROVIDER_FUNC(rmr_free_func, rmr); } -static inline DAT_RETURN dat_psp_create(DAT_IA_HANDLE ia, +static inline u32 dat_psp_create(DAT_IA_HANDLE ia, DAT_CONN_QUAL conn_qual, DAT_EVD_HANDLE evd, DAT_PSP_FLAGS flags, @@ -1729,7 +1727,7 @@ static inline DAT_RETURN dat_psp_create( psp_create_func, ia, conn_qual, evd, flags, psp); } -static inline DAT_RETURN dat_psp_create_any(DAT_IA_HANDLE ia, +static inline u32 dat_psp_create_any(DAT_IA_HANDLE ia, DAT_CONN_QUAL * conn_qual, DAT_EVD_HANDLE evd, DAT_PSP_FLAGS flags, @@ -1739,18 +1737,18 @@ static inline DAT_RETURN dat_psp_create_ psp_create_any_func, ia, conn_qual, evd, flags, psp); } -static inline DAT_RETURN dat_psp_query(DAT_PSP_HANDLE psp, +static inline u32 dat_psp_query(DAT_PSP_HANDLE psp, struct dat_psp_param *param) { return DAT_CALL_PROVIDER_FUNC(psp_query_func, psp, param); } -static inline DAT_RETURN dat_psp_free(DAT_PSP_HANDLE psp) +static inline u32 dat_psp_free(DAT_PSP_HANDLE psp) { return DAT_CALL_PROVIDER_FUNC(psp_free_func, psp); } -static inline DAT_RETURN dat_rsp_create(DAT_IA_HANDLE ia, +static inline u32 dat_rsp_create(DAT_IA_HANDLE ia, DAT_CONN_QUAL conn_qual, DAT_EP_HANDLE ep, DAT_EVD_HANDLE evd, @@ -1760,34 +1758,34 @@ static inline DAT_RETURN dat_rsp_create( rsp_create_func, ia, conn_qual, ep, evd, rsp); } -static inline DAT_RETURN dat_rsp_query(DAT_RSP_HANDLE rsp, +static inline u32 dat_rsp_query(DAT_RSP_HANDLE rsp, struct dat_rsp_param *param) { return DAT_CALL_PROVIDER_FUNC(rsp_query_func, rsp, param); } -static inline DAT_RETURN dat_rsp_free(DAT_RSP_HANDLE rsp) +static inline u32 dat_rsp_free(DAT_RSP_HANDLE rsp) { return DAT_CALL_PROVIDER_FUNC(rsp_free_func, rsp); } -static inline DAT_RETURN dat_pz_create(DAT_IA_HANDLE ia, DAT_PZ_HANDLE * pz) +static inline u32 dat_pz_create(DAT_IA_HANDLE ia, DAT_PZ_HANDLE * pz) { return DAT_CALL_PROVIDER_FUNC(pz_create_func, ia, pz); } -static inline DAT_RETURN dat_pz_query(DAT_PZ_HANDLE pz, +static inline u32 dat_pz_query(DAT_PZ_HANDLE pz, struct dat_pz_param *param) { return DAT_CALL_PROVIDER_FUNC(pz_query_func, pz, param); } -static inline DAT_RETURN dat_pz_free(DAT_PZ_HANDLE pz) +static inline u32 dat_pz_free(DAT_PZ_HANDLE pz) { return DAT_CALL_PROVIDER_FUNC(pz_free_func, pz); } -static inline DAT_RETURN dat_srq_create(DAT_IA_HANDLE ia, +static inline u32 dat_srq_create(DAT_IA_HANDLE ia, DAT_PZ_HANDLE pz, struct dat_srq_attr *attr, DAT_SRQ_HANDLE * srq) @@ -1795,12 +1793,12 @@ static inline DAT_RETURN dat_srq_create( return DAT_CALL_PROVIDER_FUNC(srq_create_func, ia, pz, attr, srq); } -static inline DAT_RETURN dat_srq_free(DAT_SRQ_HANDLE srq) +static inline u32 dat_srq_free(DAT_SRQ_HANDLE srq) { return DAT_CALL_PROVIDER_FUNC(srq_free_func, srq); } -static inline DAT_RETURN dat_srq_post_recv(DAT_SRQ_HANDLE srq, +static inline u32 dat_srq_post_recv(DAT_SRQ_HANDLE srq, DAT_COUNT num_iovs, struct dat_lmr_triplet *iovs, DAT_DTO_COOKIE cookie) @@ -1809,25 +1807,25 @@ static inline DAT_RETURN dat_srq_post_re srq_post_recv_func, srq, num_iovs, iovs, cookie); } -static inline DAT_RETURN dat_srq_query(DAT_SRQ_HANDLE srq, +static inline u32 dat_srq_query(DAT_SRQ_HANDLE srq, struct dat_srq_param *param) { return DAT_CALL_PROVIDER_FUNC(srq_query_func, srq, param); } -static inline DAT_RETURN dat_srq_resize(DAT_SRQ_HANDLE srq, +static inline u32 dat_srq_resize(DAT_SRQ_HANDLE srq, DAT_COUNT max_recv_dtos) { return DAT_CALL_PROVIDER_FUNC(srq_resize_func, srq, max_recv_dtos); } -static inline DAT_RETURN dat_srq_set_lw(DAT_SRQ_HANDLE srq, +static inline u32 dat_srq_set_lw(DAT_SRQ_HANDLE srq, DAT_COUNT low_watermark) { return DAT_CALL_PROVIDER_FUNC(srq_set_lw_func, srq, low_watermark); } -static inline DAT_RETURN dat_lmr_kcreate(DAT_IA_HANDLE ia, +static inline u32 dat_lmr_kcreate(DAT_IA_HANDLE ia, DAT_MEM_TYPE type, DAT_REGION_DESCRIPTION region, DAT_VLEN len, diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat/dictionary.c linux-kernel2/dat/dictionary.c --- linux-kernel/dat/dictionary.c 2005-05-12 07:46:08.833024000 -0700 +++ linux-kernel2/dat/dictionary.c 2005-05-12 07:51:24.928004000 -0700 @@ -39,9 +39,8 @@ * * *********************************************************************/ -static DAT_RETURN -dat_dictionary_key_dup(const struct dat_provider_info *old_key, - struct dat_provider_info *new_key); +static u32 dat_dictionary_key_dup(const struct dat_provider_info *old_key, + struct dat_provider_info *new_key); static boolean_t dat_dictionary_key_is_equal(const struct dat_provider_info *key_a, @@ -78,8 +77,8 @@ void dat_dictionary_destroy(struct dat_d * Function: dat_dictionary_size ***********************************************************************/ -DAT_RETURN dat_dictionary_size(const struct dat_dictionary *dictionary, - DAT_COUNT *size) +u32 dat_dictionary_size(const struct dat_dictionary *dictionary, + DAT_COUNT *size) { BUG_ON(NULL == dictionary); BUG_ON(NULL == size); @@ -93,15 +92,13 @@ DAT_RETURN dat_dictionary_size(const str * Function: dat_dictionary_entry_create ***********************************************************************/ -DAT_RETURN dat_dictionary_entry_create(struct dat_dictionary_entry **p_entry) +u32 dat_dictionary_entry_create(struct dat_dictionary_entry **p_entry) { struct dat_dictionary_entry *entry; - DAT_RETURN status; + u32 status = DAT_SUCCESS; BUG_ON(NULL == p_entry); - status = DAT_SUCCESS; - entry = kmalloc(sizeof(*entry), GFP_ATOMIC); if (NULL == entry) { status = @@ -128,12 +125,11 @@ void dat_dictionary_entry_destroy(const * Function: dat_dictionary_insert ***********************************************************************/ -DAT_RETURN dat_dictionary_insert(struct dat_dictionary *dictionary, - const struct dat_provider_info *key, - struct dat_dictionary_entry *entry, - void *data) +u32 dat_dictionary_insert(struct dat_dictionary *dictionary, + const struct dat_provider_info *key, + struct dat_dictionary_entry *entry, void *data) { - DAT_RETURN status; + u32 status; BUG_ON(NULL == dictionary); BUG_ON(NULL == entry); @@ -163,13 +159,12 @@ DAT_RETURN dat_dictionary_insert(struct * Function: dat_dictionary_search ***********************************************************************/ -DAT_RETURN dat_dictionary_search(struct dat_dictionary *dictionary, - const struct dat_provider_info *key, - void **p_data) +u32 dat_dictionary_search(struct dat_dictionary *dictionary, + const struct dat_provider_info *key, void **p_data) { struct list_head *cur_list; struct dat_dictionary_entry *cur_entry; - DAT_RETURN status; + u32 status; BUG_ON(NULL == dictionary); BUG_ON(NULL == key); @@ -196,14 +191,13 @@ DAT_RETURN dat_dictionary_search(struct * Function: dat_dictionary_enumerate ***********************************************************************/ -DAT_RETURN dat_dictionary_enumerate(struct dat_dictionary *dictionary, - void *array[], - DAT_COUNT array_size) +u32 dat_dictionary_enumerate(struct dat_dictionary *dictionary, void *array[], + DAT_COUNT array_size) { struct list_head *cur_list; struct dat_dictionary_entry *cur_entry; DAT_COUNT i; - DAT_RETURN status; + u32 status; BUG_ON(NULL == dictionary); BUG_ON(NULL == array); @@ -230,14 +224,13 @@ DAT_RETURN dat_dictionary_enumerate(stru * Function: dat_dictionary_remove ***********************************************************************/ -DAT_RETURN dat_dictionary_remove(struct dat_dictionary *dictionary, - const struct dat_provider_info *key, - struct dat_dictionary_entry **p_entry, - void **p_data) +u32 dat_dictionary_remove(struct dat_dictionary *dictionary, + const struct dat_provider_info *key, + struct dat_dictionary_entry **p_entry, void **p_data) { struct list_head *cur_list, *next_list; struct dat_dictionary_entry *cur_entry; - DAT_RETURN status; + u32 status; BUG_ON(NULL == dictionary); BUG_ON(NULL == key); @@ -277,8 +270,8 @@ DAT_RETURN dat_dictionary_remove(struct * Function: dat_dictionary_key_create ***********************************************************************/ -DAT_RETURN dat_dictionary_key_dup(const struct dat_provider_info *old_key, - struct dat_provider_info *new_key) +u32 dat_dictionary_key_dup(const struct dat_provider_info *old_key, + struct dat_provider_info *new_key) { BUG_ON(NULL == old_key); BUG_ON(NULL == new_key); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat/dictionary.h linux-kernel2/dat/dictionary.h --- linux-kernel/dat/dictionary.h 2005-05-12 07:46:08.907009000 -0700 +++ linux-kernel2/dat/dictionary.h 2005-05-12 07:53:32.355003000 -0700 @@ -69,31 +69,30 @@ struct dat_dictionary { extern void dat_dictionary_destroy(struct dat_dictionary *dictionary); -extern DAT_RETURN dat_dictionary_size( - const struct dat_dictionary *dictionary, DAT_COUNT *size); +extern u32 dat_dictionary_size(const struct dat_dictionary *dictionary, + DAT_COUNT *size); -extern DAT_RETURN dat_dictionary_entry_create( - struct dat_dictionary_entry **p_entry); +extern u32 dat_dictionary_entry_create(struct dat_dictionary_entry **p_entry); extern void dat_dictionary_entry_destroy( const struct dat_dictionary_entry *entry); -extern DAT_RETURN dat_dictionary_insert(struct dat_dictionary *dictionary, - const struct dat_provider_info *key, - struct dat_dictionary_entry *entry, - void *data); - -extern DAT_RETURN dat_dictionary_search(struct dat_dictionary *dictionary, - const struct dat_provider_info *key, - void **p_data); - -extern DAT_RETURN dat_dictionary_enumerate(struct dat_dictionary *dictionary, - void *array[], - const DAT_COUNT array_size); - -extern DAT_RETURN dat_dictionary_remove(struct dat_dictionary *dictionary, - const struct dat_provider_info *key, - struct dat_dictionary_entry **p_entry, - void **p_data); +extern u32 dat_dictionary_insert(struct dat_dictionary *dictionary, + const struct dat_provider_info *key, + struct dat_dictionary_entry *entry, + void *data); + +extern u32 dat_dictionary_search(struct dat_dictionary *dictionary, + const struct dat_provider_info *key, + void **p_data); + +extern u32 dat_dictionary_enumerate(struct dat_dictionary *dictionary, + void *array[], + const DAT_COUNT array_size); + +extern u32 dat_dictionary_remove(struct dat_dictionary *dictionary, + const struct dat_provider_info *key, + struct dat_dictionary_entry **p_entry, + void **p_data); #endif /* DICTIONARY_H */ diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat/dr.c linux-kernel2/dat/dr.c --- linux-kernel/dat/dr.c 2005-05-12 07:46:08.786003000 -0700 +++ linux-kernel2/dat/dr.c 2005-05-12 07:49:02.844000000 -0700 @@ -76,12 +76,12 @@ void dat_dr_fini(void) * Function: dat_dr_insert ************************************************************************/ -DAT_RETURN dat_dr_insert(const struct dat_provider_info *info, - const DAT_IA_OPEN_FUNC ia_open_func) +u32 dat_dr_insert(const struct dat_provider_info *info, + const DAT_IA_OPEN_FUNC ia_open_func) { struct dat_dr_entry *data; struct dat_dictionary_entry *dict_entry; - DAT_RETURN status; + u32 status; data = kmalloc(sizeof(*data), GFP_ATOMIC); if (NULL == data) { @@ -122,11 +122,11 @@ DAT_RETURN dat_dr_insert(const struct da * Function: dat_dr_remove ************************************************************************/ -DAT_RETURN dat_dr_remove(const struct dat_provider_info *info) +u32 dat_dr_remove(const struct dat_provider_info *info) { struct dat_dr_entry *data; struct dat_dictionary_entry *dict_entry; - DAT_RETURN status; + u32 status; dict_entry = NULL; spin_lock(&g_dr_lock); @@ -161,11 +161,11 @@ DAT_RETURN dat_dr_remove(const struct da * Function: dat_dr_provider_open ************************************************************************/ -DAT_RETURN dat_dr_provider_open(const struct dat_provider_info *info, +u32 dat_dr_provider_open(const struct dat_provider_info *info, DAT_IA_OPEN_FUNC *p_ia_open_func) { struct dat_dr_entry *data; - DAT_RETURN status; + u32 status; spin_lock(&g_dr_lock); @@ -185,10 +185,10 @@ DAT_RETURN dat_dr_provider_open(const st * Function: dat_dr_provider_close ************************************************************************/ -DAT_RETURN dat_dr_provider_close(const struct dat_provider_info *info) +u32 dat_dr_provider_close(const struct dat_provider_info *info) { struct dat_dr_entry *data; - DAT_RETURN status; + u32 status; spin_lock(&g_dr_lock); @@ -206,7 +206,7 @@ DAT_RETURN dat_dr_provider_close(const s * Function: dat_dr_size ************************************************************************/ -DAT_RETURN dat_dr_size(DAT_COUNT *size) +u32 dat_dr_size(DAT_COUNT *size) { return dat_dictionary_size(&g_dr_dictionary, size); } @@ -215,17 +215,16 @@ DAT_RETURN dat_dr_size(DAT_COUNT *size) * Function: dat_dr_list ***********************************************************************/ -DAT_RETURN dat_dr_list(DAT_COUNT max_to_return, - DAT_COUNT *entries_returned, - struct dat_provider_info *(dat_provider_list[])) +u32 dat_dr_list(DAT_COUNT max_to_return, + DAT_COUNT *entries_returned, + struct dat_provider_info *(dat_provider_list[])) { struct dat_dr_entry **array; DAT_COUNT array_size; DAT_COUNT i; - DAT_RETURN status; + u32 status = DAT_SUCCESS; array = NULL; - status = DAT_SUCCESS; /* The dictionary size may increase between the call to */ /* dat_dictionary_size() and dat_dictionary_enumerate(). */ diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat/dr.h linux-kernel2/dat/dr.h --- linux-kernel/dat/dr.h 2005-05-12 07:46:08.935003000 -0700 +++ linux-kernel2/dat/dr.h 2005-05-12 07:54:14.114000000 -0700 @@ -44,20 +44,20 @@ extern void dat_dr_fini(void); -extern DAT_RETURN dat_dr_insert(const struct dat_provider_info *info, - const DAT_IA_OPEN_FUNC ia_open_func); +extern u32 dat_dr_insert(const struct dat_provider_info *info, + const DAT_IA_OPEN_FUNC ia_open_func); -extern DAT_RETURN dat_dr_remove(const struct dat_provider_info *info); +extern u32 dat_dr_remove(const struct dat_provider_info *info); -extern DAT_RETURN dat_dr_provider_open(const struct dat_provider_info *info, - DAT_IA_OPEN_FUNC *p_ia_open_func); +extern u32 dat_dr_provider_open(const struct dat_provider_info *info, + DAT_IA_OPEN_FUNC *p_ia_open_func); -extern DAT_RETURN dat_dr_provider_close(const struct dat_provider_info *info); +extern u32 dat_dr_provider_close(const struct dat_provider_info *info); -extern DAT_RETURN dat_dr_size(DAT_COUNT *size); +extern u32 dat_dr_size(DAT_COUNT *size); -extern DAT_RETURN dat_dr_list(DAT_COUNT max_to_return, - DAT_COUNT *entries_returned, - struct dat_provider_info *(dat_provider_list[])); +extern u32 dat_dr_list(DAT_COUNT max_to_return, + DAT_COUNT *entries_returned, + struct dat_provider_info *(dat_provider_list[])); #endif /* DR_H */ diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_adapter_util.h linux-kernel2/dat-provider/dapl_adapter_util.h --- linux-kernel/dat-provider/dapl_adapter_util.h 2005-05-11 10:09:59.053024000 -0700 +++ linux-kernel2/dat-provider/dapl_adapter_util.h 2005-05-11 11:37:05.389975000 -0700 @@ -49,112 +49,93 @@ typedef enum async_handler_type { */ #define IB_CM_REJ_REASON_CONSUMER_REJ 0x001C -DAT_RETURN dapl_ib_enum_hcas(const char *vendor, - DAPL_HCA_NAME ** hca_names, - DAT_COUNT * total_hca_count); +u32 dapl_ib_enum_hcas(const char *vendor, + DAPL_HCA_NAME **hca_names, + DAT_COUNT *total_hca_count); -DAT_RETURN dapl_ib_get_instance_data(DAPL_HCA_NAME hca_name, char *instance); +u32 dapl_ib_get_instance_data(DAPL_HCA_NAME hca_name, char *instance); -DAT_RETURN dapl_ib_open_hca(char *namestr, DAPL_HCA * hca_ptr); +u32 dapl_ib_open_hca(char *namestr, DAPL_HCA *hca_ptr); -DAT_RETURN dapl_ib_close_hca(DAPL_HCA * hca_ptr); +u32 dapl_ib_close_hca(DAPL_HCA *hca_ptr); -DAT_RETURN dapl_ib_qp_alloc(DAPL_IA * ia_ptr, - DAPL_EP * ep_ptr, DAPL_EP * ep_ctx_ptr); +u32 dapl_ib_qp_alloc(DAPL_IA *ia_ptr, DAPL_EP *ep_ptr, DAPL_EP *ep_ctx_ptr); -DAT_RETURN dapl_ib_qp_free(DAPL_IA * ia_ptr, DAPL_EP * ep_ptr); +u32 dapl_ib_qp_free(DAPL_IA *ia_ptr, DAPL_EP *ep_ptr); -DAT_RETURN dapl_ib_qp_modify(DAPL_IA * ia_ptr, - DAPL_EP * ep_ptr, struct dat_ep_attr *ep_attr); +u32 dapl_ib_qp_modify(DAPL_IA *ia_ptr, DAPL_EP *ep_ptr, + struct dat_ep_attr *ep_attr); -DAT_RETURN dapl_ib_connect(DAT_EP_HANDLE ep_handle, - DAT_IA_ADDRESS_PTR remote_ia_address, - DAT_CONN_QUAL remote_conn_qual, - DAT_COUNT private_data_size, DAT_PVOID private_data); +u32 dapl_ib_connect(DAT_EP_HANDLE ep_handle, + DAT_IA_ADDRESS_PTR remote_ia_address, + DAT_CONN_QUAL remote_conn_qual, + DAT_COUNT private_data_size, DAT_PVOID private_data); -DAT_RETURN dapl_ib_disconnect(DAPL_EP * ep_ptr, DAT_CLOSE_FLAGS close_flags); +u32 dapl_ib_disconnect(DAPL_EP *ep_ptr, DAT_CLOSE_FLAGS close_flags); -DAT_RETURN dapl_ib_setup_conn_listener(DAPL_IA * ia_ptr, - u64 ServiceID, DAPL_SP * sp_ptr); +u32 dapl_ib_setup_conn_listener(DAPL_IA *ia_ptr, u64 ServiceID, + DAPL_SP *sp_ptr); -DAT_RETURN dapl_ib_remove_conn_listener(DAPL_IA * ia_ptr, DAPL_SP * sp_ptr); +u32 dapl_ib_remove_conn_listener(DAPL_IA *ia_ptr, DAPL_SP *sp_ptr); -DAT_RETURN dapl_ib_accept_connection(DAT_CR_HANDLE cr_handle, - DAT_EP_HANDLE ep_handle, - DAT_COUNT private_data_size, - const DAT_PVOID private_data); +u32 dapl_ib_accept_connection(DAT_CR_HANDLE cr_handle, + DAT_EP_HANDLE ep_handle, + DAT_COUNT private_data_size, + const DAT_PVOID private_data); -DAT_RETURN dapl_ib_reject_connection(ib_cm_handle_t cm_handle, - int reject_reason); +u32 dapl_ib_reject_connection(ib_cm_handle_t cm_handle, int reject_reason); -DAT_RETURN dapl_ib_setup_async_callback(DAPL_IA * ia_ptr, - DAPL_ASYNC_HANDLER_TYPE handler_type, - ib_async_handler_t callback, - void *context); +u32 dapl_ib_setup_async_callback(DAPL_IA *ia_ptr, + DAPL_ASYNC_HANDLER_TYPE handler_type, + ib_async_handler_t callback, void *context); -DAT_RETURN dapl_ib_cq_alloc(DAPL_IA * ia_ptr, - DAPL_EVD * evd_ptr, - DAT_COUNT * cqlen, ib_comp_handle_t callback); +u32 dapl_ib_cq_alloc(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr, DAT_COUNT *cqlen, + ib_comp_handle_t callback); -DAT_RETURN dapl_ib_cq_free(DAPL_IA * ia_ptr, DAPL_EVD * evd_ptr); +u32 dapl_ib_cq_free(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr); -DAT_RETURN dapl_set_cq_notify(DAPL_IA * ia_ptr, DAPL_EVD * evd_ptr); +u32 dapl_set_cq_notify(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr); -DAT_RETURN dapl_ib_cq_resize(DAPL_IA * ia_ptr, - DAPL_EVD * evd_ptr, DAT_COUNT * cqlen); +u32 dapl_ib_cq_resize(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr, DAT_COUNT *cqlen); -DAT_RETURN dapl_ib_pd_alloc(DAPL_IA * ia_ptr, DAPL_PZ * pz); +u32 dapl_ib_pd_alloc(DAPL_IA *ia_ptr, DAPL_PZ *pz); -DAT_RETURN dapl_ib_pd_free(DAPL_PZ * pz); +u32 dapl_ib_pd_free(DAPL_PZ *pz); -DAT_RETURN dapl_ib_mr_register(DAPL_IA * ia_ptr, - DAPL_LMR * lmr, - DAT_PVOID virt_addr, - DAT_VLEN length, DAT_MEM_PRIV_FLAGS privileges); +u32 dapl_ib_mr_register(DAPL_IA *ia_ptr, DAPL_LMR *lmr, DAT_PVOID virt_addr, + DAT_VLEN length, DAT_MEM_PRIV_FLAGS privileges); -DAT_RETURN dapl_ib_mr_register_physical(DAPL_IA * ia_ptr, - DAPL_LMR * lmr, - DAT_PVOID phys_addr, - DAT_VLEN length, - DAT_MEM_PRIV_FLAGS privileges); +u32 dapl_ib_mr_register_physical(DAPL_IA *ia_ptr, DAPL_LMR *lmr, + DAT_PVOID phys_addr, DAT_VLEN length, + DAT_MEM_PRIV_FLAGS privileges); -DAT_RETURN dapl_ib_mr_deregister(DAPL_LMR * lmr); +u32 dapl_ib_mr_deregister(DAPL_LMR *lmr); -DAT_RETURN dapl_ib_mr_register_shared(DAPL_IA * ia_ptr, - DAPL_LMR * lmr, - DAT_MEM_PRIV_FLAGS privileges); +u32 dapl_ib_mr_register_shared(DAPL_IA *ia_ptr, DAPL_LMR *lmr, + DAT_MEM_PRIV_FLAGS privileges); -DAT_RETURN dapl_ib_mw_alloc(DAPL_RMR * rmr); +u32 dapl_ib_mw_alloc(DAPL_RMR *rmr); -DAT_RETURN dapl_ib_mw_free(DAPL_RMR * rmr); +u32 dapl_ib_mw_free(DAPL_RMR *rmr); -DAT_RETURN dapl_ib_mw_bind(DAPL_RMR * rmr, - DAPL_LMR * lmr, - DAPL_EP * ep, - DAPL_COOKIE * cookie, - DAT_VADDR virtual_address, - DAT_VLEN length, - DAT_MEM_PRIV_FLAGS mem_priv, - boolean_t is_signaled); +u32 dapl_ib_mw_bind(DAPL_RMR *rmr, DAPL_LMR *lmr, DAPL_EP *ep, + DAPL_COOKIE *cookie, DAT_VADDR virtual_address, + DAT_VLEN length, DAT_MEM_PRIV_FLAGS mem_priv, + boolean_t is_signaled); -DAT_RETURN dapl_ib_mw_unbind(DAPL_RMR * rmr, - DAPL_EP * ep, - DAPL_COOKIE * cookie, boolean_t is_signaled); +u32 dapl_ib_mw_unbind(DAPL_RMR *rmr, DAPL_EP *ep, DAPL_COOKIE *cookie, + boolean_t is_signaled); -DAT_RETURN dapl_ib_query_hca(DAPL_HCA * hca_ptr, - struct dat_ia_attr *ia_attr, - struct dat_ep_attr *ep_attr, - struct sockaddr_in6 *ip_addr); +u32 dapl_ib_query_hca(DAPL_HCA *hca_ptr, struct dat_ia_attr *ia_attr, + struct dat_ep_attr *ep_attr, + struct sockaddr_in6 *ip_addr); -DAT_RETURN dapl_ib_completion_poll(DAPL_HCA * hca_ptr, - DAPL_EVD * evd_ptr, - ib_work_completion_t * cqe_ptr); +u32 dapl_ib_completion_poll(DAPL_HCA *hca_ptr, DAPL_EVD *evd_ptr, + ib_work_completion_t *cqe_ptr); -DAT_RETURN dapl_ib_completion_notify(DAPL_EVD * evd_ptr, - ib_notification_type_t type); +u32 dapl_ib_completion_notify(DAPL_EVD *evd_ptr, ib_notification_type_t type); -DAT_DTO_COMPLETION_STATUS dapl_ib_get_dto_status(ib_work_completion_t * - cqe_ptr); +DAT_DTO_COMPLETION_STATUS dapl_ib_get_dto_status(ib_work_completion_t *cqe_ptr); void dapl_ib_reinit_ep(DAPL_EP * ep_ptr); @@ -162,17 +143,16 @@ void dapl_ib_disconnect_clean(DAPL_EP * boolean_t passive, const ib_cm_events_t ib_cm_event); -DAT_RETURN dapl_ib_get_async_event(ib_error_record_t * cause_ptr, - DAT_EVENT_NUMBER * async_event); +u32 dapl_ib_get_async_event(ib_error_record_t *cause_ptr, + DAT_EVENT_NUMBER *async_event); DAT_EVENT_NUMBER dapl_ib_get_dat_event(const ib_cm_events_t ib_cm_event, boolean_t active); ib_cm_events_t dapl_ib_get_cm_event(DAT_EVENT_NUMBER dat_event_num); -DAT_RETURN dapl_ib_cm_remote_addr(DAT_HANDLE dat_handle, - struct sockaddr_in6 *remote_ia_address); - +u32 dapl_ib_cm_remote_addr(DAT_HANDLE dat_handle, + struct sockaddr_in6 *remote_ia_address); /* * Values for provider struct dat_named_attr diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_cno_util.c linux-kernel2/dat-provider/dapl_cno_util.c --- linux-kernel/dat-provider/dapl_cno_util.c 2005-05-11 08:28:12.472019000 -0700 +++ linux-kernel2/dat-provider/dapl_cno_util.c 2005-05-11 11:37:30.214963000 -0700 @@ -111,11 +111,9 @@ dapl_cno_trigger ( DAPL_CNO *cno_ptr, DAPL_EVD *evd_ptr) { - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; struct dat_event event; - dat_status = DAT_SUCCESS; - dapl_os_assert (cno_ptr->header.magic == DAPL_MAGIC_CNO); /* The spec allows NULL EVDs. kDAPL doesn't have CNOs, they * are strictly used behind the scenes diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_cno_util.c.mine linux-kernel2/dat-provider/dapl_cno_util.c.mine --- linux-kernel/dat-provider/dapl_cno_util.c.mine 2005-05-10 11:53:42.390979000 -0700 +++ linux-kernel2/dat-provider/dapl_cno_util.c.mine 1969-12-31 16:00:00.000000000 -0800 @@ -1,139 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * Manage CNO Info structure - * - * $Id: dapl_cno_util.c 2293 2005-05-10 15:18:25Z jlentini $ - */ - -#include "dapl_ia_util.h" -#include "dapl_cno_util.h" -#include "dapl_adapter_util.h" - - - -/* - * alloc and initialize an EVD struct - */ -DAPL_CNO * -dapl_cno_alloc(DAPL_IA *ia_ptr, const struct dat_upcall_object *upcall) -{ - DAPL_CNO *cno_ptr; - - cno_ptr = (DAPL_CNO *) kmalloc(sizeof (DAPL_CNO), GFP_ATOMIC); - if (!cno_ptr) - { - return NULL; - } - - /* zero the structure */ - memset (cno_ptr, 0, sizeof (DAPL_CNO)); - - /* - * Initialize the header. - */ - cno_ptr->header.provider = ia_ptr->header.provider; - cno_ptr->header.magic = DAPL_MAGIC_CNO; - cno_ptr->header.owner_ia = ia_ptr; - cno_ptr->header.user_context.as_64 = 0; - cno_ptr->header.user_context.as_ptr = NULL; - dapl_llist_init_entry (&cno_ptr->header.ia_list_entry); - spin_lock_init (&cno_ptr->header.lock); - - /* - * Initialize the body - */ - cno_ptr->cno_upcall = *upcall; - - return cno_ptr; -} - -/* - * Free the passed in CNO structure. - */ -void -dapl_cno_dealloc ( - DAPL_CNO *cno_ptr) -{ - dapl_os_assert (cno_ptr->header.magic == DAPL_MAGIC_CNO); - /* - * deinitialize the header - */ - cno_ptr->header.magic = DAPL_MAGIC_INVALID; /* reset magic to prevent reuse */ - - kfree (cno_ptr); -} - - -/* - * dapl_cno_trigger - * - * DAPL Internal routine to trigger the specified CNO. - * Called by the callback of some EVD associated with the CNO. - * - * Input: - * cno_ptr - * evd_ptr EVD triggering - * - * Output: - * None - * - * Returns: - * None - */ -void -dapl_cno_trigger ( - DAPL_CNO *cno_ptr, - DAPL_EVD *evd_ptr) -{ - DAT_RETURN dat_status; - struct dat_event event; - - dat_status = DAT_SUCCESS; - - dapl_os_assert (cno_ptr->header.magic == DAPL_MAGIC_CNO); - /* The spec allows NULL EVDs. kDAPL doesn't have CNOs, they - * are strictly used behind the scenes - */ - dapl_os_assert (evd_ptr == NULL || evd_ptr->header.magic == DAPL_MAGIC_EVD); - - spin_lock (&cno_ptr->header.lock); - - spin_unlock (&cno_ptr->header.lock); - dat_status = dapl_evd_dequeue ((DAT_EVD_HANDLE)evd_ptr, &event); - while (dat_status == DAT_SUCCESS ) { - if (cno_ptr->cno_upcall.upcall_func != (DAT_UPCALL_FUNC) NULL) { - cno_ptr->cno_upcall.upcall_func ( - cno_ptr->cno_upcall.instance_data, - &event, - FALSE); - } - dat_status = dapl_evd_dequeue ( (DAT_EVD_HANDLE)evd_ptr, &event); - } - return; -} diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_cno_util.c.r2295 linux-kernel2/dat-provider/dapl_cno_util.c.r2295 --- linux-kernel/dat-provider/dapl_cno_util.c.r2295 2005-05-10 11:53:42.382980000 -0700 +++ linux-kernel2/dat-provider/dapl_cno_util.c.r2295 1969-12-31 16:00:00.000000000 -0800 @@ -1,139 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * Manage CNO Info structure - * - * $Id: dapl_cno_util.c 2296 2005-05-10 19:00:01Z jlentini $ - */ - -#include "dapl_ia_util.h" -#include "dapl_cno_util.h" -#include "dapl_adapter_util.h" - - - -/* - * alloc and initialize an EVD struct - */ -DAPL_CNO * -dapl_cno_alloc(DAPL_IA *ia_ptr, const struct dat_upcall_object *upcall) -{ - DAPL_CNO *cno_ptr; - - cno_ptr = (DAPL_CNO *) kmalloc(sizeof (DAPL_CNO), GFP_ATOMIC); - if (!cno_ptr) - { - return NULL; - } - - /* zero the structure */ - memset (cno_ptr, 0, sizeof (DAPL_CNO)); - - /* - * Initialize the header. - */ - cno_ptr->header.provider = ia_ptr->header.provider; - cno_ptr->header.magic = DAPL_MAGIC_CNO; - cno_ptr->header.owner_ia = ia_ptr; - cno_ptr->header.user_context.as_64 = 0; - cno_ptr->header.user_context.as_ptr = NULL; - dapl_llist_init_entry (&cno_ptr->header.ia_list_entry); - spin_lock_init (&cno_ptr->header.lock); - - /* - * Initialize the body - */ - cno_ptr->cno_upcall = *upcall; - - return cno_ptr; -} - -/* - * Free the passed in CNO structure. - */ -void -dapl_cno_dealloc ( - DAPL_CNO *cno_ptr) -{ - dapl_os_assert (cno_ptr->header.magic == DAPL_MAGIC_CNO); - /* - * deinitialize the header - */ - cno_ptr->header.magic = DAPL_MAGIC_INVALID; /* reset magic to prevent reuse */ - - kfree (cno_ptr); -} - - -/* - * dapl_cno_trigger - * - * DAPL Internal routine to trigger the specified CNO. - * Called by the callback of some EVD associated with the CNO. - * - * Input: - * cno_ptr - * evd_ptr EVD triggering - * - * Output: - * None - * - * Returns: - * None - */ -void -dapl_cno_trigger ( - DAPL_CNO *cno_ptr, - DAPL_EVD *evd_ptr) -{ - DAT_RETURN dat_status; - struct dat_event event; - - dat_status = DAT_SUCCESS; - - dapl_os_assert (cno_ptr->header.magic == DAPL_MAGIC_CNO); - /* The spec allows NULL EVDs. kDAPL doesn't have CNOs, they - * are strictly used behind the scenes - */ - dapl_os_assert (evd_ptr == NULL || evd_ptr->header.magic == DAPL_MAGIC_EVD); - - spin_lock (&cno_ptr->header.lock); - - spin_unlock (&cno_ptr->header.lock); - dat_status = dapl_evd_dequeue ((DAT_EVD_HANDLE)evd_ptr, &event); - while (dat_status == DAT_SUCCESS ) { - if (cno_ptr->cno_upcall.upcall_func != (DAT_UPCALL_FUNC) NULL) { - cno_ptr->cno_upcall.upcall_func ( - cno_ptr->cno_upcall.instance_data, - &event, - DAT_FALSE); - } - dat_status = dapl_evd_dequeue ( (DAT_EVD_HANDLE)evd_ptr, &event); - } - return; -} diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_cno_util.c.r2296 linux-kernel2/dat-provider/dapl_cno_util.c.r2296 --- linux-kernel/dat-provider/dapl_cno_util.c.r2296 2005-05-10 11:53:42.385969000 -0700 +++ linux-kernel2/dat-provider/dapl_cno_util.c.r2296 1969-12-31 16:00:00.000000000 -0800 @@ -1,136 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * Manage CNO Info structure - * - * $Id: dapl_cno_util.c 2296 2005-05-10 19:00:01Z jlentini $ - */ - -#include "dapl_ia_util.h" -#include "dapl_cno_util.h" -#include "dapl_adapter_util.h" - - - -/* - * alloc and initialize an EVD struct - */ -DAPL_CNO * -dapl_cno_alloc(DAPL_IA *ia_ptr, const struct dat_upcall_object *upcall) -{ - DAPL_CNO *cno_ptr; - - cno_ptr = (DAPL_CNO *) kmalloc(sizeof (DAPL_CNO), GFP_ATOMIC); - if (!cno_ptr) - { - return NULL; - } - - /* zero the structure */ - memset (cno_ptr, 0, sizeof (DAPL_CNO)); - - /* - * Initialize the header. - */ - cno_ptr->header.provider = ia_ptr->header.provider; - cno_ptr->header.magic = DAPL_MAGIC_CNO; - cno_ptr->header.owner_ia = ia_ptr; - cno_ptr->header.user_context.as_64 = 0; - cno_ptr->header.user_context.as_ptr = NULL; - dapl_llist_init_entry (&cno_ptr->header.ia_list_entry); - spin_lock_init (&cno_ptr->header.lock); - - /* - * Initialize the body - */ - cno_ptr->cno_upcall = *upcall; - - return cno_ptr; -} - -/* - * Free the passed in CNO structure. - */ -void -dapl_cno_dealloc ( - DAPL_CNO *cno_ptr) -{ - dapl_os_assert (cno_ptr->header.magic == DAPL_MAGIC_CNO); - /* - * deinitialize the header - */ - cno_ptr->header.magic = DAPL_MAGIC_INVALID; /* reset magic to prevent reuse */ - - kfree (cno_ptr); -} - - -/* - * dapl_cno_trigger - * - * DAPL Internal routine to trigger the specified CNO. - * Called by the callback of some EVD associated with the CNO. - * - * Input: - * cno_ptr - * evd_ptr EVD triggering - * - * Output: - * None - * - * Returns: - * None - */ -void -dapl_cno_trigger ( - DAPL_CNO *cno_ptr, - DAPL_EVD *evd_ptr) -{ - DAT_RETURN dat_status; - struct dat_event event; - - dat_status = DAT_SUCCESS; - - dapl_os_assert (cno_ptr->header.magic == DAPL_MAGIC_CNO); - /* The spec allows NULL EVDs. kDAPL doesn't have CNOs, they - * are strictly used behind the scenes - */ - dapl_os_assert (evd_ptr == NULL || - evd_ptr->header.magic == DAPL_MAGIC_EVD); - - dat_status = dapl_evd_dequeue ((DAT_EVD_HANDLE)evd_ptr, &event); - while (dat_status == DAT_SUCCESS ) { - if (cno_ptr->cno_upcall.upcall_func != (DAT_UPCALL_FUNC) NULL) { - cno_ptr->cno_upcall.upcall_func ( - cno_ptr->cno_upcall.instance_data, - &event, - DAT_FALSE); - } - dat_status = dapl_evd_dequeue((DAT_EVD_HANDLE)evd_ptr, &event); - } -} diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_cookie.c linux-kernel2/dat-provider/dapl_cookie.c --- linux-kernel/dat-provider/dapl_cookie.c 2005-04-27 11:38:34.700014000 -0700 +++ linux-kernel2/dat-provider/dapl_cookie.c 2005-05-11 11:41:23.413960000 -0700 @@ -84,9 +84,9 @@ * * *********************************************************************/ -DAT_RETURN dapl_cb_get(DAPL_COOKIE_BUFFER * buffer, DAPL_COOKIE ** cookie_ptr); +u32 dapl_cb_get(DAPL_COOKIE_BUFFER *buffer, DAPL_COOKIE **cookie_ptr); -DAT_RETURN dapl_cb_put(DAPL_COOKIE_BUFFER * buffer, DAPL_COOKIE * cookie); +u32 dapl_cb_put(DAPL_COOKIE_BUFFER *buffer, DAPL_COOKIE *cookie); /********************************************************************* * * @@ -113,8 +113,7 @@ DAT_RETURN dapl_cb_put(DAPL_COOKIE_BUFFE * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN -dapl_cb_create(DAPL_COOKIE_BUFFER * buffer, DAPL_EP * ep, DAT_COUNT size) +u32 dapl_cb_create(DAPL_COOKIE_BUFFER *buffer, DAPL_EP *ep, DAT_COUNT size) { DAT_COUNT i; @@ -179,9 +178,9 @@ void dapl_cb_free(DAPL_COOKIE_BUFFER * b * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN dapl_cb_get(DAPL_COOKIE_BUFFER * buffer, DAPL_COOKIE ** cookie_ptr) +u32 dapl_cb_get(DAPL_COOKIE_BUFFER *buffer, DAPL_COOKIE **cookie_ptr) { - DAT_RETURN dat_status; + u32 dat_status; DAT_COUNT new_head; dapl_os_assert(NULL != cookie_ptr); @@ -218,7 +217,7 @@ DAT_RETURN dapl_cb_get(DAPL_COOKIE_BUFFE * DAT_INSUFFICIENT_EMPTY * */ -DAT_RETURN dapl_cb_put(DAPL_COOKIE_BUFFER * buffer, DAPL_COOKIE * cookie) +u32 dapl_cb_put(DAPL_COOKIE_BUFFER *buffer, DAPL_COOKIE *cookie) { atomic_set(&buffer->tail, cookie->index); @@ -243,13 +242,11 @@ DAT_RETURN dapl_cb_put(DAPL_COOKIE_BUFFE * DAT_INSUFFICIENT_EMPTY * */ -DAT_RETURN -dapl_rmr_cookie_alloc(DAPL_COOKIE_BUFFER * buffer, - DAPL_RMR * rmr, - DAT_RMR_COOKIE user_cookie, DAPL_COOKIE ** cookie_ptr) +u32 dapl_rmr_cookie_alloc(DAPL_COOKIE_BUFFER *buffer, DAPL_RMR *rmr, + DAT_RMR_COOKIE user_cookie, DAPL_COOKIE **cookie_ptr) { DAPL_COOKIE *cookie; - DAT_RETURN dat_status; + u32 dat_status; if (DAT_SUCCESS != dapl_cb_get(buffer, &cookie)) { *cookie_ptr = NULL; @@ -287,10 +284,8 @@ dapl_rmr_cookie_alloc(DAPL_COOKIE_BUFFER * DAT_INSUFFICIENT_EMPTY * */ -DAT_RETURN -dapl_dto_cookie_alloc(DAPL_COOKIE_BUFFER * buffer, - DAPL_DTO_TYPE type, - DAT_DTO_COOKIE user_cookie, DAPL_COOKIE ** cookie_ptr) +u32 dapl_dto_cookie_alloc(DAPL_COOKIE_BUFFER *buffer, DAPL_DTO_TYPE type, + DAT_DTO_COOKIE user_cookie, DAPL_COOKIE **cookie_ptr) { DAPL_COOKIE *cookie; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_cookie.h linux-kernel2/dat-provider/dapl_cookie.h --- linux-kernel/dat-provider/dapl_cookie.h 2005-04-20 12:42:06.513011000 -0700 +++ linux-kernel2/dat-provider/dapl_cookie.h 2005-05-11 11:42:25.658973000 -0700 @@ -36,20 +36,18 @@ #include "dapl.h" -extern DAT_RETURN -dapl_cb_create(DAPL_COOKIE_BUFFER * buffer, DAPL_EP * ep, DAT_COUNT size); +extern u32 dapl_cb_create(DAPL_COOKIE_BUFFER *buffer, DAPL_EP *ep, + DAT_COUNT size); -extern void dapl_cb_free(DAPL_COOKIE_BUFFER * buffer); +extern void dapl_cb_free(DAPL_COOKIE_BUFFER *buffer); -extern DAT_RETURN -dapl_rmr_cookie_alloc(DAPL_COOKIE_BUFFER * buffer, - DAPL_RMR * rmr, - DAT_RMR_COOKIE user_cookie, DAPL_COOKIE ** cookie_ptr); - -extern DAT_RETURN -dapl_dto_cookie_alloc(DAPL_COOKIE_BUFFER * buffer, - DAPL_DTO_TYPE type, - DAT_DTO_COOKIE user_cookie, DAPL_COOKIE ** cookie_ptr); +extern u32 dapl_rmr_cookie_alloc(DAPL_COOKIE_BUFFER *buffer, DAPL_RMR *rmr, + DAT_RMR_COOKIE user_cookie, + DAPL_COOKIE **cookie_ptr); + +extern u32 dapl_dto_cookie_alloc(DAPL_COOKIE_BUFFER *buffer, + DAPL_DTO_TYPE type, DAT_DTO_COOKIE user_cookie, + DAPL_COOKIE **cookie_ptr); extern void dapl_cookie_dealloc(DAPL_COOKIE_BUFFER * buffer, DAPL_COOKIE * cookie); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_cr_accept.c linux-kernel2/dat-provider/dapl_cr_accept.c --- linux-kernel/dat-provider/dapl_cr_accept.c 2005-05-02 08:48:17.581023000 -0700 +++ linux-kernel2/dat-provider/dapl_cr_accept.c 2005-05-11 11:42:47.385963000 -0700 @@ -39,13 +39,11 @@ * Establish a connection between active side (remote endpoint) * and passive side (local endpoint). */ -DAT_RETURN -dapl_cr_accept(DAT_CR_HANDLE cr_handle, - DAT_EP_HANDLE ep_handle, - DAT_COUNT private_data_size, const DAT_PVOID private_data) +u32 dapl_cr_accept(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, + DAT_COUNT private_data_size, const DAT_PVOID private_data) { DAPL_EP *ep_ptr; - DAT_RETURN dat_status; + u32 dat_status; DAPL_CR *cr_ptr; DAT_EP_STATE entry_ep_state; DAT_EP_HANDLE entry_ep_handle; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_cr_callback.c linux-kernel2/dat-provider/dapl_cr_callback.c --- linux-kernel/dat-provider/dapl_cr_callback.c 2005-05-11 08:28:11.511012000 -0700 +++ linux-kernel2/dat-provider/dapl_cr_callback.c 2005-05-11 11:43:26.176964000 -0700 @@ -45,9 +45,8 @@ /* * Prototypes */ -DAT_RETURN dapl_connection_request(ib_cm_handle_t ib_cm_handle, - DAPL_SP * sp_ptr, - const void *prd_ptr, DAPL_EVD * evd_ptr); +u32 dapl_connection_request(ib_cm_handle_t ib_cm_handle, DAPL_SP *sp_ptr, + const void *prd_ptr, DAPL_EVD *evd_ptr); DAPL_EP *dapl_get_sp_ep(ib_cm_handle_t ib_cm_handle, DAPL_SP * sp_ptr, DAT_EVENT_NUMBER dat_event_num); @@ -65,7 +64,7 @@ void dapl_cr_callback(ib_cm_handle_t ib_ DAPL_EVD *evd_ptr; DAPL_SP *sp_ptr; DAT_EVENT_NUMBER dat_event_num; - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_CM | DAPL_DBG_TYPE_CALLBACK, "--> dapl_cr_callback! context: %p event: %x cm_handle %p\n", @@ -266,12 +265,10 @@ void dapl_cr_callback(ib_cm_handle_t ib_ * Create a CR record and link it on to the SP so we can update it * and free it later. Create an EP if specified by the PSP flags. */ -DAT_RETURN -dapl_connection_request(ib_cm_handle_t ib_cm_handle, - DAPL_SP * sp_ptr, - const void *prd_ptr, DAPL_EVD * evd_ptr) +u32 dapl_connection_request(ib_cm_handle_t ib_cm_handle, DAPL_SP *sp_ptr, + const void *prd_ptr, DAPL_EVD *evd_ptr) { - DAT_RETURN dat_status; + u32 dat_status; DAPL_CR *cr_ptr; DAPL_EP *ep_ptr; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_cr_handoff.c linux-kernel2/dat-provider/dapl_cr_handoff.c --- linux-kernel/dat-provider/dapl_cr_handoff.c 2005-04-20 12:42:07.958024000 -0700 +++ linux-kernel2/dat-provider/dapl_cr_handoff.c 2005-05-11 11:16:06.189964000 -0700 @@ -37,7 +37,7 @@ * Hand the connection request to another Sevice pont specified by the * Connectin Qualifier. */ -DAT_RETURN dapl_cr_handoff(DAT_CR_HANDLE cr_handle, DAT_CONN_QUAL cr_handoff) +u32 dapl_cr_handoff(DAT_CR_HANDLE cr_handle, DAT_CONN_QUAL cr_handoff) { /* handoff */ return DAT_ERROR(DAT_NOT_IMPLEMENTED, 0); } diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_cr_query.c linux-kernel2/dat-provider/dapl_cr_query.c --- linux-kernel/dat-provider/dapl_cr_query.c 2005-05-10 08:13:29.281019000 -0700 +++ linux-kernel2/dat-provider/dapl_cr_query.c 2005-05-11 11:16:06.193960000 -0700 @@ -34,10 +34,10 @@ #include "dapl.h" #include "dapl_adapter_util.h" -DAT_RETURN dapl_cr_query(DAT_CR_HANDLE cr_handle, struct dat_cr_param *cr_param) +u32 dapl_cr_query(DAT_CR_HANDLE cr_handle, struct dat_cr_param *cr_param) { DAPL_CR *cr_ptr; - DAT_RETURN status; + u32 status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_cr_query (%p, %x, %p)\n", diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_cr_reject.c linux-kernel2/dat-provider/dapl_cr_reject.c --- linux-kernel/dat-provider/dapl_cr_reject.c 2005-04-20 12:42:07.641003000 -0700 +++ linux-kernel2/dat-provider/dapl_cr_reject.c 2005-05-11 11:16:06.196963000 -0700 @@ -40,14 +40,14 @@ * Reject a connection request from the active remote side requesting * an Endpoint. */ -DAT_RETURN dapl_cr_reject(DAT_CR_HANDLE cr_handle) +u32 dapl_cr_reject(DAT_CR_HANDLE cr_handle) { DAPL_CR *cr_ptr; DAPL_EP *ep_ptr; DAT_EP_STATE entry_ep_state; DAT_EP_HANDLE entry_ep_handle; DAPL_SP *sp_ptr; - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_cr_reject (%p)\n", cr_handle); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_connect.c linux-kernel2/dat-provider/dapl_ep_connect.c --- linux-kernel/dat-provider/dapl_ep_connect.c 2005-05-10 08:13:28.285058000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_connect.c 2005-05-11 11:58:33.206963000 -0700 @@ -42,18 +42,15 @@ * and a remote Endpoint. This operation is used by the active/client * side of a connection */ -DAT_RETURN -dapl_ep_connect(DAT_EP_HANDLE ep_handle, - DAT_IA_ADDRESS_PTR remote_ia_address, - DAT_CONN_QUAL remote_conn_qual, - DAT_TIMEOUT timeout, - DAT_COUNT private_data_size, - const DAT_PVOID private_data, - DAT_QOS qos, DAT_CONNECT_FLAGS connect_flags) +u32 dapl_ep_connect(DAT_EP_HANDLE ep_handle, + DAT_IA_ADDRESS_PTR remote_ia_address, + DAT_CONN_QUAL remote_conn_qual, DAT_TIMEOUT timeout, + DAT_COUNT private_data_size, const DAT_PVOID private_data, + DAT_QOS qos, DAT_CONNECT_FLAGS connect_flags) { DAPL_EP *ep_ptr; DAPL_EP alloc_ep; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API | DAPL_DBG_TYPE_CM, "dapl_ep_connect (%p, {%u.%u.%u.%u}, %X, %d, %d, %p, %x, %x)\n", @@ -66,7 +63,6 @@ dapl_ep_connect(DAT_EP_HANDLE ep_handle, timeout, private_data_size, private_data, qos, connect_flags); - dat_status = DAT_SUCCESS; ep_ptr = (DAPL_EP *) ep_handle; /* diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_create.c linux-kernel2/dat-provider/dapl_ep_create.c --- linux-kernel/dat-provider/dapl_ep_create.c 2005-05-10 08:13:28.361017000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_create.c 2005-05-11 11:59:08.236964000 -0700 @@ -63,18 +63,16 @@ * DAT_INVALID_ATTRIBUTE * DAT_MODEL_NOT_SUPPORTED */ -DAT_RETURN -dapl_ep_create(DAT_IA_HANDLE ia_handle, - DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE recv_evd_handle, - DAT_EVD_HANDLE request_evd_handle, - DAT_EVD_HANDLE connect_evd_handle, - const struct dat_ep_attr *ep_attr, DAT_EP_HANDLE * ep_handle) +u32 dapl_ep_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, + DAT_EVD_HANDLE recv_evd_handle, + DAT_EVD_HANDLE request_evd_handle, + DAT_EVD_HANDLE connect_evd_handle, + const struct dat_ep_attr *ep_attr, DAT_EP_HANDLE *ep_handle) { DAPL_IA *ia_ptr; DAPL_EP *ep_ptr; struct dat_ep_attr ep_attr_limit; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_create (%p, %p, %p, %p, %p, %p, %p)\n", @@ -85,7 +83,6 @@ dapl_ep_create(DAT_IA_HANDLE ia_handle, connect_evd_handle, ep_attr, ep_handle); ia_ptr = (DAPL_IA *) ia_handle; - dat_status = DAT_SUCCESS; /* * Verify parameters diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_create_with_srq.c linux-kernel2/dat-provider/dapl_ep_create_with_srq.c --- linux-kernel/dat-provider/dapl_ep_create_with_srq.c 2005-05-10 08:13:29.078008000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_create_with_srq.c 2005-05-11 12:00:02.787969000 -0700 @@ -69,22 +69,19 @@ * DAT_INVALID_ATTRIBUTE * DAT_MODEL_NOT_SUPPORTED */ -DAT_RETURN -dapl_ep_create_with_srq(DAT_IA_HANDLE ia_handle, - DAT_PZ_HANDLE pz_handle, - DAT_EVD_HANDLE recv_evd_handle, - DAT_EVD_HANDLE request_evd_handle, - DAT_EVD_HANDLE connect_evd_handle, - DAT_SRQ_HANDLE srq_handle, - const struct dat_ep_attr *ep_attr, - DAT_EP_HANDLE * ep_handle) +u32 dapl_ep_create_with_srq(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, + DAT_EVD_HANDLE recv_evd_handle, + DAT_EVD_HANDLE request_evd_handle, + DAT_EVD_HANDLE connect_evd_handle, + DAT_SRQ_HANDLE srq_handle, + const struct dat_ep_attr *ep_attr, + DAT_EP_HANDLE *ep_handle) { DAPL_IA *ia_ptr; DAPL_EP *ep_ptr; struct dat_ep_attr ep_attr_limit; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; - dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_create_with_srq (%p, %p, %p, %p, %p, %p, %p %p)\n", ia_handle, diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_disconnect.c linux-kernel2/dat-provider/dapl_ep_disconnect.c --- linux-kernel/dat-provider/dapl_ep_disconnect.c 2005-05-02 08:48:17.787970000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_disconnect.c 2005-05-11 12:00:23.524963000 -0700 @@ -63,14 +63,14 @@ * DAT_INSUFFICIENT_RESOURCES * DAT_INVALID_PARAMETER */ -DAT_RETURN -dapl_ep_disconnect(DAT_EP_HANDLE ep_handle, DAT_CLOSE_FLAGS disconnect_flags) +u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_handle, + DAT_CLOSE_FLAGS disconnect_flags) { DAPL_EP *ep_ptr; DAPL_EVD *evd_ptr; DAPL_CR *cr_ptr; ib_cm_events_t ib_cm_event; - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API | DAPL_DBG_TYPE_CM, "dapl_ep_disconnect (%p, %x)\n", diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_dup_connect.c linux-kernel2/dat-provider/dapl_ep_dup_connect.c --- linux-kernel/dat-provider/dapl_ep_dup_connect.c 2005-04-20 12:42:08.547017000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_dup_connect.c 2005-05-11 12:00:47.734963000 -0700 @@ -69,15 +69,12 @@ * DAT_INVALID_STATE * DAT_MODEL_NOT_SUPPORTED */ -DAT_RETURN -dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, - DAT_EP_HANDLE ep_dup_handle, - DAT_TIMEOUT timeout, - DAT_COUNT private_data_size, - const DAT_PVOID private_data, DAT_QOS qos) +u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, DAT_EP_HANDLE ep_dup_handle, + DAT_TIMEOUT timeout, DAT_COUNT private_data_size, + const DAT_PVOID private_data, DAT_QOS qos) { DAPL_EP *ep_dup_ptr; - DAT_RETURN dat_status; + u32 dat_status; DAT_IA_ADDRESS_PTR remote_ia_address_ptr; DAT_CONN_QUAL remote_conn_qual; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_free.c linux-kernel2/dat-provider/dapl_ep_free.c --- linux-kernel/dat-provider/dapl_ep_free.c 2005-05-10 08:13:28.517011000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_free.c 2005-05-11 11:16:06.224962000 -0700 @@ -61,13 +61,13 @@ * DAT_INVALID_PARAMETER * DAT_INVALID_STATE */ -DAT_RETURN dapl_ep_free(DAT_EP_HANDLE ep_handle) +u32 dapl_ep_free(DAT_EP_HANDLE ep_handle) { DAPL_EP *ep_ptr; DAPL_IA *ia_ptr; struct dat_ep_param *param; ib_qp_state_t save_qp_state; - DAT_RETURN dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_free (%p)\n", ep_handle); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_get_status.c linux-kernel2/dat-provider/dapl_ep_get_status.c --- linux-kernel/dat-provider/dapl_ep_get_status.c 2005-05-11 08:28:11.861017000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_get_status.c 2005-05-11 12:02:16.057971000 -0700 @@ -59,20 +59,17 @@ * DAT_SUCCESS * DAT_INVALID_PARAMETER */ -DAT_RETURN -dapl_ep_get_status(DAT_EP_HANDLE ep_handle, - DAT_EP_STATE * ep_state, - boolean_t *in_dto_idle, boolean_t *out_dto_idle) +u32 dapl_ep_get_status(DAT_EP_HANDLE ep_handle, DAT_EP_STATE *ep_state, + boolean_t *in_dto_idle, boolean_t *out_dto_idle) { DAPL_EP *ep_ptr; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_get_status (%p, %p, %p, %p)\n", ep_handle, ep_state, in_dto_idle, out_dto_idle); ep_ptr = (DAPL_EP *) ep_handle; - dat_status = DAT_SUCCESS; /* * Verify parameter & state diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_modify.c linux-kernel2/dat-provider/dapl_ep_modify.c --- linux-kernel/dat-provider/dapl_ep_modify.c 2005-05-11 08:28:12.424000000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_modify.c 2005-05-11 12:04:01.023960000 -0700 @@ -52,7 +52,7 @@ * */ -static inline DAT_RETURN +static inline u32 dapl_ep_modify_validate_parameters(DAT_EP_HANDLE ep_handle, DAT_EP_PARAM_MASK mask, const struct dat_ep_param *ep_param, @@ -65,11 +65,10 @@ dapl_ep_modify_validate_parameters(DAT_E struct dat_ep_attr ep_attr; struct dat_ep_attr ep_attr_limit; struct dat_ep_attr ep_attr_request; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; *ia_ptr = NULL; *ep_ptr = NULL; - dat_status = DAT_SUCCESS; if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) { dat_status = @@ -309,8 +308,8 @@ dapl_ep_modify_validate_parameters(DAT_E * DAT_INVALID_ATTRIBUTE * DAT_INVALID_STATE */ -DAT_RETURN dapl_ep_modify(DAT_EP_HANDLE ep_handle, DAT_EP_PARAM_MASK mask, - const struct dat_ep_param *ep_param) +u32 dapl_ep_modify(DAT_EP_HANDLE ep_handle, DAT_EP_PARAM_MASK mask, + const struct dat_ep_param *ep_param) { DAPL_IA *ia; DAPL_EP *ep1, *ep2; @@ -319,7 +318,7 @@ DAT_RETURN dapl_ep_modify(DAT_EP_HANDLE DAPL_EP alloc_ep; /* Holder for resources. */ DAPL_PZ *tmp_pz; DAPL_EVD *tmp_evd; - DAT_RETURN dat_status; + u32 dat_status; /* Flag indicating we've allocated a new one of these. */ boolean_t qp_allocated = FALSE; @@ -585,7 +584,7 @@ DAT_RETURN dapl_ep_modify(DAT_EP_HANDLE bail: if (qp_allocated) { - DAT_RETURN local_dat_status; + u32 local_dat_status; if (dat_status != DAT_SUCCESS || !qp_used) { local_dat_status = dapl_ib_qp_free(ia, &alloc_ep); } else { diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_post_rdma_read.c linux-kernel2/dat-provider/dapl_ep_post_rdma_read.c --- linux-kernel/dat-provider/dapl_ep_post_rdma_read.c 2005-05-10 08:13:29.096006000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_post_rdma_read.c 2005-05-11 12:04:41.106971000 -0700 @@ -66,16 +66,14 @@ * DAT_PROTECTION_VIOLATION * DAT_PRIVILEGES_VIOLATION */ -DAT_RETURN -dapl_ep_post_rdma_read(DAT_EP_HANDLE ep_handle, - DAT_COUNT num_segments, - struct dat_lmr_triplet *local_iov, - DAT_DTO_COOKIE user_cookie, - const struct dat_rmr_triplet *remote_iov, - DAT_COMPLETION_FLAGS completion_flags) +u32 dapl_ep_post_rdma_read(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, + struct dat_lmr_triplet *local_iov, + DAT_DTO_COOKIE user_cookie, + const struct dat_rmr_triplet *remote_iov, + DAT_COMPLETION_FLAGS completion_flags) { - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_post_rdma_read (%p, %d, %p, %p, %p, %x)\n", diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_post_rdma_write.c linux-kernel2/dat-provider/dapl_ep_post_rdma_write.c --- linux-kernel/dat-provider/dapl_ep_post_rdma_write.c 2005-05-10 08:13:28.342024000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_post_rdma_write.c 2005-05-11 12:05:08.228968000 -0700 @@ -66,15 +66,13 @@ * DAT_PROTECTION_VIOLATION * DAT_PRIVILEGES_VIOLATION */ -DAT_RETURN -dapl_ep_post_rdma_write(DAT_EP_HANDLE ep_handle, - DAT_COUNT num_segments, - struct dat_lmr_triplet *local_iov, - DAT_DTO_COOKIE user_cookie, - const struct dat_rmr_triplet *remote_iov, - DAT_COMPLETION_FLAGS completion_flags) +u32 dapl_ep_post_rdma_write(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, + struct dat_lmr_triplet *local_iov, + DAT_DTO_COOKIE user_cookie, + const struct dat_rmr_triplet *remote_iov, + DAT_COMPLETION_FLAGS completion_flags) { - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_post_rdma_write (%p, %d, %p, %P, %p, %x)\n", diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_post_recv.c linux-kernel2/dat-provider/dapl_ep_post_recv.c --- linux-kernel/dat-provider/dapl_ep_post_recv.c 2005-05-10 08:13:29.263024000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_post_recv.c 2005-05-11 12:05:45.066971000 -0700 @@ -66,16 +66,14 @@ * DAT_PROTECTION_VIOLATION * DAT_PROVILEGES_VIOLATION */ -DAT_RETURN -dapl_ep_post_recv(DAT_EP_HANDLE ep_handle, - DAT_COUNT num_segments, - struct dat_lmr_triplet *local_iov, - DAT_DTO_COOKIE user_cookie, - DAT_COMPLETION_FLAGS completion_flags) +u32 dapl_ep_post_recv(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, + struct dat_lmr_triplet *local_iov, + DAT_DTO_COOKIE user_cookie, + DAT_COMPLETION_FLAGS completion_flags) { DAPL_EP *ep_ptr; DAPL_COOKIE *cookie; - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_post_recv (%p, %d, %p, %P, %x)\n", diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_post_send.c linux-kernel2/dat-provider/dapl_ep_post_send.c --- linux-kernel/dat-provider/dapl_ep_post_send.c 2005-05-10 08:13:28.702022000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_post_send.c 2005-05-11 12:06:13.573960000 -0700 @@ -63,15 +63,13 @@ * DAT_PROTECTION_VIOLATION * DAT_PRIVILEGES_VIOLATION */ -DAT_RETURN -dapl_ep_post_send(DAT_EP_HANDLE ep_handle, - DAT_COUNT num_segments, - struct dat_lmr_triplet *local_iov, - DAT_DTO_COOKIE user_cookie, - DAT_COMPLETION_FLAGS completion_flags) +u32 dapl_ep_post_send(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, + struct dat_lmr_triplet *local_iov, + DAT_DTO_COOKIE user_cookie, + DAT_COMPLETION_FLAGS completion_flags) { struct dat_rmr_triplet remote_iov = { 0, 0, 0, 0 }; - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_post_send (%p, %d, %p, %P, %x)\n", diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_query.c linux-kernel2/dat-provider/dapl_ep_query.c --- linux-kernel/dat-provider/dapl_ep_query.c 2005-05-10 08:13:28.778009000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_query.c 2005-05-11 12:06:40.494963000 -0700 @@ -57,17 +57,15 @@ * DAT_SUCCESS * DAT_INVALID_PARAMETER */ -DAT_RETURN -dapl_ep_query(DAT_EP_HANDLE ep_handle, struct dat_ep_param *ep_param) +u32 dapl_ep_query(DAT_EP_HANDLE ep_handle, struct dat_ep_param *ep_param) { DAPL_EP *ep_ptr; - DAT_RETURN status; + u32 status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_query (%p, %p)\n", ep_handle, ep_param); - status = DAT_SUCCESS; ep_ptr = (DAPL_EP *) ep_handle; /* diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_recv_query.c linux-kernel2/dat-provider/dapl_ep_recv_query.c --- linux-kernel/dat-provider/dapl_ep_recv_query.c 2005-05-04 08:19:32.717975000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_recv_query.c 2005-05-11 12:07:41.941768000 -0700 @@ -54,14 +54,11 @@ * DAT_INVALID_HANDLE * DAT_MODEL_NOT_SUPPORTED */ -DAT_RETURN -dapl_ep_recv_query(DAT_EP_HANDLE ep_handle, - DAT_COUNT * nbufs_allocate, DAT_COUNT * bufs_alloc_span) +u32 dapl_ep_recv_query(DAT_EP_HANDLE ep_handle, + DAT_COUNT *nbufs_allocate, DAT_COUNT *bufs_alloc_span) { DAPL_EP *ep_ptr; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_recv_query (%p, %p, %p)\n", ep_handle, nbufs_allocate, bufs_alloc_span); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_reset.c linux-kernel2/dat-provider/dapl_ep_reset.c --- linux-kernel/dat-provider/dapl_ep_reset.c 2005-04-20 12:42:07.660015000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_reset.c 2005-05-11 12:08:05.331962000 -0700 @@ -61,12 +61,10 @@ * DAT_INVALID_PARAMETER * DAT_INVALID_STATE */ -DAT_RETURN dapl_ep_reset(DAT_EP_HANDLE ep_handle) +u32 dapl_ep_reset(DAT_EP_HANDLE ep_handle) { DAPL_EP *ep_ptr; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; ep_ptr = (DAPL_EP *) ep_handle; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_set_watermark.c linux-kernel2/dat-provider/dapl_ep_set_watermark.c --- linux-kernel/dat-provider/dapl_ep_set_watermark.c 2005-04-20 12:42:08.865015000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_set_watermark.c 2005-05-11 12:08:28.540010000 -0700 @@ -59,15 +59,12 @@ * DAT_INVALID_HANDLE * DAT_MODEL_NOT_SUPPORTED */ -DAT_RETURN -dapl_ep_set_watermark(DAT_EP_HANDLE ep_handle, - DAT_COUNT soft_high_watermark, - DAT_COUNT hard_high_watermark) +u32 dapl_ep_set_watermark(DAT_EP_HANDLE ep_handle, + DAT_COUNT soft_high_watermark, + DAT_COUNT hard_high_watermark) { DAPL_EP *ep_ptr; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_set_watermark (%p, %d, %d)\n", ep_handle, soft_high_watermark, hard_high_watermark); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_util.c linux-kernel2/dat-provider/dapl_ep_util.c --- linux-kernel/dat-provider/dapl_ep_util.c 2005-05-11 08:28:12.629009000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_util.c 2005-05-11 12:09:16.971011000 -0700 @@ -257,7 +257,7 @@ void dapl_ep_default_attrs(DAPL_EP * ep_ return; } -DAT_RETURN dapl_ep_check_recv_completion_flags(DAT_COMPLETION_FLAGS flags) +u32 dapl_ep_check_recv_completion_flags(DAT_COMPLETION_FLAGS flags) { /* @@ -276,23 +276,21 @@ DAT_RETURN dapl_ep_check_recv_completion return DAT_SUCCESS; } -DAT_RETURN dapl_ep_check_request_completion_flags(DAT_COMPLETION_FLAGS flags) +u32 dapl_ep_check_request_completion_flags(DAT_COMPLETION_FLAGS flags) { return DAT_SUCCESS; } -DAT_RETURN -dapl_ep_post_send_req(DAT_EP_HANDLE ep_handle, - DAT_COUNT num_segments, - struct dat_lmr_triplet *local_iov, - DAT_DTO_COOKIE user_cookie, - const struct dat_rmr_triplet *remote_iov, - DAT_COMPLETION_FLAGS completion_flags, - DAPL_DTO_TYPE dto_type, int op_type) +u32 dapl_ep_post_send_req(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, + struct dat_lmr_triplet *local_iov, + DAT_DTO_COOKIE user_cookie, + const struct dat_rmr_triplet *remote_iov, + DAT_COMPLETION_FLAGS completion_flags, + DAPL_DTO_TYPE dto_type, int op_type) { DAPL_EP *ep_ptr; DAPL_COOKIE *cookie; - DAT_RETURN dat_status; + u32 dat_status; if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) { dat_status = @@ -373,9 +371,9 @@ void dapl_ep_timeout(uintptr_t arg) * Return the INVALID_STATE connection subtype associated with an * INVALID_STATE on an EP. Strictly for error reporting. */ -DAT_RETURN_SUBTYPE dapl_ep_state_subtype(DAPL_EP * ep_ptr) +enum dat_return_subtype dapl_ep_state_subtype(DAPL_EP * ep_ptr) { - DAT_RETURN_SUBTYPE dat_status; + enum dat_return_subtype dat_status; switch (ep_ptr->param.ep_state) { case DAT_EP_STATE_UNCONNECTED: diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ep_util.h linux-kernel2/dat-provider/dapl_ep_util.h --- linux-kernel/dat-provider/dapl_ep_util.h 2005-05-10 08:13:29.059014000 -0700 +++ linux-kernel2/dat-provider/dapl_ep_util.h 2005-05-11 12:09:54.401014000 -0700 @@ -46,23 +46,20 @@ extern DAPL_EP *dapl_ep_alloc(DAPL_IA * extern void dapl_ep_dealloc(DAPL_EP * ep_ptr); -extern DAT_RETURN -dapl_ep_check_recv_completion_flags(DAT_COMPLETION_FLAGS flags); +extern u32 dapl_ep_check_recv_completion_flags(DAT_COMPLETION_FLAGS flags); -extern DAT_RETURN -dapl_ep_check_request_completion_flags(DAT_COMPLETION_FLAGS flags); +extern u32 dapl_ep_check_request_completion_flags(DAT_COMPLETION_FLAGS flags); -extern DAT_RETURN -dapl_ep_post_send_req(DAT_EP_HANDLE ep_handle, - DAT_COUNT num_segments, - struct dat_lmr_triplet *local_iov, - DAT_DTO_COOKIE user_cookie, - const struct dat_rmr_triplet *remote_iov, - DAT_COMPLETION_FLAGS completion_flags, - DAPL_DTO_TYPE dto_type, int op_type); +extern u32 dapl_ep_post_send_req(DAT_EP_HANDLE ep_handle, + DAT_COUNT num_segments, + struct dat_lmr_triplet *local_iov, + DAT_DTO_COOKIE user_cookie, + const struct dat_rmr_triplet *remote_iov, + DAT_COMPLETION_FLAGS completion_flags, + DAPL_DTO_TYPE dto_type, int op_type); void dapl_ep_timeout(uintptr_t arg); -DAT_RETURN_SUBTYPE dapl_ep_state_subtype(DAPL_EP * ep_ptr); +enum dat_return_subtype dapl_ep_state_subtype(DAPL_EP * ep_ptr); #endif /* DAPL_EP_UTIL_H */ diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_evd_connection_callb.c linux-kernel2/dat-provider/dapl_evd_connection_callb.c --- linux-kernel/dat-provider/dapl_evd_connection_callb.c 2005-05-11 08:28:11.694002000 -0700 +++ linux-kernel2/dat-provider/dapl_evd_connection_callb.c 2005-05-11 11:16:06.285963000 -0700 @@ -69,7 +69,7 @@ dapl_evd_connection_callback(ib_cm_handl DAPL_EP *ep_ptr; DAPL_EVD *evd_ptr; DAT_EVENT_NUMBER dat_event_num; - DAT_RETURN dat_status; + u32 dat_status; int private_data_size; dapl_dbg_log(DAPL_DBG_TYPE_CM | DAPL_DBG_TYPE_CALLBACK, diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_evd_cq_async_error_callb.c linux-kernel2/dat-provider/dapl_evd_cq_async_error_callb.c --- linux-kernel/dat-provider/dapl_evd_cq_async_error_callb.c 2005-04-20 12:42:06.412004000 -0700 +++ linux-kernel2/dat-provider/dapl_evd_cq_async_error_callb.c 2005-05-11 11:16:06.290966000 -0700 @@ -58,7 +58,7 @@ dapl_evd_cq_async_error_callback(ib_erro { DAPL_EVD *async_evd; DAPL_EVD *evd; - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK | DAPL_DBG_TYPE_EXCEPTION, "dapl_evd_cq_async_error_callback (%p, %p)\n", diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_evd_dequeue.c linux-kernel2/dat-provider/dapl_evd_dequeue.c --- linux-kernel/dat-provider/dapl_evd_dequeue.c 2005-05-10 08:13:28.796004000 -0700 +++ linux-kernel2/dat-provider/dapl_evd_dequeue.c 2005-05-11 12:11:09.256002000 -0700 @@ -62,17 +62,16 @@ * DAT_QUEUE_EMPTY */ -DAT_RETURN dapl_evd_dequeue(DAT_EVD_HANDLE evd_handle, struct dat_event *event) +u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_handle, struct dat_event *event) { DAPL_EVD *evd_ptr; struct dat_event *local_event; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_evd_dequeue (%p, %p)\n", evd_handle, event); evd_ptr = (DAPL_EVD *) evd_handle; - dat_status = DAT_SUCCESS; if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, 0); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_evd_dto_callb.c linux-kernel2/dat-provider/dapl_evd_dto_callb.c --- linux-kernel/dat-provider/dapl_evd_dto_callb.c 2005-05-09 11:07:01.636005000 -0700 +++ linux-kernel2/dat-provider/dapl_evd_dto_callb.c 2005-05-11 11:16:06.298964000 -0700 @@ -71,7 +71,7 @@ void dapl_evd_dto_callback(ib_cq_handle_t cq_handle, void *user_context) { DAPL_EVD *evd_ptr; - DAT_RETURN dat_status; + u32 dat_status; DAPL_EVD_STATE state; dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK, diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_evd_free.c linux-kernel2/dat-provider/dapl_evd_free.c --- linux-kernel/dat-provider/dapl_evd_free.c 2005-05-04 13:38:41.149012000 -0700 +++ linux-kernel2/dat-provider/dapl_evd_free.c 2005-05-11 12:12:01.796013000 -0700 @@ -39,15 +39,14 @@ /* * Destroy a specific instance of the Event Dispatcher */ -DAT_RETURN dapl_evd_free(DAT_EVD_HANDLE evd_handle) +u32 dapl_evd_free(DAT_EVD_HANDLE evd_handle) { DAPL_EVD *evd_ptr; DAPL_CNO *cno_ptr; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_evd_free (%p)\n", evd_handle); - dat_status = DAT_SUCCESS; evd_ptr = (DAPL_EVD *) evd_handle; if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_evd_kcreate.c linux-kernel2/dat-provider/dapl_evd_kcreate.c --- linux-kernel/dat-provider/dapl_evd_kcreate.c 2005-05-10 08:13:28.913024000 -0700 +++ linux-kernel2/dat-provider/dapl_evd_kcreate.c 2005-05-11 12:12:41.439041000 -0700 @@ -48,16 +48,15 @@ * even if it is not required. However, it will not be armed. */ -DAT_RETURN dapl_evd_kcreate(DAT_IA_HANDLE ia_handle, DAT_COUNT evd_min_qlen, - DAT_UPCALL_POLICY upcall_policy, /* ignore */ - const struct dat_upcall_object *upcall, - DAT_EVD_FLAGS evd_flags, - DAT_EVD_HANDLE * evd_handle) +u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_handle, DAT_COUNT evd_min_qlen, + DAT_UPCALL_POLICY upcall_policy, /* ignore */ + const struct dat_upcall_object *upcall, + DAT_EVD_FLAGS evd_flags, DAT_EVD_HANDLE *evd_handle) { DAPL_IA *ia_ptr; DAPL_EVD *evd_ptr; DAPL_CNO *cno_ptr; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; DAT_EVD_HANDLE evdhandle = (DAT_EVD_HANDLE) NULL; dapl_dbg_log(DAPL_DBG_TYPE_API, @@ -69,7 +68,6 @@ DAT_RETURN dapl_evd_kcreate(DAT_IA_HANDL cno_ptr = NULL; evd_ptr = NULL; *evd_handle = NULL; - dat_status = DAT_SUCCESS; if (DAPL_BAD_HANDLE(ia_handle, DAPL_MAGIC_IA)) { dat_status = diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_evd_kquery.c linux-kernel2/dat-provider/dapl_evd_kquery.c --- linux-kernel/dat-provider/dapl_evd_kquery.c 2005-05-10 08:13:28.949014000 -0700 +++ linux-kernel2/dat-provider/dapl_evd_kquery.c 2005-05-11 12:13:16.420015000 -0700 @@ -38,13 +38,10 @@ #include "dapl.h" -DAT_RETURN dapl_evd_kquery(DAT_EVD_HANDLE evd_handle, - struct dat_evd_param *evd_param) +u32 dapl_evd_kquery(DAT_EVD_HANDLE evd_handle, struct dat_evd_param *evd_param) { DAPL_EVD *evd_ptr; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; if (NULL == evd_param) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_evd_modify_upcall.c linux-kernel2/dat-provider/dapl_evd_modify_upcall.c --- linux-kernel/dat-provider/dapl_evd_modify_upcall.c 2005-05-10 08:13:28.739072000 -0700 +++ linux-kernel2/dat-provider/dapl_evd_modify_upcall.c 2005-05-11 12:13:46.318006000 -0700 @@ -43,19 +43,17 @@ * Modify the upcall instance on an EVD */ -DAT_RETURN dapl_evd_modify_upcall(DAT_EVD_HANDLE evd_handle, - DAT_UPCALL_POLICY upcall_policy, /* ignore */ - const struct dat_upcall_object *upcall) +u32 dapl_evd_modify_upcall(DAT_EVD_HANDLE evd_handle, + DAT_UPCALL_POLICY upcall_policy, /* ignore */ + const struct dat_upcall_object *upcall) { DAPL_EVD *evd_ptr; DAPL_CNO *cno_ptr; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_modify_upcall (%p)\n", evd_handle); - dat_status = DAT_SUCCESS; - evd_ptr = (DAPL_EVD *) evd_handle; cno_ptr = evd_ptr->cno_ptr; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_evd_post_se.c linux-kernel2/dat-provider/dapl_evd_post_se.c --- linux-kernel/dat-provider/dapl_evd_post_se.c 2005-05-10 08:13:28.553029000 -0700 +++ linux-kernel2/dat-provider/dapl_evd_post_se.c 2005-05-11 12:14:04.553039000 -0700 @@ -59,14 +59,12 @@ * DAT_INVALID_PARAMETER */ -DAT_RETURN dapl_evd_post_se(DAT_EVD_HANDLE evd_handle, - const struct dat_event *event) +u32 dapl_evd_post_se(DAT_EVD_HANDLE evd_handle, const struct dat_event *event) { DAPL_EVD *evd_ptr; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; evd_ptr = (DAPL_EVD *) evd_handle; - dat_status = DAT_SUCCESS; if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, 0); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_evd_qp_async_error_callb.c linux-kernel2/dat-provider/dapl_evd_qp_async_error_callb.c --- linux-kernel/dat-provider/dapl_evd_qp_async_error_callb.c 2005-05-04 08:19:32.774965000 -0700 +++ linux-kernel2/dat-provider/dapl_evd_qp_async_error_callb.c 2005-05-11 11:16:06.330963000 -0700 @@ -60,7 +60,7 @@ dapl_evd_qp_async_error_callback(ib_erro DAPL_EP *ep_ptr; DAPL_EVD *async_evd; DAT_EVENT_NUMBER async_event; - DAT_RETURN dat_status; + u32 dat_status; ep_ptr = (DAPL_EP *) context; async_evd = (DAPL_EVD *) ep_ptr->header.owner_ia->async_error_evd; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_evd_resize.c linux-kernel2/dat-provider/dapl_evd_resize.c --- linux-kernel/dat-provider/dapl_evd_resize.c 2005-05-09 11:07:01.737008000 -0700 +++ linux-kernel2/dat-provider/dapl_evd_resize.c 2005-05-11 11:16:06.333962000 -0700 @@ -63,12 +63,12 @@ * DAT_INVALID_STATE */ -DAT_RETURN dapl_evd_resize(DAT_EVD_HANDLE evd_handle, DAT_COUNT evd_qlen) +u32 dapl_evd_resize(DAT_EVD_HANDLE evd_handle, DAT_COUNT evd_qlen) { DAPL_IA *ia_ptr; DAPL_EVD *evd_ptr; DAT_COUNT pend_cnt; - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_evd_resize (%p, %d)\n", evd_handle, evd_qlen); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_evd_un_async_error_callb.c linux-kernel2/dat-provider/dapl_evd_un_async_error_callb.c --- linux-kernel/dat-provider/dapl_evd_un_async_error_callb.c 2005-04-20 12:42:07.893006000 -0700 +++ linux-kernel2/dat-provider/dapl_evd_un_async_error_callb.c 2005-05-11 11:16:06.338964000 -0700 @@ -59,7 +59,7 @@ dapl_evd_un_async_error_callback(ib_erro { DAPL_EVD *async_evd; DAT_EVENT_NUMBER async_event; - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK | DAPL_DBG_TYPE_EXCEPTION, "dapl_evd_un_async_error_callback (%p, %p)\n", diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_evd_util.c linux-kernel2/dat-provider/dapl_evd_util.c --- linux-kernel/dat-provider/dapl_evd_util.c 2005-05-11 08:28:12.854000000 -0700 +++ linux-kernel2/dat-provider/dapl_evd_util.c 2005-05-11 12:18:40.507003000 -0700 @@ -41,7 +41,7 @@ static __inline__ void dapl_evd_eh_print_cqe(ib_work_completion_t * cqe); -DAT_RETURN dapl_evd_event_alloc(DAPL_EVD * evd_ptr, DAT_COUNT qlen); +u32 dapl_evd_event_alloc(DAPL_EVD *evd_ptr, DAT_COUNT qlen); /* * actually create the evd. this is called after all parameter checking @@ -49,17 +49,14 @@ DAT_RETURN dapl_evd_event_alloc(DAPL_EVD * to create the default async evd. */ -DAT_RETURN -dapl_evd_internal_create(DAPL_IA * ia_ptr, - DAPL_CNO *cno_ptr, - DAT_COUNT min_qlen, - DAT_EVD_FLAGS evd_flags, DAPL_EVD ** evd_ptr_ptr) +u32 dapl_evd_internal_create(DAPL_IA *ia_ptr, DAPL_CNO *cno_ptr, + DAT_COUNT min_qlen, DAT_EVD_FLAGS evd_flags, + DAPL_EVD **evd_ptr_ptr) { DAPL_EVD *evd_ptr; DAT_COUNT cq_len; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; - dat_status = DAT_SUCCESS; *evd_ptr_ptr = NULL; cq_len = min_qlen; @@ -174,13 +171,11 @@ DAPL_EVD *dapl_evd_alloc(DAPL_IA * ia_pt /* * alloc events into an EVD. */ -DAT_RETURN dapl_evd_event_alloc(DAPL_EVD * evd_ptr, DAT_COUNT qlen) +u32 dapl_evd_event_alloc(DAPL_EVD *evd_ptr, DAT_COUNT qlen) { struct dat_event *event_ptr; DAT_COUNT i; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; /* Allocate EVENTs */ event_ptr = @@ -218,13 +213,13 @@ DAT_RETURN dapl_evd_event_alloc(DAPL_EVD /* * realloc events into an EVD. */ -DAT_RETURN dapl_evd_event_realloc(DAPL_EVD * evd_ptr, DAT_COUNT qlen) +u32 dapl_evd_event_realloc(DAPL_EVD *evd_ptr, DAT_COUNT qlen) { struct dat_event *events; DAT_COUNT old_qlen; DAT_COUNT i; intptr_t diff; - DAT_RETURN dat_status; + u32 dat_status; /* Allocate EVENTs */ events = (struct dat_event *)dapl_os_realloc(evd_ptr->events, @@ -274,13 +269,11 @@ bail: * will clean up all of the internal data structures and report the * error. */ -DAT_RETURN dapl_evd_dealloc(DAPL_EVD * evd_ptr) +u32 dapl_evd_dealloc(DAPL_EVD *evd_ptr) { - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; DAPL_IA *ia_ptr; - dat_status = DAT_SUCCESS; - dapl_os_assert(evd_ptr->header.magic == DAPL_MAGIC_EVD); dapl_os_assert(atomic_read(&evd_ptr->evd_ref_count) == 0); @@ -419,9 +412,10 @@ static struct dat_event *dapl_evd_get_ev * entry to this function. */ -static void dapl_evd_post_event(DAPL_EVD * evd_ptr, const struct dat_event * event_ptr) +static void dapl_evd_post_event(DAPL_EVD *evd_ptr, + const struct dat_event *event_ptr) { - DAT_RETURN dat_status; + u32 dat_status; DAPL_CNO * cno_to_trigger = NULL; dapl_dbg_log(DAPL_DBG_TYPE_EVD, @@ -511,12 +505,12 @@ static struct dat_event *dapl_evd_get_an return event_ptr; } -DAT_RETURN -dapl_evd_post_cr_arrival_event(DAPL_EVD * evd_ptr, - DAT_EVENT_NUMBER event_number, - DAT_SP_HANDLE sp_handle, - DAT_IA_ADDRESS_PTR ia_address_ptr, - DAT_CONN_QUAL conn_qual, DAT_CR_HANDLE cr_handle) +u32 dapl_evd_post_cr_arrival_event(DAPL_EVD *evd_ptr, + DAT_EVENT_NUMBER event_number, + DAT_SP_HANDLE sp_handle, + DAT_IA_ADDRESS_PTR ia_address_ptr, + DAT_CONN_QUAL conn_qual, + DAT_CR_HANDLE cr_handle) { struct dat_event *event_ptr; event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number); @@ -542,12 +536,11 @@ dapl_evd_post_cr_arrival_event(DAPL_EVD return DAT_SUCCESS; } -DAT_RETURN -dapl_evd_post_connection_event(DAPL_EVD * evd_ptr, - DAT_EVENT_NUMBER event_number, - DAT_EP_HANDLE ep_handle, - DAT_COUNT private_data_size, - DAT_PVOID private_data) +u32 dapl_evd_post_connection_event(DAPL_EVD *evd_ptr, + DAT_EVENT_NUMBER event_number, + DAT_EP_HANDLE ep_handle, + DAT_COUNT private_data_size, + DAT_PVOID private_data) { struct dat_event *event_ptr; event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number); @@ -572,10 +565,9 @@ dapl_evd_post_connection_event(DAPL_EVD return DAT_SUCCESS; } -DAT_RETURN -dapl_evd_post_async_error_event(DAPL_EVD * evd_ptr, - DAT_EVENT_NUMBER event_number, - DAT_IA_HANDLE ia_handle) +u32 dapl_evd_post_async_error_event(DAPL_EVD *evd_ptr, + DAT_EVENT_NUMBER event_number, + DAT_IA_HANDLE ia_handle) { struct dat_event *event_ptr; event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number); @@ -598,9 +590,9 @@ dapl_evd_post_async_error_event(DAPL_EVD return DAT_SUCCESS; } -DAT_RETURN -dapl_evd_post_software_event(DAPL_EVD * evd_ptr, - DAT_EVENT_NUMBER event_number, DAT_PVOID pointer) +u32 dapl_evd_post_software_event(DAPL_EVD *evd_ptr, + DAT_EVENT_NUMBER event_number, + DAT_PVOID pointer) { struct dat_event *event_ptr; event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number); @@ -775,10 +767,9 @@ dapl_evd_cqe_to_event(DAPL_EVD * evd_ptr * Status of operation * */ -DAT_RETURN dapl_evd_cq_poll_to_event(DAPL_EVD * evd_ptr, - struct dat_event *event) +u32 dapl_evd_cq_poll_to_event(DAPL_EVD *evd_ptr, struct dat_event *event) { - DAT_RETURN dat_status; + u32 dat_status; ib_work_completion_t cur_cqe; dat_status = dapl_ib_completion_poll(evd_ptr->header.owner_ia->hca_ptr, diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_evd_util.h linux-kernel2/dat-provider/dapl_evd_util.h --- linux-kernel/dat-provider/dapl_evd_util.h 2005-05-10 08:13:29.134055000 -0700 +++ linux-kernel2/dat-provider/dapl_evd_util.h 2005-05-11 12:21:06.564007000 -0700 @@ -40,17 +40,16 @@ #include "dapl.h" -DAT_RETURN -dapl_evd_internal_create(DAPL_IA * ia_ptr, DAPL_CNO * cno_ptr, - DAT_COUNT min_qlen, - DAT_EVD_FLAGS evd_flags, DAPL_EVD ** evd_ptr_ptr); +u32 dapl_evd_internal_create(DAPL_IA *ia_ptr, DAPL_CNO *cno_ptr, + DAT_COUNT min_qlen, DAT_EVD_FLAGS evd_flags, + DAPL_EVD **evd_ptr_ptr); DAPL_EVD *dapl_evd_alloc(DAPL_IA * ia_ptr, DAPL_CNO * cno_ptr, DAT_EVD_FLAGS evd_flags, DAT_COUNT qlen); -DAT_RETURN dapl_evd_dealloc(DAPL_EVD * evd_ptr); +u32 dapl_evd_dealloc(DAPL_EVD *evd_ptr); -DAT_RETURN dapl_evd_event_realloc(DAPL_EVD * evd_ptr, DAT_COUNT qlen); +u32 dapl_evd_event_realloc(DAPL_EVD *evd_ptr, DAT_COUNT qlen); /* * Each of these functions will retrieve a free event from @@ -63,30 +62,26 @@ DAT_RETURN dapl_evd_event_realloc(DAPL_E * DAT_SUCCESS otherwise. */ -DAT_RETURN -dapl_evd_post_cr_arrival_event(DAPL_EVD * evd_ptr, - DAT_EVENT_NUMBER event_number, - DAT_SP_HANDLE sp_handle, - DAT_IA_ADDRESS_PTR ia_address_ptr, - DAT_CONN_QUAL conn_qual, - DAT_CR_HANDLE cr_handle); - -DAT_RETURN -dapl_evd_post_connection_event(DAPL_EVD * evd_ptr, - DAT_EVENT_NUMBER event_number, - DAT_EP_HANDLE ep_handle, - DAT_COUNT private_data_size, - DAT_PVOID private_data); - -DAT_RETURN -dapl_evd_post_async_error_event(DAPL_EVD * evd_ptr, - DAT_EVENT_NUMBER event_number, - DAT_IA_HANDLE ia_handle); - -DAT_RETURN -dapl_evd_post_software_event(DAPL_EVD * evd_ptr, - DAT_EVENT_NUMBER event_number, DAT_PVOID pointer); - +u32 dapl_evd_post_cr_arrival_event(DAPL_EVD *evd_ptr, + DAT_EVENT_NUMBER event_number, + DAT_SP_HANDLE sp_handle, + DAT_IA_ADDRESS_PTR ia_address_ptr, + DAT_CONN_QUAL conn_qual, + DAT_CR_HANDLE cr_handle); + +u32 dapl_evd_post_connection_event(DAPL_EVD *evd_ptr, + DAT_EVENT_NUMBER event_number, + DAT_EP_HANDLE ep_handle, + DAT_COUNT private_data_size, + DAT_PVOID private_data); + +u32 dapl_evd_post_async_error_event(DAPL_EVD *evd_ptr, + DAT_EVENT_NUMBER event_number, + DAT_IA_HANDLE ia_handle); + +u32 dapl_evd_post_software_event(DAPL_EVD *evd_ptr, + DAT_EVENT_NUMBER event_number, + DAT_PVOID pointer); /************************************* * dapl internal callbacks functions * @@ -111,7 +106,7 @@ extern void dapl_evd_cq_async_error_call extern void dapl_evd_qp_async_error_callback(ib_error_record_t * cause_ptr, void *context); -extern DAT_RETURN dapl_evd_cq_poll_to_event(DAPL_EVD * evd_ptr, - struct dat_event *event); +extern u32 dapl_evd_cq_poll_to_event(DAPL_EVD *evd_ptr, + struct dat_event *event); #endif diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_get_consumer_context.c linux-kernel2/dat-provider/dapl_get_consumer_context.c --- linux-kernel/dat-provider/dapl_get_consumer_context.c 2005-04-20 12:42:06.961003000 -0700 +++ linux-kernel2/dat-provider/dapl_get_consumer_context.c 2005-05-11 12:21:23.159014000 -0700 @@ -55,14 +55,11 @@ * DAT_SUCCESS * DAT_INVALID_PARAMETER */ -DAT_RETURN -dapl_get_consumer_context(DAT_HANDLE dat_handle, DAT_CONTEXT * context) +u32 dapl_get_consumer_context(DAT_HANDLE dat_handle, DAT_CONTEXT *context) { - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; DAPL_HEADER *header; - dat_status = DAT_SUCCESS; - header = (DAPL_HEADER *) dat_handle; if (((header) == NULL) || ((unsigned long)(header) & 3) || diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_get_handle_type.c linux-kernel2/dat-provider/dapl_get_handle_type.c --- linux-kernel/dat-provider/dapl_get_handle_type.c 2005-04-20 12:42:08.013010000 -0700 +++ linux-kernel2/dat-provider/dapl_get_handle_type.c 2005-05-11 12:21:50.885012000 -0700 @@ -56,14 +56,11 @@ * DAT_INVALID_PARAMETER */ -DAT_RETURN -dapl_get_handle_type(DAT_HANDLE dat_handle, DAT_HANDLE_TYPE * handle_type) +u32 dapl_get_handle_type(DAT_HANDLE dat_handle, DAT_HANDLE_TYPE *handle_type) { - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; DAPL_HEADER *header; - dat_status = DAT_SUCCESS; - header = (DAPL_HEADER *) dat_handle; if (((header) == NULL) || ((unsigned long)(header) & 3) || diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl.h linux-kernel2/dat-provider/dapl.h --- linux-kernel/dat-provider/dapl.h 2005-05-11 11:08:04.453960000 -0700 +++ linux-kernel2/dat-provider/dapl.h 2005-05-11 13:23:31.584000000 -0700 @@ -138,8 +138,6 @@ typedef enum dapl_qp_state { #define DEBUG_IS_BAD_HANDLE(h, magicNum) (0) #endif -#define DAT_ERROR(Type, SubType) ((DAT_RETURN)(DAT_CLASS_ERROR | Type | SubType)) - /********************************************************************* * * * Typedefs * @@ -477,275 +475,245 @@ typedef char DAPL_HCA_NAME[DAPL_HCA_NAME * DAT API functions */ -extern DAT_RETURN dapl_ia_open(const DAT_NAME_PTR, DAT_COUNT, DAT_EVD_HANDLE *, - DAT_IA_HANDLE *); +extern u32 dapl_ia_open(const DAT_NAME_PTR, DAT_COUNT, DAT_EVD_HANDLE *, + DAT_IA_HANDLE *); -extern DAT_RETURN dapl_ia_close(DAT_IA_HANDLE, DAT_CLOSE_FLAGS); +extern u32 dapl_ia_close(DAT_IA_HANDLE, DAT_CLOSE_FLAGS); -extern DAT_RETURN dapl_ia_query(DAT_IA_HANDLE, - DAT_EVD_HANDLE *, - struct dat_ia_attr *, - struct dat_provider_attr *); +extern u32 dapl_ia_query(DAT_IA_HANDLE, DAT_EVD_HANDLE *, struct dat_ia_attr *, + struct dat_provider_attr *); /* helper functions */ -extern DAT_RETURN dapl_set_consumer_context(DAT_HANDLE, /* dat handle */ - DAT_CONTEXT); /* context */ +extern u32 dapl_set_consumer_context(DAT_HANDLE handle, DAT_CONTEXT context); -extern DAT_RETURN dapl_get_consumer_context(DAT_HANDLE, /* dat handle */ - DAT_CONTEXT *); /* context */ +extern u32 dapl_get_consumer_context(DAT_HANDLE handle, DAT_CONTEXT *context); -extern DAT_RETURN dapl_get_handle_type(DAT_HANDLE, DAT_HANDLE_TYPE *); +extern u32 dapl_get_handle_type(DAT_HANDLE handle, DAT_HANDLE_TYPE *context); /* CR Functions */ -extern DAT_RETURN dapl_cr_query(DAT_CR_HANDLE, struct dat_cr_param *); +extern u32 dapl_cr_query(DAT_CR_HANDLE, struct dat_cr_param *); -extern DAT_RETURN dapl_cr_accept(DAT_CR_HANDLE, /* cr_handle */ - DAT_EP_HANDLE, /* ep_handle */ - DAT_COUNT, /* private_data_size */ - const DAT_PVOID); /* private_data */ +extern u32 dapl_cr_accept(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, + DAT_COUNT private_data_size, + const DAT_PVOID private_data); -extern DAT_RETURN dapl_cr_reject(DAT_CR_HANDLE); +extern u32 dapl_cr_reject(DAT_CR_HANDLE cr_handle); -extern DAT_RETURN dapl_cr_handoff(DAT_CR_HANDLE, /* cr_handle */ - DAT_CONN_QUAL); /* handoff */ +extern u32 dapl_cr_handoff(DAT_CR_HANDLE cr_handle, DAT_CONN_QUAL handoff); /* EVD Functions */ -extern DAT_RETURN dapl_ia_memtype_hint(DAT_IA_HANDLE, /* ia_handle */ - DAT_MEM_TYPE, /* mem_type */ - DAT_VLEN, /* length */ - DAT_MEM_OPT, /* mem_optimization */ - DAT_VLEN *, /* suggested_length */ - DAT_VADDR *); /* suggested_alignment */ +extern u32 dapl_ia_memtype_hint(DAT_IA_HANDLE ia_handle, DAT_MEM_TYPE mem_type, + DAT_VLEN length, DAT_MEM_OPT mem_optimization, + DAT_VLEN *suggested_length, + DAT_VADDR *suggested_alignment); -extern DAT_RETURN dapl_evd_kcreate(DAT_IA_HANDLE, /* ia_handle */ - DAT_COUNT, /* evd_min_qlen */ - DAT_UPCALL_POLICY, /* upcall_policy */ - const struct dat_upcall_object *,/* upcall */ - DAT_EVD_FLAGS, /* evd_flags */ - DAT_EVD_HANDLE *); /* evd_handle */ +extern u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_handle, DAT_COUNT evd_min_qlen, + DAT_UPCALL_POLICY upcall_policy, + const struct dat_upcall_object * upcall, + DAT_EVD_FLAGS evd_flags, + DAT_EVD_HANDLE * evd_handle); -extern DAT_RETURN dapl_evd_kquery(DAT_EVD_HANDLE, /* evd_handle */ - struct dat_evd_param *); /* evd_args */ +extern u32 dapl_evd_kquery(DAT_EVD_HANDLE evd_handle, + struct dat_evd_param *evd_args); -extern DAT_RETURN dapl_evd_modify_upcall(DAT_EVD_HANDLE, /* evd_handle */ - DAT_UPCALL_POLICY, /* upcall_policy */ - const struct dat_upcall_object *); /* upcall */ +extern u32 dapl_evd_modify_upcall(DAT_EVD_HANDLE evd_handle, + DAT_UPCALL_POLICY upcall_policy, + const struct dat_upcall_object * upcall); -extern DAT_RETURN dapl_evd_enable(DAT_EVD_HANDLE); /* evd_handle */ +extern u32 dapl_evd_enable(DAT_EVD_HANDLE evd_handle); -extern DAT_RETURN dapl_evd_disable(DAT_EVD_HANDLE); /* evd_handle */ +extern u32 dapl_evd_disable(DAT_EVD_HANDLE evd_handle); -extern DAT_RETURN dapl_evd_resize(DAT_EVD_HANDLE, /* evd_handle */ - DAT_COUNT); /* evd_qlen */ +extern u32 dapl_evd_resize(DAT_EVD_HANDLE evd_handle, DAT_COUNT evd_qlen); -extern DAT_RETURN dapl_evd_post_se(DAT_EVD_HANDLE, /* evd_handle */ - const struct dat_event *); /* event */ +extern u32 dapl_evd_post_se(DAT_EVD_HANDLE evd_handle, + const struct dat_event *event); -extern DAT_RETURN dapl_evd_dequeue(DAT_EVD_HANDLE, /* evd_handle */ - struct dat_event *); /* event */ +extern u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_handle, struct dat_event *event); -extern DAT_RETURN dapl_evd_free(DAT_EVD_HANDLE); +extern u32 dapl_evd_free(DAT_EVD_HANDLE evd_handle); -extern DAT_RETURN dapl_evd_set_unwaitable(DAT_EVD_HANDLE evd_handle); +extern u32 dapl_evd_set_unwaitable(DAT_EVD_HANDLE evd_handle); -extern DAT_RETURN dapl_evd_clear_unwaitable(DAT_EVD_HANDLE evd_handle); +extern u32 dapl_evd_clear_unwaitable(DAT_EVD_HANDLE evd_handle); /* EP functions */ -extern DAT_RETURN dapl_ep_create(DAT_IA_HANDLE, /* ia_handle */ - DAT_PZ_HANDLE, /* pz_handle */ - DAT_EVD_HANDLE, /* in_dto_completion_evd_handle */ - DAT_EVD_HANDLE, /* out_dto_completion_evd_handle */ - DAT_EVD_HANDLE, /* connect_evd_handle */ - const struct dat_ep_attr *, /* ep_parameters */ - DAT_EP_HANDLE *); /* ep_handle */ - -extern DAT_RETURN dapl_ep_query(DAT_EP_HANDLE, struct dat_ep_param *); - -extern DAT_RETURN dapl_ep_modify(DAT_EP_HANDLE, /* ep_handle */ - DAT_EP_PARAM_MASK, /* ep_args_mask */ - const struct dat_ep_param *); /* ep_args */ - -extern DAT_RETURN dapl_ep_connect(DAT_EP_HANDLE, /* ep_handle */ - DAT_IA_ADDRESS_PTR, /* remote_ia_address */ - DAT_CONN_QUAL, /* remote_conn_qual */ - DAT_TIMEOUT, /* timeout */ - DAT_COUNT, /* private_data_size */ - const DAT_PVOID, /* private_data */ - DAT_QOS, /* quality_of_service */ - DAT_CONNECT_FLAGS); /* connect_flags */ - -extern DAT_RETURN dapl_ep_dup_connect(DAT_EP_HANDLE, /* ep_handle */ - DAT_EP_HANDLE, /* ep_dup_handle */ - DAT_TIMEOUT, /* timeout */ - DAT_COUNT, /* private_data_size */ - const DAT_PVOID, /* private_data */ - DAT_QOS); /* quality_of_service */ - -extern DAT_RETURN dapl_ep_disconnect(DAT_EP_HANDLE, /* ep_handle */ - DAT_CLOSE_FLAGS); /* close_flags */ - -extern DAT_RETURN dapl_ep_post_send(DAT_EP_HANDLE, /* ep_handle */ - DAT_COUNT, /* num_segments */ - struct dat_lmr_triplet *, /* local_iov */ - DAT_DTO_COOKIE, /* user_cookie */ - DAT_COMPLETION_FLAGS); /* completion_flags */ - -extern DAT_RETURN dapl_ep_post_recv(DAT_EP_HANDLE, /* ep_handle */ - DAT_COUNT, /* num_segments */ - struct dat_lmr_triplet *, /* local_iov */ - DAT_DTO_COOKIE, /* user_cookie */ - DAT_COMPLETION_FLAGS); /* completion_flags */ - -extern DAT_RETURN dapl_ep_post_rdma_read(DAT_EP_HANDLE, /* ep_handle */ - DAT_COUNT, /* num_segments */ - struct dat_lmr_triplet *, /* local_iov */ - DAT_DTO_COOKIE, /* user_cookie */ - const struct dat_rmr_triplet *, /* remote_iov */ - DAT_COMPLETION_FLAGS); /* completion_flags */ - -extern DAT_RETURN dapl_ep_post_rdma_write(DAT_EP_HANDLE, /* ep_handle */ - DAT_COUNT, /* num_segments */ - struct dat_lmr_triplet *, /* local_iov */ - DAT_DTO_COOKIE, /* user_cookie */ - const struct dat_rmr_triplet *, /* remote_iov */ - DAT_COMPLETION_FLAGS); /* completion_flags */ - -extern DAT_RETURN dapl_ep_get_status(DAT_EP_HANDLE, /* ep_handle */ - DAT_EP_STATE *, /* ep_state */ - boolean_t *, /* in_dto_idle */ - boolean_t *); /* out_dto_idle */ - -extern DAT_RETURN dapl_ep_free(DAT_EP_HANDLE); /* ep_handle */ - -extern DAT_RETURN dapl_ep_reset(DAT_EP_HANDLE); /* ep_handle */ - -extern DAT_RETURN dapl_ep_create_with_srq(DAT_IA_HANDLE, /* ia_handle */ - DAT_PZ_HANDLE, /* pz_handle */ - DAT_EVD_HANDLE, /* recv_evd_handle */ - DAT_EVD_HANDLE, /* request_evd_handle */ - DAT_EVD_HANDLE, /* connect_evd_handle */ - DAT_SRQ_HANDLE, /* srq_handle */ - const struct dat_ep_attr *, /* ep_attributes */ - DAT_EP_HANDLE *); /* ep_handle */ - -extern DAT_RETURN dapl_ep_recv_query(DAT_EP_HANDLE, /* ep_handle */ - DAT_COUNT *, /* nbufs_allocated */ - DAT_COUNT *); /* bufs_alloc_span */ - -extern DAT_RETURN dapl_ep_set_watermark(DAT_EP_HANDLE, /* ep_handle */ - DAT_COUNT, /* soft_high_watermark */ - DAT_COUNT); /* hard_high_watermark */ +extern u32 dapl_ep_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, + DAT_EVD_HANDLE in_dto_completion_evd_handle, + DAT_EVD_HANDLE out_dto_completion_evd_handle, + DAT_EVD_HANDLE connect_evd_handle, + const struct dat_ep_attr *ep_parameters, + DAT_EP_HANDLE *ep_handle); + +extern u32 dapl_ep_query(DAT_EP_HANDLE, struct dat_ep_param *); + +extern u32 dapl_ep_modify(DAT_EP_HANDLE ep_handle, + DAT_EP_PARAM_MASK ep_args_mask, + const struct dat_ep_param *ep_args); + +extern u32 dapl_ep_connect(DAT_EP_HANDLE ep_handle, + DAT_IA_ADDRESS_PTR remote_ia_address, + DAT_CONN_QUAL remote_conn_qual, + DAT_TIMEOUT timeout, + DAT_COUNT private_data_size, + const DAT_PVOID private_data, + DAT_QOS quality_of_service, + DAT_CONNECT_FLAGS connect_flags); + +extern u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, + DAT_EP_HANDLE ep_dup_handle, DAT_TIMEOUT timeout, + DAT_COUNT private_data_size, + const DAT_PVOID private_data, + DAT_QOS quality_of_service); + +extern u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_handle, + DAT_CLOSE_FLAGS close_flags); + +extern u32 dapl_ep_post_send(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, + struct dat_lmr_triplet *local_iov, + DAT_DTO_COOKIE user_cookie, + DAT_COMPLETION_FLAGS completion_flags); + +extern u32 dapl_ep_post_recv(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, + struct dat_lmr_triplet *local_iov, + DAT_DTO_COOKIE user_cookie, + DAT_COMPLETION_FLAGS completion_flags); + +extern u32 dapl_ep_post_rdma_read(DAT_EP_HANDLE ep_handle, + DAT_COUNT num_segments, + struct dat_lmr_triplet *local_iov, + DAT_DTO_COOKIE user_cookie, + const struct dat_rmr_triplet *remote_iov, + DAT_COMPLETION_FLAGS completion_flags); + +extern u32 dapl_ep_post_rdma_write(DAT_EP_HANDLE ep_handle, + DAT_COUNT num_segments, + struct dat_lmr_triplet *local_iov, + DAT_DTO_COOKIE user_cookie, + const struct dat_rmr_triplet *remote_iov, + DAT_COMPLETION_FLAGS completion_flags); + +extern u32 dapl_ep_get_status(DAT_EP_HANDLE ep_handle, DAT_EP_STATE *ep_state, + boolean_t *in_dto_idle, boolean_t *out_dto_idle); + +extern u32 dapl_ep_free(DAT_EP_HANDLE ep_handle); + +extern u32 dapl_ep_reset(DAT_EP_HANDLE ep_handle); + +extern u32 dapl_ep_create_with_srq(DAT_IA_HANDLE ia_handle, + DAT_PZ_HANDLE pz_handle, + DAT_EVD_HANDLE recv_evd_handle, + DAT_EVD_HANDLE request_evd_handle, + DAT_EVD_HANDLE connect_evd_handle, + DAT_SRQ_HANDLE srq_handle, + const struct dat_ep_attr *ep_attributes, + DAT_EP_HANDLE *ep_handle); + +extern u32 dapl_ep_recv_query(DAT_EP_HANDLE ep_handle, + DAT_COUNT *nbufs_allocated, + DAT_COUNT *bufs_alloc_span); + +extern u32 dapl_ep_set_watermark(DAT_EP_HANDLE ep_handle, + DAT_COUNT soft_high_watermark, + DAT_COUNT hard_high_watermark); /* LMR functions */ -extern DAT_RETURN dapl_lmr_kcreate(DAT_IA_HANDLE, /* ia_handle */ - DAT_MEM_TYPE, /* mem_type */ - DAT_REGION_DESCRIPTION, /* region_description */ - DAT_VLEN, /* length */ - DAT_PZ_HANDLE, /* pz_handle */ - DAT_MEM_PRIV_FLAGS, /* privileges */ - DAT_MEM_OPT, /* optimization */ - DAT_LMR_HANDLE *, /* lmr_handle */ - DAT_LMR_CONTEXT *, /* lmr_context */ - DAT_RMR_CONTEXT *, /* rmr_context */ - DAT_VLEN *, /* registered_length */ - DAT_VADDR *); /* registered_address */ - -extern DAT_RETURN dapl_lmr_query(DAT_LMR_HANDLE, struct dat_lmr_param *); - -extern DAT_RETURN dapl_lmr_free(DAT_LMR_HANDLE); - -extern DAT_RETURN dapl_lmr_sync_rdma_read(DAT_IA_HANDLE, /* ia_handle */ - const struct dat_lmr_triplet *, /* local_segments */ - DAT_VLEN); /* num_segments */ - -extern DAT_RETURN dapl_lmr_sync_rdma_write(DAT_IA_HANDLE, /* ia_handle */ - const struct dat_lmr_triplet *, /* local_segments */ - DAT_VLEN); /* num_segments */ +extern u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_handle, DAT_MEM_TYPE mem_type, + DAT_REGION_DESCRIPTION region_description, + DAT_VLEN length, DAT_PZ_HANDLE pz_handle, + DAT_MEM_PRIV_FLAGS privileges, + DAT_MEM_OPT optimization, + DAT_LMR_HANDLE *lmr_handle, + DAT_LMR_CONTEXT *lmr_context, + DAT_RMR_CONTEXT *rmr_context, + DAT_VLEN *registered_length, + DAT_VADDR *registered_address); + +extern u32 dapl_lmr_query(DAT_LMR_HANDLE, struct dat_lmr_param *); + +extern u32 dapl_lmr_free(DAT_LMR_HANDLE); + +extern u32 dapl_lmr_sync_rdma_read(DAT_IA_HANDLE ia_handle, + const struct dat_lmr_triplet *local_segments, + DAT_VLEN num_segments); + +extern u32 dapl_lmr_sync_rdma_write(DAT_IA_HANDLE ia_handle, + const struct dat_lmr_triplet *local_segments, + DAT_VLEN num_segments); /* RMR Functions */ -extern DAT_RETURN dapl_rmr_create(DAT_PZ_HANDLE, /* pz_handle */ - DAT_RMR_HANDLE *); /* rmr_handle */ +extern u32 dapl_rmr_create(DAT_PZ_HANDLE pz_handle, DAT_RMR_HANDLE *rmr_handle); -extern DAT_RETURN dapl_rmr_query(DAT_RMR_HANDLE, /* rmr_handle */ - struct dat_rmr_param *); /* rmr_args */ +extern u32 dapl_rmr_query(DAT_RMR_HANDLE rmr_handle, + struct dat_rmr_param *rmr_args); -extern DAT_RETURN dapl_rmr_bind(DAT_RMR_HANDLE, /* rmr_handle */ - const struct dat_lmr_triplet *, /* lmr_triplet */ - DAT_MEM_PRIV_FLAGS, /* mem_priv */ - DAT_EP_HANDLE, /* ep_handle */ - DAT_RMR_COOKIE, /* user_cookie */ - DAT_COMPLETION_FLAGS, /* completion_flags */ - DAT_RMR_CONTEXT *); /* context */ +extern u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_handle, + const struct dat_lmr_triplet *lmr_triplet, + DAT_MEM_PRIV_FLAGS mem_priv, DAT_EP_HANDLE ep_handle, + DAT_RMR_COOKIE user_cookie, + DAT_COMPLETION_FLAGS completion_flags, + DAT_RMR_CONTEXT *context); -extern DAT_RETURN dapl_rmr_free(DAT_RMR_HANDLE); +extern u32 dapl_rmr_free(DAT_RMR_HANDLE); /* PSP Functions */ -extern DAT_RETURN dapl_psp_create(DAT_IA_HANDLE, /* ia_handle */ - DAT_CONN_QUAL, /* conn_qual */ - DAT_EVD_HANDLE, /* evd_handle */ - DAT_PSP_FLAGS, /* psp_flags */ - DAT_PSP_HANDLE *); /* psp_handle */ - -extern DAT_RETURN dapl_psp_create_any(DAT_IA_HANDLE, /* ia_handle */ - DAT_CONN_QUAL *, /* conn_qual */ - DAT_EVD_HANDLE, /* evd_handle */ - DAT_PSP_FLAGS, /* psp_flags */ - DAT_PSP_HANDLE *); /* psp_handle */ +extern u32 dapl_psp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, + DAT_EVD_HANDLE evd_handle, DAT_PSP_FLAGS psp_flags, + DAT_PSP_HANDLE *psp_handle); + +extern u32 dapl_psp_create_any(DAT_IA_HANDLE ia_handle, + DAT_CONN_QUAL *conn_qual, + DAT_EVD_HANDLE evd_handle, + DAT_PSP_FLAGS psp_flags, + DAT_PSP_HANDLE *psp_handle); -extern DAT_RETURN dapl_psp_query(DAT_PSP_HANDLE, struct dat_psp_param *); +extern u32 dapl_psp_query(DAT_PSP_HANDLE, struct dat_psp_param *); -extern DAT_RETURN dapl_psp_free(DAT_PSP_HANDLE); /* psp_handle */ +extern u32 dapl_psp_free(DAT_PSP_HANDLE psp_handle); /* RSP Functions */ -extern DAT_RETURN dapl_rsp_create(DAT_IA_HANDLE, /* ia_handle */ - DAT_CONN_QUAL, /* conn_qual */ - DAT_EP_HANDLE, /* ep_handle */ - DAT_EVD_HANDLE, /* evd_handle */ - DAT_RSP_HANDLE *); /* rsp_handle */ +extern u32 dapl_rsp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, + DAT_EP_HANDLE ep_handle, DAT_EVD_HANDLE evd_handle, + DAT_RSP_HANDLE *rsp_handle); -extern DAT_RETURN dapl_rsp_query(DAT_RSP_HANDLE, struct dat_rsp_param *); +extern u32 dapl_rsp_query(DAT_RSP_HANDLE, struct dat_rsp_param *); -extern DAT_RETURN dapl_rsp_free(DAT_RSP_HANDLE); /* rsp_handle */ +extern u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_handle); /* PZ Functions */ -extern DAT_RETURN dapl_pz_create(DAT_IA_HANDLE, /* ia_handle */ - DAT_PZ_HANDLE *); /* pz_handle */ +extern u32 dapl_pz_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE *pz_handle); -extern DAT_RETURN dapl_pz_query(DAT_PZ_HANDLE, /* pz_handle */ - struct dat_pz_param *); /* pz_args */ +extern u32 dapl_pz_query(DAT_PZ_HANDLE pz_handle, struct dat_pz_param *pz_args); -extern DAT_RETURN dapl_pz_free(DAT_PZ_HANDLE); /* pz_handle */ +extern u32 dapl_pz_free(DAT_PZ_HANDLE pz_handle); /* SRQ functions */ -extern DAT_RETURN dapl_srq_create(DAT_IA_HANDLE, /* ia_handle */ - DAT_PZ_HANDLE, /* pz_handle */ - struct dat_srq_attr *, /* srq_attr */ - DAT_SRQ_HANDLE *); /* srq_handle */ - -extern DAT_RETURN dapl_srq_free(DAT_SRQ_HANDLE); /* srq_handle */ - -extern DAT_RETURN dapl_srq_post_recv(DAT_SRQ_HANDLE, /* srq_handle */ - DAT_COUNT, /* num_segments */ - struct dat_lmr_triplet *, /* local_iov */ - DAT_DTO_COOKIE); /* user_cookie */ +extern u32 dapl_srq_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, + struct dat_srq_attr *srq_attr, + DAT_SRQ_HANDLE *srq_handle); + +extern u32 dapl_srq_free(DAT_SRQ_HANDLE srq_handle); + +extern u32 dapl_srq_post_recv(DAT_SRQ_HANDLE srq_handle, DAT_COUNT num_segments, + struct dat_lmr_triplet *local_iov, + DAT_DTO_COOKIE user_cookie); -extern DAT_RETURN dapl_srq_query(DAT_SRQ_HANDLE, /* srq_handle */ - struct dat_srq_param *); /* srq_param */ +extern u32 dapl_srq_query(DAT_SRQ_HANDLE srq_handle, + struct dat_srq_param *srq_param); -extern DAT_RETURN dapl_srq_resize(DAT_SRQ_HANDLE, /* srq_handle */ - DAT_COUNT); /* srq_max_recv_dto */ +extern u32 dapl_srq_resize(DAT_SRQ_HANDLE srq_handle, + DAT_COUNT srq_max_recv_dto); -extern DAT_RETURN dapl_srq_set_lw(DAT_SRQ_HANDLE, /* srq_handle */ - DAT_COUNT); /* low_watermark */ +extern u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, DAT_COUNT low_watermark); /* * DAPL internal utility function prototpyes diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_hash.c linux-kernel2/dat-provider/dapl_hash.c --- linux-kernel/dat-provider/dapl_hash.c 2005-05-11 08:28:12.964000000 -0700 +++ linux-kernel2/dat-provider/dapl_hash.c 2005-05-11 13:24:53.952031000 -0700 @@ -320,15 +320,14 @@ dapl_hash_delete_element(DAPL_HASH_ELEM /* * Create a new hash table with at least 'table_size' hash buckets. */ -DAT_RETURN dapl_hash_create(DAT_COUNT table_size, DAPL_HASH_TABLE ** pp_table) +u32 dapl_hash_create(DAT_COUNT table_size, DAPL_HASH_TABLE **pp_table) { DAPL_HASH_TABLE *p_table; DAT_COUNT table_length = table_size * sizeof(DAPL_HASH_ELEM); - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; DAT_COUNT i; dapl_os_assert(pp_table); - dat_status = DAT_SUCCESS; /* Allocate hash table */ p_table = kmalloc(sizeof(DAPL_HASH_TABLE), GFP_ATOMIC); @@ -365,7 +364,7 @@ DAT_RETURN dapl_hash_create(DAT_COUNT ta /* * Destroy a hash table */ -DAT_RETURN dapl_hash_free(DAPL_HASH_TABLE * p_table) +u32 dapl_hash_free(DAPL_HASH_TABLE *p_table) { dapl_os_assert(p_table && p_table->table); @@ -380,7 +379,7 @@ DAT_RETURN dapl_hash_free(DAPL_HASH_TABL * Returns the number of elements stored in the table */ -DAT_RETURN dapl_hash_size(DAPL_HASH_TABLE * p_table, DAT_COUNT * p_size) +u32 dapl_hash_size(DAPL_HASH_TABLE *p_table, DAT_COUNT *p_size) { dapl_os_assert(p_table && p_size); @@ -394,14 +393,12 @@ DAT_RETURN dapl_hash_size(DAPL_HASH_TABL * Duplicates are not expected, and return in error, having done nothing. */ -DAT_RETURN -dapl_hash_insert(DAPL_HASH_TABLE * p_table, - DAPL_HASH_KEY key, DAPL_HASH_DATA data) +u32 dapl_hash_insert(DAPL_HASH_TABLE *p_table, DAPL_HASH_KEY key, + DAPL_HASH_DATA data) { - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; dapl_os_assert(p_table); - dat_status = DAT_SUCCESS; spin_lock(&p_table->lock); if (!dapl_hash_add(p_table, key, data, FALSE, NULL)) { @@ -419,11 +416,10 @@ dapl_hash_insert(DAPL_HASH_TABLE * p_tab * data is returned in the DAPL_HASH_DATA * pointer if that pointer is not NULL. */ -DAT_RETURN -dapl_hash_search(DAPL_HASH_TABLE * p_table, - DAPL_HASH_KEY key, DAPL_HASH_DATA * p_data) +u32 dapl_hash_search(DAPL_HASH_TABLE *p_table, DAPL_HASH_KEY key, + DAPL_HASH_DATA *p_data) { - DAT_RETURN dat_status; + u32 dat_status; void *olddatum; DAPL_HASH_ELEM *found; @@ -444,11 +440,10 @@ dapl_hash_search(DAPL_HASH_TABLE * p_tab return dat_status; } -DAT_RETURN -dapl_hash_remove(DAPL_HASH_TABLE * p_table, - DAPL_HASH_KEY key, DAPL_HASH_DATA * p_data) +u32 dapl_hash_remove(DAPL_HASH_TABLE *p_table, DAPL_HASH_KEY key, + DAPL_HASH_DATA *p_data) { - DAT_RETURN dat_status; + u32 dat_status; DAPL_HASH_KEY hashValue, save_key = key; dapl_os_assert(p_table); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_hash.h linux-kernel2/dat-provider/dapl_hash.h --- linux-kernel/dat-provider/dapl_hash.h 2005-04-20 12:42:08.701003000 -0700 +++ linux-kernel2/dat-provider/dapl_hash.h 2005-05-11 13:25:44.049020000 -0700 @@ -67,23 +67,19 @@ * * *********************************************************************/ -extern DAT_RETURN -dapl_hash_create(DAT_COUNT capacity, DAPL_HASH_TABLE ** pp_table); +extern u32 dapl_hash_create(DAT_COUNT capacity, DAPL_HASH_TABLE **pp_table); -extern DAT_RETURN dapl_hash_free(DAPL_HASH_TABLE * p_table); +extern u32 dapl_hash_free(DAPL_HASH_TABLE *p_table); -extern DAT_RETURN dapl_hash_size(DAPL_HASH_TABLE * p_table, DAT_COUNT * p_size); +extern u32 dapl_hash_size(DAPL_HASH_TABLE *p_table, DAT_COUNT *p_size); -extern DAT_RETURN -dapl_hash_insert(DAPL_HASH_TABLE * p_table, - DAPL_HASH_KEY key, DAPL_HASH_DATA data); +extern u32 dapl_hash_insert(DAPL_HASH_TABLE *p_table, DAPL_HASH_KEY key, + DAPL_HASH_DATA data); -extern DAT_RETURN -dapl_hash_search(DAPL_HASH_TABLE * p_table, - DAPL_HASH_KEY key, DAPL_HASH_DATA * p_data); +extern u32 dapl_hash_search(DAPL_HASH_TABLE *p_table, DAPL_HASH_KEY key, + DAPL_HASH_DATA *p_data); -extern DAT_RETURN -dapl_hash_remove(DAPL_HASH_TABLE * p_table, - DAPL_HASH_KEY key, DAPL_HASH_DATA * p_data); +extern u32 dapl_hash_remove(DAPL_HASH_TABLE *p_table, DAPL_HASH_KEY key, + DAPL_HASH_DATA *p_data); #endif /* DAPL_HASH_H */ diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ia_close.c linux-kernel2/dat-provider/dapl_ia_close.c --- linux-kernel/dat-provider/dapl_ia_close.c 2005-04-20 12:42:07.789026000 -0700 +++ linux-kernel2/dat-provider/dapl_ia_close.c 2005-05-11 11:16:06.367966000 -0700 @@ -57,10 +57,10 @@ * DAT_INSUFFICIENT_RESOURCES * DAT_INVALID_PARAMETER */ -DAT_RETURN dapl_ia_close(DAT_IA_HANDLE ia_handle, DAT_CLOSE_FLAGS ia_flags) +u32 dapl_ia_close(DAT_IA_HANDLE ia_handle, DAT_CLOSE_FLAGS ia_flags) { DAPL_IA *ia_ptr; - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ia_close (%p, %d)\n", ia_handle, ia_flags); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ia_memtype_hint.c linux-kernel2/dat-provider/dapl_ia_memtype_hint.c --- linux-kernel/dat-provider/dapl_ia_memtype_hint.c 2005-04-20 12:42:06.476020000 -0700 +++ linux-kernel2/dat-provider/dapl_ia_memtype_hint.c 2005-05-11 13:26:28.672013000 -0700 @@ -63,22 +63,18 @@ * DAT_INVALID_HANDLE * DAT_MODEL_NOT_SUPPORTED */ -DAT_RETURN -dapl_ia_memtype_hint(DAT_IA_HANDLE ia_handle, - DAT_MEM_TYPE mem_type, - DAT_VLEN length, - DAT_MEM_OPT mem_optimization, - DAT_VLEN * suggested_length, - DAT_VADDR * suggested_alignment) +u32 dapl_ia_memtype_hint(DAT_IA_HANDLE ia_handle, DAT_MEM_TYPE mem_type, + DAT_VLEN length, DAT_MEM_OPT mem_optimization, + DAT_VLEN *suggested_length, + DAT_VADDR *suggested_alignment) { DAPL_IA *ia_ptr; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ia_memtype_hint (%p)\n", ia_handle); ia_ptr = (DAPL_IA *) ia_handle; - dat_status = DAT_SUCCESS; if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { dat_status = diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ia_open.c linux-kernel2/dat-provider/dapl_ia_open.c --- linux-kernel/dat-provider/dapl_ia_open.c 2005-05-11 11:08:04.435965000 -0700 +++ linux-kernel2/dat-provider/dapl_ia_open.c 2005-05-11 13:26:48.054000000 -0700 @@ -51,19 +51,16 @@ * That function maps the DAT_NAME parameter of dat_ia_open to a DAT_PROVIDER, * and calls this function. */ -DAT_RETURN -dapl_ia_open(const DAT_NAME_PTR name, - DAT_COUNT async_evd_qlen, - DAT_EVD_HANDLE * async_evd_handle_ptr, - DAT_IA_HANDLE * ia_handle_ptr) +u32 dapl_ia_open(const DAT_NAME_PTR name, DAT_COUNT async_evd_qlen, + DAT_EVD_HANDLE *async_evd_handle_ptr, + DAT_IA_HANDLE *ia_handle_ptr) { - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; struct dat_provider *provider; DAPL_HCA *hca_ptr; DAPL_IA *ia_ptr; DAPL_EVD *evd_ptr; - dat_status = DAT_SUCCESS; hca_ptr = NULL; ia_ptr = NULL; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ia_query.c linux-kernel2/dat-provider/dapl_ia_query.c --- linux-kernel/dat-provider/dapl_ia_query.c 2005-05-11 08:28:11.913022000 -0700 +++ linux-kernel2/dat-provider/dapl_ia_query.c 2005-05-11 13:27:31.889018000 -0700 @@ -59,14 +59,12 @@ * DAT_SUCCESS * DAT_INVALID_PARAMETER */ -DAT_RETURN -dapl_ia_query(DAT_IA_HANDLE ia_handle, - DAT_EVD_HANDLE * async_evd_handle, - struct dat_ia_attr *ia_attr, - struct dat_provider_attr *provider_attr) +u32 dapl_ia_query(DAT_IA_HANDLE ia_handle, DAT_EVD_HANDLE *async_evd_handle, + struct dat_ia_attr *ia_attr, + struct dat_provider_attr *provider_attr) { DAPL_IA *ia_ptr; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; struct evd_merge_type { boolean_t array[6][6]; } *evd_merge; @@ -82,7 +80,6 @@ dapl_ia_query(DAT_IA_HANDLE ia_handle, provider_attr); ia_ptr = (DAPL_IA *) ia_handle; - dat_status = DAT_SUCCESS; if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { dat_status = diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ia_util.c linux-kernel2/dat-provider/dapl_ia_util.c --- linux-kernel/dat-provider/dapl_ia_util.c 2005-05-11 11:08:04.416964000 -0700 +++ linux-kernel2/dat-provider/dapl_ia_util.c 2005-05-11 13:28:41.996001000 -0700 @@ -107,9 +107,9 @@ DAPL_IA *dapl_ia_alloc(struct dat_provid * Performs an abrupt close of the IA */ -DAT_RETURN dapl_ia_abrupt_close(DAPL_IA * ia_ptr) +u32 dapl_ia_abrupt_close(DAPL_IA *ia_ptr) { - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; DAPL_EP *ep_ptr, *next_ep_ptr; DAPL_LMR *lmr_ptr, *next_lmr_ptr; DAPL_RMR *rmr_ptr, *next_rmr_ptr; @@ -119,8 +119,6 @@ DAT_RETURN dapl_ia_abrupt_close(DAPL_IA DAPL_CR *cr_ptr, *next_cr_ptr; DAPL_HCA *hca_ptr; - dat_status = DAT_SUCCESS; - /* * clear all the data structures associated with the IA. * this must be done in order (rmr,rsp) before (ep lmr psp) before @@ -351,16 +349,14 @@ DAT_RETURN dapl_ia_abrupt_close(DAPL_IA * */ -DAT_RETURN dapl_ia_graceful_close(DAPL_IA * ia_ptr) +u32 dapl_ia_graceful_close(DAPL_IA *ia_ptr) { - DAT_RETURN dat_status; - DAT_RETURN cur_dat_status; + u32 dat_status = DAT_SUCCESS; + u32 cur_dat_status; DAPL_EVD *evd_ptr; DAPL_LLIST_ENTRY *entry; DAPL_HCA *hca_ptr; - dat_status = DAT_SUCCESS; - if (!dapl_llist_is_empty(&ia_ptr->rmr_list_head) || !dapl_llist_is_empty(&ia_ptr->rsp_list_head) || !dapl_llist_is_empty(&ia_ptr->ep_list_head) || @@ -710,9 +706,9 @@ void dapl_ia_link_rsp(DAPL_IA * ia_ptr, spin_unlock(&ia_ptr->header.lock); } -DAT_RETURN dapl_ia_setup_callbacks(DAPL_IA * ia_ptr, DAPL_EVD * async_evd_ptr) +u32 dapl_ia_setup_callbacks(DAPL_IA * ia_ptr, DAPL_EVD * async_evd_ptr) { - DAT_RETURN dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; /* unaffiliated handler */ dat_status = @@ -733,9 +729,9 @@ DAT_RETURN dapl_ia_setup_callbacks(DAPL_ return dat_status; } -DAT_RETURN dapl_ia_teardown_callbacks(DAPL_IA * ia_ptr) +u32 dapl_ia_teardown_callbacks(DAPL_IA * ia_ptr) { - DAT_RETURN dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; /* unaffiliated handler */ dat_status = diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ia_util.h linux-kernel2/dat-provider/dapl_ia_util.h --- linux-kernel/dat-provider/dapl_ia_util.h 2005-05-11 08:28:12.246007000 -0700 +++ linux-kernel2/dat-provider/dapl_ia_util.h 2005-05-11 13:29:00.006004000 -0700 @@ -41,9 +41,9 @@ DAPL_IA *dapl_ia_alloc(struct dat_provider *provider, DAPL_HCA * hca_ptr); -DAT_RETURN dapl_ia_abrupt_close(DAPL_IA * ia_ptr); +u32 dapl_ia_abrupt_close(DAPL_IA *ia_ptr); -DAT_RETURN dapl_ia_graceful_close(DAPL_IA * ia_ptr); +u32 dapl_ia_graceful_close(DAPL_IA *ia_ptr); void dapl_ia_free(DAPL_IA * ia_ptr); @@ -80,8 +80,8 @@ void dapl_ia_link_rsp(DAPL_IA * ia_ptr, DAPL_SP *dapl_ia_sp_search(DAPL_IA * ia_ptr, DAT_CONN_QUAL conn_qual, boolean_t is_psp); -DAT_RETURN dapl_ia_setup_callbacks(DAPL_IA * ia_ptr, DAPL_EVD * async_evd_ptr); +u32 dapl_ia_setup_callbacks(DAPL_IA *ia_ptr, DAPL_EVD *async_evd_ptr); -DAT_RETURN dapl_ia_teardown_callbacks(DAPL_IA * ia_ptr); +u32 dapl_ia_teardown_callbacks(DAPL_IA *ia_ptr); #endif diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_lmr_free.c linux-kernel2/dat-provider/dapl_lmr_free.c --- linux-kernel/dat-provider/dapl_lmr_free.c 2005-04-27 11:38:35.485003000 -0700 +++ linux-kernel2/dat-provider/dapl_lmr_free.c 2005-05-11 11:16:06.394963000 -0700 @@ -57,10 +57,10 @@ * DAT_INVALID_STATE */ -DAT_RETURN dapl_lmr_free(DAT_LMR_HANDLE lmr_handle) +u32 dapl_lmr_free(DAT_LMR_HANDLE lmr_handle) { DAPL_LMR *lmr; - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_free (%p)\n", lmr_handle); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_lmr_kcreate.c linux-kernel2/dat-provider/dapl_lmr_kcreate.c --- linux-kernel/dat-provider/dapl_lmr_kcreate.c 2005-05-06 16:16:19.180012000 -0700 +++ linux-kernel2/dat-provider/dapl_lmr_kcreate.c 2005-05-11 13:37:00.876008000 -0700 @@ -37,71 +37,20 @@ #include "dapl_lmr_util.h" #include "dapl_adapter_util.h" -/********************************************************************* - * * - * Function Prototypes * - * * - *********************************************************************/ - -static __inline__ DAT_RETURN -dapl_lmr_create_virtual(DAPL_IA * ia, - DAT_PVOID virt_addr, - DAT_VLEN length, - DAPL_PZ * pz, - DAT_MEM_PRIV_FLAGS privileges, - DAT_LMR_HANDLE * lmr_handle, - DAT_LMR_CONTEXT * lmr_context, - DAT_RMR_CONTEXT * rmr_context, - DAT_VLEN * registered_length, - DAT_VADDR * registered_address); - -static __inline__ DAT_RETURN -dapl_lmr_create_physical(DAPL_IA * ia, - DAT_REGION_DESCRIPTION phys_addr, - DAT_VLEN length, - DAPL_PZ * pz, - DAT_MEM_PRIV_FLAGS privileges, - DAT_LMR_HANDLE * lmr_handle, - DAT_LMR_CONTEXT * lmr_context, - DAT_RMR_CONTEXT * rmr_context, - DAT_VLEN * registered_length, - DAT_VADDR * registered_address); - -static __inline__ DAT_RETURN -dapl_lmr_create_lmr(DAPL_IA * ia, - DAPL_LMR * original_lmr, - DAPL_PZ * pz, - DAT_MEM_PRIV_FLAGS privileges, - DAT_LMR_HANDLE * lmr_handle, - DAT_LMR_CONTEXT * lmr_context, - DAT_RMR_CONTEXT * rmr_context, - DAT_VLEN * registered_length, - DAT_VADDR * registered_address); - -/********************************************************************* - * * - * Function Definitions * - * * - *********************************************************************/ - -DAT_RETURN -dapl_lmr_create_virtual(DAPL_IA * ia, - DAT_PVOID virt_addr, - DAT_VLEN length, - DAPL_PZ * pz, - DAT_MEM_PRIV_FLAGS privileges, - DAT_LMR_HANDLE * lmr_handle, - DAT_LMR_CONTEXT * lmr_context, - DAT_RMR_CONTEXT * rmr_context, - DAT_VLEN * registered_length, - DAT_VADDR * registered_address) +static inline u32 dapl_lmr_create_virtual(DAPL_IA *ia, DAT_PVOID virt_addr, + DAT_VLEN length, DAPL_PZ *pz, + DAT_MEM_PRIV_FLAGS privileges, + DAT_LMR_HANDLE *lmr_handle, + DAT_LMR_CONTEXT *lmr_context, + DAT_RMR_CONTEXT *rmr_context, + DAT_VLEN *registered_length, + DAT_VADDR *registered_address) { DAPL_LMR *lmr; DAT_REGION_DESCRIPTION reg_desc; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; reg_desc.for_va = virt_addr; - dat_status = DAT_SUCCESS; lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_VIRTUAL, @@ -170,25 +119,21 @@ dapl_lmr_create_virtual(DAPL_IA * ia, } -DAT_RETURN -dapl_lmr_create_physical(DAPL_IA * ia, - DAT_REGION_DESCRIPTION phys_addr, - DAT_VLEN page_count, - DAPL_PZ * pz, - DAT_MEM_PRIV_FLAGS privileges, - DAT_LMR_HANDLE * lmr_handle, - DAT_LMR_CONTEXT * lmr_context, - DAT_RMR_CONTEXT * rmr_context, - DAT_VLEN * registered_length, - DAT_VADDR * registered_address) +static inline u32 dapl_lmr_create_physical(DAPL_IA *ia, + DAT_REGION_DESCRIPTION phys_addr, + DAT_VLEN page_count, DAPL_PZ *pz, + DAT_MEM_PRIV_FLAGS privileges, + DAT_LMR_HANDLE *lmr_handle, + DAT_LMR_CONTEXT *lmr_context, + DAT_RMR_CONTEXT *rmr_context, + DAT_VLEN *registered_length, + DAT_VADDR *registered_address) { DAPL_LMR *lmr; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; u64 *array; array = (u64 *) phys_addr.for_array; - dat_status = DAT_SUCCESS; - lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_PHYSICAL, phys_addr, @@ -258,20 +203,18 @@ dapl_lmr_create_physical(DAPL_IA * ia, return dat_status; } -DAT_RETURN -dapl_lmr_create_lmr(DAPL_IA * ia, - DAPL_LMR * original_lmr, - DAPL_PZ * pz, - DAT_MEM_PRIV_FLAGS privileges, - DAT_LMR_HANDLE * lmr_handle, - DAT_LMR_CONTEXT * lmr_context, - DAT_RMR_CONTEXT * rmr_context, - DAT_VLEN * registered_length, - DAT_VADDR * registered_address) +static inline u32 dapl_lmr_create_lmr(DAPL_IA *ia, DAPL_LMR *original_lmr, + DAPL_PZ *pz, + DAT_MEM_PRIV_FLAGS privileges, + DAT_LMR_HANDLE *lmr_handle, + DAT_LMR_CONTEXT *lmr_context, + DAT_RMR_CONTEXT *rmr_context, + DAT_VLEN *registered_length, + DAT_VADDR *registered_address) { DAPL_LMR *lmr; DAT_REGION_DESCRIPTION reg_desc; - DAT_RETURN dat_status; + u32 dat_status; dat_status = dapl_hash_search(ia->hca_ptr->lmr_hash_table, original_lmr->param.lmr_context, @@ -378,22 +321,15 @@ dapl_lmr_create_lmr(DAPL_IA * ia, * DAT_MODEL_NOT_SUPPORTED * */ -DAT_RETURN -dapl_lmr_kcreate(DAT_IA_HANDLE ia_handle, - DAT_MEM_TYPE mem_type, - DAT_REGION_DESCRIPTION region_description, - DAT_VLEN length, - DAT_PZ_HANDLE pz_handle, - DAT_MEM_PRIV_FLAGS privileges, - DAT_MEM_OPT optimization, - DAT_LMR_HANDLE * lmr_handle, - DAT_LMR_CONTEXT * lmr_context, - DAT_RMR_CONTEXT * rmr_context, - DAT_VLEN * registered_length, DAT_VADDR * registered_address) +u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_handle, DAT_MEM_TYPE mem_type, + DAT_REGION_DESCRIPTION region_description, DAT_VLEN length, + DAT_PZ_HANDLE pz_handle, DAT_MEM_PRIV_FLAGS privileges, + DAT_MEM_OPT optimization, DAT_LMR_HANDLE *lmr_handle, + DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, DAT_VLEN *registered_length, DAT_VADDR *registered_address) { DAPL_IA *ia; DAPL_PZ *pz; - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_kcreate(ia:%p, mem_type:%x, ...)\n", diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_lmr_query.c linux-kernel2/dat-provider/dapl_lmr_query.c --- linux-kernel/dat-provider/dapl_lmr_query.c 2005-05-10 08:13:28.665004000 -0700 +++ linux-kernel2/dat-provider/dapl_lmr_query.c 2005-05-11 13:37:47.609016000 -0700 @@ -36,11 +36,10 @@ #include "dapl.h" -DAT_RETURN dapl_lmr_query(DAT_LMR_HANDLE lmr_handle, - struct dat_lmr_param *lmr_param) +u32 dapl_lmr_query(DAT_LMR_HANDLE lmr_handle, struct dat_lmr_param *lmr_param) { DAPL_LMR *lmr; - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_query (%p, %p)\n", diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_lmr_sync_rdma_read.c linux-kernel2/dat-provider/dapl_lmr_sync_rdma_read.c --- linux-kernel/dat-provider/dapl_lmr_sync_rdma_read.c 2005-05-10 08:13:28.304000000 -0700 +++ linux-kernel2/dat-provider/dapl_lmr_sync_rdma_read.c 2005-05-11 13:38:06.653039000 -0700 @@ -56,15 +56,12 @@ * DAT_INVALID_HANDLE * DAT_INVALID_PARAMETER */ -DAT_RETURN -dapl_lmr_sync_rdma_read(DAT_IA_HANDLE ia_handle, - const struct dat_lmr_triplet *local_segments, - DAT_VLEN num_segments) +u32 dapl_lmr_sync_rdma_read(DAT_IA_HANDLE ia_handle, + const struct dat_lmr_triplet *local_segments, + DAT_VLEN num_segments) { DAPL_IA *ia_ptr; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dat_lmr_sync_rdma_read (%p, %p, %ld)\n", diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_lmr_sync_rdma_write.c linux-kernel2/dat-provider/dapl_lmr_sync_rdma_write.c --- linux-kernel/dat-provider/dapl_lmr_sync_rdma_write.c 2005-05-10 08:13:29.023025000 -0700 +++ linux-kernel2/dat-provider/dapl_lmr_sync_rdma_write.c 2005-05-11 13:38:29.184000000 -0700 @@ -41,15 +41,12 @@ * Ensure a region of memory is consistent by locally flushing * non-coherent cache */ -DAT_RETURN -dapl_lmr_sync_rdma_write(DAT_IA_HANDLE ia_handle, - const struct dat_lmr_triplet *local_segments, - DAT_VLEN num_segments) +u32 dapl_lmr_sync_rdma_write(DAT_IA_HANDLE ia_handle, + const struct dat_lmr_triplet *local_segments, + DAT_VLEN num_segments) { DAPL_IA *ia_ptr; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dat_lmr_sync_rdma_write (%p, %p, %ld)\n", diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_module.c linux-kernel2/dat-provider/dapl_module.c --- linux-kernel/dat-provider/dapl_module.c 2005-05-11 11:08:04.396968000 -0700 +++ linux-kernel2/dat-provider/dapl_module.c 2005-05-11 11:16:06.423960000 -0700 @@ -58,7 +58,7 @@ static void dapl_unregister_providers(vo */ static int __init dapl_init(void) { - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_UTIL, "DAPL: Started (dapl_init)\n"); @@ -96,7 +96,7 @@ module_init(dapl_init); */ static void __exit dapl_fini(void) { - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_UTIL, "DAPL: Stopped (dapl_fini)\n"); @@ -121,7 +121,7 @@ void DAT_PROVIDER_INIT_FUNC_NAME(const s { struct dat_provider *provider; DAPL_HCA *hca_ptr; - DAT_RETURN dat_status; + u32 dat_status; provider = NULL; hca_ptr = NULL; @@ -194,7 +194,7 @@ void DAT_PROVIDER_INIT_FUNC_NAME(const s void DAT_PROVIDER_FINI_FUNC_NAME(const struct dat_provider_info *provider_info) { struct dat_provider *provider; - DAT_RETURN dat_status; + u32 dat_status; dat_status = dapl_provider_list_search(provider_info->ia_name, &provider); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_openib_cm.c linux-kernel2/dat-provider/dapl_openib_cm.c --- linux-kernel/dat-provider/dapl_openib_cm.c 2005-05-11 10:09:59.074056000 -0700 +++ linux-kernel2/dat-provider/dapl_openib_cm.c 2005-05-11 13:40:22.442023000 -0700 @@ -397,10 +397,10 @@ error: * DAT_INVALID_PARAMETER * */ -DAT_RETURN dapl_ib_connect(DAT_EP_HANDLE ep_handle, - DAT_IA_ADDRESS_PTR remote_ia_address, - DAT_CONN_QUAL remote_conn_qual, - DAT_COUNT private_data_size, DAT_PVOID private_data) +u32 dapl_ib_connect(DAT_EP_HANDLE ep_handle, + DAT_IA_ADDRESS_PTR remote_ia_address, + DAT_CONN_QUAL remote_conn_qual, + DAT_COUNT private_data_size, DAT_PVOID private_data) { DAPL_IA *ia_ptr; DAPL_EP *ep_ptr; @@ -489,7 +489,7 @@ DAT_RETURN dapl_ib_connect(DAT_EP_HANDLE * DAT_INVALID_PARAMETER * */ -DAT_RETURN dapl_ib_disconnect(DAPL_EP * ep_ptr, DAT_CLOSE_FLAGS close_flags) +u32 dapl_ib_disconnect(DAPL_EP *ep_ptr, DAT_CLOSE_FLAGS close_flags) { struct dapl_cm_id *conn = ep_ptr->cm_handle; int status; @@ -622,9 +622,7 @@ void dapl_ib_reinit_ep(DAPL_EP * ep_ptr) * DAT_INVALID_PARAMETER * */ -DAT_RETURN -dapl_ib_setup_conn_listener(DAPL_IA * ia_ptr, - u64 ServiceID, DAPL_SP * sp_ptr) +u32 dapl_ib_setup_conn_listener(DAPL_IA *ia_ptr, u64 ServiceID, DAPL_SP *sp_ptr) { int status; @@ -667,7 +665,7 @@ dapl_ib_setup_conn_listener(DAPL_IA * ia * DAT_INVALID_PARAMETER * */ -DAT_RETURN dapl_ib_remove_conn_listener(DAPL_IA * ia_ptr, DAPL_SP * sp_ptr) +u32 dapl_ib_remove_conn_listener(DAPL_IA *ia_ptr, DAPL_SP *sp_ptr) { //*** This will hang if called from CM thread context... //*** Move back to using WQ... @@ -695,8 +693,7 @@ DAT_RETURN dapl_ib_remove_conn_listener( * DAT_INVALID_PARAMETER * */ -DAT_RETURN -dapl_ib_reject_connection(struct dapl_cm_id *cm_handle, int reject_reason) +u32 dapl_ib_reject_connection(struct dapl_cm_id *cm_handle, int reject_reason) { int status; @@ -738,15 +735,14 @@ dapl_ib_reject_connection(struct dapl_cm * DAT_INVALID_PARAMETER * */ -DAT_RETURN dapl_ib_accept_connection(DAT_CR_HANDLE cr_handle, - DAT_EP_HANDLE ep_handle, - DAT_COUNT private_data_size, - const DAT_PVOID prd_ptr) +u32 dapl_ib_accept_connection(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, + DAT_COUNT private_data_size, + const DAT_PVOID prd_ptr) { DAPL_CR *cr_ptr; DAPL_EP *ep_ptr; DAPL_IA *ia_ptr; - DAT_RETURN dat_status; + u32 dat_status; int status; struct ib_cm_rep_param passive_params; struct dapl_cm_id *conn; @@ -842,8 +838,8 @@ static int ib_cm_get_remote_gid(struct d * DAT_INVALID_PARAMETER * */ -DAT_RETURN dapl_ib_cm_remote_addr(DAT_HANDLE dat_handle, - struct sockaddr_in6 *remote_ia_address) +u32 dapl_ib_cm_remote_addr(DAT_HANDLE dat_handle, + struct sockaddr_in6 *remote_ia_address) { DAPL_IA *ia_ptr; int rc; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_openib_dto.h linux-kernel2/dat-provider/dapl_openib_dto.h --- linux-kernel/dat-provider/dapl_openib_dto.h 2005-05-10 08:13:28.379014000 -0700 +++ linux-kernel2/dat-provider/dapl_openib_dto.h 2005-05-11 13:41:39.620043000 -0700 @@ -50,10 +50,9 @@ * * Provider specific Post RECV function */ -static __inline__ DAT_RETURN -dapl_ib_post_recv(DAPL_EP * ep_ptr, - DAPL_COOKIE * cookie, - DAT_COUNT num_segments, struct dat_lmr_triplet *local_iov) +static __inline__ u32 dapl_ib_post_recv(DAPL_EP *ep_ptr, DAPL_COOKIE *cookie, + DAT_COUNT num_segments, + struct dat_lmr_triplet *local_iov) { struct ib_recv_wr rr_desc = { NULL }; struct ib_recv_wr *rr_desc_fail; @@ -100,14 +99,12 @@ dapl_ib_post_recv(DAPL_EP * ep_ptr, * * Provider specific Post SEND function */ -static __inline__ DAT_RETURN -dapl_ib_post_send(DAPL_EP * ep_ptr, - int op_type, - DAPL_COOKIE * cookie, - DAT_COUNT num_segments, - struct dat_lmr_triplet *local_iov, - const struct dat_rmr_triplet *remote_iov, - DAT_COMPLETION_FLAGS completion_flags) +static __inline__ u32 dapl_ib_post_send(DAPL_EP *ep_ptr, int op_type, + DAPL_COOKIE *cookie, + DAT_COUNT num_segments, + struct dat_lmr_triplet *local_iov, + const struct dat_rmr_triplet *remote_iov, + DAT_COMPLETION_FLAGS completion_flags) { struct ib_send_wr send_desc = { NULL }; struct ib_send_wr *send_desc_fail; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_openib_qp.c linux-kernel2/dat-provider/dapl_openib_qp.c --- linux-kernel/dat-provider/dapl_openib_qp.c 2005-05-10 08:13:28.894000000 -0700 +++ linux-kernel2/dat-provider/dapl_openib_qp.c 2005-05-11 13:42:06.454000000 -0700 @@ -67,8 +67,7 @@ ib_cq_handle_t dapl_get_dto_cq(DAPL_IA * * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN -dapl_ib_qp_alloc(DAPL_IA * ia_ptr, DAPL_EP * ep_ptr, DAPL_EP * ep_ctx_ptr) +u32 dapl_ib_qp_alloc(DAPL_IA *ia_ptr, DAPL_EP *ep_ptr, DAPL_EP *ep_ctx_ptr) { struct dat_ep_attr *attr; int ib_status; @@ -151,7 +150,7 @@ dapl_ib_qp_alloc(DAPL_IA * ia_ptr, DAPL_ * Status of destroy operation. * */ -DAT_RETURN dapl_ib_qp_free(DAPL_IA * ia_ptr, DAPL_EP * ep_ptr) +u32 dapl_ib_qp_free(DAPL_IA *ia_ptr, DAPL_EP *ep_ptr) { int ib_status; @@ -231,9 +230,8 @@ ib_cq_handle_t dapl_get_dto_cq(DAPL_IA * * DAT_INVALID_PARAMETER * */ -DAT_RETURN -dapl_ib_qp_modify(DAPL_IA * ia_ptr, DAPL_EP * ep_ptr, - struct dat_ep_attr *ep_attr) +u32 dapl_ib_qp_modify(DAPL_IA *ia_ptr, DAPL_EP *ep_ptr, + struct dat_ep_attr *ep_attr) { struct ib_qp_attr qp_attr; struct ib_qp *qp_handle; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_openib_util.c linux-kernel2/dat-provider/dapl_openib_util.c --- linux-kernel/dat-provider/dapl_openib_util.c 2005-05-11 10:09:59.144000000 -0700 +++ linux-kernel2/dat-provider/dapl_openib_util.c 2005-05-11 13:48:41.352007000 -0700 @@ -47,9 +47,8 @@ #include "dapl_vendor.h" #include "dapl_openib_cm.h" -DAT_RETURN -dapl_ib_get_hca_ids(ib_hca_handle_t hca, - u8 port, union ib_gid * gid, u16 * lid); +u32 dapl_ib_get_hca_ids(ib_hca_handle_t hca, u8 port, union ib_gid *gid, + u16 *lid); /* * Map all IB DTO completion codes to their DAT equivalent. @@ -162,9 +161,9 @@ dapl_ib_convert_mem_privileges(DAT_MEM_P * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN dapl_ib_open_hca(char *name, DAPL_HCA * hca_ptr) +u32 dapl_ib_open_hca(char *name, DAPL_HCA * hca_ptr) { - DAT_RETURN dat_ret; + u32 dat_ret; memset(&hca_ptr->ib_trans.gid.global, 0, sizeof(union ib_gid)); hca_ptr->ib_trans.lid = 0; @@ -199,7 +198,7 @@ DAT_RETURN dapl_ib_open_hca(char *name, * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN dapl_ib_close_hca(DAPL_HCA * hca_ptr) +u32 dapl_ib_close_hca(DAPL_HCA *hca_ptr) { return DAT_SUCCESS; } @@ -222,10 +221,8 @@ DAT_RETURN dapl_ib_close_hca(DAPL_HCA * * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN -dapl_ib_cq_alloc(DAPL_IA * ia_ptr, - DAPL_EVD * evd_ptr, - DAT_COUNT * cqlen, ib_comp_handle_t callback) +u32 dapl_ib_cq_alloc(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr, DAT_COUNT *cqlen, + ib_comp_handle_t callback) { int ib_status; @@ -258,7 +255,7 @@ dapl_ib_cq_alloc(DAPL_IA * ia_ptr, * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN dapl_ib_cq_free(DAPL_IA * ia_ptr, DAPL_EVD * evd_ptr) +u32 dapl_ib_cq_free(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr) { int ib_status; @@ -266,8 +263,7 @@ DAT_RETURN dapl_ib_cq_free(DAPL_IA * ia_ return dapl_ib_status_convert(ib_status); } -DAT_RETURN -dapl_ib_cq_resize(DAPL_IA * ia_ptr, DAPL_EVD * evd_ptr, DAT_COUNT * cqlen) +u32 dapl_ib_cq_resize(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr, DAT_COUNT *cqlen) { int ib_status; @@ -299,7 +295,7 @@ dapl_ib_cq_resize(DAPL_IA * ia_ptr, DAPL * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN dapl_set_cq_notify(DAPL_IA * ia_ptr, DAPL_EVD * evd_ptr) +u32 dapl_set_cq_notify(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr) { int ib_status; @@ -324,7 +320,7 @@ DAT_RETURN dapl_set_cq_notify(DAPL_IA * * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN dapl_ib_pd_alloc(DAPL_IA * ia, DAPL_PZ * pz) +u32 dapl_ib_pd_alloc(DAPL_IA *ia, DAPL_PZ *pz) { struct ib_pd *pd; int ib_status; @@ -359,7 +355,7 @@ DAT_RETURN dapl_ib_pd_alloc(DAPL_IA * ia * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN dapl_ib_pd_free(DAPL_PZ * pz) +u32 dapl_ib_pd_free(DAPL_PZ *pz) { int ib_status; @@ -394,21 +390,16 @@ DAT_RETURN dapl_ib_pd_free(DAPL_PZ * pz) * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN -dapl_ib_mr_register(DAPL_IA * ia, - DAPL_LMR * lmr, - DAT_PVOID virt_addr, - DAT_VLEN length, DAT_MEM_PRIV_FLAGS privileges) +u32 dapl_ib_mr_register(DAPL_IA *ia, DAPL_LMR *lmr, DAT_PVOID virt_addr, + DAT_VLEN length, DAT_MEM_PRIV_FLAGS privileges) { /* ITBD virtual memory registration ??? */ return (DAT_SUCCESS); } -DAT_RETURN dapl_ib_mr_register_physical(DAPL_IA * ia_ptr, - DAPL_LMR * lmr, - DAT_PVOID phys_addr, - DAT_VLEN length, - DAT_MEM_PRIV_FLAGS privileges) +u32 dapl_ib_mr_register_physical(DAPL_IA *ia_ptr, DAPL_LMR *lmr, + DAT_PVOID phys_addr, DAT_VLEN length, + DAT_MEM_PRIV_FLAGS privileges) { int ib_status; int i; @@ -481,7 +472,7 @@ DAT_RETURN dapl_ib_mr_register_physical( * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN dapl_ib_mr_deregister(DAPL_LMR * lmr) +u32 dapl_ib_mr_deregister(DAPL_LMR *lmr) { int ib_status; @@ -515,7 +506,7 @@ DAT_RETURN dapl_ib_mr_deregister(DAPL_LM * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN dapl_ib_mw_alloc(DAPL_RMR * rmr) +u32 dapl_ib_mw_alloc(DAPL_RMR *rmr) { int ib_status; struct ib_mw *mw; @@ -552,7 +543,7 @@ DAT_RETURN dapl_ib_mw_alloc(DAPL_RMR * r * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN dapl_ib_mw_free(DAPL_RMR * rmr) +u32 dapl_ib_mw_free(DAPL_RMR *rmr) { int ib_status; @@ -585,14 +576,10 @@ DAT_RETURN dapl_ib_mw_free(DAPL_RMR * rm * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN -dapl_ib_mw_bind(DAPL_RMR * rmr, - DAPL_LMR * lmr, - DAPL_EP * ep, - DAPL_COOKIE * cookie, - DAT_VADDR virtual_address, - DAT_VLEN length, - DAT_MEM_PRIV_FLAGS mem_priv, boolean_t is_signaled) +u32 dapl_ib_mw_bind(DAPL_RMR *rmr, DAPL_LMR *lmr, DAPL_EP *ep, + DAPL_COOKIE *cookie, DAT_VADDR virtual_address, + DAT_VLEN length, DAT_MEM_PRIV_FLAGS mem_priv, + boolean_t is_signaled) { int ib_status; struct ib_mw_bind mw_bind_prop; @@ -640,9 +627,8 @@ dapl_ib_mw_bind(DAPL_RMR * rmr, * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN -dapl_ib_mw_unbind(DAPL_RMR * rmr, - DAPL_EP * ep, DAPL_COOKIE * cookie, boolean_t is_signaled) +u32 dapl_ib_mw_unbind(DAPL_RMR *rmr, DAPL_EP *ep, DAPL_COOKIE *cookie, + boolean_t is_signaled) { int ib_status; struct ib_mw_bind mw_bind_prop; @@ -688,10 +674,9 @@ dapl_ib_mw_unbind(DAPL_RMR * rmr, * DAT_INVALID_PARAMETER * */ -DAT_RETURN -dapl_ib_setup_async_callback(DAPL_IA * ia_ptr, - DAPL_ASYNC_HANDLER_TYPE handler_type, - ib_async_handler_t callback, void *context) +u32 dapl_ib_setup_async_callback(DAPL_IA *ia_ptr, + DAPL_ASYNC_HANDLER_TYPE handler_type, + ib_async_handler_t callback, void *context) { int ib_status = 0; @@ -714,10 +699,8 @@ dapl_ib_setup_async_callback(DAPL_IA * i * DAT_SUCCESS * DAT_INVALID_PARAMETER */ -DAT_RETURN dapl_ib_query_hca(DAPL_HCA * hca_ptr, - struct dat_ia_attr *ia_attr, - struct dat_ep_attr *ep_attr, - struct sockaddr_in6 *ip_addr) +u32 dapl_ib_query_hca(DAPL_HCA *hca_ptr, struct dat_ia_attr *ia_attr, + struct dat_ep_attr *ep_attr, struct sockaddr_in6 *ip_addr) { int ib_status; struct ib_device_attr device_attr; @@ -795,9 +778,8 @@ DAT_RETURN dapl_ib_query_hca(DAPL_HCA * return DAT_SUCCESS; } -DAT_RETURN -dapl_ib_completion_poll(DAPL_HCA * hca_ptr, - DAPL_EVD * evd_ptr, ib_work_completion_t * cqe_ptr) +u32 dapl_ib_completion_poll(DAPL_HCA *hca_ptr, DAPL_EVD *evd_ptr, + ib_work_completion_t *cqe_ptr) { int ib_status; @@ -805,12 +787,11 @@ dapl_ib_completion_poll(DAPL_HCA * hca_p if (ib_status == 1) return DAT_SUCCESS; if (ib_status == 0) - return DAT_ERRORP(DAT_QUEUE_EMPTY, DAT_NO_SUBTYPE); + return DAT_ERROR(DAT_QUEUE_EMPTY, DAT_NO_SUBTYPE); return dapl_ib_status_convert(ib_status); } -DAT_RETURN -dapl_ib_completion_notify(DAPL_EVD * evd_ptr, ib_notification_type_t type) +u32 dapl_ib_completion_notify(DAPL_EVD *evd_ptr, ib_notification_type_t type) { int ib_status; @@ -877,14 +858,12 @@ DAT_DTO_COMPLETION_STATUS dapl_ib_get_dt * DAT_NOT_IMPLEMENTED Caller is not interested this event */ -DAT_RETURN -dapl_ib_get_async_event(ib_error_record_t * cause_ptr, - DAT_EVENT_NUMBER * async_event) +u32 dapl_ib_get_async_event(ib_error_record_t *cause_ptr, + DAT_EVENT_NUMBER *async_event) { ib_async_event_type event_id; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; - dat_status = DAT_SUCCESS; event_id = cause_ptr->event; switch (event_id) { @@ -921,8 +900,8 @@ dapl_ib_get_async_event(ib_error_record_ return dat_status; } -DAT_RETURN -dapl_ib_get_hca_ids(ib_hca_handle_t hca, u8 port, union ib_gid * gid, u16 * lid) +u32 dapl_ib_get_hca_ids(ib_hca_handle_t hca, u8 port, union ib_gid *gid, + u16 *lid) { int status; struct ib_port_attr port_attr; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_openib_util.h linux-kernel2/dat-provider/dapl_openib_util.h --- linux-kernel/dat-provider/dapl_openib_util.h 2005-05-11 10:09:59.163023000 -0700 +++ linux-kernel2/dat-provider/dapl_openib_util.h 2005-05-11 13:48:41.355003000 -0700 @@ -47,12 +47,7 @@ void dapl_add_one(struct ib_device *device); void dapl_remove_one(struct ib_device *device); -static __inline__ DAT_RETURN dapl_ib_status_convert(int32_t ib_status); - -/* A version of DAT_ERROR is redefined here as the order of include - * files makes it unavailable at this point - */ -#define DAT_ERRORP(Type, SubType) ((DAT_RETURN)(DAT_CLASS_ERROR | Type | SubType)) +static __inline__ u32 dapl_ib_status_convert(int32_t ib_status); /* * Typedefs to map OpenIB types to more generic 'ib' types @@ -126,28 +121,28 @@ typedef struct ib_hca_transport { * Do not return invalid Handles, the user is not able * to deal with them. */ -static __inline__ DAT_RETURN dapl_ib_status_convert(int32_t ib_status) +static __inline__ u32 dapl_ib_status_convert(int32_t ib_status) { switch (ib_status) { case 0: return DAT_SUCCESS; case -EAGAIN: - return DAT_ERRORP(DAT_QUEUE_EMPTY, DAT_NO_SUBTYPE); + return DAT_ERROR(DAT_QUEUE_EMPTY, DAT_NO_SUBTYPE); case -EBUSY: - return DAT_ERRORP(DAT_INTERNAL_ERROR, DAT_NO_SUBTYPE); + return DAT_ERROR(DAT_INTERNAL_ERROR, DAT_NO_SUBTYPE); case -ENOMEM: - return DAT_ERRORP(DAT_INSUFFICIENT_RESOURCES, DAT_NO_SUBTYPE); + return DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_NO_SUBTYPE); case -EINVAL: - return DAT_ERRORP(DAT_INVALID_HANDLE, DAT_NO_SUBTYPE); + return DAT_ERROR(DAT_INVALID_HANDLE, DAT_NO_SUBTYPE); /* * Udapl only, not sure what will be the return value from the IB * case ??????: - * return DAT_ERRORP(DAT_TIMEOUT_EXPIRED, DAT_NO_SUBTYPE); + * return DAT_ERROR(DAT_TIMEOUT_EXPIRED, DAT_NO_SUBTYPE); */ case -EINTR: - return DAT_ERRORP(DAT_INTERRUPTED_CALL, DAT_NO_SUBTYPE); + return DAT_ERROR(DAT_INTERRUPTED_CALL, DAT_NO_SUBTYPE); default: - return DAT_ERRORP(DAT_INTERNAL_ERROR, DAT_NO_SUBTYPE); + return DAT_ERROR(DAT_INTERNAL_ERROR, DAT_NO_SUBTYPE); } } diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_provider.c linux-kernel2/dat-provider/dapl_provider.c --- linux-kernel/dat-provider/dapl_provider.c 2005-05-11 08:28:12.130010000 -0700 +++ linux-kernel2/dat-provider/dapl_provider.c 2005-05-11 13:50:39.276005000 -0700 @@ -37,7 +37,7 @@ #include "dapl_provider.h" -extern DAT_RETURN dapl_not_implemented(void); +extern u32 dapl_not_implemented(void); /********************************************************************* * * @@ -144,11 +144,9 @@ dapl_provider_list_key_cmp(const char *n * * *********************************************************************/ -DAT_RETURN dapl_provider_list_create(void) +u32 dapl_provider_list_create(void) { - DAT_RETURN status; - - status = DAT_SUCCESS; + u32 status = DAT_SUCCESS; /* create the head node */ g_dapl_provider_list.head = @@ -192,7 +190,7 @@ DAT_RETURN dapl_provider_list_create(voi return status; } -DAT_RETURN dapl_provider_list_destroy(void) +u32 dapl_provider_list_destroy(void) { DAPL_PROVIDER_LIST_NODE *cur_node; @@ -211,15 +209,12 @@ DAT_COUNT dapl_provider_list_size(void) return g_dapl_provider_list.size; } -DAT_RETURN dapl_provider_list_insert(const char *name, - struct dat_provider **p_data) +u32 dapl_provider_list_insert(const char *name, struct dat_provider **p_data) { DAPL_PROVIDER_LIST_NODE *cur_node, *prev_node, *next_node; - DAT_RETURN status; + u32 status = DAT_SUCCESS; unsigned int len; - status = DAT_SUCCESS; - cur_node = kmalloc(sizeof(DAPL_PROVIDER_LIST_NODE), GFP_ATOMIC); if (NULL == cur_node) { @@ -266,13 +261,10 @@ DAT_RETURN dapl_provider_list_insert(con return status; } -DAT_RETURN dapl_provider_list_search(const char *name, - struct dat_provider **p_data) +u32 dapl_provider_list_search(const char *name, struct dat_provider **p_data) { DAPL_PROVIDER_LIST_NODE *cur_node; - DAT_RETURN status; - - status = DAT_ERROR(DAT_PROVIDER_NOT_FOUND, DAT_NAME_NOT_REGISTERED); + u32 status = DAT_ERROR(DAT_PROVIDER_NOT_FOUND, DAT_NAME_NOT_REGISTERED); for (cur_node = g_dapl_provider_list.head->next; g_dapl_provider_list.tail != cur_node; cur_node = cur_node->next) { @@ -290,12 +282,10 @@ DAT_RETURN dapl_provider_list_search(con return status; } -DAT_RETURN dapl_provider_list_remove(const char *name) +u32 dapl_provider_list_remove(const char *name) { DAPL_PROVIDER_LIST_NODE *cur_node, *prev_node, *next_node; - DAT_RETURN status; - - status = DAT_ERROR(DAT_PROVIDER_NOT_FOUND, DAT_NAME_NOT_REGISTERED); + u32 status = DAT_ERROR(DAT_PROVIDER_NOT_FOUND, DAT_NAME_NOT_REGISTERED); for (cur_node = g_dapl_provider_list.head->next; g_dapl_provider_list.tail != cur_node; cur_node = cur_node->next) { diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_provider.h linux-kernel2/dat-provider/dapl_provider.h --- linux-kernel/dat-provider/dapl_provider.h 2005-05-09 11:07:01.934000000 -0700 +++ linux-kernel2/dat-provider/dapl_provider.h 2005-05-11 13:51:03.534001000 -0700 @@ -74,18 +74,18 @@ extern struct dat_provider g_dapl_provid * * *********************************************************************/ -extern DAT_RETURN dapl_provider_list_create(void); +extern u32 dapl_provider_list_create(void); -extern DAT_RETURN dapl_provider_list_destroy(void); +extern u32 dapl_provider_list_destroy(void); extern DAT_COUNT dapl_provider_list_size(void); -extern DAT_RETURN -dapl_provider_list_insert(const char *name, struct dat_provider **p_data); +extern u32 dapl_provider_list_insert(const char *name, + struct dat_provider **p_data); -extern DAT_RETURN -dapl_provider_list_search(const char *name, struct dat_provider **p_data); +extern u32 dapl_provider_list_search(const char *name, + struct dat_provider **p_data); -extern DAT_RETURN dapl_provider_list_remove(const char *name); +extern u32 dapl_provider_list_remove(const char *name); #endif /* DAPL_PROVIDER_H */ diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_psp_create_any.c linux-kernel2/dat-provider/dapl_psp_create_any.c --- linux-kernel/dat-provider/dapl_psp_create_any.c 2005-05-11 08:28:11.754000000 -0700 +++ linux-kernel2/dat-provider/dapl_psp_create_any.c 2005-05-11 13:51:42.543040000 -0700 @@ -70,22 +70,19 @@ * DAT_CONN_QUAL_IN_USE * DAT_MODEL_NOT_SUPPORTED */ -DAT_RETURN -dapl_psp_create_any(DAT_IA_HANDLE ia_handle, - DAT_CONN_QUAL * conn_qual, - DAT_EVD_HANDLE evd_handle, - DAT_PSP_FLAGS psp_flags, DAT_PSP_HANDLE * psp_handle) +u32 dapl_psp_create_any(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL *conn_qual, + DAT_EVD_HANDLE evd_handle, DAT_PSP_FLAGS psp_flags, + DAT_PSP_HANDLE *psp_handle) { DAPL_IA *ia_ptr; DAPL_SP *sp_ptr; DAPL_EVD *evd_ptr; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; static DAT_CONN_QUAL hint_conn_qual = 1000; /* seed value */ DAT_CONN_QUAL lcl_conn_qual; DAT_CONN_QUAL limit_conn_qual; ia_ptr = (DAPL_IA *) ia_handle; - dat_status = DAT_SUCCESS; if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { dat_status = diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_psp_create.c linux-kernel2/dat-provider/dapl_psp_create.c --- linux-kernel/dat-provider/dapl_psp_create.c 2005-05-11 08:28:11.632017000 -0700 +++ linux-kernel2/dat-provider/dapl_psp_create.c 2005-05-11 13:52:10.176006000 -0700 @@ -67,20 +67,17 @@ * DAT_CONN_QUAL_IN_USE * DAT_MODEL_NOT_SUPPORTED */ -DAT_RETURN -dapl_psp_create(DAT_IA_HANDLE ia_handle, - DAT_CONN_QUAL conn_qual, - DAT_EVD_HANDLE evd_handle, - DAT_PSP_FLAGS psp_flags, DAT_PSP_HANDLE * psp_handle) +u32 dapl_psp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, + DAT_EVD_HANDLE evd_handle, DAT_PSP_FLAGS psp_flags, + DAT_PSP_HANDLE *psp_handle) { DAPL_IA *ia_ptr; DAPL_SP *sp_ptr; DAPL_EVD *evd_ptr; boolean_t sp_found; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; ia_ptr = (DAPL_IA *) ia_handle; - dat_status = DAT_SUCCESS; if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { dat_status = diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_psp_free.c linux-kernel2/dat-provider/dapl_psp_free.c --- linux-kernel/dat-provider/dapl_psp_free.c 2005-05-11 08:28:11.814001000 -0700 +++ linux-kernel2/dat-provider/dapl_psp_free.c 2005-05-11 13:52:46.992015000 -0700 @@ -58,15 +58,14 @@ * DAT_SUCCESS * DAT_INVALID_PARAMETER */ -DAT_RETURN dapl_psp_free(DAT_PSP_HANDLE psp_handle) +u32 dapl_psp_free(DAT_PSP_HANDLE psp_handle) { DAPL_IA *ia_ptr; DAPL_SP *sp_ptr; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; DAPL_SP_STATE save_state; sp_ptr = (DAPL_SP *) psp_handle; - dat_status = DAT_SUCCESS; /* * Verify handle */ diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_psp_query.c linux-kernel2/dat-provider/dapl_psp_query.c --- linux-kernel/dat-provider/dapl_psp_query.c 2005-05-11 08:28:13.131019000 -0700 +++ linux-kernel2/dat-provider/dapl_psp_query.c 2005-05-11 13:53:16.824005000 -0700 @@ -38,11 +38,10 @@ #include "dapl.h" -DAT_RETURN dapl_psp_query(DAT_PSP_HANDLE psp_handle, - struct dat_psp_param *psp_param) +u32 dapl_psp_query(DAT_PSP_HANDLE psp_handle, struct dat_psp_param *psp_param) { DAPL_SP *sp_ptr; - DAT_RETURN status; + u32 status; if (DAPL_BAD_HANDLE(psp_handle, DAPL_MAGIC_PSP) || ((DAPL_SP *) psp_handle)->listening != TRUE) { diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_pz_create.c linux-kernel2/dat-provider/dapl_pz_create.c --- linux-kernel/dat-provider/dapl_pz_create.c 2005-04-20 12:42:07.269030000 -0700 +++ linux-kernel2/dat-provider/dapl_pz_create.c 2005-05-11 13:53:40.203026000 -0700 @@ -55,16 +55,15 @@ * DAT_INVALID_PARAMETER * DAT_INVLAID_HANDLE */ -DAT_RETURN dapl_pz_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE * pz_handle) +u32 dapl_pz_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE *pz_handle) { DAPL_IA *ia; DAPL_PZ *pz; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_create (%p, %p)\n", ia_handle, pz_handle); - dat_status = DAT_SUCCESS; if (DAPL_BAD_HANDLE(ia_handle, DAPL_MAGIC_IA)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_pz_free.c linux-kernel2/dat-provider/dapl_pz_free.c --- linux-kernel/dat-provider/dapl_pz_free.c 2005-04-27 11:38:35.840017000 -0700 +++ linux-kernel2/dat-provider/dapl_pz_free.c 2005-05-11 13:53:55.799011000 -0700 @@ -54,14 +54,13 @@ * DAT_INVALID_STATE * DAT_INVALID_HANDLE */ -DAT_RETURN dapl_pz_free(DAT_PZ_HANDLE pz_handle) +u32 dapl_pz_free(DAT_PZ_HANDLE pz_handle) { DAPL_PZ *pz; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_free (%p)\n", pz_handle); - dat_status = DAT_SUCCESS; if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_pz_query.c linux-kernel2/dat-provider/dapl_pz_query.c --- linux-kernel/dat-provider/dapl_pz_query.c 2005-05-10 08:13:28.647010000 -0700 +++ linux-kernel2/dat-provider/dapl_pz_query.c 2005-05-11 11:16:06.498965000 -0700 @@ -36,10 +36,10 @@ #include "dapl.h" -DAT_RETURN dapl_pz_query(DAT_PZ_HANDLE pz_handle, struct dat_pz_param *pz_param) +u32 dapl_pz_query(DAT_PZ_HANDLE pz_handle, struct dat_pz_param *pz_param) { DAPL_PZ *pz; - DAT_RETURN status; + u32 status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_query (%p, %x, %p)\n", pz_handle, pz_param); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ring_buffer_util.c linux-kernel2/dat-provider/dapl_ring_buffer_util.c --- linux-kernel/dat-provider/dapl_ring_buffer_util.c 2005-04-27 11:38:34.816004000 -0700 +++ linux-kernel2/dat-provider/dapl_ring_buffer_util.c 2005-05-11 13:54:51.243023000 -0700 @@ -56,7 +56,7 @@ * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN dapl_rbuf_alloc(DAPL_RING_BUFFER * rbuf, DAT_COUNT size) +u32 dapl_rbuf_alloc(DAPL_RING_BUFFER *rbuf, DAT_COUNT size) { unsigned int rsize; /* real size */ @@ -104,13 +104,11 @@ DAT_RETURN dapl_rbuf_alloc(DAPL_RING_BUF * DAT_INSUFFICIENT_RESOURCES * */ -DAT_RETURN dapl_rbuf_realloc(DAPL_RING_BUFFER * rbuf, DAT_COUNT size) +u32 dapl_rbuf_realloc(DAPL_RING_BUFFER *rbuf, DAT_COUNT size) { DAPL_RING_BUFFER new_rbuf; void *entry; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; /* decreasing the size or retaining the old size is not allowed */ if (size <= rbuf->lim + 1) @@ -191,7 +189,7 @@ void dapl_rbuf_destroy(DAPL_RING_BUFFER * DAT_INSUFFICIENT_RESOURCES (queue full) * */ -DAT_RETURN dapl_rbuf_add(DAPL_RING_BUFFER * rbuf, void *entry) +u32 dapl_rbuf_add(DAPL_RING_BUFFER *rbuf, void *entry) { int pos; int val; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ring_buffer_util.h linux-kernel2/dat-provider/dapl_ring_buffer_util.h --- linux-kernel/dat-provider/dapl_ring_buffer_util.h 2005-04-27 11:38:34.911019000 -0700 +++ linux-kernel2/dat-provider/dapl_ring_buffer_util.h 2005-05-11 13:55:12.302088000 -0700 @@ -42,13 +42,13 @@ /* * Prototypes */ -DAT_RETURN dapl_rbuf_alloc(DAPL_RING_BUFFER * rbuf, DAT_COUNT size); +u32 dapl_rbuf_alloc(DAPL_RING_BUFFER *rbuf, DAT_COUNT size); -DAT_RETURN dapl_rbuf_realloc(DAPL_RING_BUFFER * rbuf, DAT_COUNT size); +u32 dapl_rbuf_realloc(DAPL_RING_BUFFER *rbuf, DAT_COUNT size); void dapl_rbuf_destroy(DAPL_RING_BUFFER * rbuf); -DAT_RETURN dapl_rbuf_add(DAPL_RING_BUFFER * rbuf, void *entry); +u32 dapl_rbuf_add(DAPL_RING_BUFFER *rbuf, void *entry); void *dapl_rbuf_remove(DAPL_RING_BUFFER * rbuf); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_rmr_bind.c linux-kernel2/dat-provider/dapl_rmr_bind.c --- linux-kernel/dat-provider/dapl_rmr_bind.c 2005-05-11 10:09:59.093023000 -0700 +++ linux-kernel2/dat-provider/dapl_rmr_bind.c 2005-05-11 13:56:47.864000000 -0700 @@ -46,18 +46,17 @@ * * *********************************************************************/ -static inline DAT_RETURN -dapl_rmr_bind_fuse(DAPL_RMR * rmr, - const struct dat_lmr_triplet *lmr_triplet, - DAT_MEM_PRIV_FLAGS mem_priv, - DAPL_EP * ep_ptr, - DAT_RMR_COOKIE user_cookie, - DAT_COMPLETION_FLAGS completion_flags, - DAT_RMR_CONTEXT * rmr_context) +static inline u32 dapl_rmr_bind_fuse(DAPL_RMR *rmr, + const struct dat_lmr_triplet *lmr_triplet, + DAT_MEM_PRIV_FLAGS mem_priv, + DAPL_EP *ep_ptr, + DAT_RMR_COOKIE user_cookie, + DAT_COMPLETION_FLAGS completion_flags, + DAT_RMR_CONTEXT *rmr_context) { DAPL_LMR *lmr; DAPL_COOKIE *cookie; - DAT_RETURN status; + u32 status; boolean_t is_signaled; status = dapl_hash_search(rmr->header.owner_ia->hca_ptr->lmr_hash_table, @@ -157,17 +156,14 @@ dapl_rmr_bind_fuse(DAPL_RMR * rmr, return status; } -static inline DAT_RETURN -dapl_rmr_bind_unfuse(DAPL_RMR * rmr, - DAPL_EP * ep_ptr, - DAT_RMR_COOKIE user_cookie, - DAT_COMPLETION_FLAGS completion_flags) +static inline u32 dapl_rmr_bind_unfuse(DAPL_RMR *rmr, DAPL_EP *ep_ptr, + DAT_RMR_COOKIE user_cookie, + DAT_COMPLETION_FLAGS completion_flags) { DAPL_COOKIE *cookie; - DAT_RETURN status; + u32 status = DAT_SUCCESS; boolean_t is_signaled; - status = DAT_SUCCESS; /* * if the ep in unconnected return an error. IB requires that the * QP be connected to change a memory window binding since: @@ -253,13 +249,12 @@ dapl_rmr_bind_unfuse(DAPL_RMR * rmr, * Input: * Output: */ -DAT_RETURN dapl_rmr_bind(DAT_RMR_HANDLE rmr_handle, - const struct dat_lmr_triplet *lmr_triplet, - DAT_MEM_PRIV_FLAGS mem_priv, - DAT_EP_HANDLE ep_handle, - DAT_RMR_COOKIE user_cookie, - DAT_COMPLETION_FLAGS completion_flags, - DAT_RMR_CONTEXT * rmr_context) +u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_handle, + const struct dat_lmr_triplet *lmr_triplet, + DAT_MEM_PRIV_FLAGS mem_priv, DAT_EP_HANDLE ep_handle, + DAT_RMR_COOKIE user_cookie, + DAT_COMPLETION_FLAGS completion_flags, + DAT_RMR_CONTEXT *rmr_context) { DAPL_RMR *rmr; DAPL_EP *ep_ptr; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_rmr_create.c linux-kernel2/dat-provider/dapl_rmr_create.c --- linux-kernel/dat-provider/dapl_rmr_create.c 2005-04-27 11:38:34.778010000 -0700 +++ linux-kernel2/dat-provider/dapl_rmr_create.c 2005-05-11 13:57:10.428101000 -0700 @@ -53,13 +53,11 @@ * DAT_INSUFFICIENT_RESOURCES * DAT_INVALID_PARAMETER */ -DAT_RETURN dapl_rmr_create(DAT_PZ_HANDLE pz_handle, DAT_RMR_HANDLE * rmr_handle) +u32 dapl_rmr_create(DAT_PZ_HANDLE pz_handle, DAT_RMR_HANDLE *rmr_handle) { DAPL_PZ *pz; DAPL_RMR *rmr; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { dat_status = diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_rmr_free.c linux-kernel2/dat-provider/dapl_rmr_free.c --- linux-kernel/dat-provider/dapl_rmr_free.c 2005-04-27 11:38:35.027007000 -0700 +++ linux-kernel2/dat-provider/dapl_rmr_free.c 2005-05-11 13:57:34.719027000 -0700 @@ -53,12 +53,10 @@ * DAT_SUCCESS * DAT_INVALID_PARAMETER */ -DAT_RETURN dapl_rmr_free(DAT_RMR_HANDLE rmr_handle) +u32 dapl_rmr_free(DAT_RMR_HANDLE rmr_handle) { DAPL_RMR *rmr; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) { dat_status = diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_rmr_query.c linux-kernel2/dat-provider/dapl_rmr_query.c --- linux-kernel/dat-provider/dapl_rmr_query.c 2005-05-10 08:13:28.721018000 -0700 +++ linux-kernel2/dat-provider/dapl_rmr_query.c 2005-05-11 13:57:51.347004000 -0700 @@ -38,13 +38,10 @@ #include "dapl.h" -DAT_RETURN dapl_rmr_query(DAT_RMR_HANDLE rmr_handle, - struct dat_rmr_param *rmr_param) +u32 dapl_rmr_query(DAT_RMR_HANDLE rmr_handle, struct dat_rmr_param *rmr_param) { DAPL_RMR *rmr; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) { dat_status = diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_rsp_create.c linux-kernel2/dat-provider/dapl_rsp_create.c --- linux-kernel/dat-provider/dapl_rsp_create.c 2005-05-11 08:28:12.578010000 -0700 +++ linux-kernel2/dat-provider/dapl_rsp_create.c 2005-05-11 13:58:29.482113000 -0700 @@ -68,20 +68,17 @@ * DAT_INVALID_STATE * DAT_CONN_QUAL_IN_USE */ -DAT_RETURN -dapl_rsp_create(DAT_IA_HANDLE ia_handle, - DAT_CONN_QUAL conn_qual, - DAT_EP_HANDLE ep_handle, - DAT_EVD_HANDLE evd_handle, DAT_RSP_HANDLE * rsp_handle) +u32 dapl_rsp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, + DAT_EP_HANDLE ep_handle, DAT_EVD_HANDLE evd_handle, + DAT_RSP_HANDLE *rsp_handle) { DAPL_IA *ia_ptr; DAPL_SP *sp_ptr; DAPL_EVD *evd_ptr; DAPL_EP *ep_ptr; boolean_t sp_found; - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; - dat_status = DAT_SUCCESS; ia_ptr = (DAPL_IA *) ia_handle; dapl_dbg_log(DAPL_DBG_TYPE_CM, diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_rsp_free.c linux-kernel2/dat-provider/dapl_rsp_free.c --- linux-kernel/dat-provider/dapl_rsp_free.c 2005-05-11 08:28:12.737008000 -0700 +++ linux-kernel2/dat-provider/dapl_rsp_free.c 2005-05-11 13:58:44.765006000 -0700 @@ -58,14 +58,12 @@ * DAT_SUCCESS * DAT_INVALID_HANDLE */ -DAT_RETURN dapl_rsp_free(DAT_RSP_HANDLE rsp_handle) +u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_handle) { DAPL_IA *ia_ptr; DAPL_SP *sp_ptr; DAPL_EP *ep_ptr; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; sp_ptr = (DAPL_SP *) rsp_handle; /* diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_rsp_query.c linux-kernel2/dat-provider/dapl_rsp_query.c --- linux-kernel/dat-provider/dapl_rsp_query.c 2005-05-10 08:13:29.299014000 -0700 +++ linux-kernel2/dat-provider/dapl_rsp_query.c 2005-05-11 13:58:53.725005000 -0700 @@ -38,11 +38,10 @@ #include "dapl.h" -DAT_RETURN dapl_rsp_query(DAT_RSP_HANDLE rsp_handle, - struct dat_rsp_param *rsp_param) +u32 dapl_rsp_query(DAT_RSP_HANDLE rsp_handle, struct dat_rsp_param *rsp_param) { DAPL_SP *sp_ptr; - DAT_RETURN status; + u32 status; if (DAPL_BAD_HANDLE(rsp_handle, DAPL_MAGIC_RSP)) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RSP); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_set_consumer_context.c linux-kernel2/dat-provider/dapl_set_consumer_context.c --- linux-kernel/dat-provider/dapl_set_consumer_context.c 2005-04-20 12:42:07.233010000 -0700 +++ linux-kernel2/dat-provider/dapl_set_consumer_context.c 2005-05-11 13:59:07.265009000 -0700 @@ -56,13 +56,11 @@ * DAT_SUCCESS * DAT_INVALID_HANDLE */ -DAT_RETURN dapl_set_consumer_context(DAT_HANDLE dat_handle, DAT_CONTEXT context) +u32 dapl_set_consumer_context(DAT_HANDLE dat_handle, DAT_CONTEXT context) { - DAT_RETURN dat_status; + u32 dat_status = DAT_SUCCESS; DAPL_HEADER *header; - dat_status = DAT_SUCCESS; - header = (DAPL_HEADER *) dat_handle; if (((header) == NULL) || ((unsigned long)(header) & 3) || diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_srq_create.c linux-kernel2/dat-provider/dapl_srq_create.c --- linux-kernel/dat-provider/dapl_srq_create.c 2005-05-10 08:13:28.397009000 -0700 +++ linux-kernel2/dat-provider/dapl_srq_create.c 2005-05-11 13:59:33.909018000 -0700 @@ -65,16 +65,12 @@ * ?DAT_INVALID_ATTRIBUTE?? * DAT_MODEL_NOT_SUPPORTED */ -DAT_RETURN -dapl_srq_create(DAT_IA_HANDLE ia_handle, - DAT_PZ_HANDLE pz_handle, - struct dat_srq_attr *srq_attr, DAT_SRQ_HANDLE * srq_handle) +u32 dapl_srq_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, + struct dat_srq_attr *srq_attr, DAT_SRQ_HANDLE *srq_handle) { DAPL_IA *ia_ptr; DAPL_SRQ *srq_ptr; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_create (%p, %p, %p, %p)\n", diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_srq_free.c linux-kernel2/dat-provider/dapl_srq_free.c --- linux-kernel/dat-provider/dapl_srq_free.c 2005-05-10 08:13:29.041072000 -0700 +++ linux-kernel2/dat-provider/dapl_srq_free.c 2005-05-11 13:59:46.496011000 -0700 @@ -59,14 +59,12 @@ * DAT_INVALID_PARAMETER * DAT_INVALID_STATE */ -DAT_RETURN dapl_srq_free(DAT_SRQ_HANDLE srq_handle) +u32 dapl_srq_free(DAT_SRQ_HANDLE srq_handle) { DAPL_SRQ *srq_ptr; DAPL_IA *ia_ptr; struct dat_srq_param *param; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_free (%p)\n", srq_handle); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_srq_post_recv.c linux-kernel2/dat-provider/dapl_srq_post_recv.c --- linux-kernel/dat-provider/dapl_srq_post_recv.c 2005-05-10 08:13:28.985006000 -0700 +++ linux-kernel2/dat-provider/dapl_srq_post_recv.c 2005-05-11 14:00:05.629017000 -0700 @@ -67,15 +67,13 @@ * DAT_PROTECTION_VIOLATION * DAT_PROVILEGES_VIOLATION */ -DAT_RETURN -dapl_srq_post_recv(DAT_SRQ_HANDLE srq_handle, - DAT_COUNT num_segments, - struct dat_lmr_triplet *local_iov, - DAT_DTO_COOKIE user_cookie) +u32 dapl_srq_post_recv(DAT_SRQ_HANDLE srq_handle, DAT_COUNT num_segments, + struct dat_lmr_triplet *local_iov, + DAT_DTO_COOKIE user_cookie) { DAPL_SRQ *srq_ptr; DAPL_COOKIE *cookie; - DAT_RETURN dat_status; + u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_post_recv (%p, %d, %p, %P)\n", diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_srq_query.c linux-kernel2/dat-provider/dapl_srq_query.c --- linux-kernel/dat-provider/dapl_srq_query.c 2005-05-10 08:13:28.434065000 -0700 +++ linux-kernel2/dat-provider/dapl_srq_query.c 2005-05-11 14:00:20.113028000 -0700 @@ -39,13 +39,10 @@ #include "dapl.h" -DAT_RETURN dapl_srq_query(DAT_SRQ_HANDLE srq_handle, - struct dat_srq_param *srq_param) +u32 dapl_srq_query(DAT_SRQ_HANDLE srq_handle, struct dat_srq_param *srq_param) { DAPL_SRQ *srq_ptr; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_query (%p, %x, %p)\n", diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_srq_resize.c linux-kernel2/dat-provider/dapl_srq_resize.c --- linux-kernel/dat-provider/dapl_srq_resize.c 2005-04-20 12:42:06.684011000 -0700 +++ linux-kernel2/dat-provider/dapl_srq_resize.c 2005-05-11 14:00:32.574007000 -0700 @@ -63,14 +63,11 @@ * DAT_INVALID_STATE */ -DAT_RETURN -dapl_srq_resize(DAT_SRQ_HANDLE srq_handle, DAT_COUNT srq_max_recv_dto) +u32 dapl_srq_resize(DAT_SRQ_HANDLE srq_handle, DAT_COUNT srq_max_recv_dto) { DAPL_IA *ia_ptr; DAPL_SRQ *srq_ptr; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_resize (%p, %d)\n", srq_handle, srq_max_recv_dto); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_srq_set_lw.c linux-kernel2/dat-provider/dapl_srq_set_lw.c --- linux-kernel/dat-provider/dapl_srq_set_lw.c 2005-04-20 12:42:07.855013000 -0700 +++ linux-kernel2/dat-provider/dapl_srq_set_lw.c 2005-05-11 14:00:43.964001000 -0700 @@ -63,12 +63,10 @@ * DAT_MODEL_NOT_SUPPORTED */ -DAT_RETURN dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, DAT_COUNT low_watermark) +u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, DAT_COUNT low_watermark) { DAPL_SRQ *srq_ptr; - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_set_lw (%p, %d)\n", srq_handle, low_watermark); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_timer_util.c linux-kernel2/dat-provider/dapl_timer_util.c --- linux-kernel/dat-provider/dapl_timer_util.c 2005-05-11 08:28:12.521015000 -0700 +++ linux-kernel2/dat-provider/dapl_timer_util.c 2005-05-11 14:01:14.200017000 -0700 @@ -91,9 +91,8 @@ void dapl_timer_init(void) * no return value * */ -DAT_RETURN -dapl_timer_set(DAPL_OS_TIMER * timer, - void (*func) (uintptr_t), void *data, DAPL_OS_TIMEVAL expires) +u32 dapl_timer_set(DAPL_OS_TIMER *timer, void (*func) (uintptr_t), + void *data, DAPL_OS_TIMEVAL expires) { DAPL_OS_TIMER *list_ptr; DAPL_OS_TIMEVAL cur_time; @@ -240,7 +239,7 @@ void dapl_timer_thread(void *arg) { DAPL_OS_TIMER *list_ptr; DAPL_OS_TIMEVAL cur_time; - DAT_RETURN dat_status; + u32 dat_status; DAPL_TIMER_HEAD *timer_head; timer_head = arg; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_timer_util.h linux-kernel2/dat-provider/dapl_timer_util.h --- linux-kernel/dat-provider/dapl_timer_util.h 2005-04-20 12:42:07.873010000 -0700 +++ linux-kernel2/dat-provider/dapl_timer_util.h 2005-05-11 14:01:34.201022000 -0700 @@ -37,8 +37,7 @@ void dapl_timer_init(void); -DAT_RETURN dapl_timer_set(DAPL_OS_TIMER * timer, - void (*func) (uintptr_t), - void *data, DAPL_OS_TIMEVAL expires); +u32 dapl_timer_set(DAPL_OS_TIMER *timer, void (*func) (uintptr_t), + void *data, DAPL_OS_TIMEVAL expires); void dapl_timer_cancel(DAPL_OS_TIMER * timer); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_util.c linux-kernel2/dat-provider/dapl_util.c --- linux-kernel/dat-provider/dapl_util.c 2005-05-06 16:22:03.058010000 -0700 +++ linux-kernel2/dat-provider/dapl_util.c 2005-05-11 14:02:26.035004000 -0700 @@ -55,7 +55,7 @@ * DAT_SUCCESS */ -DAT_RETURN dapl_os_get_time(DAPL_OS_TIMEVAL * loc) +u32 dapl_os_get_time(DAPL_OS_TIMEVAL *loc) { struct timeval tv; @@ -81,7 +81,7 @@ DAT_RETURN dapl_os_get_time(DAPL_OS_TIME * DAT_SUCCESS * DAT_INTERNAL_ERROR */ -DAT_RETURN dapl_os_wait_object_init(DAPL_OS_WAIT_OBJECT * wait_obj) +u32 dapl_os_wait_object_init(DAPL_OS_WAIT_OBJECT *wait_obj) { init_waitqueue_head(&wait_obj->wait_queue); @@ -101,13 +101,10 @@ DAT_RETURN dapl_os_wait_object_init(DAPL * DAT_TIMEOUT -- the specified time limit was reached. */ -DAT_RETURN -dapl_os_wait_object_wait(DAPL_OS_WAIT_OBJECT * wait_obj, - DAT_TIMEOUT timeout_val) +u32 dapl_os_wait_object_wait(DAPL_OS_WAIT_OBJECT *wait_obj, + DAT_TIMEOUT timeout_val) { - DAT_RETURN dat_status; - - dat_status = DAT_SUCCESS; + u32 dat_status = DAT_SUCCESS; if (DAT_TIMEOUT_INFINITE == timeout_val) { interruptible_sleep_on(&wait_obj->wait_queue); @@ -137,7 +134,7 @@ dapl_os_wait_object_wait(DAPL_OS_WAIT_OB * DAT_SUCCESS * DAT_INTERNAL_ERROR */ -DAT_RETURN dapl_os_wait_object_wakeup(DAPL_OS_WAIT_OBJECT * wait_obj) +u32 dapl_os_wait_object_wakeup(DAPL_OS_WAIT_OBJECT *wait_obj) { wake_up_interruptible(&wait_obj->wait_queue); @@ -156,7 +153,7 @@ DAT_RETURN dapl_os_wait_object_wakeup(DA * DAT_SUCCESS * DAT_INTERNAL_ERROR */ -DAT_RETURN dapl_os_wait_object_destroy(DAPL_OS_WAIT_OBJECT * wait_obj) +u32 dapl_os_wait_object_destroy(DAPL_OS_WAIT_OBJECT *wait_obj) { return DAT_SUCCESS; @@ -188,9 +185,8 @@ void dapl_thread_init(struct thread_draf * Returns: * DAT_SUCCESS */ -DAT_RETURN -dapl_os_thread_create(void (*func) (void *), - void *data, DAPL_OS_THREAD * thread_id) +u32 dapl_os_thread_create(void (*func) (void *), void *data, + DAPL_OS_THREAD * thread_id) { struct thread_draft *thread_draft; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_util.h linux-kernel2/dat-provider/dapl_util.h --- linux-kernel/dat-provider/dapl_util.h 2005-05-06 16:22:03.118012000 -0700 +++ linux-kernel2/dat-provider/dapl_util.h 2005-05-11 14:03:17.730035000 -0700 @@ -125,9 +125,8 @@ dapl_os_atomic_assign(atomic_t * v, DAT_ */ typedef int DAPL_OS_THREAD; -DAT_RETURN -dapl_os_thread_create(void (*func) (void *), - void *data, DAPL_OS_THREAD * thread_id); +u32 dapl_os_thread_create(void (*func) (void *), void *data, + DAPL_OS_THREAD *thread_id); /* * Wait Objects @@ -144,15 +143,14 @@ typedef struct _dapl_os_wait_object_ { } DAPL_OS_WAIT_OBJECT; /* function prototypes */ -DAT_RETURN dapl_os_wait_object_init(DAPL_OS_WAIT_OBJECT * wait_obj); +u32 dapl_os_wait_object_init(DAPL_OS_WAIT_OBJECT *wait_obj); -DAT_RETURN -dapl_os_wait_object_wait(DAPL_OS_WAIT_OBJECT * wait_obj, - DAT_TIMEOUT timeout_val); +u32 dapl_os_wait_object_wait(DAPL_OS_WAIT_OBJECT *wait_obj, + DAT_TIMEOUT timeout_val); -DAT_RETURN dapl_os_wait_object_wakeup(DAPL_OS_WAIT_OBJECT * wait_obj); +u32 dapl_os_wait_object_wakeup(DAPL_OS_WAIT_OBJECT *wait_obj); -DAT_RETURN dapl_os_wait_object_destroy(DAPL_OS_WAIT_OBJECT * wait_obj); +u32 dapl_os_wait_object_destroy(DAPL_OS_WAIT_OBJECT *wait_obj); /* * Memory Functions @@ -193,7 +191,7 @@ typedef struct dapl_timer_entry DAPL_OS_ typedef unsigned long long int DAPL_OS_TICKS; /* timer function prototype */ -DAT_RETURN dapl_os_get_time(DAPL_OS_TIMEVAL *); +u32 dapl_os_get_time(DAPL_OS_TIMEVAL *); /* * *printf format helpers. We use the C string constant concatenation @@ -224,12 +222,11 @@ static __inline__ long dapl_os_strtol(co /* * OS inter module functions to obtain and release verbs list */ -DAT_RETURN dapl_os_get_verbs_vector(void **verbs_list, - char *list_name, char *driver_name); +u32 dapl_os_get_verbs_vector(void **verbs_list, char *list_name, + char *driver_name); -DAT_RETURN -dapl_os_release_verbs_vector(void **verbs_list, - char *list_name, char *driver_name); +u32 dapl_os_release_verbs_vector(void **verbs_list, char *list_name, + char *driver_name); /* * helper routines diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/include/dapl_proto.h linux-kernel2/test/dapltest/include/dapl_proto.h --- linux-kernel/test/dapltest/include/dapl_proto.h 2005-05-11 08:52:07.424000000 -0700 +++ linux-kernel2/test/dapltest/include/dapl_proto.h 2005-05-11 11:16:06.658962000 -0700 @@ -523,7 +523,7 @@ boolean_t DT_check_params (Pe void DT_Test_Error (void); /* dapl_util.c */ -const char *DT_RetToString (DAT_RETURN ret_value); +const char *DT_RetToString (u32 ret_value); const char *DT_TransferTypeToString (DT_Transfer_Type type); @@ -640,7 +640,7 @@ void DT_mem_test (Params_t *params_p /* dapl_fft_queryinfo.c */ int DT_queryinfo_basic (Params_t *params_ptr, FFT_Cmd_t *cmd, FFT_query_enum object_to_query, - DAT_RETURN result_wanted); + u32 result_wanted); int DT_queryinfo_case0 (Params_t *params_ptr, FFT_Cmd_t *cmd); int DT_queryinfo_case1 (Params_t *params_ptr, FFT_Cmd_t *cmd); int DT_queryinfo_case2 (Params_t *params_ptr, FFT_Cmd_t *cmd); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/include/dapl_tdep.h linux-kernel2/test/dapltest/include/dapl_tdep.h --- linux-kernel/test/dapltest/include/dapl_tdep.h 2005-05-10 08:13:26.796006000 -0700 +++ linux-kernel2/test/dapltest/include/dapl_tdep.h 2005-05-11 11:16:06.664963000 -0700 @@ -44,21 +44,21 @@ DT_Tdep_End ( void ) ; int DT_Tdep_Execute_Test ( Params_t *params_ptr ) ; -DAT_RETURN +u32 DT_Tdep_evd_create (DAT_IA_HANDLE ia_handle, DAT_COUNT evd_min_qlen, DAT_CNO_HANDLE cno_handle, DAT_EVD_FLAGS evd_flags, DAT_EVD_HANDLE *evd_handle_ptr); -DAT_RETURN +u32 DT_Tdep_evd_free (DAT_EVD_HANDLE evd_handle); -DAT_RETURN +u32 DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_handle, DAT_TIMEOUT timeout, struct dat_event *event); -DAT_RETURN +u32 DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_handle, struct dat_event *event); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c linux-kernel2/test/dapltest/kdapl/kdapl_tdep_evd.c --- linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c 2005-05-10 10:19:18.034000000 -0700 +++ linux-kernel2/test/dapltest/kdapl/kdapl_tdep_evd.c 2005-05-11 11:16:06.670966000 -0700 @@ -94,7 +94,7 @@ KDT_Evd_Destroy(void) DT_Mdep_LockDestroy (&DT_Evd_Lock); } -DAT_RETURN +u32 DT_Tdep_evd_create (DAT_IA_HANDLE ia_handle, DAT_COUNT evd_min_qlen, DAT_CNO_HANDLE cno_handle, @@ -102,7 +102,7 @@ DT_Tdep_evd_create (DAT_IA_HANDLE ia_ha DAT_EVD_HANDLE *evd_handle_ptr) { - DAT_RETURN dat_status; + u32 dat_status; struct dat_upcall_object upcall; Tdep_Evd *evd_ptr; @@ -154,11 +154,11 @@ error: } return (dat_status); } -DAT_RETURN +u32 DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_handle, struct dat_event *dat_event) { - DAT_RETURN dat_status; + u32 dat_status; Tdep_Evd *evd_ptr; Tdep_Event *event; @@ -212,12 +212,12 @@ DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_ return dat_status; } -DAT_RETURN +u32 DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_handle, DAT_TIMEOUT timeout, struct dat_event *dat_event) { - DAT_RETURN dat_status; + u32 dat_status; Tdep_Evd *evd_ptr; Tdep_Event *event; int rc; @@ -290,7 +290,7 @@ DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_han return dat_status; } -DAT_RETURN +u32 DT_Tdep_evd_free (DAT_EVD_HANDLE evd_handle) { Tdep_Evd *next; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_bpool.c linux-kernel2/test/dapltest/test/dapl_bpool.c --- linux-kernel/test/dapltest/test/dapl_bpool.c 2005-05-11 09:01:40.530003000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_bpool.c 2005-05-11 11:16:06.675963000 -0700 @@ -101,7 +101,7 @@ DT_BpoolAlloc ( Bpool *bpool_ptr = NULL; DAT_COUNT alloc_size, bpool_size; DAT_REGION_DESCRIPTION region; - DAT_RETURN ret = DAT_SUCCESS; + u32 ret = DAT_SUCCESS; DAT_VLEN bp_len; DAT_MEM_TYPE DT_mem_type; u64 buf_region[10]; @@ -328,7 +328,7 @@ DT_Bpool_Destroy (Per_Test_Data_t * pt_p { struct dat_lmr_triplet iov; DAT_RMR_COOKIE cookie; - DAT_RETURN ret; + u32 ret; iov.virtual_address = bpool_ptr->reg_addr; iov.segment_length = 0; /* un-bind */ @@ -359,7 +359,7 @@ DT_Bpool_Destroy (Per_Test_Data_t * pt_p if (bpool_ptr->lmr_handle) { - DAT_RETURN ret = dat_lmr_free (bpool_ptr->lmr_handle); + u32 ret = dat_lmr_free (bpool_ptr->lmr_handle); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_client.c linux-kernel2/test/dapltest/test/dapl_client.c --- linux-kernel/test/dapltest/test/dapl_client.c 2005-05-11 08:59:09.049005000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_client.c 2005-05-11 11:16:06.680966000 -0700 @@ -63,7 +63,7 @@ DT_cs_Client (Params_t * params_ptr, struct dat_dto_completion_event_data dto_stat; DAT_EVENT_NUMBER event_num; unsigned char * buffp; - DAT_RETURN ret; + u32 ret; DT_Tdep_Print_Head *phead; int status = 0; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_cnxn.c linux-kernel2/test/dapltest/test/dapl_cnxn.c --- linux-kernel/test/dapltest/test/dapl_cnxn.c 2005-05-10 10:19:18.044000000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_cnxn.c 2005-05-11 11:16:06.687963000 -0700 @@ -34,7 +34,7 @@ get_ep_connection_state (DT_Tdep_Print_H DAT_EP_STATE ep_state; boolean_t in_dto_idle; boolean_t out_dto_idle; - DAT_RETURN ret; + u32 ret; char *recv_status = "Idle"; char *req_status = "Idle"; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_fft_connmgt.c linux-kernel2/test/dapltest/test/dapl_fft_connmgt.c --- linux-kernel/test/dapltest/test/dapl_fft_connmgt.c 2005-04-06 22:02:35.416990000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_fft_connmgt.c 2005-05-11 11:16:06.693960000 -0700 @@ -31,7 +31,7 @@ int DT_connmgt_case0 (Params_t *params_p { FFT_Connection_t conn; int res = 1; - DAT_RETURN rc = 0; + u32 rc = 0; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; @@ -54,7 +54,7 @@ int DT_connmgt_case1 (Params_t *params_p { FFT_Connection_t conn; int res = 1; - DAT_RETURN rc; + u32 rc; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_fft_dataxfer.c linux-kernel2/test/dapltest/test/dapl_fft_dataxfer.c --- linux-kernel/test/dapltest/test/dapl_fft_dataxfer.c 2005-04-06 22:02:35.866991000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_fft_dataxfer.c 2005-05-11 11:16:06.696963000 -0700 @@ -31,7 +31,7 @@ int DT_dataxfer_generic ( DT_Tdep_Print_Head * phead, FFT_Cmd_t *cmd, int test_case) { FFT_Connection_t conn; - DAT_RETURN rc=0; + u32 rc=0; int res=1; DT_fft_init_server (phead, cmd, &conn); DT_assert (phead, NULL != conn.ia_handle); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_fft_dataxfer_client.c linux-kernel2/test/dapltest/test/dapl_fft_dataxfer_client.c --- linux-kernel/test/dapltest/test/dapl_fft_dataxfer_client.c 2005-05-11 08:59:09.052008000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_fft_dataxfer_client.c 2005-05-11 11:16:06.700967000 -0700 @@ -37,7 +37,7 @@ int DT_dataxfer_client_generic (DT_Tdep_ { int res=1; FFT_Connection_t conn; - DAT_RETURN rc=0; + u32 rc=0; DT_fft_init_client (phead, cmd, &conn); DT_assert_dat (phead, conn.ia_handle != NULL) diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_fft_endpoint.c linux-kernel2/test/dapltest/test/dapl_fft_endpoint.c --- linux-kernel/test/dapltest/test/dapl_fft_endpoint.c 2005-05-11 08:59:09.055001000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_fft_endpoint.c 2005-05-11 11:16:06.705962000 -0700 @@ -43,7 +43,7 @@ int DT_endpoint_generic (Params_t *param DAT_EVD_HANDLE conn_evd_handle; DAT_EVD_HANDLE send_evd_handle; DAT_EVD_HANDLE recv_evd_handle; - DAT_RETURN rc, wanted; + u32 rc, wanted; int res; DT_Tdep_Print_Head *phead; @@ -171,7 +171,7 @@ int DT_endpoint_case2 (Params_t *params_ struct dat_event event; Bpool *bpool; int res; - DAT_RETURN rc; + u32 rc; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; DT_Tdep_PT_Printf (phead, "\ diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_fft_hwconn.c linux-kernel2/test/dapltest/test/dapl_fft_hwconn.c --- linux-kernel/test/dapltest/test/dapl_fft_hwconn.c 2005-05-05 13:35:54.484001000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_fft_hwconn.c 2005-05-11 11:16:06.710964000 -0700 @@ -33,7 +33,7 @@ int DT_hwconn_case0 ( Params_t *params_p char* dev_name; DAT_IA_HANDLE nic_handle; DAT_EVD_HANDLE evd_handle; - DAT_RETURN rc; + u32 rc; int res = 1; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; @@ -59,7 +59,7 @@ cleanup: int DT_hwconn_case1 ( Params_t *params_ptr, FFT_Cmd_t *cmd) { DAT_IA_HANDLE nic_handle; - DAT_RETURN rc; + u32 rc; DAT_EVD_HANDLE evd_handle; char dev_name[100]; int i; @@ -121,7 +121,7 @@ int DT_hwconn_case1 ( Params_t *params_p int DT_hwconn_case2 (Params_t *params_ptr, FFT_Cmd_t *cmd) { DAT_IA_HANDLE nic_handle; - DAT_RETURN rc; + u32 rc; int res=1; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; @@ -140,7 +140,7 @@ cleanup: int DT_hwconn_case3 (Params_t *params_ptr, FFT_Cmd_t *cmd) { FFT_Connection_t conn; - DAT_RETURN rc; + u32 rc; int res; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_fft_mem.c linux-kernel2/test/dapltest/test/dapl_fft_mem.c --- linux-kernel/test/dapltest/test/dapl_fft_mem.c 2005-05-02 12:52:13.731015000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_fft_mem.c 2005-05-11 11:16:06.715962000 -0700 @@ -33,7 +33,7 @@ /*--------------------------------------------------------*/ int DT_mem_generic (Params_t *params_ptr, FFT_Cmd_t *cmd, int flag) { - DAT_RETURN rc, expect; + u32 rc, expect; FFT_Connection_t conn; DAT_REGION_DESCRIPTION region; DAT_VLEN reg_size; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_fft_pz.c linux-kernel2/test/dapltest/test/dapl_fft_pz.c --- linux-kernel/test/dapltest/test/dapl_fft_pz.c 2005-05-11 08:59:09.058004000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_fft_pz.c 2005-05-11 11:16:06.722964000 -0700 @@ -38,7 +38,7 @@ int DT_pz_case0 ( Params_t *params_ptr, DAT_IA_HANDLE ia_handle; DAT_PZ_HANDLE pz_handle; DAT_EVD_HANDLE evd_handle; - DAT_RETURN rc; + u32 rc; int res; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; @@ -79,7 +79,7 @@ int DT_pz_case1 (Params_t *params_ptr, F DAT_PZ_HANDLE pz_handle; DAT_EP_HANDLE ep_handle; DAT_EVD_HANDLE conn_evd, send_evd, recv_evd, cr_evd; - DAT_RETURN rc; + u32 rc; int res; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; @@ -163,7 +163,7 @@ int DT_pz_case2 (Params_t *params_ptr, F DAT_IA_HANDLE ia_handle; DAT_PZ_HANDLE pz_handle; Bpool *bpool; - DAT_RETURN rc; + u32 rc; int res; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c linux-kernel2/test/dapltest/test/dapl_fft_queryinfo.c --- linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c 2005-05-10 08:16:11.583008000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_fft_queryinfo.c 2005-05-11 11:16:06.726963000 -0700 @@ -41,7 +41,7 @@ static DAT_OS_WAIT_PROXY_AGENT NULLPROXY int DT_queryinfo_basic (Params_t *params_ptr, FFT_Cmd_t *cmd, FFT_query_enum object_to_query, - DAT_RETURN result_wanted) + u32 result_wanted) { char *dev_name; DAT_IA_HANDLE ia_handle; @@ -77,7 +77,7 @@ int DT_queryinfo_basic (Params_t *params unsigned char *alloc_ptr; DT_Tdep_Print_Head *phead; - DAT_RETURN rc; + u32 rc; int res = 1; buffer_size = BUFFSIZE * sizeof (unsigned char); phead = params_ptr->phead; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_fft_util.c linux-kernel2/test/dapltest/test/dapl_fft_util.c --- linux-kernel/test/dapltest/test/dapl_fft_util.c 2005-05-11 08:59:09.062007000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_fft_util.c 2005-05-11 11:16:06.731964000 -0700 @@ -72,7 +72,7 @@ int DT_ep_create (Params_t *params_ptr, DAT_EVD_HANDLE *recv_evd, DAT_EP_HANDLE *ep_handle) { - DAT_RETURN status; + u32 status; DT_Tdep_Print_Head *phead; *conn_evd = NULL; *send_evd = NULL; @@ -146,7 +146,7 @@ void DT_fft_init_conn_struct (FFT_Connec void DT_fft_init_client (Params_t *params_ptr, FFT_Cmd_t *cmd, FFT_Connection_t *conn) { int res; - DAT_RETURN rc=0; + u32 rc=0; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; @@ -194,7 +194,7 @@ cleanup: int DT_fft_destroy_conn_struct (Params_t *params_ptr, FFT_Connection_t *conn) { - DAT_RETURN rc = DAT_SUCCESS; + u32 rc = DAT_SUCCESS; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; @@ -264,7 +264,7 @@ int DT_fft_destroy_conn_struct (Params_t void DT_fft_init_server (Params_t *params_ptr, FFT_Cmd_t *cmd, FFT_Connection_t *conn) { int res; - DAT_RETURN rc=0; + u32 rc=0; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; @@ -308,7 +308,7 @@ cleanup: void DT_fft_listen (Params_t *params_ptr, FFT_Connection_t *conn) { int res; - DAT_RETURN rc=0; + u32 rc=0; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; @@ -341,7 +341,7 @@ int DT_fft_connect (Params_t *params_ptr { int wait_count; int res; - DAT_RETURN rc=0; + u32 rc=0; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_limit.c linux-kernel2/test/dapltest/test/dapl_limit.c --- linux-kernel/test/dapltest/test/dapl_limit.c 2005-05-11 09:04:51.554002000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_limit.c 2005-05-11 11:16:06.742963000 -0700 @@ -105,7 +105,7 @@ limit_test ( DT_Tdep_Print_Head *phead, DAT_OS_WAIT_PROXY_AGENT DAT_OS_WAIT_PROXY_AGENT_NULL = {NULL, NULL}; #endif - DAT_RETURN ret; + u32 ret; # define DFLT_QLEN 10 /* a small event queue size */ # define START_COUNT 1024 /* initial # handles */ # define DFLT_BUFFSZ 4096 /* default size for buffer */ diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_performance_client.c linux-kernel2/test/dapltest/test/dapl_performance_client.c --- linux-kernel/test/dapltest/test/dapl_performance_client.c 2005-05-11 09:04:51.559004000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_performance_client.c 2005-05-11 11:16:06.745962000 -0700 @@ -91,7 +91,7 @@ DT_Performance_Test_Client_Connect ( DT_Tdep_Print_Head *phead, Performance_Test_t *test_ptr) { - DAT_RETURN ret; + u32 ret; DAT_EVENT_NUMBER event_num; unsigned int retry_cnt = 0; @@ -256,7 +256,7 @@ DT_Performance_Test_Client_Phase2 ( struct dat_rmr_triplet rmr_triplet; DAT_DTO_COOKIE cookie; struct dat_event event; - DAT_RETURN ret; + u32 ret; Performance_Ep_Context_t *ep_context; Performance_Test_Op_t *op; DT_Mdep_TimeStamp pre_ts; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_performance_server.c linux-kernel2/test/dapltest/test/dapl_performance_server.c --- linux-kernel/test/dapltest/test/dapl_performance_server.c 2005-05-11 09:04:51.566003000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_performance_server.c 2005-05-12 07:55:42.534000000 -0700 @@ -99,7 +99,7 @@ DT_Performance_Test_Server_Connect ( DT_Tdep_Print_Head *phead, Performance_Test_t *test_ptr) { - DAT_RETURN ret; + u32 ret; boolean_t status; DAT_RSP_HANDLE rsp_handle; DAT_PSP_HANDLE psp_handle; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_performance_util.c linux-kernel2/test/dapltest/test/dapl_performance_util.c --- linux-kernel/test/dapltest/test/dapl_performance_util.c 2005-05-11 09:04:51.571006000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_performance_util.c 2005-05-11 11:16:06.756962000 -0700 @@ -41,7 +41,7 @@ DT_Performance_Test_Create ( { Performance_Test_t *test_ptr; DAT_COUNT pipeline_len; - DAT_RETURN ret; + u32 ret; DT_Tdep_Print_Head *phead; phead = pt_ptr->Params.phead; @@ -268,7 +268,7 @@ DT_Performance_Test_Destroy ( Performance_Test_t *test_ptr, boolean_t is_server) { - DAT_RETURN ret; + u32 ret; DAT_EP_HANDLE ep_handle; DT_Tdep_Print_Head *phead; int status = 0; @@ -427,7 +427,7 @@ DT_performance_post_rdma_op ( DT_Mdep_TimeStamp pre_ts; DT_Mdep_TimeStamp post_ts; DAT_DTO_COOKIE cookie; - DAT_RETURN ret; + u32 ret; Performance_Test_Op_t *op = &ep_context->op; struct dat_lmr_triplet *iov = DT_Bpool_GetIOV(op->bp, 0); struct dat_rmr_triplet rmr_triplet; @@ -525,7 +525,7 @@ DT_performance_wait ( { DAT_COUNT i; DAT_COUNT queue_size; - DAT_RETURN ret; + u32 ret; struct dat_event event; unsigned long pre_ctxt_num; unsigned long post_ctxt_num; @@ -614,7 +614,7 @@ DT_performance_poll ( DAT_EVD_HANDLE evd_handle, Performance_Stats_t *stats) { - DAT_RETURN ret; + u32 ret; struct dat_event event; unsigned long pre_ctxt_num; unsigned long post_ctxt_num; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_server.c linux-kernel2/test/dapltest/test/dapl_server.c --- linux-kernel/test/dapltest/test/dapl_server.c 2005-05-11 08:59:09.094002000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_server.c 2005-05-11 11:16:06.763960000 -0700 @@ -52,7 +52,7 @@ DT_cs_Server (Params_t * params_ptr) DAT_DTO_COOKIE dto_cookie; struct dat_dto_completion_event_data dto_stat; - DAT_RETURN ret; + u32 ret; DT_Tdep_Print_Head *phead; phead = params_ptr->phead; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_test_util.c linux-kernel2/test/dapltest/test/dapl_test_util.c --- linux-kernel/test/dapltest/test/dapl_test_util.c 2005-05-11 09:04:51.577002000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_test_util.c 2005-05-11 11:16:06.768963000 -0700 @@ -38,7 +38,7 @@ DT_query ( Per_Test_Data_t *pt_ptr, unsigned char *module = "DT_query"; DAT_EVD_HANDLE async_evd_hdl; /* not used */ struct dat_ep_param ep_params; - DAT_RETURN ret; + u32 ret; DT_Tdep_Print_Head *phead; phead = pt_ptr->Params.phead; @@ -169,7 +169,7 @@ DT_post_recv_buffer (DT_Tdep_Print_Head struct dat_lmr_triplet *iov = DT_Bpool_GetIOV(bp, index); DAT_LMR_CONTEXT lmr_c = DT_Bpool_GetLMR (bp, index); DAT_DTO_COOKIE cookie; - DAT_RETURN ret; + u32 ret; /* * Prep the inputs @@ -214,7 +214,7 @@ DT_post_send_buffer (DT_Tdep_Print_Head struct dat_lmr_triplet *iov = DT_Bpool_GetIOV(bp, index); DAT_LMR_CONTEXT lmr_c = DT_Bpool_GetLMR (bp, index); DAT_DTO_COOKIE cookie; - DAT_RETURN ret; + u32 ret; /* * Prep the inputs @@ -258,7 +258,7 @@ DT_cr_event_wait ( DT_Tdep_Print_Head *p for (;;) { - DAT_RETURN ret; + u32 ret; struct dat_event event; ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); @@ -313,7 +313,7 @@ DT_conn_event_wait (DT_Tdep_Print_Head * { for (;;) { - DAT_RETURN ret; + u32 ret; struct dat_event event; ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); @@ -366,7 +366,7 @@ DT_disco_event_wait ( DT_Tdep_Print_Head { for (;;) { - DAT_RETURN ret; + u32 ret; struct dat_event event; ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); @@ -437,7 +437,7 @@ DT_dto_event_poll (DT_Tdep_Print_Head *p { for (;;) { - DAT_RETURN ret; + u32 ret; struct dat_event event; ret = DT_Tdep_evd_dequeue ( evd_handle, @@ -489,7 +489,7 @@ DT_dto_event_wait (DT_Tdep_Print_Head *p { for (;;) { - DAT_RETURN ret; + u32 ret; struct dat_event event; ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); @@ -533,7 +533,7 @@ DT_rmr_event_wait (DT_Tdep_Print_Head *p { for (;;) { - DAT_RETURN ret; + u32 ret; struct dat_event event; ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); @@ -664,7 +664,7 @@ DT_cr_check ( DT_Tdep_Print_Head *phead DAT_CR_HANDLE *cr_handlep, char *message) { - DAT_RETURN ret; + u32 ret; if (cr_handlep) { diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_transaction_test.c linux-kernel2/test/dapltest/test/dapl_transaction_test.c --- linux-kernel/test/dapltest/test/dapl_transaction_test.c 2005-05-11 09:04:51.593002000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_transaction_test.c 2005-05-11 11:16:06.783962000 -0700 @@ -229,7 +229,7 @@ void DT_Transaction_Main (void *param) { Transaction_Test_t *test_ptr = (Transaction_Test_t *) param; - DAT_RETURN ret; + u32 ret; u32 i, j; boolean_t success = FALSE; Per_Test_Data_t *pt_ptr; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_transaction_util.c linux-kernel2/test/dapltest/test/dapl_transaction_util.c --- linux-kernel/test/dapltest/test/dapl_transaction_util.c 2005-05-11 09:04:51.599004000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_transaction_util.c 2005-05-11 11:16:06.789964000 -0700 @@ -43,7 +43,7 @@ DT_handle_post_recv_buf (DT_Tdep_Print_H Transaction_Test_Op_t *op = &ep_context[i].op[op_indx]; struct dat_lmr_triplet *iov = DT_Bpool_GetIOV(op->bp, 0); DAT_DTO_COOKIE cookie; - DAT_RETURN ret; + u32 ret; /* Prep the inputs */ for (j = 0; j < op->num_segs; j++) @@ -105,7 +105,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p Transaction_Test_Op_t *op = &ep_context[i].op[op_indx]; struct dat_lmr_triplet *iov = DT_Bpool_GetIOV(op->bp, 0); DAT_DTO_COOKIE cookie; - DAT_RETURN ret; + u32 ret; /* Prep the inputs */ for (j = 0; j < op->num_segs; j++) @@ -470,7 +470,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p boolean_t poll) { unsigned int i, j; - DAT_RETURN ret; + u32 ret; unsigned char *completion_reaped; completion_reaped = DT_Mdep_Malloc (num_eps * sizeof (unsigned char)); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_util.c linux-kernel2/test/dapltest/test/dapl_util.c --- linux-kernel/test/dapltest/test/dapl_util.c 2005-05-05 09:42:06.416005000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_util.c 2005-05-11 11:16:06.794962000 -0700 @@ -28,11 +28,11 @@ #include "dapl_proto.h" /* - * Map DAT_RETURN values to readable strings, + * Map u32 values to readable strings, * but don't assume the values are zero-based or contiguous. */ const char * -DT_RetToString (DAT_RETURN ret_value) +DT_RetToString (u32 ret_value) { const char *major_msg, *minor_msg; int sz; @@ -61,7 +61,7 @@ DT_AsyncErr2Str (DAT_EVENT_NUMBER error_ static struct { const char *name; - DAT_RETURN value; + u32 value; } dat_errors[] = { # define DATxx(x) { # x, x } @@ -106,7 +106,7 @@ DT_EventToSTr (DAT_EVENT_NUMBER event_co static struct { const char *name; - DAT_RETURN value; + u32 value; } dat_events[] = { @@ -154,7 +154,7 @@ DT_State2Str (DAT_EP_STATE state_code) static struct { const char *name; - DAT_RETURN value; + u32 value; } dat_state[] = { From halr at voltaire.com Thu May 12 08:29:41 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 12 May 2005 11:29:41 -0400 Subject: [openib-general] [PATCH] [TRIVIAL] Eliminate comment for unused parameter to dapl_evd_connection_callback Message-ID: <1115911780.4479.15.camel@localhost.localdomain> Eliminate comment for unused parameter to dapl_evd_connection_callback Signed-off-by: Hal Rosenstock Index: dapl_evd_connection_callb.c =================================================================== --- dapl_evd_connection_callb.c (revision 2315) +++ dapl_evd_connection_callb.c (working copy) @@ -54,7 +54,6 @@ * ib_cm_event * private_data_ptr * context (evd) - * cr_pp * * Output: * None From halr at voltaire.com Thu May 12 08:43:17 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 12 May 2005 11:43:17 -0400 Subject: [openib-general] Latest kdapl load problem Message-ID: <1115912597.4479.29.camel@localhost.localdomain> Just rebuilt with the latest from the tree and get this when insmod'ing kdapltest after modprobe'ing dat and the ib_dat_provider. dmesg shows: KDAPL : Added device mthca0a to the registry KDAPL : Added device mthca0b to the registry kdapltest: disagrees about version of symbol dat_strerror kdapltest: Unknown symbol dat_strerror kdapltest: disagrees about version of symbol dat_ia_openv kdapltest: Unknown symbol dat_ia_openv kdapltest: disagrees about version of symbol dat_ia_close kdapltest: Unknown symbol dat_ia_close Any ideas ? -- Hal From mshefty at ichips.intel.com Thu May 12 08:52:44 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Thu, 12 May 2005 08:52:44 -0700 Subject: [openib-general] Re: DAPL CM Reject Questions In-Reply-To: <1115901591.4766.38.camel@localhost.localdomain> References: <1115901591.4766.38.camel@localhost.localdomain> Message-ID: <42837BCC.20003@ichips.intel.com> Hal Rosenstock wrote: > On the passive side, an RTU is received but the QP cannot be transition > to RTS so a REJ is attempted as below: > > static int do_rtu_recv(struct dapl_cm_id *conn) > { > int status; > > status = dapl_modify_qp_state_to_rts(conn->cm_id, conn->ep->qp_handle); > if (status) { > dapl_dbg_log(DAPL_DBG_TYPE_ERR, " do_rtu_recv: could not " > "modify QP state to RTS status %d\n", status); > goto reject; > } > ... > > reject: > ib_send_cm_rej(conn->cm_id, IB_CM_REJ_CONSUMER_DEFINED, NULL, 0, > NULL, 0); This should call ib_send_cm_dreq(), rather than a reject. > If RTU is received, the connection would be in ESTABLISHED state. Is > sending REJ valid there ? What's odd to me is that the CM state machine > in the IBA spec shows REJ being received in the established state on the > active side (and moving to TIMEWAIT) but there is no action out of > established on the passive side showing send REJ. On the passive side, the user can time out waiting for an RTU and issue a reject. But the RTU may have been sent (and lost) on the active side. So, the active side would be in the ESTABLISHED state when the REJ is received. This cannot happen on the passive side. > If sending REJ is valid on the passive side in established state, does > the current CM handle this ? It looks to me like it returns -EINVAL for > this case. See above... I think that the CM code is correct, and my kDAPL implementation is in error. - Sean From jlentini at netapp.com Thu May 12 08:52:52 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 12 May 2005 11:52:52 -0400 (EDT) Subject: [openib-general] Re: Latest kdapl load problem In-Reply-To: <1115912597.4479.29.camel@localhost.localdomain> References: <1115912597.4479.29.camel@localhost.localdomain> Message-ID: Did you rebuild the kdapltest module? On Thu, 12 May 2005, Hal Rosenstock wrote: halr> Just rebuilt with the latest from the tree and get this when insmod'ing halr> kdapltest after modprobe'ing dat and the ib_dat_provider. halr> halr> dmesg shows: halr> KDAPL : Added device mthca0a to the registry halr> KDAPL : Added device mthca0b to the registry halr> kdapltest: disagrees about version of symbol dat_strerror halr> kdapltest: Unknown symbol dat_strerror halr> kdapltest: disagrees about version of symbol dat_ia_openv halr> kdapltest: Unknown symbol dat_ia_openv halr> kdapltest: disagrees about version of symbol dat_ia_close halr> kdapltest: Unknown symbol dat_ia_close halr> halr> Any ideas ? halr> halr> -- Hal halr> From iod00d at hp.com Thu May 12 08:55:50 2005 From: iod00d at hp.com (Grant Grundler) Date: Thu, 12 May 2005 08:55:50 -0700 Subject: [openib-general] Re: user-mode verbs on Itanium In-Reply-To: <20050511072138.GB13349@mellanox.co.il> References: <20050429183846.GH24871@esmail.cup.hp.com> <20050504005534.GA13574@esmail.cup.hp.com> <20050504070910.GC9674@mellanox.co.il> <20050506190947.GA23958@esmail.cup.hp.com> <20050507053706.GN23958@esmail.cup.hp.com> <20050508142345.GQ9941@mellanox.co.il> <20050510152428.GA5495@esmail.cup.hp.com> <20050510154030.GH2474@mellanox.co.il> <20050511012148.GE5495@esmail.cup.hp.com> <20050511072138.GB13349@mellanox.co.il> Message-ID: <20050512155550.GA14123@esmail.cup.hp.com> On Wed, May 11, 2005 at 10:21:38AM +0300, Michael S. Tsirkin wrote: > You mean just sample the clock before the first post? Yes. > That would be easy, but lets look at what does it measure: > > On the client, you just get the time it takes to perform the > first post_send operation. If this is interesting to you, I agree, > but lets take the timestamps around each post send operation to > make it statistically relevant. > Let me know - and need to think of a way to do this without > affecting latency numbers. Yes, I'd like to try that. But a few more of my pending patches need to either be rejected or (preferably) accepted. Given the roundtrip time is in the 10000-20000 cycle range, the additional cost of reading the ITC should be light weight enough on most architectures. ISTR it's ~36 cycles on IA64. We could do two get_cycle calls back-to-back at the start and substract that from any measurements. > On the server, you get that, plus > the latency, plus time it takes the OS to wake the client from a blocking > socket operation. It's the last component that isnt IB-related and that > makes it a non-interesting. I think it's interesting but doesn't have to be part of this test. If we meaure it, people will be aware of it when they use rdma_lat.c as an example for their own code. > By the way, what do you use for statistics? I normally use netperf which includes statistical-aware sampling. I haven't used anything for this perf test. But the results look fairly reliable. > > ok. Any other theories on why it takes ~12x longer (4.7 vs 63usec)? The wakeup is *probably* a big chunk of the 63usec...but I'm still just guessing. > Hard to tell for sure. That needs to be investigated. Ok. I don't have the HW or time to chase that. I'm just hoping that by getting the test to measure the right things, it will provide the people who can, some ideas on what needs to be investigated. thanks, grant From halr at voltaire.com Thu May 12 09:02:42 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 12 May 2005 12:02:42 -0400 Subject: [openib-general] Re: Latest kdapl load problem In-Reply-To: References: <1115912597.4479.29.camel@localhost.localdomain> Message-ID: <1115913762.4479.40.camel@localhost.localdomain> On Thu, 2005-05-12 at 11:52, James Lentini wrote: > Did you rebuild the kdapltest module? Thanks. I forgot to do that. Does kdapltest build fine for you ? I'm getting some errors on building the user portion. -- Hal From jlentini at netapp.com Thu May 12 09:08:31 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 12 May 2005 12:08:31 -0400 (EDT) Subject: [openib-general] Re: Latest kdapl load problem In-Reply-To: <1115913762.4479.40.camel@localhost.localdomain> References: <1115912597.4479.29.camel@localhost.localdomain> <1115913762.4479.40.camel@localhost.localdomain> Message-ID: If you do an update those should disappear. On Thu, 12 May 2005, Hal Rosenstock wrote: halr> On Thu, 2005-05-12 at 11:52, James Lentini wrote: halr> > Did you rebuild the kdapltest module? halr> halr> Thanks. I forgot to do that. halr> halr> Does kdapltest build fine for you ? I'm getting some errors on building halr> the user portion. halr> halr> -- Hal halr> From halr at voltaire.com Thu May 12 09:06:04 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 12 May 2005 12:06:04 -0400 Subject: [openib-general] Re: Latest kdapl load problem In-Reply-To: <1115913762.4479.40.camel@localhost.localdomain> References: <1115912597.4479.29.camel@localhost.localdomain> <1115913762.4479.40.camel@localhost.localdomain> Message-ID: <1115913964.4479.42.camel@localhost.localdomain> On Thu, 2005-05-12 at 12:02, Hal Rosenstock wrote: > On Thu, 2005-05-12 at 11:52, James Lentini wrote: > > Did you rebuild the kdapltest module? > > Thanks. I forgot to do that. > > Does kdapltest build fine for you ? I'm getting some errors on building > the user portion. OK. r2318 fixes that. -- Hal From tduffy at sun.com Thu May 12 09:33:28 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 12 May 2005 09:33:28 -0700 Subject: [openib-general] Re: [PATCHv2] kDAPL: remove DAT_RETURN In-Reply-To: References: <1115910070.25530.2.camel@duffman> Message-ID: <1115915608.9070.0.camel@duffman> On Thu, 2005-05-12 at 11:50 -0400, James Lentini wrote: > Thanks for the updated patch Tom. It is committed in revision 2316. > > Eventually we will need to remove our dat_return_* enums and replace > our return values with errno values. Correct? That is right. This is just a first pass at cleaning it up. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From info at qsv18.com Thu May 12 07:07:47 2005 From: info at qsv18.com (info at qsv18.com) Date: 12 May 2005 23:07:47 +0900 Subject: [openib-general] (no subject) Message-ID: <20050512140747.15109.qmail@mail.qsv18.com> From info at qsv18.com Thu May 12 07:07:56 2005 From: info at qsv18.com (info at qsv18.com) Date: 12 May 2005 23:07:56 +0900 Subject: [openib-general] YUMI$B!!$5$s$+$i $B5U!{=u4uK>$N?M:J$5$s$+$i5.J}08$G%a%C%;!<%8$rD:$-$^$7$?!*(B $BAw?.0MMj $B%?%$%H%k(B:$B$3$s$K$A$O!#(B $BK\J8(B: $B!!FMA3$N%a!<%k$OLBOG$+$J!A!);d$O!!$f$_!!$H$$$$$^$9!#(B $B!!C6Fa$OKG0W;v6H$N7P1D$r$7$F$*$j!";E;vJA!"#1G/$N$&$A#2#5#0F|0J>e$O3$30(B $B$K9T$C$F$*$j$^$9!#0JA0!"%5!<%/%k$GCN$j9g$C$?J}$HNI$$$*IU$-9g$$$r$5$;$F(B $B$$$?$@$$$?$N$G$9$,!"H>G/4V$K#1 at iK|$b$7(B $B$G$9!#7n$K#1#0#0$^$s0J>e$O=P$5$J$$$H7h$a$F$*$j$^$9$N$G!"$=$l$GK~B-$7$F(B $B$$$?$@$1$kJ}!"$*BT$A$7$F$*$j$^$9!#(B $B!!$"!"$f$_$N7HBSHV9fM_$7$$$J$i$$$D$G$b8@$C$F$M$'!A(B $B!a!a!a!a!a!a!a!a!a!a!a!a!a!a!a!a!a!a!a!a!a!a!a!a!a!a!a!a!a!a!a!a!a!a!a(B $B!!!!(B $B"#$f$_!!$5$s$N8D?M>pJs"#(B $B!!!!!!!!!!!!!!!!!!!!!!(BID$BHV9f!'(B103369 $B!!!!!!!!!!!!!!!!!!!!!!G/Np!'!!;0==Be(B $B!!!!!!!!!!!!!!!!!!!!!!D>-d!'!!!{!!HsDLCN!'!_(B $B!!!!!!!!!!!!!!!!!!!!!!R2pR2pR2p$9$k$N$G!"CO0h$N(B $B!!!!ItJ,$r at 53N$K$45-F~$N>eJV?.$r$J$5$k$h$&8f4j$$CW$7$^$9!#4uK>=w(B $B!!!!@-$N References: <1115910070.25530.2.camel@duffman> <1115915608.9070.0.camel@duffman> <469958e005051209465c1b299@mail.gmail.com> Message-ID: <1115919025.9070.6.camel@duffman> [[ Please, Caitlin, keep discussion on list ]] On Thu, 2005-05-12 at 09:46 -0700, Caitlin Bestler wrote: > Which of the following are you thinking of? > > a) changing enums to #defines? > b) changing the value of the enums > c) eliminating the entire bit-stuffed layered error code approach I think c. As this is how the rest of the kernel deals with return values and errnos. > If either of the latter two are contemplated, how will the code > interoperate with user mode applications coded with the > reference dat include files? A shim layer? Not a kernel problem. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From tduffy at sun.com Thu May 12 10:35:33 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 12 May 2005 10:35:33 -0700 Subject: [openib-general] [PATCH] kDAPL: remove use of DAT_COUNT Message-ID: <1115919333.9070.9.camel@duffman> Remove DAT_COUNT, just use int. Plus a few whitespace fixes thrown in for good measure. I am trying to clean these things up as I see them. Signed-off-by: Tom Duffy Index: linux-kernel/test/dapltest/test/dapl_performance_util.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_performance_util.c (revision 2321) +++ linux-kernel/test/dapltest/test/dapl_performance_util.c (working copy) @@ -40,7 +40,7 @@ DT_Performance_Test_Create ( Performance_Test_t **perf_test) { Performance_Test_t *test_ptr; - DAT_COUNT pipeline_len; + int pipeline_len; u32 ret; DT_Tdep_Print_Head *phead; @@ -523,8 +523,8 @@ DT_performance_wait ( DAT_EVD_HANDLE evd_handle, Performance_Stats_t *stats) { - DAT_COUNT i; - DAT_COUNT queue_size; + int i; + int queue_size; u32 ret; struct dat_event event; unsigned long pre_ctxt_num; Index: linux-kernel/test/dapltest/test/dapl_limit.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_limit.c (revision 2321) +++ linux-kernel/test/dapltest/test/dapl_limit.c (working copy) @@ -1122,8 +1122,8 @@ limit_test ( DT_Tdep_Print_Head *phead, */ if (depth == LIM_SIZE_LMR) { - DAT_COUNT last_size = 0; - DAT_COUNT test_size = DFLT_BUFFSZ; + int last_size = 0; + int test_size = DFLT_BUFFSZ; Bpool *test_bpool; for (;;) { Index: linux-kernel/test/dapltest/test/dapl_transaction_test.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_transaction_test.c (revision 2321) +++ linux-kernel/test/dapltest/test/dapl_transaction_test.c (working copy) @@ -742,7 +742,7 @@ retry: */ { struct dat_event event; - DAT_COUNT drained = 0; + int drained = 0; dat_ep_reset (test_ptr->ep_context[i].ep_handle); do Index: linux-kernel/test/dapltest/test/dapl_bpool.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_bpool.c (revision 2321) +++ linux-kernel/test/dapltest/test/dapl_bpool.c (working copy) @@ -90,16 +90,16 @@ DT_BpoolAlloc ( DAT_PZ_HANDLE pz_handle, DAT_EP_HANDLE ep_handle, DAT_EVD_HANDLE rmr_evd_handle, - DAT_COUNT seg_size, - DAT_COUNT num_segs, - DAT_COUNT alignment, + int seg_size, + int num_segs, + int alignment, boolean_t enable_rdma_write, boolean_t enable_rdma_read) { unsigned char *module = "DT_BpoolAlloc"; unsigned char *alloc_ptr = NULL; Bpool *bpool_ptr = NULL; - DAT_COUNT alloc_size, bpool_size; + int alloc_size, bpool_size; DAT_REGION_DESCRIPTION region; u32 ret = DAT_SUCCESS; DAT_VLEN bp_len; @@ -392,7 +392,7 @@ DT_Bpool_GetRegBuff (Bpool * bpool_ptr, } /*****************************************************************************/ -DAT_COUNT +int DT_Bpool_GetBuffSize (Bpool * bpool_ptr, int index) { return ( bpool_ptr->seg_size ); Index: linux-kernel/test/dapltest/test/dapl_test_util.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_test_util.c (revision 2321) +++ linux-kernel/test/dapltest/test/dapl_test_util.c (working copy) @@ -573,7 +573,7 @@ boolean_t DT_dto_check ( DT_Tdep_Print_Head *phead, struct dat_dto_completion_event_data *dto_p, DAT_EP_HANDLE ep_expected, - DAT_COUNT len_expected, + int len_expected, DAT_DTO_COOKIE cookie_expected, char *message ) { Index: linux-kernel/test/dapltest/test/dapl_util.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_util.c (revision 2321) +++ linux-kernel/test/dapltest/test/dapl_util.c (working copy) @@ -189,7 +189,7 @@ DT_State2Str (DAT_EP_STATE state_code) * and do the correct thing if align == 0. */ unsigned char * -DT_AlignPtr (void * val, DAT_COUNT align) +DT_AlignPtr (void * val, int align) { if (align) { @@ -199,8 +199,8 @@ DT_AlignPtr (void * val, DAT_COUNT align return (val); } -DAT_COUNT -DT_RoundSize (DAT_COUNT val, DAT_COUNT align) +int +DT_RoundSize (int val, int align) { if (align) { Index: linux-kernel/test/dapltest/test/dapl_client.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_client.c (revision 2321) +++ linux-kernel/test/dapltest/test/dapl_client.c (working copy) @@ -276,7 +276,7 @@ retry: if ( event_num == DAT_CONNECTION_EVENT_PEER_REJECTED ) { struct dat_event event; - DAT_COUNT drained = 0; + int drained = 0; DT_Mdep_Sleep (1000); DT_Tdep_PT_Printf (phead, Index: linux-kernel/test/dapltest/include/dapl_tdep.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_tdep.h (revision 2321) +++ linux-kernel/test/dapltest/include/dapl_tdep.h (working copy) @@ -46,7 +46,7 @@ DT_Tdep_Execute_Test ( Params_t *params_ u32 DT_Tdep_evd_create (DAT_IA_HANDLE ia_handle, - DAT_COUNT evd_min_qlen, + int evd_min_qlen, DAT_CNO_HANDLE cno_handle, DAT_EVD_FLAGS evd_flags, DAT_EVD_HANDLE *evd_handle_ptr); Index: linux-kernel/test/dapltest/include/dapl_global.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_global.h (revision 2321) +++ linux-kernel/test/dapltest/include/dapl_global.h (working copy) @@ -30,7 +30,7 @@ #include "dapl_proto.h" -extern DAT_COUNT DT_dapltest_debug; +extern int DT_dapltest_debug; extern boolean_t DT_local_is_little_endian; #endif Index: linux-kernel/test/dapltest/include/dapl_performance_test.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_performance_test.h (revision 2321) +++ linux-kernel/test/dapltest/include/dapl_performance_test.h (working copy) @@ -57,7 +57,7 @@ typedef struct DAT_EP_HANDLE ep_handle; struct dat_ep_attr ep_attr; DAT_CONN_QUAL port; - DAT_COUNT pipeline_len; + int pipeline_len; Bpool *bp; Performance_Test_Op_t op; } Performance_Ep_Context_t; @@ -73,13 +73,13 @@ typedef struct DAT_IA_HANDLE ia_handle; DAT_PZ_HANDLE pz_handle; DAT_CNO_HANDLE cno_handle; - DAT_COUNT reqt_evd_length; + int reqt_evd_length; DAT_EVD_HANDLE reqt_evd_hdl; /* request+rmr */ - DAT_COUNT recv_evd_length; + int recv_evd_length; DAT_EVD_HANDLE recv_evd_hdl; /* receive */ - DAT_COUNT conn_evd_length; + int conn_evd_length; DAT_EVD_HANDLE conn_evd_hdl; /* connect */ - DAT_COUNT creq_evd_length; + int creq_evd_length; DAT_EVD_HANDLE creq_evd_hdl; /* "" request */ Performance_Ep_Context_t ep_context; } Performance_Test_t; Index: linux-kernel/test/dapltest/include/dapl_proto.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_proto.h (revision 2321) +++ linux-kernel/test/dapltest/include/dapl_proto.h (working copy) @@ -85,9 +85,9 @@ Bpool * DT_BpoolAlloc (Per_Test_ DAT_PZ_HANDLE pz_handle, DAT_EP_HANDLE ep_handle, DAT_EVD_HANDLE rmr_evd_handle, - DAT_COUNT seg_size, - DAT_COUNT num_segs, - DAT_COUNT alignment, + int seg_size, + int num_segs, + int alignment, boolean_t enable_rdma_write, boolean_t enable_rdma_read); @@ -97,7 +97,7 @@ boolean_t DT_Bpool_Destroy (P unsigned char *DT_Bpool_GetBuffer (Bpool * bpool_ptr, int index); DAT_VADDR DT_Bpool_GetRegBuff (Bpool * bpool_ptr, int index); -DAT_COUNT DT_Bpool_GetBuffSize (Bpool * bpool_ptr, int index); +int DT_Bpool_GetBuffSize (Bpool * bpool_ptr, int index); struct dat_lmr_triplet *DT_Bpool_GetIOV (Bpool * bpool_ptr, int index); DAT_LMR_CONTEXT DT_Bpool_GetLMR (Bpool * bpool_ptr, int index); DAT_RMR_CONTEXT DT_Bpool_GetRMR (Bpool * bpool_ptr, int index); @@ -398,7 +398,7 @@ boolean_t DT_rmr_event_wait ( boolean_t DT_dto_check ( DT_Tdep_Print_Head* phead, struct dat_dto_completion_event_data *dto_p, DAT_EP_HANDLE ep_expected, - DAT_COUNT len_expected, + int len_expected, DAT_DTO_COOKIE cookie_expected, char *message); @@ -537,9 +537,9 @@ DAT_QOS DT_ParseQoS (char *arg); DAT_MEM_TYPE DT_ParseMemType (char *arg); -unsigned char *DT_AlignPtr (void * val, DAT_COUNT align); +unsigned char *DT_AlignPtr (void * val, int align); -DAT_COUNT DT_RoundSize (DAT_COUNT val, DAT_COUNT align); +int DT_RoundSize (int val, int align); /* dapl_limit_cmd.c */ void DT_Limit_Cmd_Init ( Limit_Cmd_t * cmd); Index: linux-kernel/test/dapltest/include/dapl_bpool.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_bpool.h (revision 2321) +++ linux-kernel/test/dapltest/include/dapl_bpool.h (working copy) @@ -39,8 +39,8 @@ struct Bpool_tag { unsigned char *alloc_ptr; uint32_t alloc_size; DAT_PZ_HANDLE pz_handle; - DAT_COUNT seg_size; - DAT_COUNT num_segs; /* num segments */ + int seg_size; + int num_segs; /* num segments */ unsigned char *buffer_start; /* Start of buffer area */ DAT_VLEN buffer_size; /* Size of data buffer (rounded) */ DAT_VADDR reg_addr; /* start of registered area */ Index: linux-kernel/test/dapltest/include/dapl_fft_util.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_fft_util.h (revision 2321) +++ linux-kernel/test/dapltest/include/dapl_fft_util.h (working copy) @@ -61,7 +61,7 @@ typedef struct DAT_EP_HANDLE ep_handle; DAT_EVD_HANDLE cr_evd, conn_evd, send_evd, recv_evd; struct dat_event event; - DAT_COUNT count; + int count; DAT_CR_HANDLE cr_handle; Bpool *bpool; struct dat_cr_arrival_event_data cr_stat; Index: linux-kernel/test/dapltest/include/dapl_performance_stats.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_performance_stats.h (revision 2321) +++ linux-kernel/test/dapltest/include/dapl_performance_stats.h (working copy) @@ -58,7 +58,7 @@ typedef struct Performance_Stats_Data_t latency; DT_CpuStat pre_cpu_stat; DT_CpuStat post_cpu_stat; - DAT_COUNT pipeline_len; + int pipeline_len; } Performance_Stats_t; #endif Index: linux-kernel/test/dapltest/include/dapl_transaction_test.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_transaction_test.h (revision 2321) +++ linux-kernel/test/dapltest/include/dapl_transaction_test.h (working copy) @@ -80,7 +80,7 @@ typedef struct DAT_IA_ADDRESS_PTR remote_ia_addr; DAT_CONN_QUAL base_port; DAT_TIMEOUT time_out; - DAT_COUNT evd_length; + int evd_length; Thread *thread; /* This group set up by each thread in DT_Transaction_Main() */ Index: linux-kernel/test/dapltest/cmd/dapl_test_data.c =================================================================== --- linux-kernel/test/dapltest/cmd/dapl_test_data.c (revision 2321) +++ linux-kernel/test/dapltest/cmd/dapl_test_data.c (working copy) @@ -27,7 +27,7 @@ #include "dapl_proto.h" -DAT_COUNT DT_dapltest_debug = 0; +int DT_dapltest_debug = 0; /* * check memory leaking int alloc_count; DT_Mdep_LockType * Alloc_Count_Lock; Index: linux-kernel/test/dapltest/common/dapl_global.c =================================================================== --- linux-kernel/test/dapltest/common/dapl_global.c (revision 2321) +++ linux-kernel/test/dapltest/common/dapl_global.c (working copy) @@ -28,6 +28,6 @@ #include "dapl_proto.h" boolean_t DT_local_is_little_endian; -DAT_COUNT DT_dapltest_debug = 0; +int DT_dapltest_debug = 0; Index: linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c =================================================================== --- linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c (revision 2321) +++ linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c (working copy) @@ -96,7 +96,7 @@ KDT_Evd_Destroy(void) u32 DT_Tdep_evd_create (DAT_IA_HANDLE ia_handle, - DAT_COUNT evd_min_qlen, + int evd_min_qlen, DAT_CNO_HANDLE cno_handle, DAT_EVD_FLAGS evd_flags, DAT_EVD_HANDLE *evd_handle_ptr) Index: linux-kernel/dat-provider/dapl_cookie.c =================================================================== --- linux-kernel/dat-provider/dapl_cookie.c (revision 2321) +++ linux-kernel/dat-provider/dapl_cookie.c (working copy) @@ -113,9 +113,9 @@ u32 dapl_cb_put(DAPL_COOKIE_BUFFER *buff * DAT_INSUFFICIENT_RESOURCES * */ -u32 dapl_cb_create(DAPL_COOKIE_BUFFER *buffer, DAPL_EP *ep, DAT_COUNT size) +u32 dapl_cb_create(DAPL_COOKIE_BUFFER *buffer, DAPL_EP *ep, int size) { - DAT_COUNT i; + int i; /* * allocate one additional entry so that the tail @@ -181,7 +181,7 @@ void dapl_cb_free(DAPL_COOKIE_BUFFER * b u32 dapl_cb_get(DAPL_COOKIE_BUFFER *buffer, DAPL_COOKIE **cookie_ptr) { u32 dat_status; - DAT_COUNT new_head; + int new_head; dapl_os_assert(NULL != cookie_ptr); Index: linux-kernel/dat-provider/dapl_ep_connect.c =================================================================== --- linux-kernel/dat-provider/dapl_ep_connect.c (revision 2321) +++ linux-kernel/dat-provider/dapl_ep_connect.c (working copy) @@ -45,7 +45,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_handle, DAT_IA_ADDRESS_PTR remote_ia_address, DAT_CONN_QUAL remote_conn_qual, DAT_TIMEOUT timeout, - DAT_COUNT private_data_size, const void *private_data, + int private_data_size, const void *private_data, DAT_QOS qos, DAT_CONNECT_FLAGS connect_flags) { DAPL_EP *ep_ptr; Index: linux-kernel/dat-provider/dapl_cookie.h =================================================================== --- linux-kernel/dat-provider/dapl_cookie.h (revision 2321) +++ linux-kernel/dat-provider/dapl_cookie.h (working copy) @@ -37,7 +37,7 @@ #include "dapl.h" extern u32 dapl_cb_create(DAPL_COOKIE_BUFFER *buffer, DAPL_EP *ep, - DAT_COUNT size); + int size); extern void dapl_cb_free(DAPL_COOKIE_BUFFER *buffer); Index: linux-kernel/dat-provider/dapl_module.c =================================================================== --- linux-kernel/dat-provider/dapl_module.c (revision 2321) +++ linux-kernel/dat-provider/dapl_module.c (working copy) @@ -174,7 +174,7 @@ void DAT_PROVIDER_INIT_FUNC_NAME(const s } printk("KDAPL : Added device %s to the registry\n", provider_info->ia_name); - bail: +bail: if (DAT_SUCCESS != dat_status) { if (NULL != provider) { (void)dapl_provider_list_remove(provider_info->ia_name); @@ -220,7 +220,7 @@ void DAT_PROVIDER_FINI_FUNC_NAME(const s */ dapl_hca_free(provider->extension); - (void)dapl_provider_list_remove(provider_info->ia_name); + dapl_provider_list_remove(provider_info->ia_name); } static struct ib_client dapl_client = { @@ -237,9 +237,8 @@ static char *dev_name_suffix_table[3] = void dapl_add_one(struct ib_device *device) { - DAT_COUNT i; + int i, length; struct dat_provider_info provider_info; - int length; dapl_dbg_log(DAPL_DBG_TYPE_UTIL, "Inside dapl_add_one called by %s\n", device->name); @@ -254,14 +253,12 @@ void dapl_add_one(struct ib_device *devi provider_info.is_thread_safe = FALSE; DAT_PROVIDER_INIT_FUNC_NAME(&provider_info, device, i); } - return; } void dapl_remove_one(struct ib_device *device) { - DAT_COUNT i; + int i, length; struct dat_provider_info provider_info; - int length; dapl_dbg_log(DAPL_DBG_TYPE_UTIL, "Inside dapl_remove_one called by %s\n", device->name); @@ -269,14 +266,12 @@ void dapl_remove_one(struct ib_device *d length = strlen(device->name); for (i = 0; i < device->phys_port_cnt; i++) { strncpy(provider_info.ia_name, device->name, length + 1); - (void)strcat(provider_info.ia_name, - dev_name_suffix_table[i + 1]); + strcat(provider_info.ia_name, dev_name_suffix_table[i + 1]); provider_info.dat_version_major = DAT_VERSION_MAJOR; provider_info.dat_version_minor = DAT_VERSION_MINOR; provider_info.is_thread_safe = FALSE; DAT_PROVIDER_FINI_FUNC_NAME(&provider_info); } - return; } /* @@ -289,11 +284,9 @@ static void dapl_register_providers(void status = ib_register_client(&dapl_client); if (status < 0) printk("ib_register_client () failed: %x\n", status); - return; }; static void dapl_unregister_providers(void) { - (void)ib_unregister_client(&dapl_client); - return; + ib_unregister_client(&dapl_client); } Index: linux-kernel/dat-provider/dapl_ring_buffer_util.c =================================================================== --- linux-kernel/dat-provider/dapl_ring_buffer_util.c (revision 2321) +++ linux-kernel/dat-provider/dapl_ring_buffer_util.c (working copy) @@ -56,7 +56,7 @@ * DAT_INSUFFICIENT_RESOURCES * */ -u32 dapl_rbuf_alloc(DAPL_RING_BUFFER *rbuf, DAT_COUNT size) +u32 dapl_rbuf_alloc(DAPL_RING_BUFFER *rbuf, int size) { unsigned int rsize; /* real size */ @@ -68,7 +68,7 @@ u32 dapl_rbuf_alloc(DAPL_RING_BUFFER *rb /* Put size on a power of 2 boundary */ rsize = 1; - while ((DAT_COUNT) rsize < size) { + while ((int) rsize < size) { rsize <<= 1; } @@ -104,7 +104,7 @@ u32 dapl_rbuf_alloc(DAPL_RING_BUFFER *rb * DAT_INSUFFICIENT_RESOURCES * */ -u32 dapl_rbuf_realloc(DAPL_RING_BUFFER *rbuf, DAT_COUNT size) +u32 dapl_rbuf_realloc(DAPL_RING_BUFFER *rbuf, int size) { DAPL_RING_BUFFER new_rbuf; void *entry; @@ -261,11 +261,9 @@ void *dapl_rbuf_remove(DAPL_RING_BUFFER * count of entries * */ -DAT_COUNT dapl_rbuf_count(DAPL_RING_BUFFER * rbuf) +int dapl_rbuf_count(DAPL_RING_BUFFER * rbuf) { - DAT_COUNT count; - int head; - int tail; + int count, head, tail; head = atomic_read(&rbuf->head) & rbuf->lim; tail = atomic_read(&rbuf->tail) & rbuf->lim; Index: linux-kernel/dat-provider/dapl_srq_resize.c =================================================================== --- linux-kernel/dat-provider/dapl_srq_resize.c (revision 2321) +++ linux-kernel/dat-provider/dapl_srq_resize.c (working copy) @@ -63,7 +63,7 @@ * DAT_INVALID_STATE */ -u32 dapl_srq_resize(DAT_SRQ_HANDLE srq_handle, DAT_COUNT srq_max_recv_dto) +u32 dapl_srq_resize(DAT_SRQ_HANDLE srq_handle, int srq_max_recv_dto) { DAPL_IA *ia_ptr; DAPL_SRQ *srq_ptr; Index: linux-kernel/dat-provider/dapl_ep_post_rdma_write.c =================================================================== --- linux-kernel/dat-provider/dapl_ep_post_rdma_write.c (revision 2321) +++ linux-kernel/dat-provider/dapl_ep_post_rdma_write.c (working copy) @@ -66,7 +66,7 @@ * DAT_PROTECTION_VIOLATION * DAT_PRIVILEGES_VIOLATION */ -u32 dapl_ep_post_rdma_write(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, +u32 dapl_ep_post_rdma_write(DAT_EP_HANDLE ep_handle, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, Index: linux-kernel/dat-provider/dapl_openib_dto.h =================================================================== --- linux-kernel/dat-provider/dapl_openib_dto.h (revision 2321) +++ linux-kernel/dat-provider/dapl_openib_dto.h (working copy) @@ -51,7 +51,7 @@ * Provider specific Post RECV function */ static __inline__ u32 dapl_ib_post_recv(DAPL_EP *ep_ptr, DAPL_COOKIE *cookie, - DAT_COUNT num_segments, + int num_segments, struct dat_lmr_triplet *local_iov) { struct ib_recv_wr rr_desc = { NULL }; @@ -101,7 +101,7 @@ static __inline__ u32 dapl_ib_post_recv( */ static __inline__ u32 dapl_ib_post_send(DAPL_EP *ep_ptr, int op_type, DAPL_COOKIE *cookie, - DAT_COUNT num_segments, + int num_segments, struct dat_lmr_triplet *local_iov, const struct dat_rmr_triplet *remote_iov, DAT_COMPLETION_FLAGS completion_flags) Index: linux-kernel/dat-provider/dapl_ring_buffer_util.h =================================================================== --- linux-kernel/dat-provider/dapl_ring_buffer_util.h (revision 2321) +++ linux-kernel/dat-provider/dapl_ring_buffer_util.h (working copy) @@ -42,9 +42,9 @@ /* * Prototypes */ -u32 dapl_rbuf_alloc(DAPL_RING_BUFFER *rbuf, DAT_COUNT size); +u32 dapl_rbuf_alloc(DAPL_RING_BUFFER *rbuf, int size); -u32 dapl_rbuf_realloc(DAPL_RING_BUFFER *rbuf, DAT_COUNT size); +u32 dapl_rbuf_realloc(DAPL_RING_BUFFER *rbuf, int size); void dapl_rbuf_destroy(DAPL_RING_BUFFER * rbuf); @@ -52,7 +52,7 @@ u32 dapl_rbuf_add(DAPL_RING_BUFFER *rbuf void *dapl_rbuf_remove(DAPL_RING_BUFFER * rbuf); -DAT_COUNT dapl_rbuf_count(DAPL_RING_BUFFER * rbuf); +int dapl_rbuf_count(DAPL_RING_BUFFER *rbuf); void dapl_rbuf_adjust(DAPL_RING_BUFFER * rbuf, intptr_t offset); Index: linux-kernel/dat-provider/dapl_evd_resize.c =================================================================== --- linux-kernel/dat-provider/dapl_evd_resize.c (revision 2321) +++ linux-kernel/dat-provider/dapl_evd_resize.c (working copy) @@ -63,11 +63,11 @@ * DAT_INVALID_STATE */ -u32 dapl_evd_resize(DAT_EVD_HANDLE evd_handle, DAT_COUNT evd_qlen) +u32 dapl_evd_resize(DAT_EVD_HANDLE evd_handle, int evd_qlen) { DAPL_IA *ia_ptr; DAPL_EVD *evd_ptr; - DAT_COUNT pend_cnt; + int pend_cnt; u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_evd_resize (%p, %d)\n", Index: linux-kernel/dat-provider/dapl_adapter_util.h =================================================================== --- linux-kernel/dat-provider/dapl_adapter_util.h (revision 2321) +++ linux-kernel/dat-provider/dapl_adapter_util.h (working copy) @@ -51,7 +51,7 @@ typedef enum async_handler_type { u32 dapl_ib_enum_hcas(const char *vendor, DAPL_HCA_NAME **hca_names, - DAT_COUNT *total_hca_count); + int *total_hca_count); u32 dapl_ib_get_instance_data(DAPL_HCA_NAME hca_name, char *instance); @@ -69,7 +69,7 @@ u32 dapl_ib_qp_modify(DAPL_IA *ia_ptr, D u32 dapl_ib_connect(DAT_EP_HANDLE ep_handle, DAT_IA_ADDRESS_PTR remote_ia_address, DAT_CONN_QUAL remote_conn_qual, - DAT_COUNT private_data_size, void *private_data); + int private_data_size, void *private_data); u32 dapl_ib_disconnect(DAPL_EP *ep_ptr, DAT_CLOSE_FLAGS close_flags); @@ -80,7 +80,7 @@ u32 dapl_ib_remove_conn_listener(DAPL_IA u32 dapl_ib_accept_connection(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, - DAT_COUNT private_data_size, + int private_data_size, const void *private_data); u32 dapl_ib_reject_connection(ib_cm_handle_t cm_handle, int reject_reason); @@ -89,14 +89,14 @@ u32 dapl_ib_setup_async_callback(DAPL_IA DAPL_ASYNC_HANDLER_TYPE handler_type, ib_async_handler_t callback, void *context); -u32 dapl_ib_cq_alloc(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr, DAT_COUNT *cqlen, +u32 dapl_ib_cq_alloc(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr, int *cqlen, ib_comp_handle_t callback); u32 dapl_ib_cq_free(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr); u32 dapl_set_cq_notify(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr); -u32 dapl_ib_cq_resize(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr, DAT_COUNT *cqlen); +u32 dapl_ib_cq_resize(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr, int *cqlen); u32 dapl_ib_pd_alloc(DAPL_IA *ia_ptr, DAPL_PZ *pz); Index: linux-kernel/dat-provider/dapl_ep_post_send.c =================================================================== --- linux-kernel/dat-provider/dapl_ep_post_send.c (revision 2321) +++ linux-kernel/dat-provider/dapl_ep_post_send.c (working copy) @@ -63,7 +63,7 @@ * DAT_PROTECTION_VIOLATION * DAT_PRIVILEGES_VIOLATION */ -u32 dapl_ep_post_send(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, +u32 dapl_ep_post_send(DAT_EP_HANDLE ep_handle, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, DAT_COMPLETION_FLAGS completion_flags) Index: linux-kernel/dat-provider/dapl_provider.c =================================================================== --- linux-kernel/dat-provider/dapl_provider.c (revision 2321) +++ linux-kernel/dat-provider/dapl_provider.c (working copy) @@ -204,7 +204,7 @@ u32 dapl_provider_list_destroy(void) return DAT_SUCCESS; } -DAT_COUNT dapl_provider_list_size(void) +int dapl_provider_list_size(void) { return g_dapl_provider_list.size; } Index: linux-kernel/dat-provider/dapl_openib_cm.c =================================================================== --- linux-kernel/dat-provider/dapl_openib_cm.c (revision 2321) +++ linux-kernel/dat-provider/dapl_openib_cm.c (working copy) @@ -396,7 +396,7 @@ error: u32 dapl_ib_connect(DAT_EP_HANDLE ep_handle, DAT_IA_ADDRESS_PTR remote_ia_address, DAT_CONN_QUAL remote_conn_qual, - DAT_COUNT private_data_size, void *private_data) + int private_data_size, void *private_data) { DAPL_IA *ia_ptr; DAPL_EP *ep_ptr; @@ -732,7 +732,7 @@ u32 dapl_ib_reject_connection(struct dap * */ u32 dapl_ib_accept_connection(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, - DAT_COUNT private_data_size, + int private_data_size, const void *prd_ptr) { DAPL_CR *cr_ptr; Index: linux-kernel/dat-provider/dapl_provider.h =================================================================== --- linux-kernel/dat-provider/dapl_provider.h (revision 2321) +++ linux-kernel/dat-provider/dapl_provider.h (working copy) @@ -56,7 +56,7 @@ typedef struct DAPL_PROVIDER_LIST_NODE { typedef struct DAPL_PROVIDER_LIST { DAPL_PROVIDER_LIST_NODE *head; DAPL_PROVIDER_LIST_NODE *tail; - DAT_COUNT size; + int size; } DAPL_PROVIDER_LIST; /********************************************************************* @@ -78,7 +78,7 @@ extern u32 dapl_provider_list_create(voi extern u32 dapl_provider_list_destroy(void); -extern DAT_COUNT dapl_provider_list_size(void); +extern int dapl_provider_list_size(void); extern u32 dapl_provider_list_insert(const char *name, struct dat_provider **p_data); Index: linux-kernel/dat-provider/dapl_ep_recv_query.c =================================================================== --- linux-kernel/dat-provider/dapl_ep_recv_query.c (revision 2321) +++ linux-kernel/dat-provider/dapl_ep_recv_query.c (working copy) @@ -55,7 +55,7 @@ * DAT_MODEL_NOT_SUPPORTED */ u32 dapl_ep_recv_query(DAT_EP_HANDLE ep_handle, - DAT_COUNT *nbufs_allocate, DAT_COUNT *bufs_alloc_span) + int *nbufs_allocate, int *bufs_alloc_span) { DAPL_EP *ep_ptr; u32 dat_status = DAT_SUCCESS; Index: linux-kernel/dat-provider/dapl_openib_util.c =================================================================== --- linux-kernel/dat-provider/dapl_openib_util.c (revision 2321) +++ linux-kernel/dat-provider/dapl_openib_util.c (working copy) @@ -221,7 +221,7 @@ u32 dapl_ib_close_hca(DAPL_HCA *hca_ptr) * DAT_INSUFFICIENT_RESOURCES * */ -u32 dapl_ib_cq_alloc(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr, DAT_COUNT *cqlen, +u32 dapl_ib_cq_alloc(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr, int *cqlen, ib_comp_handle_t callback) { int ib_status; @@ -263,7 +263,7 @@ u32 dapl_ib_cq_free(DAPL_IA *ia_ptr, DAP return dapl_ib_status_convert(ib_status); } -u32 dapl_ib_cq_resize(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr, DAT_COUNT *cqlen) +u32 dapl_ib_cq_resize(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr, int *cqlen) { int ib_status; Index: linux-kernel/dat-provider/dapl_evd_kcreate.c =================================================================== --- linux-kernel/dat-provider/dapl_evd_kcreate.c (revision 2321) +++ linux-kernel/dat-provider/dapl_evd_kcreate.c (working copy) @@ -48,7 +48,7 @@ * even if it is not required. However, it will not be armed. */ -u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_handle, DAT_COUNT evd_min_qlen, +u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_handle, int evd_min_qlen, DAT_UPCALL_POLICY upcall_policy, /* ignore */ const struct dat_upcall_object *upcall, DAT_EVD_FLAGS evd_flags, DAT_EVD_HANDLE *evd_handle) Index: linux-kernel/dat-provider/dapl_srq_set_lw.c =================================================================== --- linux-kernel/dat-provider/dapl_srq_set_lw.c (revision 2321) +++ linux-kernel/dat-provider/dapl_srq_set_lw.c (working copy) @@ -63,7 +63,7 @@ * DAT_MODEL_NOT_SUPPORTED */ -u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, DAT_COUNT low_watermark) +u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, int low_watermark) { DAPL_SRQ *srq_ptr; u32 dat_status = DAT_SUCCESS; Index: linux-kernel/dat-provider/dapl_cr_accept.c =================================================================== --- linux-kernel/dat-provider/dapl_cr_accept.c (revision 2321) +++ linux-kernel/dat-provider/dapl_cr_accept.c (working copy) @@ -40,7 +40,7 @@ * and passive side (local endpoint). */ u32 dapl_cr_accept(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, - DAT_COUNT private_data_size, const void *private_data) + int private_data_size, const void *private_data) { DAPL_EP *ep_ptr; u32 dat_status; Index: linux-kernel/dat-provider/dapl_srq_post_recv.c =================================================================== --- linux-kernel/dat-provider/dapl_srq_post_recv.c (revision 2321) +++ linux-kernel/dat-provider/dapl_srq_post_recv.c (working copy) @@ -67,7 +67,7 @@ * DAT_PROTECTION_VIOLATION * DAT_PROVILEGES_VIOLATION */ -u32 dapl_srq_post_recv(DAT_SRQ_HANDLE srq_handle, DAT_COUNT num_segments, +u32 dapl_srq_post_recv(DAT_SRQ_HANDLE srq_handle, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie) { Index: linux-kernel/dat-provider/dapl_ep_util.c =================================================================== --- linux-kernel/dat-provider/dapl_ep_util.c (revision 2321) +++ linux-kernel/dat-provider/dapl_ep_util.c (working copy) @@ -281,7 +281,7 @@ u32 dapl_ep_check_request_completion_fla return DAT_SUCCESS; } -u32 dapl_ep_post_send_req(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, +u32 dapl_ep_post_send_req(DAT_EP_HANDLE ep_handle, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, Index: linux-kernel/dat-provider/dapl_ep_util.h =================================================================== --- linux-kernel/dat-provider/dapl_ep_util.h (revision 2321) +++ linux-kernel/dat-provider/dapl_ep_util.h (working copy) @@ -50,8 +50,7 @@ extern u32 dapl_ep_check_recv_completion extern u32 dapl_ep_check_request_completion_flags(DAT_COMPLETION_FLAGS flags); -extern u32 dapl_ep_post_send_req(DAT_EP_HANDLE ep_handle, - DAT_COUNT num_segments, +extern u32 dapl_ep_post_send_req(DAT_EP_HANDLE ep_handle, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, Index: linux-kernel/dat-provider/dapl_ep_post_rdma_read.c =================================================================== --- linux-kernel/dat-provider/dapl_ep_post_rdma_read.c (revision 2321) +++ linux-kernel/dat-provider/dapl_ep_post_rdma_read.c (working copy) @@ -66,7 +66,7 @@ * DAT_PROTECTION_VIOLATION * DAT_PRIVILEGES_VIOLATION */ -u32 dapl_ep_post_rdma_read(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, +u32 dapl_ep_post_rdma_read(DAT_EP_HANDLE ep_handle, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, Index: linux-kernel/dat-provider/dapl_ia_open.c =================================================================== --- linux-kernel/dat-provider/dapl_ia_open.c (revision 2321) +++ linux-kernel/dat-provider/dapl_ia_open.c (working copy) @@ -51,7 +51,7 @@ * That function maps the DAT_NAME parameter of dat_ia_open to a DAT_PROVIDER, * and calls this function. */ -u32 dapl_ia_open(const DAT_NAME_PTR name, DAT_COUNT async_evd_qlen, +u32 dapl_ia_open(const DAT_NAME_PTR name, int async_evd_qlen, DAT_EVD_HANDLE *async_evd_handle_ptr, DAT_IA_HANDLE *ia_handle_ptr) { Index: linux-kernel/dat-provider/dapl_evd_util.c =================================================================== --- linux-kernel/dat-provider/dapl_evd_util.c (revision 2321) +++ linux-kernel/dat-provider/dapl_evd_util.c (working copy) @@ -41,7 +41,7 @@ static __inline__ void dapl_evd_eh_print_cqe(ib_work_completion_t * cqe); -u32 dapl_evd_event_alloc(DAPL_EVD *evd_ptr, DAT_COUNT qlen); +u32 dapl_evd_event_alloc(DAPL_EVD *evd_ptr, int qlen); /* * actually create the evd. this is called after all parameter checking @@ -50,11 +50,11 @@ u32 dapl_evd_event_alloc(DAPL_EVD *evd_p */ u32 dapl_evd_internal_create(DAPL_IA *ia_ptr, DAPL_CNO *cno_ptr, - DAT_COUNT min_qlen, DAT_EVD_FLAGS evd_flags, + int min_qlen, DAT_EVD_FLAGS evd_flags, DAPL_EVD **evd_ptr_ptr) { DAPL_EVD *evd_ptr; - DAT_COUNT cq_len; + int cq_len; u32 dat_status = DAT_SUCCESS; *evd_ptr_ptr = NULL; @@ -113,7 +113,7 @@ u32 dapl_evd_internal_create(DAPL_IA *ia dapl_ia_link_evd(ia_ptr, evd_ptr); *evd_ptr_ptr = evd_ptr; - bail: +bail: if (dat_status != DAT_SUCCESS) { if (evd_ptr) { dapl_evd_dealloc(evd_ptr); @@ -127,7 +127,7 @@ u32 dapl_evd_internal_create(DAPL_IA *ia * alloc and initialize an EVD struct */ DAPL_EVD *dapl_evd_alloc(DAPL_IA * ia_ptr, DAPL_CNO * cno_ptr, - DAT_EVD_FLAGS evd_flags, DAT_COUNT qlen) + DAT_EVD_FLAGS evd_flags, int qlen) { DAPL_EVD *evd_ptr; @@ -164,17 +164,17 @@ DAPL_EVD *dapl_evd_alloc(DAPL_IA * ia_pt evd_ptr->qlen = qlen; evd_ptr->cno_ptr = cno_ptr; - bail: +bail: return evd_ptr; } /* * alloc events into an EVD. */ -u32 dapl_evd_event_alloc(DAPL_EVD *evd_ptr, DAT_COUNT qlen) +u32 dapl_evd_event_alloc(DAPL_EVD *evd_ptr, int qlen) { struct dat_event *event_ptr; - DAT_COUNT i; + int i; u32 dat_status = DAT_SUCCESS; /* Allocate EVENTs */ @@ -206,62 +206,56 @@ u32 dapl_evd_event_alloc(DAPL_EVD *evd_p event_ptr++; } - bail: +bail: return dat_status; } /* * realloc events into an EVD. */ -u32 dapl_evd_event_realloc(DAPL_EVD *evd_ptr, DAT_COUNT qlen) +u32 dapl_evd_event_realloc(DAPL_EVD *evd_ptr, int qlen) { - struct dat_event *events; - DAT_COUNT old_qlen; - DAT_COUNT i; - intptr_t diff; + struct dat_event *events; + int i, old_qlen, diff; u32 dat_status; - /* Allocate EVENTs */ - events = (struct dat_event *)dapl_os_realloc(evd_ptr->events, - qlen * sizeof *events); - if ( NULL == events ) - { - dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); - goto bail; - } + /* Allocate EVENTs */ + events = (struct dat_event *)dapl_os_realloc(evd_ptr->events, + qlen * sizeof *events); + if (!events) { + dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, + DAT_RESOURCE_MEMORY); + goto bail; + } - diff = events - evd_ptr->events; - evd_ptr->events = events; + diff = events - evd_ptr->events; + evd_ptr->events = events; - old_qlen = evd_ptr->qlen; - evd_ptr->qlen = qlen; + old_qlen = evd_ptr->qlen; + evd_ptr->qlen = qlen; - /* reallocate free event queue */ - dat_status = dapl_rbuf_realloc(&evd_ptr->free_event_queue, qlen); - if (dat_status != DAT_SUCCESS) - { - goto bail; - } - dapl_rbuf_adjust(&evd_ptr->free_event_queue, diff); + /* reallocate free event queue */ + dat_status = dapl_rbuf_realloc(&evd_ptr->free_event_queue, qlen); + if (dat_status != DAT_SUCCESS) + goto bail; - /* reallocate pending event queue */ - dat_status = dapl_rbuf_realloc(&evd_ptr->pending_event_queue, qlen); - if (dat_status != DAT_SUCCESS) - { - goto bail; - } - dapl_rbuf_adjust(&evd_ptr->pending_event_queue, diff); + dapl_rbuf_adjust(&evd_ptr->free_event_queue, diff); - /* - * add new events to free event queue. - */ - for (i = old_qlen; i < qlen; i++) - { - dapl_rbuf_add(&evd_ptr->free_event_queue, (void *) &events[i]); - } + /* reallocate pending event queue */ + dat_status = dapl_rbuf_realloc(&evd_ptr->pending_event_queue, qlen); + if (dat_status != DAT_SUCCESS) + goto bail; + + dapl_rbuf_adjust(&evd_ptr->pending_event_queue, diff); + + /* + * add new events to free event queue. + */ + for (i = old_qlen; i < qlen; i++) + dapl_rbuf_add(&evd_ptr->free_event_queue, (void *) &events[i]); bail: - return dat_status; + return dat_status; } /* @@ -297,10 +291,8 @@ u32 dapl_evd_dealloc(DAPL_EVD *evd_ptr) evd_ptr->header.magic = DAPL_MAGIC_INVALID; /* Release reference on the CNO if it exists */ - if ( evd_ptr->cno_ptr != NULL ) - { + if (evd_ptr->cno_ptr != NULL) evd_ptr->cno_ptr = NULL; - } /* If the ring buffer allocation failed, then the dapl_rbuf_destroy */ /* function will detect that the ring buffer's internal data (ex. base */ @@ -314,7 +306,7 @@ u32 dapl_evd_dealloc(DAPL_EVD *evd_ptr) kfree(evd_ptr); - bail: +bail: return dat_status; } @@ -539,7 +531,7 @@ u32 dapl_evd_post_cr_arrival_event(DAPL_ u32 dapl_evd_post_connection_event(DAPL_EVD *evd_ptr, DAT_EVENT_NUMBER event_number, DAT_EP_HANDLE ep_handle, - DAT_COUNT private_data_size, + int private_data_size, void *private_data) { struct dat_event *event_ptr; Index: linux-kernel/dat-provider/dapl_evd_util.h =================================================================== --- linux-kernel/dat-provider/dapl_evd_util.h (revision 2321) +++ linux-kernel/dat-provider/dapl_evd_util.h (working copy) @@ -41,15 +41,15 @@ #include "dapl.h" u32 dapl_evd_internal_create(DAPL_IA *ia_ptr, DAPL_CNO *cno_ptr, - DAT_COUNT min_qlen, DAT_EVD_FLAGS evd_flags, + int min_qlen, DAT_EVD_FLAGS evd_flags, DAPL_EVD **evd_ptr_ptr); DAPL_EVD *dapl_evd_alloc(DAPL_IA * ia_ptr, DAPL_CNO * cno_ptr, - DAT_EVD_FLAGS evd_flags, DAT_COUNT qlen); + DAT_EVD_FLAGS evd_flags, int qlen); u32 dapl_evd_dealloc(DAPL_EVD *evd_ptr); -u32 dapl_evd_event_realloc(DAPL_EVD *evd_ptr, DAT_COUNT qlen); +u32 dapl_evd_event_realloc(DAPL_EVD *evd_ptr, int qlen); /* * Each of these functions will retrieve a free event from @@ -72,7 +72,7 @@ u32 dapl_evd_post_cr_arrival_event(DAPL_ u32 dapl_evd_post_connection_event(DAPL_EVD *evd_ptr, DAT_EVENT_NUMBER event_number, DAT_EP_HANDLE ep_handle, - DAT_COUNT private_data_size, + int private_data_size, void *private_data); u32 dapl_evd_post_async_error_event(DAPL_EVD *evd_ptr, Index: linux-kernel/dat-provider/dapl_util.h =================================================================== --- linux-kernel/dat-provider/dapl_util.h (revision 2321) +++ linux-kernel/dat-provider/dapl_util.h (working copy) @@ -73,13 +73,6 @@ typedef long intptr_t; #define dapl_ip_addr6(sockaddr) (((struct sockaddr_in6 *)sockaddr)->sin6_addr.s6_addr32) -/* - * Atomic operations - */ - -static __inline__ DAT_COUNT -dapl_os_atomic_assign(atomic_t * v, DAT_COUNT match_value, DAT_COUNT new_value); - /* dapl_os_atomic_assign * * assign 'new_value' to '*v' if the current value @@ -90,14 +83,14 @@ dapl_os_atomic_assign(atomic_t * v, DAT_ * Return the current value in any case. * * This matches the IBTA atomic operation compare & swap - * except that it is for local memory and a DAT_COUNT may + * except that it is for local memory and a int may * be only 32 bits, rather than 64. */ -static __inline__ DAT_COUNT -dapl_os_atomic_assign(atomic_t * v, DAT_COUNT match_value, DAT_COUNT new_value) +static inline int dapl_os_atomic_assign(atomic_t * v, int match_value, + int new_value) { - DAT_COUNT current_value; + int current_value; /* * Use the Pentium compare and exchange instruction Index: linux-kernel/dat-provider/dapl_hash.c =================================================================== --- linux-kernel/dat-provider/dapl_hash.c (revision 2321) +++ linux-kernel/dat-provider/dapl_hash.c (working copy) @@ -320,12 +320,12 @@ dapl_hash_delete_element(DAPL_HASH_ELEM /* * Create a new hash table with at least 'table_size' hash buckets. */ -u32 dapl_hash_create(DAT_COUNT table_size, DAPL_HASH_TABLE **pp_table) +u32 dapl_hash_create(int table_size, DAPL_HASH_TABLE **pp_table) { DAPL_HASH_TABLE *p_table; - DAT_COUNT table_length = table_size * sizeof(DAPL_HASH_ELEM); + int table_length = table_size * sizeof(DAPL_HASH_ELEM); u32 dat_status = DAT_SUCCESS; - DAT_COUNT i; + int i; dapl_os_assert(pp_table); @@ -379,7 +379,7 @@ u32 dapl_hash_free(DAPL_HASH_TABLE *p_ta * Returns the number of elements stored in the table */ -u32 dapl_hash_size(DAPL_HASH_TABLE *p_table, DAT_COUNT *p_size) +u32 dapl_hash_size(DAPL_HASH_TABLE *p_table, int *p_size) { dapl_os_assert(p_table && p_size); Index: linux-kernel/dat-provider/dapl_ep_dup_connect.c =================================================================== --- linux-kernel/dat-provider/dapl_ep_dup_connect.c (revision 2321) +++ linux-kernel/dat-provider/dapl_ep_dup_connect.c (working copy) @@ -70,7 +70,7 @@ * DAT_MODEL_NOT_SUPPORTED */ u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, DAT_EP_HANDLE ep_dup_handle, - DAT_TIMEOUT timeout, DAT_COUNT private_data_size, + DAT_TIMEOUT timeout, int private_data_size, const void *private_data, DAT_QOS qos) { DAPL_EP *ep_dup_ptr; Index: linux-kernel/dat-provider/dapl.h =================================================================== --- linux-kernel/dat-provider/dapl.h (revision 2321) +++ linux-kernel/dat-provider/dapl.h (working copy) @@ -193,14 +193,14 @@ struct dapl_llist_entry { struct dapl_ring_buffer { void **base; /* base of element array */ - DAT_COUNT lim; /* mask, number of entries - 1 */ + int lim; /* mask, number of entries - 1 */ atomic_t head; /* head pointer index */ atomic_t tail; /* tail pointer index */ }; struct dapl_cookie_buffer { DAPL_COOKIE *pool; - DAT_COUNT pool_size; + int pool_size; atomic_t head; atomic_t tail; }; @@ -257,33 +257,32 @@ struct dapl_cno { }; /* DAPL_EVD maps to DAT_EVD_HANDLE */ -struct dapl_evd -{ - DAPL_HEADER header; - - DAPL_EVD_STATE evd_state; - DAT_EVD_FLAGS evd_flags; - - /* Derived from evd_flags; see dapls_evd_internal_create. */ - boolean_t evd_producer_locking_needed; - - /* Every EVD has a CQ unless it is a SOFTWARE_EVENT only EVD */ - ib_cq_handle_t ib_cq_handle; - - /* An Event Dispatcher cannot be freed while - * it is referenced elsewhere. - */ - atomic_t evd_ref_count; - - /* Set if there has been a catastrophic overflow */ - boolean_t catastrophic_overflow; - - /* the actual events */ - DAT_COUNT qlen; - struct dat_event *events; - DAPL_RING_BUFFER free_event_queue; - DAPL_RING_BUFFER pending_event_queue; - DAPL_CNO *cno_ptr; +struct dapl_evd { + DAPL_HEADER header; + DAPL_EVD_STATE evd_state; + DAT_EVD_FLAGS evd_flags; + + /* Derived from evd_flags; see dapls_evd_internal_create. */ + boolean_t evd_producer_locking_needed; + + /* Every EVD has a CQ unless it is a SOFTWARE_EVENT only EVD */ + ib_cq_handle_t ib_cq_handle; + + /* + * An Event Dispatcher cannot be freed while + * it is referenced elsewhere. + */ + atomic_t evd_ref_count; + + /* Set if there has been a catastrophic overflow */ + boolean_t catastrophic_overflow; + + /* the actual events */ + int qlen; + struct dat_event *events; + DAPL_RING_BUFFER free_event_queue; + DAPL_RING_BUFFER pending_event_queue; + DAPL_CNO *cno_ptr; }; /* uDAPL timer entry, used to queue timeouts */ @@ -329,10 +328,10 @@ struct dapl_ep { DAPL_COOKIE_BUFFER recv_buffer; ib_data_segment_t *recv_iov; - DAT_COUNT recv_iov_num; + int recv_iov_num; ib_data_segment_t *send_iov; - DAT_COUNT send_iov_num; + int send_iov_num; }; /* DAPL_SRQ maps to DAT_SRQ_HANDLE */ @@ -394,7 +393,7 @@ struct dapl_sp { boolean_t listening; /* PSP is registered & active */ ib_cm_srvc_handle_t cm_srvc_handle; /* Used by CM */ DAPL_LLIST_HEAD cr_list_head; /* CR pending queue */ - DAT_COUNT cr_list_count; /* count of CRs on queue */ + int cr_list_count; /* count of CRs on queue */ }; /* DAPL_CR maps to DAT_CR_HANDLE */ @@ -439,7 +438,7 @@ typedef enum dapl_cookie_type { struct dapl_dto_cookie { DAPL_DTO_TYPE type; DAT_DTO_COOKIE cookie; - DAT_COUNT size; /* used for SEND and RDMA write */ + int size; /* used for SEND and RDMA write */ }; /* DAPL_RMR_COOKIE used as context for bind WQEs */ @@ -452,7 +451,7 @@ struct dapl_rmr_cookie { struct dapl_cookie { DAPL_COOKIE_TYPE type; /* Must be first, to define struct. */ DAPL_EP *ep; - DAT_COUNT index; + int index; union { DAPL_DTO_COOKIE dto; DAPL_RMR_COOKIE rmr; @@ -475,8 +474,9 @@ typedef char DAPL_HCA_NAME[DAPL_HCA_NAME * DAT API functions */ -extern u32 dapl_ia_open(const DAT_NAME_PTR, DAT_COUNT, DAT_EVD_HANDLE *, - DAT_IA_HANDLE *); +extern u32 dapl_ia_open(const DAT_NAME_PTR name, int async_evd_qlen, + DAT_EVD_HANDLE *async_evd_handle_ptr, + DAT_IA_HANDLE *ia_handle_ptr); extern u32 dapl_ia_close(DAT_IA_HANDLE, DAT_CLOSE_FLAGS); @@ -496,7 +496,7 @@ extern u32 dapl_get_handle_type(DAT_HAND extern u32 dapl_cr_query(DAT_CR_HANDLE, struct dat_cr_param *); extern u32 dapl_cr_accept(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, - DAT_COUNT private_data_size, + int private_data_size, const void *private_data); extern u32 dapl_cr_reject(DAT_CR_HANDLE cr_handle); @@ -509,7 +509,7 @@ extern u32 dapl_ia_memtype_hint(DAT_IA_H DAT_VLEN *suggested_length, DAT_VADDR *suggested_alignment); -extern u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_handle, DAT_COUNT evd_min_qlen, +extern u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_handle, int evd_min_qlen, DAT_UPCALL_POLICY upcall_policy, const struct dat_upcall_object * upcall, DAT_EVD_FLAGS evd_flags, @@ -526,7 +526,7 @@ extern u32 dapl_evd_enable(DAT_EVD_HANDL extern u32 dapl_evd_disable(DAT_EVD_HANDLE evd_handle); -extern u32 dapl_evd_resize(DAT_EVD_HANDLE evd_handle, DAT_COUNT evd_qlen); +extern u32 dapl_evd_resize(DAT_EVD_HANDLE evd_handle, int evd_qlen); extern u32 dapl_evd_post_se(DAT_EVD_HANDLE evd_handle, const struct dat_event *event); @@ -558,39 +558,39 @@ extern u32 dapl_ep_connect(DAT_EP_HANDLE DAT_IA_ADDRESS_PTR remote_ia_address, DAT_CONN_QUAL remote_conn_qual, DAT_TIMEOUT timeout, - DAT_COUNT private_data_size, + int private_data_size, const void *private_data, DAT_QOS quality_of_service, DAT_CONNECT_FLAGS connect_flags); extern u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, DAT_EP_HANDLE ep_dup_handle, DAT_TIMEOUT timeout, - DAT_COUNT private_data_size, + int private_data_size, const void *private_data, DAT_QOS quality_of_service); extern u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_handle, DAT_CLOSE_FLAGS close_flags); -extern u32 dapl_ep_post_send(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, +extern u32 dapl_ep_post_send(DAT_EP_HANDLE ep_handle, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, DAT_COMPLETION_FLAGS completion_flags); -extern u32 dapl_ep_post_recv(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, +extern u32 dapl_ep_post_recv(DAT_EP_HANDLE ep_handle, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, DAT_COMPLETION_FLAGS completion_flags); extern u32 dapl_ep_post_rdma_read(DAT_EP_HANDLE ep_handle, - DAT_COUNT num_segments, + int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, DAT_COMPLETION_FLAGS completion_flags); extern u32 dapl_ep_post_rdma_write(DAT_EP_HANDLE ep_handle, - DAT_COUNT num_segments, + int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, @@ -613,12 +613,12 @@ extern u32 dapl_ep_create_with_srq(DAT_I DAT_EP_HANDLE *ep_handle); extern u32 dapl_ep_recv_query(DAT_EP_HANDLE ep_handle, - DAT_COUNT *nbufs_allocated, - DAT_COUNT *bufs_alloc_span); + int *nbufs_allocated, + int *bufs_alloc_span); extern u32 dapl_ep_set_watermark(DAT_EP_HANDLE ep_handle, - DAT_COUNT soft_high_watermark, - DAT_COUNT hard_high_watermark); + int soft_high_watermark, + int hard_high_watermark); /* LMR functions */ @@ -703,7 +703,7 @@ extern u32 dapl_srq_create(DAT_IA_HANDLE extern u32 dapl_srq_free(DAT_SRQ_HANDLE srq_handle); -extern u32 dapl_srq_post_recv(DAT_SRQ_HANDLE srq_handle, DAT_COUNT num_segments, +extern u32 dapl_srq_post_recv(DAT_SRQ_HANDLE srq_handle, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie); @@ -711,9 +711,9 @@ extern u32 dapl_srq_query(DAT_SRQ_HANDLE struct dat_srq_param *srq_param); extern u32 dapl_srq_resize(DAT_SRQ_HANDLE srq_handle, - DAT_COUNT srq_max_recv_dto); + int srq_max_recv_dto); -extern u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, DAT_COUNT low_watermark); +extern u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, int low_watermark); /* * DAPL internal utility function prototpyes Index: linux-kernel/dat-provider/dapl_hash.h =================================================================== --- linux-kernel/dat-provider/dapl_hash.h (revision 2321) +++ linux-kernel/dat-provider/dapl_hash.h (working copy) @@ -67,11 +67,11 @@ * * *********************************************************************/ -extern u32 dapl_hash_create(DAT_COUNT capacity, DAPL_HASH_TABLE **pp_table); +extern u32 dapl_hash_create(int capacity, DAPL_HASH_TABLE **pp_table); extern u32 dapl_hash_free(DAPL_HASH_TABLE *p_table); -extern u32 dapl_hash_size(DAPL_HASH_TABLE *p_table, DAT_COUNT *p_size); +extern u32 dapl_hash_size(DAPL_HASH_TABLE *p_table, int *p_size); extern u32 dapl_hash_insert(DAPL_HASH_TABLE *p_table, DAPL_HASH_KEY key, DAPL_HASH_DATA data); Index: linux-kernel/dat-provider/dapl_ep_post_recv.c =================================================================== --- linux-kernel/dat-provider/dapl_ep_post_recv.c (revision 2321) +++ linux-kernel/dat-provider/dapl_ep_post_recv.c (working copy) @@ -66,7 +66,7 @@ * DAT_PROTECTION_VIOLATION * DAT_PROVILEGES_VIOLATION */ -u32 dapl_ep_post_recv(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, +u32 dapl_ep_post_recv(DAT_EP_HANDLE ep_handle, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, DAT_COMPLETION_FLAGS completion_flags) Index: linux-kernel/dat-provider/dapl_ep_set_watermark.c =================================================================== --- linux-kernel/dat-provider/dapl_ep_set_watermark.c (revision 2321) +++ linux-kernel/dat-provider/dapl_ep_set_watermark.c (working copy) @@ -60,8 +60,8 @@ * DAT_MODEL_NOT_SUPPORTED */ u32 dapl_ep_set_watermark(DAT_EP_HANDLE ep_handle, - DAT_COUNT soft_high_watermark, - DAT_COUNT hard_high_watermark) + int soft_high_watermark, + int hard_high_watermark) { DAPL_EP *ep_ptr; u32 dat_status = DAT_SUCCESS; Index: linux-kernel/dat/dictionary.c =================================================================== --- linux-kernel/dat/dictionary.c (revision 2321) +++ linux-kernel/dat/dictionary.c (working copy) @@ -75,8 +75,7 @@ void dat_dictionary_destroy(struct dat_d * Function: dat_dictionary_size ***********************************************************************/ -u32 dat_dictionary_size(const struct dat_dictionary *dictionary, - DAT_COUNT *size) +u32 dat_dictionary_size(const struct dat_dictionary *dictionary, int *size) { BUG_ON(NULL == dictionary); BUG_ON(NULL == size); @@ -190,11 +189,11 @@ u32 dat_dictionary_search(struct dat_dic ***********************************************************************/ u32 dat_dictionary_enumerate(struct dat_dictionary *dictionary, void *array[], - DAT_COUNT array_size) + int array_size) { struct list_head *cur_list; struct dat_dictionary_entry *cur_entry; - DAT_COUNT i; + int i; u32 status; BUG_ON(NULL == dictionary); Index: linux-kernel/dat/dr.c =================================================================== --- linux-kernel/dat/dr.c (revision 2321) +++ linux-kernel/dat/dr.c (working copy) @@ -42,7 +42,7 @@ *********************************************************************/ struct dat_dr_entry { - DAT_COUNT ref_count; + int ref_count; struct dat_provider_info info; DAT_IA_OPEN_FUNC ia_open_func; }; @@ -206,7 +206,7 @@ u32 dat_dr_provider_close(const struct d * Function: dat_dr_size ************************************************************************/ -u32 dat_dr_size(DAT_COUNT *size) +u32 dat_dr_size(int *size) { return dat_dictionary_size(&g_dr_dictionary, size); } @@ -215,17 +215,13 @@ u32 dat_dr_size(DAT_COUNT *size) * Function: dat_dr_list ***********************************************************************/ -u32 dat_dr_list(DAT_COUNT max_to_return, - DAT_COUNT *entries_returned, +u32 dat_dr_list(int max_to_return, int *entries_returned, struct dat_provider_info *(dat_provider_list[])) { - struct dat_dr_entry **array; - DAT_COUNT array_size; - DAT_COUNT i; + struct dat_dr_entry **array = NULL; + int i, array_size; u32 status = DAT_SUCCESS; - array = NULL; - /* The dictionary size may increase between the call to */ /* dat_dictionary_size() and dat_dictionary_enumerate(). */ /* Therefore we loop until a successful enumeration is made. */ Index: linux-kernel/dat/api.c =================================================================== --- linux-kernel/dat/api.c (revision 2321) +++ linux-kernel/dat/api.c (working copy) @@ -575,7 +575,7 @@ static u32 dat_strerror_minor(u32 value, * Function: dat_ia_openv ***********************************************************************/ -u32 dat_ia_openv(const DAT_NAME_PTR name, DAT_COUNT async_event_qlen, +u32 dat_ia_openv(const DAT_NAME_PTR name, int async_event_qlen, DAT_EVD_HANDLE *async_event_handle, DAT_IA_HANDLE *ia_handle, u32 dat_major, u32 dat_minor, boolean_t thread_safety) { @@ -756,8 +756,7 @@ EXPORT_SYMBOL(dat_registry_add_provider) * Function: dat_registry_list_providers ***********************************************************************/ -u32 dat_registry_list_providers(DAT_COUNT max_to_return, - DAT_COUNT *entries_returned, +u32 dat_registry_list_providers(int max_to_return, int *entries_returned, struct dat_provider_info *(dat_provider_list[])) { u32 status = DAT_SUCCESS; Index: linux-kernel/dat/dr.h =================================================================== --- linux-kernel/dat/dr.h (revision 2321) +++ linux-kernel/dat/dr.h (working copy) @@ -54,10 +54,9 @@ extern u32 dat_dr_provider_open(const st extern u32 dat_dr_provider_close(const struct dat_provider_info *info); -extern u32 dat_dr_size(DAT_COUNT *size); +extern u32 dat_dr_size(int *size); -extern u32 dat_dr_list(DAT_COUNT max_to_return, - DAT_COUNT *entries_returned, +extern u32 dat_dr_list(int max_to_return, int *entries_returned, struct dat_provider_info *(dat_provider_list[])); #endif /* DR_H */ Index: linux-kernel/dat/dictionary.h =================================================================== --- linux-kernel/dat/dictionary.h (revision 2321) +++ linux-kernel/dat/dictionary.h (working copy) @@ -50,7 +50,7 @@ struct dat_dictionary_entry { struct dat_dictionary { struct list_head list; - DAT_COUNT size; + int size; }; /********************************************************************* @@ -70,7 +70,7 @@ struct dat_dictionary { extern void dat_dictionary_destroy(struct dat_dictionary *dictionary); extern u32 dat_dictionary_size(const struct dat_dictionary *dictionary, - DAT_COUNT *size); + int *size); extern u32 dat_dictionary_entry_create(struct dat_dictionary_entry **p_entry); @@ -87,8 +87,7 @@ extern u32 dat_dictionary_search(struct void **p_data); extern u32 dat_dictionary_enumerate(struct dat_dictionary *dictionary, - void *array[], - const DAT_COUNT array_size); + void *array[], const int array_size); extern u32 dat_dictionary_remove(struct dat_dictionary *dictionary, const struct dat_provider_info *key, Index: linux-kernel/dat/dat.h =================================================================== --- linux-kernel/dat/dat.h (revision 2321) +++ linux-kernel/dat/dat.h (working copy) @@ -40,7 +40,6 @@ /* unsigned longest native to compiler */ typedef unsigned long long DAT_UVERYLONG; -typedef int DAT_COUNT; typedef u64 DAT_PADDR; /* @@ -575,18 +574,18 @@ struct dat_ep_attr { DAT_QOS qos; DAT_COMPLETION_FLAGS recv_completion_flags; DAT_COMPLETION_FLAGS request_completion_flags; - DAT_COUNT max_recv_dtos; - DAT_COUNT max_request_dtos; - DAT_COUNT max_recv_iov; - DAT_COUNT max_request_iov; - DAT_COUNT max_rdma_read_in; - DAT_COUNT max_rdma_read_out; - DAT_COUNT srq_soft_hw; - DAT_COUNT max_rdma_read_iov; - DAT_COUNT max_rdma_write_iov; - DAT_COUNT ep_transport_specific_count; + int max_recv_dtos; + int max_request_dtos; + int max_recv_iov; + int max_request_iov; + int max_rdma_read_in; + int max_rdma_read_out; + int srq_soft_hw; + int max_rdma_read_iov; + int max_rdma_write_iov; + int ep_transport_specific_count; struct dat_named_attr *ep_transport_specific; - DAT_COUNT ep_provider_specific_count; + int ep_provider_specific_count; struct dat_named_attr *ep_provider_specific; }; @@ -656,7 +655,7 @@ typedef enum dat_ep_param_mask { DAT_EP_FIELD_ALL = 0x3FFFF7FF } DAT_EP_PARAM_MASK; -#define DAT_WATERMARK_INFINITE ((DAT_COUNT)~0) +#define DAT_WATERMARK_INFINITE ((int)~0) #define DAT_HW_DEFAULT DAT_WATERMARK_INFINITE #define DAT_SRQ_LW_DEFAULT 0x0 @@ -665,23 +664,23 @@ typedef enum dat_srq_state { DAT_SRQ_STATE_ERROR } DAT_SRQ_STATE; -#define DAT_VALUE_UNKNOWN (((DAT_COUNT) ~0)-1) +#define DAT_VALUE_UNKNOWN (((int) ~0)-1) struct dat_srq_attr { - DAT_COUNT max_recv_dtos; - DAT_COUNT max_recv_iov; - DAT_COUNT low_watermark; + int max_recv_dtos; + int max_recv_iov; + int low_watermark; }; struct dat_srq_param { DAT_IA_HANDLE ia_handle; DAT_SRQ_STATE srq_state; DAT_PZ_HANDLE pz_handle; - DAT_COUNT max_recv_dtos; - DAT_COUNT max_recv_iov; - DAT_COUNT low_watermark; - DAT_COUNT available_dto_count; - DAT_COUNT outstanding_dto_count; + int max_recv_dtos; + int max_recv_iov; + int low_watermark; + int available_dto_count; + int outstanding_dto_count; }; /* PZ Parameters */ @@ -723,7 +722,7 @@ struct dat_cr_param { DAT_PORT_QUAL remote_port_qual; /* Size of the Private Data. */ - DAT_COUNT private_data_size; + int private_data_size; /* Pointer to the Private Data passed by remote side in the Connection * Request. @@ -820,7 +819,7 @@ struct dat_cr_arrival_event_data { /* Connection event data */ struct dat_connection_event_data { DAT_EP_HANDLE ep_handle; - DAT_COUNT private_data_size; + int private_data_size; void *private_data; }; @@ -836,7 +835,7 @@ struct dat_connection_event_data { struct dat_asynch_error_event_data { DAT_HANDLE dat_handle; /* either IA, EP, EVD, SRQ, */ /* LMR, RMR, or PZ handle */ - DAT_COUNT reason; /* object specific */ + int reason; /* object specific */ }; /* The reason is object type specific and its values are defined below. */ @@ -954,7 +953,7 @@ struct dat_provider_info { struct dat_evd_param { DAT_IA_HANDLE ia_handle; - DAT_COUNT evd_qlen; + int evd_qlen; DAT_UPCALL_POLICY upcall_policy; struct dat_upcall_object upcall; DAT_EVD_FLAGS evd_flags; @@ -1010,33 +1009,33 @@ struct dat_ia_attr { u32 firmware_version_major; u32 firmware_version_minor; DAT_IA_ADDRESS_PTR ia_address_ptr; - DAT_COUNT max_eps; - DAT_COUNT max_dto_per_ep; - DAT_COUNT max_rdma_read_per_ep_in; - DAT_COUNT max_rdma_read_per_ep_out; - DAT_COUNT max_evds; - DAT_COUNT max_evd_qlen; - DAT_COUNT max_iov_segments_per_dto; - DAT_COUNT max_lmrs; + int max_eps; + int max_dto_per_ep; + int max_rdma_read_per_ep_in; + int max_rdma_read_per_ep_out; + int max_evds; + int max_evd_qlen; + int max_iov_segments_per_dto; + int max_lmrs; DAT_VLEN max_lmr_block_size; DAT_VADDR max_lmr_virtual_address; - DAT_COUNT max_pzs; + int max_pzs; DAT_VLEN max_message_size; DAT_VLEN max_rdma_size; - DAT_COUNT max_rmrs; + int max_rmrs; DAT_VADDR max_rmr_target_address; - DAT_COUNT max_srqs; - DAT_COUNT max_ep_per_srq; - DAT_COUNT max_recv_per_srq; - DAT_COUNT max_iov_segments_per_rdma_read; - DAT_COUNT max_iov_segments_per_rdma_write; - DAT_COUNT max_rdma_read_in; - DAT_COUNT max_rdma_read_out; + int max_srqs; + int max_ep_per_srq; + int max_recv_per_srq; + int max_iov_segments_per_rdma_read; + int max_iov_segments_per_rdma_write; + int max_rdma_read_in; + int max_rdma_read_out; boolean_t max_rdma_read_per_ep_in_guaranteed; boolean_t max_rdma_read_per_ep_out_guaranteed; - DAT_COUNT num_transport_attr; + int num_transport_attr; struct dat_named_attr *transport_attr; - DAT_COUNT num_vendor_attr; + int num_vendor_attr; struct dat_named_attr *vendor_attr; }; @@ -1079,21 +1078,21 @@ struct dat_provider_attr { DAT_QOS dat_qos_supported; DAT_COMPLETION_FLAGS completion_flags_supported; boolean_t is_thread_safe; - DAT_COUNT max_private_data_size; + int max_private_data_size; boolean_t supports_multipath; DAT_EP_CREATOR_FOR_PSP ep_creator; DAT_UPCALL_POLICY upcall_policy; u32 optimal_buffer_alignment; const boolean_t evd_stream_merging_supported[6][6]; boolean_t srq_supported; - DAT_COUNT srq_watermarks_supported; + int srq_watermarks_supported; boolean_t srq_ep_pz_difference_supported; - DAT_COUNT srq_info_supported; - DAT_COUNT ep_recv_info_supported; + int srq_info_supported; + int ep_recv_info_supported; boolean_t lmr_sync_req; boolean_t dto_async_return_guaranteed; boolean_t rdma_write_for_rdma_read_req; - DAT_COUNT num_provider_specific_attr; + int num_provider_specific_attr; struct dat_named_attr *provider_specific_attr; }; @@ -1105,10 +1104,10 @@ struct dat_provider_attr { ****************************************************************/ typedef u32 (*DAT_IA_OPEN_FUNC) ( - const DAT_NAME_PTR, DAT_COUNT, DAT_EVD_HANDLE *, DAT_IA_HANDLE *); + const DAT_NAME_PTR, int, DAT_EVD_HANDLE *, DAT_IA_HANDLE *); typedef u32 (*DAT_IA_OPENV_FUNC) ( - const DAT_NAME_PTR, DAT_COUNT, DAT_EVD_HANDLE *, DAT_IA_HANDLE *, + const DAT_NAME_PTR, int, DAT_EVD_HANDLE *, DAT_IA_HANDLE *, u32, u32, boolean_t); typedef u32 (*DAT_IA_CLOSE_FUNC) (DAT_IA_HANDLE, DAT_CLOSE_FLAGS); @@ -1125,14 +1124,14 @@ typedef u32 (*DAT_GET_HANDLE_TYPE_FUNC) typedef u32 (*DAT_CR_QUERY_FUNC) (DAT_CR_HANDLE, struct dat_cr_param *); -typedef u32 (*DAT_CR_ACCEPT_FUNC) (DAT_CR_HANDLE, DAT_EP_HANDLE, DAT_COUNT, +typedef u32 (*DAT_CR_ACCEPT_FUNC) (DAT_CR_HANDLE, DAT_EP_HANDLE, int, const void*); typedef u32 (*DAT_CR_REJECT_FUNC) (DAT_CR_HANDLE); typedef u32 (*DAT_CR_HANDOFF_FUNC) (DAT_CR_HANDLE, DAT_CONN_QUAL); -typedef u32 (*DAT_EVD_RESIZE_FUNC) (DAT_EVD_HANDLE, DAT_COUNT); +typedef u32 (*DAT_EVD_RESIZE_FUNC) (DAT_EVD_HANDLE, int); typedef u32 (*DAT_EVD_POST_SE_FUNC) (DAT_EVD_HANDLE, const struct dat_event *); @@ -1158,29 +1157,29 @@ typedef u32 (*DAT_EP_MODIFY_FUNC) ( DAT_EP_HANDLE, DAT_EP_PARAM_MASK, const struct dat_ep_param *); typedef u32 (*DAT_EP_CONNECT_FUNC) (DAT_EP_HANDLE, DAT_IA_ADDRESS_PTR, - DAT_CONN_QUAL, DAT_TIMEOUT, DAT_COUNT, + DAT_CONN_QUAL, DAT_TIMEOUT, int, const void *, DAT_QOS, DAT_CONNECT_FLAGS); typedef u32 (*DAT_EP_DUP_CONNECT_FUNC) (DAT_EP_HANDLE, DAT_EP_HANDLE, - DAT_TIMEOUT, DAT_COUNT, const void *, + DAT_TIMEOUT, int, const void *, DAT_QOS); typedef u32 (*DAT_EP_DISCONNECT_FUNC) (DAT_EP_HANDLE, DAT_CLOSE_FLAGS); typedef u32 (*DAT_EP_POST_SEND_FUNC) ( - DAT_EP_HANDLE, DAT_COUNT, struct dat_lmr_triplet *, DAT_DTO_COOKIE, + DAT_EP_HANDLE, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, DAT_COMPLETION_FLAGS); typedef u32 (*DAT_EP_POST_RECV_FUNC) ( - DAT_EP_HANDLE, DAT_COUNT, struct dat_lmr_triplet *, DAT_DTO_COOKIE, + DAT_EP_HANDLE, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, DAT_COMPLETION_FLAGS); typedef u32 (*DAT_EP_POST_RDMA_READ_FUNC) ( - DAT_EP_HANDLE, DAT_COUNT, struct dat_lmr_triplet *, DAT_DTO_COOKIE, + DAT_EP_HANDLE, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, const struct dat_rmr_triplet *, DAT_COMPLETION_FLAGS); typedef u32 (*DAT_EP_POST_RDMA_WRITE_FUNC) ( - DAT_EP_HANDLE, DAT_COUNT, struct dat_lmr_triplet *, DAT_DTO_COOKIE, + DAT_EP_HANDLE, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, const struct dat_rmr_triplet *, DAT_COMPLETION_FLAGS); typedef u32 (*DAT_EP_GET_STATUS_FUNC) ( @@ -1191,10 +1190,10 @@ typedef u32 (*DAT_EP_FREE_FUNC) (DAT_EP_ typedef u32 (*DAT_EP_RESET_FUNC) (DAT_EP_HANDLE); typedef u32 (*DAT_EP_RECV_QUERY_FUNC) ( - DAT_EP_HANDLE, DAT_COUNT *, DAT_COUNT *); + DAT_EP_HANDLE, int *, int *); typedef u32 (*DAT_EP_SET_WATERMARK_FUNC) ( - DAT_EP_HANDLE, DAT_COUNT, DAT_COUNT); + DAT_EP_HANDLE, int, int); typedef u32 (*DAT_LMR_FREE_FUNC) (DAT_LMR_HANDLE); @@ -1248,17 +1247,17 @@ typedef u32 (*DAT_PZ_FREE_FUNC) (DAT_PZ_ typedef u32 (*DAT_SRQ_CREATE_FUNC) ( DAT_IA_HANDLE, DAT_PZ_HANDLE, struct dat_srq_attr *, DAT_SRQ_HANDLE *); -typedef u32 (*DAT_SRQ_SET_LW_FUNC) (DAT_SRQ_HANDLE, DAT_COUNT); +typedef u32 (*DAT_SRQ_SET_LW_FUNC) (DAT_SRQ_HANDLE, int); typedef u32 (*DAT_SRQ_FREE_FUNC) (DAT_SRQ_HANDLE); typedef u32 (*DAT_SRQ_QUERY_FUNC) (DAT_SRQ_HANDLE, struct dat_srq_param *); -typedef u32 (*DAT_SRQ_RESIZE_FUNC) (DAT_SRQ_HANDLE, DAT_COUNT); +typedef u32 (*DAT_SRQ_RESIZE_FUNC) (DAT_SRQ_HANDLE, int); typedef u32 (*DAT_SRQ_POST_RECV_FUNC) ( - DAT_SRQ_HANDLE, DAT_COUNT, struct dat_lmr_triplet *, DAT_DTO_COOKIE); + DAT_SRQ_HANDLE, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE); typedef u32 (*DAT_LMR_KCREATE_FUNC) ( DAT_IA_HANDLE, DAT_MEM_TYPE, DAT_REGION_DESCRIPTION, DAT_VLEN, @@ -1271,7 +1270,7 @@ typedef u32 (*DAT_IA_MEMTYPE_HINT_FUNC) DAT_VLEN *, DAT_VADDR *); typedef u32 (*DAT_EVD_KCREATE_FUNC) ( - DAT_IA_HANDLE, DAT_COUNT, DAT_UPCALL_POLICY, + DAT_IA_HANDLE, int, DAT_UPCALL_POLICY, const struct dat_upcall_object *, DAT_EVD_FLAGS, DAT_EVD_HANDLE *); @@ -1378,9 +1377,10 @@ extern u32 dat_registry_remove_provider( /* * DAT registry functions for consumers */ -extern u32 dat_ia_openv(const DAT_NAME_PTR, DAT_COUNT, - DAT_EVD_HANDLE *, DAT_IA_HANDLE *, - u32, u32, boolean_t); +extern u32 dat_ia_openv(const DAT_NAME_PTR name, int async_event_qlen, + DAT_EVD_HANDLE *async_event_handle, + DAT_IA_HANDLE *ia_handle, u32 dat_major, u32 dat_minor, + boolean_t thread_safety); #define dat_ia_open(name, qlen, async_evd, ia) \ dat_ia_openv((name), (qlen), (async_evd), (ia), \ @@ -1391,8 +1391,8 @@ extern u32 dat_ia_close(DAT_IA_HANDLE, D extern u32 dat_strerror(u32, const char **, const char **); -extern u32 dat_registry_list_providers( - DAT_COUNT, DAT_COUNT *, struct dat_provider_info *dat_provider_list[]); +extern u32 dat_registry_list_providers(int max_to_return, int *entries_returned, + struct dat_provider_info *dat_provider_list[]); /* * inline functions for consumers @@ -1402,64 +1402,57 @@ extern u32 dat_registry_list_providers( #define DAT_CALL_PROVIDER_FUNC(func, handle, ...) \ DAT_HANDLE_TO_PROVIDER(handle)->func(handle, ##__VA_ARGS__) -static inline u32 dat_set_consumer_context(DAT_HANDLE handle, - DAT_CONTEXT context) +static inline u32 dat_set_consumer_context(DAT_HANDLE handle, + DAT_CONTEXT context) { - return DAT_CALL_PROVIDER_FUNC( - set_consumer_context_func, handle, context); + return DAT_CALL_PROVIDER_FUNC(set_consumer_context_func, handle, + context); } -static inline u32 dat_get_consumer_context(DAT_HANDLE handle, - DAT_CONTEXT *context) +static inline u32 dat_get_consumer_context(DAT_HANDLE handle, + DAT_CONTEXT *context) { - return DAT_CALL_PROVIDER_FUNC( - get_consumer_context_func, handle, context); + return DAT_CALL_PROVIDER_FUNC(get_consumer_context_func, handle, + context); } -static inline u32 dat_get_handle_type(DAT_HANDLE handle, - DAT_HANDLE_TYPE *type) +static inline u32 dat_get_handle_type(DAT_HANDLE handle, DAT_HANDLE_TYPE *type) { return DAT_CALL_PROVIDER_FUNC(get_handle_type_func, handle, type); } -static inline u32 dat_ia_memtype_hint(DAT_IA_HANDLE ia, - DAT_MEM_TYPE mem_type, - DAT_VLEN length, - DAT_MEM_OPTIMIZE_FLAGS mem_optimize, - DAT_VLEN * preferred_length, - DAT_VADDR * preferred_alignment) -{ - return DAT_CALL_PROVIDER_FUNC( - ia_memtype_hint_func, ia, mem_type, length, mem_optimize, - preferred_length, preferred_alignment); +static inline u32 dat_ia_memtype_hint(DAT_IA_HANDLE ia, DAT_MEM_TYPE mem_type, + DAT_VLEN length, + DAT_MEM_OPTIMIZE_FLAGS mem_optimize, + DAT_VLEN *preferred_length, + DAT_VADDR *preferred_alignment) +{ + return DAT_CALL_PROVIDER_FUNC(ia_memtype_hint_func, ia, mem_type, + length, mem_optimize, preferred_length, + preferred_alignment); } -static inline u32 dat_ia_query(DAT_IA_HANDLE ia, - DAT_EVD_HANDLE * async_evd, - struct dat_ia_attr *ia_attr, - struct dat_provider_attr *provider_attr) +static inline u32 dat_ia_query(DAT_IA_HANDLE ia, DAT_EVD_HANDLE *async_evd, + struct dat_ia_attr *ia_attr, + struct dat_provider_attr *provider_attr) { return DAT_CALL_PROVIDER_FUNC( ia_query_func, ia, async_evd, ia_attr, provider_attr); } -static inline u32 dat_cr_accept(DAT_CR_HANDLE cr, - DAT_EP_HANDLE ep, - DAT_COUNT private_data_size, - const void *private_data) +static inline u32 dat_cr_accept(DAT_CR_HANDLE cr, DAT_EP_HANDLE ep, + int private_data_size, const void *private_data) { return DAT_CALL_PROVIDER_FUNC( cr_accept_func, cr, ep, private_data_size, private_data); } -static inline u32 dat_cr_handoff(DAT_CR_HANDLE cr, - DAT_CONN_QUAL handoff) +static inline u32 dat_cr_handoff(DAT_CR_HANDLE cr, DAT_CONN_QUAL handoff) { return DAT_CALL_PROVIDER_FUNC(cr_handoff_func, cr, handoff); } -static inline u32 dat_cr_query(DAT_CR_HANDLE cr, - struct dat_cr_param *param) +static inline u32 dat_cr_query(DAT_CR_HANDLE cr, struct dat_cr_param *param) { return DAT_CALL_PROVIDER_FUNC(cr_query_func, cr, param); } @@ -1469,8 +1462,7 @@ static inline u32 dat_cr_reject(DAT_CR_H return DAT_CALL_PROVIDER_FUNC(cr_reject_func, cr); } -static inline u32 dat_evd_dequeue(DAT_EVD_HANDLE evd, - struct dat_event *event) +static inline u32 dat_evd_dequeue(DAT_EVD_HANDLE evd, struct dat_event *event) { return DAT_CALL_PROVIDER_FUNC(evd_dequeue_func, evd, event); } @@ -1480,94 +1472,81 @@ static inline u32 dat_evd_free(DAT_EVD_H return DAT_CALL_PROVIDER_FUNC(evd_free_func, evd); } -static inline u32 dat_evd_kcreate(DAT_IA_HANDLE ia, - DAT_COUNT qlen, - DAT_UPCALL_POLICY policy, - const struct dat_upcall_object *upcall, - DAT_EVD_FLAGS flags, - DAT_EVD_HANDLE * evd) +static inline u32 dat_evd_kcreate(DAT_IA_HANDLE ia, int qlen, + DAT_UPCALL_POLICY policy, + const struct dat_upcall_object *upcall, + DAT_EVD_FLAGS flags, DAT_EVD_HANDLE * evd) { - return DAT_CALL_PROVIDER_FUNC( - evd_kcreate_func, ia, qlen, policy, upcall, flags, evd); + return DAT_CALL_PROVIDER_FUNC(evd_kcreate_func, ia, qlen, policy, + upcall, flags, evd); } -static inline u32 dat_evd_modify_upcall(DAT_EVD_HANDLE evd, - DAT_UPCALL_POLICY policy, - const struct dat_upcall_object *upcall) +static inline u32 dat_evd_modify_upcall(DAT_EVD_HANDLE evd, + DAT_UPCALL_POLICY policy, + const struct dat_upcall_object *upcall) { - return DAT_CALL_PROVIDER_FUNC( - evd_modify_upcall_func, evd, policy, upcall); + return DAT_CALL_PROVIDER_FUNC(evd_modify_upcall_func, evd, policy, + upcall); } -static inline u32 dat_evd_post_se(DAT_EVD_HANDLE evd, - const struct dat_event *event) +static inline u32 dat_evd_post_se(DAT_EVD_HANDLE evd, + const struct dat_event *event) { return DAT_CALL_PROVIDER_FUNC(evd_post_se_func, evd, event); } -static inline u32 dat_evd_query(DAT_EVD_HANDLE evd, - struct dat_evd_param *param) +static inline u32 dat_evd_query(DAT_EVD_HANDLE evd, struct dat_evd_param *param) { return DAT_CALL_PROVIDER_FUNC(evd_query_func, evd, param); } -static inline u32 dat_evd_resize(DAT_EVD_HANDLE evd, DAT_COUNT qlen) +static inline u32 dat_evd_resize(DAT_EVD_HANDLE evd, int qlen) { return DAT_CALL_PROVIDER_FUNC(evd_resize_func, evd, qlen); } -static inline u32 dat_ep_connect(DAT_EP_HANDLE ep, - DAT_IA_ADDRESS_PTR ia_addr, - DAT_CONN_QUAL conn_qual, - DAT_TIMEOUT timeout, - DAT_COUNT private_data_size, - const void *private_data, - DAT_QOS qos, - DAT_CONNECT_FLAGS flags) -{ - return DAT_CALL_PROVIDER_FUNC( - ep_connect_func, ep, ia_addr, conn_qual, timeout, - private_data_size, private_data, qos, flags); -} - -static inline u32 dat_ep_create(DAT_IA_HANDLE ia, - DAT_PZ_HANDLE pz, - DAT_EVD_HANDLE in_evd, - DAT_EVD_HANDLE out_evd, - DAT_EVD_HANDLE connect_evd, - const struct dat_ep_attr *attr, - DAT_EP_HANDLE * ep) -{ - return DAT_CALL_PROVIDER_FUNC( - ep_create_func, ia, pz, in_evd, out_evd, connect_evd, attr, ep); +static inline u32 dat_ep_connect(DAT_EP_HANDLE ep, DAT_IA_ADDRESS_PTR ia_addr, + DAT_CONN_QUAL conn_qual, DAT_TIMEOUT timeout, + int private_data_size, + const void *private_data, DAT_QOS qos, + DAT_CONNECT_FLAGS flags) +{ + return DAT_CALL_PROVIDER_FUNC(ep_connect_func, ep, ia_addr, conn_qual, + timeout, private_data_size, private_data, + qos, flags); +} + +static inline u32 dat_ep_create(DAT_IA_HANDLE ia, DAT_PZ_HANDLE pz, + DAT_EVD_HANDLE in_evd, DAT_EVD_HANDLE out_evd, + DAT_EVD_HANDLE connect_evd, + const struct dat_ep_attr *attr, + DAT_EP_HANDLE *ep) +{ + return DAT_CALL_PROVIDER_FUNC(ep_create_func, ia, pz, in_evd, out_evd, + connect_evd, attr, ep); } -static inline u32 dat_ep_create_with_srq(DAT_IA_HANDLE ia, - DAT_PZ_HANDLE pz, - DAT_EVD_HANDLE in_evd, - DAT_EVD_HANDLE out_evd, - DAT_EVD_HANDLE connect_evd, - DAT_SRQ_HANDLE srq, - const struct dat_ep_attr *attr, - DAT_EP_HANDLE * ep) +static inline u32 dat_ep_create_with_srq(DAT_IA_HANDLE ia, DAT_PZ_HANDLE pz, + DAT_EVD_HANDLE in_evd, + DAT_EVD_HANDLE out_evd, + DAT_EVD_HANDLE connect_evd, + DAT_SRQ_HANDLE srq, + const struct dat_ep_attr *attr, + DAT_EP_HANDLE *ep) { return DAT_CALL_PROVIDER_FUNC(ep_create_with_srq_func, ia, pz, in_evd, out_evd, connect_evd, srq, attr, ep); } -static inline u32 dat_ep_disconnect(DAT_EP_HANDLE ep, - DAT_CLOSE_FLAGS flags) +static inline u32 dat_ep_disconnect(DAT_EP_HANDLE ep, DAT_CLOSE_FLAGS flags) { return DAT_CALL_PROVIDER_FUNC(ep_disconnect_func, ep, flags); } -static inline u32 dat_ep_dup_connect(DAT_EP_HANDLE ep, - DAT_EP_HANDLE dup_ep, - DAT_TIMEOUT timeout, - DAT_COUNT private_data_size, - const void *private_data, - DAT_QOS qos) +static inline u32 dat_ep_dup_connect(DAT_EP_HANDLE ep, DAT_EP_HANDLE dup_ep, + DAT_TIMEOUT timeout, int private_data_size, + const void *private_data, DAT_QOS qos) { return DAT_CALL_PROVIDER_FUNC(ep_dup_connect_func, ep, dup_ep, timeout, private_data_size, private_data, qos); @@ -1578,56 +1557,50 @@ static inline u32 dat_ep_free(DAT_EP_HAN return DAT_CALL_PROVIDER_FUNC(ep_free_func, ep); } -static inline u32 dat_ep_get_status(DAT_EP_HANDLE ep, - DAT_EP_STATE * state, - boolean_t *recv_idle, - boolean_t *req_idle) +static inline u32 dat_ep_get_status(DAT_EP_HANDLE ep, DAT_EP_STATE *state, + boolean_t *recv_idle, boolean_t *req_idle) { - return DAT_CALL_PROVIDER_FUNC( - ep_get_status_func, ep, state, recv_idle, req_idle); + return DAT_CALL_PROVIDER_FUNC(ep_get_status_func, ep, state, recv_idle, + req_idle); } -static inline u32 dat_ep_modify(DAT_EP_HANDLE ep, - DAT_EP_PARAM_MASK mask, - const struct dat_ep_param *param) +static inline u32 dat_ep_modify(DAT_EP_HANDLE ep, DAT_EP_PARAM_MASK mask, + const struct dat_ep_param *param) { return DAT_CALL_PROVIDER_FUNC(ep_modify_func, ep, mask, param); } -static inline u32 dat_ep_post_rdma_read(DAT_EP_HANDLE ep, - DAT_COUNT size, - struct dat_lmr_triplet *local_iov, - DAT_DTO_COOKIE cookie, - const struct dat_rmr_triplet *remote_iov, - DAT_COMPLETION_FLAGS flags) +static inline u32 dat_ep_post_rdma_read(DAT_EP_HANDLE ep, int size, + struct dat_lmr_triplet *local_iov, + DAT_DTO_COOKIE cookie, + const struct dat_rmr_triplet *remote_iov, + DAT_COMPLETION_FLAGS flags) { return DAT_CALL_PROVIDER_FUNC(ep_post_rdma_read_func, ep, size, local_iov, cookie, remote_iov, flags); } -static inline u32 dat_ep_post_rdma_write(DAT_EP_HANDLE ep, - DAT_COUNT size, - struct dat_lmr_triplet *local_iov, - DAT_DTO_COOKIE cookie, - const struct dat_rmr_triplet *remote_iov, - DAT_COMPLETION_FLAGS flags) +static inline u32 dat_ep_post_rdma_write(DAT_EP_HANDLE ep, int size, + struct dat_lmr_triplet *local_iov, + DAT_DTO_COOKIE cookie, + const struct dat_rmr_triplet *remote_iov, + DAT_COMPLETION_FLAGS flags) { return DAT_CALL_PROVIDER_FUNC(ep_post_rdma_write_func, ep, size, local_iov, cookie, remote_iov, flags); } -static inline u32 dat_ep_post_recv(DAT_EP_HANDLE ep, - DAT_COUNT size, - struct dat_lmr_triplet *local_iov, - DAT_DTO_COOKIE cookie, - DAT_COMPLETION_FLAGS flags) +static inline u32 dat_ep_post_recv(DAT_EP_HANDLE ep, int size, + struct dat_lmr_triplet *local_iov, + DAT_DTO_COOKIE cookie, + DAT_COMPLETION_FLAGS flags) { - return DAT_CALL_PROVIDER_FUNC( - ep_post_recv_func, ep, size, local_iov, cookie, flags); + return DAT_CALL_PROVIDER_FUNC(ep_post_recv_func, ep, size, local_iov, + cookie, flags); } static inline u32 dat_ep_post_send(DAT_EP_HANDLE ep, - DAT_COUNT size, + int size, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, DAT_COMPLETION_FLAGS flags) @@ -1641,12 +1614,11 @@ static inline u32 dat_ep_query(DAT_EP_HA return DAT_CALL_PROVIDER_FUNC(ep_query_func, ep, param); } -static inline u32 dat_ep_recv_query(DAT_EP_HANDLE ep, - DAT_COUNT * bufs_alloc, - DAT_COUNT * bufs_avail) +static inline u32 dat_ep_recv_query(DAT_EP_HANDLE ep, int *bufs_alloc, + int *bufs_avail) { - return DAT_CALL_PROVIDER_FUNC( - ep_recv_query_func, ep, bufs_alloc, bufs_avail); + return DAT_CALL_PROVIDER_FUNC(ep_recv_query_func, ep, bufs_alloc, + bufs_avail); } static inline u32 dat_ep_reset(DAT_EP_HANDLE ep) @@ -1654,16 +1626,15 @@ static inline u32 dat_ep_reset(DAT_EP_HA return DAT_CALL_PROVIDER_FUNC(ep_reset_func, ep); } -static inline u32 dat_ep_set_watermark(DAT_EP_HANDLE ep, - DAT_COUNT soft_high_watermark, - DAT_COUNT hard_high_watermark) +static inline u32 dat_ep_set_watermark(DAT_EP_HANDLE ep, + int soft_high_watermark, + int hard_high_watermark) { return DAT_CALL_PROVIDER_FUNC(ep_set_watermark_func, ep, soft_high_watermark, hard_high_watermark); } -static inline u32 dat_lmr_query(DAT_LMR_HANDLE lmr, - struct dat_lmr_param *param) +static inline u32 dat_lmr_query(DAT_LMR_HANDLE lmr, struct dat_lmr_param *param) { return DAT_CALL_PROVIDER_FUNC(lmr_query_func, lmr, param); } @@ -1673,23 +1644,23 @@ static inline u32 dat_lmr_free(DAT_LMR_H return DAT_CALL_PROVIDER_FUNC(lmr_free_func, lmr); } -static inline u32 dat_lmr_sync_rdma_read(DAT_IA_HANDLE ia, - const struct dat_lmr_triplet *iovs, - DAT_VLEN num_iovs) +static inline u32 dat_lmr_sync_rdma_read(DAT_IA_HANDLE ia, + const struct dat_lmr_triplet *iovs, + DAT_VLEN num_iovs) { - return DAT_CALL_PROVIDER_FUNC( - lmr_sync_rdma_read_func, ia, iovs, num_iovs); + return DAT_CALL_PROVIDER_FUNC(lmr_sync_rdma_read_func, ia, iovs, + num_iovs); } static inline u32 dat_lmr_sync_rdma_write(DAT_IA_HANDLE ia, - const struct dat_lmr_triplet *iovs, - DAT_VLEN num_iovs) + const struct dat_lmr_triplet *iovs, + DAT_VLEN num_iovs) { - return DAT_CALL_PROVIDER_FUNC( - lmr_sync_rdma_write_func, ia, iovs, num_iovs); + return DAT_CALL_PROVIDER_FUNC(lmr_sync_rdma_write_func, ia, iovs, + num_iovs); } -static inline u32 dat_rmr_create(DAT_PZ_HANDLE pz, DAT_RMR_HANDLE * rmr) +static inline u32 dat_rmr_create(DAT_PZ_HANDLE pz, DAT_RMR_HANDLE *rmr) { return DAT_CALL_PROVIDER_FUNC(rmr_create_func, pz, rmr); } @@ -1699,13 +1670,12 @@ static inline u32 dat_rmr_query(DAT_RMR_ return DAT_CALL_PROVIDER_FUNC(rmr_query_func, rmr, param); } -static inline u32 dat_rmr_bind(DAT_RMR_HANDLE rmr, - const struct dat_lmr_triplet *iov, - DAT_MEM_PRIV_FLAGS mem_flags, - DAT_EP_HANDLE ep, - DAT_RMR_COOKIE cookie, - DAT_COMPLETION_FLAGS comp_flags, - DAT_RMR_CONTEXT * context) +static inline u32 dat_rmr_bind(DAT_RMR_HANDLE rmr, + const struct dat_lmr_triplet *iov, + DAT_MEM_PRIV_FLAGS mem_flags, + DAT_EP_HANDLE ep, DAT_RMR_COOKIE cookie, + DAT_COMPLETION_FLAGS comp_flags, + DAT_RMR_CONTEXT *context) { return DAT_CALL_PROVIDER_FUNC(rmr_bind_func, rmr, iov, mem_flags, ep, cookie, comp_flags, context); @@ -1716,28 +1686,23 @@ static inline u32 dat_rmr_free(DAT_RMR_H return DAT_CALL_PROVIDER_FUNC(rmr_free_func, rmr); } -static inline u32 dat_psp_create(DAT_IA_HANDLE ia, - DAT_CONN_QUAL conn_qual, - DAT_EVD_HANDLE evd, - DAT_PSP_FLAGS flags, - DAT_PSP_HANDLE * psp) +static inline u32 dat_psp_create(DAT_IA_HANDLE ia, DAT_CONN_QUAL conn_qual, + DAT_EVD_HANDLE evd, DAT_PSP_FLAGS flags, + DAT_PSP_HANDLE *psp) { - return DAT_CALL_PROVIDER_FUNC( - psp_create_func, ia, conn_qual, evd, flags, psp); + return DAT_CALL_PROVIDER_FUNC(psp_create_func, ia, conn_qual, evd, + flags, psp); } -static inline u32 dat_psp_create_any(DAT_IA_HANDLE ia, - DAT_CONN_QUAL * conn_qual, - DAT_EVD_HANDLE evd, - DAT_PSP_FLAGS flags, - DAT_PSP_HANDLE * psp) +static inline u32 dat_psp_create_any(DAT_IA_HANDLE ia, DAT_CONN_QUAL *conn_qual, + DAT_EVD_HANDLE evd, DAT_PSP_FLAGS flags, + DAT_PSP_HANDLE *psp) { - return DAT_CALL_PROVIDER_FUNC( - psp_create_any_func, ia, conn_qual, evd, flags, psp); + return DAT_CALL_PROVIDER_FUNC(psp_create_any_func, ia, conn_qual, evd, + flags, psp); } -static inline u32 dat_psp_query(DAT_PSP_HANDLE psp, - struct dat_psp_param *param) +static inline u32 dat_psp_query(DAT_PSP_HANDLE psp, struct dat_psp_param *param) { return DAT_CALL_PROVIDER_FUNC(psp_query_func, psp, param); } @@ -1747,18 +1712,15 @@ static inline u32 dat_psp_free(DAT_PSP_H return DAT_CALL_PROVIDER_FUNC(psp_free_func, psp); } -static inline u32 dat_rsp_create(DAT_IA_HANDLE ia, - DAT_CONN_QUAL conn_qual, - DAT_EP_HANDLE ep, - DAT_EVD_HANDLE evd, - DAT_RSP_HANDLE * rsp) +static inline u32 dat_rsp_create(DAT_IA_HANDLE ia, DAT_CONN_QUAL conn_qual, + DAT_EP_HANDLE ep, DAT_EVD_HANDLE evd, + DAT_RSP_HANDLE *rsp) { - return DAT_CALL_PROVIDER_FUNC( - rsp_create_func, ia, conn_qual, ep, evd, rsp); + return DAT_CALL_PROVIDER_FUNC(rsp_create_func, ia, conn_qual, ep, evd, + rsp); } -static inline u32 dat_rsp_query(DAT_RSP_HANDLE rsp, - struct dat_rsp_param *param) +static inline u32 dat_rsp_query(DAT_RSP_HANDLE rsp, struct dat_rsp_param *param) { return DAT_CALL_PROVIDER_FUNC(rsp_query_func, rsp, param); } @@ -1768,13 +1730,12 @@ static inline u32 dat_rsp_free(DAT_RSP_H return DAT_CALL_PROVIDER_FUNC(rsp_free_func, rsp); } -static inline u32 dat_pz_create(DAT_IA_HANDLE ia, DAT_PZ_HANDLE * pz) +static inline u32 dat_pz_create(DAT_IA_HANDLE ia, DAT_PZ_HANDLE *pz) { return DAT_CALL_PROVIDER_FUNC(pz_create_func, ia, pz); } -static inline u32 dat_pz_query(DAT_PZ_HANDLE pz, - struct dat_pz_param *param) +static inline u32 dat_pz_query(DAT_PZ_HANDLE pz, struct dat_pz_param *param) { return DAT_CALL_PROVIDER_FUNC(pz_query_func, pz, param); } @@ -1784,10 +1745,9 @@ static inline u32 dat_pz_free(DAT_PZ_HAN return DAT_CALL_PROVIDER_FUNC(pz_free_func, pz); } -static inline u32 dat_srq_create(DAT_IA_HANDLE ia, - DAT_PZ_HANDLE pz, - struct dat_srq_attr *attr, - DAT_SRQ_HANDLE * srq) +static inline u32 dat_srq_create(DAT_IA_HANDLE ia, DAT_PZ_HANDLE pz, + struct dat_srq_attr *attr, + DAT_SRQ_HANDLE *srq) { return DAT_CALL_PROVIDER_FUNC(srq_create_func, ia, pz, attr, srq); } @@ -1797,50 +1757,44 @@ static inline u32 dat_srq_free(DAT_SRQ_H return DAT_CALL_PROVIDER_FUNC(srq_free_func, srq); } -static inline u32 dat_srq_post_recv(DAT_SRQ_HANDLE srq, - DAT_COUNT num_iovs, - struct dat_lmr_triplet *iovs, - DAT_DTO_COOKIE cookie) +static inline u32 dat_srq_post_recv(DAT_SRQ_HANDLE srq, int num_iovs, + struct dat_lmr_triplet *iovs, + DAT_DTO_COOKIE cookie) { - return DAT_CALL_PROVIDER_FUNC( - srq_post_recv_func, srq, num_iovs, iovs, cookie); + return DAT_CALL_PROVIDER_FUNC(srq_post_recv_func, srq, num_iovs, iovs, + cookie); } -static inline u32 dat_srq_query(DAT_SRQ_HANDLE srq, - struct dat_srq_param *param) +static inline u32 dat_srq_query(DAT_SRQ_HANDLE srq, struct dat_srq_param *param) { return DAT_CALL_PROVIDER_FUNC(srq_query_func, srq, param); } -static inline u32 dat_srq_resize(DAT_SRQ_HANDLE srq, - DAT_COUNT max_recv_dtos) +static inline u32 dat_srq_resize(DAT_SRQ_HANDLE srq, int max_recv_dtos) { return DAT_CALL_PROVIDER_FUNC(srq_resize_func, srq, max_recv_dtos); } -static inline u32 dat_srq_set_lw(DAT_SRQ_HANDLE srq, - DAT_COUNT low_watermark) +static inline u32 dat_srq_set_lw(DAT_SRQ_HANDLE srq, int low_watermark) { return DAT_CALL_PROVIDER_FUNC(srq_set_lw_func, srq, low_watermark); } -static inline u32 dat_lmr_kcreate(DAT_IA_HANDLE ia, - DAT_MEM_TYPE type, - DAT_REGION_DESCRIPTION region, - DAT_VLEN len, - DAT_PZ_HANDLE pz, - DAT_MEM_PRIV_FLAGS privileges, - DAT_MEM_OPTIMIZE_FLAGS optimization, - DAT_LMR_HANDLE * lmr_handle, - DAT_LMR_CONTEXT * lmr_context, - DAT_RMR_CONTEXT * rmr_context, - DAT_VLEN * registered_length, - DAT_VADDR *registered_address) -{ - return DAT_CALL_PROVIDER_FUNC( - lmr_kcreate_func, ia, type, region, len, pz, privileges, - optimization, lmr_handle, lmr_context, rmr_context, - registered_length, registered_address); +static inline u32 dat_lmr_kcreate(DAT_IA_HANDLE ia, DAT_MEM_TYPE type, + DAT_REGION_DESCRIPTION region, DAT_VLEN len, + DAT_PZ_HANDLE pz, + DAT_MEM_PRIV_FLAGS privileges, + DAT_MEM_OPTIMIZE_FLAGS optimization, + DAT_LMR_HANDLE *lmr_handle, + DAT_LMR_CONTEXT *lmr_context, + DAT_RMR_CONTEXT *rmr_context, + DAT_VLEN *registered_length, + DAT_VADDR *registered_address) +{ + return DAT_CALL_PROVIDER_FUNC(lmr_kcreate_func, ia, type, region, len, + pz, privileges, optimization, lmr_handle, + lmr_context, rmr_context, + registered_length, registered_address); } #endif /* DAT_H */ From tduffy at sun.com Thu May 12 11:19:54 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 12 May 2005 11:19:54 -0700 Subject: [openib-general] Re: Latest kdapl load problem In-Reply-To: <1115913762.4479.40.camel@localhost.localdomain> References: <1115912597.4479.29.camel@localhost.localdomain> <1115913762.4479.40.camel@localhost.localdomain> Message-ID: <1115921994.9070.19.camel@duffman> On Thu, 2005-05-12 at 12:02 -0400, Hal Rosenstock wrote: > On Thu, 2005-05-12 at 11:52, James Lentini wrote: > > Did you rebuild the kdapltest module? > > Thanks. I forgot to do that. > > Does kdapltest build fine for you ? I'm getting some errors on building > the user portion. Sorry, that was my fault. Forgot to check that my changes still allowed user to build. > Modified: gen2/users/jlentini/linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.h > =================================================================== > --- gen2/users/jlentini/linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.h 2005-05-12 15:55:55 UTC (rev 2316) > +++ gen2/users/jlentini/linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.h 2005-05-12 15:58:00 UTC (rev 2317) > @@ -50,7 +50,15 @@ > #endif > #endif > > +typedef enum { > +#undef FALSE > + FALSE = 0, > +#undef TRUE > + TRUE = 1 > +} boolean_t; > > +typedef uint32_t u32; > + > /* Default Device Name */ > #define DT_MdepDeviceName "ia0a" > > @@ -72,7 +80,7 @@ BTW, u32 should not be used in user stuff. For that we should use the standard uint32_t. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From tduffy at sun.com Thu May 12 11:24:38 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 12 May 2005 11:24:38 -0700 Subject: [openib-general] [PATCH] kDAPL: remove typedef DAT_NAME_PTR Message-ID: <1115922278.9070.23.camel@duffman> Remove typedef DAT_NAME_PTR. Also get rid of DAT_UVERYLONG & DAT_PADDR that are not used anywhere outside of dat.h. Also, a little more whitespace cleaning that should have been done with the DAT_RETURN patch. Signed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat/api.c linux-kernel2/dat/api.c --- linux-kernel/dat/api.c 2005-05-12 10:08:24.774000000 -0700 +++ linux-kernel2/dat/api.c 2005-05-12 11:02:59.847003000 -0700 @@ -575,7 +575,7 @@ static u32 dat_strerror_minor(u32 value, * Function: dat_ia_openv ***********************************************************************/ -u32 dat_ia_openv(const DAT_NAME_PTR name, int async_event_qlen, +u32 dat_ia_openv(const char *name, int async_event_qlen, DAT_EVD_HANDLE *async_event_handle, DAT_IA_HANDLE *ia_handle, u32 dat_major, u32 dat_minor, boolean_t thread_safety) { diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat/dat.h linux-kernel2/dat/dat.h --- linux-kernel/dat/dat.h 2005-05-12 10:25:26.111007000 -0700 +++ linux-kernel2/dat/dat.h 2005-05-12 11:14:17.374001000 -0700 @@ -37,11 +37,6 @@ #define DAT_VERSION_MINOR 2 #define DAT_THREADSAFE FALSE -/* unsigned longest native to compiler */ -typedef unsigned long long DAT_UVERYLONG; - -typedef u64 DAT_PADDR; - /* * All return codes are actually a 3-way tuple: * @@ -352,7 +347,6 @@ typedef enum dat_handle_type { DAT_HANDLE_TYPE_SRQ } DAT_HANDLE_TYPE; -typedef char *DAT_NAME_PTR; #define DAT_NAME_MAX_LENGTH 256 struct dat_named_attr { @@ -370,7 +364,7 @@ typedef enum { typedef union dat_context { void *as_ptr; u64 as_64; - DAT_UVERYLONG as_index; + unsigned long long as_index; } DAT_CONTEXT; typedef DAT_CONTEXT DAT_DTO_COOKIE; @@ -983,7 +977,7 @@ typedef union dat_region_description { DAT_LMR_HANDLE for_lmr_handle; void *for_pointer; /* For kdapl only */ void *for_array; /* For kdapl only */ - DAT_PADDR for_pa; /* For kdapl only */ + u64 for_pa; /* For kdapl only */ } DAT_REGION_DESCRIPTION; /* LMR Arguments */ @@ -1103,179 +1097,176 @@ struct dat_provider_attr { * ****************************************************************/ -typedef u32 (*DAT_IA_OPEN_FUNC) ( - const DAT_NAME_PTR, int, DAT_EVD_HANDLE *, DAT_IA_HANDLE *); +typedef u32 (*DAT_IA_OPEN_FUNC)(const char *, int, DAT_EVD_HANDLE *, + DAT_IA_HANDLE *); -typedef u32 (*DAT_IA_OPENV_FUNC) ( - const DAT_NAME_PTR, int, DAT_EVD_HANDLE *, DAT_IA_HANDLE *, - u32, u32, boolean_t); +typedef u32 (*DAT_IA_OPENV_FUNC)(const char *, int, DAT_EVD_HANDLE *, + DAT_IA_HANDLE *, u32, u32, boolean_t); -typedef u32 (*DAT_IA_CLOSE_FUNC) (DAT_IA_HANDLE, DAT_CLOSE_FLAGS); +typedef u32 (*DAT_IA_CLOSE_FUNC)(DAT_IA_HANDLE, DAT_CLOSE_FLAGS); -typedef u32 (*DAT_IA_QUERY_FUNC) (DAT_IA_HANDLE, DAT_EVD_HANDLE *, - struct dat_ia_attr *, - struct dat_provider_attr *); +typedef u32 (*DAT_IA_QUERY_FUNC)(DAT_IA_HANDLE, DAT_EVD_HANDLE *, + struct dat_ia_attr *, + struct dat_provider_attr *); -typedef u32 (*DAT_SET_CONSUMER_CONTEXT_FUNC) (DAT_HANDLE, DAT_CONTEXT); +typedef u32 (*DAT_SET_CONSUMER_CONTEXT_FUNC)(DAT_HANDLE, DAT_CONTEXT); -typedef u32 (*DAT_GET_CONSUMER_CONTEXT_FUNC) (DAT_HANDLE, DAT_CONTEXT *); +typedef u32 (*DAT_GET_CONSUMER_CONTEXT_FUNC)(DAT_HANDLE, DAT_CONTEXT *); -typedef u32 (*DAT_GET_HANDLE_TYPE_FUNC) (DAT_HANDLE, DAT_HANDLE_TYPE *); +typedef u32 (*DAT_GET_HANDLE_TYPE_FUNC)(DAT_HANDLE, DAT_HANDLE_TYPE *); -typedef u32 (*DAT_CR_QUERY_FUNC) (DAT_CR_HANDLE, struct dat_cr_param *); +typedef u32 (*DAT_CR_QUERY_FUNC)(DAT_CR_HANDLE, struct dat_cr_param *); -typedef u32 (*DAT_CR_ACCEPT_FUNC) (DAT_CR_HANDLE, DAT_EP_HANDLE, int, - const void*); +typedef u32 (*DAT_CR_ACCEPT_FUNC)(DAT_CR_HANDLE, DAT_EP_HANDLE, int, + const void*); -typedef u32 (*DAT_CR_REJECT_FUNC) (DAT_CR_HANDLE); +typedef u32 (*DAT_CR_REJECT_FUNC)(DAT_CR_HANDLE); -typedef u32 (*DAT_CR_HANDOFF_FUNC) (DAT_CR_HANDLE, DAT_CONN_QUAL); +typedef u32 (*DAT_CR_HANDOFF_FUNC)(DAT_CR_HANDLE, DAT_CONN_QUAL); -typedef u32 (*DAT_EVD_RESIZE_FUNC) (DAT_EVD_HANDLE, int); +typedef u32 (*DAT_EVD_RESIZE_FUNC)(DAT_EVD_HANDLE, int); -typedef u32 (*DAT_EVD_POST_SE_FUNC) (DAT_EVD_HANDLE, - const struct dat_event *); +typedef u32 (*DAT_EVD_POST_SE_FUNC)(DAT_EVD_HANDLE, const struct dat_event *); -typedef u32 (*DAT_EVD_DEQUEUE_FUNC) (DAT_EVD_HANDLE, struct dat_event *); +typedef u32 (*DAT_EVD_DEQUEUE_FUNC)(DAT_EVD_HANDLE, struct dat_event *); -typedef u32 (*DAT_EVD_FREE_FUNC) (DAT_EVD_HANDLE); +typedef u32 (*DAT_EVD_FREE_FUNC)(DAT_EVD_HANDLE); -typedef u32 (*DAT_EVD_QUERY_FUNC) (DAT_EVD_HANDLE, - struct dat_evd_param *); +typedef u32 (*DAT_EVD_QUERY_FUNC)(DAT_EVD_HANDLE, struct dat_evd_param *); -typedef u32 (*DAT_EP_CREATE_FUNC) ( - DAT_IA_HANDLE, DAT_PZ_HANDLE, DAT_EVD_HANDLE, DAT_EVD_HANDLE, - DAT_EVD_HANDLE, const struct dat_ep_attr *, DAT_EP_HANDLE *); +typedef u32 (*DAT_EP_CREATE_FUNC)(DAT_IA_HANDLE, DAT_PZ_HANDLE, DAT_EVD_HANDLE, + DAT_EVD_HANDLE, DAT_EVD_HANDLE, + const struct dat_ep_attr *, DAT_EP_HANDLE *); -typedef u32 (*DAT_EP_CREATE_WITH_SRQ_FUNC) ( - DAT_IA_HANDLE, DAT_PZ_HANDLE, DAT_EVD_HANDLE, DAT_EVD_HANDLE, - DAT_EVD_HANDLE, DAT_SRQ_HANDLE, const struct dat_ep_attr *, DAT_EP_HANDLE *); +typedef u32 (*DAT_EP_CREATE_WITH_SRQ_FUNC)(DAT_IA_HANDLE, DAT_PZ_HANDLE, + DAT_EVD_HANDLE, DAT_EVD_HANDLE, + DAT_EVD_HANDLE, DAT_SRQ_HANDLE, + const struct dat_ep_attr *, + DAT_EP_HANDLE *); -typedef u32 (*DAT_EP_QUERY_FUNC) (DAT_EP_HANDLE, struct dat_ep_param *); +typedef u32 (*DAT_EP_QUERY_FUNC)(DAT_EP_HANDLE, struct dat_ep_param *); -typedef u32 (*DAT_EP_MODIFY_FUNC) ( - DAT_EP_HANDLE, DAT_EP_PARAM_MASK, const struct dat_ep_param *); +typedef u32 (*DAT_EP_MODIFY_FUNC)(DAT_EP_HANDLE, DAT_EP_PARAM_MASK, + const struct dat_ep_param *); -typedef u32 (*DAT_EP_CONNECT_FUNC) (DAT_EP_HANDLE, DAT_IA_ADDRESS_PTR, - DAT_CONN_QUAL, DAT_TIMEOUT, int, - const void *, DAT_QOS, DAT_CONNECT_FLAGS); +typedef u32 (*DAT_EP_CONNECT_FUNC)(DAT_EP_HANDLE, DAT_IA_ADDRESS_PTR, + DAT_CONN_QUAL, DAT_TIMEOUT, int, + const void *, DAT_QOS, DAT_CONNECT_FLAGS); -typedef u32 (*DAT_EP_DUP_CONNECT_FUNC) (DAT_EP_HANDLE, DAT_EP_HANDLE, - DAT_TIMEOUT, int, const void *, - DAT_QOS); +typedef u32 (*DAT_EP_DUP_CONNECT_FUNC)(DAT_EP_HANDLE, DAT_EP_HANDLE, + DAT_TIMEOUT, int, const void *, DAT_QOS); -typedef u32 (*DAT_EP_DISCONNECT_FUNC) (DAT_EP_HANDLE, DAT_CLOSE_FLAGS); +typedef u32 (*DAT_EP_DISCONNECT_FUNC)(DAT_EP_HANDLE, DAT_CLOSE_FLAGS); -typedef u32 (*DAT_EP_POST_SEND_FUNC) ( - DAT_EP_HANDLE, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, - DAT_COMPLETION_FLAGS); +typedef u32 (*DAT_EP_POST_SEND_FUNC)(DAT_EP_HANDLE, int, + struct dat_lmr_triplet *, DAT_DTO_COOKIE, + DAT_COMPLETION_FLAGS); -typedef u32 (*DAT_EP_POST_RECV_FUNC) ( - DAT_EP_HANDLE, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, - DAT_COMPLETION_FLAGS); +typedef u32 (*DAT_EP_POST_RECV_FUNC)(DAT_EP_HANDLE, int, + struct dat_lmr_triplet *, DAT_DTO_COOKIE, + DAT_COMPLETION_FLAGS); -typedef u32 (*DAT_EP_POST_RDMA_READ_FUNC) ( - DAT_EP_HANDLE, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, - const struct dat_rmr_triplet *, DAT_COMPLETION_FLAGS); +typedef u32 (*DAT_EP_POST_RDMA_READ_FUNC)(DAT_EP_HANDLE, int, + struct dat_lmr_triplet *, + DAT_DTO_COOKIE, + const struct dat_rmr_triplet *, + DAT_COMPLETION_FLAGS); -typedef u32 (*DAT_EP_POST_RDMA_WRITE_FUNC) ( - DAT_EP_HANDLE, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, - const struct dat_rmr_triplet *, DAT_COMPLETION_FLAGS); +typedef u32 (*DAT_EP_POST_RDMA_WRITE_FUNC)(DAT_EP_HANDLE, int, + struct dat_lmr_triplet *, + DAT_DTO_COOKIE, + const struct dat_rmr_triplet *, + DAT_COMPLETION_FLAGS); -typedef u32 (*DAT_EP_GET_STATUS_FUNC) ( - DAT_EP_HANDLE, DAT_EP_STATE *, boolean_t *, boolean_t *); +typedef u32 (*DAT_EP_GET_STATUS_FUNC)(DAT_EP_HANDLE, DAT_EP_STATE *, + boolean_t *, boolean_t *); -typedef u32 (*DAT_EP_FREE_FUNC) (DAT_EP_HANDLE); +typedef u32 (*DAT_EP_FREE_FUNC)(DAT_EP_HANDLE); -typedef u32 (*DAT_EP_RESET_FUNC) (DAT_EP_HANDLE); +typedef u32 (*DAT_EP_RESET_FUNC)(DAT_EP_HANDLE); -typedef u32 (*DAT_EP_RECV_QUERY_FUNC) ( - DAT_EP_HANDLE, int *, int *); +typedef u32 (*DAT_EP_RECV_QUERY_FUNC)(DAT_EP_HANDLE, int *, int *); -typedef u32 (*DAT_EP_SET_WATERMARK_FUNC) ( - DAT_EP_HANDLE, int, int); +typedef u32 (*DAT_EP_SET_WATERMARK_FUNC)(DAT_EP_HANDLE, int, int); -typedef u32 (*DAT_LMR_FREE_FUNC) (DAT_LMR_HANDLE); +typedef u32 (*DAT_LMR_FREE_FUNC)(DAT_LMR_HANDLE); -typedef u32 (*DAT_LMR_QUERY_FUNC) (DAT_LMR_HANDLE, - struct dat_lmr_param *); +typedef u32 (*DAT_LMR_QUERY_FUNC)(DAT_LMR_HANDLE, struct dat_lmr_param *); -typedef u32 (*DAT_LMR_SYNC_RDMA_READ_FUNC) ( - DAT_IA_HANDLE, const struct dat_lmr_triplet *, DAT_VLEN); +typedef u32 (*DAT_LMR_SYNC_RDMA_READ_FUNC)(DAT_IA_HANDLE, + const struct dat_lmr_triplet *, + DAT_VLEN); -typedef u32 (*DAT_LMR_SYNC_RDMA_WRITE_FUNC) ( - DAT_IA_HANDLE, const struct dat_lmr_triplet *, DAT_VLEN); +typedef u32 (*DAT_LMR_SYNC_RDMA_WRITE_FUNC)(DAT_IA_HANDLE, + const struct dat_lmr_triplet *, + DAT_VLEN); -typedef u32 (*DAT_RMR_CREATE_FUNC) (DAT_PZ_HANDLE, DAT_RMR_HANDLE *); +typedef u32 (*DAT_RMR_CREATE_FUNC)(DAT_PZ_HANDLE, DAT_RMR_HANDLE *); -typedef u32 (*DAT_RMR_QUERY_FUNC) (DAT_RMR_HANDLE, struct dat_rmr_param *); +typedef u32 (*DAT_RMR_QUERY_FUNC)(DAT_RMR_HANDLE, struct dat_rmr_param *); -typedef u32 (*DAT_RMR_BIND_FUNC) ( - DAT_RMR_HANDLE, const struct dat_lmr_triplet *, DAT_MEM_PRIV_FLAGS, - DAT_EP_HANDLE, DAT_RMR_COOKIE, DAT_COMPLETION_FLAGS, DAT_RMR_CONTEXT *); +typedef u32 (*DAT_RMR_BIND_FUNC)(DAT_RMR_HANDLE, const struct dat_lmr_triplet *, + DAT_MEM_PRIV_FLAGS, DAT_EP_HANDLE, + DAT_RMR_COOKIE, DAT_COMPLETION_FLAGS, + DAT_RMR_CONTEXT *); -typedef u32 (*DAT_RMR_FREE_FUNC) (DAT_RMR_HANDLE); +typedef u32 (*DAT_RMR_FREE_FUNC)(DAT_RMR_HANDLE); -typedef u32 (*DAT_PSP_CREATE_FUNC) ( - DAT_IA_HANDLE, DAT_CONN_QUAL, DAT_EVD_HANDLE, DAT_PSP_FLAGS, - DAT_PSP_HANDLE *); +typedef u32 (*DAT_PSP_CREATE_FUNC)(DAT_IA_HANDLE, DAT_CONN_QUAL, DAT_EVD_HANDLE, + DAT_PSP_FLAGS, DAT_PSP_HANDLE *); -typedef u32 (*DAT_PSP_CREATE_ANY_FUNC) ( - DAT_IA_HANDLE, DAT_CONN_QUAL *, DAT_EVD_HANDLE, DAT_PSP_FLAGS, - DAT_PSP_HANDLE *); +typedef u32 (*DAT_PSP_CREATE_ANY_FUNC)(DAT_IA_HANDLE, DAT_CONN_QUAL *, + DAT_EVD_HANDLE, DAT_PSP_FLAGS, + DAT_PSP_HANDLE *); -typedef u32 (*DAT_PSP_QUERY_FUNC) (DAT_PSP_HANDLE, - struct dat_psp_param *); +typedef u32 (*DAT_PSP_QUERY_FUNC)(DAT_PSP_HANDLE, struct dat_psp_param *); -typedef u32 (*DAT_PSP_FREE_FUNC) (DAT_PSP_HANDLE); +typedef u32 (*DAT_PSP_FREE_FUNC)(DAT_PSP_HANDLE); -typedef u32 (*DAT_RSP_CREATE_FUNC) ( - DAT_IA_HANDLE, DAT_CONN_QUAL, DAT_EP_HANDLE, DAT_EVD_HANDLE, - DAT_RSP_HANDLE *); +typedef u32 (*DAT_RSP_CREATE_FUNC)(DAT_IA_HANDLE, DAT_CONN_QUAL, DAT_EP_HANDLE, + DAT_EVD_HANDLE, DAT_RSP_HANDLE *); -typedef u32 (*DAT_RSP_QUERY_FUNC) (DAT_RSP_HANDLE, - struct dat_rsp_param *); +typedef u32 (*DAT_RSP_QUERY_FUNC)(DAT_RSP_HANDLE, struct dat_rsp_param *); -typedef u32 (*DAT_RSP_FREE_FUNC) (DAT_RSP_HANDLE); +typedef u32 (*DAT_RSP_FREE_FUNC)(DAT_RSP_HANDLE); -typedef u32 (*DAT_PZ_CREATE_FUNC) (DAT_IA_HANDLE, DAT_PZ_HANDLE *); +typedef u32 (*DAT_PZ_CREATE_FUNC)(DAT_IA_HANDLE, DAT_PZ_HANDLE *); -typedef u32 (*DAT_PZ_QUERY_FUNC) (DAT_PZ_HANDLE, struct dat_pz_param *); +typedef u32 (*DAT_PZ_QUERY_FUNC)(DAT_PZ_HANDLE, struct dat_pz_param *); -typedef u32 (*DAT_PZ_FREE_FUNC) (DAT_PZ_HANDLE); +typedef u32 (*DAT_PZ_FREE_FUNC)(DAT_PZ_HANDLE); -typedef u32 (*DAT_SRQ_CREATE_FUNC) ( - DAT_IA_HANDLE, DAT_PZ_HANDLE, struct dat_srq_attr *, DAT_SRQ_HANDLE *); +typedef u32 (*DAT_SRQ_CREATE_FUNC)(DAT_IA_HANDLE, DAT_PZ_HANDLE, + struct dat_srq_attr *, DAT_SRQ_HANDLE *); -typedef u32 (*DAT_SRQ_SET_LW_FUNC) (DAT_SRQ_HANDLE, int); +typedef u32 (*DAT_SRQ_SET_LW_FUNC)(DAT_SRQ_HANDLE, int); -typedef u32 (*DAT_SRQ_FREE_FUNC) (DAT_SRQ_HANDLE); +typedef u32 (*DAT_SRQ_FREE_FUNC)(DAT_SRQ_HANDLE); -typedef u32 (*DAT_SRQ_QUERY_FUNC) (DAT_SRQ_HANDLE, - struct dat_srq_param *); +typedef u32 (*DAT_SRQ_QUERY_FUNC)(DAT_SRQ_HANDLE, struct dat_srq_param *); -typedef u32 (*DAT_SRQ_RESIZE_FUNC) (DAT_SRQ_HANDLE, int); +typedef u32 (*DAT_SRQ_RESIZE_FUNC)(DAT_SRQ_HANDLE, int); -typedef u32 (*DAT_SRQ_POST_RECV_FUNC) ( - DAT_SRQ_HANDLE, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE); +typedef u32 (*DAT_SRQ_POST_RECV_FUNC)(DAT_SRQ_HANDLE, int, + struct dat_lmr_triplet *, DAT_DTO_COOKIE); -typedef u32 (*DAT_LMR_KCREATE_FUNC) ( - DAT_IA_HANDLE, DAT_MEM_TYPE, DAT_REGION_DESCRIPTION, DAT_VLEN, - DAT_PZ_HANDLE, DAT_MEM_PRIV_FLAGS, DAT_MEM_OPTIMIZE_FLAGS, - DAT_LMR_HANDLE *, DAT_LMR_CONTEXT *, DAT_RMR_CONTEXT *, DAT_VLEN *, - DAT_VADDR *); +typedef u32 (*DAT_LMR_KCREATE_FUNC)(DAT_IA_HANDLE, DAT_MEM_TYPE, + DAT_REGION_DESCRIPTION, DAT_VLEN, + DAT_PZ_HANDLE, DAT_MEM_PRIV_FLAGS, + DAT_MEM_OPTIMIZE_FLAGS, DAT_LMR_HANDLE *, + DAT_LMR_CONTEXT *, DAT_RMR_CONTEXT *, + DAT_VLEN *, DAT_VADDR *); -typedef u32 (*DAT_IA_MEMTYPE_HINT_FUNC) ( - DAT_IA_HANDLE, DAT_MEM_TYPE, DAT_VLEN, DAT_MEM_OPTIMIZE_FLAGS, - DAT_VLEN *, DAT_VADDR *); +typedef u32 (*DAT_IA_MEMTYPE_HINT_FUNC)(DAT_IA_HANDLE, DAT_MEM_TYPE, DAT_VLEN, + DAT_MEM_OPTIMIZE_FLAGS, DAT_VLEN *, + DAT_VADDR *); -typedef u32 (*DAT_EVD_KCREATE_FUNC) ( - DAT_IA_HANDLE, int, DAT_UPCALL_POLICY, - const struct dat_upcall_object *, - DAT_EVD_FLAGS, DAT_EVD_HANDLE *); +typedef u32 (*DAT_EVD_KCREATE_FUNC) (DAT_IA_HANDLE, int, DAT_UPCALL_POLICY, + const struct dat_upcall_object *, + DAT_EVD_FLAGS, DAT_EVD_HANDLE *); -typedef u32 (*DAT_EVD_MODIFY_UPCALL_FUNC) ( - DAT_EVD_HANDLE, DAT_UPCALL_POLICY, const struct dat_upcall_object *); +typedef u32 (*DAT_EVD_MODIFY_UPCALL_FUNC)(DAT_EVD_HANDLE, DAT_UPCALL_POLICY, + const struct dat_upcall_object *); struct dat_provider { const char *device_name; @@ -1377,7 +1368,7 @@ extern u32 dat_registry_remove_provider( /* * DAT registry functions for consumers */ -extern u32 dat_ia_openv(const DAT_NAME_PTR name, int async_event_qlen, +extern u32 dat_ia_openv(const char *name, int async_event_qlen, DAT_EVD_HANDLE *async_event_handle, DAT_IA_HANDLE *ia_handle, u32 dat_major, u32 dat_minor, boolean_t thread_safety); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl.h linux-kernel2/dat-provider/dapl.h --- linux-kernel/dat-provider/dapl.h 2005-05-12 10:06:07.536001000 -0700 +++ linux-kernel2/dat-provider/dapl.h 2005-05-12 11:02:59.834000000 -0700 @@ -474,7 +474,7 @@ typedef char DAPL_HCA_NAME[DAPL_HCA_NAME * DAT API functions */ -extern u32 dapl_ia_open(const DAT_NAME_PTR name, int async_evd_qlen, +extern u32 dapl_ia_open(const char *name, int async_evd_qlen, DAT_EVD_HANDLE *async_evd_handle_ptr, DAT_IA_HANDLE *ia_handle_ptr); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ia_open.c linux-kernel2/dat-provider/dapl_ia_open.c --- linux-kernel/dat-provider/dapl_ia_open.c 2005-05-12 09:34:59.157003000 -0700 +++ linux-kernel2/dat-provider/dapl_ia_open.c 2005-05-12 11:02:59.837003000 -0700 @@ -51,7 +51,7 @@ * That function maps the DAT_NAME parameter of dat_ia_open to a DAT_PROVIDER, * and calls this function. */ -u32 dapl_ia_open(const DAT_NAME_PTR name, int async_evd_qlen, +u32 dapl_ia_open(const char *name, int async_evd_qlen, DAT_EVD_HANDLE *async_evd_handle_ptr, DAT_IA_HANDLE *ia_handle_ptr) { diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/cmd/dapl_netaddr.c linux-kernel2/test/dapltest/cmd/dapl_netaddr.c --- linux-kernel/test/dapltest/cmd/dapl_netaddr.c 2005-05-11 08:53:17.661015000 -0700 +++ linux-kernel2/test/dapltest/cmd/dapl_netaddr.c 2005-05-12 11:02:59.874000000 -0700 @@ -50,7 +50,7 @@ DT_NetAddrFree (Per_Test_Data_t * pt_ptr boolean_t DT_NetAddrLookupHostAddress (DAT_IA_ADDRESS_PTR to_netaddr, - DAT_NAME_PTR hostname) + char * hostname) { struct addrinfo *target; int rval; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/include/dapl_proto.h linux-kernel2/test/dapltest/include/dapl_proto.h --- linux-kernel/test/dapltest/include/dapl_proto.h 2005-05-12 09:34:59.322003000 -0700 +++ linux-kernel2/test/dapltest/include/dapl_proto.h 2005-05-12 11:02:59.881011000 -0700 @@ -604,7 +604,7 @@ void DT_assert_fail (DT_Tdep_Print_H char *file, char *baseFile, int line); -int DT_ia_open (DAT_NAME_PTR dev_name, DAT_IA_HANDLE *ia_handle); +int DT_ia_open (char *dev_name, DAT_IA_HANDLE *ia_handle); int DT_ep_create (Params_t *params_ptr, DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_fft_endpoint.c linux-kernel2/test/dapltest/test/dapl_fft_endpoint.c --- linux-kernel/test/dapltest/test/dapl_fft_endpoint.c 2005-05-12 09:28:43.503023000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_fft_endpoint.c 2005-05-12 11:03:25.510004000 -0700 @@ -59,7 +59,7 @@ int DT_endpoint_generic (Params_t *param evd_handle = DAT_HANDLE_NULL; phead = params_ptr->phead; - rc = dat_ia_open ((const DAT_NAME_PTR)dev_name, + rc = dat_ia_open ((const char *)dev_name, DEFAULT_QUEUE_LEN, &evd_handle, &ia_handle); DT_assert_dat (phead, rc == DAT_SUCCESS); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_fft_hwconn.c linux-kernel2/test/dapltest/test/dapl_fft_hwconn.c --- linux-kernel/test/dapltest/test/dapl_fft_hwconn.c 2005-05-12 09:28:43.633078000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_fft_hwconn.c 2005-05-12 11:03:25.515001000 -0700 @@ -45,7 +45,7 @@ int DT_hwconn_case0 ( Params_t *params_p nic_handle = NULL; evd_handle = DAT_HANDLE_NULL; - rc=dat_ia_open ((const DAT_NAME_PTR)dev_name, 10, &evd_handle, &nic_handle); + rc=dat_ia_open ((const char *)dev_name, 10, &evd_handle, &nic_handle); DT_assert_dat (phead, rc == DAT_SUCCESS); rc=dat_ia_close (nic_handle, DAT_CLOSE_ABRUPT_FLAG); @@ -95,7 +95,7 @@ int DT_hwconn_case1 ( Params_t *params_p } evd_handle = DAT_HANDLE_NULL; - rc=dat_ia_open ((const DAT_NAME_PTR)dev_name, 10, &evd_handle, &nic_handle); + rc=dat_ia_open ((const char *)dev_name, 10, &evd_handle, &nic_handle); if (DAT_GET_TYPE (rc) != DAT_PROVIDER_NOT_FOUND) { const char *major_msg, *minor_msg; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c linux-kernel2/test/dapltest/test/dapl_fft_queryinfo.c --- linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c 2005-05-12 09:28:43.523022000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_fft_queryinfo.c 2005-05-12 11:03:25.523003000 -0700 @@ -94,7 +94,7 @@ int DT_queryinfo_basic (Params_t *params dev_name = cmd->device_name; /* All functions require an ia_handle to be created */ - rc = dat_ia_open ((const DAT_NAME_PTR)dev_name, + rc = dat_ia_open ((const char *)dev_name, DEFAULT_QUEUE_LEN, &evd_handle, &ia_handle); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_fft_util.c linux-kernel2/test/dapltest/test/dapl_fft_util.c --- linux-kernel/test/dapltest/test/dapl_fft_util.c 2005-05-12 09:28:43.375003000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_fft_util.c 2005-05-12 11:02:59.887005000 -0700 @@ -55,7 +55,7 @@ void DT_assert_fail (DT_Tdep_Print_Head } /* helper function to open an IA */ -int DT_ia_open (DAT_NAME_PTR dev_name, DAT_IA_HANDLE *ia_handle) +int DT_ia_open (char *dev_name, DAT_IA_HANDLE *ia_handle) { DAT_EVD_HANDLE evd_handle; evd_handle = DAT_HANDLE_NULL; From halr at voltaire.com Thu May 12 11:26:39 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 12 May 2005 14:26:39 -0400 Subject: [openib-general] Re: Latest kdapl load problem In-Reply-To: <1115921994.9070.19.camel@duffman> References: <1115912597.4479.29.camel@localhost.localdomain> <1115913762.4479.40.camel@localhost.localdomain> <1115921994.9070.19.camel@duffman> Message-ID: <1115922398.4493.48.camel@localhost.localdomain> On Thu, 2005-05-12 at 14:19, Tom Duffy wrote: > On Thu, 2005-05-12 at 12:02 -0400, Hal Rosenstock wrote: > > On Thu, 2005-05-12 at 11:52, James Lentini wrote: > > > Did you rebuild the kdapltest module? > > > > Thanks. I forgot to do that. > > > > Does kdapltest build fine for you ? I'm getting some errors on building > > the user portion. > > Sorry, that was my fault. Forgot to check that my changes still allowed > user to build. It got fixed quickly. A minor hiccup. > > Modified: gen2/users/jlentini/linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.h > > =================================================================== > > --- gen2/users/jlentini/linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.h 2005-05-12 15:55:55 UTC (rev 2316) > > +++ gen2/users/jlentini/linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.h 2005-05-12 15:58:00 UTC (rev 2317) > > @@ -50,7 +50,15 @@ > > #endif > > #endif > > > > +typedef enum { > > +#undef FALSE > > + FALSE = 0, > > +#undef TRUE > > + TRUE = 1 > > +} boolean_t; > > > > +typedef uint32_t u32; > > + > > /* Default Device Name */ > > #define DT_MdepDeviceName "ia0a" > > > > @@ -72,7 +80,7 @@ > > BTW, u32 should not be used in user stuff. For that we should use the > standard uint32_t. Yup. -- Hal From jlentini at netapp.com Thu May 12 12:03:41 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 12 May 2005 15:03:41 -0400 (EDT) Subject: [openib-general] Re: Latest kdapl load problem In-Reply-To: <1115921994.9070.19.camel@duffman> References: <1115912597.4479.29.camel@localhost.localdomain> <1115913762.4479.40.camel@localhost.localdomain> <1115921994.9070.19.camel@duffman> Message-ID: tduffy> On Thu, 2005-05-12 at 12:02 -0400, Hal Rosenstock wrote: tduffy> > On Thu, 2005-05-12 at 11:52, James Lentini wrote: tduffy> > > Did you rebuild the kdapltest module? tduffy> > tduffy> > Thanks. I forgot to do that. tduffy> > tduffy> > Does kdapltest build fine for you ? I'm getting some errors on building tduffy> > the user portion. tduffy> tduffy> Sorry, that was my fault. Forgot to check that my changes still allowed tduffy> user to build. tduffy> tduffy> > Modified: gen2/users/jlentini/linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.h tduffy> > =================================================================== tduffy> > --- gen2/users/jlentini/linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.h 2005-05-12 15:55:55 UTC (rev 2316) tduffy> > +++ gen2/users/jlentini/linux-kernel/test/dapltest/mdep/linux/dapl_mdep_user.h 2005-05-12 15:58:00 UTC (rev 2317) tduffy> > @@ -50,7 +50,15 @@ tduffy> > #endif tduffy> > #endif tduffy> > tduffy> > +typedef enum { tduffy> > +#undef FALSE tduffy> > + FALSE = 0, tduffy> > +#undef TRUE tduffy> > + TRUE = 1 tduffy> > +} boolean_t; tduffy> > tduffy> > +typedef uint32_t u32; tduffy> > + tduffy> > /* Default Device Name */ tduffy> > #define DT_MdepDeviceName "ia0a" tduffy> > tduffy> > @@ -72,7 +80,7 @@ tduffy> tduffy> BTW, u32 should not be used in user stuff. For that we should use the tduffy> standard uint32_t. This is partially a result of the user and kernel portions of kdapltest being intertwined. Ultimately I'd like to see the directory structure of kdapltest updated to seperate the kernel and user code. I'll go through the user code and change the u32 to uint32_t. From sean.hefty at intel.com Thu May 12 12:10:01 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Thu, 12 May 2005 12:10:01 -0700 Subject: [openib-general] [PATCH] [CM] convert CM to use MAD send helper routines Message-ID: Converts the CM to make use of the newer MAD send helper data structure and helper routines. Signed-off-by: Sean Hefty Index: cm.c =================================================================== --- cm.c (revision 2320) +++ cm.c (working copy) @@ -86,15 +86,6 @@ struct cm_device { struct cm_port port[0]; }; -struct cm_msg { - struct cm_id_private *cm_id_priv; - struct ib_send_wr send_wr; - struct ib_sge sge; - enum ib_cm_state sent_state; - DECLARE_PCI_UNMAP_ADDR(mapping) - struct ib_mad mad; -}; - struct cm_av { struct cm_port *port; union ib_gid dgid; @@ -132,7 +123,7 @@ struct cm_id_private { wait_queue_head_t wait; atomic_t refcount; - struct cm_msg *msg; + struct ib_mad_send_buf *msg; struct cm_timewait_info *timewait_info; /* todo: use alternate port on send failure */ struct cm_av av; @@ -165,61 +156,40 @@ static inline void cm_deref_id(struct cm } static int cm_alloc_msg(struct cm_id_private *cm_id_priv, - struct cm_msg **msg) + struct ib_mad_send_buf **msg) { struct ib_mad_agent *mad_agent; - struct cm_msg *m; - int ret; - - m = kmalloc(sizeof *m, GFP_ATOMIC); - if (!m) - return -ENOMEM; - memset(m, 0, sizeof *m); + struct ib_mad_send_buf *m; + struct ib_ah *ah; mad_agent = cm_id_priv->av.port->mad_agent; - m->send_wr.wr.ud.ah = ib_create_ah(mad_agent->qp->pd, - &cm_id_priv->av.ah_attr); - if (IS_ERR(m->send_wr.wr.ud.ah)) { - ret = PTR_ERR(m->send_wr.wr.ud.ah); - kfree(m); - return ret; + ah = ib_create_ah(mad_agent->qp->pd, &cm_id_priv->av.ah_attr); + if (IS_ERR(ah)) + return PTR_ERR(ah); + + m = ib_create_send_mad(mad_agent, 1, cm_id_priv->av.pkey_index, + ah, 0, sizeof(struct ib_mad_hdr), + sizeof(struct ib_mad)-sizeof(struct ib_mad_hdr), + GFP_ATOMIC); + if (IS_ERR(m)) { + ib_destroy_ah(ah); + return PTR_ERR(m); } - m->sge.addr = dma_map_single(mad_agent->device->dma_device, - &m->mad, sizeof m->mad, - DMA_TO_DEVICE); - pci_unmap_addr_set(m, mapping, m->sge.addr); - m->sge.length = sizeof m->mad; - m->sge.lkey = cm_id_priv->av.port->mad_agent->mr->lkey; - - m->send_wr.wr_id = (unsigned long) m; - m->send_wr.sg_list = &m->sge; - m->send_wr.num_sge = 1; - m->send_wr.opcode = IB_WR_SEND; - m->send_wr.send_flags = IB_SEND_SIGNALED; - m->send_wr.wr.ud.mad_hdr = &m->mad.mad_hdr; - m->send_wr.wr.ud.remote_qpn = 1; - m->send_wr.wr.ud.remote_qkey = IB_QP1_QKEY; /* Timeout set by caller if response is expected. */ m->send_wr.wr.ud.retries = cm_id_priv->max_cm_retries; - m->send_wr.wr.ud.pkey_index = cm_id_priv->av.pkey_index; atomic_inc(&cm_id_priv->refcount); - m->cm_id_priv = cm_id_priv; + m->context[0] = cm_id_priv; *msg = m; return 0; } -static void cm_free_msg(struct cm_msg *msg) +static void cm_free_msg(struct ib_mad_send_buf *msg) { - struct ib_device *device; - ib_destroy_ah(msg->send_wr.wr.ud.ah); - device = msg->cm_id_priv->av.port->mad_agent->device; - dma_unmap_single(device->dma_device, pci_unmap_addr(msg, mapping), - sizeof msg->mad, DMA_TO_DEVICE); - cm_deref_id(msg->cm_id_priv); - kfree(msg); + cm_deref_id(msg->context[0]); + ib_free_send_mad(msg); } static void cm_set_ah_attr(struct ib_ah_attr *ah_attr, u8 port_num, @@ -852,10 +822,10 @@ int ib_send_cm_req(struct ib_cm_id *cm_i if (ret) goto out; - req_msg = (struct cm_req_msg *)&cm_id_priv->msg->mad; + req_msg = (struct cm_req_msg *) cm_id_priv->msg->mad; cm_format_req(req_msg, cm_id_priv, param); cm_id_priv->msg->send_wr.wr.ud.timeout_ms = cm_id_priv->timeout_ms; - cm_id_priv->msg->sent_state = IB_CM_REQ_SENT; + cm_id_priv->msg->context[1] = (void *) (unsigned long) IB_CM_REQ_SENT; cm_id_priv->local_qpn = cm_req_get_local_qpn(req_msg); cm_id_priv->rq_psn = cm_req_get_starting_psn(req_msg); @@ -1126,7 +1096,7 @@ int ib_send_cm_rep(struct ib_cm_id *cm_i struct ib_cm_rep_param *param) { struct cm_id_private *cm_id_priv; - struct cm_msg *msg; + struct ib_mad_send_buf *msg; struct cm_rep_msg *rep_msg; struct ib_send_wr *bad_send_wr; unsigned long flags; @@ -1143,10 +1113,10 @@ int ib_send_cm_rep(struct ib_cm_id *cm_i if (ret) goto out; - rep_msg = (struct cm_rep_msg *)&msg->mad; + rep_msg = (struct cm_rep_msg *) msg->mad; cm_format_rep(rep_msg, cm_id_priv, param); msg->send_wr.wr.ud.timeout_ms = cm_id_priv->timeout_ms; - msg->sent_state = IB_CM_REP_SENT; + msg->context[1] = (void *) (unsigned long) IB_CM_REP_SENT; spin_lock_irqsave(&cm_id_priv->lock, flags); if (cm_id->state == IB_CM_REQ_RCVD || @@ -1191,7 +1161,7 @@ static void cm_format_rtu(struct cm_rtu_ static void cm_resend_rtu(struct cm_id_private *cm_id_priv) { - struct cm_msg *msg; + struct ib_mad_send_buf *msg; struct ib_send_wr *bad_send_wr; int ret; @@ -1199,7 +1169,7 @@ static void cm_resend_rtu(struct cm_id_p if (ret) return; - cm_format_rtu((struct cm_rtu_msg *)&msg->mad, cm_id_priv, NULL, 0); + cm_format_rtu((struct cm_rtu_msg *) msg->mad, cm_id_priv, NULL, 0); ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent, &msg->send_wr, &bad_send_wr); if (ret) @@ -1211,7 +1181,7 @@ int ib_send_cm_rtu(struct ib_cm_id *cm_i u8 private_data_len) { struct cm_id_private *cm_id_priv; - struct cm_msg *msg; + struct ib_mad_send_buf *msg; struct ib_send_wr *bad_send_wr; unsigned long flags; int ret; @@ -1224,7 +1194,7 @@ int ib_send_cm_rtu(struct ib_cm_id *cm_i if (ret) goto out; - cm_format_rtu((struct cm_rtu_msg *)&msg->mad, cm_id_priv, + cm_format_rtu((struct cm_rtu_msg *) msg->mad, cm_id_priv, private_data, private_data_len); spin_lock_irqsave(&cm_id_priv->lock, flags); @@ -1467,7 +1437,7 @@ int ib_send_cm_dreq(struct ib_cm_id *cm_ u8 private_data_len) { struct cm_id_private *cm_id_priv; - struct cm_msg *msg; + struct ib_mad_send_buf *msg; struct ib_send_wr *bad_send_wr; unsigned long flags; int msg_ret, ret; @@ -1478,10 +1448,10 @@ int ib_send_cm_dreq(struct ib_cm_id *cm_ cm_id_priv = container_of(cm_id, struct cm_id_private, id); msg_ret = cm_alloc_msg(cm_id_priv, &msg); if (!msg_ret) { - cm_format_dreq((struct cm_dreq_msg *)&msg->mad, cm_id_priv, + cm_format_dreq((struct cm_dreq_msg *) msg->mad, cm_id_priv, private_data, private_data_len); msg->send_wr.wr.ud.timeout_ms = cm_id_priv->timeout_ms; - msg->sent_state = IB_CM_DREQ_SENT; + msg->context[1] = (void *) (unsigned long) IB_CM_DREQ_SENT; } spin_lock_irqsave(&cm_id_priv->lock, flags); @@ -1523,7 +1493,7 @@ static void cm_format_drep(struct cm_dre static void cm_resend_drep(struct cm_id_private *cm_id_priv) { - struct cm_msg *msg; + struct ib_mad_send_buf *msg; struct ib_send_wr *bad_send_wr; int ret; @@ -1531,7 +1501,7 @@ static void cm_resend_drep(struct cm_id_ if (ret) return; - cm_format_drep((struct cm_drep_msg *)&msg->mad, cm_id_priv, NULL, 0); + cm_format_drep((struct cm_drep_msg *) msg->mad, cm_id_priv, NULL, 0); ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent, &msg->send_wr, &bad_send_wr); if (ret) @@ -1543,7 +1513,7 @@ int ib_send_cm_drep(struct ib_cm_id *cm_ u8 private_data_len) { struct cm_id_private *cm_id_priv; - struct cm_msg *msg; + struct ib_mad_send_buf *msg; struct ib_send_wr *bad_send_wr; unsigned long flags; int msg_ret, ret; @@ -1554,7 +1524,7 @@ int ib_send_cm_drep(struct ib_cm_id *cm_ cm_id_priv = container_of(cm_id, struct cm_id_private, id); msg_ret = cm_alloc_msg(cm_id_priv, &msg); if (!msg_ret) - cm_format_drep((struct cm_drep_msg *)&msg->mad, cm_id_priv, + cm_format_drep((struct cm_drep_msg *) msg->mad, cm_id_priv, private_data, private_data_len); spin_lock_irqsave(&cm_id_priv->lock, flags); @@ -1711,7 +1681,7 @@ int ib_send_cm_rej(struct ib_cm_id *cm_i u8 private_data_len) { struct cm_id_private *cm_id_priv; - struct cm_msg *msg; + struct ib_mad_send_buf *msg; struct ib_send_wr *bad_send_wr; unsigned long flags; int msg_ret, ret; @@ -1723,7 +1693,7 @@ int ib_send_cm_rej(struct ib_cm_id *cm_i cm_id_priv = container_of(cm_id, struct cm_id_private, id); msg_ret = cm_alloc_msg(cm_id_priv, &msg); if (!msg_ret) - cm_format_rej((struct cm_rej_msg *) &msg->mad, cm_id_priv, + cm_format_rej((struct cm_rej_msg *) msg->mad, cm_id_priv, reason, ari, ari_length, private_data, private_data_len); @@ -1910,7 +1880,7 @@ int ib_send_cm_mra(struct ib_cm_id *cm_i u8 private_data_len) { struct cm_id_private *cm_id_priv; - struct cm_msg *msg; + struct ib_mad_send_buf *msg; struct ib_send_wr *bad_send_wr; unsigned long flags; int ret; @@ -1923,7 +1893,7 @@ int ib_send_cm_mra(struct ib_cm_id *cm_i if (ret) goto out; - cm_format_mra((struct cm_mra_msg *)&msg->mad, cm_id_priv, + cm_format_mra((struct cm_mra_msg *) msg->mad, cm_id_priv, service_timeout, private_data, private_data_len); spin_lock_irqsave(&cm_id_priv->lock, flags); @@ -2008,7 +1978,7 @@ int ib_send_cm_lap(struct ib_cm_id *cm_i u8 private_data_len) { struct cm_id_private *cm_id_priv; - struct cm_msg *msg; + struct ib_mad_send_buf *msg; struct ib_send_wr *bad_send_wr; unsigned long flags; int ret; @@ -2021,10 +1991,10 @@ int ib_send_cm_lap(struct ib_cm_id *cm_i if (ret) goto out; - cm_format_lap((struct cm_lap_msg *)&msg->mad, cm_id_priv, + cm_format_lap((struct cm_lap_msg *) msg->mad, cm_id_priv, alternate_path, private_data, private_data_len); msg->send_wr.wr.ud.timeout_ms = cm_id_priv->timeout_ms; - msg->sent_state = IB_CM_ESTABLISHED; + msg->context[1] = (void *) (unsigned long) IB_CM_ESTABLISHED; spin_lock_irqsave(&cm_id_priv->lock, flags); if (cm_id->state == IB_CM_ESTABLISHED && @@ -2144,7 +2114,7 @@ int ib_send_cm_apr(struct ib_cm_id *cm_i u8 private_data_len) { struct cm_id_private *cm_id_priv; - struct cm_msg *msg; + struct ib_mad_send_buf *msg; struct ib_send_wr *bad_send_wr; unsigned long flags; int ret; @@ -2158,7 +2128,7 @@ int ib_send_cm_apr(struct ib_cm_id *cm_i if (ret) goto out; - cm_format_apr((struct cm_apr_msg *)&msg->mad, cm_id_priv, status, + cm_format_apr((struct cm_apr_msg *) msg->mad, cm_id_priv, status, info, info_length, private_data, private_data_len); spin_lock_irqsave(&cm_id_priv->lock, flags); @@ -2284,7 +2254,7 @@ int ib_send_cm_sidr_req(struct ib_cm_id struct ib_cm_sidr_req_param *param) { struct cm_id_private *cm_id_priv; - struct cm_msg *msg; + struct ib_mad_send_buf *msg; struct ib_send_wr *bad_send_wr; unsigned long flags; int ret; @@ -2306,10 +2276,10 @@ int ib_send_cm_sidr_req(struct ib_cm_id if (ret) goto out; - cm_format_sidr_req((struct cm_sidr_req_msg *)&msg->mad, cm_id_priv, + cm_format_sidr_req((struct cm_sidr_req_msg *) msg->mad, cm_id_priv, param); msg->send_wr.wr.ud.timeout_ms = cm_id_priv->timeout_ms; - msg->sent_state = IB_CM_SIDR_REQ_SENT; + msg->context[1] = (void *) (unsigned long) IB_CM_SIDR_REQ_SENT; spin_lock_irqsave(&cm_id_priv->lock, flags); if (cm_id->state == IB_CM_IDLE) @@ -2427,7 +2397,7 @@ int ib_send_cm_sidr_rep(struct ib_cm_id struct ib_cm_sidr_rep_param *param) { struct cm_id_private *cm_id_priv; - struct cm_msg *msg; + struct ib_mad_send_buf *msg; struct ib_send_wr *bad_send_wr; unsigned long flags; int ret; @@ -2442,7 +2412,7 @@ int ib_send_cm_sidr_rep(struct ib_cm_id if (ret) goto out; - cm_format_sidr_rep((struct cm_sidr_rep_msg *)&msg->mad, cm_id_priv, + cm_format_sidr_rep((struct cm_sidr_rep_msg *) msg->mad, cm_id_priv, param); spin_lock_irqsave(&cm_id_priv->lock, flags); @@ -2514,23 +2484,25 @@ out: return -EINVAL; } -static void cm_process_send_error(struct cm_msg *msg, +static void cm_process_send_error(struct ib_mad_send_buf *msg, enum ib_wc_status wc_status) { struct cm_id_private *cm_id_priv; struct ib_cm_event cm_event; + enum ib_cm_state state; unsigned long flags; int ret; memset(&cm_event, 0, sizeof cm_event); - cm_id_priv = msg->cm_id_priv; + cm_id_priv = msg->context[0]; + state = (enum ib_cm_state) (unsigned long) msg->context[1]; /* Discard old sends or ones without a response. */ spin_lock_irqsave(&cm_id_priv->lock, flags); - if (msg != cm_id_priv->msg || msg->sent_state != cm_id_priv->id.state) + if (msg != cm_id_priv->msg || state != cm_id_priv->id.state) goto discard; - switch (cm_id_priv->id.state) { + switch (state) { case IB_CM_REQ_SENT: cm_reset_to_idle(cm_id_priv); cm_event.event = IB_CM_REQ_ERROR; @@ -2567,9 +2539,9 @@ discard: static void cm_send_handler(struct ib_mad_agent *mad_agent, struct ib_mad_send_wc *mad_send_wc) { - struct cm_msg *msg; + struct ib_mad_send_buf *msg; - msg = (struct cm_msg *)(unsigned long)mad_send_wc->wr_id; + msg = (struct ib_mad_send_buf *)(unsigned long)mad_send_wc->wr_id; switch (mad_send_wc->status) { case IB_WC_SUCCESS: From jlentini at netapp.com Thu May 12 12:46:19 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 12 May 2005 15:46:19 -0400 (EDT) Subject: [openib-general] Re: kdapltest works again!, still having slab corruption In-Reply-To: <1115832917.4490.128.camel@localhost.localdomain> References: <1115411060.4476.346.camel@localhost.localdomain> <1115413920.4476.1.camel@localhost.localdomain> <1115425669.19661.14.camel@duffman> <1115445245.4733.26.camel@localhost.localdomain> <1115481438.5263.4.camel@duffman> <1115650013.4490.16.camel@localhost.localdomain> <1115832762.11344.8.camel@duffman> <1115832917.4490.128.camel@localhost.localdomain> Message-ID: Hal, Did you reproduce this on x86 or x86_64? On Wed, 11 May 2005, Hal Rosenstock wrote: halr> On Wed, 2005-05-11 at 13:32, Tom Duffy wrote: halr> > Can you reproduce this on your end? I am still seeing it every time I halr> > run the kdapltest program on x86_64 with slab debugging turned on. With halr> > the latest bits (v2305). halr> halr> Yes, I can reproduce it and did spend some time looking at it but do not halr> have a workable theory on this right now. halr> halr> -- Hal halr> From halr at voltaire.com Thu May 12 12:49:27 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 12 May 2005 15:49:27 -0400 Subject: [openib-general] Re: kdapltest works again!, still having slab corruption In-Reply-To: References: <1115411060.4476.346.camel@localhost.localdomain> <1115413920.4476.1.camel@localhost.localdomain> <1115425669.19661.14.camel@duffman> <1115445245.4733.26.camel@localhost.localdomain> <1115481438.5263.4.camel@duffman> <1115650013.4490.16.camel@localhost.localdomain> <1115832762.11344.8.camel@duffman> <1115832917.4490.128.camel@localhost.localdomain> Message-ID: <1115927366.4493.119.camel@localhost.localdomain> On Thu, 2005-05-12 at 15:46, James Lentini wrote: > Hal, > > Did you reproduce this on x86 or x86_64? I did it on x86. I didn't try on x86_64 but could. -- Hal > > On Wed, 11 May 2005, Hal Rosenstock wrote: > > halr> On Wed, 2005-05-11 at 13:32, Tom Duffy wrote: > halr> > Can you reproduce this on your end? I am still seeing it every time I > halr> > run the kdapltest program on x86_64 with slab debugging turned on. With > halr> > the latest bits (v2305). > halr> > halr> Yes, I can reproduce it and did spend some time looking at it but do not > halr> have a workable theory on this right now. > halr> > halr> -- Hal > halr> From caitlin.bestler at gmail.com Thu May 12 12:56:07 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Thu, 12 May 2005 12:56:07 -0700 Subject: [openib-general] Re: [PATCHv2] kDAPL: remove DAT_RETURN In-Reply-To: <1115915608.9070.0.camel@duffman> References: <1115910070.25530.2.camel@duffman> <1115915608.9070.0.camel@duffman> Message-ID: <469958e00505121256cd8c04d@mail.gmail.com> DAT_RETURN serves two distinct purposes: it is an immediate return to a kernel specific caller, and it is a standardized error encoding for clients outside of the linux kernel (loadable modules and or proxied user consumers). Even if the functional return to in-kernel clients is converted to comply with kernel coding conventions, the information currently encoded in the DAT_RETURN needs to be deliverable to DAPL clients that are outside of the linux kernel. Therefore the current enums need to be retained, and there should be some mechanism for these precise diagnostics to be relayed to non-kernel clients. An external shim cannot do this *after* the information is lost by using a standard error return. (I am assuming that creating a list of error returns that encompass everything that DAT_RETURN *Can* encode is out of the question, there's a reason it's multi-layer now). One possibility would be to add an external_client_dat_return * out parameter that would allow a shim to restore the full DAT defined error code for non-kernel clients. On 5/12/05, Tom Duffy wrote: > On Thu, 2005-05-12 at 11:50 -0400, James Lentini wrote: > > Thanks for the updated patch Tom. It is committed in revision 2316. > > > > Eventually we will need to remove our dat_return_* enums and replace > > our return values with errno values. Correct? > > That is right. This is just a first pass at cleaning it up. > > -tduffy > > > BodyID:10548341.2.n.logpart (stored separately) > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > > From tduffy at sun.com Thu May 12 14:26:30 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 12 May 2005 14:26:30 -0700 Subject: [openib-general] Re: questions on how to create patches to linux (openib) In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35AD4@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35AD4@mtvex01.mtv.mtl.com> Message-ID: <1115933190.16144.21.camel@duffman> [CC'd the list to see if they know a better way for your last question] On Thu, 2005-05-12 at 23:56 +0300, Itamar Rabenstein wrote: > Hi Tom, > > I would like to ask some questions related to how to craete patches to Linux > (openib). > > As you can see in my last patch I have learned how to send the patch > direct from Linux shell and not from my outlook. > so it better from the first patch were is was warped my the mail client. Cool! Glad that works for you now. Now that you have made the first step (admitting you have a problem), you are on the path to dumping Outlook all together ;-) > My question is what is the rule of indentation in multiple parameters for > function I don't think Documentation/CodingStyle says anything specific about this, but you should use the convention that we use in the rest of openib. > 1) how many params in each line ? As many as can fit. Although if you are hitting the end of the line, it generally means either your function name is too long, the names of your parameters are too long, or you have too many parameters passing into the function. Remember that most architectures have a limited amount of registers reserved for function passing. Nobody is going to shoot you if you put one parameter per line (as long as all the parameters couldn't fit on one line). > 2) in the second line what is the indentation rule ? > for example: > > void my_function (int param1, int param2, > int param3); > > or > > void my_function (int param1, int param2, > int param3); > > or You almost have this right. What you should do is tab in as much as you can, then use spaces for the rest until the param3 lines up with param 1. For instance: void my_function(int param1, int param2, int param3, int param4, int param5 int param6); Also note the lack of a space between the function name and the open parenthesis. > void my_function (int param1, > int param2, > int param3); > > my second question is how do you create a set of patches were the second one > should be > apply on top of the first one. > when I try to craete it the second patch include the first. > how can I create the second patch if I want it to include only "new things" > on top of the first patch? subversion does not have an automagic way to do this, AFAIK. What I normally do is copy the whole directory to a new one and then diff -Nurp the two directories. When you start getting to many patches that all have dependencies/ordering on each other, subversion is not the tool to use. For that, I would recommend quilt or maybe git (haven't played with it much yet). -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From jlentini at netapp.com Thu May 12 14:31:42 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 12 May 2005 17:31:42 -0400 (EDT) Subject: [openib-general] Re: DAPL CM Reject Questions In-Reply-To: <42837BCC.20003@ichips.intel.com> References: <1115901591.4766.38.camel@localhost.localdomain> <42837BCC.20003@ichips.intel.com> Message-ID: > Hal Rosenstock wrote: >> On the passive side, an RTU is received but the QP cannot be transition >> to RTS so a REJ is attempted as below: >> >> static int do_rtu_recv(struct dapl_cm_id *conn) >> { >> int status; >> >> status = dapl_modify_qp_state_to_rts(conn->cm_id, >> conn->ep->qp_handle); >> if (status) { >> dapl_dbg_log(DAPL_DBG_TYPE_ERR, " do_rtu_recv: could not " >> "modify QP state to RTS status %d\n", status); >> goto reject; >> } >> ... >> >> reject: >> ib_send_cm_rej(conn->cm_id, IB_CM_REJ_CONSUMER_DEFINED, NULL, 0, >> NULL, 0); > > This should call ib_send_cm_dreq(), rather than a reject. I've changed the above line to (void) ib_send_cm_dreq(conn->cm_id, NULL, 0); in revision 2329. >> If RTU is received, the connection would be in ESTABLISHED state. Is >> sending REJ valid there ? What's odd to me is that the CM state machine >> in the IBA spec shows REJ being received in the established state on the >> active side (and moving to TIMEWAIT) but there is no action out of >> established on the passive side showing send REJ. > > On the passive side, the user can time out waiting for an RTU and issue a > reject. But the RTU may have been sent (and lost) on the active side. So, > the active side would be in the ESTABLISHED state when the REJ is received. > > This cannot happen on the passive side. > >> If sending REJ is valid on the passive side in established state, does >> the current CM handle this ? It looks to me like it returns -EINVAL for >> this case. > > See above... I think that the CM code is correct, and my kDAPL > implementation is in error. Were either of you going to work on a fix for this? From sean.hefty at intel.com Thu May 12 14:35:50 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Thu, 12 May 2005 14:35:50 -0700 Subject: [openib-general] Re: DAPL CM Reject Questions In-Reply-To: Message-ID: >> This should call ib_send_cm_dreq(), rather than a reject. > >I've changed the above line to > >(void) ib_send_cm_dreq(conn->cm_id, NULL, 0); > >in revision 2329. >> See above... I think that the CM code is correct, and my kDAPL >> implementation is in error. > >Were either of you going to work on a fix for this? It sounds like you just fixed this. Was there something else that needed to be done? - Sean From libor at topspin.com Thu May 12 14:15:54 2005 From: libor at topspin.com (Libor Michalek) Date: Thu, 12 May 2005 14:15:54 -0700 Subject: [openib-general] [PATCH][SDP] address for fmr must be page aligned Message-ID: <20050512141554.B23734@topspin.com> The IO address passed ib_fmr_pool_map_phys must be page aligned, here is a patch to make AIO with unaligned buffers work. By default ttcp.aio.c aligns buffers on page boundaries, which is why it was not noticed before. Signed-off-by: Libor Michalek Index: sdp_iocb.c =================================================================== --- sdp_iocb.c (revision 2326) +++ sdp_iocb.c (working copy) @@ -432,12 +432,12 @@ int sdp_iocb_register(struct sdpc_iocb * /* * prime io address with physical address of first byte? */ - iocb->io_addr = iocb->addr_array[0] + iocb->page_offset; + iocb->io_addr = iocb->addr_array[0]; /* * register IOCBs physical memory */ iocb->mem = ib_fmr_pool_map_phys(conn->fmr_pool, - (u64 *)iocb->addr_array, + iocb->addr_array, iocb->page_count, &iocb->io_addr); if (IS_ERR(iocb->mem)) { @@ -458,6 +458,7 @@ int sdp_iocb_register(struct sdpc_iocb * * some data may have already been consumed, adjust the io address * to take this into account */ + iocb->io_addr += iocb->page_offset; iocb->io_addr += iocb->post; iocb->flags |= SDP_IOCB_F_REG; From libor at topspin.com Thu May 12 14:13:04 2005 From: libor at topspin.com (Libor Michalek) Date: Thu, 12 May 2005 14:13:04 -0700 Subject: [openib-general] Re: [PATCH] sdpc_buff trivial changes In-Reply-To: <20050510100021.GA2474@mellanox.co.il>; from mst@mellanox.co.il on Tue, May 10, 2005 at 01:00:21PM +0300 References: <20050510100021.GA2474@mellanox.co.il> Message-ID: <20050512141304.A23734@topspin.com> On Tue, May 10, 2005 at 01:00:21PM +0300, Michael S. Tsirkin wrote: > struct sdpc_buff has a copy of ib_sge and uses casts to turn that into > a real ib_sge. The right thing is obviously to keep ib_sge > as part of sdpc_buff. Thanks, applied and checked in. -Libor From jlentini at netapp.com Thu May 12 14:42:04 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 12 May 2005 17:42:04 -0400 (EDT) Subject: [openib-general] Re: DAPL CM Reject Questions In-Reply-To: References: Message-ID: On Thu, 12 May 2005, Sean Hefty wrote: >>> This should call ib_send_cm_dreq(), rather than a reject. >> >> I've changed the above line to >> >> (void) ib_send_cm_dreq(conn->cm_id, NULL, 0); >> >> in revision 2329. > > >>> See above... I think that the CM code is correct, and my kDAPL >>> implementation is in error. >> >> Were either of you going to work on a fix for this? > > > It sounds like you just fixed this. Was there something else that > needed to be done? I misread your earlier mail. I thought that there was more to do than just the one liner. From mshefty at ichips.intel.com Thu May 12 14:43:12 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Thu, 12 May 2005 14:43:12 -0700 Subject: [openib-general] Re: questions on how to create patches to linux (openib) In-Reply-To: <1115933190.16144.21.camel@duffman> References: <91DB792C7985D411BEC300B40080D29CC35AD4@mtvex01.mtv.mtl.com> <1115933190.16144.21.camel@duffman> Message-ID: <4283CDF0.4030607@ichips.intel.com> Tom Duffy wrote: >>As you can see in my last patch I have learned how to send the patch >>direct from Linux shell and not from my outlook. >>so it better from the first patch were is was warped my the mail client. You can avoid wrapping (and breaking threading) in Outlook if you connect to your Exchange server using IMAP, and send using that. - Sean From tduffy at sun.com Thu May 12 14:55:50 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 12 May 2005 14:55:50 -0700 Subject: [openib-general] Re: DAPL CM Reject Questions In-Reply-To: References: <1115901591.4766.38.camel@localhost.localdomain> <42837BCC.20003@ichips.intel.com> Message-ID: <1115934950.16144.22.camel@duffman> On Thu, 2005-05-12 at 17:31 -0400, James Lentini wrote: > I've changed the above line to > > (void) ib_send_cm_dreq(conn->cm_id, NULL, 0); Is there a reason for the leading (void)? -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From surs at cse.ohio-state.edu Thu May 12 15:18:04 2005 From: surs at cse.ohio-state.edu (Sayantan Sur) Date: Thu, 12 May 2005 18:18:04 -0400 Subject: [openib-general] inline support in Gen2 Message-ID: <20050512221803.GA14851@cse.ohio-state.edu> Hi Roland, We had a discussion a few weeks back about the absence of "query_qp" verbs in Gen2. You had indicated that on Gen2, the `ibv_create_qp' itself can be used to pass back the max value of inline data size. The old thread is here: http://openib.org/pipermail/openib-general/2005-April/005594.html I was wondering whether this feature was incorporated? If it was, then what verb do I use to use this inline feature? (I couldn't find any query_qp verbs in Gen2 as of today's code). By the way, we were looking for equivalents to these VAPI calls, but couldn't find them on Gen2. 1) attach QP to multicast group ( VAPI_attach_to_multicast ) 2) detach QP from multicast ( VAPI_detach_from_multicast ) How does a user process use IB hardware multicast using Gen2 verbs? Thanks, Sayantan. -- http://www.cse.ohio-state.edu/~surs From libor at topspin.com Thu May 12 15:10:51 2005 From: libor at topspin.com (Libor Michalek) Date: Thu, 12 May 2005 15:10:51 -0700 Subject: [openib-general] Re: [PATCH] (repost) sdp: replace mlock with get_user_pages In-Reply-To: <20050510143232.GF2474@mellanox.co.il>; from mst@mellanox.co.il on Tue, May 10, 2005 at 05:32:32PM +0300 References: <20050505110158.GH30653@mellanox.co.il> <20050506175415.I26823@topspin.com> <20050510143232.GF2474@mellanox.co.il> Message-ID: <20050512151051.C23734@topspin.com> On Tue, May 10, 2005 at 05:32:32PM +0300, Michael S. Tsirkin wrote: > Quoting r. Libor Michalek : > > On Thu, May 05, 2005 at 02:01:58PM +0300, Michael S. Tsirkin wrote: > > > > > > The following patch replaces the mlock hack with call > > > to get_user_pages. Since the application could have forked > > > while an iocb is outstanding, when an iocb is done > > > I do get_user_pages for a second time and copy data if > > > the physical address has changed. > > > > > > Thus, changing ulimit is no longer required to get aio > > > working, processes are also allowed to fork and to call mlock/munlock > > > on the buffer. > > > > In the latest kernel what happens to a page that's held > > with a get_user_page reference on fork? Which is the only > > case left. Since there is an iocv flag parameter it would > > be prefered to use a bit there instead of is_receive. Also, > > please leave spaces around '='. Finally it would be nice > > if we could get rid of having to schedule a task to unlock > > the iocb in complete, I've noticed this having a negative > > effect on system CPU utilization on older kernels. > > I dont see what would prevent the page > from being copied on a COW event. Do you? Certainly the patch in 2.6.7 > that fixed the swap-out problem does not affect COW. > With regard to avoiding task scheduling, since current code already > needs this, so we are not breaking anything. > I put in comments so that we remember to do this research at some point > in time. > > In the patch below the style issues are addressed: whitespace fixed, > is_receive changed to flag. Michael, I'm seeing a SDP bug and the only solution I've come up with so far conflicts directly with your patch. The problem. When using ttcp.aio.c with buffers below the zcopy threshold, multiple aio buffers outstanding, and verification turned on either send and/or receive side very quickly gets a mismatch error. The mismatch error occurs when the order of buffers returned by io_getevents() differs from the order in which they are passed to io_submit(). Here's the params I was using: ./ttcp.aio.x -r -s -l 3276 -f m -a 20 -v ./ttcp.aio.x -t -s -l 3276 -n 50000 -a 20 -f m -v 192.168.0.191 The issue. There is a race condition between completing IOs synchronously and asynchronously using aio_complete(). The race condition is the result of iocb_complete() spawning a work thread to unlock memory and call aio_complete(). It's possible for a new IO to get serviced synchronously before the do_iocb_complete() task executes which means that the two IOs will appear on the event queue in the reverse order of being submitted. Ths solution. Always call aio_complete() immediately when iocb_complete() is called, and only spawn the work thread to unlock the memory after the aio_complete() call. The patch is below. The problem is that your patch relies on access to the page tables occurs before the aio_complete() call. Which is the reverse of what is in the patch, and is incompatible with calling aio_complete before scheduling a task... -Libor Index: sdp_send.c =================================================================== --- sdp_send.c (revision 2326) +++ sdp_send.c (working copy) @@ -2059,7 +2059,8 @@ int sdp_inet_send(struct kiocb *req, str * they are smaller then the zopy threshold, but only if there is * no buffer write space. */ - if (!(conn->src_zthresh > size) && !is_sync_kiocb(req)) + if (!is_sync_kiocb(req) && (sdp_desc_q_size(&conn->send_queue) || + !(conn->src_zthresh > size))) goto skip; /* * clear ASYN space bit, it'll be reset if there is no space. Index: sdp_iocb.c =================================================================== --- sdp_iocb.c (revision 2328) +++ sdp_iocb.c (working copy) @@ -496,14 +496,6 @@ static void do_iocb_complete(void *arg) { struct sdpc_iocb *iocb = (struct sdpc_iocb *)arg; int result; - long value; - /* - * release memory - */ - result = sdp_iocb_release(iocb); - if (result < 0) - sdp_dbg_err("Error <%d> releasing IOCB <%d> memory <%ld>", - result, iocb->key, iocb->addr); /* * unlock now, after aio_complete the mm reference will be released. */ @@ -512,25 +504,12 @@ static void do_iocb_complete(void *arg) sdp_dbg_err("Error <%d> unlocking IOCB <%d memory <%ld>>", result, iocb->key, iocb->addr); /* - * callback to complete IOCB - */ - value = (iocb->post > 0) ? iocb->post : iocb->status; - - sdp_dbg_data(NULL, "IOCB complete. <%d:%d:%08lx> value <%ld>", - iocb->req->ki_users, iocb->req->ki_key, - iocb->req->ki_flags, value); - /* - * valid result can be 0 or 1 for complete so - * we ignore the value. - */ - (void)aio_complete(iocb->req, value, 0); - /* * delete IOCB */ result = sdp_iocb_destroy(iocb); if (result < 0) - sdp_dbg_err("Error <%d> deleting IOCB <%d> of status <%Zu>", - result, iocb->key, iocb->status); + sdp_dbg_err("Error <%d> deleting IOCB <%d:%08lx>", + result, iocb->key, iocb->addr); } /* @@ -538,7 +517,28 @@ static void do_iocb_complete(void *arg) */ int sdp_iocb_complete(struct sdpc_iocb *iocb, ssize_t status) { - iocb->status = status; + int result; + long value; + /* + * release memory + */ + result = sdp_iocb_release(iocb); + if (result < 0) + sdp_dbg_err("Error <%d> releasing IOCB <%d> memory <%ld>", + result, iocb->key, iocb->addr); + /* + * callback to complete IOCB + */ + value = (iocb->post > 0) ? iocb->post : status; + + sdp_dbg_data(NULL, "IOCB complete. <%d:%d:%08lx> value <%08lx:%ld>", + iocb->req->ki_users, iocb->req->ki_key, + iocb->req->ki_flags, iocb->addr, value); + /* + * valid result can be 0 or 1 for complete so + * we ignore the value. + */ + (void)aio_complete(iocb->req, value, 0); if (in_atomic() || irqs_disabled()) { INIT_WORK(&iocb->completion, do_iocb_complete, (void *)iocb); Index: sdp_iocb.h =================================================================== --- sdp_iocb.h (revision 2326) +++ sdp_iocb.h (working copy) @@ -88,7 +88,6 @@ struct sdpc_iocb { int len; /* space left in the user buffer */ int post; /* amount of data requested so far. */ u64 wrid; /* work request completing this IOCB */ - ssize_t status; /* status of completed iocb */ /* * IB specific information for zcopy. */ From halr at voltaire.com Thu May 12 15:40:18 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 12 May 2005 18:40:18 -0400 Subject: [openib-general] inline support in Gen2 In-Reply-To: <20050512221803.GA14851@cse.ohio-state.edu> References: <20050512221803.GA14851@cse.ohio-state.edu> Message-ID: <1115937617.4479.6.camel@localhost.localdomain> On Thu, 2005-05-12 at 18:18, Sayantan Sur wrote: > By the way, we were looking for equivalents to these VAPI calls, but > couldn't find them on Gen2. > > 1) attach QP to multicast group ( VAPI_attach_to_multicast ) > 2) detach QP from multicast ( VAPI_detach_from_multicast ) > > How does a user process use IB hardware multicast using Gen2 verbs? These are on the list to be added to gen2. -- Hal From libor at topspin.com Thu May 12 15:48:57 2005 From: libor at topspin.com (Libor Michalek) Date: Thu, 12 May 2005 15:48:57 -0700 Subject: [PATCH] Re: [openib-general] 0 op factor In-Reply-To: <20050510174700.A5174@topspin.com>; from libor@topspin.com on Tue, May 10, 2005 at 05:47:00PM -0700 References: <20050505155803.GS15688@aon.at> <20050505172122.B26428@topspin.com> <20050507174718.GQ15688@aon.at> <20050510174700.A5174@topspin.com> Message-ID: <20050512154857.D23734@topspin.com> On Tue, May 10, 2005 at 05:47:00PM -0700, Libor Michalek wrote: > On Sat, May 07, 2005 at 07:47:18PM +0200, Bernhard Fischer wrote: > > > - remove expect from _sdp_cm_path_complete(). > > When I said that the SDP_EXPECT should eventually be removed, I meant > that the functions which have their return values checked by SDP_EXPECT > should either be turned into void return functions, or something > intelligent should be done with the return value, such as error recovery > or propogation. The functions which should be turned into void functions > are the ones which will never return anything but success. To expand on the last point. There are a lot of functions, for example those in sdp_buff.c, which check for incorrect function usage, such as checking that a buffer is not already in a queue before inserting it into a queue. These checks could be removed entirely or the checks could be made and a call to BUG() in case the condition is met. I prefer the later since the former would corrupt the entire queue. I'm not sure which is preferable to everyone else, either way the result is that the function becomes a void. -Libor From roland at topspin.com Thu May 12 16:14:22 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 12 May 2005 16:14:22 -0700 Subject: [openib-general] Re: inline support in Gen2 In-Reply-To: <20050512221803.GA14851@cse.ohio-state.edu> (Sayantan Sur's message of "Thu, 12 May 2005 18:18:04 -0400") References: <20050512221803.GA14851@cse.ohio-state.edu> Message-ID: <52br7gt4tt.fsf@topspin.com> Sayantan> Hi Roland, We had a discussion a few weeks back about Sayantan> the absence of "query_qp" verbs in Gen2. You had Sayantan> indicated that on Gen2, the `ibv_create_qp' itself can Sayantan> be used to pass back the max value of inline data size. I have patches from Michael Tsirkin that I am integrating now, which will set the value in the init_attr->cap.max_inline_data returned from the ibv_create_qp() verb. This should be done by tomorrow. Sayantan> 1) attach QP to multicast group ( Sayantan> VAPI_attach_to_multicast ) 2) detach QP from multicast ( Sayantan> VAPI_detach_from_multicast ) Sayantan> How does a user process use IB hardware multicast using Sayantan> Gen2 verbs? This still needs to be implemented... there will be functions int ibv_attach_mcast(struct ibv_qp *qp, union ibv_gid *gid, uint16_t lid); int ibv_detach_mcast(struct ibv_qp *qp, union ibv_gid *gid, uint16_t lid); these should also be done soon. Thanks, Roland From roland at topspin.com Thu May 12 16:11:12 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 12 May 2005 16:11:12 -0700 Subject: [openib-general] Re: [PATCH] (repost) sdp: replace mlock with get_user_pages In-Reply-To: <20050512151051.C23734@topspin.com> (Libor Michalek's message of "Thu, 12 May 2005 15:10:51 -0700") References: <20050505110158.GH30653@mellanox.co.il> <20050506175415.I26823@topspin.com> <20050510143232.GF2474@mellanox.co.il> <20050512151051.C23734@topspin.com> Message-ID: <52fywst4z3.fsf@topspin.com> Libor> Always call aio_complete() immediately when Libor> iocb_complete() is called, and only spawn the work thread Libor> to unlock the memory after the aio_complete() call. The Libor> patch is below. Another alternative would be to always complete aios asynchronously, which should preserve the order. I guess this would hurt latency for small ios... - R. From surs at cse.ohio-state.edu Thu May 12 16:58:05 2005 From: surs at cse.ohio-state.edu (Sayantan Sur) Date: Thu, 12 May 2005 19:58:05 -0400 Subject: [openib-general] Re: inline support in Gen2 In-Reply-To: <52br7gt4tt.fsf@topspin.com> References: <20050512221803.GA14851@cse.ohio-state.edu> <52br7gt4tt.fsf@topspin.com> Message-ID: <4283ED8D.2020101@cse.ohio-state.edu> Roland & Hal, Roland Dreier wrote: > Sayantan> Hi Roland, We had a discussion a few weeks back about > Sayantan> the absence of "query_qp" verbs in Gen2. You had > Sayantan> indicated that on Gen2, the `ibv_create_qp' itself can > Sayantan> be used to pass back the max value of inline data size. > > I have patches from Michael Tsirkin that I am integrating now, which > will set the value in the init_attr->cap.max_inline_data returned from > the ibv_create_qp() verb. This should be done by tomorrow. > > Sayantan> 1) attach QP to multicast group ( > Sayantan> VAPI_attach_to_multicast ) 2) detach QP from multicast ( > Sayantan> VAPI_detach_from_multicast ) > > Sayantan> How does a user process use IB hardware multicast using > Sayantan> Gen2 verbs? > > This still needs to be implemented... there will be functions > > int ibv_attach_mcast(struct ibv_qp *qp, union ibv_gid *gid, uint16_t lid); > int ibv_detach_mcast(struct ibv_qp *qp, union ibv_gid *gid, uint16_t lid); > > these should also be done soon. Thank you for the information. Looking forward to having these features in Gen2. Thanks, Sayantan. > > Thanks, > Roland > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general -- http://www.cse.ohio-state.edu/~surs From jlentini at netapp.com Thu May 12 19:00:47 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 12 May 2005 22:00:47 -0400 (EDT) Subject: [openib-general] [PATCH] make private data pointers const Message-ID: This patch affects ib_cm.h and cm.c, which appear to be maintained by Sean, and ucm.c, which appears to be maintained by Libor. The patch adds the const qualifier to private data pointers passed into the IB layer. Signed-off-by: James Lentini Index: include/ib_cm.h =================================================================== --- include/ib_cm.h (revision 2325) +++ include/ib_cm.h (working copy) @@ -333,7 +333,7 @@ u32 qp_num; enum ib_qp_type qp_type; u32 starting_psn; - void *private_data; + const void *private_data; u8 private_data_len; u8 peer_to_peer; u8 responder_resources; @@ -360,7 +360,7 @@ struct ib_cm_rep_param { u32 qp_num; u32 starting_psn; - void *private_data; + const void *private_data; u8 private_data_len; u8 responder_resources; u8 initiator_depth; @@ -391,7 +391,7 @@ * @private_data_len: Size of the private data buffer, in bytes. */ int ib_send_cm_rtu(struct ib_cm_id *cm_id, - void *private_data, + const void *private_data, u8 private_data_len); /** @@ -404,7 +404,7 @@ * @private_data_len: Size of the private data buffer, in bytes. */ int ib_send_cm_dreq(struct ib_cm_id *cm_id, - void *private_data, + const void *private_data, u8 private_data_len); /** @@ -416,7 +416,7 @@ * @private_data_len: Size of the private data buffer, in bytes. */ int ib_send_cm_drep(struct ib_cm_id *cm_id, - void *private_data, + const void *private_data, u8 private_data_len); /** @@ -444,7 +444,7 @@ enum ib_cm_rej_reason reason, void *ari, u8 ari_length, - void *private_data, + const void *private_data, u8 private_data_len); /** @@ -459,7 +459,7 @@ */ int ib_send_cm_mra(struct ib_cm_id *cm_id, u8 service_timeout, - void *private_data, + const void *private_data, u8 private_data_len); /** @@ -474,7 +474,7 @@ */ int ib_send_cm_lap(struct ib_cm_id *cm_id, struct ib_sa_path_rec *alternate_path, - void *private_data, + const void *private_data, u8 private_data_len); /** @@ -513,14 +513,14 @@ enum ib_cm_apr_status status, void *info, u8 info_length, - void *private_data, + const void *private_data, u8 private_data_len); struct ib_cm_sidr_req_param { struct ib_sa_path_rec *path; u64 service_id; int timeout_ms; - void *private_data; + const void *private_data; u8 private_data_len; u8 max_cm_retries; u16 pkey; @@ -540,9 +540,9 @@ u32 qp_num; u32 qkey; enum ib_cm_sidr_status status; - void *info; + const void *info; u8 info_length; - void *private_data; + const void *private_data; u8 private_data_len; }; Index: core/cm.c =================================================================== --- core/cm.c (revision 2331) +++ core/cm.c (working copy) @@ -1146,7 +1146,7 @@ static void cm_format_rtu(struct cm_rtu_msg *rtu_msg, struct cm_id_private *cm_id_priv, - void *private_data, + const void *private_data, u8 private_data_len) { cm_format_mad_hdr(&rtu_msg->hdr, cm_id_priv, @@ -1177,7 +1177,7 @@ } int ib_send_cm_rtu(struct ib_cm_id *cm_id, - void *private_data, + const void *private_data, u8 private_data_len) { struct cm_id_private *cm_id_priv; @@ -1418,7 +1418,7 @@ static void cm_format_dreq(struct cm_dreq_msg *dreq_msg, struct cm_id_private *cm_id_priv, - void *private_data, + const void *private_data, u8 private_data_len) { cm_format_mad_hdr(&dreq_msg->hdr, cm_id_priv, @@ -1433,7 +1433,7 @@ } int ib_send_cm_dreq(struct ib_cm_id *cm_id, - void *private_data, + const void *private_data, u8 private_data_len) { struct cm_id_private *cm_id_priv; @@ -1478,7 +1478,7 @@ static void cm_format_drep(struct cm_drep_msg *drep_msg, struct cm_id_private *cm_id_priv, - void *private_data, + const void *private_data, u8 private_data_len) { cm_format_mad_hdr(&drep_msg->hdr, cm_id_priv, @@ -1509,7 +1509,7 @@ } int ib_send_cm_drep(struct ib_cm_id *cm_id, - void *private_data, + const void *private_data, u8 private_data_len) { struct cm_id_private *cm_id_priv; @@ -1640,7 +1640,7 @@ enum ib_cm_rej_reason reason, void *ari, u8 ari_length, - void *private_data, + const void *private_data, u8 private_data_len) { cm_format_mad_hdr(&rej_msg->hdr, cm_id_priv, @@ -1677,7 +1677,7 @@ enum ib_cm_rej_reason reason, void *ari, u8 ari_length, - void *private_data, + const void *private_data, u8 private_data_len) { struct cm_id_private *cm_id_priv; @@ -1836,7 +1836,7 @@ static void cm_format_mra(struct cm_mra_msg *mra_msg, struct cm_id_private *cm_id_priv, u8 service_timeout, - void *private_data, + const void *private_data, u8 private_data_len) { enum cm_msg_sequence msg_sequence; @@ -1876,7 +1876,7 @@ int ib_send_cm_mra(struct ib_cm_id *cm_id, u8 service_timeout, - void *private_data, + const void *private_data, u8 private_data_len) { struct cm_id_private *cm_id_priv; @@ -1944,7 +1944,7 @@ static void cm_format_lap(struct cm_lap_msg *lap_msg, struct cm_id_private *cm_id_priv, struct ib_sa_path_rec *alternate_path, - void *private_data, + const void *private_data, u8 private_data_len) { cm_format_mad_hdr(&lap_msg->hdr, cm_id_priv, @@ -1974,7 +1974,7 @@ int ib_send_cm_lap(struct ib_cm_id *cm_id, struct ib_sa_path_rec *alternate_path, - void *private_data, + const void *private_data, u8 private_data_len) { struct cm_id_private *cm_id_priv; @@ -2087,7 +2087,7 @@ enum ib_cm_apr_status status, void *info, u8 info_length, - void *private_data, + const void *private_data, u8 private_data_len) { cm_format_mad_hdr(&apr_msg->hdr, cm_id_priv, @@ -2110,7 +2110,7 @@ enum ib_cm_apr_status status, void *info, u8 info_length, - void *private_data, + const void *private_data, u8 private_data_len) { struct cm_id_private *cm_id_priv; Index: core/ucm.c =================================================================== --- core/ucm.c (revision 2331) +++ core/ucm.c (working copy) @@ -721,7 +721,7 @@ return result; } -static int ib_ucm_alloc_data(void **dest, u64 src, u32 len) +static int ib_ucm_alloc_data(const void **dest, u64 src, u32 len) { void *data; @@ -913,12 +913,12 @@ static ssize_t ib_ucm_send_private_data(struct ib_ucm_file *file, const char __user *inbuf, int in_len, int (*func)(struct ib_cm_id *cm_id, - void *private_data, + const void *private_data, u8 private_data_len)) { struct ib_ucm_private_data cmd; struct ib_ucm_context *ctx; - void *private_data = NULL; + const void *private_data = NULL; int result; if (copy_from_user(&cmd, inbuf, sizeof(cmd))) @@ -974,15 +974,15 @@ const char __user *inbuf, int in_len, int (*func)(struct ib_cm_id *cm_id, int status, - void *info, + const void *info, u8 info_len, - void *data, + const void *data, u8 data_len)) { struct ib_ucm_context *ctx; struct ib_ucm_info cmd; - void *data = NULL; - void *info = NULL; + const void *data = NULL; + const void *info = NULL; int result; if (copy_from_user(&cmd, inbuf, sizeof(cmd))) @@ -1041,7 +1041,7 @@ { struct ib_ucm_context *ctx; struct ib_ucm_mra cmd; - void *data = NULL; + const void *data = NULL; int result; if (copy_from_user(&cmd, inbuf, sizeof(cmd))) @@ -1080,7 +1080,7 @@ struct ib_ucm_context *ctx; struct ib_sa_path_rec *path = NULL; struct ib_ucm_lap cmd; - void *data = NULL; + const void *data = NULL; int result; if (copy_from_user(&cmd, inbuf, sizeof(cmd))) From FutpCfs at hotmail.com Thu May 12 16:26:38 2005 From: FutpCfs at hotmail.com (Maryann Sophia) Date: Fri, 13 May 2005 02:26:38 +0300 Subject: [openib-general] PPV for free! Message-ID: <20030301153818.GA16823@gala .eu.org> Unlock Digital Cable *PPV channels *Boxing and any other Sports Event on PPV *Adult Channels *Movies *And anything else on PPV channels. http://JZv6.1clickresults.info This is the NEWEST AND BEST Digital CATV Filter/Descrambler that can test your digital cable PPV purchase functions along with eliminating unwanted interference caused by your broadband connection. This is a "True" universal product. It will work on 99% of all digital cable systems in use today. http://D7jG.1clickresults.info This little filter has many different uses: Interference filter, PPV tester and "Digital Descrambler". These will work with the following receivers: Scientific Atlanta Jerrold/General Instruments Zenith Tocom Starcom Explorer Series Motorola http://1IgZ.1clickresults.info Simple to Install. ATTENTION: Because the Cable Company has no way of telling you are using this product you need to notify them of any movie purchases. If you are getting interference on your television set while connected to your broadband connection,simply installing this device prior to your cable converter will dramatically improve your reception. http://AJN7.1clickresults.info Try us out! http://wIlt.1clickresults.info Regards, Maryann Sophia From gamboge at doneasy.com Thu May 12 21:42:39 2005 From: gamboge at doneasy.com (Matt Bourgeois) Date: Fri, 13 May 2005 08:42:39 +0400 Subject: [openib-general] Lowest rates in 45 years Message-ID: <118441.2414.gamboge@doneasy.com> Hello, We tried contacting you awhile ago about your low interest morta(ge rate. You have qualified for the lowest rate in years... You could get over $380,000 for as little as $500 a month! Ba(d credit? Doesn't matter, low rates are fixed no matter what! To get a free, no obli,gation consultation click below: http://www.ch00s3.com/sign.asp Best Regards, Terry Schaefer to be remov(ed: http://www.ch00s3.com/gone.asp this process takes one week, so please be patient. we do our best to take your email/s off but you have to fill out a rem/ove or else you will continue to recieve email/s. From info at susukino-5you.net Thu May 12 06:59:45 2005 From: info at susukino-5you.net (info at susukino-5you.net) Date: 12 May 2005 22:59:45 +0900 Subject: [openib-general] $B?75,EPO?$NJ}$O(B10000$B1_J,L5NA(B Message-ID: <20050512135945.19117.qmail@mail.susukino-5you.net> $B0lK|1_J,!*40A4L5NA$G;HMQ(BOK $B"#:#$J$i4V$K9g$&(BW$B%A%c%s%9!*L5NA%]%$%s%H%"%C%W3Z$7$5(B100$BG\!*"#(B $B"(EPO?=w at -2q0w$N%8%c%s%kJL?M5$%i%s%-%s%0(B $B0l0L!'5U1g!J(B189$BL>EPO?!K(B $BFs0L!'1g8r!J(B429$BL>EPO?!K(B $B;00L!'(BSM$B5U1g!J(B243$BL>EPO?!K(B $B;M0L!'%;%U%l!J(B1038$BL>EPO?!K(B $B8^0L!'ITNQ!J(B2421$BL>EPO?!K(B $BO;0L!'%F%l%(%C%A!J(B3463$BL>EPO?!K(B $B!&(B $B!&(B $B!&(B $B0lK|1_J,40A4L5NA$G$*;n$7$O(B $B"-"-"-(B http://awg.webchu.com/?summer27 $BB~:#!*?75,EPO?$7$?J}$K$OL5NA$G(B10,000$B1_J,$4MxMQ=PMh$^$9!#(B $B!ZL5NA%]%$%s%HFb$G==J,$K%Q!<%H%J!<$rC5$9;v$,=PMh$^$9!#![(B $B"((B18$B:P0J>e$NHkL)87l9g$O2<5-(BURL$B$KJV?.2<$5$$!#(B henshin_senyou at ok.kz From blist at aon.at Fri May 13 00:32:12 2005 From: blist at aon.at (Bernhard Fischer) Date: Fri, 13 May 2005 09:32:12 +0200 Subject: [PATCH] Re: [openib-general] 0 op factor In-Reply-To: <20050510174700.A5174@topspin.com> References: <20050505155803.GS15688@aon.at> <20050505172122.B26428@topspin.com> <20050507174718.GQ15688@aon.at> <20050510174700.A5174@topspin.com> Message-ID: <20050513073212.GA18105@aon.at> On Tue, May 10, 2005 at 05:47:00PM -0700, Libor Michalek wrote: >On Sat, May 07, 2005 at 07:47:18PM +0200, Bernhard Fischer wrote: >> - remove '0 operator factor' statements. >> - a bit of whitespace removal. >> - remove return at end of void functions. > > If you're removing the 'return' why not remove the blank line before >it as well. Should have done that too, yes. Would you ditch those blank lines while ci the whitespace stuff? If you want, i could send you a patchlet which does remove them sometimes not before end of next week. Please just drop the removal of the expect, it shouldn't have been it in the patch i sent. Thank you, Bernhard From blist at aon.at Fri May 13 00:35:19 2005 From: blist at aon.at (Bernhard Fischer) Date: Fri, 13 May 2005 09:35:19 +0200 Subject: [PATCH] Re: [openib-general] 0 op factor In-Reply-To: <20050512154857.D23734@topspin.com> References: <20050505155803.GS15688@aon.at> <20050505172122.B26428@topspin.com> <20050507174718.GQ15688@aon.at> <20050510174700.A5174@topspin.com> <20050512154857.D23734@topspin.com> Message-ID: <20050513073519.GB18105@aon.at> On Thu, May 12, 2005 at 03:48:57PM -0700, Libor Michalek wrote: >On Tue, May 10, 2005 at 05:47:00PM -0700, Libor Michalek wrote: >> On Sat, May 07, 2005 at 07:47:18PM +0200, Bernhard Fischer wrote: >> >> > - remove expect from _sdp_cm_path_complete(). >> >> When I said that the SDP_EXPECT should eventually be removed, I meant >> that the functions which have their return values checked by SDP_EXPECT >> should either be turned into void return functions, or something >> intelligent should be done with the return value, such as error recovery >> or propogation. The functions which should be turned into void functions >> are the ones which will never return anything but success. > > To expand on the last point. There are a lot of functions, for >example those in sdp_buff.c, which check for incorrect function >usage, such as checking that a buffer is not already in a queue before >inserting it into a queue. These checks could be removed entirely or >the checks could be made and a call to BUG() in case the condition >is met. I prefer the later since the former would corrupt the entire >queue. I'm not sure which is preferable to everyone else, either way >the result is that the function becomes a void. BUG_ON would be fine with me, fwiw. Guess others might have a stronger opinion regarding either, though. From halr at voltaire.com Fri May 13 05:52:41 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 13 May 2005 08:52:41 -0400 Subject: [openib-general] [PATCH] [kdapl]: Reduce CM retry count Message-ID: <1115988739.10509.25.camel@localhost.localdomain> Reduce CM retry count Signed-off-by: Hal Rosenstock Index: dapl_openib_cm.h =================================================================== -- dapl_openib_cm.h (revision 2331) +++ dapl_openib_cm.h (working copy) @@ -40,7 +40,7 @@ #define DAPL_OPENIB_RC_RETRY_COUNT 7 #define DAPL_OPENIB_RNR_RETRY_COUNT 6 #define DAPL_OPENIB_CM_RESPONSE_TIMEOUT 20 /* 4 sec */ -#define DAPL_OPENIB_MAX_CM_RETRIES 15 +#define DAPL_OPENIB_MAX_CM_RETRIES 4 typedef enum { IB_CME_CONNECTED, From halr at voltaire.com Fri May 13 05:55:54 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 13 May 2005 08:55:54 -0400 Subject: [openib-general] [PATCH][TRIVIAL] [kdapl] Minor CM Changes Message-ID: <1115988748.10509.27.camel@localhost.localdomain> Minor CM changes Change label in do_rep_recv to more appropriate name Signed-off-by: Hal Rosenstock Index: dapl_openib_cm.c =================================================================== -- dapl_openib_cm.c (revision 2331) +++ dapl_openib_cm.c (working copy) @@ -68,7 +68,7 @@ if (conn->ep->qp_handle == IB_INVALID_HANDLE) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, " do_rep_recv: invalid qp " "handle\n"); - goto reject; + goto disc; } /* First, transition QP to RTR */ @@ -76,7 +76,7 @@ if (status) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, " do_rep_recv: could not " "modify QP state to RTR status %d\n", status); - goto reject; + goto disc; } /* Now, transition QP to RTS */ @@ -84,23 +84,22 @@ if (status) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, " do_rep_recv: could not " "modify QP state to RTS status %d\n", status); - goto reject; + goto disc; } status = ib_send_cm_rtu(conn->cm_id, NULL, 0); if (status) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, " do_rep_recv: ib_send_cm_rtu " "failed: %d\n", status); - goto reject; + goto disc; } dapl_evd_connection_callback(conn, IB_CME_CONNECTED, rep_recv_param->private_data, conn->ep); return; -reject: - ib_send_cm_rej(conn->cm_id, IB_CM_REJ_CONSUMER_DEFINED, NULL, 0, - NULL, 0); +disc: + ib_send_cm_dreq(conn->cm_id, NULL, 0); /* ??? */ dapl_evd_connection_callback(conn, IB_CME_LOCAL_FAILURE, NULL, conn->ep); } @@ -456,7 +455,7 @@ if (status < 0) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, " ib_at_route_by_ip failed " "with status %d\n", status); - dapl_destroy_cm_id(conn); + dapl_destroy_cm_id(conn); /* ??? */ return DAT_INTERNAL_ERROR; } From halr at voltaire.com Fri May 13 05:59:24 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 13 May 2005 08:59:24 -0400 Subject: [openib-general] [PATCH] user_mad: Support RMPP on send side Message-ID: <1115989127.10509.33.camel@localhost.localdomain> user_mad: Support RMPP on send side Note that this change will need a coordinated change to OpenSM and some userspace/management libraries which will be done as soon as possible once this patch is accepted. Receive side support for RMPP will be added separately. Signed-off-by: Hal Rosenstock Index: infiniband/include/ib_user_mad.h =================================================================== --- infiniband/include/ib_user_mad.h (revision 2331) +++ infiniband/include/ib_user_mad.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Voltaire, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -42,7 +43,7 @@ * Increment this value if any changes that break userspace ABI * compatibility are made. */ -#define IB_USER_MAD_ABI_VERSION 2 +#define IB_USER_MAD_ABI_VERSION 3 /* * Make sure that all structs defined in this file remain laid out so @@ -51,8 +52,7 @@ */ /** - * ib_user_mad - MAD packet - * @data - Contents of MAD + * ib_user_mad_hdr - MAD packet header * @id - ID of agent MAD received with/to be sent with * @status - 0 on successful receive, ETIMEDOUT if no response * received (transaction ID in data[] will be set to TID of original @@ -72,8 +72,7 @@ * * All multi-byte quantities are stored in network (big endian) byte order. */ -struct ib_user_mad { - __u8 data[256]; +struct ib_user_mad_hdr { __u32 id; __u32 status; __u32 timeout_ms; @@ -91,6 +90,17 @@ }; /** + * ib_user_mad - MAD packet + * @hdr - MAD packet header + * @data - Contents of MAD + * + */ +struct ib_user_mad { + struct ib_user_mad_hdr hdr; + __u8 data[0]; +}; + +/** * ib_user_mad_reg_req - MAD registration request * @id - Set by the kernel; used to identify agent in future requests. * @qpn - Queue pair number; must be 0 or 1. @@ -103,6 +113,8 @@ * management class to receive. * @oui: Indicates IEEE OUI when mgmt_class is a vendor class * in the range from 0x30 to 0x4f. Otherwise not used. + * @rmpp_version: If set, indicates the RMPP version used. + * */ struct ib_user_mad_reg_req { __u32 id; @@ -111,6 +123,7 @@ __u8 mgmt_class; __u8 mgmt_class_version; __u8 oui[3]; + __u8 rmpp_version; }; #define IB_IOCTL_MAGIC 0x1b Index: infiniband/core/user_mad.c =================================================================== --- infiniband/core/user_mad.c (revision 2331) +++ infiniband/core/user_mad.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Voltaire, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -94,10 +95,12 @@ }; struct ib_umad_packet { - struct ib_user_mad mad; struct ib_ah *ah; + struct ib_mad_send_buf *msg; struct list_head list; + int length; DECLARE_PCI_UNMAP_ADDR(mapping) + struct ib_user_mad mad; }; static const dev_t base_dev = MKDEV(IB_UMAD_MAJOR, IB_UMAD_MINOR_BASE); @@ -114,10 +117,10 @@ int ret = 1; down_read(&file->agent_mutex); - for (packet->mad.id = 0; - packet->mad.id < IB_UMAD_MAX_AGENTS; - packet->mad.id++) - if (agent == file->agent[packet->mad.id]) { + for (packet->mad.hdr.id = 0; + packet->mad.hdr.id < IB_UMAD_MAX_AGENTS; + packet->mad.hdr.id++) + if (agent == file->agent[packet->mad.hdr.id]) { spin_lock_irq(&file->recv_lock); list_add_tail(&packet->list, &file->recv_list); spin_unlock_irq(&file->recv_lock); @@ -138,14 +141,11 @@ struct ib_umad_packet *packet = (void *) (unsigned long) send_wc->wr_id; - dma_unmap_single(agent->device->dma_device, - pci_unmap_addr(packet, mapping), - sizeof packet->mad.data, - DMA_TO_DEVICE); - ib_destroy_ah(packet->ah); + ib_free_send_mad(packet->msg); + ib_destroy_ah(packet->msg->send_wr.wr.ud.ah); if (send_wc->status == IB_WC_RESP_TIMEOUT_ERR) { - packet->mad.status = ETIMEDOUT; + packet->mad.hdr.status = ETIMEDOUT; if (!queue_packet(file, agent, packet)) return; @@ -159,30 +159,34 @@ { struct ib_umad_file *file = agent->context; struct ib_umad_packet *packet; + int length; + if (mad_recv_wc->wc->status != IB_WC_SUCCESS) goto out; - packet = kmalloc(sizeof *packet, GFP_KERNEL); + length = 256; /* until RMPP is supported */ + packet = kmalloc(sizeof *packet + length, GFP_KERNEL); if (!packet) goto out; - memset(packet, 0, sizeof *packet); + memset(packet, 0, sizeof *packet + length); + packet->length = length; - memcpy(packet->mad.data, mad_recv_wc->recv_buf.mad, sizeof packet->mad.data); - packet->mad.status = 0; - packet->mad.qpn = cpu_to_be32(mad_recv_wc->wc->src_qp); - packet->mad.lid = cpu_to_be16(mad_recv_wc->wc->slid); - packet->mad.sl = mad_recv_wc->wc->sl; - packet->mad.path_bits = mad_recv_wc->wc->dlid_path_bits; - packet->mad.grh_present = !!(mad_recv_wc->wc->wc_flags & IB_WC_GRH); - if (packet->mad.grh_present) { + memcpy(packet->mad.data, mad_recv_wc->recv_buf.mad, length); + packet->mad.hdr.status = 0; + packet->mad.hdr.qpn = cpu_to_be32(mad_recv_wc->wc->src_qp); + packet->mad.hdr.lid = cpu_to_be16(mad_recv_wc->wc->slid); + packet->mad.hdr.sl = mad_recv_wc->wc->sl; + packet->mad.hdr.path_bits = mad_recv_wc->wc->dlid_path_bits; + packet->mad.hdr.grh_present = !!(mad_recv_wc->wc->wc_flags & IB_WC_GRH); + if (packet->mad.hdr.grh_present) { /* XXX parse GRH */ - packet->mad.gid_index = 0; - packet->mad.hop_limit = 0; - packet->mad.traffic_class = 0; - memset(packet->mad.gid, 0, 16); - packet->mad.flow_label = 0; + packet->mad.hdr.gid_index = 0; + packet->mad.hdr.hop_limit = 0; + packet->mad.hdr.traffic_class = 0; + memset(packet->mad.hdr.gid, 0, 16); + packet->mad.hdr.flow_label = 0; } if (queue_packet(file, agent, packet)) @@ -199,7 +203,7 @@ struct ib_umad_packet *packet; ssize_t ret; - if (count < sizeof (struct ib_user_mad)) + if (count < sizeof (struct ib_user_mad) + 256) /* until RMPP supported */ return -EINVAL; spin_lock_irq(&file->recv_lock); @@ -222,10 +226,12 @@ spin_unlock_irq(&file->recv_lock); - if (copy_to_user(buf, &packet->mad, sizeof packet->mad)) + if (copy_to_user(buf, &packet->mad, + min(count, packet->length + + sizeof (struct ib_user_mad)))) ret = -EFAULT; else - ret = sizeof packet->mad; + ret = count; kfree(packet); return ret; @@ -238,107 +244,155 @@ struct ib_umad_packet *packet; struct ib_mad_agent *agent; struct ib_ah_attr ah_attr; - struct ib_sge gather_list; - struct ib_send_wr *bad_wr, wr = { - .opcode = IB_WR_SEND, - .sg_list = &gather_list, - .num_sge = 1, - .send_flags = IB_SEND_SIGNALED, - }; + struct ib_send_wr *bad_wr; + struct ib_rmpp_mad *rmpp_mad; u8 method; u64 *tid; - int ret; + int ret, length, hdr_len, data_len, rmpp_hdr_size; + int rmpp_active = 0; if (count < sizeof (struct ib_user_mad)) return -EINVAL; - packet = kmalloc(sizeof *packet, GFP_KERNEL); + length = count - sizeof (struct ib_user_mad); + packet = kmalloc(sizeof *packet + sizeof(struct ib_mad_hdr) + + sizeof(struct ib_rmpp_hdr), GFP_KERNEL); if (!packet) return -ENOMEM; - if (copy_from_user(&packet->mad, buf, sizeof packet->mad)) { - kfree(packet); - return -EFAULT; + if (copy_from_user(&packet->mad, buf, + sizeof (struct ib_user_mad) + + sizeof(struct ib_mad_hdr) + + sizeof(struct ib_rmpp_hdr))) { + ret = -EFAULT; + goto err; } - if (packet->mad.id < 0 || packet->mad.id >= IB_UMAD_MAX_AGENTS) { + if (packet->mad.hdr.id < 0 || + packet->mad.hdr.id >= IB_UMAD_MAX_AGENTS) { ret = -EINVAL; goto err; } + packet->length = length; + down_read(&file->agent_mutex); - agent = file->agent[packet->mad.id]; + agent = file->agent[packet->mad.hdr.id]; if (!agent) { ret = -EINVAL; goto err_up; } + memset(&ah_attr, 0, sizeof ah_attr); + ah_attr.dlid = be16_to_cpu(packet->mad.hdr.lid); + ah_attr.sl = packet->mad.hdr.sl; + ah_attr.src_path_bits = packet->mad.hdr.path_bits; + ah_attr.port_num = file->port->port_num; + if (packet->mad.hdr.grh_present) { + ah_attr.ah_flags = IB_AH_GRH; + memcpy(ah_attr.grh.dgid.raw, packet->mad.hdr.gid, 16); + ah_attr.grh.flow_label = packet->mad.hdr.flow_label; + ah_attr.grh.hop_limit = packet->mad.hdr.hop_limit; + ah_attr.grh.traffic_class = packet->mad.hdr.traffic_class; + } + + packet->ah = ib_create_ah(agent->qp->pd, &ah_attr); + if (IS_ERR(packet->ah)) { + ret = PTR_ERR(packet->ah); + goto err_up; + } + + rmpp_mad = (struct ib_rmpp_mad *) packet->mad.data; + if (ib_get_rmpp_flags(&rmpp_mad->rmpp_hdr) & IB_MGMT_RMPP_FLAG_ACTIVE) { + /* RMPP active */ + if (!agent->rmpp_version) { + ret = -EINVAL; + goto err_ah; + } + /* Validate that management class can support RMPP */ + if (rmpp_mad->mad_hdr.mgmt_class == IB_MGMT_CLASS_SUBN_ADM) { + hdr_len = offsetof(struct ib_sa_mad, data); + data_len = length; + } else if ((rmpp_mad->mad_hdr.mgmt_class >= IB_MGMT_CLASS_VENDOR_RANGE2_START) && + (rmpp_mad->mad_hdr.mgmt_class <= IB_MGMT_CLASS_VENDOR_RANGE2_END)) { + hdr_len = offsetof(struct ib_vendor_mad, data); + data_len = length - hdr_len; + } else { + ret = -EINVAL; + goto err_ah; + } + rmpp_active = 1; + } else { + if (length > sizeof(struct ib_mad)) { + ret = -EINVAL; + goto err_ah; + } + hdr_len = offsetof(struct ib_mad, data); + data_len = length - hdr_len; + } + + packet->msg = ib_create_send_mad(agent, + be32_to_cpu(packet->mad.hdr.qpn), + 0, packet->ah, rmpp_active, + hdr_len, data_len, + GFP_KERNEL); + if (IS_ERR(packet->msg)) { + ret = PTR_ERR(packet->msg); + goto err_ah; + } + + packet->msg->send_wr.wr.ud.retries = 0; + packet->msg->send_wr.wr.ud.timeout_ms = packet->mad.hdr.timeout_ms; + /* Override send WR WRID created by ib_create_send_mad */ + packet->msg->send_wr.wr_id = (unsigned long) packet; + + if (!rmpp_active) { + /* Copy message from user into send buffer */ + copy_from_user(packet->msg->mad, + buf + sizeof(struct ib_user_mad), length); + } else { + rmpp_hdr_size = sizeof(struct ib_mad_hdr) + + sizeof(struct ib_rmpp_hdr); + /* Only copy MAD headers (RMPP header in place) */ + memcpy(packet->msg->mad, packet->mad.data, + sizeof(struct ib_mad_hdr)); + /* Now, copy rest of message from user into send buffer */ + copy_from_user(((struct ib_rmpp_mad *)packet->msg->mad)->data, + buf + sizeof(struct ib_user_mad) + rmpp_hdr_size, + length - rmpp_hdr_size); + } + /* * If userspace is generating a request that will generate a * response, we need to make sure the high-order part of the * transaction ID matches the agent being used to send the * MAD. */ - method = ((struct ib_mad_hdr *) packet->mad.data)->method; + method = packet->msg->mad->mad_hdr.method; if (!(method & IB_MGMT_METHOD_RESP) && method != IB_MGMT_METHOD_TRAP_REPRESS && method != IB_MGMT_METHOD_SEND) { - tid = &((struct ib_mad_hdr *) packet->mad.data)->tid; + tid = &packet->msg->mad->mad_hdr.tid; *tid = cpu_to_be64(((u64) agent->hi_tid) << 32 | (be64_to_cpup(tid) & 0xffffffff)); } - memset(&ah_attr, 0, sizeof ah_attr); - ah_attr.dlid = be16_to_cpu(packet->mad.lid); - ah_attr.sl = packet->mad.sl; - ah_attr.src_path_bits = packet->mad.path_bits; - ah_attr.port_num = file->port->port_num; - if (packet->mad.grh_present) { - ah_attr.ah_flags = IB_AH_GRH; - memcpy(ah_attr.grh.dgid.raw, packet->mad.gid, 16); - ah_attr.grh.flow_label = packet->mad.flow_label; - ah_attr.grh.hop_limit = packet->mad.hop_limit; - ah_attr.grh.traffic_class = packet->mad.traffic_class; - } + ret = ib_post_send_mad(agent, &packet->msg->send_wr, &bad_wr); + if (ret) + goto err_msg; - packet->ah = ib_create_ah(agent->qp->pd, &ah_attr); - if (IS_ERR(packet->ah)) { - ret = PTR_ERR(packet->ah); - goto err_up; - } + up_read(&file->agent_mutex); - gather_list.addr = dma_map_single(agent->device->dma_device, - packet->mad.data, - sizeof packet->mad.data, - DMA_TO_DEVICE); - gather_list.length = sizeof packet->mad.data; - gather_list.lkey = file->mr[packet->mad.id]->lkey; - pci_unmap_addr_set(packet, mapping, gather_list.addr); + return sizeof (struct ib_user_mad_hdr) + packet->length; - wr.wr.ud.mad_hdr = (struct ib_mad_hdr *) packet->mad.data; - wr.wr.ud.ah = packet->ah; - wr.wr.ud.remote_qpn = be32_to_cpu(packet->mad.qpn); - wr.wr.ud.remote_qkey = be32_to_cpu(packet->mad.qkey); - wr.wr.ud.timeout_ms = packet->mad.timeout_ms; - wr.wr.ud.retries = 0; +err_msg: + ib_free_send_mad(packet->msg); - wr.wr_id = (unsigned long) packet; +err_ah: + ib_destroy_ah(packet->ah); - ret = ib_post_send_mad(agent, &wr, &bad_wr); - if (ret) { - dma_unmap_single(agent->device->dma_device, - pci_unmap_addr(packet, mapping), - sizeof packet->mad.data, - DMA_TO_DEVICE); - goto err_up; - } - - up_read(&file->agent_mutex); - - return sizeof packet->mad; - err_up: up_read(&file->agent_mutex); @@ -400,7 +454,8 @@ agent = ib_register_mad_agent(file->port->ib_dev, file->port->port_num, ureq.qpn ? IB_QPT_GSI : IB_QPT_SMI, ureq.mgmt_class ? &req : NULL, - 0, send_handler, recv_handler, file); + ureq.rmpp_version, + send_handler, recv_handler, file); if (IS_ERR(agent)) { ret = PTR_ERR(agent); goto out; Index: docs/user_mad.txt =================================================================== --- docs/user_mad.txt (revision 2331) +++ docs/user_mad.txt (working copy) @@ -28,11 +28,13 @@ Receiving MADs - MADs are received using read(). The buffer passed to read() must be - large enough to hold at least one struct ib_user_mad. For example: + MADs are received using read(). The receive side does not currently + support RMPP so the buffer passed to read() must be at least one + struct ib_user_mad + 256 bytes. For example: - struct ib_user_mad mad; - ret = read(fd, &mad, sizeof mad); + struct ib_user_mad *mad; + mad = malloc(sizeof *mad + 256); + ret = read(fd, mad, sizeof *mad + 256); if (ret != sizeof mad) perror("read"); @@ -50,18 +52,21 @@ MADs are sent using write(). The agent ID for sending should be filled into the id field of the MAD, the destination LID should be - filled into the lid field, and so on. For example: + filled into the lid field, and so on. The send side does support + RMPP so arbitrary length MAD can be sent. For example: - struct ib_user_mad mad; + struct ib_user_mad *mad; - /* fill in mad.data */ + mad = malloc(sizeof *mad + mad_length); - mad.id = my_agent; /* req.id from agent registration */ - mad.lid = my_dest; /* in network byte order... */ + /* fill in mad->data */ + + mad->hdr.id = my_agent; /* req.id from agent registration */ + mad->hdr.lid = my_dest; /* in network byte order... */ /* etc. */ - ret = write(fd, &mad, sizeof mad); - if (ret != sizeof mad) + ret = write(fd, &mad, sizeof *mad + mad_length); + if (ret != sizeof *mad + mad_length) perror("write"); Setting IsSM Capability Bit From mst at mellanox.co.il Fri May 13 06:51:45 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Fri, 13 May 2005 16:51:45 +0300 Subject: [openib-general] Re: [PATCH] (repost) sdp: replace mlock with get_user_pages In-Reply-To: <52fywst4z3.fsf@topspin.com> References: <20050505110158.GH30653@mellanox.co.il> <20050506175415.I26823@topspin.com> <20050510143232.GF2474@mellanox.co.il> <20050512151051.C23734@topspin.com> <52fywst4z3.fsf@topspin.com> Message-ID: <20050513135145.GA320@mellanox.co.il> Quoting r. Roland Dreier : > Subject: Re: [openib-general] Re: [PATCH] (repost) sdp: replace mlock with get_user_pages > > Libor> Always call aio_complete() immediately when > Libor> iocb_complete() is called, and only spawn the work thread > Libor> to unlock the memory after the aio_complete() call. The > Libor> patch is below. Libor, I dont think its a good idea - this will break other assumptions, like the assumption that the task mm isnt destroyed before we unlock the memory. > Another alternative would be to always complete aios asynchronously, > which should preserve the order. I guess this would hurt latency for > small ios... > > - R. > To avoid hurting latency, lets count the number of outstanding asynchronous AIOs, and if there are asynchronous AIOs complete all of them asynchronously. Does this make sense? -- MST - Michael S. Tsirkin From mst at mellanox.co.il Fri May 13 06:54:09 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Fri, 13 May 2005 16:54:09 +0300 Subject: [openib-general] Re: user-mode verbs on Itanium In-Reply-To: <20050512155550.GA14123@esmail.cup.hp.com> References: <20050504005534.GA13574@esmail.cup.hp.com> <20050504070910.GC9674@mellanox.co.il> <20050506190947.GA23958@esmail.cup.hp.com> <20050507053706.GN23958@esmail.cup.hp.com> <20050508142345.GQ9941@mellanox.co.il> <20050510152428.GA5495@esmail.cup.hp.com> <20050510154030.GH2474@mellanox.co.il> <20050511012148.GE5495@esmail.cup.hp.com> <20050511072138.GB13349@mellanox.co.il> <20050512155550.GA14123@esmail.cup.hp.com> Message-ID: <20050513135409.GB320@mellanox.co.il> Quoting r. Grant Grundler : > Subject: Re: user-mode verbs on Itanium > > On Wed, May 11, 2005 at 10:21:38AM +0300, Michael S. Tsirkin wrote: > > You mean just sample the clock before the first post? > > Yes. > > > That would be easy, but lets look at what does it measure: > > > > On the client, you just get the time it takes to perform the > > first post_send operation. If this is interesting to you, I agree, > > but lets take the timestamps around each post send operation to > > make it statistically relevant. > > Let me know - and need to think of a way to do this without > > affecting latency numbers. > > Yes, I'd like to try that. > But a few more of my pending patches need to either be rejected > or (preferably) accepted. I plan to go over that on Sunday. -- MST - Michael S. Tsirkin From mst at mellanox.co.il Fri May 13 07:06:00 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Fri, 13 May 2005 17:06:00 +0300 Subject: [openib-general] use of virt_to_phys in sdp Message-ID: <20050513140600.GC320@mellanox.co.il> Use of virt_to_phys looks wrong in sdp_recv and sdp_send. Must not the DMA API be used to get the physical address to be put in ib_sge? -- MST - Michael S. Tsirkin From mst at mellanox.co.il Fri May 13 07:06:55 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Fri, 13 May 2005 17:06:55 +0300 Subject: [openib-general] Re: [PATCH] Re: 0 op factor In-Reply-To: <20050512154857.D23734@topspin.com> References: <20050505155803.GS15688@aon.at> <20050505172122.B26428@topspin.com> <20050507174718.GQ15688@aon.at> <20050510174700.A5174@topspin.com> <20050512154857.D23734@topspin.com> Message-ID: <20050513140655.GD320@mellanox.co.il> Quoting r. Libor Michalek : > Subject: Re: [PATCH] Re: 0 op factor > > On Tue, May 10, 2005 at 05:47:00PM -0700, Libor Michalek wrote: > > On Sat, May 07, 2005 at 07:47:18PM +0200, Bernhard Fischer wrote: > > > > > - remove expect from _sdp_cm_path_complete(). > > > > When I said that the SDP_EXPECT should eventually be removed, I meant > > that the functions which have their return values checked by SDP_EXPECT > > should either be turned into void return functions, or something > > intelligent should be done with the return value, such as error recovery > > or propogation. The functions which should be turned into void functions > > are the ones which will never return anything but success. > > To expand on the last point. There are a lot of functions, for > example those in sdp_buff.c, which check for incorrect function > usage, such as checking that a buffer is not already in a queue before > inserting it into a queue. These checks could be removed entirely or > the checks could be made and a call to BUG() in case the condition > is met. I prefer the later since the former would corrupt the entire > queue. I'm not sure which is preferable to everyone else, either way > the result is that the function becomes a void. > > -Libor IMO BUG_ON is the way to go. -- MST - Michael S. Tsirkin From halr at voltaire.com Fri May 13 07:22:53 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 13 May 2005 10:22:53 -0400 Subject: [openib-general] [PATCH] libibcm building Message-ID: <1115994173.10509.79.camel@localhost.localdomain> Hi Libor, A couple of minor libibcm building comments: 1. When running autogen.sh, it warns about: Makefile.am:19: variable `ucm_version_script' not defined 2. example/simple is not built by the current makefile generated. Below is a patch for this. -- Hal Add building of simple example for user CM to generated makefile Signed-off-by: Hal Rosenstock Index: Makefile.am =================================================================== --- Makefile.am (revision 2331) +++ Makefile.am (working copy) @@ -5,7 +5,7 @@ ibcmlibdir = $(libdir)/infiniband -ibcmlib_LTLIBRARIES = src/ibcm.la +ibcmlib_LTLIBRARIES = src/libibcm.la src_ibcm_la_CFLAGS = -g -Wall -D_GNU_SOURCE @@ -15,9 +15,13 @@ ibcm_version_script = endif -src_ibcm_la_SOURCES = src/cm.c -src_ibcm_la_LDFLAGS = -avoid-version -module $(ucm_version_script) +src_libibcm_la_SOURCES = src/cm.c +src_libibcm_la_LDFLAGS = -avoid-version -module $(ucm_version_script) +bin_PROGRAMS = examples/ucm_simple +examples_ucm_simple_SOURCES = examples/simple.c +examples_ucm_simple_LDADD = $(top_builddir)/src/libibcm.la + libibcmincludedir = $(includedir)/infiniband libibcminclude_HEADERS = include/infiniband/cm_abi.h \ From jlentini at netapp.com Fri May 13 07:42:12 2005 From: jlentini at netapp.com (James Lentini) Date: Fri, 13 May 2005 10:42:12 -0400 (EDT) Subject: [openib-general] Re: DAPL CM Reject Questions In-Reply-To: <1115934950.16144.22.camel@duffman> References: <1115901591.4766.38.camel@localhost.localdomain> <42837BCC.20003@ichips.intel.com> <1115934950.16144.22.camel@duffman> Message-ID: If I don't use a function's return value, I cast it to void to say "I know this function returns something but I don't want to use it". I'm not sure were I picked up this habit, but I think it is fairly common. Some compilers will give a warning if you don't perform the cast, although gcc does not appear to. Here are some of the places I've found this construct in the 2.6.11 sources: arch/i386/kernel/apm.c (line 924) arch/i386/mach-voyager/voyager_smp.c (line 1606) crypto/tcrypt.c (line 547) drivers/acpi/events/evxface.c (line 95) On Thu, 12 May 2005, Tom Duffy wrote: tduffy> On Thu, 2005-05-12 at 17:31 -0400, James Lentini wrote: tduffy> > I've changed the above line to tduffy> > tduffy> > (void) ib_send_cm_dreq(conn->cm_id, NULL, 0); tduffy> tduffy> Is there a reason for the leading (void)? tduffy> tduffy> -tduffy tduffy> From halr at voltaire.com Fri May 13 07:41:57 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 13 May 2005 10:41:57 -0400 Subject: [openib-general] [PATCH] libibcm/examples/simple.c: Eliminate a couple of warnings Message-ID: <1115995316.10509.87.camel@localhost.localdomain> libibcm/examples/simple.c: Eliminate a couple of warnings Signed-off-by: Hal Rosenstock Index: simple.c =================================================================== --- simple.c (revision 2331) +++ simple.c (working copy) @@ -45,6 +45,7 @@ #include #include #include +#include #include #include @@ -63,7 +64,6 @@ struct ib_sa_path_rec sa; union ibv_gid *dst; union ibv_gid *src; - int result; param.qp_type = IBV_QPT_RC; param.qp_num = 0xff00; From adarme at emailaccount.com Fri May 13 09:23:24 2005 From: adarme at emailaccount.com (Carlene Kent) Date: Fri, 13 May 2005 15:23:24 -0100 Subject: [openib-general] Super low mortage rate Message-ID: <110341.5920.adarme@emailaccount.com> Hello, We tried contacting you awhile ago about your low interest morta(ge rate.= You have qualified for the lowest rate in years... You could get over $380,000 for as little as $500 a month! Ba(d credit? Doesn't matter, low rates are fixed no matter what! To get a free, no obli,gation consultation click below: http://www.ch00s3.com/sign.asp Best Regards, Jeremiah Snyder to be remov(ed: http://www.ch00s3.com/gone.asp this process takes one week, so please be patient. we do our best to take your email/s off but you have to fill out a rem/ove or else you will continue to recieve email/s. From info at kqsv19.com Fri May 13 05:20:21 2005 From: info at kqsv19.com (info at kqsv19.com) Date: 13 May 2005 21:20:21 +0900 Subject: [openib-general] $B?75,EPO?$NJ}$O(B10000$B1_J,L5NA(B Message-ID: <20050513122021.15489.qmail@mail.kqsv19.com> $B0lK|1_J,!*40A4L5NA$G;HMQ(BOK $B"#:#$J$i4V$K9g$&(BW$B%A%c%s%9!*L5NA%]%$%s%H%"%C%W3Z$7$5(B100$BG\!*"#(B $B"(EPO?=w at -2q0w$N%8%c%s%kJL?M5$%i%s%-%s%0(B $B0l0L!'5U1g!J(B189$BL>EPO?!K(B $BFs0L!'1g8r!J(B429$BL>EPO?!K(B $B;00L!'(BSM$B5U1g!J(B243$BL>EPO?!K(B $B;M0L!'%;%U%l!J(B1038$BL>EPO?!K(B $B8^0L!'ITNQ!J(B2421$BL>EPO?!K(B $BO;0L!'%F%l%(%C%A!J(B3463$BL>EPO?!K(B $B!&(B $B!&(B $B!&(B $B0lK|1_J,40A4L5NA$G$*;n$7$O(B $B"-"-"-(B http://awg.webchu.com/?summer19 $BB~:#!*?75,EPO?$7$?J}$K$OL5NA$G(B10,000$B1_J,$4MxMQ=PMh$^$9!#(B $B!ZL5NA%]%$%s%HFb$G==J,$K%Q!<%H%J!<$rC5$9;v$,=PMh$^$9!#![(B $B"((B18$B:P0J>e$NHkL)87l9g$O2<5-(BURL$B$KJV?.2<$5$$!#(B henshin_senyou at ok.kz From tduffy at sun.com Fri May 13 08:27:58 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 13 May 2005 08:27:58 -0700 Subject: [openib-general] [PATCH] SDP & CM: change some functions to return void Message-ID: <1115998078.10240.8.camel@duffman> This patch changes some functions to return void rather than an int since they were always returning 0, thus making checking return values rather pointless. The patch touches both SDP and CM. Signed-off-by: Tom Duffy Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_link.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_link.c (revision 2334) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_link.c (working copy) @@ -720,19 +720,7 @@ int sdp_link_addr_init(void) int result; sdp_dbg_init("Link level services initialization."); -#if 0 - /* - * Install async event handler, to clear cache on port down - */ - result = tsIbAsynchronousEventHandlerRegister(TS_IB_PORT_ERROR, - _tsSdpAsyncEventFunc, - NULL, - &_tsSdpAsyncHandle); - if (result) { - - goto error_async; - } -#endif + info_cache = kmem_cache_create("sdp_path_info", sizeof(struct sdp_path_info), 0, SLAB_HWCACHE_ALIGN, @@ -784,7 +772,7 @@ error_path: /* * sdp_link_addr_cleanup - cleanup the link address retrival code */ -int sdp_link_addr_cleanup(void) +void sdp_link_addr_cleanup(void) { struct sdp_path_info *info; @@ -809,12 +797,4 @@ int sdp_link_addr_cleanup(void) */ kmem_cache_destroy(info_cache); kmem_cache_destroy(wait_cache); -#if 0 - /* - * release async event handler - */ - (void)tsIbAsynchronousEventHandlerUnregister(_tsSdpAsyncHandle); -#endif - - return 0; } Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_inet.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_inet.c (revision 2334) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_inet.c (working copy) @@ -1553,15 +1553,15 @@ static int __init sdp_init(void) return 0; error_sock: - (void)sdp_conn_table_clear(); + sdp_conn_table_clear(); error_conn: - (void)sdp_buff_pool_destroy(); + sdp_buff_pool_destroy(); error_buff: - (void)sdp_link_addr_cleanup(); + sdp_link_addr_cleanup(); error_link: sdp_main_advt_cleanup(); error_advt: - (void)sdp_main_proc_cleanup(); + sdp_main_proc_cleanup(); error_proc: return result; /* success */ } @@ -1579,15 +1579,15 @@ static void __exit sdp_exit(void) /* * connection table */ - (void)sdp_conn_table_clear(); + sdp_conn_table_clear(); /* * delete buffer memory */ - (void)sdp_buff_pool_destroy(); + sdp_buff_pool_destroy(); /* * delete link information */ - (void)sdp_link_addr_cleanup(); + sdp_link_addr_cleanup(); /* * delete advertisment table */ @@ -1595,7 +1595,7 @@ static void __exit sdp_exit(void) /* * proc tables */ - (void)sdp_main_proc_cleanup(); + sdp_main_proc_cleanup(); } module_init(sdp_init); Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_proto.h =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_proto.h (revision 2334) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_proto.h (working copy) @@ -186,13 +186,13 @@ void sdp_iocb_q_remove(struct sdpc_iocb int sdp_iocb_register(struct sdpc_iocb *iocb, struct sdp_opt *conn); -int sdp_iocb_release(struct sdpc_iocb *iocb); +void sdp_iocb_release(struct sdpc_iocb *iocb); int sdp_iocb_complete(struct sdpc_iocb *iocb, ssize_t status); int sdp_iocb_lock(struct sdpc_iocb *iocb); -int sdp_iocb_unlock(struct sdpc_iocb *iocb); +void sdp_iocb_unlock(struct sdpc_iocb *iocb); /* * Generic object managment @@ -236,7 +236,7 @@ void sdp_desc_q_clear(struct sdpc_desc_q */ int sdp_main_proc_init(void); -int sdp_main_proc_cleanup(void); +void sdp_main_proc_cleanup(void); /* * connection table @@ -249,7 +249,7 @@ int sdp_conn_table_init(int proto_family int send_buff_max, int send_usig_max); -int sdp_conn_table_clear(void); +void sdp_conn_table_clear(void); int sdp_proc_dump_conn_main(char *buffer, int max_size, @@ -434,7 +434,7 @@ int sdp_link_path_lookup(u32 dst_addr, int sdp_link_addr_init(void); -int sdp_link_addr_cleanup(void); +void sdp_link_addr_cleanup(void); /* * Function types Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_conn.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_conn.c (revision 2334) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_conn.c (working copy) @@ -613,7 +613,7 @@ static void sdp_desc_q_cancel_iocb(struc NULL))) { sdp_iocb_q_remove(iocb); - (void)sdp_iocb_complete(iocb, error); + sdp_iocb_complete(iocb, error); } } @@ -741,11 +741,7 @@ void sdp_conn_destruct(struct sdp_opt *c */ if (conn->cm_id) { sdp_dbg_warn(conn, "destroying CM ID."); - - result = ib_destroy_cm_id(conn->cm_id); - if (result) - sdp_dbg_warn(conn, "Error <%d> detroying CM ID", - result); + ib_destroy_cm_id(conn->cm_id); } /* * check consistancy @@ -1826,7 +1822,7 @@ error: } if (!IS_ERR(hca->fmr_pool)) - (void)ib_destroy_fmr_pool(hca->fmr_pool); + ib_destroy_fmr_pool(hca->fmr_pool); if (!IS_ERR(hca->mem_h)) (void)ib_dereg_mr(hca->mem_h); @@ -1861,7 +1857,7 @@ static void sdp_device_remove_one(struct } if (!IS_ERR(hca->fmr_pool)) - (void)ib_destroy_fmr_pool(hca->fmr_pool); + ib_destroy_fmr_pool(hca->fmr_pool); if (!IS_ERR(hca->mem_h)) (void)ib_dereg_mr(hca->mem_h); @@ -1999,7 +1995,7 @@ int sdp_conn_table_init(int proto_family return 0; error_listen: - (void)ib_destroy_cm_id(dev_root_s.listen_id); + ib_destroy_cm_id(dev_root_s.listen_id); error_cm_id: kmem_cache_destroy(dev_root_s.sock_cache); error_sock: @@ -2019,8 +2015,8 @@ error_hca: /* * sdp_conn_table_clear - destroy connection managment and tables */ -int sdp_conn_table_clear(void) - { +void sdp_conn_table_clear(void) +{ sdp_dbg_init("Deleting connection tables."); #if 0 struct sdp_opt *conn; @@ -2055,7 +2051,5 @@ int sdp_conn_table_clear(void) /* * stop listening */ - (void)ib_destroy_cm_id(dev_root_s.listen_id); - - return 0; + ib_destroy_cm_id(dev_root_s.listen_id); } Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_wall.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_wall.c (revision 2334) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_wall.c (working copy) @@ -422,8 +422,6 @@ void sdp_wall_recv_drop(struct sdp_opt * break; } - - return; } /* Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_recv.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_recv.c (revision 2334) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_recv.c (working copy) @@ -447,7 +447,7 @@ release: iocb->flags &= ~SDP_IOCB_F_ACTIVE; iocb->flags &= ~SDP_IOCB_F_RDMA_W; - (void)sdp_iocb_release(iocb); + sdp_iocb_release(iocb); error: return result; } Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_proc.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_proc.c (revision 2334) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_proc.c (working copy) @@ -127,7 +127,7 @@ static struct sdpc_proc_ent file_entry_l /* * sdp_main_proc_cleanup - cleanup the proc filesystem entries */ -int sdp_main_proc_cleanup(void) +void sdp_main_proc_cleanup(void) { struct sdpc_proc_ent *sub_entry; int counter; @@ -149,8 +149,6 @@ int sdp_main_proc_cleanup(void) dir_root = NULL; sdp_dbg_init("/proc filesystem cleanup complete."); - - return 0; } /* @@ -212,6 +210,6 @@ int sdp_main_proc_init(void) return 0; /* success */ error: - (void)sdp_main_proc_cleanup(); + sdp_main_proc_cleanup(); return result; } Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_iocb.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_iocb.c (revision 2334) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_iocb.c (working copy) @@ -47,7 +47,7 @@ static do_mlock_ptr_t mlock_ptr = NULL; /* * do_iocb_unlock - unlock the memory for an IOCB */ -static int do_iocb_unlock(struct sdpc_iocb *iocb) +static void do_iocb_unlock(struct sdpc_iocb *iocb) { struct vm_area_struct *vma; @@ -90,28 +90,24 @@ static int do_iocb_unlock(struct sdpc_io else vma = NULL; } - - return 0; } /* * sdp_iocb_unlock - unlock the memory for an IOCB */ -int sdp_iocb_unlock(struct sdpc_iocb *iocb) +void sdp_iocb_unlock(struct sdpc_iocb *iocb) { - int result; - /* * check if IOCB is locked. */ if (!(iocb->flags & SDP_IOCB_F_LOCKED)) - return 0; + return; /* * spin lock since this could be from interrupt context. */ down_write(&iocb->mm->mmap_sem); - result = do_iocb_unlock(iocb); + do_iocb_unlock(iocb); up_write(&iocb->mm->mmap_sem); @@ -125,8 +121,6 @@ int sdp_iocb_unlock(struct sdpc_iocb *io * mark IOCB unlocked. */ iocb->flags &= ~SDP_IOCB_F_LOCKED; - - return result; } /* @@ -331,7 +325,7 @@ int sdp_iocb_lock(struct sdpc_iocb *iocb return 0; err_save: - (void)do_iocb_unlock(iocb); + do_iocb_unlock(iocb); err_lock: /* * unlock the mm and restore capabilities. @@ -402,15 +396,13 @@ done: /* * sdp_mem_lock_cleanup - cleanup the memory locking tables */ -static int sdp_mem_lock_cleanup(void) +static void sdp_mem_lock_cleanup(void) { sdp_dbg_init("Memory Locking cleanup."); /* * null out entries. */ mlock_ptr = NULL; - - return 0; } /* @@ -472,12 +464,12 @@ error: /* * sdp_iocb_release - unregister an IOCBs memory */ -int sdp_iocb_release(struct sdpc_iocb *iocb) +void sdp_iocb_release(struct sdpc_iocb *iocb) { int result; if (!(iocb->flags & SDP_IOCB_F_REG)) - return 0; + return; result = ib_fmr_pool_unmap(iocb->mem); if (result < 0) @@ -485,8 +477,6 @@ int sdp_iocb_release(struct sdpc_iocb *i result, iocb->key, iocb->addr); iocb->flags &= ~(SDP_IOCB_F_REG); - - return 0; } /* @@ -500,17 +490,11 @@ static void do_iocb_complete(void *arg) /* * release memory */ - result = sdp_iocb_release(iocb); - if (result < 0) - sdp_dbg_err("Error <%d> releasing IOCB <%d> memory <%ld>", - result, iocb->key, iocb->addr); + sdp_iocb_release(iocb); /* * unlock now, after aio_complete the mm reference will be released. */ - result = sdp_iocb_unlock(iocb); - if (result < 0) - sdp_dbg_err("Error <%d> unlocking IOCB <%d memory <%ld>>", - result, iocb->key, iocb->addr); + sdp_iocb_unlock(iocb); /* * callback to complete IOCB */ @@ -633,11 +617,11 @@ int sdp_iocb_destroy(struct sdpc_iocb *i /* * release iocb registered memory */ - (void)sdp_iocb_release(iocb); + sdp_iocb_release(iocb); /* * unlock IOCB memory */ - (void)sdp_iocb_unlock(iocb); + sdp_iocb_unlock(iocb); /* * array dealloc */ @@ -870,7 +854,7 @@ int sdp_main_iocb_init(void) return 0; error_iocb_c: - (void)sdp_mem_lock_cleanup(); + sdp_mem_lock_cleanup(); return result; } @@ -891,5 +875,5 @@ void sdp_main_iocb_cleanup(void) /* * cleanup memory locking */ - (void)sdp_mem_lock_cleanup(); + sdp_mem_lock_cleanup(); } Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_event.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_event.c (revision 2334) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_event.c (working copy) @@ -143,7 +143,7 @@ void sdp_cq_event_handler(struct ib_cq * if (!conn) { sdp_dbg_warn(conn, "Unknown connection <%d> for cq event", hashent); - goto done; + return; } /* * lock the bottom half of the socket. If the connection is in use, @@ -190,8 +190,6 @@ void sdp_cq_event_handler(struct ib_cq * unlock: SDP_CONN_UNLOCK_IRQ(conn, flags); sdp_conn_put(conn); -done: - return; } /* Index: linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_buff.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_buff.c (revision 2334) +++ linux-2.6.11-openib/drivers/infiniband/ulp/sdp/sdp_buff.c (working copy) @@ -724,7 +724,7 @@ int sdp_buff_pool_chain_put(struct sdpc_ main_pool->pool.size += count; - (void)sdp_buff_pool_release_check(main_pool); + sdp_buff_pool_release_check(main_pool); spin_unlock_irqrestore(&main_pool->lock, flags); Index: linux-2.6.11-openib/drivers/infiniband/include/ib_cm.h =================================================================== --- linux-2.6.11-openib/drivers/infiniband/include/ib_cm.h (revision 2334) +++ linux-2.6.11-openib/drivers/infiniband/include/ib_cm.h (working copy) @@ -309,7 +309,7 @@ struct ib_cm_id *ib_create_cm_id(ib_cm_h * * This call blocks until the connection identifier is destroyed. */ -int ib_destroy_cm_id(struct ib_cm_id *cm_id); +void ib_destroy_cm_id(struct ib_cm_id *cm_id); /** * ib_cm_listen - Initiates listening on the specified service ID for Index: linux-2.6.11-openib/drivers/infiniband/include/ib_fmr_pool.h =================================================================== --- linux-2.6.11-openib/drivers/infiniband/include/ib_fmr_pool.h (revision 2334) +++ linux-2.6.11-openib/drivers/infiniband/include/ib_fmr_pool.h (working copy) @@ -78,7 +78,7 @@ struct ib_pool_fmr { struct ib_fmr_pool *ib_create_fmr_pool(struct ib_pd *pd, struct ib_fmr_pool_param *params); -int ib_destroy_fmr_pool(struct ib_fmr_pool *pool); +void ib_destroy_fmr_pool(struct ib_fmr_pool *pool); int ib_flush_fmr_pool(struct ib_fmr_pool *pool); Index: linux-2.6.11-openib/drivers/infiniband/core/cm.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/core/cm.c (revision 2334) +++ linux-2.6.11-openib/drivers/infiniband/core/cm.c (working copy) @@ -580,7 +580,7 @@ static void cm_reset_to_idle(struct cm_i } } -int ib_destroy_cm_id(struct ib_cm_id *cm_id) +void ib_destroy_cm_id(struct ib_cm_id *cm_id) { struct cm_id_private *cm_id_priv; struct cm_work *work; @@ -644,7 +644,6 @@ retest: while ((work = cm_dequeue_work(cm_id_priv)) != NULL) cm_free_work(work); kfree(cm_id_priv); - return 0; } EXPORT_SYMBOL(ib_destroy_cm_id); Index: linux-2.6.11-openib/drivers/infiniband/core/fmr_pool.c =================================================================== --- linux-2.6.11-openib/drivers/infiniband/core/fmr_pool.c (revision 2334) +++ linux-2.6.11-openib/drivers/infiniband/core/fmr_pool.c (working copy) @@ -329,7 +329,7 @@ EXPORT_SYMBOL(ib_create_fmr_pool); * * Destroy an FMR pool and free all associated resources. */ -int ib_destroy_fmr_pool(struct ib_fmr_pool *pool) +void ib_destroy_fmr_pool(struct ib_fmr_pool *pool) { struct ib_pool_fmr *fmr; struct ib_pool_fmr *tmp; @@ -352,8 +352,6 @@ int ib_destroy_fmr_pool(struct ib_fmr_po kfree(pool->cache_bucket); kfree(pool); - - return 0; } EXPORT_SYMBOL(ib_destroy_fmr_pool); From kkhjLOBQj4dE at hotmail.com Fri May 13 06:49:24 2005 From: kkhjLOBQj4dE at hotmail.com (Rosie Hank) Date: Fri, 13 May 2005 15:49:24 +0200 Subject: [openib-general] PPV for free! Message-ID: <603z7fzlsc.fsf@calle82.net> We offer three products at this point of time, please read the complete email for more information on each. Digital Cable Filters: http://kCRU.1clickresults.info Fuel Savers: http://QLIS.1clicktime.info Ebay Business Traning Program: http://yxOA.1clickaccess.info Free PPV: Unlock: *PPV channels *Boxing and any other Sports Event on PPV *Adult Channels *Movies *And anything else on PPV channels. http://WXmd.1clickresults.info This is the NEWEST AND BEST Digital CATV Filter/Descrambler that can test your digital cable PPV purchase functions along with eliminating unwanted interference caused by your broadband connection. This is a "True" universal product. It will work on 99% of all digital cable systems in use today. This little filter has many different uses: Interference filter, PPV tester and "Digital Descrambler". These will work with the following receivers: Scientific Atlanta Jerrold/General Instruments Zenith Tocom Starcom Explorer Series Motorola Simple to Install. ATTENTION: Because the Cable Company has no way of telling you are using this product you need to notify them of any movie purchases. If you are getting interference on your television set while connected to your broadband connection,simply installing this device prior to your cable converter will dramatically improve your reception. Try us out, You wont regret it! http://rBTw.1clickresults.info --- Maybe your looking for something else? Fuel savers? http://Abub.1clicktime.info Installs in seconds (No tools or experience required) Fuel savings of upto 27% - more milage. One time purchase. The super Ecnofuels that is proven to increase gas mileage upto 27%, it snaps on easily over the fuel line, no tools or experience required. It is a magnetic frequency resonater- Using its patented Neodymium super-conducters it generates the specific resonance frequency, that when installed over the fuel line, makes combustion more efficient. http://JHVf.1clicktime.info -- Or maybe your looking for an interactive eBay Business Training Program on CD? http://SmcS.1clickaccess.info Check us out! Digital Cable Filters: http://mKeP.1clickresults.info Fuel Savers: http://HMuy.1clicktime.info Ebay Business Traning Program: http://Bwuw.1clickaccess.info Regards, Rosie Hank From mshefty at ichips.intel.com Fri May 13 09:30:32 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Fri, 13 May 2005 09:30:32 -0700 Subject: [openib-general] Re: [PATCH] SDP & CM: change some functions to return void In-Reply-To: <1115998078.10240.8.camel@duffman> References: <1115998078.10240.8.camel@duffman> Message-ID: <4284D628.4010801@ichips.intel.com> Tom Duffy wrote: > This patch changes some functions to return void rather than an int > since they were always returning 0, thus making checking return values > rather pointless. The patch touches both SDP and CM. Thanks, Tom. Since the majority of these changes affect SDP, and the check-ins should be coordinated, I'll ask Libor if he can commit the CM changes when he commits the SDP changes. Libor, can you commit the CM changes when you commit the SDP changes? :) There are also some (void)ib_destroy_cm_id casts in umc.c that could lose the casts with this change as well. - Sean From mshefty at ichips.intel.com Fri May 13 09:50:51 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Fri, 13 May 2005 09:50:51 -0700 Subject: [openib-general] [PATCH] CM: add invalid service ID handling In-Reply-To: <78d18e205051012572aec71c9@mail.gmail.com> References: <78d18e205051012572aec71c9@mail.gmail.com> Message-ID: <4284DAEB.6080509@ichips.intel.com> William Jordan wrote: > Patch to respond to invalid REQs (no listener) with a REJ message. As just an FYI in case you're working on other CM areas, I'm working on sending REJs in the case of a stale connection, followed by adding in MRA support. I'm hoping to finish the state connection handling today or Monday. - Sean From jlentini at netapp.com Fri May 13 10:11:50 2005 From: jlentini at netapp.com (James Lentini) Date: Fri, 13 May 2005 13:11:50 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: remove use of DAT_COUNT In-Reply-To: <1115919333.9070.9.camel@duffman> References: <1115919333.9070.9.camel@duffman> Message-ID: Committed in revision 2335. On Thu, 12 May 2005, Tom Duffy wrote: tduffy> Remove DAT_COUNT, just use int. tduffy> tduffy> Plus a few whitespace fixes thrown in for good measure. I am trying to tduffy> clean these things up as I see them. tduffy> tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> Index: linux-kernel/test/dapltest/test/dapl_performance_util.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_performance_util.c (revision 2321) tduffy> +++ linux-kernel/test/dapltest/test/dapl_performance_util.c (working copy) tduffy> @@ -40,7 +40,7 @@ DT_Performance_Test_Create ( tduffy> Performance_Test_t **perf_test) tduffy> { tduffy> Performance_Test_t *test_ptr; tduffy> - DAT_COUNT pipeline_len; tduffy> + int pipeline_len; tduffy> u32 ret; tduffy> DT_Tdep_Print_Head *phead; tduffy> tduffy> @@ -523,8 +523,8 @@ DT_performance_wait ( tduffy> DAT_EVD_HANDLE evd_handle, tduffy> Performance_Stats_t *stats) tduffy> { tduffy> - DAT_COUNT i; tduffy> - DAT_COUNT queue_size; tduffy> + int i; tduffy> + int queue_size; tduffy> u32 ret; tduffy> struct dat_event event; tduffy> unsigned long pre_ctxt_num; tduffy> Index: linux-kernel/test/dapltest/test/dapl_limit.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_limit.c (revision 2321) tduffy> +++ linux-kernel/test/dapltest/test/dapl_limit.c (working copy) tduffy> @@ -1122,8 +1122,8 @@ limit_test ( DT_Tdep_Print_Head *phead, tduffy> */ tduffy> if (depth == LIM_SIZE_LMR) tduffy> { tduffy> - DAT_COUNT last_size = 0; tduffy> - DAT_COUNT test_size = DFLT_BUFFSZ; tduffy> + int last_size = 0; tduffy> + int test_size = DFLT_BUFFSZ; tduffy> Bpool *test_bpool; tduffy> for (;;) tduffy> { tduffy> Index: linux-kernel/test/dapltest/test/dapl_transaction_test.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_transaction_test.c (revision 2321) tduffy> +++ linux-kernel/test/dapltest/test/dapl_transaction_test.c (working copy) tduffy> @@ -742,7 +742,7 @@ retry: tduffy> */ tduffy> { tduffy> struct dat_event event; tduffy> - DAT_COUNT drained = 0; tduffy> + int drained = 0; tduffy> tduffy> dat_ep_reset (test_ptr->ep_context[i].ep_handle); tduffy> do tduffy> Index: linux-kernel/test/dapltest/test/dapl_bpool.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_bpool.c (revision 2321) tduffy> +++ linux-kernel/test/dapltest/test/dapl_bpool.c (working copy) tduffy> @@ -90,16 +90,16 @@ DT_BpoolAlloc ( tduffy> DAT_PZ_HANDLE pz_handle, tduffy> DAT_EP_HANDLE ep_handle, tduffy> DAT_EVD_HANDLE rmr_evd_handle, tduffy> - DAT_COUNT seg_size, tduffy> - DAT_COUNT num_segs, tduffy> - DAT_COUNT alignment, tduffy> + int seg_size, tduffy> + int num_segs, tduffy> + int alignment, tduffy> boolean_t enable_rdma_write, tduffy> boolean_t enable_rdma_read) tduffy> { tduffy> unsigned char *module = "DT_BpoolAlloc"; tduffy> unsigned char *alloc_ptr = NULL; tduffy> Bpool *bpool_ptr = NULL; tduffy> - DAT_COUNT alloc_size, bpool_size; tduffy> + int alloc_size, bpool_size; tduffy> DAT_REGION_DESCRIPTION region; tduffy> u32 ret = DAT_SUCCESS; tduffy> DAT_VLEN bp_len; tduffy> @@ -392,7 +392,7 @@ DT_Bpool_GetRegBuff (Bpool * bpool_ptr, tduffy> } tduffy> tduffy> /*****************************************************************************/ tduffy> -DAT_COUNT tduffy> +int tduffy> DT_Bpool_GetBuffSize (Bpool * bpool_ptr, int index) tduffy> { tduffy> return ( bpool_ptr->seg_size ); tduffy> Index: linux-kernel/test/dapltest/test/dapl_test_util.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_test_util.c (revision 2321) tduffy> +++ linux-kernel/test/dapltest/test/dapl_test_util.c (working copy) tduffy> @@ -573,7 +573,7 @@ boolean_t tduffy> DT_dto_check ( DT_Tdep_Print_Head *phead, tduffy> struct dat_dto_completion_event_data *dto_p, tduffy> DAT_EP_HANDLE ep_expected, tduffy> - DAT_COUNT len_expected, tduffy> + int len_expected, tduffy> DAT_DTO_COOKIE cookie_expected, tduffy> char *message ) tduffy> { tduffy> Index: linux-kernel/test/dapltest/test/dapl_util.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_util.c (revision 2321) tduffy> +++ linux-kernel/test/dapltest/test/dapl_util.c (working copy) tduffy> @@ -189,7 +189,7 @@ DT_State2Str (DAT_EP_STATE state_code) tduffy> * and do the correct thing if align == 0. tduffy> */ tduffy> unsigned char * tduffy> -DT_AlignPtr (void * val, DAT_COUNT align) tduffy> +DT_AlignPtr (void * val, int align) tduffy> { tduffy> if (align) tduffy> { tduffy> @@ -199,8 +199,8 @@ DT_AlignPtr (void * val, DAT_COUNT align tduffy> return (val); tduffy> } tduffy> tduffy> -DAT_COUNT tduffy> -DT_RoundSize (DAT_COUNT val, DAT_COUNT align) tduffy> +int tduffy> +DT_RoundSize (int val, int align) tduffy> { tduffy> if (align) tduffy> { tduffy> Index: linux-kernel/test/dapltest/test/dapl_client.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/test/dapl_client.c (revision 2321) tduffy> +++ linux-kernel/test/dapltest/test/dapl_client.c (working copy) tduffy> @@ -276,7 +276,7 @@ retry: tduffy> if ( event_num == DAT_CONNECTION_EVENT_PEER_REJECTED ) tduffy> { tduffy> struct dat_event event; tduffy> - DAT_COUNT drained = 0; tduffy> + int drained = 0; tduffy> tduffy> DT_Mdep_Sleep (1000); tduffy> DT_Tdep_PT_Printf (phead, tduffy> Index: linux-kernel/test/dapltest/include/dapl_tdep.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/include/dapl_tdep.h (revision 2321) tduffy> +++ linux-kernel/test/dapltest/include/dapl_tdep.h (working copy) tduffy> @@ -46,7 +46,7 @@ DT_Tdep_Execute_Test ( Params_t *params_ tduffy> tduffy> u32 tduffy> DT_Tdep_evd_create (DAT_IA_HANDLE ia_handle, tduffy> - DAT_COUNT evd_min_qlen, tduffy> + int evd_min_qlen, tduffy> DAT_CNO_HANDLE cno_handle, tduffy> DAT_EVD_FLAGS evd_flags, tduffy> DAT_EVD_HANDLE *evd_handle_ptr); tduffy> Index: linux-kernel/test/dapltest/include/dapl_global.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/include/dapl_global.h (revision 2321) tduffy> +++ linux-kernel/test/dapltest/include/dapl_global.h (working copy) tduffy> @@ -30,7 +30,7 @@ tduffy> tduffy> #include "dapl_proto.h" tduffy> tduffy> -extern DAT_COUNT DT_dapltest_debug; tduffy> +extern int DT_dapltest_debug; tduffy> extern boolean_t DT_local_is_little_endian; tduffy> tduffy> #endif tduffy> Index: linux-kernel/test/dapltest/include/dapl_performance_test.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/include/dapl_performance_test.h (revision 2321) tduffy> +++ linux-kernel/test/dapltest/include/dapl_performance_test.h (working copy) tduffy> @@ -57,7 +57,7 @@ typedef struct tduffy> DAT_EP_HANDLE ep_handle; tduffy> struct dat_ep_attr ep_attr; tduffy> DAT_CONN_QUAL port; tduffy> - DAT_COUNT pipeline_len; tduffy> + int pipeline_len; tduffy> Bpool *bp; tduffy> Performance_Test_Op_t op; tduffy> } Performance_Ep_Context_t; tduffy> @@ -73,13 +73,13 @@ typedef struct tduffy> DAT_IA_HANDLE ia_handle; tduffy> DAT_PZ_HANDLE pz_handle; tduffy> DAT_CNO_HANDLE cno_handle; tduffy> - DAT_COUNT reqt_evd_length; tduffy> + int reqt_evd_length; tduffy> DAT_EVD_HANDLE reqt_evd_hdl; /* request+rmr */ tduffy> - DAT_COUNT recv_evd_length; tduffy> + int recv_evd_length; tduffy> DAT_EVD_HANDLE recv_evd_hdl; /* receive */ tduffy> - DAT_COUNT conn_evd_length; tduffy> + int conn_evd_length; tduffy> DAT_EVD_HANDLE conn_evd_hdl; /* connect */ tduffy> - DAT_COUNT creq_evd_length; tduffy> + int creq_evd_length; tduffy> DAT_EVD_HANDLE creq_evd_hdl; /* "" request */ tduffy> Performance_Ep_Context_t ep_context; tduffy> } Performance_Test_t; tduffy> Index: linux-kernel/test/dapltest/include/dapl_proto.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/include/dapl_proto.h (revision 2321) tduffy> +++ linux-kernel/test/dapltest/include/dapl_proto.h (working copy) tduffy> @@ -85,9 +85,9 @@ Bpool * DT_BpoolAlloc (Per_Test_ tduffy> DAT_PZ_HANDLE pz_handle, tduffy> DAT_EP_HANDLE ep_handle, tduffy> DAT_EVD_HANDLE rmr_evd_handle, tduffy> - DAT_COUNT seg_size, tduffy> - DAT_COUNT num_segs, tduffy> - DAT_COUNT alignment, tduffy> + int seg_size, tduffy> + int num_segs, tduffy> + int alignment, tduffy> boolean_t enable_rdma_write, tduffy> boolean_t enable_rdma_read); tduffy> tduffy> @@ -97,7 +97,7 @@ boolean_t DT_Bpool_Destroy (P tduffy> tduffy> unsigned char *DT_Bpool_GetBuffer (Bpool * bpool_ptr, int index); tduffy> DAT_VADDR DT_Bpool_GetRegBuff (Bpool * bpool_ptr, int index); tduffy> -DAT_COUNT DT_Bpool_GetBuffSize (Bpool * bpool_ptr, int index); tduffy> +int DT_Bpool_GetBuffSize (Bpool * bpool_ptr, int index); tduffy> struct dat_lmr_triplet *DT_Bpool_GetIOV (Bpool * bpool_ptr, int index); tduffy> DAT_LMR_CONTEXT DT_Bpool_GetLMR (Bpool * bpool_ptr, int index); tduffy> DAT_RMR_CONTEXT DT_Bpool_GetRMR (Bpool * bpool_ptr, int index); tduffy> @@ -398,7 +398,7 @@ boolean_t DT_rmr_event_wait ( tduffy> boolean_t DT_dto_check ( DT_Tdep_Print_Head* phead, tduffy> struct dat_dto_completion_event_data *dto_p, tduffy> DAT_EP_HANDLE ep_expected, tduffy> - DAT_COUNT len_expected, tduffy> + int len_expected, tduffy> DAT_DTO_COOKIE cookie_expected, tduffy> char *message); tduffy> tduffy> @@ -537,9 +537,9 @@ DAT_QOS DT_ParseQoS (char *arg); tduffy> tduffy> DAT_MEM_TYPE DT_ParseMemType (char *arg); tduffy> tduffy> -unsigned char *DT_AlignPtr (void * val, DAT_COUNT align); tduffy> +unsigned char *DT_AlignPtr (void * val, int align); tduffy> tduffy> -DAT_COUNT DT_RoundSize (DAT_COUNT val, DAT_COUNT align); tduffy> +int DT_RoundSize (int val, int align); tduffy> tduffy> /* dapl_limit_cmd.c */ tduffy> void DT_Limit_Cmd_Init ( Limit_Cmd_t * cmd); tduffy> Index: linux-kernel/test/dapltest/include/dapl_bpool.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/include/dapl_bpool.h (revision 2321) tduffy> +++ linux-kernel/test/dapltest/include/dapl_bpool.h (working copy) tduffy> @@ -39,8 +39,8 @@ struct Bpool_tag { tduffy> unsigned char *alloc_ptr; tduffy> uint32_t alloc_size; tduffy> DAT_PZ_HANDLE pz_handle; tduffy> - DAT_COUNT seg_size; tduffy> - DAT_COUNT num_segs; /* num segments */ tduffy> + int seg_size; tduffy> + int num_segs; /* num segments */ tduffy> unsigned char *buffer_start; /* Start of buffer area */ tduffy> DAT_VLEN buffer_size; /* Size of data buffer (rounded) */ tduffy> DAT_VADDR reg_addr; /* start of registered area */ tduffy> Index: linux-kernel/test/dapltest/include/dapl_fft_util.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/include/dapl_fft_util.h (revision 2321) tduffy> +++ linux-kernel/test/dapltest/include/dapl_fft_util.h (working copy) tduffy> @@ -61,7 +61,7 @@ typedef struct tduffy> DAT_EP_HANDLE ep_handle; tduffy> DAT_EVD_HANDLE cr_evd, conn_evd, send_evd, recv_evd; tduffy> struct dat_event event; tduffy> - DAT_COUNT count; tduffy> + int count; tduffy> DAT_CR_HANDLE cr_handle; tduffy> Bpool *bpool; tduffy> struct dat_cr_arrival_event_data cr_stat; tduffy> Index: linux-kernel/test/dapltest/include/dapl_performance_stats.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/include/dapl_performance_stats.h (revision 2321) tduffy> +++ linux-kernel/test/dapltest/include/dapl_performance_stats.h (working copy) tduffy> @@ -58,7 +58,7 @@ typedef struct tduffy> Performance_Stats_Data_t latency; tduffy> DT_CpuStat pre_cpu_stat; tduffy> DT_CpuStat post_cpu_stat; tduffy> - DAT_COUNT pipeline_len; tduffy> + int pipeline_len; tduffy> } Performance_Stats_t; tduffy> tduffy> #endif tduffy> Index: linux-kernel/test/dapltest/include/dapl_transaction_test.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/include/dapl_transaction_test.h (revision 2321) tduffy> +++ linux-kernel/test/dapltest/include/dapl_transaction_test.h (working copy) tduffy> @@ -80,7 +80,7 @@ typedef struct tduffy> DAT_IA_ADDRESS_PTR remote_ia_addr; tduffy> DAT_CONN_QUAL base_port; tduffy> DAT_TIMEOUT time_out; tduffy> - DAT_COUNT evd_length; tduffy> + int evd_length; tduffy> Thread *thread; tduffy> tduffy> /* This group set up by each thread in DT_Transaction_Main() */ tduffy> Index: linux-kernel/test/dapltest/cmd/dapl_test_data.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/cmd/dapl_test_data.c (revision 2321) tduffy> +++ linux-kernel/test/dapltest/cmd/dapl_test_data.c (working copy) tduffy> @@ -27,7 +27,7 @@ tduffy> tduffy> #include "dapl_proto.h" tduffy> tduffy> -DAT_COUNT DT_dapltest_debug = 0; tduffy> +int DT_dapltest_debug = 0; tduffy> /* tduffy> * check memory leaking int alloc_count; DT_Mdep_LockType tduffy> * Alloc_Count_Lock; tduffy> Index: linux-kernel/test/dapltest/common/dapl_global.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/common/dapl_global.c (revision 2321) tduffy> +++ linux-kernel/test/dapltest/common/dapl_global.c (working copy) tduffy> @@ -28,6 +28,6 @@ tduffy> #include "dapl_proto.h" tduffy> tduffy> boolean_t DT_local_is_little_endian; tduffy> -DAT_COUNT DT_dapltest_debug = 0; tduffy> +int DT_dapltest_debug = 0; tduffy> tduffy> tduffy> Index: linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c (revision 2321) tduffy> +++ linux-kernel/test/dapltest/kdapl/kdapl_tdep_evd.c (working copy) tduffy> @@ -96,7 +96,7 @@ KDT_Evd_Destroy(void) tduffy> tduffy> u32 tduffy> DT_Tdep_evd_create (DAT_IA_HANDLE ia_handle, tduffy> - DAT_COUNT evd_min_qlen, tduffy> + int evd_min_qlen, tduffy> DAT_CNO_HANDLE cno_handle, tduffy> DAT_EVD_FLAGS evd_flags, tduffy> DAT_EVD_HANDLE *evd_handle_ptr) tduffy> Index: linux-kernel/dat-provider/dapl_cookie.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_cookie.c (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl_cookie.c (working copy) tduffy> @@ -113,9 +113,9 @@ u32 dapl_cb_put(DAPL_COOKIE_BUFFER *buff tduffy> * DAT_INSUFFICIENT_RESOURCES tduffy> * tduffy> */ tduffy> -u32 dapl_cb_create(DAPL_COOKIE_BUFFER *buffer, DAPL_EP *ep, DAT_COUNT size) tduffy> +u32 dapl_cb_create(DAPL_COOKIE_BUFFER *buffer, DAPL_EP *ep, int size) tduffy> { tduffy> - DAT_COUNT i; tduffy> + int i; tduffy> tduffy> /* tduffy> * allocate one additional entry so that the tail tduffy> @@ -181,7 +181,7 @@ void dapl_cb_free(DAPL_COOKIE_BUFFER * b tduffy> u32 dapl_cb_get(DAPL_COOKIE_BUFFER *buffer, DAPL_COOKIE **cookie_ptr) tduffy> { tduffy> u32 dat_status; tduffy> - DAT_COUNT new_head; tduffy> + int new_head; tduffy> tduffy> dapl_os_assert(NULL != cookie_ptr); tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_ep_connect.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_ep_connect.c (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl_ep_connect.c (working copy) tduffy> @@ -45,7 +45,7 @@ tduffy> u32 dapl_ep_connect(DAT_EP_HANDLE ep_handle, tduffy> DAT_IA_ADDRESS_PTR remote_ia_address, tduffy> DAT_CONN_QUAL remote_conn_qual, DAT_TIMEOUT timeout, tduffy> - DAT_COUNT private_data_size, const void *private_data, tduffy> + int private_data_size, const void *private_data, tduffy> DAT_QOS qos, DAT_CONNECT_FLAGS connect_flags) tduffy> { tduffy> DAPL_EP *ep_ptr; tduffy> Index: linux-kernel/dat-provider/dapl_cookie.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_cookie.h (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl_cookie.h (working copy) tduffy> @@ -37,7 +37,7 @@ tduffy> #include "dapl.h" tduffy> tduffy> extern u32 dapl_cb_create(DAPL_COOKIE_BUFFER *buffer, DAPL_EP *ep, tduffy> - DAT_COUNT size); tduffy> + int size); tduffy> tduffy> extern void dapl_cb_free(DAPL_COOKIE_BUFFER *buffer); tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_module.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_module.c (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl_module.c (working copy) tduffy> @@ -174,7 +174,7 @@ void DAT_PROVIDER_INIT_FUNC_NAME(const s tduffy> } tduffy> printk("KDAPL : Added device %s to the registry\n", tduffy> provider_info->ia_name); tduffy> - bail: tduffy> +bail: tduffy> if (DAT_SUCCESS != dat_status) { tduffy> if (NULL != provider) { tduffy> (void)dapl_provider_list_remove(provider_info->ia_name); tduffy> @@ -220,7 +220,7 @@ void DAT_PROVIDER_FINI_FUNC_NAME(const s tduffy> */ tduffy> dapl_hca_free(provider->extension); tduffy> tduffy> - (void)dapl_provider_list_remove(provider_info->ia_name); tduffy> + dapl_provider_list_remove(provider_info->ia_name); tduffy> } tduffy> tduffy> static struct ib_client dapl_client = { tduffy> @@ -237,9 +237,8 @@ static char *dev_name_suffix_table[3] = tduffy> tduffy> void dapl_add_one(struct ib_device *device) tduffy> { tduffy> - DAT_COUNT i; tduffy> + int i, length; tduffy> struct dat_provider_info provider_info; tduffy> - int length; tduffy> dapl_dbg_log(DAPL_DBG_TYPE_UTIL, tduffy> "Inside dapl_add_one called by %s\n", device->name); tduffy> tduffy> @@ -254,14 +253,12 @@ void dapl_add_one(struct ib_device *devi tduffy> provider_info.is_thread_safe = FALSE; tduffy> DAT_PROVIDER_INIT_FUNC_NAME(&provider_info, device, i); tduffy> } tduffy> - return; tduffy> } tduffy> tduffy> void dapl_remove_one(struct ib_device *device) tduffy> { tduffy> - DAT_COUNT i; tduffy> + int i, length; tduffy> struct dat_provider_info provider_info; tduffy> - int length; tduffy> dapl_dbg_log(DAPL_DBG_TYPE_UTIL, tduffy> "Inside dapl_remove_one called by %s\n", device->name); tduffy> tduffy> @@ -269,14 +266,12 @@ void dapl_remove_one(struct ib_device *d tduffy> length = strlen(device->name); tduffy> for (i = 0; i < device->phys_port_cnt; i++) { tduffy> strncpy(provider_info.ia_name, device->name, length + 1); tduffy> - (void)strcat(provider_info.ia_name, tduffy> - dev_name_suffix_table[i + 1]); tduffy> + strcat(provider_info.ia_name, dev_name_suffix_table[i + 1]); tduffy> provider_info.dat_version_major = DAT_VERSION_MAJOR; tduffy> provider_info.dat_version_minor = DAT_VERSION_MINOR; tduffy> provider_info.is_thread_safe = FALSE; tduffy> DAT_PROVIDER_FINI_FUNC_NAME(&provider_info); tduffy> } tduffy> - return; tduffy> } tduffy> tduffy> /* tduffy> @@ -289,11 +284,9 @@ static void dapl_register_providers(void tduffy> status = ib_register_client(&dapl_client); tduffy> if (status < 0) tduffy> printk("ib_register_client () failed: %x\n", status); tduffy> - return; tduffy> }; tduffy> tduffy> static void dapl_unregister_providers(void) tduffy> { tduffy> - (void)ib_unregister_client(&dapl_client); tduffy> - return; tduffy> + ib_unregister_client(&dapl_client); tduffy> } tduffy> Index: linux-kernel/dat-provider/dapl_ring_buffer_util.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_ring_buffer_util.c (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl_ring_buffer_util.c (working copy) tduffy> @@ -56,7 +56,7 @@ tduffy> * DAT_INSUFFICIENT_RESOURCES tduffy> * tduffy> */ tduffy> -u32 dapl_rbuf_alloc(DAPL_RING_BUFFER *rbuf, DAT_COUNT size) tduffy> +u32 dapl_rbuf_alloc(DAPL_RING_BUFFER *rbuf, int size) tduffy> { tduffy> unsigned int rsize; /* real size */ tduffy> tduffy> @@ -68,7 +68,7 @@ u32 dapl_rbuf_alloc(DAPL_RING_BUFFER *rb tduffy> tduffy> /* Put size on a power of 2 boundary */ tduffy> rsize = 1; tduffy> - while ((DAT_COUNT) rsize < size) { tduffy> + while ((int) rsize < size) { tduffy> rsize <<= 1; tduffy> } tduffy> tduffy> @@ -104,7 +104,7 @@ u32 dapl_rbuf_alloc(DAPL_RING_BUFFER *rb tduffy> * DAT_INSUFFICIENT_RESOURCES tduffy> * tduffy> */ tduffy> -u32 dapl_rbuf_realloc(DAPL_RING_BUFFER *rbuf, DAT_COUNT size) tduffy> +u32 dapl_rbuf_realloc(DAPL_RING_BUFFER *rbuf, int size) tduffy> { tduffy> DAPL_RING_BUFFER new_rbuf; tduffy> void *entry; tduffy> @@ -261,11 +261,9 @@ void *dapl_rbuf_remove(DAPL_RING_BUFFER tduffy> * count of entries tduffy> * tduffy> */ tduffy> -DAT_COUNT dapl_rbuf_count(DAPL_RING_BUFFER * rbuf) tduffy> +int dapl_rbuf_count(DAPL_RING_BUFFER * rbuf) tduffy> { tduffy> - DAT_COUNT count; tduffy> - int head; tduffy> - int tail; tduffy> + int count, head, tail; tduffy> tduffy> head = atomic_read(&rbuf->head) & rbuf->lim; tduffy> tail = atomic_read(&rbuf->tail) & rbuf->lim; tduffy> Index: linux-kernel/dat-provider/dapl_srq_resize.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_srq_resize.c (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl_srq_resize.c (working copy) tduffy> @@ -63,7 +63,7 @@ tduffy> * DAT_INVALID_STATE tduffy> */ tduffy> tduffy> -u32 dapl_srq_resize(DAT_SRQ_HANDLE srq_handle, DAT_COUNT srq_max_recv_dto) tduffy> +u32 dapl_srq_resize(DAT_SRQ_HANDLE srq_handle, int srq_max_recv_dto) tduffy> { tduffy> DAPL_IA *ia_ptr; tduffy> DAPL_SRQ *srq_ptr; tduffy> Index: linux-kernel/dat-provider/dapl_ep_post_rdma_write.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_ep_post_rdma_write.c (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl_ep_post_rdma_write.c (working copy) tduffy> @@ -66,7 +66,7 @@ tduffy> * DAT_PROTECTION_VIOLATION tduffy> * DAT_PRIVILEGES_VIOLATION tduffy> */ tduffy> -u32 dapl_ep_post_rdma_write(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, tduffy> +u32 dapl_ep_post_rdma_write(DAT_EP_HANDLE ep_handle, int num_segments, tduffy> struct dat_lmr_triplet *local_iov, tduffy> DAT_DTO_COOKIE user_cookie, tduffy> const struct dat_rmr_triplet *remote_iov, tduffy> Index: linux-kernel/dat-provider/dapl_openib_dto.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_openib_dto.h (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl_openib_dto.h (working copy) tduffy> @@ -51,7 +51,7 @@ tduffy> * Provider specific Post RECV function tduffy> */ tduffy> static __inline__ u32 dapl_ib_post_recv(DAPL_EP *ep_ptr, DAPL_COOKIE *cookie, tduffy> - DAT_COUNT num_segments, tduffy> + int num_segments, tduffy> struct dat_lmr_triplet *local_iov) tduffy> { tduffy> struct ib_recv_wr rr_desc = { NULL }; tduffy> @@ -101,7 +101,7 @@ static __inline__ u32 dapl_ib_post_recv( tduffy> */ tduffy> static __inline__ u32 dapl_ib_post_send(DAPL_EP *ep_ptr, int op_type, tduffy> DAPL_COOKIE *cookie, tduffy> - DAT_COUNT num_segments, tduffy> + int num_segments, tduffy> struct dat_lmr_triplet *local_iov, tduffy> const struct dat_rmr_triplet *remote_iov, tduffy> DAT_COMPLETION_FLAGS completion_flags) tduffy> Index: linux-kernel/dat-provider/dapl_ring_buffer_util.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_ring_buffer_util.h (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl_ring_buffer_util.h (working copy) tduffy> @@ -42,9 +42,9 @@ tduffy> /* tduffy> * Prototypes tduffy> */ tduffy> -u32 dapl_rbuf_alloc(DAPL_RING_BUFFER *rbuf, DAT_COUNT size); tduffy> +u32 dapl_rbuf_alloc(DAPL_RING_BUFFER *rbuf, int size); tduffy> tduffy> -u32 dapl_rbuf_realloc(DAPL_RING_BUFFER *rbuf, DAT_COUNT size); tduffy> +u32 dapl_rbuf_realloc(DAPL_RING_BUFFER *rbuf, int size); tduffy> tduffy> void dapl_rbuf_destroy(DAPL_RING_BUFFER * rbuf); tduffy> tduffy> @@ -52,7 +52,7 @@ u32 dapl_rbuf_add(DAPL_RING_BUFFER *rbuf tduffy> tduffy> void *dapl_rbuf_remove(DAPL_RING_BUFFER * rbuf); tduffy> tduffy> -DAT_COUNT dapl_rbuf_count(DAPL_RING_BUFFER * rbuf); tduffy> +int dapl_rbuf_count(DAPL_RING_BUFFER *rbuf); tduffy> tduffy> void dapl_rbuf_adjust(DAPL_RING_BUFFER * rbuf, intptr_t offset); tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_evd_resize.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_evd_resize.c (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl_evd_resize.c (working copy) tduffy> @@ -63,11 +63,11 @@ tduffy> * DAT_INVALID_STATE tduffy> */ tduffy> tduffy> -u32 dapl_evd_resize(DAT_EVD_HANDLE evd_handle, DAT_COUNT evd_qlen) tduffy> +u32 dapl_evd_resize(DAT_EVD_HANDLE evd_handle, int evd_qlen) tduffy> { tduffy> DAPL_IA *ia_ptr; tduffy> DAPL_EVD *evd_ptr; tduffy> - DAT_COUNT pend_cnt; tduffy> + int pend_cnt; tduffy> u32 dat_status; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_evd_resize (%p, %d)\n", tduffy> Index: linux-kernel/dat-provider/dapl_adapter_util.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_adapter_util.h (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl_adapter_util.h (working copy) tduffy> @@ -51,7 +51,7 @@ typedef enum async_handler_type { tduffy> tduffy> u32 dapl_ib_enum_hcas(const char *vendor, tduffy> DAPL_HCA_NAME **hca_names, tduffy> - DAT_COUNT *total_hca_count); tduffy> + int *total_hca_count); tduffy> tduffy> u32 dapl_ib_get_instance_data(DAPL_HCA_NAME hca_name, char *instance); tduffy> tduffy> @@ -69,7 +69,7 @@ u32 dapl_ib_qp_modify(DAPL_IA *ia_ptr, D tduffy> u32 dapl_ib_connect(DAT_EP_HANDLE ep_handle, tduffy> DAT_IA_ADDRESS_PTR remote_ia_address, tduffy> DAT_CONN_QUAL remote_conn_qual, tduffy> - DAT_COUNT private_data_size, void *private_data); tduffy> + int private_data_size, void *private_data); tduffy> tduffy> u32 dapl_ib_disconnect(DAPL_EP *ep_ptr, DAT_CLOSE_FLAGS close_flags); tduffy> tduffy> @@ -80,7 +80,7 @@ u32 dapl_ib_remove_conn_listener(DAPL_IA tduffy> tduffy> u32 dapl_ib_accept_connection(DAT_CR_HANDLE cr_handle, tduffy> DAT_EP_HANDLE ep_handle, tduffy> - DAT_COUNT private_data_size, tduffy> + int private_data_size, tduffy> const void *private_data); tduffy> tduffy> u32 dapl_ib_reject_connection(ib_cm_handle_t cm_handle, int reject_reason); tduffy> @@ -89,14 +89,14 @@ u32 dapl_ib_setup_async_callback(DAPL_IA tduffy> DAPL_ASYNC_HANDLER_TYPE handler_type, tduffy> ib_async_handler_t callback, void *context); tduffy> tduffy> -u32 dapl_ib_cq_alloc(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr, DAT_COUNT *cqlen, tduffy> +u32 dapl_ib_cq_alloc(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr, int *cqlen, tduffy> ib_comp_handle_t callback); tduffy> tduffy> u32 dapl_ib_cq_free(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr); tduffy> tduffy> u32 dapl_set_cq_notify(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr); tduffy> tduffy> -u32 dapl_ib_cq_resize(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr, DAT_COUNT *cqlen); tduffy> +u32 dapl_ib_cq_resize(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr, int *cqlen); tduffy> tduffy> u32 dapl_ib_pd_alloc(DAPL_IA *ia_ptr, DAPL_PZ *pz); tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_ep_post_send.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_ep_post_send.c (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl_ep_post_send.c (working copy) tduffy> @@ -63,7 +63,7 @@ tduffy> * DAT_PROTECTION_VIOLATION tduffy> * DAT_PRIVILEGES_VIOLATION tduffy> */ tduffy> -u32 dapl_ep_post_send(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, tduffy> +u32 dapl_ep_post_send(DAT_EP_HANDLE ep_handle, int num_segments, tduffy> struct dat_lmr_triplet *local_iov, tduffy> DAT_DTO_COOKIE user_cookie, tduffy> DAT_COMPLETION_FLAGS completion_flags) tduffy> Index: linux-kernel/dat-provider/dapl_provider.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_provider.c (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl_provider.c (working copy) tduffy> @@ -204,7 +204,7 @@ u32 dapl_provider_list_destroy(void) tduffy> return DAT_SUCCESS; tduffy> } tduffy> tduffy> -DAT_COUNT dapl_provider_list_size(void) tduffy> +int dapl_provider_list_size(void) tduffy> { tduffy> return g_dapl_provider_list.size; tduffy> } tduffy> Index: linux-kernel/dat-provider/dapl_openib_cm.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_openib_cm.c (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl_openib_cm.c (working copy) tduffy> @@ -396,7 +396,7 @@ error: tduffy> u32 dapl_ib_connect(DAT_EP_HANDLE ep_handle, tduffy> DAT_IA_ADDRESS_PTR remote_ia_address, tduffy> DAT_CONN_QUAL remote_conn_qual, tduffy> - DAT_COUNT private_data_size, void *private_data) tduffy> + int private_data_size, void *private_data) tduffy> { tduffy> DAPL_IA *ia_ptr; tduffy> DAPL_EP *ep_ptr; tduffy> @@ -732,7 +732,7 @@ u32 dapl_ib_reject_connection(struct dap tduffy> * tduffy> */ tduffy> u32 dapl_ib_accept_connection(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, tduffy> - DAT_COUNT private_data_size, tduffy> + int private_data_size, tduffy> const void *prd_ptr) tduffy> { tduffy> DAPL_CR *cr_ptr; tduffy> Index: linux-kernel/dat-provider/dapl_provider.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_provider.h (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl_provider.h (working copy) tduffy> @@ -56,7 +56,7 @@ typedef struct DAPL_PROVIDER_LIST_NODE { tduffy> typedef struct DAPL_PROVIDER_LIST { tduffy> DAPL_PROVIDER_LIST_NODE *head; tduffy> DAPL_PROVIDER_LIST_NODE *tail; tduffy> - DAT_COUNT size; tduffy> + int size; tduffy> } DAPL_PROVIDER_LIST; tduffy> tduffy> /********************************************************************* tduffy> @@ -78,7 +78,7 @@ extern u32 dapl_provider_list_create(voi tduffy> tduffy> extern u32 dapl_provider_list_destroy(void); tduffy> tduffy> -extern DAT_COUNT dapl_provider_list_size(void); tduffy> +extern int dapl_provider_list_size(void); tduffy> tduffy> extern u32 dapl_provider_list_insert(const char *name, tduffy> struct dat_provider **p_data); tduffy> Index: linux-kernel/dat-provider/dapl_ep_recv_query.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_ep_recv_query.c (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl_ep_recv_query.c (working copy) tduffy> @@ -55,7 +55,7 @@ tduffy> * DAT_MODEL_NOT_SUPPORTED tduffy> */ tduffy> u32 dapl_ep_recv_query(DAT_EP_HANDLE ep_handle, tduffy> - DAT_COUNT *nbufs_allocate, DAT_COUNT *bufs_alloc_span) tduffy> + int *nbufs_allocate, int *bufs_alloc_span) tduffy> { tduffy> DAPL_EP *ep_ptr; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> Index: linux-kernel/dat-provider/dapl_openib_util.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_openib_util.c (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl_openib_util.c (working copy) tduffy> @@ -221,7 +221,7 @@ u32 dapl_ib_close_hca(DAPL_HCA *hca_ptr) tduffy> * DAT_INSUFFICIENT_RESOURCES tduffy> * tduffy> */ tduffy> -u32 dapl_ib_cq_alloc(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr, DAT_COUNT *cqlen, tduffy> +u32 dapl_ib_cq_alloc(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr, int *cqlen, tduffy> ib_comp_handle_t callback) tduffy> { tduffy> int ib_status; tduffy> @@ -263,7 +263,7 @@ u32 dapl_ib_cq_free(DAPL_IA *ia_ptr, DAP tduffy> return dapl_ib_status_convert(ib_status); tduffy> } tduffy> tduffy> -u32 dapl_ib_cq_resize(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr, DAT_COUNT *cqlen) tduffy> +u32 dapl_ib_cq_resize(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr, int *cqlen) tduffy> { tduffy> int ib_status; tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_evd_kcreate.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_evd_kcreate.c (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl_evd_kcreate.c (working copy) tduffy> @@ -48,7 +48,7 @@ tduffy> * even if it is not required. However, it will not be armed. tduffy> */ tduffy> tduffy> -u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_handle, DAT_COUNT evd_min_qlen, tduffy> +u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_handle, int evd_min_qlen, tduffy> DAT_UPCALL_POLICY upcall_policy, /* ignore */ tduffy> const struct dat_upcall_object *upcall, tduffy> DAT_EVD_FLAGS evd_flags, DAT_EVD_HANDLE *evd_handle) tduffy> Index: linux-kernel/dat-provider/dapl_srq_set_lw.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_srq_set_lw.c (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl_srq_set_lw.c (working copy) tduffy> @@ -63,7 +63,7 @@ tduffy> * DAT_MODEL_NOT_SUPPORTED tduffy> */ tduffy> tduffy> -u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, DAT_COUNT low_watermark) tduffy> +u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, int low_watermark) tduffy> { tduffy> DAPL_SRQ *srq_ptr; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> Index: linux-kernel/dat-provider/dapl_cr_accept.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_cr_accept.c (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl_cr_accept.c (working copy) tduffy> @@ -40,7 +40,7 @@ tduffy> * and passive side (local endpoint). tduffy> */ tduffy> u32 dapl_cr_accept(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, tduffy> - DAT_COUNT private_data_size, const void *private_data) tduffy> + int private_data_size, const void *private_data) tduffy> { tduffy> DAPL_EP *ep_ptr; tduffy> u32 dat_status; tduffy> Index: linux-kernel/dat-provider/dapl_srq_post_recv.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_srq_post_recv.c (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl_srq_post_recv.c (working copy) tduffy> @@ -67,7 +67,7 @@ tduffy> * DAT_PROTECTION_VIOLATION tduffy> * DAT_PROVILEGES_VIOLATION tduffy> */ tduffy> -u32 dapl_srq_post_recv(DAT_SRQ_HANDLE srq_handle, DAT_COUNT num_segments, tduffy> +u32 dapl_srq_post_recv(DAT_SRQ_HANDLE srq_handle, int num_segments, tduffy> struct dat_lmr_triplet *local_iov, tduffy> DAT_DTO_COOKIE user_cookie) tduffy> { tduffy> Index: linux-kernel/dat-provider/dapl_ep_util.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_ep_util.c (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl_ep_util.c (working copy) tduffy> @@ -281,7 +281,7 @@ u32 dapl_ep_check_request_completion_fla tduffy> return DAT_SUCCESS; tduffy> } tduffy> tduffy> -u32 dapl_ep_post_send_req(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, tduffy> +u32 dapl_ep_post_send_req(DAT_EP_HANDLE ep_handle, int num_segments, tduffy> struct dat_lmr_triplet *local_iov, tduffy> DAT_DTO_COOKIE user_cookie, tduffy> const struct dat_rmr_triplet *remote_iov, tduffy> Index: linux-kernel/dat-provider/dapl_ep_util.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_ep_util.h (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl_ep_util.h (working copy) tduffy> @@ -50,8 +50,7 @@ extern u32 dapl_ep_check_recv_completion tduffy> tduffy> extern u32 dapl_ep_check_request_completion_flags(DAT_COMPLETION_FLAGS flags); tduffy> tduffy> -extern u32 dapl_ep_post_send_req(DAT_EP_HANDLE ep_handle, tduffy> - DAT_COUNT num_segments, tduffy> +extern u32 dapl_ep_post_send_req(DAT_EP_HANDLE ep_handle, int num_segments, tduffy> struct dat_lmr_triplet *local_iov, tduffy> DAT_DTO_COOKIE user_cookie, tduffy> const struct dat_rmr_triplet *remote_iov, tduffy> Index: linux-kernel/dat-provider/dapl_ep_post_rdma_read.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_ep_post_rdma_read.c (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl_ep_post_rdma_read.c (working copy) tduffy> @@ -66,7 +66,7 @@ tduffy> * DAT_PROTECTION_VIOLATION tduffy> * DAT_PRIVILEGES_VIOLATION tduffy> */ tduffy> -u32 dapl_ep_post_rdma_read(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, tduffy> +u32 dapl_ep_post_rdma_read(DAT_EP_HANDLE ep_handle, int num_segments, tduffy> struct dat_lmr_triplet *local_iov, tduffy> DAT_DTO_COOKIE user_cookie, tduffy> const struct dat_rmr_triplet *remote_iov, tduffy> Index: linux-kernel/dat-provider/dapl_ia_open.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_ia_open.c (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl_ia_open.c (working copy) tduffy> @@ -51,7 +51,7 @@ tduffy> * That function maps the DAT_NAME parameter of dat_ia_open to a DAT_PROVIDER, tduffy> * and calls this function. tduffy> */ tduffy> -u32 dapl_ia_open(const DAT_NAME_PTR name, DAT_COUNT async_evd_qlen, tduffy> +u32 dapl_ia_open(const DAT_NAME_PTR name, int async_evd_qlen, tduffy> DAT_EVD_HANDLE *async_evd_handle_ptr, tduffy> DAT_IA_HANDLE *ia_handle_ptr) tduffy> { tduffy> Index: linux-kernel/dat-provider/dapl_evd_util.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_evd_util.c (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl_evd_util.c (working copy) tduffy> @@ -41,7 +41,7 @@ tduffy> tduffy> static __inline__ void dapl_evd_eh_print_cqe(ib_work_completion_t * cqe); tduffy> tduffy> -u32 dapl_evd_event_alloc(DAPL_EVD *evd_ptr, DAT_COUNT qlen); tduffy> +u32 dapl_evd_event_alloc(DAPL_EVD *evd_ptr, int qlen); tduffy> tduffy> /* tduffy> * actually create the evd. this is called after all parameter checking tduffy> @@ -50,11 +50,11 @@ u32 dapl_evd_event_alloc(DAPL_EVD *evd_p tduffy> */ tduffy> tduffy> u32 dapl_evd_internal_create(DAPL_IA *ia_ptr, DAPL_CNO *cno_ptr, tduffy> - DAT_COUNT min_qlen, DAT_EVD_FLAGS evd_flags, tduffy> + int min_qlen, DAT_EVD_FLAGS evd_flags, tduffy> DAPL_EVD **evd_ptr_ptr) tduffy> { tduffy> DAPL_EVD *evd_ptr; tduffy> - DAT_COUNT cq_len; tduffy> + int cq_len; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> *evd_ptr_ptr = NULL; tduffy> @@ -113,7 +113,7 @@ u32 dapl_evd_internal_create(DAPL_IA *ia tduffy> dapl_ia_link_evd(ia_ptr, evd_ptr); tduffy> *evd_ptr_ptr = evd_ptr; tduffy> tduffy> - bail: tduffy> +bail: tduffy> if (dat_status != DAT_SUCCESS) { tduffy> if (evd_ptr) { tduffy> dapl_evd_dealloc(evd_ptr); tduffy> @@ -127,7 +127,7 @@ u32 dapl_evd_internal_create(DAPL_IA *ia tduffy> * alloc and initialize an EVD struct tduffy> */ tduffy> DAPL_EVD *dapl_evd_alloc(DAPL_IA * ia_ptr, DAPL_CNO * cno_ptr, tduffy> - DAT_EVD_FLAGS evd_flags, DAT_COUNT qlen) tduffy> + DAT_EVD_FLAGS evd_flags, int qlen) tduffy> { tduffy> DAPL_EVD *evd_ptr; tduffy> tduffy> @@ -164,17 +164,17 @@ DAPL_EVD *dapl_evd_alloc(DAPL_IA * ia_pt tduffy> evd_ptr->qlen = qlen; tduffy> evd_ptr->cno_ptr = cno_ptr; tduffy> tduffy> - bail: tduffy> +bail: tduffy> return evd_ptr; tduffy> } tduffy> tduffy> /* tduffy> * alloc events into an EVD. tduffy> */ tduffy> -u32 dapl_evd_event_alloc(DAPL_EVD *evd_ptr, DAT_COUNT qlen) tduffy> +u32 dapl_evd_event_alloc(DAPL_EVD *evd_ptr, int qlen) tduffy> { tduffy> struct dat_event *event_ptr; tduffy> - DAT_COUNT i; tduffy> + int i; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> /* Allocate EVENTs */ tduffy> @@ -206,62 +206,56 @@ u32 dapl_evd_event_alloc(DAPL_EVD *evd_p tduffy> event_ptr++; tduffy> } tduffy> tduffy> - bail: tduffy> +bail: tduffy> return dat_status; tduffy> } tduffy> tduffy> /* tduffy> * realloc events into an EVD. tduffy> */ tduffy> -u32 dapl_evd_event_realloc(DAPL_EVD *evd_ptr, DAT_COUNT qlen) tduffy> +u32 dapl_evd_event_realloc(DAPL_EVD *evd_ptr, int qlen) tduffy> { tduffy> - struct dat_event *events; tduffy> - DAT_COUNT old_qlen; tduffy> - DAT_COUNT i; tduffy> - intptr_t diff; tduffy> + struct dat_event *events; tduffy> + int i, old_qlen, diff; tduffy> u32 dat_status; tduffy> tduffy> - /* Allocate EVENTs */ tduffy> - events = (struct dat_event *)dapl_os_realloc(evd_ptr->events, tduffy> - qlen * sizeof *events); tduffy> - if ( NULL == events ) tduffy> - { tduffy> - dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> - goto bail; tduffy> - } tduffy> + /* Allocate EVENTs */ tduffy> + events = (struct dat_event *)dapl_os_realloc(evd_ptr->events, tduffy> + qlen * sizeof *events); tduffy> + if (!events) { tduffy> + dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, tduffy> + DAT_RESOURCE_MEMORY); tduffy> + goto bail; tduffy> + } tduffy> tduffy> - diff = events - evd_ptr->events; tduffy> - evd_ptr->events = events; tduffy> + diff = events - evd_ptr->events; tduffy> + evd_ptr->events = events; tduffy> tduffy> - old_qlen = evd_ptr->qlen; tduffy> - evd_ptr->qlen = qlen; tduffy> + old_qlen = evd_ptr->qlen; tduffy> + evd_ptr->qlen = qlen; tduffy> tduffy> - /* reallocate free event queue */ tduffy> - dat_status = dapl_rbuf_realloc(&evd_ptr->free_event_queue, qlen); tduffy> - if (dat_status != DAT_SUCCESS) tduffy> - { tduffy> - goto bail; tduffy> - } tduffy> - dapl_rbuf_adjust(&evd_ptr->free_event_queue, diff); tduffy> + /* reallocate free event queue */ tduffy> + dat_status = dapl_rbuf_realloc(&evd_ptr->free_event_queue, qlen); tduffy> + if (dat_status != DAT_SUCCESS) tduffy> + goto bail; tduffy> tduffy> - /* reallocate pending event queue */ tduffy> - dat_status = dapl_rbuf_realloc(&evd_ptr->pending_event_queue, qlen); tduffy> - if (dat_status != DAT_SUCCESS) tduffy> - { tduffy> - goto bail; tduffy> - } tduffy> - dapl_rbuf_adjust(&evd_ptr->pending_event_queue, diff); tduffy> + dapl_rbuf_adjust(&evd_ptr->free_event_queue, diff); tduffy> tduffy> - /* tduffy> - * add new events to free event queue. tduffy> - */ tduffy> - for (i = old_qlen; i < qlen; i++) tduffy> - { tduffy> - dapl_rbuf_add(&evd_ptr->free_event_queue, (void *) &events[i]); tduffy> - } tduffy> + /* reallocate pending event queue */ tduffy> + dat_status = dapl_rbuf_realloc(&evd_ptr->pending_event_queue, qlen); tduffy> + if (dat_status != DAT_SUCCESS) tduffy> + goto bail; tduffy> + tduffy> + dapl_rbuf_adjust(&evd_ptr->pending_event_queue, diff); tduffy> + tduffy> + /* tduffy> + * add new events to free event queue. tduffy> + */ tduffy> + for (i = old_qlen; i < qlen; i++) tduffy> + dapl_rbuf_add(&evd_ptr->free_event_queue, (void *) &events[i]); tduffy> tduffy> bail: tduffy> - return dat_status; tduffy> + return dat_status; tduffy> } tduffy> tduffy> /* tduffy> @@ -297,10 +291,8 @@ u32 dapl_evd_dealloc(DAPL_EVD *evd_ptr) tduffy> evd_ptr->header.magic = DAPL_MAGIC_INVALID; tduffy> tduffy> /* Release reference on the CNO if it exists */ tduffy> - if ( evd_ptr->cno_ptr != NULL ) tduffy> - { tduffy> + if (evd_ptr->cno_ptr != NULL) tduffy> evd_ptr->cno_ptr = NULL; tduffy> - } tduffy> tduffy> /* If the ring buffer allocation failed, then the dapl_rbuf_destroy */ tduffy> /* function will detect that the ring buffer's internal data (ex. base */ tduffy> @@ -314,7 +306,7 @@ u32 dapl_evd_dealloc(DAPL_EVD *evd_ptr) tduffy> tduffy> kfree(evd_ptr); tduffy> tduffy> - bail: tduffy> +bail: tduffy> return dat_status; tduffy> } tduffy> tduffy> @@ -539,7 +531,7 @@ u32 dapl_evd_post_cr_arrival_event(DAPL_ tduffy> u32 dapl_evd_post_connection_event(DAPL_EVD *evd_ptr, tduffy> DAT_EVENT_NUMBER event_number, tduffy> DAT_EP_HANDLE ep_handle, tduffy> - DAT_COUNT private_data_size, tduffy> + int private_data_size, tduffy> void *private_data) tduffy> { tduffy> struct dat_event *event_ptr; tduffy> Index: linux-kernel/dat-provider/dapl_evd_util.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_evd_util.h (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl_evd_util.h (working copy) tduffy> @@ -41,15 +41,15 @@ tduffy> #include "dapl.h" tduffy> tduffy> u32 dapl_evd_internal_create(DAPL_IA *ia_ptr, DAPL_CNO *cno_ptr, tduffy> - DAT_COUNT min_qlen, DAT_EVD_FLAGS evd_flags, tduffy> + int min_qlen, DAT_EVD_FLAGS evd_flags, tduffy> DAPL_EVD **evd_ptr_ptr); tduffy> tduffy> DAPL_EVD *dapl_evd_alloc(DAPL_IA * ia_ptr, DAPL_CNO * cno_ptr, tduffy> - DAT_EVD_FLAGS evd_flags, DAT_COUNT qlen); tduffy> + DAT_EVD_FLAGS evd_flags, int qlen); tduffy> tduffy> u32 dapl_evd_dealloc(DAPL_EVD *evd_ptr); tduffy> tduffy> -u32 dapl_evd_event_realloc(DAPL_EVD *evd_ptr, DAT_COUNT qlen); tduffy> +u32 dapl_evd_event_realloc(DAPL_EVD *evd_ptr, int qlen); tduffy> tduffy> /* tduffy> * Each of these functions will retrieve a free event from tduffy> @@ -72,7 +72,7 @@ u32 dapl_evd_post_cr_arrival_event(DAPL_ tduffy> u32 dapl_evd_post_connection_event(DAPL_EVD *evd_ptr, tduffy> DAT_EVENT_NUMBER event_number, tduffy> DAT_EP_HANDLE ep_handle, tduffy> - DAT_COUNT private_data_size, tduffy> + int private_data_size, tduffy> void *private_data); tduffy> tduffy> u32 dapl_evd_post_async_error_event(DAPL_EVD *evd_ptr, tduffy> Index: linux-kernel/dat-provider/dapl_util.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_util.h (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl_util.h (working copy) tduffy> @@ -73,13 +73,6 @@ typedef long intptr_t; tduffy> tduffy> #define dapl_ip_addr6(sockaddr) (((struct sockaddr_in6 *)sockaddr)->sin6_addr.s6_addr32) tduffy> tduffy> -/* tduffy> - * Atomic operations tduffy> - */ tduffy> - tduffy> -static __inline__ DAT_COUNT tduffy> -dapl_os_atomic_assign(atomic_t * v, DAT_COUNT match_value, DAT_COUNT new_value); tduffy> - tduffy> /* dapl_os_atomic_assign tduffy> * tduffy> * assign 'new_value' to '*v' if the current value tduffy> @@ -90,14 +83,14 @@ dapl_os_atomic_assign(atomic_t * v, DAT_ tduffy> * Return the current value in any case. tduffy> * tduffy> * This matches the IBTA atomic operation compare & swap tduffy> - * except that it is for local memory and a DAT_COUNT may tduffy> + * except that it is for local memory and a int may tduffy> * be only 32 bits, rather than 64. tduffy> */ tduffy> tduffy> -static __inline__ DAT_COUNT tduffy> -dapl_os_atomic_assign(atomic_t * v, DAT_COUNT match_value, DAT_COUNT new_value) tduffy> +static inline int dapl_os_atomic_assign(atomic_t * v, int match_value, tduffy> + int new_value) tduffy> { tduffy> - DAT_COUNT current_value; tduffy> + int current_value; tduffy> tduffy> /* tduffy> * Use the Pentium compare and exchange instruction tduffy> Index: linux-kernel/dat-provider/dapl_hash.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_hash.c (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl_hash.c (working copy) tduffy> @@ -320,12 +320,12 @@ dapl_hash_delete_element(DAPL_HASH_ELEM tduffy> /* tduffy> * Create a new hash table with at least 'table_size' hash buckets. tduffy> */ tduffy> -u32 dapl_hash_create(DAT_COUNT table_size, DAPL_HASH_TABLE **pp_table) tduffy> +u32 dapl_hash_create(int table_size, DAPL_HASH_TABLE **pp_table) tduffy> { tduffy> DAPL_HASH_TABLE *p_table; tduffy> - DAT_COUNT table_length = table_size * sizeof(DAPL_HASH_ELEM); tduffy> + int table_length = table_size * sizeof(DAPL_HASH_ELEM); tduffy> u32 dat_status = DAT_SUCCESS; tduffy> - DAT_COUNT i; tduffy> + int i; tduffy> tduffy> dapl_os_assert(pp_table); tduffy> tduffy> @@ -379,7 +379,7 @@ u32 dapl_hash_free(DAPL_HASH_TABLE *p_ta tduffy> * Returns the number of elements stored in the table tduffy> */ tduffy> tduffy> -u32 dapl_hash_size(DAPL_HASH_TABLE *p_table, DAT_COUNT *p_size) tduffy> +u32 dapl_hash_size(DAPL_HASH_TABLE *p_table, int *p_size) tduffy> { tduffy> dapl_os_assert(p_table && p_size); tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_ep_dup_connect.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_ep_dup_connect.c (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl_ep_dup_connect.c (working copy) tduffy> @@ -70,7 +70,7 @@ tduffy> * DAT_MODEL_NOT_SUPPORTED tduffy> */ tduffy> u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, DAT_EP_HANDLE ep_dup_handle, tduffy> - DAT_TIMEOUT timeout, DAT_COUNT private_data_size, tduffy> + DAT_TIMEOUT timeout, int private_data_size, tduffy> const void *private_data, DAT_QOS qos) tduffy> { tduffy> DAPL_EP *ep_dup_ptr; tduffy> Index: linux-kernel/dat-provider/dapl.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl.h (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl.h (working copy) tduffy> @@ -193,14 +193,14 @@ struct dapl_llist_entry { tduffy> tduffy> struct dapl_ring_buffer { tduffy> void **base; /* base of element array */ tduffy> - DAT_COUNT lim; /* mask, number of entries - 1 */ tduffy> + int lim; /* mask, number of entries - 1 */ tduffy> atomic_t head; /* head pointer index */ tduffy> atomic_t tail; /* tail pointer index */ tduffy> }; tduffy> tduffy> struct dapl_cookie_buffer { tduffy> DAPL_COOKIE *pool; tduffy> - DAT_COUNT pool_size; tduffy> + int pool_size; tduffy> atomic_t head; tduffy> atomic_t tail; tduffy> }; tduffy> @@ -257,33 +257,32 @@ struct dapl_cno { tduffy> }; tduffy> tduffy> /* DAPL_EVD maps to DAT_EVD_HANDLE */ tduffy> -struct dapl_evd tduffy> -{ tduffy> - DAPL_HEADER header; tduffy> - tduffy> - DAPL_EVD_STATE evd_state; tduffy> - DAT_EVD_FLAGS evd_flags; tduffy> - tduffy> - /* Derived from evd_flags; see dapls_evd_internal_create. */ tduffy> - boolean_t evd_producer_locking_needed; tduffy> - tduffy> - /* Every EVD has a CQ unless it is a SOFTWARE_EVENT only EVD */ tduffy> - ib_cq_handle_t ib_cq_handle; tduffy> - tduffy> - /* An Event Dispatcher cannot be freed while tduffy> - * it is referenced elsewhere. tduffy> - */ tduffy> - atomic_t evd_ref_count; tduffy> - tduffy> - /* Set if there has been a catastrophic overflow */ tduffy> - boolean_t catastrophic_overflow; tduffy> - tduffy> - /* the actual events */ tduffy> - DAT_COUNT qlen; tduffy> - struct dat_event *events; tduffy> - DAPL_RING_BUFFER free_event_queue; tduffy> - DAPL_RING_BUFFER pending_event_queue; tduffy> - DAPL_CNO *cno_ptr; tduffy> +struct dapl_evd { tduffy> + DAPL_HEADER header; tduffy> + DAPL_EVD_STATE evd_state; tduffy> + DAT_EVD_FLAGS evd_flags; tduffy> + tduffy> + /* Derived from evd_flags; see dapls_evd_internal_create. */ tduffy> + boolean_t evd_producer_locking_needed; tduffy> + tduffy> + /* Every EVD has a CQ unless it is a SOFTWARE_EVENT only EVD */ tduffy> + ib_cq_handle_t ib_cq_handle; tduffy> + tduffy> + /* tduffy> + * An Event Dispatcher cannot be freed while tduffy> + * it is referenced elsewhere. tduffy> + */ tduffy> + atomic_t evd_ref_count; tduffy> + tduffy> + /* Set if there has been a catastrophic overflow */ tduffy> + boolean_t catastrophic_overflow; tduffy> + tduffy> + /* the actual events */ tduffy> + int qlen; tduffy> + struct dat_event *events; tduffy> + DAPL_RING_BUFFER free_event_queue; tduffy> + DAPL_RING_BUFFER pending_event_queue; tduffy> + DAPL_CNO *cno_ptr; tduffy> }; tduffy> tduffy> /* uDAPL timer entry, used to queue timeouts */ tduffy> @@ -329,10 +328,10 @@ struct dapl_ep { tduffy> DAPL_COOKIE_BUFFER recv_buffer; tduffy> tduffy> ib_data_segment_t *recv_iov; tduffy> - DAT_COUNT recv_iov_num; tduffy> + int recv_iov_num; tduffy> tduffy> ib_data_segment_t *send_iov; tduffy> - DAT_COUNT send_iov_num; tduffy> + int send_iov_num; tduffy> }; tduffy> tduffy> /* DAPL_SRQ maps to DAT_SRQ_HANDLE */ tduffy> @@ -394,7 +393,7 @@ struct dapl_sp { tduffy> boolean_t listening; /* PSP is registered & active */ tduffy> ib_cm_srvc_handle_t cm_srvc_handle; /* Used by CM */ tduffy> DAPL_LLIST_HEAD cr_list_head; /* CR pending queue */ tduffy> - DAT_COUNT cr_list_count; /* count of CRs on queue */ tduffy> + int cr_list_count; /* count of CRs on queue */ tduffy> }; tduffy> tduffy> /* DAPL_CR maps to DAT_CR_HANDLE */ tduffy> @@ -439,7 +438,7 @@ typedef enum dapl_cookie_type { tduffy> struct dapl_dto_cookie { tduffy> DAPL_DTO_TYPE type; tduffy> DAT_DTO_COOKIE cookie; tduffy> - DAT_COUNT size; /* used for SEND and RDMA write */ tduffy> + int size; /* used for SEND and RDMA write */ tduffy> }; tduffy> tduffy> /* DAPL_RMR_COOKIE used as context for bind WQEs */ tduffy> @@ -452,7 +451,7 @@ struct dapl_rmr_cookie { tduffy> struct dapl_cookie { tduffy> DAPL_COOKIE_TYPE type; /* Must be first, to define struct. */ tduffy> DAPL_EP *ep; tduffy> - DAT_COUNT index; tduffy> + int index; tduffy> union { tduffy> DAPL_DTO_COOKIE dto; tduffy> DAPL_RMR_COOKIE rmr; tduffy> @@ -475,8 +474,9 @@ typedef char DAPL_HCA_NAME[DAPL_HCA_NAME tduffy> * DAT API functions tduffy> */ tduffy> tduffy> -extern u32 dapl_ia_open(const DAT_NAME_PTR, DAT_COUNT, DAT_EVD_HANDLE *, tduffy> - DAT_IA_HANDLE *); tduffy> +extern u32 dapl_ia_open(const DAT_NAME_PTR name, int async_evd_qlen, tduffy> + DAT_EVD_HANDLE *async_evd_handle_ptr, tduffy> + DAT_IA_HANDLE *ia_handle_ptr); tduffy> tduffy> extern u32 dapl_ia_close(DAT_IA_HANDLE, DAT_CLOSE_FLAGS); tduffy> tduffy> @@ -496,7 +496,7 @@ extern u32 dapl_get_handle_type(DAT_HAND tduffy> extern u32 dapl_cr_query(DAT_CR_HANDLE, struct dat_cr_param *); tduffy> tduffy> extern u32 dapl_cr_accept(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, tduffy> - DAT_COUNT private_data_size, tduffy> + int private_data_size, tduffy> const void *private_data); tduffy> tduffy> extern u32 dapl_cr_reject(DAT_CR_HANDLE cr_handle); tduffy> @@ -509,7 +509,7 @@ extern u32 dapl_ia_memtype_hint(DAT_IA_H tduffy> DAT_VLEN *suggested_length, tduffy> DAT_VADDR *suggested_alignment); tduffy> tduffy> -extern u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_handle, DAT_COUNT evd_min_qlen, tduffy> +extern u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_handle, int evd_min_qlen, tduffy> DAT_UPCALL_POLICY upcall_policy, tduffy> const struct dat_upcall_object * upcall, tduffy> DAT_EVD_FLAGS evd_flags, tduffy> @@ -526,7 +526,7 @@ extern u32 dapl_evd_enable(DAT_EVD_HANDL tduffy> tduffy> extern u32 dapl_evd_disable(DAT_EVD_HANDLE evd_handle); tduffy> tduffy> -extern u32 dapl_evd_resize(DAT_EVD_HANDLE evd_handle, DAT_COUNT evd_qlen); tduffy> +extern u32 dapl_evd_resize(DAT_EVD_HANDLE evd_handle, int evd_qlen); tduffy> tduffy> extern u32 dapl_evd_post_se(DAT_EVD_HANDLE evd_handle, tduffy> const struct dat_event *event); tduffy> @@ -558,39 +558,39 @@ extern u32 dapl_ep_connect(DAT_EP_HANDLE tduffy> DAT_IA_ADDRESS_PTR remote_ia_address, tduffy> DAT_CONN_QUAL remote_conn_qual, tduffy> DAT_TIMEOUT timeout, tduffy> - DAT_COUNT private_data_size, tduffy> + int private_data_size, tduffy> const void *private_data, tduffy> DAT_QOS quality_of_service, tduffy> DAT_CONNECT_FLAGS connect_flags); tduffy> tduffy> extern u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, tduffy> DAT_EP_HANDLE ep_dup_handle, DAT_TIMEOUT timeout, tduffy> - DAT_COUNT private_data_size, tduffy> + int private_data_size, tduffy> const void *private_data, tduffy> DAT_QOS quality_of_service); tduffy> tduffy> extern u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_handle, tduffy> DAT_CLOSE_FLAGS close_flags); tduffy> tduffy> -extern u32 dapl_ep_post_send(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, tduffy> +extern u32 dapl_ep_post_send(DAT_EP_HANDLE ep_handle, int num_segments, tduffy> struct dat_lmr_triplet *local_iov, tduffy> DAT_DTO_COOKIE user_cookie, tduffy> DAT_COMPLETION_FLAGS completion_flags); tduffy> tduffy> -extern u32 dapl_ep_post_recv(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, tduffy> +extern u32 dapl_ep_post_recv(DAT_EP_HANDLE ep_handle, int num_segments, tduffy> struct dat_lmr_triplet *local_iov, tduffy> DAT_DTO_COOKIE user_cookie, tduffy> DAT_COMPLETION_FLAGS completion_flags); tduffy> tduffy> extern u32 dapl_ep_post_rdma_read(DAT_EP_HANDLE ep_handle, tduffy> - DAT_COUNT num_segments, tduffy> + int num_segments, tduffy> struct dat_lmr_triplet *local_iov, tduffy> DAT_DTO_COOKIE user_cookie, tduffy> const struct dat_rmr_triplet *remote_iov, tduffy> DAT_COMPLETION_FLAGS completion_flags); tduffy> tduffy> extern u32 dapl_ep_post_rdma_write(DAT_EP_HANDLE ep_handle, tduffy> - DAT_COUNT num_segments, tduffy> + int num_segments, tduffy> struct dat_lmr_triplet *local_iov, tduffy> DAT_DTO_COOKIE user_cookie, tduffy> const struct dat_rmr_triplet *remote_iov, tduffy> @@ -613,12 +613,12 @@ extern u32 dapl_ep_create_with_srq(DAT_I tduffy> DAT_EP_HANDLE *ep_handle); tduffy> tduffy> extern u32 dapl_ep_recv_query(DAT_EP_HANDLE ep_handle, tduffy> - DAT_COUNT *nbufs_allocated, tduffy> - DAT_COUNT *bufs_alloc_span); tduffy> + int *nbufs_allocated, tduffy> + int *bufs_alloc_span); tduffy> tduffy> extern u32 dapl_ep_set_watermark(DAT_EP_HANDLE ep_handle, tduffy> - DAT_COUNT soft_high_watermark, tduffy> - DAT_COUNT hard_high_watermark); tduffy> + int soft_high_watermark, tduffy> + int hard_high_watermark); tduffy> tduffy> /* LMR functions */ tduffy> tduffy> @@ -703,7 +703,7 @@ extern u32 dapl_srq_create(DAT_IA_HANDLE tduffy> tduffy> extern u32 dapl_srq_free(DAT_SRQ_HANDLE srq_handle); tduffy> tduffy> -extern u32 dapl_srq_post_recv(DAT_SRQ_HANDLE srq_handle, DAT_COUNT num_segments, tduffy> +extern u32 dapl_srq_post_recv(DAT_SRQ_HANDLE srq_handle, int num_segments, tduffy> struct dat_lmr_triplet *local_iov, tduffy> DAT_DTO_COOKIE user_cookie); tduffy> tduffy> @@ -711,9 +711,9 @@ extern u32 dapl_srq_query(DAT_SRQ_HANDLE tduffy> struct dat_srq_param *srq_param); tduffy> tduffy> extern u32 dapl_srq_resize(DAT_SRQ_HANDLE srq_handle, tduffy> - DAT_COUNT srq_max_recv_dto); tduffy> + int srq_max_recv_dto); tduffy> tduffy> -extern u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, DAT_COUNT low_watermark); tduffy> +extern u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, int low_watermark); tduffy> tduffy> /* tduffy> * DAPL internal utility function prototpyes tduffy> Index: linux-kernel/dat-provider/dapl_hash.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_hash.h (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl_hash.h (working copy) tduffy> @@ -67,11 +67,11 @@ tduffy> * * tduffy> *********************************************************************/ tduffy> tduffy> -extern u32 dapl_hash_create(DAT_COUNT capacity, DAPL_HASH_TABLE **pp_table); tduffy> +extern u32 dapl_hash_create(int capacity, DAPL_HASH_TABLE **pp_table); tduffy> tduffy> extern u32 dapl_hash_free(DAPL_HASH_TABLE *p_table); tduffy> tduffy> -extern u32 dapl_hash_size(DAPL_HASH_TABLE *p_table, DAT_COUNT *p_size); tduffy> +extern u32 dapl_hash_size(DAPL_HASH_TABLE *p_table, int *p_size); tduffy> tduffy> extern u32 dapl_hash_insert(DAPL_HASH_TABLE *p_table, DAPL_HASH_KEY key, tduffy> DAPL_HASH_DATA data); tduffy> Index: linux-kernel/dat-provider/dapl_ep_post_recv.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_ep_post_recv.c (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl_ep_post_recv.c (working copy) tduffy> @@ -66,7 +66,7 @@ tduffy> * DAT_PROTECTION_VIOLATION tduffy> * DAT_PROVILEGES_VIOLATION tduffy> */ tduffy> -u32 dapl_ep_post_recv(DAT_EP_HANDLE ep_handle, DAT_COUNT num_segments, tduffy> +u32 dapl_ep_post_recv(DAT_EP_HANDLE ep_handle, int num_segments, tduffy> struct dat_lmr_triplet *local_iov, tduffy> DAT_DTO_COOKIE user_cookie, tduffy> DAT_COMPLETION_FLAGS completion_flags) tduffy> Index: linux-kernel/dat-provider/dapl_ep_set_watermark.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_ep_set_watermark.c (revision 2321) tduffy> +++ linux-kernel/dat-provider/dapl_ep_set_watermark.c (working copy) tduffy> @@ -60,8 +60,8 @@ tduffy> * DAT_MODEL_NOT_SUPPORTED tduffy> */ tduffy> u32 dapl_ep_set_watermark(DAT_EP_HANDLE ep_handle, tduffy> - DAT_COUNT soft_high_watermark, tduffy> - DAT_COUNT hard_high_watermark) tduffy> + int soft_high_watermark, tduffy> + int hard_high_watermark) tduffy> { tduffy> DAPL_EP *ep_ptr; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> Index: linux-kernel/dat/dictionary.c tduffy> =================================================================== tduffy> --- linux-kernel/dat/dictionary.c (revision 2321) tduffy> +++ linux-kernel/dat/dictionary.c (working copy) tduffy> @@ -75,8 +75,7 @@ void dat_dictionary_destroy(struct dat_d tduffy> * Function: dat_dictionary_size tduffy> ***********************************************************************/ tduffy> tduffy> -u32 dat_dictionary_size(const struct dat_dictionary *dictionary, tduffy> - DAT_COUNT *size) tduffy> +u32 dat_dictionary_size(const struct dat_dictionary *dictionary, int *size) tduffy> { tduffy> BUG_ON(NULL == dictionary); tduffy> BUG_ON(NULL == size); tduffy> @@ -190,11 +189,11 @@ u32 dat_dictionary_search(struct dat_dic tduffy> ***********************************************************************/ tduffy> tduffy> u32 dat_dictionary_enumerate(struct dat_dictionary *dictionary, void *array[], tduffy> - DAT_COUNT array_size) tduffy> + int array_size) tduffy> { tduffy> struct list_head *cur_list; tduffy> struct dat_dictionary_entry *cur_entry; tduffy> - DAT_COUNT i; tduffy> + int i; tduffy> u32 status; tduffy> tduffy> BUG_ON(NULL == dictionary); tduffy> Index: linux-kernel/dat/dr.c tduffy> =================================================================== tduffy> --- linux-kernel/dat/dr.c (revision 2321) tduffy> +++ linux-kernel/dat/dr.c (working copy) tduffy> @@ -42,7 +42,7 @@ tduffy> *********************************************************************/ tduffy> tduffy> struct dat_dr_entry { tduffy> - DAT_COUNT ref_count; tduffy> + int ref_count; tduffy> struct dat_provider_info info; tduffy> DAT_IA_OPEN_FUNC ia_open_func; tduffy> }; tduffy> @@ -206,7 +206,7 @@ u32 dat_dr_provider_close(const struct d tduffy> * Function: dat_dr_size tduffy> ************************************************************************/ tduffy> tduffy> -u32 dat_dr_size(DAT_COUNT *size) tduffy> +u32 dat_dr_size(int *size) tduffy> { tduffy> return dat_dictionary_size(&g_dr_dictionary, size); tduffy> } tduffy> @@ -215,17 +215,13 @@ u32 dat_dr_size(DAT_COUNT *size) tduffy> * Function: dat_dr_list tduffy> ***********************************************************************/ tduffy> tduffy> -u32 dat_dr_list(DAT_COUNT max_to_return, tduffy> - DAT_COUNT *entries_returned, tduffy> +u32 dat_dr_list(int max_to_return, int *entries_returned, tduffy> struct dat_provider_info *(dat_provider_list[])) tduffy> { tduffy> - struct dat_dr_entry **array; tduffy> - DAT_COUNT array_size; tduffy> - DAT_COUNT i; tduffy> + struct dat_dr_entry **array = NULL; tduffy> + int i, array_size; tduffy> u32 status = DAT_SUCCESS; tduffy> tduffy> - array = NULL; tduffy> - tduffy> /* The dictionary size may increase between the call to */ tduffy> /* dat_dictionary_size() and dat_dictionary_enumerate(). */ tduffy> /* Therefore we loop until a successful enumeration is made. */ tduffy> Index: linux-kernel/dat/api.c tduffy> =================================================================== tduffy> --- linux-kernel/dat/api.c (revision 2321) tduffy> +++ linux-kernel/dat/api.c (working copy) tduffy> @@ -575,7 +575,7 @@ static u32 dat_strerror_minor(u32 value, tduffy> * Function: dat_ia_openv tduffy> ***********************************************************************/ tduffy> tduffy> -u32 dat_ia_openv(const DAT_NAME_PTR name, DAT_COUNT async_event_qlen, tduffy> +u32 dat_ia_openv(const DAT_NAME_PTR name, int async_event_qlen, tduffy> DAT_EVD_HANDLE *async_event_handle, DAT_IA_HANDLE *ia_handle, tduffy> u32 dat_major, u32 dat_minor, boolean_t thread_safety) tduffy> { tduffy> @@ -756,8 +756,7 @@ EXPORT_SYMBOL(dat_registry_add_provider) tduffy> * Function: dat_registry_list_providers tduffy> ***********************************************************************/ tduffy> tduffy> -u32 dat_registry_list_providers(DAT_COUNT max_to_return, tduffy> - DAT_COUNT *entries_returned, tduffy> +u32 dat_registry_list_providers(int max_to_return, int *entries_returned, tduffy> struct dat_provider_info *(dat_provider_list[])) tduffy> { tduffy> u32 status = DAT_SUCCESS; tduffy> Index: linux-kernel/dat/dr.h tduffy> =================================================================== tduffy> --- linux-kernel/dat/dr.h (revision 2321) tduffy> +++ linux-kernel/dat/dr.h (working copy) tduffy> @@ -54,10 +54,9 @@ extern u32 dat_dr_provider_open(const st tduffy> tduffy> extern u32 dat_dr_provider_close(const struct dat_provider_info *info); tduffy> tduffy> -extern u32 dat_dr_size(DAT_COUNT *size); tduffy> +extern u32 dat_dr_size(int *size); tduffy> tduffy> -extern u32 dat_dr_list(DAT_COUNT max_to_return, tduffy> - DAT_COUNT *entries_returned, tduffy> +extern u32 dat_dr_list(int max_to_return, int *entries_returned, tduffy> struct dat_provider_info *(dat_provider_list[])); tduffy> tduffy> #endif /* DR_H */ tduffy> Index: linux-kernel/dat/dictionary.h tduffy> =================================================================== tduffy> --- linux-kernel/dat/dictionary.h (revision 2321) tduffy> +++ linux-kernel/dat/dictionary.h (working copy) tduffy> @@ -50,7 +50,7 @@ struct dat_dictionary_entry { tduffy> tduffy> struct dat_dictionary { tduffy> struct list_head list; tduffy> - DAT_COUNT size; tduffy> + int size; tduffy> }; tduffy> tduffy> /********************************************************************* tduffy> @@ -70,7 +70,7 @@ struct dat_dictionary { tduffy> extern void dat_dictionary_destroy(struct dat_dictionary *dictionary); tduffy> tduffy> extern u32 dat_dictionary_size(const struct dat_dictionary *dictionary, tduffy> - DAT_COUNT *size); tduffy> + int *size); tduffy> tduffy> extern u32 dat_dictionary_entry_create(struct dat_dictionary_entry **p_entry); tduffy> tduffy> @@ -87,8 +87,7 @@ extern u32 dat_dictionary_search(struct tduffy> void **p_data); tduffy> tduffy> extern u32 dat_dictionary_enumerate(struct dat_dictionary *dictionary, tduffy> - void *array[], tduffy> - const DAT_COUNT array_size); tduffy> + void *array[], const int array_size); tduffy> tduffy> extern u32 dat_dictionary_remove(struct dat_dictionary *dictionary, tduffy> const struct dat_provider_info *key, tduffy> Index: linux-kernel/dat/dat.h tduffy> =================================================================== tduffy> --- linux-kernel/dat/dat.h (revision 2321) tduffy> +++ linux-kernel/dat/dat.h (working copy) tduffy> @@ -40,7 +40,6 @@ tduffy> /* unsigned longest native to compiler */ tduffy> typedef unsigned long long DAT_UVERYLONG; tduffy> tduffy> -typedef int DAT_COUNT; tduffy> typedef u64 DAT_PADDR; tduffy> tduffy> /* tduffy> @@ -575,18 +574,18 @@ struct dat_ep_attr { tduffy> DAT_QOS qos; tduffy> DAT_COMPLETION_FLAGS recv_completion_flags; tduffy> DAT_COMPLETION_FLAGS request_completion_flags; tduffy> - DAT_COUNT max_recv_dtos; tduffy> - DAT_COUNT max_request_dtos; tduffy> - DAT_COUNT max_recv_iov; tduffy> - DAT_COUNT max_request_iov; tduffy> - DAT_COUNT max_rdma_read_in; tduffy> - DAT_COUNT max_rdma_read_out; tduffy> - DAT_COUNT srq_soft_hw; tduffy> - DAT_COUNT max_rdma_read_iov; tduffy> - DAT_COUNT max_rdma_write_iov; tduffy> - DAT_COUNT ep_transport_specific_count; tduffy> + int max_recv_dtos; tduffy> + int max_request_dtos; tduffy> + int max_recv_iov; tduffy> + int max_request_iov; tduffy> + int max_rdma_read_in; tduffy> + int max_rdma_read_out; tduffy> + int srq_soft_hw; tduffy> + int max_rdma_read_iov; tduffy> + int max_rdma_write_iov; tduffy> + int ep_transport_specific_count; tduffy> struct dat_named_attr *ep_transport_specific; tduffy> - DAT_COUNT ep_provider_specific_count; tduffy> + int ep_provider_specific_count; tduffy> struct dat_named_attr *ep_provider_specific; tduffy> }; tduffy> tduffy> @@ -656,7 +655,7 @@ typedef enum dat_ep_param_mask { tduffy> DAT_EP_FIELD_ALL = 0x3FFFF7FF tduffy> } DAT_EP_PARAM_MASK; tduffy> tduffy> -#define DAT_WATERMARK_INFINITE ((DAT_COUNT)~0) tduffy> +#define DAT_WATERMARK_INFINITE ((int)~0) tduffy> #define DAT_HW_DEFAULT DAT_WATERMARK_INFINITE tduffy> #define DAT_SRQ_LW_DEFAULT 0x0 tduffy> tduffy> @@ -665,23 +664,23 @@ typedef enum dat_srq_state { tduffy> DAT_SRQ_STATE_ERROR tduffy> } DAT_SRQ_STATE; tduffy> tduffy> -#define DAT_VALUE_UNKNOWN (((DAT_COUNT) ~0)-1) tduffy> +#define DAT_VALUE_UNKNOWN (((int) ~0)-1) tduffy> tduffy> struct dat_srq_attr { tduffy> - DAT_COUNT max_recv_dtos; tduffy> - DAT_COUNT max_recv_iov; tduffy> - DAT_COUNT low_watermark; tduffy> + int max_recv_dtos; tduffy> + int max_recv_iov; tduffy> + int low_watermark; tduffy> }; tduffy> tduffy> struct dat_srq_param { tduffy> DAT_IA_HANDLE ia_handle; tduffy> DAT_SRQ_STATE srq_state; tduffy> DAT_PZ_HANDLE pz_handle; tduffy> - DAT_COUNT max_recv_dtos; tduffy> - DAT_COUNT max_recv_iov; tduffy> - DAT_COUNT low_watermark; tduffy> - DAT_COUNT available_dto_count; tduffy> - DAT_COUNT outstanding_dto_count; tduffy> + int max_recv_dtos; tduffy> + int max_recv_iov; tduffy> + int low_watermark; tduffy> + int available_dto_count; tduffy> + int outstanding_dto_count; tduffy> }; tduffy> tduffy> /* PZ Parameters */ tduffy> @@ -723,7 +722,7 @@ struct dat_cr_param { tduffy> DAT_PORT_QUAL remote_port_qual; tduffy> tduffy> /* Size of the Private Data. */ tduffy> - DAT_COUNT private_data_size; tduffy> + int private_data_size; tduffy> tduffy> /* Pointer to the Private Data passed by remote side in the Connection tduffy> * Request. tduffy> @@ -820,7 +819,7 @@ struct dat_cr_arrival_event_data { tduffy> /* Connection event data */ tduffy> struct dat_connection_event_data { tduffy> DAT_EP_HANDLE ep_handle; tduffy> - DAT_COUNT private_data_size; tduffy> + int private_data_size; tduffy> void *private_data; tduffy> }; tduffy> tduffy> @@ -836,7 +835,7 @@ struct dat_connection_event_data { tduffy> struct dat_asynch_error_event_data { tduffy> DAT_HANDLE dat_handle; /* either IA, EP, EVD, SRQ, */ tduffy> /* LMR, RMR, or PZ handle */ tduffy> - DAT_COUNT reason; /* object specific */ tduffy> + int reason; /* object specific */ tduffy> }; tduffy> tduffy> /* The reason is object type specific and its values are defined below. */ tduffy> @@ -954,7 +953,7 @@ struct dat_provider_info { tduffy> tduffy> struct dat_evd_param { tduffy> DAT_IA_HANDLE ia_handle; tduffy> - DAT_COUNT evd_qlen; tduffy> + int evd_qlen; tduffy> DAT_UPCALL_POLICY upcall_policy; tduffy> struct dat_upcall_object upcall; tduffy> DAT_EVD_FLAGS evd_flags; tduffy> @@ -1010,33 +1009,33 @@ struct dat_ia_attr { tduffy> u32 firmware_version_major; tduffy> u32 firmware_version_minor; tduffy> DAT_IA_ADDRESS_PTR ia_address_ptr; tduffy> - DAT_COUNT max_eps; tduffy> - DAT_COUNT max_dto_per_ep; tduffy> - DAT_COUNT max_rdma_read_per_ep_in; tduffy> - DAT_COUNT max_rdma_read_per_ep_out; tduffy> - DAT_COUNT max_evds; tduffy> - DAT_COUNT max_evd_qlen; tduffy> - DAT_COUNT max_iov_segments_per_dto; tduffy> - DAT_COUNT max_lmrs; tduffy> + int max_eps; tduffy> + int max_dto_per_ep; tduffy> + int max_rdma_read_per_ep_in; tduffy> + int max_rdma_read_per_ep_out; tduffy> + int max_evds; tduffy> + int max_evd_qlen; tduffy> + int max_iov_segments_per_dto; tduffy> + int max_lmrs; tduffy> DAT_VLEN max_lmr_block_size; tduffy> DAT_VADDR max_lmr_virtual_address; tduffy> - DAT_COUNT max_pzs; tduffy> + int max_pzs; tduffy> DAT_VLEN max_message_size; tduffy> DAT_VLEN max_rdma_size; tduffy> - DAT_COUNT max_rmrs; tduffy> + int max_rmrs; tduffy> DAT_VADDR max_rmr_target_address; tduffy> - DAT_COUNT max_srqs; tduffy> - DAT_COUNT max_ep_per_srq; tduffy> - DAT_COUNT max_recv_per_srq; tduffy> - DAT_COUNT max_iov_segments_per_rdma_read; tduffy> - DAT_COUNT max_iov_segments_per_rdma_write; tduffy> - DAT_COUNT max_rdma_read_in; tduffy> - DAT_COUNT max_rdma_read_out; tduffy> + int max_srqs; tduffy> + int max_ep_per_srq; tduffy> + int max_recv_per_srq; tduffy> + int max_iov_segments_per_rdma_read; tduffy> + int max_iov_segments_per_rdma_write; tduffy> + int max_rdma_read_in; tduffy> + int max_rdma_read_out; tduffy> boolean_t max_rdma_read_per_ep_in_guaranteed; tduffy> boolean_t max_rdma_read_per_ep_out_guaranteed; tduffy> - DAT_COUNT num_transport_attr; tduffy> + int num_transport_attr; tduffy> struct dat_named_attr *transport_attr; tduffy> - DAT_COUNT num_vendor_attr; tduffy> + int num_vendor_attr; tduffy> struct dat_named_attr *vendor_attr; tduffy> }; tduffy> tduffy> @@ -1079,21 +1078,21 @@ struct dat_provider_attr { tduffy> DAT_QOS dat_qos_supported; tduffy> DAT_COMPLETION_FLAGS completion_flags_supported; tduffy> boolean_t is_thread_safe; tduffy> - DAT_COUNT max_private_data_size; tduffy> + int max_private_data_size; tduffy> boolean_t supports_multipath; tduffy> DAT_EP_CREATOR_FOR_PSP ep_creator; tduffy> DAT_UPCALL_POLICY upcall_policy; tduffy> u32 optimal_buffer_alignment; tduffy> const boolean_t evd_stream_merging_supported[6][6]; tduffy> boolean_t srq_supported; tduffy> - DAT_COUNT srq_watermarks_supported; tduffy> + int srq_watermarks_supported; tduffy> boolean_t srq_ep_pz_difference_supported; tduffy> - DAT_COUNT srq_info_supported; tduffy> - DAT_COUNT ep_recv_info_supported; tduffy> + int srq_info_supported; tduffy> + int ep_recv_info_supported; tduffy> boolean_t lmr_sync_req; tduffy> boolean_t dto_async_return_guaranteed; tduffy> boolean_t rdma_write_for_rdma_read_req; tduffy> - DAT_COUNT num_provider_specific_attr; tduffy> + int num_provider_specific_attr; tduffy> struct dat_named_attr *provider_specific_attr; tduffy> }; tduffy> tduffy> @@ -1105,10 +1104,10 @@ struct dat_provider_attr { tduffy> ****************************************************************/ tduffy> tduffy> typedef u32 (*DAT_IA_OPEN_FUNC) ( tduffy> - const DAT_NAME_PTR, DAT_COUNT, DAT_EVD_HANDLE *, DAT_IA_HANDLE *); tduffy> + const DAT_NAME_PTR, int, DAT_EVD_HANDLE *, DAT_IA_HANDLE *); tduffy> tduffy> typedef u32 (*DAT_IA_OPENV_FUNC) ( tduffy> - const DAT_NAME_PTR, DAT_COUNT, DAT_EVD_HANDLE *, DAT_IA_HANDLE *, tduffy> + const DAT_NAME_PTR, int, DAT_EVD_HANDLE *, DAT_IA_HANDLE *, tduffy> u32, u32, boolean_t); tduffy> tduffy> typedef u32 (*DAT_IA_CLOSE_FUNC) (DAT_IA_HANDLE, DAT_CLOSE_FLAGS); tduffy> @@ -1125,14 +1124,14 @@ typedef u32 (*DAT_GET_HANDLE_TYPE_FUNC) tduffy> tduffy> typedef u32 (*DAT_CR_QUERY_FUNC) (DAT_CR_HANDLE, struct dat_cr_param *); tduffy> tduffy> -typedef u32 (*DAT_CR_ACCEPT_FUNC) (DAT_CR_HANDLE, DAT_EP_HANDLE, DAT_COUNT, tduffy> +typedef u32 (*DAT_CR_ACCEPT_FUNC) (DAT_CR_HANDLE, DAT_EP_HANDLE, int, tduffy> const void*); tduffy> tduffy> typedef u32 (*DAT_CR_REJECT_FUNC) (DAT_CR_HANDLE); tduffy> tduffy> typedef u32 (*DAT_CR_HANDOFF_FUNC) (DAT_CR_HANDLE, DAT_CONN_QUAL); tduffy> tduffy> -typedef u32 (*DAT_EVD_RESIZE_FUNC) (DAT_EVD_HANDLE, DAT_COUNT); tduffy> +typedef u32 (*DAT_EVD_RESIZE_FUNC) (DAT_EVD_HANDLE, int); tduffy> tduffy> typedef u32 (*DAT_EVD_POST_SE_FUNC) (DAT_EVD_HANDLE, tduffy> const struct dat_event *); tduffy> @@ -1158,29 +1157,29 @@ typedef u32 (*DAT_EP_MODIFY_FUNC) ( tduffy> DAT_EP_HANDLE, DAT_EP_PARAM_MASK, const struct dat_ep_param *); tduffy> tduffy> typedef u32 (*DAT_EP_CONNECT_FUNC) (DAT_EP_HANDLE, DAT_IA_ADDRESS_PTR, tduffy> - DAT_CONN_QUAL, DAT_TIMEOUT, DAT_COUNT, tduffy> + DAT_CONN_QUAL, DAT_TIMEOUT, int, tduffy> const void *, DAT_QOS, DAT_CONNECT_FLAGS); tduffy> tduffy> typedef u32 (*DAT_EP_DUP_CONNECT_FUNC) (DAT_EP_HANDLE, DAT_EP_HANDLE, tduffy> - DAT_TIMEOUT, DAT_COUNT, const void *, tduffy> + DAT_TIMEOUT, int, const void *, tduffy> DAT_QOS); tduffy> tduffy> typedef u32 (*DAT_EP_DISCONNECT_FUNC) (DAT_EP_HANDLE, DAT_CLOSE_FLAGS); tduffy> tduffy> typedef u32 (*DAT_EP_POST_SEND_FUNC) ( tduffy> - DAT_EP_HANDLE, DAT_COUNT, struct dat_lmr_triplet *, DAT_DTO_COOKIE, tduffy> + DAT_EP_HANDLE, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, tduffy> DAT_COMPLETION_FLAGS); tduffy> tduffy> typedef u32 (*DAT_EP_POST_RECV_FUNC) ( tduffy> - DAT_EP_HANDLE, DAT_COUNT, struct dat_lmr_triplet *, DAT_DTO_COOKIE, tduffy> + DAT_EP_HANDLE, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, tduffy> DAT_COMPLETION_FLAGS); tduffy> tduffy> typedef u32 (*DAT_EP_POST_RDMA_READ_FUNC) ( tduffy> - DAT_EP_HANDLE, DAT_COUNT, struct dat_lmr_triplet *, DAT_DTO_COOKIE, tduffy> + DAT_EP_HANDLE, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, tduffy> const struct dat_rmr_triplet *, DAT_COMPLETION_FLAGS); tduffy> tduffy> typedef u32 (*DAT_EP_POST_RDMA_WRITE_FUNC) ( tduffy> - DAT_EP_HANDLE, DAT_COUNT, struct dat_lmr_triplet *, DAT_DTO_COOKIE, tduffy> + DAT_EP_HANDLE, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, tduffy> const struct dat_rmr_triplet *, DAT_COMPLETION_FLAGS); tduffy> tduffy> typedef u32 (*DAT_EP_GET_STATUS_FUNC) ( tduffy> @@ -1191,10 +1190,10 @@ typedef u32 (*DAT_EP_FREE_FUNC) (DAT_EP_ tduffy> typedef u32 (*DAT_EP_RESET_FUNC) (DAT_EP_HANDLE); tduffy> tduffy> typedef u32 (*DAT_EP_RECV_QUERY_FUNC) ( tduffy> - DAT_EP_HANDLE, DAT_COUNT *, DAT_COUNT *); tduffy> + DAT_EP_HANDLE, int *, int *); tduffy> tduffy> typedef u32 (*DAT_EP_SET_WATERMARK_FUNC) ( tduffy> - DAT_EP_HANDLE, DAT_COUNT, DAT_COUNT); tduffy> + DAT_EP_HANDLE, int, int); tduffy> tduffy> typedef u32 (*DAT_LMR_FREE_FUNC) (DAT_LMR_HANDLE); tduffy> tduffy> @@ -1248,17 +1247,17 @@ typedef u32 (*DAT_PZ_FREE_FUNC) (DAT_PZ_ tduffy> typedef u32 (*DAT_SRQ_CREATE_FUNC) ( tduffy> DAT_IA_HANDLE, DAT_PZ_HANDLE, struct dat_srq_attr *, DAT_SRQ_HANDLE *); tduffy> tduffy> -typedef u32 (*DAT_SRQ_SET_LW_FUNC) (DAT_SRQ_HANDLE, DAT_COUNT); tduffy> +typedef u32 (*DAT_SRQ_SET_LW_FUNC) (DAT_SRQ_HANDLE, int); tduffy> tduffy> typedef u32 (*DAT_SRQ_FREE_FUNC) (DAT_SRQ_HANDLE); tduffy> tduffy> typedef u32 (*DAT_SRQ_QUERY_FUNC) (DAT_SRQ_HANDLE, tduffy> struct dat_srq_param *); tduffy> tduffy> -typedef u32 (*DAT_SRQ_RESIZE_FUNC) (DAT_SRQ_HANDLE, DAT_COUNT); tduffy> +typedef u32 (*DAT_SRQ_RESIZE_FUNC) (DAT_SRQ_HANDLE, int); tduffy> tduffy> typedef u32 (*DAT_SRQ_POST_RECV_FUNC) ( tduffy> - DAT_SRQ_HANDLE, DAT_COUNT, struct dat_lmr_triplet *, DAT_DTO_COOKIE); tduffy> + DAT_SRQ_HANDLE, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE); tduffy> tduffy> typedef u32 (*DAT_LMR_KCREATE_FUNC) ( tduffy> DAT_IA_HANDLE, DAT_MEM_TYPE, DAT_REGION_DESCRIPTION, DAT_VLEN, tduffy> @@ -1271,7 +1270,7 @@ typedef u32 (*DAT_IA_MEMTYPE_HINT_FUNC) tduffy> DAT_VLEN *, DAT_VADDR *); tduffy> tduffy> typedef u32 (*DAT_EVD_KCREATE_FUNC) ( tduffy> - DAT_IA_HANDLE, DAT_COUNT, DAT_UPCALL_POLICY, tduffy> + DAT_IA_HANDLE, int, DAT_UPCALL_POLICY, tduffy> const struct dat_upcall_object *, tduffy> DAT_EVD_FLAGS, DAT_EVD_HANDLE *); tduffy> tduffy> @@ -1378,9 +1377,10 @@ extern u32 dat_registry_remove_provider( tduffy> /* tduffy> * DAT registry functions for consumers tduffy> */ tduffy> -extern u32 dat_ia_openv(const DAT_NAME_PTR, DAT_COUNT, tduffy> - DAT_EVD_HANDLE *, DAT_IA_HANDLE *, tduffy> - u32, u32, boolean_t); tduffy> +extern u32 dat_ia_openv(const DAT_NAME_PTR name, int async_event_qlen, tduffy> + DAT_EVD_HANDLE *async_event_handle, tduffy> + DAT_IA_HANDLE *ia_handle, u32 dat_major, u32 dat_minor, tduffy> + boolean_t thread_safety); tduffy> tduffy> #define dat_ia_open(name, qlen, async_evd, ia) \ tduffy> dat_ia_openv((name), (qlen), (async_evd), (ia), \ tduffy> @@ -1391,8 +1391,8 @@ extern u32 dat_ia_close(DAT_IA_HANDLE, D tduffy> tduffy> extern u32 dat_strerror(u32, const char **, const char **); tduffy> tduffy> -extern u32 dat_registry_list_providers( tduffy> - DAT_COUNT, DAT_COUNT *, struct dat_provider_info *dat_provider_list[]); tduffy> +extern u32 dat_registry_list_providers(int max_to_return, int *entries_returned, tduffy> + struct dat_provider_info *dat_provider_list[]); tduffy> tduffy> /* tduffy> * inline functions for consumers tduffy> @@ -1402,64 +1402,57 @@ extern u32 dat_registry_list_providers( tduffy> #define DAT_CALL_PROVIDER_FUNC(func, handle, ...) \ tduffy> DAT_HANDLE_TO_PROVIDER(handle)->func(handle, ##__VA_ARGS__) tduffy> tduffy> -static inline u32 dat_set_consumer_context(DAT_HANDLE handle, tduffy> - DAT_CONTEXT context) tduffy> +static inline u32 dat_set_consumer_context(DAT_HANDLE handle, tduffy> + DAT_CONTEXT context) tduffy> { tduffy> - return DAT_CALL_PROVIDER_FUNC( tduffy> - set_consumer_context_func, handle, context); tduffy> + return DAT_CALL_PROVIDER_FUNC(set_consumer_context_func, handle, tduffy> + context); tduffy> } tduffy> tduffy> -static inline u32 dat_get_consumer_context(DAT_HANDLE handle, tduffy> - DAT_CONTEXT *context) tduffy> +static inline u32 dat_get_consumer_context(DAT_HANDLE handle, tduffy> + DAT_CONTEXT *context) tduffy> { tduffy> - return DAT_CALL_PROVIDER_FUNC( tduffy> - get_consumer_context_func, handle, context); tduffy> + return DAT_CALL_PROVIDER_FUNC(get_consumer_context_func, handle, tduffy> + context); tduffy> } tduffy> tduffy> -static inline u32 dat_get_handle_type(DAT_HANDLE handle, tduffy> - DAT_HANDLE_TYPE *type) tduffy> +static inline u32 dat_get_handle_type(DAT_HANDLE handle, DAT_HANDLE_TYPE *type) tduffy> { tduffy> return DAT_CALL_PROVIDER_FUNC(get_handle_type_func, handle, type); tduffy> } tduffy> tduffy> -static inline u32 dat_ia_memtype_hint(DAT_IA_HANDLE ia, tduffy> - DAT_MEM_TYPE mem_type, tduffy> - DAT_VLEN length, tduffy> - DAT_MEM_OPTIMIZE_FLAGS mem_optimize, tduffy> - DAT_VLEN * preferred_length, tduffy> - DAT_VADDR * preferred_alignment) tduffy> -{ tduffy> - return DAT_CALL_PROVIDER_FUNC( tduffy> - ia_memtype_hint_func, ia, mem_type, length, mem_optimize, tduffy> - preferred_length, preferred_alignment); tduffy> +static inline u32 dat_ia_memtype_hint(DAT_IA_HANDLE ia, DAT_MEM_TYPE mem_type, tduffy> + DAT_VLEN length, tduffy> + DAT_MEM_OPTIMIZE_FLAGS mem_optimize, tduffy> + DAT_VLEN *preferred_length, tduffy> + DAT_VADDR *preferred_alignment) tduffy> +{ tduffy> + return DAT_CALL_PROVIDER_FUNC(ia_memtype_hint_func, ia, mem_type, tduffy> + length, mem_optimize, preferred_length, tduffy> + preferred_alignment); tduffy> } tduffy> tduffy> -static inline u32 dat_ia_query(DAT_IA_HANDLE ia, tduffy> - DAT_EVD_HANDLE * async_evd, tduffy> - struct dat_ia_attr *ia_attr, tduffy> - struct dat_provider_attr *provider_attr) tduffy> +static inline u32 dat_ia_query(DAT_IA_HANDLE ia, DAT_EVD_HANDLE *async_evd, tduffy> + struct dat_ia_attr *ia_attr, tduffy> + struct dat_provider_attr *provider_attr) tduffy> { tduffy> return DAT_CALL_PROVIDER_FUNC( tduffy> ia_query_func, ia, async_evd, ia_attr, provider_attr); tduffy> } tduffy> tduffy> -static inline u32 dat_cr_accept(DAT_CR_HANDLE cr, tduffy> - DAT_EP_HANDLE ep, tduffy> - DAT_COUNT private_data_size, tduffy> - const void *private_data) tduffy> +static inline u32 dat_cr_accept(DAT_CR_HANDLE cr, DAT_EP_HANDLE ep, tduffy> + int private_data_size, const void *private_data) tduffy> { tduffy> return DAT_CALL_PROVIDER_FUNC( tduffy> cr_accept_func, cr, ep, private_data_size, private_data); tduffy> } tduffy> tduffy> -static inline u32 dat_cr_handoff(DAT_CR_HANDLE cr, tduffy> - DAT_CONN_QUAL handoff) tduffy> +static inline u32 dat_cr_handoff(DAT_CR_HANDLE cr, DAT_CONN_QUAL handoff) tduffy> { tduffy> return DAT_CALL_PROVIDER_FUNC(cr_handoff_func, cr, handoff); tduffy> } tduffy> tduffy> -static inline u32 dat_cr_query(DAT_CR_HANDLE cr, tduffy> - struct dat_cr_param *param) tduffy> +static inline u32 dat_cr_query(DAT_CR_HANDLE cr, struct dat_cr_param *param) tduffy> { tduffy> return DAT_CALL_PROVIDER_FUNC(cr_query_func, cr, param); tduffy> } tduffy> @@ -1469,8 +1462,7 @@ static inline u32 dat_cr_reject(DAT_CR_H tduffy> return DAT_CALL_PROVIDER_FUNC(cr_reject_func, cr); tduffy> } tduffy> tduffy> -static inline u32 dat_evd_dequeue(DAT_EVD_HANDLE evd, tduffy> - struct dat_event *event) tduffy> +static inline u32 dat_evd_dequeue(DAT_EVD_HANDLE evd, struct dat_event *event) tduffy> { tduffy> return DAT_CALL_PROVIDER_FUNC(evd_dequeue_func, evd, event); tduffy> } tduffy> @@ -1480,94 +1472,81 @@ static inline u32 dat_evd_free(DAT_EVD_H tduffy> return DAT_CALL_PROVIDER_FUNC(evd_free_func, evd); tduffy> } tduffy> tduffy> -static inline u32 dat_evd_kcreate(DAT_IA_HANDLE ia, tduffy> - DAT_COUNT qlen, tduffy> - DAT_UPCALL_POLICY policy, tduffy> - const struct dat_upcall_object *upcall, tduffy> - DAT_EVD_FLAGS flags, tduffy> - DAT_EVD_HANDLE * evd) tduffy> +static inline u32 dat_evd_kcreate(DAT_IA_HANDLE ia, int qlen, tduffy> + DAT_UPCALL_POLICY policy, tduffy> + const struct dat_upcall_object *upcall, tduffy> + DAT_EVD_FLAGS flags, DAT_EVD_HANDLE * evd) tduffy> { tduffy> - return DAT_CALL_PROVIDER_FUNC( tduffy> - evd_kcreate_func, ia, qlen, policy, upcall, flags, evd); tduffy> + return DAT_CALL_PROVIDER_FUNC(evd_kcreate_func, ia, qlen, policy, tduffy> + upcall, flags, evd); tduffy> } tduffy> tduffy> -static inline u32 dat_evd_modify_upcall(DAT_EVD_HANDLE evd, tduffy> - DAT_UPCALL_POLICY policy, tduffy> - const struct dat_upcall_object *upcall) tduffy> +static inline u32 dat_evd_modify_upcall(DAT_EVD_HANDLE evd, tduffy> + DAT_UPCALL_POLICY policy, tduffy> + const struct dat_upcall_object *upcall) tduffy> { tduffy> - return DAT_CALL_PROVIDER_FUNC( tduffy> - evd_modify_upcall_func, evd, policy, upcall); tduffy> + return DAT_CALL_PROVIDER_FUNC(evd_modify_upcall_func, evd, policy, tduffy> + upcall); tduffy> } tduffy> tduffy> -static inline u32 dat_evd_post_se(DAT_EVD_HANDLE evd, tduffy> - const struct dat_event *event) tduffy> +static inline u32 dat_evd_post_se(DAT_EVD_HANDLE evd, tduffy> + const struct dat_event *event) tduffy> { tduffy> return DAT_CALL_PROVIDER_FUNC(evd_post_se_func, evd, event); tduffy> } tduffy> tduffy> -static inline u32 dat_evd_query(DAT_EVD_HANDLE evd, tduffy> - struct dat_evd_param *param) tduffy> +static inline u32 dat_evd_query(DAT_EVD_HANDLE evd, struct dat_evd_param *param) tduffy> { tduffy> return DAT_CALL_PROVIDER_FUNC(evd_query_func, evd, param); tduffy> } tduffy> tduffy> -static inline u32 dat_evd_resize(DAT_EVD_HANDLE evd, DAT_COUNT qlen) tduffy> +static inline u32 dat_evd_resize(DAT_EVD_HANDLE evd, int qlen) tduffy> { tduffy> return DAT_CALL_PROVIDER_FUNC(evd_resize_func, evd, qlen); tduffy> } tduffy> tduffy> -static inline u32 dat_ep_connect(DAT_EP_HANDLE ep, tduffy> - DAT_IA_ADDRESS_PTR ia_addr, tduffy> - DAT_CONN_QUAL conn_qual, tduffy> - DAT_TIMEOUT timeout, tduffy> - DAT_COUNT private_data_size, tduffy> - const void *private_data, tduffy> - DAT_QOS qos, tduffy> - DAT_CONNECT_FLAGS flags) tduffy> -{ tduffy> - return DAT_CALL_PROVIDER_FUNC( tduffy> - ep_connect_func, ep, ia_addr, conn_qual, timeout, tduffy> - private_data_size, private_data, qos, flags); tduffy> -} tduffy> - tduffy> -static inline u32 dat_ep_create(DAT_IA_HANDLE ia, tduffy> - DAT_PZ_HANDLE pz, tduffy> - DAT_EVD_HANDLE in_evd, tduffy> - DAT_EVD_HANDLE out_evd, tduffy> - DAT_EVD_HANDLE connect_evd, tduffy> - const struct dat_ep_attr *attr, tduffy> - DAT_EP_HANDLE * ep) tduffy> -{ tduffy> - return DAT_CALL_PROVIDER_FUNC( tduffy> - ep_create_func, ia, pz, in_evd, out_evd, connect_evd, attr, ep); tduffy> +static inline u32 dat_ep_connect(DAT_EP_HANDLE ep, DAT_IA_ADDRESS_PTR ia_addr, tduffy> + DAT_CONN_QUAL conn_qual, DAT_TIMEOUT timeout, tduffy> + int private_data_size, tduffy> + const void *private_data, DAT_QOS qos, tduffy> + DAT_CONNECT_FLAGS flags) tduffy> +{ tduffy> + return DAT_CALL_PROVIDER_FUNC(ep_connect_func, ep, ia_addr, conn_qual, tduffy> + timeout, private_data_size, private_data, tduffy> + qos, flags); tduffy> +} tduffy> + tduffy> +static inline u32 dat_ep_create(DAT_IA_HANDLE ia, DAT_PZ_HANDLE pz, tduffy> + DAT_EVD_HANDLE in_evd, DAT_EVD_HANDLE out_evd, tduffy> + DAT_EVD_HANDLE connect_evd, tduffy> + const struct dat_ep_attr *attr, tduffy> + DAT_EP_HANDLE *ep) tduffy> +{ tduffy> + return DAT_CALL_PROVIDER_FUNC(ep_create_func, ia, pz, in_evd, out_evd, tduffy> + connect_evd, attr, ep); tduffy> } tduffy> tduffy> tduffy> -static inline u32 dat_ep_create_with_srq(DAT_IA_HANDLE ia, tduffy> - DAT_PZ_HANDLE pz, tduffy> - DAT_EVD_HANDLE in_evd, tduffy> - DAT_EVD_HANDLE out_evd, tduffy> - DAT_EVD_HANDLE connect_evd, tduffy> - DAT_SRQ_HANDLE srq, tduffy> - const struct dat_ep_attr *attr, tduffy> - DAT_EP_HANDLE * ep) tduffy> +static inline u32 dat_ep_create_with_srq(DAT_IA_HANDLE ia, DAT_PZ_HANDLE pz, tduffy> + DAT_EVD_HANDLE in_evd, tduffy> + DAT_EVD_HANDLE out_evd, tduffy> + DAT_EVD_HANDLE connect_evd, tduffy> + DAT_SRQ_HANDLE srq, tduffy> + const struct dat_ep_attr *attr, tduffy> + DAT_EP_HANDLE *ep) tduffy> { tduffy> return DAT_CALL_PROVIDER_FUNC(ep_create_with_srq_func, ia, pz, in_evd, tduffy> out_evd, connect_evd, srq, attr, ep); tduffy> } tduffy> tduffy> -static inline u32 dat_ep_disconnect(DAT_EP_HANDLE ep, tduffy> - DAT_CLOSE_FLAGS flags) tduffy> +static inline u32 dat_ep_disconnect(DAT_EP_HANDLE ep, DAT_CLOSE_FLAGS flags) tduffy> { tduffy> return DAT_CALL_PROVIDER_FUNC(ep_disconnect_func, ep, flags); tduffy> } tduffy> tduffy> -static inline u32 dat_ep_dup_connect(DAT_EP_HANDLE ep, tduffy> - DAT_EP_HANDLE dup_ep, tduffy> - DAT_TIMEOUT timeout, tduffy> - DAT_COUNT private_data_size, tduffy> - const void *private_data, tduffy> - DAT_QOS qos) tduffy> +static inline u32 dat_ep_dup_connect(DAT_EP_HANDLE ep, DAT_EP_HANDLE dup_ep, tduffy> + DAT_TIMEOUT timeout, int private_data_size, tduffy> + const void *private_data, DAT_QOS qos) tduffy> { tduffy> return DAT_CALL_PROVIDER_FUNC(ep_dup_connect_func, ep, dup_ep, timeout, tduffy> private_data_size, private_data, qos); tduffy> @@ -1578,56 +1557,50 @@ static inline u32 dat_ep_free(DAT_EP_HAN tduffy> return DAT_CALL_PROVIDER_FUNC(ep_free_func, ep); tduffy> } tduffy> tduffy> -static inline u32 dat_ep_get_status(DAT_EP_HANDLE ep, tduffy> - DAT_EP_STATE * state, tduffy> - boolean_t *recv_idle, tduffy> - boolean_t *req_idle) tduffy> +static inline u32 dat_ep_get_status(DAT_EP_HANDLE ep, DAT_EP_STATE *state, tduffy> + boolean_t *recv_idle, boolean_t *req_idle) tduffy> { tduffy> - return DAT_CALL_PROVIDER_FUNC( tduffy> - ep_get_status_func, ep, state, recv_idle, req_idle); tduffy> + return DAT_CALL_PROVIDER_FUNC(ep_get_status_func, ep, state, recv_idle, tduffy> + req_idle); tduffy> } tduffy> tduffy> -static inline u32 dat_ep_modify(DAT_EP_HANDLE ep, tduffy> - DAT_EP_PARAM_MASK mask, tduffy> - const struct dat_ep_param *param) tduffy> +static inline u32 dat_ep_modify(DAT_EP_HANDLE ep, DAT_EP_PARAM_MASK mask, tduffy> + const struct dat_ep_param *param) tduffy> { tduffy> return DAT_CALL_PROVIDER_FUNC(ep_modify_func, ep, mask, param); tduffy> } tduffy> tduffy> -static inline u32 dat_ep_post_rdma_read(DAT_EP_HANDLE ep, tduffy> - DAT_COUNT size, tduffy> - struct dat_lmr_triplet *local_iov, tduffy> - DAT_DTO_COOKIE cookie, tduffy> - const struct dat_rmr_triplet *remote_iov, tduffy> - DAT_COMPLETION_FLAGS flags) tduffy> +static inline u32 dat_ep_post_rdma_read(DAT_EP_HANDLE ep, int size, tduffy> + struct dat_lmr_triplet *local_iov, tduffy> + DAT_DTO_COOKIE cookie, tduffy> + const struct dat_rmr_triplet *remote_iov, tduffy> + DAT_COMPLETION_FLAGS flags) tduffy> { tduffy> return DAT_CALL_PROVIDER_FUNC(ep_post_rdma_read_func, ep, size, tduffy> local_iov, cookie, remote_iov, flags); tduffy> } tduffy> tduffy> -static inline u32 dat_ep_post_rdma_write(DAT_EP_HANDLE ep, tduffy> - DAT_COUNT size, tduffy> - struct dat_lmr_triplet *local_iov, tduffy> - DAT_DTO_COOKIE cookie, tduffy> - const struct dat_rmr_triplet *remote_iov, tduffy> - DAT_COMPLETION_FLAGS flags) tduffy> +static inline u32 dat_ep_post_rdma_write(DAT_EP_HANDLE ep, int size, tduffy> + struct dat_lmr_triplet *local_iov, tduffy> + DAT_DTO_COOKIE cookie, tduffy> + const struct dat_rmr_triplet *remote_iov, tduffy> + DAT_COMPLETION_FLAGS flags) tduffy> { tduffy> return DAT_CALL_PROVIDER_FUNC(ep_post_rdma_write_func, ep, size, tduffy> local_iov, cookie, remote_iov, flags); tduffy> } tduffy> tduffy> -static inline u32 dat_ep_post_recv(DAT_EP_HANDLE ep, tduffy> - DAT_COUNT size, tduffy> - struct dat_lmr_triplet *local_iov, tduffy> - DAT_DTO_COOKIE cookie, tduffy> - DAT_COMPLETION_FLAGS flags) tduffy> +static inline u32 dat_ep_post_recv(DAT_EP_HANDLE ep, int size, tduffy> + struct dat_lmr_triplet *local_iov, tduffy> + DAT_DTO_COOKIE cookie, tduffy> + DAT_COMPLETION_FLAGS flags) tduffy> { tduffy> - return DAT_CALL_PROVIDER_FUNC( tduffy> - ep_post_recv_func, ep, size, local_iov, cookie, flags); tduffy> + return DAT_CALL_PROVIDER_FUNC(ep_post_recv_func, ep, size, local_iov, tduffy> + cookie, flags); tduffy> } tduffy> tduffy> static inline u32 dat_ep_post_send(DAT_EP_HANDLE ep, tduffy> - DAT_COUNT size, tduffy> + int size, tduffy> struct dat_lmr_triplet *local_iov, tduffy> DAT_DTO_COOKIE cookie, tduffy> DAT_COMPLETION_FLAGS flags) tduffy> @@ -1641,12 +1614,11 @@ static inline u32 dat_ep_query(DAT_EP_HA tduffy> return DAT_CALL_PROVIDER_FUNC(ep_query_func, ep, param); tduffy> } tduffy> tduffy> -static inline u32 dat_ep_recv_query(DAT_EP_HANDLE ep, tduffy> - DAT_COUNT * bufs_alloc, tduffy> - DAT_COUNT * bufs_avail) tduffy> +static inline u32 dat_ep_recv_query(DAT_EP_HANDLE ep, int *bufs_alloc, tduffy> + int *bufs_avail) tduffy> { tduffy> - return DAT_CALL_PROVIDER_FUNC( tduffy> - ep_recv_query_func, ep, bufs_alloc, bufs_avail); tduffy> + return DAT_CALL_PROVIDER_FUNC(ep_recv_query_func, ep, bufs_alloc, tduffy> + bufs_avail); tduffy> } tduffy> tduffy> static inline u32 dat_ep_reset(DAT_EP_HANDLE ep) tduffy> @@ -1654,16 +1626,15 @@ static inline u32 dat_ep_reset(DAT_EP_HA tduffy> return DAT_CALL_PROVIDER_FUNC(ep_reset_func, ep); tduffy> } tduffy> tduffy> -static inline u32 dat_ep_set_watermark(DAT_EP_HANDLE ep, tduffy> - DAT_COUNT soft_high_watermark, tduffy> - DAT_COUNT hard_high_watermark) tduffy> +static inline u32 dat_ep_set_watermark(DAT_EP_HANDLE ep, tduffy> + int soft_high_watermark, tduffy> + int hard_high_watermark) tduffy> { tduffy> return DAT_CALL_PROVIDER_FUNC(ep_set_watermark_func, ep, tduffy> soft_high_watermark, hard_high_watermark); tduffy> } tduffy> tduffy> -static inline u32 dat_lmr_query(DAT_LMR_HANDLE lmr, tduffy> - struct dat_lmr_param *param) tduffy> +static inline u32 dat_lmr_query(DAT_LMR_HANDLE lmr, struct dat_lmr_param *param) tduffy> { tduffy> return DAT_CALL_PROVIDER_FUNC(lmr_query_func, lmr, param); tduffy> } tduffy> @@ -1673,23 +1644,23 @@ static inline u32 dat_lmr_free(DAT_LMR_H tduffy> return DAT_CALL_PROVIDER_FUNC(lmr_free_func, lmr); tduffy> } tduffy> tduffy> -static inline u32 dat_lmr_sync_rdma_read(DAT_IA_HANDLE ia, tduffy> - const struct dat_lmr_triplet *iovs, tduffy> - DAT_VLEN num_iovs) tduffy> +static inline u32 dat_lmr_sync_rdma_read(DAT_IA_HANDLE ia, tduffy> + const struct dat_lmr_triplet *iovs, tduffy> + DAT_VLEN num_iovs) tduffy> { tduffy> - return DAT_CALL_PROVIDER_FUNC( tduffy> - lmr_sync_rdma_read_func, ia, iovs, num_iovs); tduffy> + return DAT_CALL_PROVIDER_FUNC(lmr_sync_rdma_read_func, ia, iovs, tduffy> + num_iovs); tduffy> } tduffy> tduffy> static inline u32 dat_lmr_sync_rdma_write(DAT_IA_HANDLE ia, tduffy> - const struct dat_lmr_triplet *iovs, tduffy> - DAT_VLEN num_iovs) tduffy> + const struct dat_lmr_triplet *iovs, tduffy> + DAT_VLEN num_iovs) tduffy> { tduffy> - return DAT_CALL_PROVIDER_FUNC( tduffy> - lmr_sync_rdma_write_func, ia, iovs, num_iovs); tduffy> + return DAT_CALL_PROVIDER_FUNC(lmr_sync_rdma_write_func, ia, iovs, tduffy> + num_iovs); tduffy> } tduffy> tduffy> -static inline u32 dat_rmr_create(DAT_PZ_HANDLE pz, DAT_RMR_HANDLE * rmr) tduffy> +static inline u32 dat_rmr_create(DAT_PZ_HANDLE pz, DAT_RMR_HANDLE *rmr) tduffy> { tduffy> return DAT_CALL_PROVIDER_FUNC(rmr_create_func, pz, rmr); tduffy> } tduffy> @@ -1699,13 +1670,12 @@ static inline u32 dat_rmr_query(DAT_RMR_ tduffy> return DAT_CALL_PROVIDER_FUNC(rmr_query_func, rmr, param); tduffy> } tduffy> tduffy> -static inline u32 dat_rmr_bind(DAT_RMR_HANDLE rmr, tduffy> - const struct dat_lmr_triplet *iov, tduffy> - DAT_MEM_PRIV_FLAGS mem_flags, tduffy> - DAT_EP_HANDLE ep, tduffy> - DAT_RMR_COOKIE cookie, tduffy> - DAT_COMPLETION_FLAGS comp_flags, tduffy> - DAT_RMR_CONTEXT * context) tduffy> +static inline u32 dat_rmr_bind(DAT_RMR_HANDLE rmr, tduffy> + const struct dat_lmr_triplet *iov, tduffy> + DAT_MEM_PRIV_FLAGS mem_flags, tduffy> + DAT_EP_HANDLE ep, DAT_RMR_COOKIE cookie, tduffy> + DAT_COMPLETION_FLAGS comp_flags, tduffy> + DAT_RMR_CONTEXT *context) tduffy> { tduffy> return DAT_CALL_PROVIDER_FUNC(rmr_bind_func, rmr, iov, mem_flags, ep, tduffy> cookie, comp_flags, context); tduffy> @@ -1716,28 +1686,23 @@ static inline u32 dat_rmr_free(DAT_RMR_H tduffy> return DAT_CALL_PROVIDER_FUNC(rmr_free_func, rmr); tduffy> } tduffy> tduffy> -static inline u32 dat_psp_create(DAT_IA_HANDLE ia, tduffy> - DAT_CONN_QUAL conn_qual, tduffy> - DAT_EVD_HANDLE evd, tduffy> - DAT_PSP_FLAGS flags, tduffy> - DAT_PSP_HANDLE * psp) tduffy> +static inline u32 dat_psp_create(DAT_IA_HANDLE ia, DAT_CONN_QUAL conn_qual, tduffy> + DAT_EVD_HANDLE evd, DAT_PSP_FLAGS flags, tduffy> + DAT_PSP_HANDLE *psp) tduffy> { tduffy> - return DAT_CALL_PROVIDER_FUNC( tduffy> - psp_create_func, ia, conn_qual, evd, flags, psp); tduffy> + return DAT_CALL_PROVIDER_FUNC(psp_create_func, ia, conn_qual, evd, tduffy> + flags, psp); tduffy> } tduffy> tduffy> -static inline u32 dat_psp_create_any(DAT_IA_HANDLE ia, tduffy> - DAT_CONN_QUAL * conn_qual, tduffy> - DAT_EVD_HANDLE evd, tduffy> - DAT_PSP_FLAGS flags, tduffy> - DAT_PSP_HANDLE * psp) tduffy> +static inline u32 dat_psp_create_any(DAT_IA_HANDLE ia, DAT_CONN_QUAL *conn_qual, tduffy> + DAT_EVD_HANDLE evd, DAT_PSP_FLAGS flags, tduffy> + DAT_PSP_HANDLE *psp) tduffy> { tduffy> - return DAT_CALL_PROVIDER_FUNC( tduffy> - psp_create_any_func, ia, conn_qual, evd, flags, psp); tduffy> + return DAT_CALL_PROVIDER_FUNC(psp_create_any_func, ia, conn_qual, evd, tduffy> + flags, psp); tduffy> } tduffy> tduffy> -static inline u32 dat_psp_query(DAT_PSP_HANDLE psp, tduffy> - struct dat_psp_param *param) tduffy> +static inline u32 dat_psp_query(DAT_PSP_HANDLE psp, struct dat_psp_param *param) tduffy> { tduffy> return DAT_CALL_PROVIDER_FUNC(psp_query_func, psp, param); tduffy> } tduffy> @@ -1747,18 +1712,15 @@ static inline u32 dat_psp_free(DAT_PSP_H tduffy> return DAT_CALL_PROVIDER_FUNC(psp_free_func, psp); tduffy> } tduffy> tduffy> -static inline u32 dat_rsp_create(DAT_IA_HANDLE ia, tduffy> - DAT_CONN_QUAL conn_qual, tduffy> - DAT_EP_HANDLE ep, tduffy> - DAT_EVD_HANDLE evd, tduffy> - DAT_RSP_HANDLE * rsp) tduffy> +static inline u32 dat_rsp_create(DAT_IA_HANDLE ia, DAT_CONN_QUAL conn_qual, tduffy> + DAT_EP_HANDLE ep, DAT_EVD_HANDLE evd, tduffy> + DAT_RSP_HANDLE *rsp) tduffy> { tduffy> - return DAT_CALL_PROVIDER_FUNC( tduffy> - rsp_create_func, ia, conn_qual, ep, evd, rsp); tduffy> + return DAT_CALL_PROVIDER_FUNC(rsp_create_func, ia, conn_qual, ep, evd, tduffy> + rsp); tduffy> } tduffy> tduffy> -static inline u32 dat_rsp_query(DAT_RSP_HANDLE rsp, tduffy> - struct dat_rsp_param *param) tduffy> +static inline u32 dat_rsp_query(DAT_RSP_HANDLE rsp, struct dat_rsp_param *param) tduffy> { tduffy> return DAT_CALL_PROVIDER_FUNC(rsp_query_func, rsp, param); tduffy> } tduffy> @@ -1768,13 +1730,12 @@ static inline u32 dat_rsp_free(DAT_RSP_H tduffy> return DAT_CALL_PROVIDER_FUNC(rsp_free_func, rsp); tduffy> } tduffy> tduffy> -static inline u32 dat_pz_create(DAT_IA_HANDLE ia, DAT_PZ_HANDLE * pz) tduffy> +static inline u32 dat_pz_create(DAT_IA_HANDLE ia, DAT_PZ_HANDLE *pz) tduffy> { tduffy> return DAT_CALL_PROVIDER_FUNC(pz_create_func, ia, pz); tduffy> } tduffy> tduffy> -static inline u32 dat_pz_query(DAT_PZ_HANDLE pz, tduffy> - struct dat_pz_param *param) tduffy> +static inline u32 dat_pz_query(DAT_PZ_HANDLE pz, struct dat_pz_param *param) tduffy> { tduffy> return DAT_CALL_PROVIDER_FUNC(pz_query_func, pz, param); tduffy> } tduffy> @@ -1784,10 +1745,9 @@ static inline u32 dat_pz_free(DAT_PZ_HAN tduffy> return DAT_CALL_PROVIDER_FUNC(pz_free_func, pz); tduffy> } tduffy> tduffy> -static inline u32 dat_srq_create(DAT_IA_HANDLE ia, tduffy> - DAT_PZ_HANDLE pz, tduffy> - struct dat_srq_attr *attr, tduffy> - DAT_SRQ_HANDLE * srq) tduffy> +static inline u32 dat_srq_create(DAT_IA_HANDLE ia, DAT_PZ_HANDLE pz, tduffy> + struct dat_srq_attr *attr, tduffy> + DAT_SRQ_HANDLE *srq) tduffy> { tduffy> return DAT_CALL_PROVIDER_FUNC(srq_create_func, ia, pz, attr, srq); tduffy> } tduffy> @@ -1797,50 +1757,44 @@ static inline u32 dat_srq_free(DAT_SRQ_H tduffy> return DAT_CALL_PROVIDER_FUNC(srq_free_func, srq); tduffy> } tduffy> tduffy> -static inline u32 dat_srq_post_recv(DAT_SRQ_HANDLE srq, tduffy> - DAT_COUNT num_iovs, tduffy> - struct dat_lmr_triplet *iovs, tduffy> - DAT_DTO_COOKIE cookie) tduffy> +static inline u32 dat_srq_post_recv(DAT_SRQ_HANDLE srq, int num_iovs, tduffy> + struct dat_lmr_triplet *iovs, tduffy> + DAT_DTO_COOKIE cookie) tduffy> { tduffy> - return DAT_CALL_PROVIDER_FUNC( tduffy> - srq_post_recv_func, srq, num_iovs, iovs, cookie); tduffy> + return DAT_CALL_PROVIDER_FUNC(srq_post_recv_func, srq, num_iovs, iovs, tduffy> + cookie); tduffy> } tduffy> tduffy> -static inline u32 dat_srq_query(DAT_SRQ_HANDLE srq, tduffy> - struct dat_srq_param *param) tduffy> +static inline u32 dat_srq_query(DAT_SRQ_HANDLE srq, struct dat_srq_param *param) tduffy> { tduffy> return DAT_CALL_PROVIDER_FUNC(srq_query_func, srq, param); tduffy> } tduffy> tduffy> -static inline u32 dat_srq_resize(DAT_SRQ_HANDLE srq, tduffy> - DAT_COUNT max_recv_dtos) tduffy> +static inline u32 dat_srq_resize(DAT_SRQ_HANDLE srq, int max_recv_dtos) tduffy> { tduffy> return DAT_CALL_PROVIDER_FUNC(srq_resize_func, srq, max_recv_dtos); tduffy> } tduffy> tduffy> -static inline u32 dat_srq_set_lw(DAT_SRQ_HANDLE srq, tduffy> - DAT_COUNT low_watermark) tduffy> +static inline u32 dat_srq_set_lw(DAT_SRQ_HANDLE srq, int low_watermark) tduffy> { tduffy> return DAT_CALL_PROVIDER_FUNC(srq_set_lw_func, srq, low_watermark); tduffy> } tduffy> tduffy> -static inline u32 dat_lmr_kcreate(DAT_IA_HANDLE ia, tduffy> - DAT_MEM_TYPE type, tduffy> - DAT_REGION_DESCRIPTION region, tduffy> - DAT_VLEN len, tduffy> - DAT_PZ_HANDLE pz, tduffy> - DAT_MEM_PRIV_FLAGS privileges, tduffy> - DAT_MEM_OPTIMIZE_FLAGS optimization, tduffy> - DAT_LMR_HANDLE * lmr_handle, tduffy> - DAT_LMR_CONTEXT * lmr_context, tduffy> - DAT_RMR_CONTEXT * rmr_context, tduffy> - DAT_VLEN * registered_length, tduffy> - DAT_VADDR *registered_address) tduffy> -{ tduffy> - return DAT_CALL_PROVIDER_FUNC( tduffy> - lmr_kcreate_func, ia, type, region, len, pz, privileges, tduffy> - optimization, lmr_handle, lmr_context, rmr_context, tduffy> - registered_length, registered_address); tduffy> +static inline u32 dat_lmr_kcreate(DAT_IA_HANDLE ia, DAT_MEM_TYPE type, tduffy> + DAT_REGION_DESCRIPTION region, DAT_VLEN len, tduffy> + DAT_PZ_HANDLE pz, tduffy> + DAT_MEM_PRIV_FLAGS privileges, tduffy> + DAT_MEM_OPTIMIZE_FLAGS optimization, tduffy> + DAT_LMR_HANDLE *lmr_handle, tduffy> + DAT_LMR_CONTEXT *lmr_context, tduffy> + DAT_RMR_CONTEXT *rmr_context, tduffy> + DAT_VLEN *registered_length, tduffy> + DAT_VADDR *registered_address) tduffy> +{ tduffy> + return DAT_CALL_PROVIDER_FUNC(lmr_kcreate_func, ia, type, region, len, tduffy> + pz, privileges, optimization, lmr_handle, tduffy> + lmr_context, rmr_context, tduffy> + registered_length, registered_address); tduffy> } tduffy> tduffy> #endif /* DAT_H */ tduffy> From libor at topspin.com Fri May 13 10:01:49 2005 From: libor at topspin.com (Libor Michalek) Date: Fri, 13 May 2005 10:01:49 -0700 Subject: [openib-general] Re: [PATCH] (repost) sdp: replace mlock with get_user_pages In-Reply-To: <20050513135145.GA320@mellanox.co.il>; from mst@mellanox.co.il on Fri, May 13, 2005 at 04:51:45PM +0300 References: <20050505110158.GH30653@mellanox.co.il> <20050506175415.I26823@topspin.com> <20050510143232.GF2474@mellanox.co.il> <20050512151051.C23734@topspin.com> <52fywst4z3.fsf@topspin.com> <20050513135145.GA320@mellanox.co.il> Message-ID: <20050513100149.B24123@topspin.com> On Fri, May 13, 2005 at 04:51:45PM +0300, Michael S. Tsirkin wrote: > Quoting r. Roland Dreier : > > > > Libor> Always call aio_complete() immediately when > > Libor> iocb_complete() is called, and only spawn the work thread > > Libor> to unlock the memory after the aio_complete() call. The > > Libor> patch is below. > > Libor, I dont think its a good idea - this will break other assumptions, > like the assumption that the task mm isnt destroyed before we unlock > the memory. That's a good point. > > Another alternative would be to always complete aios asynchronously, > > which should preserve the order. I guess this would hurt latency for > > small ios... > > To avoid hurting latency, lets count the number of outstanding > asynchronous AIOs, and if there are asynchronous AIOs complete > all of them asynchronously. > > Does this make sense? Yes, except that the current iocb code does not reference individual sockets anywhere, and do_iocb_complete would have to be the function which decremented the per connection counter of outstanding AIOs. Also, since we don't have to do get_user_pages a second time on the send path, this would only need to be done on the recv path. -Libor From libor at topspin.com Fri May 13 09:47:26 2005 From: libor at topspin.com (Libor Michalek) Date: Fri, 13 May 2005 09:47:26 -0700 Subject: [openib-general] Re: use of virt_to_phys in sdp In-Reply-To: <20050513140600.GC320@mellanox.co.il>; from mst@mellanox.co.il on Fri, May 13, 2005 at 05:06:00PM +0300 References: <20050513140600.GC320@mellanox.co.il> Message-ID: <20050513094726.A24123@topspin.com> On Fri, May 13, 2005 at 05:06:00PM +0300, Michael S. Tsirkin wrote: > Use of virt_to_phys looks wrong in sdp_recv and sdp_send. > Must not the DMA API be used to get the physical address > to be put in ib_sge? It is incorrect. The regular send/recv path uses the DMA API correctly, but RDMA to internal buffers on both the send and receive path is still using plain old virt_to_phys. Also, does FMR map/unmap use the DMA API, or does that need to be taken care of as well? -Libor From jlentini at netapp.com Fri May 13 10:24:45 2005 From: jlentini at netapp.com (James Lentini) Date: Fri, 13 May 2005 13:24:45 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: remove typedef DAT_NAME_PTR In-Reply-To: <1115922278.9070.23.camel@duffman> References: <1115922278.9070.23.camel@duffman> Message-ID: Committed in revision 2336. On Thu, 12 May 2005, Tom Duffy wrote: tduffy> Remove typedef DAT_NAME_PTR. Also get rid of DAT_UVERYLONG & DAT_PADDR tduffy> that are not used anywhere outside of dat.h. Also, a little more tduffy> whitespace cleaning that should have been done with the DAT_RETURN tduffy> patch. tduffy> tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat/api.c linux-kernel2/dat/api.c tduffy> --- linux-kernel/dat/api.c 2005-05-12 10:08:24.774000000 -0700 tduffy> +++ linux-kernel2/dat/api.c 2005-05-12 11:02:59.847003000 -0700 tduffy> @@ -575,7 +575,7 @@ static u32 dat_strerror_minor(u32 value, tduffy> * Function: dat_ia_openv tduffy> ***********************************************************************/ tduffy> tduffy> -u32 dat_ia_openv(const DAT_NAME_PTR name, int async_event_qlen, tduffy> +u32 dat_ia_openv(const char *name, int async_event_qlen, tduffy> DAT_EVD_HANDLE *async_event_handle, DAT_IA_HANDLE *ia_handle, tduffy> u32 dat_major, u32 dat_minor, boolean_t thread_safety) tduffy> { tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat/dat.h linux-kernel2/dat/dat.h tduffy> --- linux-kernel/dat/dat.h 2005-05-12 10:25:26.111007000 -0700 tduffy> +++ linux-kernel2/dat/dat.h 2005-05-12 11:14:17.374001000 -0700 tduffy> @@ -37,11 +37,6 @@ tduffy> #define DAT_VERSION_MINOR 2 tduffy> #define DAT_THREADSAFE FALSE tduffy> tduffy> -/* unsigned longest native to compiler */ tduffy> -typedef unsigned long long DAT_UVERYLONG; tduffy> - tduffy> -typedef u64 DAT_PADDR; tduffy> - tduffy> /* tduffy> * All return codes are actually a 3-way tuple: tduffy> * tduffy> @@ -352,7 +347,6 @@ typedef enum dat_handle_type { tduffy> DAT_HANDLE_TYPE_SRQ tduffy> } DAT_HANDLE_TYPE; tduffy> tduffy> -typedef char *DAT_NAME_PTR; tduffy> #define DAT_NAME_MAX_LENGTH 256 tduffy> tduffy> struct dat_named_attr { tduffy> @@ -370,7 +364,7 @@ typedef enum { tduffy> typedef union dat_context { tduffy> void *as_ptr; tduffy> u64 as_64; tduffy> - DAT_UVERYLONG as_index; tduffy> + unsigned long long as_index; tduffy> } DAT_CONTEXT; tduffy> tduffy> typedef DAT_CONTEXT DAT_DTO_COOKIE; tduffy> @@ -983,7 +977,7 @@ typedef union dat_region_description { tduffy> DAT_LMR_HANDLE for_lmr_handle; tduffy> void *for_pointer; /* For kdapl only */ tduffy> void *for_array; /* For kdapl only */ tduffy> - DAT_PADDR for_pa; /* For kdapl only */ tduffy> + u64 for_pa; /* For kdapl only */ tduffy> } DAT_REGION_DESCRIPTION; tduffy> tduffy> /* LMR Arguments */ tduffy> @@ -1103,179 +1097,176 @@ struct dat_provider_attr { tduffy> * tduffy> ****************************************************************/ tduffy> tduffy> -typedef u32 (*DAT_IA_OPEN_FUNC) ( tduffy> - const DAT_NAME_PTR, int, DAT_EVD_HANDLE *, DAT_IA_HANDLE *); tduffy> +typedef u32 (*DAT_IA_OPEN_FUNC)(const char *, int, DAT_EVD_HANDLE *, tduffy> + DAT_IA_HANDLE *); tduffy> tduffy> -typedef u32 (*DAT_IA_OPENV_FUNC) ( tduffy> - const DAT_NAME_PTR, int, DAT_EVD_HANDLE *, DAT_IA_HANDLE *, tduffy> - u32, u32, boolean_t); tduffy> +typedef u32 (*DAT_IA_OPENV_FUNC)(const char *, int, DAT_EVD_HANDLE *, tduffy> + DAT_IA_HANDLE *, u32, u32, boolean_t); tduffy> tduffy> -typedef u32 (*DAT_IA_CLOSE_FUNC) (DAT_IA_HANDLE, DAT_CLOSE_FLAGS); tduffy> +typedef u32 (*DAT_IA_CLOSE_FUNC)(DAT_IA_HANDLE, DAT_CLOSE_FLAGS); tduffy> tduffy> -typedef u32 (*DAT_IA_QUERY_FUNC) (DAT_IA_HANDLE, DAT_EVD_HANDLE *, tduffy> - struct dat_ia_attr *, tduffy> - struct dat_provider_attr *); tduffy> +typedef u32 (*DAT_IA_QUERY_FUNC)(DAT_IA_HANDLE, DAT_EVD_HANDLE *, tduffy> + struct dat_ia_attr *, tduffy> + struct dat_provider_attr *); tduffy> tduffy> -typedef u32 (*DAT_SET_CONSUMER_CONTEXT_FUNC) (DAT_HANDLE, DAT_CONTEXT); tduffy> +typedef u32 (*DAT_SET_CONSUMER_CONTEXT_FUNC)(DAT_HANDLE, DAT_CONTEXT); tduffy> tduffy> -typedef u32 (*DAT_GET_CONSUMER_CONTEXT_FUNC) (DAT_HANDLE, DAT_CONTEXT *); tduffy> +typedef u32 (*DAT_GET_CONSUMER_CONTEXT_FUNC)(DAT_HANDLE, DAT_CONTEXT *); tduffy> tduffy> -typedef u32 (*DAT_GET_HANDLE_TYPE_FUNC) (DAT_HANDLE, DAT_HANDLE_TYPE *); tduffy> +typedef u32 (*DAT_GET_HANDLE_TYPE_FUNC)(DAT_HANDLE, DAT_HANDLE_TYPE *); tduffy> tduffy> -typedef u32 (*DAT_CR_QUERY_FUNC) (DAT_CR_HANDLE, struct dat_cr_param *); tduffy> +typedef u32 (*DAT_CR_QUERY_FUNC)(DAT_CR_HANDLE, struct dat_cr_param *); tduffy> tduffy> -typedef u32 (*DAT_CR_ACCEPT_FUNC) (DAT_CR_HANDLE, DAT_EP_HANDLE, int, tduffy> - const void*); tduffy> +typedef u32 (*DAT_CR_ACCEPT_FUNC)(DAT_CR_HANDLE, DAT_EP_HANDLE, int, tduffy> + const void*); tduffy> tduffy> -typedef u32 (*DAT_CR_REJECT_FUNC) (DAT_CR_HANDLE); tduffy> +typedef u32 (*DAT_CR_REJECT_FUNC)(DAT_CR_HANDLE); tduffy> tduffy> -typedef u32 (*DAT_CR_HANDOFF_FUNC) (DAT_CR_HANDLE, DAT_CONN_QUAL); tduffy> +typedef u32 (*DAT_CR_HANDOFF_FUNC)(DAT_CR_HANDLE, DAT_CONN_QUAL); tduffy> tduffy> -typedef u32 (*DAT_EVD_RESIZE_FUNC) (DAT_EVD_HANDLE, int); tduffy> +typedef u32 (*DAT_EVD_RESIZE_FUNC)(DAT_EVD_HANDLE, int); tduffy> tduffy> -typedef u32 (*DAT_EVD_POST_SE_FUNC) (DAT_EVD_HANDLE, tduffy> - const struct dat_event *); tduffy> +typedef u32 (*DAT_EVD_POST_SE_FUNC)(DAT_EVD_HANDLE, const struct dat_event *); tduffy> tduffy> -typedef u32 (*DAT_EVD_DEQUEUE_FUNC) (DAT_EVD_HANDLE, struct dat_event *); tduffy> +typedef u32 (*DAT_EVD_DEQUEUE_FUNC)(DAT_EVD_HANDLE, struct dat_event *); tduffy> tduffy> -typedef u32 (*DAT_EVD_FREE_FUNC) (DAT_EVD_HANDLE); tduffy> +typedef u32 (*DAT_EVD_FREE_FUNC)(DAT_EVD_HANDLE); tduffy> tduffy> -typedef u32 (*DAT_EVD_QUERY_FUNC) (DAT_EVD_HANDLE, tduffy> - struct dat_evd_param *); tduffy> +typedef u32 (*DAT_EVD_QUERY_FUNC)(DAT_EVD_HANDLE, struct dat_evd_param *); tduffy> tduffy> -typedef u32 (*DAT_EP_CREATE_FUNC) ( tduffy> - DAT_IA_HANDLE, DAT_PZ_HANDLE, DAT_EVD_HANDLE, DAT_EVD_HANDLE, tduffy> - DAT_EVD_HANDLE, const struct dat_ep_attr *, DAT_EP_HANDLE *); tduffy> +typedef u32 (*DAT_EP_CREATE_FUNC)(DAT_IA_HANDLE, DAT_PZ_HANDLE, DAT_EVD_HANDLE, tduffy> + DAT_EVD_HANDLE, DAT_EVD_HANDLE, tduffy> + const struct dat_ep_attr *, DAT_EP_HANDLE *); tduffy> tduffy> -typedef u32 (*DAT_EP_CREATE_WITH_SRQ_FUNC) ( tduffy> - DAT_IA_HANDLE, DAT_PZ_HANDLE, DAT_EVD_HANDLE, DAT_EVD_HANDLE, tduffy> - DAT_EVD_HANDLE, DAT_SRQ_HANDLE, const struct dat_ep_attr *, DAT_EP_HANDLE *); tduffy> +typedef u32 (*DAT_EP_CREATE_WITH_SRQ_FUNC)(DAT_IA_HANDLE, DAT_PZ_HANDLE, tduffy> + DAT_EVD_HANDLE, DAT_EVD_HANDLE, tduffy> + DAT_EVD_HANDLE, DAT_SRQ_HANDLE, tduffy> + const struct dat_ep_attr *, tduffy> + DAT_EP_HANDLE *); tduffy> tduffy> -typedef u32 (*DAT_EP_QUERY_FUNC) (DAT_EP_HANDLE, struct dat_ep_param *); tduffy> +typedef u32 (*DAT_EP_QUERY_FUNC)(DAT_EP_HANDLE, struct dat_ep_param *); tduffy> tduffy> -typedef u32 (*DAT_EP_MODIFY_FUNC) ( tduffy> - DAT_EP_HANDLE, DAT_EP_PARAM_MASK, const struct dat_ep_param *); tduffy> +typedef u32 (*DAT_EP_MODIFY_FUNC)(DAT_EP_HANDLE, DAT_EP_PARAM_MASK, tduffy> + const struct dat_ep_param *); tduffy> tduffy> -typedef u32 (*DAT_EP_CONNECT_FUNC) (DAT_EP_HANDLE, DAT_IA_ADDRESS_PTR, tduffy> - DAT_CONN_QUAL, DAT_TIMEOUT, int, tduffy> - const void *, DAT_QOS, DAT_CONNECT_FLAGS); tduffy> +typedef u32 (*DAT_EP_CONNECT_FUNC)(DAT_EP_HANDLE, DAT_IA_ADDRESS_PTR, tduffy> + DAT_CONN_QUAL, DAT_TIMEOUT, int, tduffy> + const void *, DAT_QOS, DAT_CONNECT_FLAGS); tduffy> tduffy> -typedef u32 (*DAT_EP_DUP_CONNECT_FUNC) (DAT_EP_HANDLE, DAT_EP_HANDLE, tduffy> - DAT_TIMEOUT, int, const void *, tduffy> - DAT_QOS); tduffy> +typedef u32 (*DAT_EP_DUP_CONNECT_FUNC)(DAT_EP_HANDLE, DAT_EP_HANDLE, tduffy> + DAT_TIMEOUT, int, const void *, DAT_QOS); tduffy> tduffy> -typedef u32 (*DAT_EP_DISCONNECT_FUNC) (DAT_EP_HANDLE, DAT_CLOSE_FLAGS); tduffy> +typedef u32 (*DAT_EP_DISCONNECT_FUNC)(DAT_EP_HANDLE, DAT_CLOSE_FLAGS); tduffy> tduffy> -typedef u32 (*DAT_EP_POST_SEND_FUNC) ( tduffy> - DAT_EP_HANDLE, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, tduffy> - DAT_COMPLETION_FLAGS); tduffy> +typedef u32 (*DAT_EP_POST_SEND_FUNC)(DAT_EP_HANDLE, int, tduffy> + struct dat_lmr_triplet *, DAT_DTO_COOKIE, tduffy> + DAT_COMPLETION_FLAGS); tduffy> tduffy> -typedef u32 (*DAT_EP_POST_RECV_FUNC) ( tduffy> - DAT_EP_HANDLE, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, tduffy> - DAT_COMPLETION_FLAGS); tduffy> +typedef u32 (*DAT_EP_POST_RECV_FUNC)(DAT_EP_HANDLE, int, tduffy> + struct dat_lmr_triplet *, DAT_DTO_COOKIE, tduffy> + DAT_COMPLETION_FLAGS); tduffy> tduffy> -typedef u32 (*DAT_EP_POST_RDMA_READ_FUNC) ( tduffy> - DAT_EP_HANDLE, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, tduffy> - const struct dat_rmr_triplet *, DAT_COMPLETION_FLAGS); tduffy> +typedef u32 (*DAT_EP_POST_RDMA_READ_FUNC)(DAT_EP_HANDLE, int, tduffy> + struct dat_lmr_triplet *, tduffy> + DAT_DTO_COOKIE, tduffy> + const struct dat_rmr_triplet *, tduffy> + DAT_COMPLETION_FLAGS); tduffy> tduffy> -typedef u32 (*DAT_EP_POST_RDMA_WRITE_FUNC) ( tduffy> - DAT_EP_HANDLE, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, tduffy> - const struct dat_rmr_triplet *, DAT_COMPLETION_FLAGS); tduffy> +typedef u32 (*DAT_EP_POST_RDMA_WRITE_FUNC)(DAT_EP_HANDLE, int, tduffy> + struct dat_lmr_triplet *, tduffy> + DAT_DTO_COOKIE, tduffy> + const struct dat_rmr_triplet *, tduffy> + DAT_COMPLETION_FLAGS); tduffy> tduffy> -typedef u32 (*DAT_EP_GET_STATUS_FUNC) ( tduffy> - DAT_EP_HANDLE, DAT_EP_STATE *, boolean_t *, boolean_t *); tduffy> +typedef u32 (*DAT_EP_GET_STATUS_FUNC)(DAT_EP_HANDLE, DAT_EP_STATE *, tduffy> + boolean_t *, boolean_t *); tduffy> tduffy> -typedef u32 (*DAT_EP_FREE_FUNC) (DAT_EP_HANDLE); tduffy> +typedef u32 (*DAT_EP_FREE_FUNC)(DAT_EP_HANDLE); tduffy> tduffy> -typedef u32 (*DAT_EP_RESET_FUNC) (DAT_EP_HANDLE); tduffy> +typedef u32 (*DAT_EP_RESET_FUNC)(DAT_EP_HANDLE); tduffy> tduffy> -typedef u32 (*DAT_EP_RECV_QUERY_FUNC) ( tduffy> - DAT_EP_HANDLE, int *, int *); tduffy> +typedef u32 (*DAT_EP_RECV_QUERY_FUNC)(DAT_EP_HANDLE, int *, int *); tduffy> tduffy> -typedef u32 (*DAT_EP_SET_WATERMARK_FUNC) ( tduffy> - DAT_EP_HANDLE, int, int); tduffy> +typedef u32 (*DAT_EP_SET_WATERMARK_FUNC)(DAT_EP_HANDLE, int, int); tduffy> tduffy> -typedef u32 (*DAT_LMR_FREE_FUNC) (DAT_LMR_HANDLE); tduffy> +typedef u32 (*DAT_LMR_FREE_FUNC)(DAT_LMR_HANDLE); tduffy> tduffy> -typedef u32 (*DAT_LMR_QUERY_FUNC) (DAT_LMR_HANDLE, tduffy> - struct dat_lmr_param *); tduffy> +typedef u32 (*DAT_LMR_QUERY_FUNC)(DAT_LMR_HANDLE, struct dat_lmr_param *); tduffy> tduffy> -typedef u32 (*DAT_LMR_SYNC_RDMA_READ_FUNC) ( tduffy> - DAT_IA_HANDLE, const struct dat_lmr_triplet *, DAT_VLEN); tduffy> +typedef u32 (*DAT_LMR_SYNC_RDMA_READ_FUNC)(DAT_IA_HANDLE, tduffy> + const struct dat_lmr_triplet *, tduffy> + DAT_VLEN); tduffy> tduffy> -typedef u32 (*DAT_LMR_SYNC_RDMA_WRITE_FUNC) ( tduffy> - DAT_IA_HANDLE, const struct dat_lmr_triplet *, DAT_VLEN); tduffy> +typedef u32 (*DAT_LMR_SYNC_RDMA_WRITE_FUNC)(DAT_IA_HANDLE, tduffy> + const struct dat_lmr_triplet *, tduffy> + DAT_VLEN); tduffy> tduffy> -typedef u32 (*DAT_RMR_CREATE_FUNC) (DAT_PZ_HANDLE, DAT_RMR_HANDLE *); tduffy> +typedef u32 (*DAT_RMR_CREATE_FUNC)(DAT_PZ_HANDLE, DAT_RMR_HANDLE *); tduffy> tduffy> -typedef u32 (*DAT_RMR_QUERY_FUNC) (DAT_RMR_HANDLE, struct dat_rmr_param *); tduffy> +typedef u32 (*DAT_RMR_QUERY_FUNC)(DAT_RMR_HANDLE, struct dat_rmr_param *); tduffy> tduffy> -typedef u32 (*DAT_RMR_BIND_FUNC) ( tduffy> - DAT_RMR_HANDLE, const struct dat_lmr_triplet *, DAT_MEM_PRIV_FLAGS, tduffy> - DAT_EP_HANDLE, DAT_RMR_COOKIE, DAT_COMPLETION_FLAGS, DAT_RMR_CONTEXT *); tduffy> +typedef u32 (*DAT_RMR_BIND_FUNC)(DAT_RMR_HANDLE, const struct dat_lmr_triplet *, tduffy> + DAT_MEM_PRIV_FLAGS, DAT_EP_HANDLE, tduffy> + DAT_RMR_COOKIE, DAT_COMPLETION_FLAGS, tduffy> + DAT_RMR_CONTEXT *); tduffy> tduffy> -typedef u32 (*DAT_RMR_FREE_FUNC) (DAT_RMR_HANDLE); tduffy> +typedef u32 (*DAT_RMR_FREE_FUNC)(DAT_RMR_HANDLE); tduffy> tduffy> -typedef u32 (*DAT_PSP_CREATE_FUNC) ( tduffy> - DAT_IA_HANDLE, DAT_CONN_QUAL, DAT_EVD_HANDLE, DAT_PSP_FLAGS, tduffy> - DAT_PSP_HANDLE *); tduffy> +typedef u32 (*DAT_PSP_CREATE_FUNC)(DAT_IA_HANDLE, DAT_CONN_QUAL, DAT_EVD_HANDLE, tduffy> + DAT_PSP_FLAGS, DAT_PSP_HANDLE *); tduffy> tduffy> -typedef u32 (*DAT_PSP_CREATE_ANY_FUNC) ( tduffy> - DAT_IA_HANDLE, DAT_CONN_QUAL *, DAT_EVD_HANDLE, DAT_PSP_FLAGS, tduffy> - DAT_PSP_HANDLE *); tduffy> +typedef u32 (*DAT_PSP_CREATE_ANY_FUNC)(DAT_IA_HANDLE, DAT_CONN_QUAL *, tduffy> + DAT_EVD_HANDLE, DAT_PSP_FLAGS, tduffy> + DAT_PSP_HANDLE *); tduffy> tduffy> -typedef u32 (*DAT_PSP_QUERY_FUNC) (DAT_PSP_HANDLE, tduffy> - struct dat_psp_param *); tduffy> +typedef u32 (*DAT_PSP_QUERY_FUNC)(DAT_PSP_HANDLE, struct dat_psp_param *); tduffy> tduffy> -typedef u32 (*DAT_PSP_FREE_FUNC) (DAT_PSP_HANDLE); tduffy> +typedef u32 (*DAT_PSP_FREE_FUNC)(DAT_PSP_HANDLE); tduffy> tduffy> -typedef u32 (*DAT_RSP_CREATE_FUNC) ( tduffy> - DAT_IA_HANDLE, DAT_CONN_QUAL, DAT_EP_HANDLE, DAT_EVD_HANDLE, tduffy> - DAT_RSP_HANDLE *); tduffy> +typedef u32 (*DAT_RSP_CREATE_FUNC)(DAT_IA_HANDLE, DAT_CONN_QUAL, DAT_EP_HANDLE, tduffy> + DAT_EVD_HANDLE, DAT_RSP_HANDLE *); tduffy> tduffy> -typedef u32 (*DAT_RSP_QUERY_FUNC) (DAT_RSP_HANDLE, tduffy> - struct dat_rsp_param *); tduffy> +typedef u32 (*DAT_RSP_QUERY_FUNC)(DAT_RSP_HANDLE, struct dat_rsp_param *); tduffy> tduffy> -typedef u32 (*DAT_RSP_FREE_FUNC) (DAT_RSP_HANDLE); tduffy> +typedef u32 (*DAT_RSP_FREE_FUNC)(DAT_RSP_HANDLE); tduffy> tduffy> -typedef u32 (*DAT_PZ_CREATE_FUNC) (DAT_IA_HANDLE, DAT_PZ_HANDLE *); tduffy> +typedef u32 (*DAT_PZ_CREATE_FUNC)(DAT_IA_HANDLE, DAT_PZ_HANDLE *); tduffy> tduffy> -typedef u32 (*DAT_PZ_QUERY_FUNC) (DAT_PZ_HANDLE, struct dat_pz_param *); tduffy> +typedef u32 (*DAT_PZ_QUERY_FUNC)(DAT_PZ_HANDLE, struct dat_pz_param *); tduffy> tduffy> -typedef u32 (*DAT_PZ_FREE_FUNC) (DAT_PZ_HANDLE); tduffy> +typedef u32 (*DAT_PZ_FREE_FUNC)(DAT_PZ_HANDLE); tduffy> tduffy> -typedef u32 (*DAT_SRQ_CREATE_FUNC) ( tduffy> - DAT_IA_HANDLE, DAT_PZ_HANDLE, struct dat_srq_attr *, DAT_SRQ_HANDLE *); tduffy> +typedef u32 (*DAT_SRQ_CREATE_FUNC)(DAT_IA_HANDLE, DAT_PZ_HANDLE, tduffy> + struct dat_srq_attr *, DAT_SRQ_HANDLE *); tduffy> tduffy> -typedef u32 (*DAT_SRQ_SET_LW_FUNC) (DAT_SRQ_HANDLE, int); tduffy> +typedef u32 (*DAT_SRQ_SET_LW_FUNC)(DAT_SRQ_HANDLE, int); tduffy> tduffy> -typedef u32 (*DAT_SRQ_FREE_FUNC) (DAT_SRQ_HANDLE); tduffy> +typedef u32 (*DAT_SRQ_FREE_FUNC)(DAT_SRQ_HANDLE); tduffy> tduffy> -typedef u32 (*DAT_SRQ_QUERY_FUNC) (DAT_SRQ_HANDLE, tduffy> - struct dat_srq_param *); tduffy> +typedef u32 (*DAT_SRQ_QUERY_FUNC)(DAT_SRQ_HANDLE, struct dat_srq_param *); tduffy> tduffy> -typedef u32 (*DAT_SRQ_RESIZE_FUNC) (DAT_SRQ_HANDLE, int); tduffy> +typedef u32 (*DAT_SRQ_RESIZE_FUNC)(DAT_SRQ_HANDLE, int); tduffy> tduffy> -typedef u32 (*DAT_SRQ_POST_RECV_FUNC) ( tduffy> - DAT_SRQ_HANDLE, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE); tduffy> +typedef u32 (*DAT_SRQ_POST_RECV_FUNC)(DAT_SRQ_HANDLE, int, tduffy> + struct dat_lmr_triplet *, DAT_DTO_COOKIE); tduffy> tduffy> -typedef u32 (*DAT_LMR_KCREATE_FUNC) ( tduffy> - DAT_IA_HANDLE, DAT_MEM_TYPE, DAT_REGION_DESCRIPTION, DAT_VLEN, tduffy> - DAT_PZ_HANDLE, DAT_MEM_PRIV_FLAGS, DAT_MEM_OPTIMIZE_FLAGS, tduffy> - DAT_LMR_HANDLE *, DAT_LMR_CONTEXT *, DAT_RMR_CONTEXT *, DAT_VLEN *, tduffy> - DAT_VADDR *); tduffy> +typedef u32 (*DAT_LMR_KCREATE_FUNC)(DAT_IA_HANDLE, DAT_MEM_TYPE, tduffy> + DAT_REGION_DESCRIPTION, DAT_VLEN, tduffy> + DAT_PZ_HANDLE, DAT_MEM_PRIV_FLAGS, tduffy> + DAT_MEM_OPTIMIZE_FLAGS, DAT_LMR_HANDLE *, tduffy> + DAT_LMR_CONTEXT *, DAT_RMR_CONTEXT *, tduffy> + DAT_VLEN *, DAT_VADDR *); tduffy> tduffy> -typedef u32 (*DAT_IA_MEMTYPE_HINT_FUNC) ( tduffy> - DAT_IA_HANDLE, DAT_MEM_TYPE, DAT_VLEN, DAT_MEM_OPTIMIZE_FLAGS, tduffy> - DAT_VLEN *, DAT_VADDR *); tduffy> +typedef u32 (*DAT_IA_MEMTYPE_HINT_FUNC)(DAT_IA_HANDLE, DAT_MEM_TYPE, DAT_VLEN, tduffy> + DAT_MEM_OPTIMIZE_FLAGS, DAT_VLEN *, tduffy> + DAT_VADDR *); tduffy> tduffy> -typedef u32 (*DAT_EVD_KCREATE_FUNC) ( tduffy> - DAT_IA_HANDLE, int, DAT_UPCALL_POLICY, tduffy> - const struct dat_upcall_object *, tduffy> - DAT_EVD_FLAGS, DAT_EVD_HANDLE *); tduffy> +typedef u32 (*DAT_EVD_KCREATE_FUNC) (DAT_IA_HANDLE, int, DAT_UPCALL_POLICY, tduffy> + const struct dat_upcall_object *, tduffy> + DAT_EVD_FLAGS, DAT_EVD_HANDLE *); tduffy> tduffy> -typedef u32 (*DAT_EVD_MODIFY_UPCALL_FUNC) ( tduffy> - DAT_EVD_HANDLE, DAT_UPCALL_POLICY, const struct dat_upcall_object *); tduffy> +typedef u32 (*DAT_EVD_MODIFY_UPCALL_FUNC)(DAT_EVD_HANDLE, DAT_UPCALL_POLICY, tduffy> + const struct dat_upcall_object *); tduffy> tduffy> struct dat_provider { tduffy> const char *device_name; tduffy> @@ -1377,7 +1368,7 @@ extern u32 dat_registry_remove_provider( tduffy> /* tduffy> * DAT registry functions for consumers tduffy> */ tduffy> -extern u32 dat_ia_openv(const DAT_NAME_PTR name, int async_event_qlen, tduffy> +extern u32 dat_ia_openv(const char *name, int async_event_qlen, tduffy> DAT_EVD_HANDLE *async_event_handle, tduffy> DAT_IA_HANDLE *ia_handle, u32 dat_major, u32 dat_minor, tduffy> boolean_t thread_safety); tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl.h linux-kernel2/dat-provider/dapl.h tduffy> --- linux-kernel/dat-provider/dapl.h 2005-05-12 10:06:07.536001000 -0700 tduffy> +++ linux-kernel2/dat-provider/dapl.h 2005-05-12 11:02:59.834000000 -0700 tduffy> @@ -474,7 +474,7 @@ typedef char DAPL_HCA_NAME[DAPL_HCA_NAME tduffy> * DAT API functions tduffy> */ tduffy> tduffy> -extern u32 dapl_ia_open(const DAT_NAME_PTR name, int async_evd_qlen, tduffy> +extern u32 dapl_ia_open(const char *name, int async_evd_qlen, tduffy> DAT_EVD_HANDLE *async_evd_handle_ptr, tduffy> DAT_IA_HANDLE *ia_handle_ptr); tduffy> tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ia_open.c linux-kernel2/dat-provider/dapl_ia_open.c tduffy> --- linux-kernel/dat-provider/dapl_ia_open.c 2005-05-12 09:34:59.157003000 -0700 tduffy> +++ linux-kernel2/dat-provider/dapl_ia_open.c 2005-05-12 11:02:59.837003000 -0700 tduffy> @@ -51,7 +51,7 @@ tduffy> * That function maps the DAT_NAME parameter of dat_ia_open to a DAT_PROVIDER, tduffy> * and calls this function. tduffy> */ tduffy> -u32 dapl_ia_open(const DAT_NAME_PTR name, int async_evd_qlen, tduffy> +u32 dapl_ia_open(const char *name, int async_evd_qlen, tduffy> DAT_EVD_HANDLE *async_evd_handle_ptr, tduffy> DAT_IA_HANDLE *ia_handle_ptr) tduffy> { tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/cmd/dapl_netaddr.c linux-kernel2/test/dapltest/cmd/dapl_netaddr.c tduffy> --- linux-kernel/test/dapltest/cmd/dapl_netaddr.c 2005-05-11 08:53:17.661015000 -0700 tduffy> +++ linux-kernel2/test/dapltest/cmd/dapl_netaddr.c 2005-05-12 11:02:59.874000000 -0700 tduffy> @@ -50,7 +50,7 @@ DT_NetAddrFree (Per_Test_Data_t * pt_ptr tduffy> tduffy> boolean_t tduffy> DT_NetAddrLookupHostAddress (DAT_IA_ADDRESS_PTR to_netaddr, tduffy> - DAT_NAME_PTR hostname) tduffy> + char * hostname) tduffy> { tduffy> struct addrinfo *target; tduffy> int rval; tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/include/dapl_proto.h linux-kernel2/test/dapltest/include/dapl_proto.h tduffy> --- linux-kernel/test/dapltest/include/dapl_proto.h 2005-05-12 09:34:59.322003000 -0700 tduffy> +++ linux-kernel2/test/dapltest/include/dapl_proto.h 2005-05-12 11:02:59.881011000 -0700 tduffy> @@ -604,7 +604,7 @@ void DT_assert_fail (DT_Tdep_Print_H tduffy> char *file, tduffy> char *baseFile, tduffy> int line); tduffy> -int DT_ia_open (DAT_NAME_PTR dev_name, DAT_IA_HANDLE *ia_handle); tduffy> +int DT_ia_open (char *dev_name, DAT_IA_HANDLE *ia_handle); tduffy> int DT_ep_create (Params_t *params_ptr, tduffy> DAT_IA_HANDLE ia_handle, tduffy> DAT_PZ_HANDLE pz_handle, tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_fft_endpoint.c linux-kernel2/test/dapltest/test/dapl_fft_endpoint.c tduffy> --- linux-kernel/test/dapltest/test/dapl_fft_endpoint.c 2005-05-12 09:28:43.503023000 -0700 tduffy> +++ linux-kernel2/test/dapltest/test/dapl_fft_endpoint.c 2005-05-12 11:03:25.510004000 -0700 tduffy> @@ -59,7 +59,7 @@ int DT_endpoint_generic (Params_t *param tduffy> evd_handle = DAT_HANDLE_NULL; tduffy> phead = params_ptr->phead; tduffy> tduffy> - rc = dat_ia_open ((const DAT_NAME_PTR)dev_name, tduffy> + rc = dat_ia_open ((const char *)dev_name, tduffy> DEFAULT_QUEUE_LEN, &evd_handle, &ia_handle); tduffy> DT_assert_dat (phead, rc == DAT_SUCCESS); tduffy> tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_fft_hwconn.c linux-kernel2/test/dapltest/test/dapl_fft_hwconn.c tduffy> --- linux-kernel/test/dapltest/test/dapl_fft_hwconn.c 2005-05-12 09:28:43.633078000 -0700 tduffy> +++ linux-kernel2/test/dapltest/test/dapl_fft_hwconn.c 2005-05-12 11:03:25.515001000 -0700 tduffy> @@ -45,7 +45,7 @@ int DT_hwconn_case0 ( Params_t *params_p tduffy> nic_handle = NULL; tduffy> evd_handle = DAT_HANDLE_NULL; tduffy> tduffy> - rc=dat_ia_open ((const DAT_NAME_PTR)dev_name, 10, &evd_handle, &nic_handle); tduffy> + rc=dat_ia_open ((const char *)dev_name, 10, &evd_handle, &nic_handle); tduffy> DT_assert_dat (phead, rc == DAT_SUCCESS); tduffy> tduffy> rc=dat_ia_close (nic_handle, DAT_CLOSE_ABRUPT_FLAG); tduffy> @@ -95,7 +95,7 @@ int DT_hwconn_case1 ( Params_t *params_p tduffy> } tduffy> tduffy> evd_handle = DAT_HANDLE_NULL; tduffy> - rc=dat_ia_open ((const DAT_NAME_PTR)dev_name, 10, &evd_handle, &nic_handle); tduffy> + rc=dat_ia_open ((const char *)dev_name, 10, &evd_handle, &nic_handle); tduffy> if (DAT_GET_TYPE (rc) != DAT_PROVIDER_NOT_FOUND) tduffy> { tduffy> const char *major_msg, *minor_msg; tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c linux-kernel2/test/dapltest/test/dapl_fft_queryinfo.c tduffy> --- linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c 2005-05-12 09:28:43.523022000 -0700 tduffy> +++ linux-kernel2/test/dapltest/test/dapl_fft_queryinfo.c 2005-05-12 11:03:25.523003000 -0700 tduffy> @@ -94,7 +94,7 @@ int DT_queryinfo_basic (Params_t *params tduffy> dev_name = cmd->device_name; tduffy> tduffy> /* All functions require an ia_handle to be created */ tduffy> - rc = dat_ia_open ((const DAT_NAME_PTR)dev_name, tduffy> + rc = dat_ia_open ((const char *)dev_name, tduffy> DEFAULT_QUEUE_LEN, tduffy> &evd_handle, tduffy> &ia_handle); tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_fft_util.c linux-kernel2/test/dapltest/test/dapl_fft_util.c tduffy> --- linux-kernel/test/dapltest/test/dapl_fft_util.c 2005-05-12 09:28:43.375003000 -0700 tduffy> +++ linux-kernel2/test/dapltest/test/dapl_fft_util.c 2005-05-12 11:02:59.887005000 -0700 tduffy> @@ -55,7 +55,7 @@ void DT_assert_fail (DT_Tdep_Print_Head tduffy> } tduffy> tduffy> /* helper function to open an IA */ tduffy> -int DT_ia_open (DAT_NAME_PTR dev_name, DAT_IA_HANDLE *ia_handle) tduffy> +int DT_ia_open (char *dev_name, DAT_IA_HANDLE *ia_handle) tduffy> { tduffy> DAT_EVD_HANDLE evd_handle; tduffy> evd_handle = DAT_HANDLE_NULL; tduffy> From roland at topspin.com Fri May 13 10:23:44 2005 From: roland at topspin.com (Roland Dreier) Date: Fri, 13 May 2005 10:23:44 -0700 Subject: [openib-general] Re: use of virt_to_phys in sdp In-Reply-To: <20050513094726.A24123@topspin.com> (Libor Michalek's message of "Fri, 13 May 2005 09:47:26 -0700") References: <20050513140600.GC320@mellanox.co.il> <20050513094726.A24123@topspin.com> Message-ID: <52y8ajrqe7.fsf@topspin.com> Libor> It is incorrect. The regular send/recv path uses the DMA Libor> API correctly, but RDMA to internal buffers on both the Libor> send and receive path is still using plain old Libor> virt_to_phys. Also, does FMR map/unmap use the DMA API, or Libor> does that need to be taken care of as well? The FMR verbs functions just take the addresses passed in by the consumer and write them directly into the HCA's page table. So the consumer needs to use the DMA API to get bus addresses. - R. From blist at aon.at Fri May 13 10:58:01 2005 From: blist at aon.at (Bernhard Fischer) Date: Fri, 13 May 2005 19:58:01 +0200 Subject: [openib-general] Re: user-mode verbs on Itanium In-Reply-To: <20050512155550.GA14123@esmail.cup.hp.com> References: <20050504005534.GA13574@esmail.cup.hp.com> <20050504070910.GC9674@mellanox.co.il> <20050506190947.GA23958@esmail.cup.hp.com> <20050507053706.GN23958@esmail.cup.hp.com> <20050508142345.GQ9941@mellanox.co.il> <20050510152428.GA5495@esmail.cup.hp.com> <20050510154030.GH2474@mellanox.co.il> <20050511012148.GE5495@esmail.cup.hp.com> <20050511072138.GB13349@mellanox.co.il> <20050512155550.GA14123@esmail.cup.hp.com> Message-ID: <20050513175801.GS15688@aon.at> On Thu, May 12, 2005 at 08:55:50AM -0700, Grant Grundler wrote: >On Wed, May 11, 2005 at 10:21:38AM +0300, Michael S. Tsirkin wrote: >> You mean just sample the clock before the first post? > >Yes. > >> That would be easy, but lets look at what does it measure: >> >> On the client, you just get the time it takes to perform the >> first post_send operation. If this is interesting to you, I agree, >> but lets take the timestamps around each post send operation to >> make it statistically relevant. >> Let me know - and need to think of a way to do this without >> affecting latency numbers. > >Yes, I'd like to try that. >But a few more of my pending patches need to either be rejected >or (preferably) accepted. Grant, please add the perftest.05 patch to the batch since without it the test is broken for i386. TIA, Bernhard From tduffy at sun.com Fri May 13 11:04:36 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 13 May 2005 11:04:36 -0700 Subject: [openib-general] Re: [openib-commits] r2337 - gen2/users/jlentini/linux-kernel In-Reply-To: <20050513175112.78CED2283D8@openib.ca.sandia.gov> References: <20050513175112.78CED2283D8@openib.ca.sandia.gov> Message-ID: <1116007476.10240.14.camel@duffman> On Fri, 2005-05-13 at 10:51 -0700, jlentini at openib.org wrote: > Modified: gen2/users/jlentini/linux-kernel/TODO > =================================================================== > --- gen2/users/jlentini/linux-kernel/TODO 2005-05-13 17:31:24 UTC (rev 2336) > +++ gen2/users/jlentini/linux-kernel/TODO 2005-05-13 17:51:11 UTC (rev 2337) > @@ -1,5 +1,5 @@ > *) Test static compilation of DAT and IB DAT PROVIDER > -*) consider using spin_lock_irqsave() instead of spin_lock() > +*) consider using spin_lock_irqsave() instead of spin_lock_irqsave() huh? -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From tduffy at sun.com Fri May 13 11:08:44 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 13 May 2005 11:08:44 -0700 Subject: [openib-general] Re: [openib-commits] r2337 - gen2/users/jlentini/linux-kernel In-Reply-To: <1116007476.10240.14.camel@duffman> References: <20050513175112.78CED2283D8@openib.ca.sandia.gov> <1116007476.10240.14.camel@duffman> Message-ID: <1116007724.10240.15.camel@duffman> On Fri, 2005-05-13 at 11:04 -0700, Tom Duffy wrote: > On Fri, 2005-05-13 at 10:51 -0700, jlentini at openib.org wrote: > > Modified: gen2/users/jlentini/linux-kernel/TODO > > =================================================================== > > --- gen2/users/jlentini/linux-kernel/TODO 2005-05-13 17:31:24 UTC (rev 2336) > > +++ gen2/users/jlentini/linux-kernel/TODO 2005-05-13 17:51:11 UTC (rev 2337) > > @@ -1,5 +1,5 @@ > > *) Test static compilation of DAT and IB DAT PROVIDER > > -*) consider using spin_lock_irqsave() instead of spin_lock() > > +*) consider using spin_lock_irqsave() instead of spin_lock_irqsave() > > huh? nevermind. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From jlentini at netapp.com Fri May 13 11:10:42 2005 From: jlentini at netapp.com (James Lentini) Date: Fri, 13 May 2005 14:10:42 -0400 (EDT) Subject: [openib-general] Re: [openib-commits] r2337 - gen2/users/jlentini/linux-kernel In-Reply-To: <1116007476.10240.14.camel@duffman> References: <20050513175112.78CED2283D8@openib.ca.sandia.gov> <1116007476.10240.14.camel@duffman> Message-ID: On Fri, 13 May 2005, Tom Duffy wrote: tduffy> On Fri, 2005-05-13 at 10:51 -0700, jlentini at openib.org wrote: tduffy> > Modified: gen2/users/jlentini/linux-kernel/TODO tduffy> > =================================================================== tduffy> > --- gen2/users/jlentini/linux-kernel/TODO 2005-05-13 17:31:24 UTC (rev 2336) tduffy> > +++ gen2/users/jlentini/linux-kernel/TODO 2005-05-13 17:51:11 UTC (rev 2337) tduffy> > @@ -1,5 +1,5 @@ tduffy> > *) Test static compilation of DAT and IB DAT PROVIDER tduffy> > -*) consider using spin_lock_irqsave() instead of spin_lock() tduffy> > +*) consider using spin_lock_irqsave() instead of spin_lock_irqsave() tduffy> tduffy> huh? I know :) I don't know what happened there. That change was not intentional. Hal pointed it out and I've fixed it. I had a hard time checking in this update. Subversion kept spitting out messages that my text-base file didn't have the correct checksum value. Has anyone ever seen that? james From info at tan-tan-tanuki.net Fri May 13 03:41:50 2005 From: info at tan-tan-tanuki.net (info at tan-tan-tanuki.net) Date: 13 May 2005 19:41:50 +0900 Subject: [openib-general] $B?75,EPO?$NJ}$O(B10000$B1_J,L5NA(B Message-ID: <20050513104150.28393.qmail@mail.tan-tan-tanuki.net> $B0lK|1_J,!*40A4L5NA$G;HMQ(BOK $B"#:#$J$i4V$K9g$&(BW$B%A%c%s%9!*L5NA%]%$%s%H%"%C%W3Z$7$5(B100$BG\!*"#(B $B"(EPO?=w at -2q0w$N%8%c%s%kJL?M5$%i%s%-%s%0(B $B0l0L!'5U1g!J(B189$BL>EPO?!K(B $BFs0L!'1g8r!J(B429$BL>EPO?!K(B $B;00L!'(BSM$B5U1g!J(B243$BL>EPO?!K(B $B;M0L!'%;%U%l!J(B1038$BL>EPO?!K(B $B8^0L!'ITNQ!J(B2421$BL>EPO?!K(B $BO;0L!'%F%l%(%C%A!J(B3463$BL>EPO?!K(B $B!&(B $B!&(B $B!&(B $B0lK|1_J,40A4L5NA$G$*;n$7$O(B $B"-"-"-(B http://awg.webchu.com/?summer29 $BB~:#!*?75,EPO?$7$?J}$K$OL5NA$G(B10,000$B1_J,$4MxMQ=PMh$^$9!#(B $B!ZL5NA%]%$%s%HFb$G==J,$K%Q!<%H%J!<$rC5$9;v$,=PMh$^$9!#![(B $B"((B18$B:P0J>e$NHkL)87l9g$O2<5-(BURL$B$KJV?.2<$5$$!#(B henshin_senyou at ok.kz From jlentini at netapp.com Fri May 13 11:35:59 2005 From: jlentini at netapp.com (James Lentini) Date: Fri, 13 May 2005 14:35:59 -0400 (EDT) Subject: [openib-general] Re: [PATCHv2] kDAPL: remove DAT_RETURN In-Reply-To: <469958e00505121256cd8c04d@mail.gmail.com> References: <1115910070.25530.2.camel@duffman> <1115915608.9070.0.camel@duffman> <469958e00505121256cd8c04d@mail.gmail.com> Message-ID: caitlin> DAT_RETURN serves two distinct purposes: it is an immediate return caitlin> to a kernel specific caller, and it is a standardized error encoding for caitlin> clients outside of the linux kernel (loadable modules and or proxied caitlin> user consumers). caitlin> caitlin> Even if the functional return to in-kernel clients is converted to comply caitlin> with kernel coding conventions, the information currently encoded caitlin> in the DAT_RETURN needs to be deliverable to DAPL clients that caitlin> are outside of the linux kernel. DAPL clients outside the linux kernel, userspace clients, should use the uDAPL API. No changes are being proposed for the uDAPL API. caitlin> Therefore the current enums need to be retained, and there should caitlin> be some mechanism for these precise diagnostics to be relayed caitlin> to non-kernel clients. An external shim cannot do this *after* caitlin> the information is lost by using a standard error return. (I am caitlin> assuming that creating a list of error returns that encompass caitlin> everything that DAT_RETURN *Can* encode is out of the caitlin> question, there's a reason it's multi-layer now). Many of the DAT_RETURN values are useful for debugging (e.g. the ability to say that parameter number 4 is invalid). However, I can't imagine a DAT API consumer that would actually provide error recovery code for each of the 22 major * 102 minor = 2244 different possible error values, let alone warning values, for every API call. I think that some of this information can be logged or obtained during debugging. caitlin> One possibility would be to add an external_client_dat_return * caitlin> out parameter that would allow a shim to restore the full caitlin> DAT defined error code for non-kernel clients. Adding an additional out parameter to every API function would clutter the interface. caitlin> On 5/12/05, Tom Duffy wrote: caitlin> > On Thu, 2005-05-12 at 11:50 -0400, James Lentini wrote: caitlin> > > Thanks for the updated patch Tom. It is committed in revision 2316. caitlin> > > caitlin> > > Eventually we will need to remove our dat_return_* enums and replace caitlin> > > our return values with errno values. Correct? caitlin> > caitlin> > That is right. This is just a first pass at cleaning it up. caitlin> > caitlin> > -tduffy From libor at topspin.com Fri May 13 11:37:13 2005 From: libor at topspin.com (Libor Michalek) Date: Fri, 13 May 2005 11:37:13 -0700 Subject: [openib-general] Re: [PATCH] make private data pointers const In-Reply-To: ; from jlentini@netapp.com on Thu, May 12, 2005 at 10:00:47PM -0400 References: Message-ID: <20050513113713.D24123@topspin.com> On Thu, May 12, 2005 at 10:00:47PM -0400, James Lentini wrote: > > This patch affects ib_cm.h and cm.c, which appear to be maintained > by Sean, and ucm.c, which appears to be maintained by Libor. > > The patch adds the const qualifier to private data pointers passed > into the IB layer. Sean, if you're going to check in the CM portion, go ahead and check in ucm.c as well. Thanks. -Libor From libor at topspin.com Fri May 13 11:32:58 2005 From: libor at topspin.com (Libor Michalek) Date: Fri, 13 May 2005 11:32:58 -0700 Subject: [openib-general] Re: [PATCH] SDP & CM: change some functions to return void In-Reply-To: <4284D628.4010801@ichips.intel.com>; from mshefty@ichips.intel.com on Fri, May 13, 2005 at 09:30:32AM -0700 References: <1115998078.10240.8.camel@duffman> <4284D628.4010801@ichips.intel.com> Message-ID: <20050513113258.C24123@topspin.com> On Fri, May 13, 2005 at 09:30:32AM -0700, Sean Hefty wrote: > Tom Duffy wrote: > > This patch changes some functions to return void rather than an int > > since they were always returning 0, thus making checking return values > > rather pointless. The patch touches both SDP and CM. > > Thanks, Tom. Since the majority of these changes affect SDP, and the > check-ins should be coordinated, I'll ask Libor if he can commit the CM > changes when he commits the SDP changes. > > Libor, can you commit the CM changes when you commit the SDP changes? :) > There are also some (void)ib_destroy_cm_id casts in umc.c that could lose > the casts with this change as well. OK, I applied and commited the patch. Thanks Tom. -Libor From caitlin.bestler at gmail.com Fri May 13 11:52:34 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Fri, 13 May 2005 11:52:34 -0700 Subject: [openib-general] Re: [PATCHv2] kDAPL: remove DAT_RETURN In-Reply-To: References: <1115910070.25530.2.camel@duffman> <1115915608.9070.0.camel@duffman> <469958e00505121256cd8c04d@mail.gmail.com> Message-ID: <469958e00505131152573cd93a@mail.gmail.com> On 5/13/05, James Lentini wrote: > > caitlin> DAT_RETURN serves two distinct purposes: it is an immediate return > caitlin> to a kernel specific caller, and it is a standardized error encoding for > caitlin> clients outside of the linux kernel (loadable modules and or proxied > caitlin> user consumers). > caitlin> > caitlin> Even if the functional return to in-kernel clients is converted to comply > caitlin> with kernel coding conventions, the information currently encoded > caitlin> in the DAT_RETURN needs to be deliverable to DAPL clients that > caitlin> are outside of the linux kernel. > > DAPL clients outside the linux kernel, userspace clients, should use > the uDAPL API. No changes are being proposed for the uDAPL API. > What about kernel clients that are compiled separately from the kernel that want to be OS neutral and use kdat.h as published by the DAT Collaborative? Secondly, many routines are identical in user and kernel (in fact the common directory is bigger than either the udapl or kdapl directory on the sourceforge implementation) requiring that this code be maintained in parallel for the kernel and partner udapl librarires seems like a lot of extra long term work. From sean.hefty at intel.com Fri May 13 11:53:42 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Fri, 13 May 2005 11:53:42 -0700 Subject: [openib-general] Re: [PATCH] make private data pointers const In-Reply-To: <20050513113713.D24123@topspin.com> Message-ID: >> This patch affects ib_cm.h and cm.c, which appear to be maintained >> by Sean, and ucm.c, which appears to be maintained by Libor. >> >> The patch adds the const qualifier to private data pointers passed >> into the IB layer. > >Sean, if you're going to check in the CM portion, go ahead and check >in ucm.c as well. Can do. I'll do this within the next couple of work days. - Sean From tduffy at sun.com Fri May 13 11:58:29 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 13 May 2005 11:58:29 -0700 Subject: [openib-general] Re: [PATCHv2] kDAPL: remove DAT_RETURN In-Reply-To: <469958e00505131152573cd93a@mail.gmail.com> References: <1115910070.25530.2.camel@duffman> <1115915608.9070.0.camel@duffman> <469958e00505121256cd8c04d@mail.gmail.com> <469958e00505131152573cd93a@mail.gmail.com> Message-ID: <1116010709.20109.1.camel@duffman> On Fri, 2005-05-13 at 11:52 -0700, Caitlin Bestler wrote: > What about kernel clients that are compiled separately from the kernel > that want to be OS neutral and use kdat.h as published by the DAT > Collaborative? That never want to be put back into mainline Linux? Too bad. Not our problem. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From jlentini at netapp.com Fri May 13 12:02:52 2005 From: jlentini at netapp.com (James Lentini) Date: Fri, 13 May 2005 15:02:52 -0400 (EDT) Subject: [openib-general] Re: [PATCH] [kdapl]: Reduce CM retry count In-Reply-To: <1115988739.10509.25.camel@localhost.localdomain> References: <1115988739.10509.25.camel@localhost.localdomain> Message-ID: Committed in revision 2341. On Fri, 13 May 2005, Hal Rosenstock wrote: halr> Reduce CM retry count halr> halr> Signed-off-by: Hal Rosenstock halr> halr> Index: dapl_openib_cm.h halr> =================================================================== halr> -- dapl_openib_cm.h (revision 2331) halr> +++ dapl_openib_cm.h (working copy) halr> @@ -40,7 +40,7 @@ halr> #define DAPL_OPENIB_RC_RETRY_COUNT 7 halr> #define DAPL_OPENIB_RNR_RETRY_COUNT 6 halr> #define DAPL_OPENIB_CM_RESPONSE_TIMEOUT 20 /* 4 sec */ halr> -#define DAPL_OPENIB_MAX_CM_RETRIES 15 halr> +#define DAPL_OPENIB_MAX_CM_RETRIES 4 halr> halr> typedef enum { halr> IB_CME_CONNECTED, halr> From Tom.Duffy at Sun.COM Fri May 13 12:03:37 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Fri, 13 May 2005 12:03:37 -0700 Subject: [openib-general] Re: [openib-commits] r2337 - gen2/users/jlentini/linux-kernel In-Reply-To: References: <20050513175112.78CED2283D8@openib.ca.sandia.gov> <1116007476.10240.14.camel@duffman> Message-ID: <1116011017.20109.4.camel@duffman> On Fri, 2005-05-13 at 14:10 -0400, James Lentini wrote: > I had a hard time checking in this update. Subversion kept spitting > out messages that my text-base file didn't have the correct checksum > value. Has anyone ever seen that? I haven't. I might suggest blowing away your whole archive and pulling a fresh copy. Also, probably a good idea to make sure you are using the latest version of subversion. [tduffy at duffman dat-provider]$ svn --version svn, version 1.1.4 (r13838) compiled Apr 13 2005, 12:25:53 -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From iod00d at hp.com Fri May 13 12:08:58 2005 From: iod00d at hp.com (Grant Grundler) Date: Fri, 13 May 2005 12:08:58 -0700 Subject: [openib-general] Re: [PATCHv2] kDAPL: remove DAT_RETURN In-Reply-To: <469958e00505131152573cd93a@mail.gmail.com> References: <1115910070.25530.2.camel@duffman> <1115915608.9070.0.camel@duffman> <469958e00505121256cd8c04d@mail.gmail.com> <469958e00505131152573cd93a@mail.gmail.com> Message-ID: <20050513190858.GE19012@esmail.cup.hp.com> Caitlin, there has been sufficient discussion about user vs kernel space on this list in the past. I'll recap key points. But you'll want to use google (or something) to read some of the previous discussion on this topic. On Fri, May 13, 2005 at 11:52:34AM -0700, Caitlin Bestler wrote: > What about kernel clients that are compiled separately from the kernel > that want to be OS neutral and use kdat.h as published by the DAT > Collaborative? There is no such thing as OS neutral code when we are talking about kernel support. Some code ports more easily between OSs. Open source community is sensitive to this issue only because the support model is different. > Secondly, many routines are identical in user and kernel (in fact the > common directory is bigger than either the udapl or kdapl directory > on the sourceforge implementation) requiring that this code be > maintained in parallel for the kernel and partner udapl librarires > seems like a lot of extra long term work. kernel and user space do not have the same requirements. kernel is ultimately responsible for enforcing userspace policies (e.g. security, fairness, access rights, etc). Some of the code can be shared with userspace, but don't expect that shared code to be OS agnostic as well. Bottomline is some of the code will need to be maintained in parallel. grant From halr at voltaire.com Fri May 13 12:16:32 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 13 May 2005 15:16:32 -0400 Subject: [openib-general] MAD module removal and MAD cache Message-ID: <1116011792.4724.10.camel@localhost.localdomain> Hi Sean, The MAD send code can place a send request on the send list, the QP's send queue, or overflow list. On port close, I see the receive queue being cleaned up but not these. Am I missing something ? If I'm not, then this may be the MAD cache leak which causes a problem when running SM, killing it, and trying to remove the modules. -- Hal From libor at topspin.com Fri May 13 12:06:37 2005 From: libor at topspin.com (Libor Michalek) Date: Fri, 13 May 2005 12:06:37 -0700 Subject: [openib-general] Re: [PATCH] libibcm/examples/simple.c: Eliminate a couple of warnings In-Reply-To: <1115995316.10509.87.camel@localhost.localdomain>; from halr@voltaire.com on Fri, May 13, 2005 at 10:41:57AM -0400 References: <1115995316.10509.87.camel@localhost.localdomain> Message-ID: <20050513120637.F24123@topspin.com> On Fri, May 13, 2005 at 10:41:57AM -0400, Hal Rosenstock wrote: > libibcm/examples/simple.c: Eliminate a couple of warnings > > Signed-off-by: Hal Rosenstock Applied, but including netinet/in.h required me to rename connect() because of the type conflict in socket.h. Also I rmeoved the ifdef'd code. Thanks. -Libor From roland at topspin.com Fri May 13 12:13:45 2005 From: roland at topspin.com (Roland Dreier) Date: Fri, 13 May 2005 12:13:45 -0700 Subject: [openib-general] Re: [openib-commits] r2337 - gen2/users/jlentini/linux-kernel In-Reply-To: (James Lentini's message of "Fri, 13 May 2005 14:10:42 -0400 (EDT)") References: <20050513175112.78CED2283D8@openib.ca.sandia.gov> <1116007476.10240.14.camel@duffman> Message-ID: <52d5rvrlau.fsf@topspin.com> James> I had a hard time checking in this update. Subversion kept James> spitting out messages that my text-base file didn't have James> the correct checksum value. Has anyone ever seen that? Is it possible that you modified a file in a .svn directory somewhere? Based on the typo checked into the TODO, it seems like maybe you did a recursive search and replace of 'spin_lock' with 'spin_lock_irqsave' in your source tree, which could screw up subversion's private original copy of the TODO file. - R. From libor at topspin.com Fri May 13 12:04:03 2005 From: libor at topspin.com (Libor Michalek) Date: Fri, 13 May 2005 12:04:03 -0700 Subject: [openib-general] Re: [PATCH] libibcm building In-Reply-To: <1115994173.10509.79.camel@localhost.localdomain>; from halr@voltaire.com on Fri, May 13, 2005 at 10:22:53AM -0400 References: <1115994173.10509.79.camel@localhost.localdomain> Message-ID: <20050513120403.E24123@topspin.com> On Fri, May 13, 2005 at 10:22:53AM -0400, Hal Rosenstock wrote: > Hi Libor, > > A couple of minor libibcm building comments: > > 1. When running autogen.sh, it warns about: > Makefile.am:19: variable `ucm_version_script' not defined > > 2. example/simple is not built by the current makefile generated. Below > is a patch for this. > > Add building of simple example for user CM to generated makefile > > Signed-off-by: Hal Rosenstock Thanks, applied and checked in. -Libor From mshefty at ichips.intel.com Fri May 13 12:31:15 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Fri, 13 May 2005 12:31:15 -0700 Subject: [openib-general] Re: MAD module removal and MAD cache In-Reply-To: <1116011792.4724.10.camel@localhost.localdomain> References: <1116011792.4724.10.camel@localhost.localdomain> Message-ID: <42850083.5080706@ichips.intel.com> Hal Rosenstock wrote: > Hi Sean, > > The MAD send code can place a send request on the send list, the QP's > send queue, or overflow list. > > On port close, I see the receive queue being cleaned up but not these. > > Am I missing something ? If I'm not, then this may be the MAD cache leak > which causes a problem when running SM, killing it, and trying to remove > the modules. When a MAD is sent, a reference is taken on the mad_agent. Cleanup of these lists should occur automatically as a send operation completes. We need to be careful about trying to perform cleanup during mad_agent destruction, since the MAD may be posted to the QP. If it is, we need to wait until we see its completion. When a mad_agent is destroyed, all outstanding MADs on its send_list are canceled. This will complete all MADs waiting for a response, but those on the QP's send_queue or overflow_list still need to wait for a completion. If the mad_agent destruction is proceeding, then it sounds like these lists are empty. To verify, you could add a check that the mad_agent's send_list is empty before deregister returns. - Sean From jlentini at netapp.com Fri May 13 13:23:04 2005 From: jlentini at netapp.com (James Lentini) Date: Fri, 13 May 2005 16:23:04 -0400 (EDT) Subject: [openib-general] Re: [PATCH][TRIVIAL] [kdapl] Minor CM Changes In-Reply-To: <1115988748.10509.27.camel@localhost.localdomain> References: <1115988748.10509.27.camel@localhost.localdomain> Message-ID: Committed in revision 2346. On Fri, 13 May 2005, Hal Rosenstock wrote: halr> Minor CM changes halr> Change label in do_rep_recv to more appropriate name halr> halr> Signed-off-by: Hal Rosenstock halr> halr> Index: dapl_openib_cm.c halr> =================================================================== halr> -- dapl_openib_cm.c (revision 2331) halr> +++ dapl_openib_cm.c (working copy) halr> @@ -68,7 +68,7 @@ halr> if (conn->ep->qp_handle == IB_INVALID_HANDLE) { halr> dapl_dbg_log(DAPL_DBG_TYPE_ERR, " do_rep_recv: invalid qp " halr> "handle\n"); halr> - goto reject; halr> + goto disc; halr> } halr> halr> /* First, transition QP to RTR */ halr> @@ -76,7 +76,7 @@ halr> if (status) { halr> dapl_dbg_log(DAPL_DBG_TYPE_ERR, " do_rep_recv: could not " halr> "modify QP state to RTR status %d\n", status); halr> - goto reject; halr> + goto disc; halr> } halr> halr> /* Now, transition QP to RTS */ halr> @@ -84,23 +84,22 @@ halr> if (status) { halr> dapl_dbg_log(DAPL_DBG_TYPE_ERR, " do_rep_recv: could not " halr> "modify QP state to RTS status %d\n", status); halr> - goto reject; halr> + goto disc; halr> } halr> halr> status = ib_send_cm_rtu(conn->cm_id, NULL, 0); halr> if (status) { halr> dapl_dbg_log(DAPL_DBG_TYPE_ERR, " do_rep_recv: ib_send_cm_rtu " halr> "failed: %d\n", status); halr> - goto reject; halr> + goto disc; halr> } halr> halr> dapl_evd_connection_callback(conn, IB_CME_CONNECTED, halr> rep_recv_param->private_data, conn->ep); halr> return; halr> halr> -reject: halr> - ib_send_cm_rej(conn->cm_id, IB_CM_REJ_CONSUMER_DEFINED, NULL, 0, halr> - NULL, 0); halr> +disc: halr> + ib_send_cm_dreq(conn->cm_id, NULL, 0); /* ??? */ halr> dapl_evd_connection_callback(conn, IB_CME_LOCAL_FAILURE, NULL, halr> conn->ep); halr> } halr> @@ -456,7 +455,7 @@ halr> if (status < 0) { halr> dapl_dbg_log(DAPL_DBG_TYPE_ERR, " ib_at_route_by_ip failed " halr> "with status %d\n", status); halr> - dapl_destroy_cm_id(conn); halr> + dapl_destroy_cm_id(conn); /* ??? */ halr> return DAT_INTERNAL_ERROR; halr> } halr> halr> halr> From sean.hefty at intel.com Fri May 13 13:36:52 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Fri, 13 May 2005 13:36:52 -0700 Subject: [openib-general] [PATCH] [CM] reject stale connections Message-ID: The following patch will reject stale connection requests. It also fixes a couple of minor error handling cleanup issues found during testing, and fixes a bug handling a stale connection reject message where the cm_id would access an invalid timewait pointer. Signed-off-by: Sean Hefty Index: core/cm.c =================================================================== --- core/cm.c (revision 2346) +++ core/cm.c (working copy) @@ -185,10 +185,35 @@ static int cm_alloc_msg(struct cm_id_pri return 0; } +static int cm_alloc_response_msg(struct cm_port *port, + struct ib_mad_recv_wc *mad_recv_wc, + struct ib_mad_send_buf **msg) +{ + struct ib_mad_send_buf *m; + struct ib_ah *ah; + + ah = ib_create_ah_from_wc(port->mad_agent->qp->pd, mad_recv_wc->wc, + mad_recv_wc->recv_buf.grh, port->port_num); + if (IS_ERR(ah)) + return PTR_ERR(ah); + + m = ib_create_send_mad(port->mad_agent, 1, mad_recv_wc->wc->pkey_index, + ah, 0, sizeof(struct ib_mad_hdr), + sizeof(struct ib_mad)-sizeof(struct ib_mad_hdr), + GFP_ATOMIC); + if (IS_ERR(m)) { + ib_destroy_ah(ah); + return PTR_ERR(m); + } + *msg = m; + return 0; +} + static void cm_free_msg(struct ib_mad_send_buf *msg) { ib_destroy_ah(msg->send_wr.wr.ud.ah); - cm_deref_id(msg->context[0]); + if (msg->context[0]) + cm_deref_id(msg->context[0]); ib_free_send_mad(msg); } @@ -531,10 +556,7 @@ static void cm_cleanup_timewait(struct c spin_unlock_irqrestore(&cm.lock, flags); } -static struct cm_timewait_info * cm_create_timewait_info(u32 local_id, - u32 remote_id, - u64 remote_ca_guid, - u32 remote_qpn) +static struct cm_timewait_info * cm_create_timewait_info(u32 local_id) { struct cm_timewait_info *timewait_info; @@ -544,10 +566,6 @@ static struct cm_timewait_info * cm_crea memset(timewait_info, 0, sizeof *timewait_info); timewait_info->work.local_id = local_id; - timewait_info->work.remote_id = remote_id; - timewait_info->remote_ca_guid = remote_ca_guid; - timewait_info->remote_qpn = remote_qpn; - INIT_WORK(&timewait_info->work.work, cm_work_handler, &timewait_info->work); timewait_info->work.cm_event.event = IB_CM_TIMEWAIT_EXIT; @@ -674,30 +692,33 @@ int ib_cm_listen(struct ib_cm_id *cm_id, } EXPORT_SYMBOL(ib_cm_listen); -static void cm_format_mad_hdr(struct ib_mad_hdr *hdr, - struct cm_id_private *cm_id_priv, - enum cm_msg_attr_id attr_id, - enum cm_msg_sequence msg_seq) +static u64 cm_form_tid(struct cm_id_private *cm_id_priv, + enum cm_msg_sequence msg_seq) { u64 hi_tid, low_tid; + hi_tid = ((u64) cm_id_priv->av.port->mad_agent->hi_tid) << 32; + low_tid = (u64) (cm_id_priv->id.local_id | (msg_seq << 30)); + return cpu_to_be64(hi_tid | low_tid); +} + +static void cm_format_mad_hdr(struct ib_mad_hdr *hdr, + enum cm_msg_attr_id attr_id, u64 tid) +{ hdr->base_version = IB_MGMT_BASE_VERSION; hdr->mgmt_class = IB_MGMT_CLASS_CM; hdr->class_version = IB_CM_CLASS_VERSION; hdr->method = IB_MGMT_METHOD_SEND; hdr->attr_id = attr_id; - - hi_tid = ((u64) cm_id_priv->av.port->mad_agent->hi_tid) << 32; - low_tid = (u64) (cm_id_priv->id.local_id | (msg_seq << 30)); - hdr->tid = cpu_to_be64(hi_tid | low_tid); + hdr->tid = tid; } static void cm_format_req(struct cm_req_msg *req_msg, struct cm_id_private *cm_id_priv, struct ib_cm_req_param *param) { - cm_format_mad_hdr(&req_msg->hdr, cm_id_priv, - CM_REQ_ATTR_ID, CM_MSG_SEQUENCE_REQ); + cm_format_mad_hdr(&req_msg->hdr, CM_REQ_ATTR_ID, + cm_form_tid(cm_id_priv, CM_MSG_SEQUENCE_REQ)); req_msg->local_comm_id = cm_id_priv->id.local_id; req_msg->service_id = param->service_id; @@ -755,6 +776,10 @@ static void cm_format_req(struct cm_req_ static inline int cm_validate_req_param(struct ib_cm_req_param *param) { + /* peer-to-peer not supported */ + if (param->peer_to_peer) + return -EINVAL; + if (!param->primary_path) return -EINVAL; @@ -796,14 +821,19 @@ int ib_send_cm_req(struct ib_cm_id *cm_i } spin_unlock_irqrestore(&cm_id_priv->lock, flags); + cm_id_priv->timewait_info = cm_create_timewait_info(cm_id_priv-> + id.local_id); + if (IS_ERR(cm_id_priv->timewait_info)) + goto out; + ret = cm_init_av_by_path(param->primary_path, &cm_id_priv->av); if (ret) - goto out; + goto error1; if (param->alternate_path) { ret = cm_init_av_by_path(param->alternate_path, &cm_id_priv->alt_av); if (ret) - goto out; + goto error1; } cm_id->service_id = param->service_id; cm_id->service_mask = ~0ULL; @@ -819,7 +849,7 @@ int ib_send_cm_req(struct ib_cm_id *cm_i ret = cm_alloc_msg(cm_id_priv, &cm_id_priv->msg); if (ret) - goto out; + goto error1; req_msg = (struct cm_req_msg *) cm_id_priv->msg->mad; cm_format_req(req_msg, cm_id_priv, param); @@ -831,35 +861,61 @@ int ib_send_cm_req(struct ib_cm_id *cm_i cm_id_priv->local_ack_timeout = cm_req_get_primary_local_ack_timeout(req_msg); - /* - * Received REQs won't match until we're in REQ_SENT state. This - * simplifies error recovery if the send fails. - */ - if (param->peer_to_peer) { - ret = -EINVAL; - goto out; - } - spin_lock_irqsave(&cm_id_priv->lock, flags); ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent, &cm_id_priv->msg->send_wr, &bad_send_wr); - if (ret) { spin_unlock_irqrestore(&cm_id_priv->lock, flags); - /* if (param->peer_to_peer) { - cleanup peer_service_table - } */ - cm_free_msg(cm_id_priv->msg); - goto out; + goto error2; } BUG_ON(cm_id->state != IB_CM_IDLE); cm_id->state = IB_CM_REQ_SENT; spin_unlock_irqrestore(&cm_id_priv->lock, flags); -out: - return ret; + return 0; + +error2: cm_free_msg(cm_id_priv->msg); +error1: kfree(cm_id_priv->timewait_info); +out: return ret; } EXPORT_SYMBOL(ib_send_cm_req); +static int cm_issue_rej(struct cm_port *port, + struct ib_mad_recv_wc *mad_recv_wc, + enum ib_cm_rej_reason reason, + enum cm_msg_response msg_rejected, + void *ari, u8 ari_length) +{ + struct ib_mad_send_buf *msg; + struct ib_send_wr *bad_send_wr; + struct cm_rej_msg *rej_msg, *rcv_msg; + int ret; + + ret = cm_alloc_response_msg(port, mad_recv_wc, &msg); + if (ret) + return ret; + + /* We just need common CM header information. Cast to any message. */ + rcv_msg = (struct cm_rej_msg *) mad_recv_wc->recv_buf.mad; + rej_msg = (struct cm_rej_msg *) msg->mad; + + cm_format_mad_hdr(&rej_msg->hdr, CM_REJ_ATTR_ID, rcv_msg->hdr.tid); + rej_msg->remote_comm_id = rcv_msg->local_comm_id; + rej_msg->local_comm_id = rcv_msg->remote_comm_id; + cm_rej_set_msg_rejected(rej_msg, msg_rejected); + rej_msg->reason = reason; + + if (ari && ari_length) { + cm_rej_set_reject_info_len(rej_msg, ari_length); + memcpy(rej_msg->ari, ari, ari_length); + } + + ret = ib_post_send_mad(port->mad_agent, &msg->send_wr, &bad_send_wr); + if (ret) + cm_free_msg(msg); + + return ret; +} + static inline int cm_is_active_peer(u64 local_ca_guid, u64 remote_ca_guid, u32 local_qpn, u32 remote_qpn) { @@ -992,29 +1048,31 @@ static int cm_req_handler(struct cm_work cm_id_priv->id.remote_id = req_msg->local_comm_id; cm_init_av_for_response(work->port, work->mad_recv_wc->wc, &cm_id_priv->av); - cm_id_priv->timewait_info = cm_create_timewait_info( - cm_id_priv->id.local_id, - cm_id_priv->id.remote_id, - req_msg->local_ca_guid, - cm_req_get_local_qpn(req_msg)); + cm_id_priv->timewait_info = cm_create_timewait_info(cm_id_priv-> + id.local_id); if (IS_ERR(cm_id_priv->timewait_info)) { ret = PTR_ERR(cm_id_priv->timewait_info); goto error1; } + cm_id_priv->timewait_info->work.remote_id = req_msg->local_comm_id; + cm_id_priv->timewait_info->remote_ca_guid = req_msg->local_ca_guid; + cm_id_priv->timewait_info->remote_qpn = cm_req_get_local_qpn(req_msg); spin_lock_irqsave(&cm.lock, flags); /* Check for duplicate REQ. */ if (cm_insert_remote_id(cm_id_priv->timewait_info)) { spin_unlock_irqrestore(&cm.lock, flags); ret = -EINVAL; - goto error1; + goto error2; } /* Check for a stale connection. */ if (cm_insert_remote_qpn(cm_id_priv->timewait_info)) { spin_unlock_irqrestore(&cm.lock, flags); - /* todo: reject as stale */ + cm_issue_rej(work->port, work->mad_recv_wc, + IB_CM_REJ_STALE_CONN, CM_MSG_RESPONSE_REQ, + NULL, 0); ret = -EINVAL; - goto error1; + goto error2; } /* Find matching listen request. */ listen_cm_id_priv = cm_find_listen(req_msg->service_id); @@ -1035,11 +1093,11 @@ static int cm_req_handler(struct cm_work cm_format_paths_from_req(req_msg, &work->path[0], &work->path[1]); ret = cm_init_av_by_path(&work->path[0], &cm_id_priv->av); if (ret) - goto error2; + goto error3; if (req_msg->alt_local_lid) { ret = cm_init_av_by_path(&work->path[1], &cm_id_priv->alt_av); if (ret) - goto error2; + goto error3; } cm_id_priv->timeout_ms = cm_convert_to_ms( cm_req_get_local_resp_timeout(req_msg)); @@ -1058,11 +1116,12 @@ static int cm_req_handler(struct cm_work cm_process_work(cm_id_priv, work); cm_deref_id(listen_cm_id_priv); return 0; -error2: - atomic_dec(&cm_id_priv->refcount); + +error3: atomic_dec(&cm_id_priv->refcount); cm_deref_id(listen_cm_id_priv); -error1: - ib_destroy_cm_id(&cm_id_priv->id); +error2: cm_cleanup_timewait(cm_id_priv->timewait_info); + kfree(cm_id_priv->timewait_info); +error1: ib_destroy_cm_id(&cm_id_priv->id); return ret; } @@ -1070,8 +1129,9 @@ static void cm_format_rep(struct cm_rep_ struct cm_id_private *cm_id_priv, struct ib_cm_rep_param *param) { - cm_format_mad_hdr(&rep_msg->hdr, cm_id_priv, - CM_REP_ATTR_ID, CM_MSG_SEQUENCE_REQ); + /* todo: TID should match received REQ */ + cm_format_mad_hdr(&rep_msg->hdr, CM_REP_ATTR_ID, + cm_form_tid(cm_id_priv, CM_MSG_SEQUENCE_REQ)); rep_msg->local_comm_id = cm_id_priv->id.local_id; rep_msg->remote_comm_id = cm_id_priv->id.remote_id; @@ -1148,8 +1208,8 @@ static void cm_format_rtu(struct cm_rtu_ void *private_data, u8 private_data_len) { - cm_format_mad_hdr(&rtu_msg->hdr, cm_id_priv, - CM_RTU_ATTR_ID, CM_MSG_SEQUENCE_REQ); + cm_format_mad_hdr(&rtu_msg->hdr, CM_RTU_ATTR_ID, + cm_form_tid(cm_id_priv, CM_MSG_SEQUENCE_REQ)); rtu_msg->local_comm_id = cm_id_priv->id.local_id; rtu_msg->remote_comm_id = cm_id_priv->id.remote_id; @@ -1262,7 +1322,6 @@ static void cm_dup_rep_handler(struct cm static int cm_rep_handler(struct cm_work *work) { struct cm_id_private *cm_id_priv; - struct cm_timewait_info *timewait_info; struct cm_rep_msg *rep_msg; unsigned long flags; int ret; @@ -1274,27 +1333,25 @@ static int cm_rep_handler(struct cm_work return -EINVAL; } - timewait_info = cm_create_timewait_info(cm_id_priv->id.local_id, - rep_msg->local_comm_id, - rep_msg->local_ca_guid, - cm_rep_get_local_qpn(rep_msg)); - if (IS_ERR(timewait_info)) { - ret = PTR_ERR(timewait_info); - goto error1; - } + cm_id_priv->timewait_info->work.remote_id = rep_msg->local_comm_id; + cm_id_priv->timewait_info->remote_ca_guid = rep_msg->local_ca_guid; + cm_id_priv->timewait_info->remote_qpn = cm_rep_get_local_qpn(rep_msg); + spin_lock_irqsave(&cm.lock, flags); /* Check for duplicate REP. */ - if (cm_insert_remote_id(timewait_info)) { + if (cm_insert_remote_id(cm_id_priv->timewait_info)) { spin_unlock_irqrestore(&cm.lock, flags); ret = -EINVAL; - goto error2; + goto error; } /* Check for a stale connection. */ - if (cm_insert_remote_qpn(timewait_info)) { + if (cm_insert_remote_qpn(cm_id_priv->timewait_info)) { spin_unlock_irqrestore(&cm.lock, flags); - /* todo: reject as stale */ + cm_issue_rej(work->port, work->mad_recv_wc, + IB_CM_REJ_STALE_CONN, CM_MSG_RESPONSE_REP, + NULL, 0); ret = -EINVAL; - goto error2; + goto error; } spin_unlock_irqrestore(&cm.lock, flags); @@ -1308,7 +1365,7 @@ static int cm_rep_handler(struct cm_work default: spin_unlock_irqrestore(&cm_id_priv->lock, flags); ret = -EINVAL; - goto error2; + goto error; } cm_id_priv->id.state = IB_CM_REP_RCVD; cm_id_priv->id.remote_id = rep_msg->local_comm_id; @@ -1317,7 +1374,6 @@ static int cm_rep_handler(struct cm_work cm_id_priv->responder_resources = rep_msg->initiator_depth; cm_id_priv->sq_psn = cm_rep_get_starting_psn(rep_msg); cm_id_priv->rnr_retry_count = cm_rep_get_rnr_retry_count(rep_msg); - cm_id_priv->timewait_info = timewait_info; /* todo: handle peer_to_peer */ @@ -1333,10 +1389,8 @@ static int cm_rep_handler(struct cm_work else cm_deref_id(cm_id_priv); return 0; -error2: - cm_cleanup_timewait(timewait_info); - kfree(timewait_info); -error1: + +error: cm_cleanup_timewait(cm_id_priv->timewait_info); cm_deref_id(cm_id_priv); return ret; } @@ -1420,8 +1474,8 @@ static void cm_format_dreq(struct cm_dre void *private_data, u8 private_data_len) { - cm_format_mad_hdr(&dreq_msg->hdr, cm_id_priv, - CM_DREQ_ATTR_ID, CM_MSG_SEQUENCE_DREQ); + cm_format_mad_hdr(&dreq_msg->hdr, CM_DREQ_ATTR_ID, + cm_form_tid(cm_id_priv, CM_MSG_SEQUENCE_DREQ)); dreq_msg->local_comm_id = cm_id_priv->id.local_id; dreq_msg->remote_comm_id = cm_id_priv->id.remote_id; @@ -1480,8 +1534,9 @@ static void cm_format_drep(struct cm_dre void *private_data, u8 private_data_len) { - cm_format_mad_hdr(&drep_msg->hdr, cm_id_priv, - CM_DREP_ATTR_ID, CM_MSG_SEQUENCE_DREQ); + /* todo: TID should match received DREQ */ + cm_format_mad_hdr(&drep_msg->hdr, CM_DREP_ATTR_ID, + cm_form_tid(cm_id_priv, CM_MSG_SEQUENCE_DREQ)); drep_msg->local_comm_id = cm_id_priv->id.local_id; drep_msg->remote_comm_id = cm_id_priv->id.remote_id; @@ -1642,8 +1697,9 @@ static void cm_format_rej(struct cm_rej_ void *private_data, u8 private_data_len) { - cm_format_mad_hdr(&rej_msg->hdr, cm_id_priv, - CM_REJ_ATTR_ID, CM_MSG_SEQUENCE_REQ); + /* todo: TID should match received REQ */ + cm_format_mad_hdr(&rej_msg->hdr, CM_REJ_ATTR_ID, + cm_form_tid(cm_id_priv, CM_MSG_SEQUENCE_REQ)); rej_msg->remote_comm_id = cm_id_priv->id.remote_id; @@ -1861,8 +1917,9 @@ static void cm_format_mra(struct cm_mra_ return; } spin_unlock_irqrestore(&cm_id_priv->lock, flags); - cm_format_mad_hdr(&mra_msg->hdr, cm_id_priv, - CM_MRA_ATTR_ID, msg_sequence); + /* todo: TID should matched REQ or LAP */ + cm_format_mad_hdr(&mra_msg->hdr, CM_MRA_ATTR_ID, + cm_form_tid(cm_id_priv, msg_sequence)); cm_mra_set_msg_mraed(mra_msg, msg_mraed); mra_msg->local_comm_id = cm_id_priv->id.local_id; @@ -1946,8 +2003,8 @@ static void cm_format_lap(struct cm_lap_ void *private_data, u8 private_data_len) { - cm_format_mad_hdr(&lap_msg->hdr, cm_id_priv, - CM_LAP_ATTR_ID, CM_MSG_SEQUENCE_LAP); + cm_format_mad_hdr(&lap_msg->hdr, CM_LAP_ATTR_ID, + cm_form_tid(cm_id_priv, CM_MSG_SEQUENCE_LAP)); lap_msg->local_comm_id = cm_id_priv->id.local_id; lap_msg->remote_comm_id = cm_id_priv->id.remote_id; @@ -2089,8 +2146,9 @@ static void cm_format_apr(struct cm_apr_ void *private_data, u8 private_data_len) { - cm_format_mad_hdr(&apr_msg->hdr, cm_id_priv, - CM_APR_ATTR_ID, CM_MSG_SEQUENCE_LAP); + /* todo: TID should match received LAP */ + cm_format_mad_hdr(&apr_msg->hdr, CM_APR_ATTR_ID, + cm_form_tid(cm_id_priv, CM_MSG_SEQUENCE_LAP)); apr_msg->local_comm_id = cm_id_priv->id.local_id; apr_msg->remote_comm_id = cm_id_priv->id.remote_id; @@ -2237,8 +2295,8 @@ static void cm_format_sidr_req(struct cm struct cm_id_private *cm_id_priv, struct ib_cm_sidr_req_param *param) { - cm_format_mad_hdr(&sidr_req_msg->hdr, cm_id_priv, - CM_SIDR_REQ_ATTR_ID, CM_MSG_SEQUENCE_SIDR); + cm_format_mad_hdr(&sidr_req_msg->hdr, CM_SIDR_REQ_ATTR_ID, + cm_form_tid(cm_id_priv, CM_MSG_SEQUENCE_SIDR)); sidr_req_msg->request_id = cm_id_priv->id.local_id; sidr_req_msg->pkey = param->pkey; @@ -2351,7 +2409,7 @@ static int cm_sidr_req_handler(struct cm if (!cur_cm_id_priv) { rb_erase(&cm_id_priv->sidr_id_node, &cm.remote_sidr_table); spin_unlock_irqrestore(&cm.lock, flags); - /* todo: reject with no match */ + /* todo: reply with no match */ goto out; /* No match. */ } atomic_inc(&cur_cm_id_priv->refcount); @@ -2375,8 +2433,9 @@ static void cm_format_sidr_rep(struct cm struct cm_id_private *cm_id_priv, struct ib_cm_sidr_rep_param *param) { - cm_format_mad_hdr(&sidr_rep_msg->hdr, cm_id_priv, - CM_SIDR_REP_ATTR_ID, CM_MSG_SEQUENCE_SIDR); + /* todo: TID should match received SIDR REQ */ + cm_format_mad_hdr(&sidr_rep_msg->hdr, CM_SIDR_REP_ATTR_ID, + cm_form_tid(cm_id_priv, CM_MSG_SEQUENCE_SIDR)); sidr_rep_msg->request_id = cm_id_priv->id.remote_id; sidr_rep_msg->status = param->status; From halr at voltaire.com Fri May 13 13:41:14 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 13 May 2005 16:41:14 -0400 Subject: [openib-general] Re: MAD module removal and MAD cache In-Reply-To: <42850083.5080706@ichips.intel.com> References: <1116011792.4724.10.camel@localhost.localdomain> <42850083.5080706@ichips.intel.com> Message-ID: <1116016873.4476.37.camel@localhost.localdomain> On Fri, 2005-05-13 at 15:31, Sean Hefty wrote: > Hal Rosenstock wrote: > > Hi Sean, > > > > The MAD send code can place a send request on the send list, the QP's > > send queue, or overflow list. > > > > On port close, I see the receive queue being cleaned up but not these. > > > > Am I missing something ? If I'm not, then this may be the MAD cache leak > > which causes a problem when running SM, killing it, and trying to remove > > the modules. > > When a MAD is sent, a reference is taken on the mad_agent. Cleanup of these > lists should occur automatically as a send operation completes. We need to > be careful about trying to perform cleanup during mad_agent destruction, > since the MAD may be posted to the QP. If it is, we need to wait until we > see its completion. > > When a mad_agent is destroyed, all outstanding MADs on its send_list are > canceled. This will complete all MADs waiting for a response, but those on > the QP's send_queue or overflow_list still need to wait for a completion. > > If the mad_agent destruction is proceeding, then it sounds like these lists > are empty. To verify, you could add a check that the mad_agent's send_list > is empty before deregister returns. I forgot that was how it worked.. I used to know this. The completions are not occuring in that case: May 13 16:17:47 localhost kernel: ib_mad: handle_outgoing_dr_smp: MAD 0xce9a4030 allocated May 13 16:17:47 localhost kernel: ib_mad: handle_outgoing_dr_smp: MAD 0xce9a41c0 allocated May 13 16:17:47 localhost kernel: ib_mad: handle_outgoing_dr_smp: MAD 0xce9a4350 allocated May 13 16:17:47 localhost kernel: ib_mad: handle_outgoing_dr_smp: MAD 0xce9a44e0 allocated May 13 16:17:47 localhost kernel: ib_mad: handle_outgoing_dr_smp: MAD 0xc2ea5e40 allocated I think this is because handle_outgoing_dr_smp isn't "playing by the rules" in ib_post_send_mad so it is not being tracked appropriately :-( My bad... -- Hal From roland at topspin.com Fri May 13 13:37:47 2005 From: roland at topspin.com (Roland Dreier) Date: Fri, 13 May 2005 13:37:47 -0700 Subject: [openib-general] [PATCH 2/2] libmthca: max_inline_data support In-Reply-To: <20050509141631.GI9941@mellanox.co.il> (Michael S. Tsirkin's message of "Mon, 9 May 2005 17:16:31 +0300") References: <20050509141631.GI9941@mellanox.co.il> Message-ID: <52zmuyrhes.fsf@topspin.com> Thanks, I committed the following patch. I fixed the computation + qp->sq.max_gs = align(cap->max_inline_data + + sizeof (struct mthca_inline_seg), + sizeof (struct mthca_data_seg)); to divide by sizeof (struct mthca_data_seg) -- otherwise it seems that max_gs will end up far too large. I also added some code to return the actual values allocated back to the consumer. - R. --- libmthca/src/qp.c (revision 2206) +++ libmthca/src/qp.c (working copy) @@ -718,10 +718,17 @@ out: return ret; } -int mthca_alloc_qp_buf(struct ibv_pd *pd, struct mthca_qp *qp) +int mthca_alloc_qp_buf(struct ibv_pd *pd, struct ibv_qp_cap *cap, + struct mthca_qp *qp) { int size; + qp->rq.max_gs = cap->max_recv_sge; + qp->sq.max_gs = align(cap->max_inline_data + sizeof (struct mthca_inline_seg), + sizeof (struct mthca_data_seg)) / sizeof (struct mthca_data_seg); + if (qp->sq.max_gs < cap->max_send_sge) + qp->sq.max_gs = cap->max_send_sge; + qp->wrid = malloc((qp->rq.max + qp->sq.max) * sizeof (uint64_t)); if (!qp->wrid) return -1; @@ -796,6 +803,39 @@ int mthca_alloc_qp_buf(struct ibv_pd *pd return 0; } +void mthca_return_cap(struct ibv_pd *pd, struct mthca_qp *qp, struct ibv_qp_cap *cap) +{ + /* + * Maximum inline data size is the full WQE size less the size + * of the next segment, inline segment and other non-data segments. + */ + cap->max_inline_data = (1 << qp->sq.wqe_shift) - + sizeof (struct mthca_next_seg) - + sizeof (struct mthca_inline_seg); + + switch (qp->qpt) { + case IBV_QPT_UD: + if (mthca_is_memfree(pd->context)) + cap->max_inline_data -= sizeof (struct mthca_arbel_ud_seg); + else + cap->max_inline_data -= sizeof (struct mthca_tavor_ud_seg); + break; + + default: + /* + * inline data won't be used in the same WQE as an + * atomic or bind segment, so we don't have to + * subtract anything off here. + */ + break; + } + + cap->max_send_wr = qp->sq.max; + cap->max_recv_wr = qp->rq.max; + cap->max_send_sge = qp->sq.max_gs; + cap->max_recv_sge = qp->rq.max_gs; +} + struct mthca_qp *mthca_find_qp(struct mthca_context *ctx, uint32_t qpn) { int tind = (qpn & (ctx->num_qps - 1)) >> ctx->qp_table_shift; --- libmthca/src/verbs.c (revision 2206) +++ libmthca/src/verbs.c (working copy) @@ -270,7 +270,6 @@ struct ibv_qp *mthca_create_qp(struct ib qp->qpt = attr->qp_type; qp->sq.max = align_qp_size(pd->context, attr->cap.max_send_wr); - qp->sq.max_gs = attr->cap.max_send_sge; qp->sq.next_ind = 0; qp->sq.last_comp = qp->sq.max - 1; qp->sq.head = 0; @@ -278,14 +277,13 @@ struct ibv_qp *mthca_create_qp(struct ib qp->sq.last = NULL; qp->rq.max = align_qp_size(pd->context, attr->cap.max_recv_wr); - qp->rq.max_gs = attr->cap.max_recv_sge; qp->rq.next_ind = 0; qp->rq.last_comp = qp->rq.max - 1; qp->rq.head = 0; qp->rq.tail = 0; qp->rq.last = NULL; - if (mthca_alloc_qp_buf(pd, qp)) + if (mthca_alloc_qp_buf(pd, &attr->cap, qp)) goto err; if (pthread_spin_init(&qp->sq.lock, PTHREAD_PROCESS_PRIVATE) || @@ -332,6 +330,8 @@ struct ibv_qp *mthca_create_qp(struct ib if (ret) goto err_destroy; + mthca_return_cap(pd, qp, &attr->cap); + return &qp->ibv_qp; err_destroy: --- libmthca/src/mthca.h (revision 2190) +++ libmthca/src/mthca.h (working copy) @@ -284,7 +284,10 @@ extern int mthca_arbel_post_send(struct struct ibv_send_wr **bad_wr); extern int mthca_arbel_post_recv(struct ibv_qp *ibqp, struct ibv_recv_wr *wr, struct ibv_recv_wr **bad_wr); -extern int mthca_alloc_qp_buf(struct ibv_pd *pd, struct mthca_qp *qp); +extern int mthca_alloc_qp_buf(struct ibv_pd *pd, struct ibv_qp_cap *cap, + struct mthca_qp *qp); +extern void mthca_return_cap(struct ibv_pd *pd, struct mthca_qp *qp, + struct ibv_qp_cap *cap); extern struct mthca_qp *mthca_find_qp(struct mthca_context *ctx, uint32_t qpn); extern int mthca_store_qp(struct mthca_context *ctx, uint32_t qpn, struct mthca_qp *qp); extern void mthca_clear_qp(struct mthca_context *ctx, uint32_t qpn); From roland at topspin.com Fri May 13 13:36:26 2005 From: roland at topspin.com (Roland Dreier) Date: Fri, 13 May 2005 13:36:26 -0700 Subject: [openib-general] Re: [PATCH 1/2] (fixed) mthca: max_inline_data support In-Reply-To: <20050510153543.GG2474@mellanox.co.il> (Michael S. Tsirkin's message of "Tue, 10 May 2005 18:35:43 +0300") References: <20050509141456.GH9941@mellanox.co.il> <52fyww1h90.fsf@topspin.com> <20050510153543.GG2474@mellanox.co.il> Message-ID: <524qd6sw1h.fsf@topspin.com> Thanks, I ended up committing this slightly modified patch: --- infiniband/hw/mthca/mthca_dev.h (revision 2287) +++ infiniband/hw/mthca/mthca_dev.h (working copy) @@ -439,12 +439,14 @@ int mthca_alloc_qp(struct mthca_dev *dev struct mthca_cq *recv_cq, enum ib_qp_type type, enum ib_sig_type send_policy, + struct ib_qp_cap *cap, struct mthca_qp *qp); int mthca_alloc_sqp(struct mthca_dev *dev, struct mthca_pd *pd, struct mthca_cq *send_cq, struct mthca_cq *recv_cq, enum ib_sig_type send_policy, + struct ib_qp_cap *cap, int qpn, int port, struct mthca_sqp *sqp); --- infiniband/hw/mthca/mthca_provider.c (revision 2287) +++ infiniband/hw/mthca/mthca_provider.c (working copy) @@ -486,16 +486,11 @@ static struct ib_qp *mthca_create_qp(str qp->rq.db_index = ucmd.rq_db_index; } - qp->sq.max = init_attr->cap.max_send_wr; - qp->rq.max = init_attr->cap.max_recv_wr; - qp->sq.max_gs = init_attr->cap.max_send_sge; - qp->rq.max_gs = init_attr->cap.max_recv_sge; - err = mthca_alloc_qp(to_mdev(pd->device), to_mpd(pd), to_mcq(init_attr->send_cq), to_mcq(init_attr->recv_cq), init_attr->qp_type, init_attr->sq_sig_type, - qp); + &init_attr->cap, qp); if (err && pd->uobject) { context = to_mucontext(pd->uobject->context); @@ -524,17 +519,12 @@ static struct ib_qp *mthca_create_qp(str if (!qp) return ERR_PTR(-ENOMEM); - qp->sq.max = init_attr->cap.max_send_wr; - qp->rq.max = init_attr->cap.max_recv_wr; - qp->sq.max_gs = init_attr->cap.max_send_sge; - qp->rq.max_gs = init_attr->cap.max_recv_sge; - qp->ibqp.qp_num = init_attr->qp_type == IB_QPT_SMI ? 0 : 1; err = mthca_alloc_sqp(to_mdev(pd->device), to_mpd(pd), to_mcq(init_attr->send_cq), to_mcq(init_attr->recv_cq), - init_attr->sq_sig_type, + init_attr->sq_sig_type, &init_attr->cap, qp->ibqp.qp_num, init_attr->port_num, to_msqp(qp)); break; @@ -552,6 +542,8 @@ static struct ib_qp *mthca_create_qp(str init_attr->cap.max_inline_data = 0; init_attr->cap.max_send_wr = qp->sq.max; init_attr->cap.max_recv_wr = qp->rq.max; + init_attr->cap.max_send_sge = qp->sq.max_gs; + init_attr->cap.max_recv_sge = qp->rq.max_gs; return &qp->ibqp; } --- infiniband/hw/mthca/mthca_qp.c (revision 2287) +++ infiniband/hw/mthca/mthca_qp.c (working copy) @@ -46,7 +46,9 @@ enum { MTHCA_MAX_DIRECT_QP_SIZE = 4 * PAGE_SIZE, MTHCA_ACK_REQ_FREQ = 10, MTHCA_FLIGHT_LIMIT = 9, - MTHCA_UD_HEADER_SIZE = 72 /* largest UD header possible */ + MTHCA_UD_HEADER_SIZE = 72, /* largest UD header possible */ + MTHCA_INLINE_HEADER_SIZE = 4, /* data segment overhead for inline */ + MTHCA_INLINE_CHUNK_SIZE = 16 /* inline data segment chunk */ }; enum { @@ -1205,22 +1207,31 @@ static int mthca_alloc_qp_common(struct return 0; } -static void mthca_align_qp_size(struct mthca_dev *dev, struct mthca_qp *qp) +static int mthca_set_qp_size(struct mthca_dev *dev, struct ib_qp_cap *cap, + struct mthca_qp *qp) { - int i; - - if (!mthca_is_memfree(dev)) - return; - - for (i = 0; 1 << i < qp->rq.max; ++i) - ; /* nothing */ + if (mthca_is_memfree(dev)) { + qp->rq.max = roundup_pow_of_two(cap->max_recv_wr); + qp->sq.max = roundup_pow_of_two(cap->max_send_wr); + } else { + qp->rq.max = cap->max_recv_wr; + qp->sq.max = cap->max_send_wr; + } - qp->rq.max = 1 << i; + qp->rq.max_gs = cap->max_recv_sge; + qp->sq.max_gs = max_t(int, cap->max_send_sge, + ALIGN(cap->max_inline_data + MTHCA_INLINE_HEADER_SIZE, + MTHCA_INLINE_CHUNK_SIZE) / sizeof (struct mthca_data_seg)); - for (i = 0; 1 << i < qp->sq.max; ++i) - ; /* nothing */ + /* + * For MLX transport we need 2 extra S/G entries: + * one for the header and one for the checksum at the end + */ + if ((qp->transport == MLX && qp->sq.max_gs + 2 > dev->limits.max_sg) || + qp->sq.max_gs > dev->limits.max_sg || qp->rq.max_gs > dev->limits.max_sg) + return -EINVAL; - qp->sq.max = 1 << i; + return 0; } int mthca_alloc_qp(struct mthca_dev *dev, @@ -1229,11 +1240,14 @@ int mthca_alloc_qp(struct mthca_dev *dev struct mthca_cq *recv_cq, enum ib_qp_type type, enum ib_sig_type send_policy, + struct ib_qp_cap *cap, struct mthca_qp *qp) { int err; - mthca_align_qp_size(dev, qp); + err = mthca_set_qp_size(dev, cap, qp); + if (err) + return err; switch (type) { case IB_QPT_RC: qp->transport = RC; break; @@ -1266,14 +1280,17 @@ int mthca_alloc_sqp(struct mthca_dev *de struct mthca_cq *send_cq, struct mthca_cq *recv_cq, enum ib_sig_type send_policy, + struct ib_qp_cap *cap, int qpn, int port, struct mthca_sqp *sqp) { - int err = 0; u32 mqpn = qpn * 2 + dev->qp_table.sqp_start + port - 1; + int err; - mthca_align_qp_size(dev, &sqp->qp); + err = mthca_set_qp_size(dev, cap, &sqp->qp); + if (err) + return err; sqp->header_buf_size = sqp->qp.sq.max * MTHCA_UD_HEADER_SIZE; sqp->header_buf = dma_alloc_coherent(&dev->pdev->dev, sqp->header_buf_size, From tduffy at sun.com Fri May 13 14:04:30 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 13 May 2005 14:04:30 -0700 Subject: [openib-general] [PATCH] kDAPL: change EXPORT_SYMBOL to EXPORT_SYMBOL_GPL In-Reply-To: <469958e00505131152573cd93a@mail.gmail.com> References: <1115910070.25530.2.camel@duffman> <1115915608.9070.0.camel@duffman> <469958e00505121256cd8c04d@mail.gmail.com> <469958e00505131152573cd93a@mail.gmail.com> Message-ID: <1116018270.20109.8.camel@duffman> On Fri, 2005-05-13 at 11:52 -0700, Caitlin Bestler wrote: > What about kernel clients that are compiled separately from the kernel > that want to be OS neutral and use kdat.h as published by the DAT > Collaborative? Make sure this doesn't happen. Signed-off-by: Tom Duffy Index: linux-kernel-gpl/dat/api.c =================================================================== --- linux-kernel-gpl/dat/api.c (revision 2338) +++ linux-kernel-gpl/dat/api.c (working copy) @@ -619,7 +619,7 @@ u32 dat_ia_openv(const char *name, int a name, async_event_qlen, async_event_handle, ia_handle); } -EXPORT_SYMBOL(dat_ia_openv); +EXPORT_SYMBOL_GPL(dat_ia_openv); /*********************************************************************** * Function: dat_ia_close @@ -673,7 +673,7 @@ u32 dat_ia_close(DAT_IA_HANDLE ia_handle return status; } -EXPORT_SYMBOL(dat_ia_close); +EXPORT_SYMBOL_GPL(dat_ia_close); /*********************************************************************** * Function: dat_strerror @@ -706,7 +706,7 @@ u32 dat_strerror(u32 value, const char * return DAT_SUCCESS; } -EXPORT_SYMBOL(dat_strerror); +EXPORT_SYMBOL_GPL(dat_strerror); /*********************************************************************** * Function: dat_registry_add_provider @@ -731,7 +731,7 @@ u32 dat_registry_add_provider(const stru return dat_dr_insert(provider_info, provider->ia_open_func); } -EXPORT_SYMBOL(dat_registry_remove_provider); +EXPORT_SYMBOL_GPL(dat_registry_remove_provider); /*********************************************************************** * Function: dat_registry_remove_provider @@ -750,7 +750,7 @@ u32 dat_registry_remove_provider(const s return dat_dr_remove(provider_info); } -EXPORT_SYMBOL(dat_registry_add_provider); +EXPORT_SYMBOL_GPL(dat_registry_add_provider); /*********************************************************************** * Function: dat_registry_list_providers @@ -787,4 +787,4 @@ u32 dat_registry_list_providers(int max_ return status; } -EXPORT_SYMBOL(dat_registry_list_providers); +EXPORT_SYMBOL_GPL(dat_registry_list_providers); From halr at voltaire.com Fri May 13 14:32:23 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 13 May 2005 17:32:23 -0400 Subject: [openib-general] [PATCH] [kdapl] CM: Eliminate explicit send DREQ on error in do_rep_recv Message-ID: <1116019630.4729.2.camel@localhost.localdomain> CM: Eliminate explicit send DREQ on error in do_rep_recv Signed-off-by: Hal Rosenstock Index: dapl_openib_cm.c =================================================================== --- dapl_openib_cm.c (revision 2348) +++ dapl_openib_cm.c (working copy) @@ -99,7 +99,6 @@ return; disc: - ib_send_cm_dreq(conn->cm_id, NULL, 0); /* ??? */ dapl_evd_connection_callback(conn, IB_CME_LOCAL_FAILURE, NULL, conn->ep); } From roland at topspin.com Fri May 13 14:19:48 2005 From: roland at topspin.com (Roland Dreier) Date: Fri, 13 May 2005 14:19:48 -0700 Subject: [openib-general] Re: inline support in Gen2 In-Reply-To: <4283ED8D.2020101@cse.ohio-state.edu> (Sayantan Sur's message of "Thu, 12 May 2005 19:58:05 -0400") References: <20050512221803.GA14851@cse.ohio-state.edu> <52br7gt4tt.fsf@topspin.com> <4283ED8D.2020101@cse.ohio-state.edu> Message-ID: <52vf5mrfgr.fsf@topspin.com> The max_inline_data changes are now checked in. You can pass a request for a max_inline_data value into ibv_create_qp() via qp_attr.cap.max_inline_data, and the actual value (guaranteed to be at least as much as you requested) will be returned in the same location. - R. From halr at voltaire.com Fri May 13 16:14:16 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 13 May 2005 19:14:16 -0400 Subject: [openib-general] Re: slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects In-Reply-To: <1115398343.4476.154.camel@localhost.localdomain> References: <20050502084013.GR16918@mellanox.co.il> <1115036059.4477.5703.camel@localhost.localdomain> <42765027.10605@ichips.intel.com> <20050502163327.GB9939@mellanox.co.il> <1115066730.4476.139.camel@localhost.localdomain> <52u0llfiot.fsf@topspin.com> <20050503072727.GD9674@mellanox.co.il> <5264y0e49p.fsf@topspin.com> <1115134226.4476.618.camel@localhost.localdomain> <20050504065759.GA9674@mellanox.co.il> <1115398343.4476.154.camel@localhost.localdomain> Message-ID: <1116026056.4480.6.camel@localhost.localdomain> On Fri, 2005-05-06 at 12:52, Hal Rosenstock wrote: > On Wed, 2005-05-04 at 02:57, Michael S. Tsirkin wrote: > > > If this only occurs on machines which are running OpenSM, it could be a > > > problem with the local MAD handling where ib_free_recv_mad is not called > > > for some case where it should. I will look into this if this is the > > > case. > > > > > > -- Hal > > > > > > > This is the case for me: the error only occurs on machines which are > > running OpenSM. > > So assuming mthca sets the bits properly for local MAD handling (and I > have no reason to think otherwise), this is likely a local MAD handling > error in mad.c where some path does not return an allocation to the MAD > cache. I inspected the code to try and find it but couldn't. I will need > to dig deeper. This will take some more time. (I will add some > accounting in and see what this shows up). Please bear with me. This patch works for me. Can you try it out ? If it works for you, I will check it in. -- Hal Index: mad.c =================================================================== -- mad.c (revision 2331) +++ mad.c (working copy) @@ -1602,7 +1602,6 @@ INIT_LIST_HEAD(&mad_recv_wc->rmpp_list); list_add(&mad_recv_wc->recv_buf.list, &mad_recv_wc->rmpp_list); - if (mad_agent_priv->agent.rmpp_version) { mad_recv_wc = ib_process_rmpp_recv_wc(mad_agent_priv, mad_recv_wc); @@ -2054,6 +2053,8 @@ /* Empty wait list to prevent receives from finding a request */ list_splice_init(&mad_agent_priv->wait_list, &cancel_list); + /* Empty local completion list as well */ + list_splice_init(&mad_agent_priv->local_list, &cancel_list); spin_unlock_irqrestore(&mad_agent_priv->lock, flags); /* Report all cancelled requests */ @@ -2168,6 +2169,7 @@ struct ib_mad_local_private *local; struct ib_mad_agent_private *recv_mad_agent; unsigned long flags; + int recv = 0; struct ib_wc wc; struct ib_mad_send_wc mad_send_wc; @@ -2183,10 +2185,10 @@ recv_mad_agent = local->recv_mad_agent; if (!recv_mad_agent) { printk(KERN_ERR PFX "No receive MAD agent for local completion\n"); - kmem_cache_free(ib_mad_cache, local->mad_priv); goto local_send_completion; } + recv = 1; /* * Defined behavior is to complete response * before request @@ -2199,6 +2201,8 @@ local->mad_priv->header.recv_wc.mad_len = sizeof(struct ib_mad); INIT_LIST_HEAD(&local->mad_priv->header.recv_wc.rmpp_list); + list_add(&local->mad_priv->header.recv_wc.recv_buf.list, + &local->mad_priv->header.recv_wc.rmpp_list); local->mad_priv->header.recv_wc.recv_buf.grh = NULL; local->mad_priv->header.recv_wc.recv_buf.mad = &local->mad_priv->mad.mad; @@ -2229,6 +2233,8 @@ spin_lock_irqsave(&mad_agent_priv->lock, flags); list_del(&local->completion_list); atomic_dec(&mad_agent_priv->refcount); + if (!recv) + kmem_cache_free(ib_mad_cache, local->mad_priv); kfree(local); } spin_unlock_irqrestore(&mad_agent_priv->lock, flags); From roland at topspin.com Fri May 13 19:49:29 2005 From: roland at topspin.com (Roland Dreier) Date: Fri, 13 May 2005 19:49:29 -0700 Subject: [openib-general] Re: inline support in Gen2 In-Reply-To: <4283ED8D.2020101@cse.ohio-state.edu> (Sayantan Sur's message of "Thu, 12 May 2005 19:58:05 -0400") References: <20050512221803.GA14851@cse.ohio-state.edu> <52br7gt4tt.fsf@topspin.com> <4283ED8D.2020101@cse.ohio-state.edu> Message-ID: <52mzqyo72e.fsf@topspin.com> I just checked in implementations of ibv_attach_mcast() and ibv_detach_mcast(). I didn't really test them, so please let me know if you see any problems using them. - R. From boas1 at llnl.gov Fri May 13 16:16:43 2005 From: boas1 at llnl.gov (Bill Boas) Date: Fri, 13 May 2005 16:16:43 -0700 Subject: [openib-general] Survey to find Exhibitors requesting Infiniband in SCinet at SC05 Message-ID: <6.1.2.0.2.20050513144615.0c7fd330@mail-lc.llnl.gov> This email is being sent to the booth contacts for Industry and Research Exhibitors at SC05 by the OpenIB Alliance - see www.openib.org. The purpose of the attached survey is to find out which exhibitors would request from SCinet a 10 Gbps (Infiniband 4X) connection for their booth if SCinet was to offer it. If you are an industry exhibitor please forward with the attachment to your marketing and technical support people that will be deciding what you demonstrate on your booth at SC and will also know which partners or customers you would be likely to be working with on joint demos, etc. using Infiniband if it was available. Costs and connection information are in the attachment. If you are a research exhibitor please forward with the attachment to your technical team that will be deciding what demos and connections you will need at SC05. Costs and connection information are in the attachment. Please contact me with any questions you may have and also please respond even if you have no interest. Your response by May 30, 2005 is essential to allow enough time for planning etc. Bill Boas bboas at llnl.gov ICCD LLNL, B-453, R-2018 Wk: 925-422-4110 7000 East Ave, L-555 Cell: 925-337-2224 Livermore, CA 94551 Pgr: 877-203-2248 -------------- next part -------------- A non-text attachment was scrubbed... Name: Survey to Determine SC05 Exhibitor Interest in IB in SCinet.doc Type: application/msword Size: 101376 bytes Desc: not available URL: From mst at mellanox.co.il Sat May 14 12:19:30 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Sat, 14 May 2005 22:19:30 +0300 Subject: [openib-general] Re: Re: [PATCH] (repost) sdp: replace mlock with get_user_pages In-Reply-To: <20050513100149.B24123@topspin.com> References: <20050505110158.GH30653@mellanox.co.il> <20050506175415.I26823@topspin.com> <20050510143232.GF2474@mellanox.co.il> <20050512151051.C23734@topspin.com> <52fywst4z3.fsf@topspin.com> <20050513135145.GA320@mellanox.co.il> <20050513100149.B24123@topspin.com> Message-ID: <20050514191930.GB10140@mellanox.co.il> Quoting r. Libor Michalek : > Subject: Re: Re: [PATCH] (repost) sdp: replace mlock with get_user_pages > > On Fri, May 13, 2005 at 04:51:45PM +0300, Michael S. Tsirkin wrote: > > Quoting r. Roland Dreier : > > > > > > Libor> Always call aio_complete() immediately when > > > Libor> iocb_complete() is called, and only spawn the work thread > > > Libor> to unlock the memory after the aio_complete() call. The > > > Libor> patch is below. > > > > Libor, I dont think its a good idea - this will break other assumptions, > > like the assumption that the task mm isnt destroyed before we unlock > > the memory. > > That's a good point. > > > > Another alternative would be to always complete aios asynchronously, > > > which should preserve the order. I guess this would hurt latency for > > > small ios... > > > > To avoid hurting latency, lets count the number of outstanding > > asynchronous AIOs, and if there are asynchronous AIOs complete > > all of them asynchronously. > > > > Does this make sense? > > Yes, except that the current iocb code does not reference individual > sockets anywhere, and do_iocb_complete would have to be the function > which decremented the per connection counter of outstanding AIOs. But do_iocb_complete is always called with socket locked, is it not? If so its not hard to add that counting. > Also, > since we don't have to do get_user_pages a second time on the send > path, this would only need to be done on the recv path. > > -Libor > I agree this is a possible optimization, although of course everything still works if we do this for both send and receive. -- MST - Michael S. Tsirkin From mst at mellanox.co.il Sat May 14 12:57:03 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Sat, 14 May 2005 22:57:03 +0300 Subject: [openib-general] Re: [PATCH 1/2] (fixed) mthca: max_inline_data support In-Reply-To: <524qd6sw1h.fsf@topspin.com> References: <20050509141456.GH9941@mellanox.co.il> <52fyww1h90.fsf@topspin.com> <20050510153543.GG2474@mellanox.co.il> <524qd6sw1h.fsf@topspin.com> Message-ID: <20050514195703.GC10140@mellanox.co.il> Quoting r. Roland Dreier : > Subject: Re: [PATCH 1/2] (fixed) mthca: max_inline_data support > > Thanks, I ended up committing this slightly modified patch: > > --- infiniband/hw/mthca/mthca_dev.h (revision 2287) > +++ infiniband/hw/mthca/mthca_dev.h (working copy) > @@ -439,12 +439,14 @@ int mthca_alloc_qp(struct mthca_dev *dev > struct mthca_cq *recv_cq, > enum ib_qp_type type, > enum ib_sig_type send_policy, > + struct ib_qp_cap *cap, > struct mthca_qp *qp); > int mthca_alloc_sqp(struct mthca_dev *dev, > struct mthca_pd *pd, > struct mthca_cq *send_cq, > struct mthca_cq *recv_cq, > enum ib_sig_type send_policy, > + struct ib_qp_cap *cap, > int qpn, > int port, > struct mthca_sqp *sqp); > --- infiniband/hw/mthca/mthca_provider.c (revision 2287) > +++ infiniband/hw/mthca/mthca_provider.c (working copy) > @@ -486,16 +486,11 @@ static struct ib_qp *mthca_create_qp(str > qp->rq.db_index = ucmd.rq_db_index; > } > > - qp->sq.max = init_attr->cap.max_send_wr; > - qp->rq.max = init_attr->cap.max_recv_wr; > - qp->sq.max_gs = init_attr->cap.max_send_sge; > - qp->rq.max_gs = init_attr->cap.max_recv_sge; > - > err = mthca_alloc_qp(to_mdev(pd->device), to_mpd(pd), > to_mcq(init_attr->send_cq), > to_mcq(init_attr->recv_cq), > init_attr->qp_type, init_attr->sq_sig_type, > - qp); > + &init_attr->cap, qp); > > if (err && pd->uobject) { > context = to_mucontext(pd->uobject->context); > @@ -524,17 +519,12 @@ static struct ib_qp *mthca_create_qp(str > if (!qp) > return ERR_PTR(-ENOMEM); > > - qp->sq.max = init_attr->cap.max_send_wr; > - qp->rq.max = init_attr->cap.max_recv_wr; > - qp->sq.max_gs = init_attr->cap.max_send_sge; > - qp->rq.max_gs = init_attr->cap.max_recv_sge; > - > qp->ibqp.qp_num = init_attr->qp_type == IB_QPT_SMI ? 0 : 1; > > err = mthca_alloc_sqp(to_mdev(pd->device), to_mpd(pd), > to_mcq(init_attr->send_cq), > to_mcq(init_attr->recv_cq), > - init_attr->sq_sig_type, > + init_attr->sq_sig_type, &init_attr->cap, > qp->ibqp.qp_num, init_attr->port_num, > to_msqp(qp)); > break; > @@ -552,6 +542,8 @@ static struct ib_qp *mthca_create_qp(str > init_attr->cap.max_inline_data = 0; > init_attr->cap.max_send_wr = qp->sq.max; > init_attr->cap.max_recv_wr = qp->rq.max; > + init_attr->cap.max_send_sge = qp->sq.max_gs; > + init_attr->cap.max_recv_sge = qp->rq.max_gs; > > return &qp->ibqp; > } > --- infiniband/hw/mthca/mthca_qp.c (revision 2287) > +++ infiniband/hw/mthca/mthca_qp.c (working copy) > @@ -46,7 +46,9 @@ enum { > MTHCA_MAX_DIRECT_QP_SIZE = 4 * PAGE_SIZE, > MTHCA_ACK_REQ_FREQ = 10, > MTHCA_FLIGHT_LIMIT = 9, > - MTHCA_UD_HEADER_SIZE = 72 /* largest UD header possible */ > + MTHCA_UD_HEADER_SIZE = 72, /* largest UD header possible */ > + MTHCA_INLINE_HEADER_SIZE = 4, /* data segment overhead for inline */ > + MTHCA_INLINE_CHUNK_SIZE = 16 /* inline data segment chunk */ > }; > > enum { > @@ -1205,22 +1207,31 @@ static int mthca_alloc_qp_common(struct > return 0; > } > > -static void mthca_align_qp_size(struct mthca_dev *dev, struct mthca_qp *qp) > +static int mthca_set_qp_size(struct mthca_dev *dev, struct ib_qp_cap *cap, > + struct mthca_qp *qp) > { > - int i; > - > - if (!mthca_is_memfree(dev)) > - return; > - > - for (i = 0; 1 << i < qp->rq.max; ++i) > - ; /* nothing */ > + if (mthca_is_memfree(dev)) { > + qp->rq.max = roundup_pow_of_two(cap->max_recv_wr); > + qp->sq.max = roundup_pow_of_two(cap->max_send_wr); Roland, roundup_pow_of_two assumes that it gets a non-0 parameter, while I think its legal to have one of max_recv_wr / max_send_wr equal 0. Is that right? > + } else { > + qp->rq.max = cap->max_recv_wr; > + qp->sq.max = cap->max_send_wr; > + } > > - qp->rq.max = 1 << i; > + qp->rq.max_gs = cap->max_recv_sge; > + qp->sq.max_gs = max_t(int, cap->max_send_sge, > + ALIGN(cap->max_inline_data + MTHCA_INLINE_HEADER_SIZE, > + MTHCA_INLINE_CHUNK_SIZE) / sizeof (struct mthca_data_seg)); Lets keep lines <= 80, please? > > - for (i = 0; 1 << i < qp->sq.max; ++i) > - ; /* nothing */ > + /* > + * For MLX transport we need 2 extra S/G entries: > + * one for the header and one for the checksum at the end > + */ > + if ((qp->transport == MLX && qp->sq.max_gs + 2 > dev->limits.max_sg) || > + qp->sq.max_gs > dev->limits.max_sg || qp->rq.max_gs > dev->limits.max_sg) Same. > + return -EINVAL; > > - qp->sq.max = 1 << i; > + return 0; > } > > int mthca_alloc_qp(struct mthca_dev *dev, > @@ -1229,11 +1240,14 @@ int mthca_alloc_qp(struct mthca_dev *dev > struct mthca_cq *recv_cq, > enum ib_qp_type type, > enum ib_sig_type send_policy, > + struct ib_qp_cap *cap, > struct mthca_qp *qp) > { > int err; > > - mthca_align_qp_size(dev, qp); > + err = mthca_set_qp_size(dev, cap, qp); > + if (err) > + return err; > > switch (type) { > case IB_QPT_RC: qp->transport = RC; break; > @@ -1266,14 +1280,17 @@ int mthca_alloc_sqp(struct mthca_dev *de > struct mthca_cq *send_cq, > struct mthca_cq *recv_cq, > enum ib_sig_type send_policy, > + struct ib_qp_cap *cap, > int qpn, > int port, > struct mthca_sqp *sqp) > { > - int err = 0; > u32 mqpn = qpn * 2 + dev->qp_table.sqp_start + port - 1; > + int err; > > - mthca_align_qp_size(dev, &sqp->qp); > + err = mthca_set_qp_size(dev, cap, &sqp->qp); > + if (err) > + return err; > > sqp->header_buf_size = sqp->qp.sq.max * MTHCA_UD_HEADER_SIZE; > sqp->header_buf = dma_alloc_coherent(&dev->pdev->dev, sqp->header_buf_size, > -- MST - Michael S. Tsirkin From info at newwaycalls.com Sat May 14 19:32:43 2005 From: info at newwaycalls.com (Info New Way Calls) Date: Sun, 15 May 2005 03:32:43 +0100 Subject: [openib-general] How to call for free! Message-ID: <20050515013602.017E61A0842E0@csmtp.b-one.net> How to call for free! Como llamar gratis! Comment appelé gratuitement! Как звонить бесплатно! NEW WAY CALLS Make FREE Mobile Phone Calls with Cellwireless Telecommunication when you help your Family and Friends SAVE on their mobile calls! Best regards! Team New Way Calls. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.gif Type: image/gif Size: 9862 bytes Desc: not available URL: From mst at mellanox.co.il Sun May 15 01:05:03 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Sun, 15 May 2005 11:05:03 +0300 Subject: [openib-general] Re: [PATCH] (repost) sdp: replace mlock with get_user_pages In-Reply-To: <20050512151051.C23734@topspin.com> References: <20050505110158.GH30653@mellanox.co.il> <20050506175415.I26823@topspin.com> <20050510143232.GF2474@mellanox.co.il> <20050512151051.C23734@topspin.com> Message-ID: <20050515080503.GB13718@mellanox.co.il> Quoting r. Libor Michalek : > Subject: Re: [PATCH] (repost) sdp: replace mlock with get_user_pages > > On Tue, May 10, 2005 at 05:32:32PM +0300, Michael S. Tsirkin wrote: > > Quoting r. Libor Michalek : > > > On Thu, May 05, 2005 at 02:01:58PM +0300, Michael S. Tsirkin wrote: > > > > > > > > The following patch replaces the mlock hack with call > > > > to get_user_pages. Since the application could have forked > > > > while an iocb is outstanding, when an iocb is done > > > > I do get_user_pages for a second time and copy data if > > > > the physical address has changed. > > > > > > > > Thus, changing ulimit is no longer required to get aio > > > > working, processes are also allowed to fork and to call mlock/munlock > > > > on the buffer. > > > > > > In the latest kernel what happens to a page that's held > > > with a get_user_page reference on fork? Which is the only > > > case left. Since there is an iocv flag parameter it would > > > be prefered to use a bit there instead of is_receive. Also, > > > please leave spaces around '='. Finally it would be nice > > > if we could get rid of having to schedule a task to unlock > > > the iocb in complete, I've noticed this having a negative > > > effect on system CPU utilization on older kernels. > > > > I dont see what would prevent the page > > from being copied on a COW event. Do you? Certainly the patch in 2.6.7 > > that fixed the swap-out problem does not affect COW. > > With regard to avoiding task scheduling, since current code already > > needs this, so we are not breaking anything. > > I put in comments so that we remember to do this research at some point > > in time. > > > > In the patch below the style issues are addressed: whitespace fixed, > > is_receive changed to flag. > > Michael, > > I'm seeing a SDP bug and the only solution I've come up with so far > conflicts directly with your patch. > > The problem. > > When using ttcp.aio.c with buffers below the zcopy threshold, > multiple aio buffers outstanding, and verification turned on either > send and/or receive side very quickly gets a mismatch error. The > mismatch error occurs when the order of buffers returned by > io_getevents() differs from the order in which they are passed to > io_submit(). Here's the params I was using: > > ./ttcp.aio.x -r -s -l 3276 -f m -a 20 -v > ./ttcp.aio.x -t -s -l 3276 -n 50000 -a 20 -f m -v 192.168.0.191 > > The issue. > > There is a race condition between completing IOs synchronously and > asynchronously using aio_complete(). The race condition is the result > of iocb_complete() spawning a work thread to unlock memory and call > aio_complete(). It's possible for a new IO to get serviced synchronously > before the do_iocb_complete() task executes which means that the two IOs > will appear on the event queue in the reverse order of being submitted. > > Ths solution. > > Always call aio_complete() immediately when iocb_complete() is called, > and only spawn the work thread to unlock the memory after the > aio_complete() call. The patch is below. > > The problem is that your patch relies on access to the page tables > occurs before the aio_complete() call. Which is the reverse of what > is in the patch, and is incompatible with calling aio_complete before > scheduling a task... > > -Libor > > @@ -538,7 +517,28 @@ static void do_iocb_complete(void *arg) > */ > int sdp_iocb_complete(struct sdpc_iocb *iocb, ssize_t status) > { > - iocb->status = status; > + int result; > + long value; > + /* > + * release memory > + */ > + result = sdp_iocb_release(iocb); > + if (result < 0) > + sdp_dbg_err("Error <%d> releasing IOCB <%d> memory <%ld>", > + result, iocb->key, iocb->addr); > + /* > + * callback to complete IOCB > + */ > + value = (iocb->post > 0) ? iocb->post : status; > + > + sdp_dbg_data(NULL, "IOCB complete. <%d:%d:%08lx> value <%08lx:%ld>", > + iocb->req->ki_users, iocb->req->ki_key, > + iocb->req->ki_flags, iocb->addr, value); > + /* > + * valid result can be 0 or 1 for complete so > + * we ignore the value. > + */ > + (void)aio_complete(iocb->req, value, 0); > > if (in_atomic() || irqs_disabled()) { > INIT_WORK(&iocb->completion, do_iocb_complete, (void *)iocb); By the way, as was already discussed here, in_atomic() may not catch all cases where its illegal to sleep. If the iocb functions get passed the socket parameter, the iocb can use the conn->lock.users flag to figure out that its OK to sleep. -- MST - Michael S. Tsirkin From mst at mellanox.co.il Sun May 15 01:26:54 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Sun, 15 May 2005 11:26:54 +0300 Subject: [openib-general] [PATCH] libmthca: qp.c trailing space Message-ID: <20050515082654.GD13718@mellanox.co.il> Remove a trailing space. Signed-off-by: Michael S. Tsirkin Index: src/userspace/libmthca/src/qp.c =================================================================== --- src/userspace/libmthca/src/qp.c (revision 2350) +++ src/userspace/libmthca/src/qp.c (working copy) @@ -834,7 +834,7 @@ void mthca_return_cap(struct ibv_pd *pd, cap->max_recv_wr = qp->rq.max; cap->max_send_sge = qp->sq.max_gs; cap->max_recv_sge = qp->rq.max_gs; -} +} struct mthca_qp *mthca_find_qp(struct mthca_context *ctx, uint32_t qpn) { -- MST - Michael S. Tsirkin From mst at mellanox.co.il Sun May 15 02:31:37 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Sun, 15 May 2005 12:31:37 +0300 Subject: [openib-general] change req_notify_cq to return void Message-ID: <20050515093137.GG13718@mellanox.co.il> Hi! According to IB spec, the output modifiers of req_notify_cq are: . Verb Results: . Operation completed successfully. . Invalid HCA handle. . Invalid CQ handle. . Invalid completion notification type. In other words, there's no way req_notify_cq can fail. I propose changing the function return type to void and so save consumers the need to check it. Does this make sense to people here? -- MST - Michael S. Tsirkin From mst at mellanox.co.il Sun May 15 06:16:45 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Sun, 15 May 2005 16:16:45 +0300 Subject: [openib-general] Re: slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects In-Reply-To: <1116026056.4480.6.camel@localhost.localdomain> References: <42765027.10605@ichips.intel.com> <20050502163327.GB9939@mellanox.co.il> <1115066730.4476.139.camel@localhost.localdomain> <52u0llfiot.fsf@topspin.com> <20050503072727.GD9674@mellanox.co.il> <5264y0e49p.fsf@topspin.com> <1115134226.4476.618.camel@localhost.localdomain> <20050504065759.GA9674@mellanox.co.il> <1115398343.4476.154.camel@localhost.localdomain> <1116026056.4480.6.camel@localhost.localdomain> Message-ID: <20050515131645.GM13718@mellanox.co.il> Quoting r. Hal Rosenstock : > Subject: Re: [openib-general] Re: slab error in kmem_cache_destroy(): cache `ib_mad': Can't free all objects > > On Fri, 2005-05-06 at 12:52, Hal Rosenstock wrote: > > On Wed, 2005-05-04 at 02:57, Michael S. Tsirkin wrote: > > > > If this only occurs on machines which are running OpenSM, it could be a > > > > problem with the local MAD handling where ib_free_recv_mad is not called > > > > for some case where it should. I will look into this if this is the > > > > case. > > > > > > > > -- Hal > > > > > > > > > > This is the case for me: the error only occurs on machines which are > > > running OpenSM. > > > > So assuming mthca sets the bits properly for local MAD handling (and I > > have no reason to think otherwise), this is likely a local MAD handling > > error in mad.c where some path does not return an allocation to the MAD > > cache. I inspected the code to try and find it but couldn't. I will need > > to dig deeper. This will take some more time. (I will add some > > accounting in and see what this shows up). Please bear with me. > > This patch works for me. Can you try it out ? If it works for you, I > will check it in. > > -- Hal > Index: mad.c > =================================================================== > -- mad.c (revision 2331) > +++ mad.c (working copy) > @@ -1602,7 +1602,6 @@ > > INIT_LIST_HEAD(&mad_recv_wc->rmpp_list); > list_add(&mad_recv_wc->recv_buf.list, &mad_recv_wc->rmpp_list); > - > if (mad_agent_priv->agent.rmpp_version) { > mad_recv_wc = ib_process_rmpp_recv_wc(mad_agent_priv, > mad_recv_wc); > @@ -2054,6 +2053,8 @@ > > /* Empty wait list to prevent receives from finding a request */ > list_splice_init(&mad_agent_priv->wait_list, &cancel_list); > + /* Empty local completion list as well */ > + list_splice_init(&mad_agent_priv->local_list, &cancel_list); > spin_unlock_irqrestore(&mad_agent_priv->lock, flags); > > /* Report all cancelled requests */ > @@ -2168,6 +2169,7 @@ > struct ib_mad_local_private *local; > struct ib_mad_agent_private *recv_mad_agent; > unsigned long flags; > + int recv = 0; > struct ib_wc wc; > struct ib_mad_send_wc mad_send_wc; > > @@ -2183,10 +2185,10 @@ > recv_mad_agent = local->recv_mad_agent; > if (!recv_mad_agent) { > printk(KERN_ERR PFX "No receive MAD agent for local completion\n"); > - kmem_cache_free(ib_mad_cache, local->mad_priv); > goto local_send_completion; > } > > + recv = 1; > /* > * Defined behavior is to complete response > * before request > @@ -2199,6 +2201,8 @@ > local->mad_priv->header.recv_wc.mad_len = > sizeof(struct ib_mad); > INIT_LIST_HEAD(&local->mad_priv->header.recv_wc.rmpp_list); > + list_add(&local->mad_priv->header.recv_wc.recv_buf.list, > + &local->mad_priv->header.recv_wc.rmpp_list); > local->mad_priv->header.recv_wc.recv_buf.grh = NULL; > local->mad_priv->header.recv_wc.recv_buf.mad = > &local->mad_priv->mad.mad; > @@ -2229,6 +2233,8 @@ > spin_lock_irqsave(&mad_agent_priv->lock, flags); > list_del(&local->completion_list); > atomic_dec(&mad_agent_priv->refcount); > + if (!recv) > + kmem_cache_free(ib_mad_cache, local->mad_priv); > kfree(local); > } > spin_unlock_irqrestore(&mad_agent_priv->lock, flags); > This patch fixes the slab error for me. Please commit. Thanks, -- MST - Michael S. Tsirkin From info at kqsv12.com Sat May 14 23:53:07 2005 From: info at kqsv12.com (info at kqsv12.com) Date: 15 May 2005 15:53:07 +0900 Subject: [openib-general] $B?75,EPO?$NJ}$O(B10000$B1_J,L5NA(B Message-ID: <20050515065307.8137.qmail@mail.kqsv12.com> $B0lK|1_J,!*40A4L5NA$G;HMQ(BOK $B"#:#$J$i4V$K9g$&(BW$B%A%c%s%9!*L5NA%]%$%s%H%"%C%W3Z$7$5(B100$BG\!*"#(B $B"(EPO?=w at -2q0w$N%8%c%s%kJL?M5$%i%s%-%s%0(B $B0l0L!'5U1g!J(B189$BL>EPO?!K(B $BFs0L!'1g8r!J(B429$BL>EPO?!K(B $B;00L!'(BSM$B5U1g!J(B243$BL>EPO?!K(B $B;M0L!'%;%U%l!J(B1038$BL>EPO?!K(B $B8^0L!'ITNQ!J(B2421$BL>EPO?!K(B $BO;0L!'%F%l%(%C%A!J(B3463$BL>EPO?!K(B $B!&(B $B!&(B $B!&(B $B0lK|1_J,40A4L5NA$G$*;n$7$O(B $B"-"-"-(B http://awg.webchu.com/?summer12 $BB~:#!*?75,EPO?$7$?J}$K$OL5NA$G(B10,000$B1_J,$4MxMQ=PMh$^$9!#(B $B!ZL5NA%]%$%s%HFb$G==J,$K%Q!<%H%J!<$rC5$9;v$,=PMh$^$9!#![(B $B"((B18$B:P0J>e$NHkL)87l9g$O2<5-(BURL$B$KJV?.2<$5$$!#(B henshin_senyou at ok.kz From mst at mellanox.co.il Sun May 15 06:57:33 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Sun, 15 May 2005 16:57:33 +0300 Subject: [openib-general] missing check in uverbs? Message-ID: <20050515135733.GQ13718@mellanox.co.il> Hi, Roland! I had a mix of an updated kernel level code (rev 2350) and an outdated userspace library. Running ibv_pingpong client I got the following: Unable to handle kernel NULL pointer dereference at 0000000000000080 RIP: {:ib_mthca:mthca_mmap_uar+48} PGD 14e8ee067 PUD 14e8e7067 PMD 0 Oops: 0000 [1] SMP CPU 1 Modules linked in: ib_uverbs ib_umad ib_mthca ib_mad ib_core Pid: 3297, comm: ibv_pingpong Not tainted 2.6.11-openib RIP: 0010:[] {:ib_mthca:mthca_mmap_uar+48} RSP: 0018:ffff81014f399e88 EFLAGS: 00010286 RAX: 00002aaaab311000 RBX: ffff81015619fcb8 RCX: 0000000000001000 RDX: 00000000ffffffea RSI: 00002aaaab311000 RDI: 0000000000000000 RBP: 0000000000001000 R08: 800000000000003f R09: ffff81015619fcb8 R10: ffff81015619fcb8 R11: 00000000000000b0 R12: 00002aaaab311000 R13: ffff81014e945280 R14: ffff8101573aa4c0 R15: 00000000001000fa FS: 00002aaaab20a0a0(0000) GS:ffffffff80522d00(0000) knlGS:0000000000000000 CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b CR2: 0000000000000080 CR3: 000000014edfe000 CR4: 00000000000006e0 Process ibv_pingpong (pid: 3297, threadinfo ffff81014f398000, task ffff8101555520f0) Stack: ffff81014e945280 ffffffff8802f022 0000000000000292 ffffffff8015f6cb ffff8101573aa4c0 ffff8101555520f0 ffffffff8012de31 ffff810005d26560 0000000000000000 0000000100000001 Call Trace:{:ib_uverbs:ib_uverbs_mmap+32} {do_mmap_pgoff+1328} {finish_task_switch+57} {sys_mmap+142} {system_call+126} Code: 48 8b 97 80 00 00 00 4c 89 cf e8 18 c0 13 f8 83 f8 01 19 d2 RIP {:ib_mthca:mthca_mmap_uar+48} RSP CR2: 0000000000000080 This is on 2.6.11, x86_64. Unfortunately I updated userspace and the problem went away, and I didnt note what version my userspace was (IIRC something from last Wednesday). This seems to indicate some sanity check missing in uverbs, does it not? -- MST - Michael S. Tsirkin From mst at mellanox.co.il Sun May 15 08:45:53 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Sun, 15 May 2005 18:45:53 +0300 Subject: [openib-general] opensm / hot-plug problem Message-ID: <20050515154553.GT13718@mellanox.co.il> Hello! opensm currently keeps multiple files open under /sys/class/infiniband/mthca%d/ports/ In particular, this will prevent the OS from performing device hot-plug removal. In my opinion, it would be better for sm to either avoid working directly with low-level drivers, or close the sysfs files after reading them. The change to close the files is probably easier to implement. Does this make sense to people here? -- MST - Michael S. Tsirkin From Tom.Duffy at Sun.COM Sun May 15 11:27:16 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Sun, 15 May 2005 11:27:16 -0700 Subject: [openib-general] [PATCH][3/19] kDAPL: Remove typedef DAT_QOS In-Reply-To: <11161816352496@sun.com> References: <11161816352658@sun.com> <11161816351137@sun.com> <11161816352496@sun.com> Message-ID: <11161816361781@sun.com> Signed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/dat/dat.h linux-kernel3/dat/dat.h --- linux-kernel2/dat/dat.h 2005-05-13 11:45:56.934001000 -0700 +++ linux-kernel3/dat/dat.h 2005-05-13 12:33:26.944001000 -0700 @@ -412,7 +412,7 @@ typedef u64 DAT_CONN_QUAL; typedef u64 DAT_PORT_QUAL; /* QOS definitions */ -typedef enum dat_qos { +enum dat_qos { DAT_QOS_BEST_EFFORT = 0x00, DAT_QOS_HIGH_THROUGHPUT = 0x01, DAT_QOS_LOW_LATENCY = 0x02, @@ -420,7 +420,7 @@ typedef enum dat_qos { DAT_QOS_ECONOMY = 0x04, /* both low latency and high throughput */ DAT_QOS_PREMIUM = 0x08 -} DAT_QOS; +}; /* * FLAGS @@ -560,7 +560,7 @@ struct dat_rmr_param { struct dat_ep_attr { u64 max_message_size; u64 max_rdma_size; - DAT_QOS qos; + enum dat_qos qos; DAT_COMPLETION_FLAGS recv_completion_flags; DAT_COMPLETION_FLAGS request_completion_flags; int max_recv_dtos; @@ -1064,7 +1064,7 @@ struct dat_provider_attr { u32 dat_version_minor; DAT_MEM_TYPE lmr_mem_types_supported; DAT_IOV_OWNERSHIP iov_ownership_on_return; - DAT_QOS dat_qos_supported; + enum dat_qos dat_qos_supported; DAT_COMPLETION_FLAGS completion_flags_supported; boolean_t is_thread_safe; int max_private_data_size; @@ -1146,10 +1146,12 @@ typedef u32 (*DAT_EP_MODIFY_FUNC)(DAT_EP typedef u32 (*DAT_EP_CONNECT_FUNC)(DAT_EP_HANDLE, struct sockaddr *, DAT_CONN_QUAL, DAT_TIMEOUT, int, - const void *, DAT_QOS, DAT_CONNECT_FLAGS); + const void *, enum dat_qos, + DAT_CONNECT_FLAGS); typedef u32 (*DAT_EP_DUP_CONNECT_FUNC)(DAT_EP_HANDLE, DAT_EP_HANDLE, - DAT_TIMEOUT, int, const void *, DAT_QOS); + DAT_TIMEOUT, int, const void *, + enum dat_qos); typedef u32 (*DAT_EP_DISCONNECT_FUNC)(DAT_EP_HANDLE, DAT_CLOSE_FLAGS); @@ -1494,7 +1496,7 @@ static inline u32 dat_evd_resize(DAT_EVD static inline u32 dat_ep_connect(DAT_EP_HANDLE ep, struct sockaddr *ia_addr, DAT_CONN_QUAL conn_qual, DAT_TIMEOUT timeout, int private_data_size, - const void *private_data, DAT_QOS qos, + const void *private_data, enum dat_qos qos, DAT_CONNECT_FLAGS flags) { return DAT_CALL_PROVIDER_FUNC(ep_connect_func, ep, ia_addr, conn_qual, @@ -1532,7 +1534,7 @@ static inline u32 dat_ep_disconnect(DAT_ static inline u32 dat_ep_dup_connect(DAT_EP_HANDLE ep, DAT_EP_HANDLE dup_ep, DAT_TIMEOUT timeout, int private_data_size, - const void *private_data, DAT_QOS qos) + const void *private_data, enum dat_qos qos) { return DAT_CALL_PROVIDER_FUNC(ep_dup_connect_func, ep, dup_ep, timeout, private_data_size, private_data, qos); diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/dat-provider/dapl_ep_connect.c linux-kernel3/dat-provider/dapl_ep_connect.c --- linux-kernel2/dat-provider/dapl_ep_connect.c 2005-05-13 11:10:25.786002000 -0700 +++ linux-kernel3/dat-provider/dapl_ep_connect.c 2005-05-13 12:33:26.934000000 -0700 @@ -46,7 +46,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han struct sockaddr *remote_ia_address, DAT_CONN_QUAL remote_conn_qual, DAT_TIMEOUT timeout, int private_data_size, const void *private_data, - DAT_QOS qos, DAT_CONNECT_FLAGS connect_flags) + enum dat_qos qos, DAT_CONNECT_FLAGS connect_flags) { DAPL_EP *ep_ptr; DAPL_EP alloc_ep; diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/dat-provider/dapl_ep_dup_connect.c linux-kernel3/dat-provider/dapl_ep_dup_connect.c --- linux-kernel2/dat-provider/dapl_ep_dup_connect.c 2005-05-13 11:10:25.788006000 -0700 +++ linux-kernel3/dat-provider/dapl_ep_dup_connect.c 2005-05-13 12:33:26.936004000 -0700 @@ -71,7 +71,7 @@ */ u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, DAT_EP_HANDLE ep_dup_handle, DAT_TIMEOUT timeout, int private_data_size, - const void *private_data, DAT_QOS qos) + const void *private_data, enum dat_qos qos) { DAPL_EP *ep_dup_ptr; u32 dat_status; diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/dat-provider/dapl.h linux-kernel3/dat-provider/dapl.h --- linux-kernel2/dat-provider/dapl.h 2005-05-13 11:38:08.137002000 -0700 +++ linux-kernel3/dat-provider/dapl.h 2005-05-13 12:33:26.930003000 -0700 @@ -560,14 +560,14 @@ extern u32 dapl_ep_connect(DAT_EP_HANDLE DAT_TIMEOUT timeout, int private_data_size, const void *private_data, - DAT_QOS quality_of_service, + enum dat_qos quality_of_service, DAT_CONNECT_FLAGS connect_flags); extern u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, DAT_EP_HANDLE ep_dup_handle, DAT_TIMEOUT timeout, int private_data_size, const void *private_data, - DAT_QOS quality_of_service); + enum dat_qos quality_of_service); extern u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_handle, DAT_CLOSE_FLAGS close_flags); diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/test/dapltest/cmd/dapl_qos_util.c linux-kernel3/test/dapltest/cmd/dapl_qos_util.c --- linux-kernel2/test/dapltest/cmd/dapl_qos_util.c 2005-04-29 10:19:04.297007000 -0700 +++ linux-kernel3/test/dapltest/cmd/dapl_qos_util.c 2005-05-13 12:36:09.268006000 -0700 @@ -28,11 +28,11 @@ #include "dapl_proto.h" /* - * Parse a QOS argument into a DAT_QOS. + * Parse a QOS argument into an enum dat_qos. * * Returns no errors: defaults to best effort. */ -DAT_QOS +enum dat_qos DT_ParseQoS (char *arg) { if (0 == strcmp (arg, "HT")) diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/test/dapltest/include/dapl_fft_cmd.h linux-kernel3/test/dapltest/include/dapl_fft_cmd.h --- linux-kernel2/test/dapltest/include/dapl_fft_cmd.h 2005-05-11 08:51:30.515003000 -0700 +++ linux-kernel3/test/dapltest/include/dapl_fft_cmd.h 2005-05-13 12:36:28.910012000 -0700 @@ -63,7 +63,7 @@ typedef struct int num_iter; /* -i */ int num_threads; /* -t */ int num_vis; /* -v */ - DAT_QOS ReliabilityLevel; /* -R */ + enum dat_qos ReliabilityLevel; /* -R */ DAT_MEM_TYPE DT_mem_type; /* -M */ } FFT_Cmd_t; diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/test/dapltest/include/dapl_limit_cmd.h linux-kernel3/test/dapltest/include/dapl_limit_cmd.h --- linux-kernel2/test/dapltest/include/dapl_limit_cmd.h 2005-05-05 13:55:19.271974000 -0700 +++ linux-kernel3/test/dapltest/include/dapl_limit_cmd.h 2005-05-13 12:36:53.685003000 -0700 @@ -52,7 +52,7 @@ typedef enum typedef struct { char device_name[256]; /* -D */ - DAT_QOS ReliabilityLevel; /* -R */ + enum dat_qos ReliabilityLevel; /* -R */ uint32_t width; /* -w */ uint32_t debug; /* -d */ uint32_t maximum; /* -m */ diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/test/dapltest/include/dapl_params.h linux-kernel3/test/dapltest/include/dapl_params.h --- linux-kernel2/test/dapltest/include/dapl_params.h 2005-05-11 08:51:30.518006000 -0700 +++ linux-kernel3/test/dapltest/include/dapl_params.h 2005-05-13 12:33:26.960013000 -0700 @@ -61,7 +61,7 @@ typedef struct } u; /* Needed here due to structure of command processing */ - DAT_QOS ReliabilityLevel; + enum dat_qos ReliabilityLevel; struct sockaddr server_netaddr; void * phead; boolean_t local_is_little_endian; diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/test/dapltest/include/dapl_performance_cmd.h linux-kernel3/test/dapltest/include/dapl_performance_cmd.h --- linux-kernel2/test/dapltest/include/dapl_performance_cmd.h 2005-05-05 13:54:46.392978000 -0700 +++ linux-kernel3/test/dapltest/include/dapl_performance_cmd.h 2005-05-13 12:37:40.908007000 -0700 @@ -50,7 +50,7 @@ typedef struct { uint32_t client_is_little_endian; char server_name[NAME_SZ]; /* -s */ char dapl_name[NAME_SZ]; /* -D */ - DAT_QOS qos; + enum dat_qos qos; uint32_t debug; /* -d */ Performance_Mode_Type mode; /* -m */ uint32_t num_iterations; /* -i */ diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/test/dapltest/include/dapl_proto.h linux-kernel3/test/dapltest/include/dapl_proto.h --- linux-kernel2/test/dapltest/include/dapl_proto.h 2005-05-13 11:37:44.339003000 -0700 +++ linux-kernel3/test/dapltest/include/dapl_proto.h 2005-05-13 12:38:11.981004000 -0700 @@ -533,7 +533,7 @@ const char *DT_EventToSTr (DAT_EVEN const char *DT_State2Str (DAT_EP_STATE state_code); -DAT_QOS DT_ParseQoS (char *arg); +enum dat_qos DT_ParseQoS (char *arg); DAT_MEM_TYPE DT_ParseMemType (char *arg); diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/test/dapltest/include/dapl_quit_cmd.h linux-kernel3/test/dapltest/include/dapl_quit_cmd.h --- linux-kernel2/test/dapltest/include/dapl_quit_cmd.h 2005-05-05 13:55:36.851973000 -0700 +++ linux-kernel3/test/dapltest/include/dapl_quit_cmd.h 2005-05-13 12:33:26.970006000 -0700 @@ -34,7 +34,7 @@ typedef struct char server_name[256]; /* -s */ char device_name[256]; /* -D */ uint32_t debug; /* -d */ - DAT_QOS ReliabilityLevel; /* -R */ + enum dat_qos ReliabilityLevel; /* -R */ DAT_MEM_TYPE DT_mem_type; /* -M */ } Quit_Cmd_t; #pragma pack () diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/test/dapltest/include/dapl_server_cmd.h linux-kernel3/test/dapltest/include/dapl_server_cmd.h --- linux-kernel2/test/dapltest/include/dapl_server_cmd.h 2005-05-11 08:52:07.427003000 -0700 +++ linux-kernel3/test/dapltest/include/dapl_server_cmd.h 2005-05-13 12:38:35.892014000 -0700 @@ -32,9 +32,9 @@ typedef struct { - boolean_t debug; /* -d */ + boolean_t debug; /* -d */ char dapl_name[256]; /* -D device name */ - DAT_QOS ReliabilityLevel; /* -R */ + enum dat_qos ReliabilityLevel; /* -R */ DAT_MEM_TYPE DT_mem_type; /* -M */ } Server_Cmd_t; #pragma pack () diff -Nurp -X /home/tduffy/dontdiff linux-kernel2/test/dapltest/include/dapl_transaction_cmd.h linux-kernel3/test/dapltest/include/dapl_transaction_cmd.h --- linux-kernel2/test/dapltest/include/dapl_transaction_cmd.h 2005-05-05 13:53:14.500974000 -0700 +++ linux-kernel3/test/dapltest/include/dapl_transaction_cmd.h 2005-05-13 12:33:26.976005000 -0700 @@ -55,7 +55,7 @@ typedef struct { uint32_t validate; /* -V */ uint32_t poll; /* -P */ char dapl_name[NAME_SZ]; /* -D */ - DAT_QOS ReliabilityLevel; + enum dat_qos ReliabilityLevel; uint32_t num_ops; Transaction_Cmd_Op_t op[MAX_OPS]; DAT_MEM_TYPE DT_mem_type; /* -M */ From Tom.Duffy at Sun.COM Sun May 15 11:27:15 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Sun, 15 May 2005 11:27:15 -0700 Subject: [openib-general] [PATCH][1/19] kDAPL: Remove typedef DAT_IA_ADDRESS_PTR In-Reply-To: <11161816352658@sun.com> References: <11161816352658@sun.com> Message-ID: <11161816351137@sun.com> Signed-off-by: Tom Duffy Index: linux-kernel/test/dapltest/test/dapl_performance_util.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_performance_util.c (revision 2338) +++ linux-kernel/test/dapltest/test/dapl_performance_util.c (working copy) @@ -34,7 +34,7 @@ boolean_t DT_Performance_Test_Create ( Per_Test_Data_t *pt_ptr, DAT_IA_HANDLE *ia_handle, - DAT_IA_ADDRESS_PTR remote_ia_addr, + struct sockaddr * remote_ia_addr, boolean_t is_server, boolean_t is_remote_little_endian, Performance_Test_t **perf_test) Index: linux-kernel/test/dapltest/test/dapl_performance_client.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_performance_client.c (revision 2338) +++ linux-kernel/test/dapltest/test/dapl_performance_client.c (working copy) @@ -35,7 +35,7 @@ DT_Performance_Test_Client ( Params_t *params_ptr, Per_Test_Data_t *pt_ptr, DAT_IA_HANDLE *ia_handle, - DAT_IA_ADDRESS_PTR remote_ia_addr) + struct sockaddr * remote_ia_addr) { Performance_Test_t *test_ptr = NULL; int connected = 1; Index: linux-kernel/test/dapltest/test/dapl_transaction_test.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_transaction_test.c (revision 2338) +++ linux-kernel/test/dapltest/test/dapl_transaction_test.c (working copy) @@ -48,7 +48,7 @@ int DT_Transaction_Test_Client (Per_Test_Data_t * pt_ptr, DAT_IA_HANDLE ia_handle, - DAT_IA_ADDRESS_PTR remote_ia_addr) + struct sockaddr *remote_ia_addr) { Transaction_Cmd_t *cmd = &pt_ptr->Params.u.Transaction_Cmd; unsigned int i; @@ -114,7 +114,7 @@ DT_Transaction_Test_Server (void *params TRUE, port_num, pt_ptr->Client_Info.is_little_endian, - (DAT_IA_ADDRESS_PTR) 0)) + (struct sockaddr *) 0)) { DT_Tdep_PT_Printf (phead, "Server: Cannot Create Test!\n"); status = 1; @@ -162,7 +162,7 @@ DT_Transaction_Create_Test (Per_Test_Dat boolean_t is_server, unsigned int port_num, boolean_t remote_is_little_endian, - DAT_IA_ADDRESS_PTR remote_ia_addr) + struct sockaddr *remote_ia_addr) { Transaction_Test_t *test_ptr; DT_Tdep_Print_Head *phead; Index: linux-kernel/test/dapltest/test/dapl_performance_server.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_performance_server.c (revision 2338) +++ linux-kernel/test/dapltest/test/dapl_performance_server.c (working copy) @@ -42,7 +42,7 @@ DT_Performance_Test_Server ( if ( !DT_Performance_Test_Create (pt_ptr, pt_ptr->ps_ptr->ia_handle, - (DAT_IA_ADDRESS_PTR) 0, + (struct sockaddr *) 0, TRUE, pt_ptr->Client_Info.is_little_endian, &test_ptr) ) Index: linux-kernel/test/dapltest/test/dapl_client.c =================================================================== --- linux-kernel/test/dapltest/test/dapl_client.c (revision 2338) +++ linux-kernel/test/dapltest/test/dapl_client.c (working copy) @@ -53,7 +53,7 @@ DT_cs_Client (Params_t * params_ptr, Quit_Cmd_t *Quit_Cmd = NULL; Performance_Cmd_t *Performance_Cmd = NULL; Bpool *bpool = NULL; - DAT_IA_ADDRESS_PTR server_netaddr = NULL; + struct sockaddr * server_netaddr = NULL; unsigned char *module = "DT_cs_Client"; unsigned int did_connect = 0; unsigned int try_connect = 0; Index: linux-kernel/test/dapltest/include/dapl_performance_test.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_performance_test.h (revision 2338) +++ linux-kernel/test/dapltest/include/dapl_performance_test.h (working copy) @@ -66,7 +66,7 @@ typedef struct { Per_Test_Data_t *pt_ptr; Performance_Cmd_t *cmd; - DAT_IA_ADDRESS_PTR remote_ia_addr; + struct sockaddr * remote_ia_addr; boolean_t is_remote_little_endian; DAT_CONN_QUAL base_port; struct dat_ia_attr ia_attr; Index: linux-kernel/test/dapltest/include/dapl_proto.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_proto.h (revision 2338) +++ linux-kernel/test/dapltest/include/dapl_proto.h (working copy) @@ -208,13 +208,13 @@ void DT_MemListFree (Per_Test void DT_PrintMemList (Per_Test_Data_t * pt_ptr); /* dapl_netaddr.c */ -boolean_t DT_NetAddrLookupHostAddress (DAT_IA_ADDRESS_PTR to_netaddr, +boolean_t DT_NetAddrLookupHostAddress (struct sockaddr *to_netaddr, char *hostname); -DAT_IA_ADDRESS_PTR DT_NetAddrAlloc (Per_Test_Data_t * pt_ptr); +struct sockaddr *DT_NetAddrAlloc (Per_Test_Data_t * pt_ptr); void DT_NetAddrFree (Per_Test_Data_t * pt_ptr, - DAT_IA_ADDRESS_PTR netaddr); + struct sockaddr *netaddr); /* dapl_params.c */ boolean_t DT_Params_Parse (int argc, @@ -241,7 +241,7 @@ void DT_Performance_Cmd_Endia int DT_Performance_Test_Client ( Params_t *params_ptr, Per_Test_Data_t * pt_ptr, DAT_IA_HANDLE * ia_handle, - DAT_IA_ADDRESS_PTR remote); + struct sockaddr *remote); boolean_t DT_Performance_Test_Client_Connect ( DT_Tdep_Print_Head *phead, @@ -266,7 +266,7 @@ boolean_t DT_Performance_Test /* dapl_performance_util.c */ boolean_t DT_Performance_Test_Create (Per_Test_Data_t * pt_ptr, DAT_IA_HANDLE * ia_handle, - DAT_IA_ADDRESS_PTR remote_ia_addr, + struct sockaddr *remote_ia_addr, boolean_t is_server, boolean_t is_remote_little_endian, Performance_Test_t **perf_test); @@ -465,7 +465,7 @@ void DT_Transaction_Cmd_Endia /* dapl_transaction_test.c */ int DT_Transaction_Test_Client (Per_Test_Data_t * pt_ptr, DAT_IA_HANDLE ia_handle, - DAT_IA_ADDRESS_PTR remote); + struct sockaddr *remote); void DT_Transaction_Test_Server (void *params); @@ -474,7 +474,7 @@ boolean_t DT_Transaction_Crea boolean_t is_server, unsigned int port_num, boolean_t remote_is_little_endian, - DAT_IA_ADDRESS_PTR remote_ia_addr); + struct sockaddr *remote_ia_addr); void DT_Transaction_Main (void *param); boolean_t DT_Transaction_Run (DT_Tdep_Print_Head* phead, Index: linux-kernel/test/dapltest/include/dapl_fft_util.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_fft_util.h (revision 2338) +++ linux-kernel/test/dapltest/include/dapl_fft_util.h (working copy) @@ -66,7 +66,7 @@ typedef struct Bpool *bpool; struct dat_cr_arrival_event_data cr_stat; DAT_EVENT_NUMBER event_num; - DAT_IA_ADDRESS_PTR remote_netaddr; + struct sockaddr *remote_netaddr; Per_Test_Data_t *pt_ptr; boolean_t connected; } FFT_Connection_t; Index: linux-kernel/test/dapltest/include/dapl_transaction_test.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_transaction_test.h (revision 2338) +++ linux-kernel/test/dapltest/include/dapl_transaction_test.h (working copy) @@ -77,7 +77,7 @@ typedef struct Per_Test_Data_t *pt_ptr; DAT_IA_HANDLE ia_handle; Transaction_Cmd_t *cmd; - DAT_IA_ADDRESS_PTR remote_ia_addr; + struct sockaddr * remote_ia_addr; DAT_CONN_QUAL base_port; DAT_TIMEOUT time_out; int evd_length; Index: linux-kernel/test/dapltest/cmd/dapl_netaddr.c =================================================================== --- linux-kernel/test/dapltest/cmd/dapl_netaddr.c (revision 2338) +++ linux-kernel/test/dapltest/cmd/dapl_netaddr.c (working copy) @@ -27,12 +27,12 @@ #include "dapl_proto.h" -DAT_IA_ADDRESS_PTR +struct sockaddr * DT_NetAddrAlloc (Per_Test_Data_t * pt_ptr) { - DAT_IA_ADDRESS_PTR netaddr; + struct sockaddr *netaddr; - netaddr = (DAT_IA_ADDRESS_PTR) DT_Mdep_Malloc (sizeof (struct sockaddr)); + netaddr = (struct sockaddr *) DT_Mdep_Malloc (sizeof (struct sockaddr)); if (!netaddr) { DT_Mdep_printf ("dapltest: No Memory to create netaddr!\n"); @@ -42,14 +42,14 @@ DT_NetAddrAlloc (Per_Test_Data_t * pt_pt void -DT_NetAddrFree (Per_Test_Data_t * pt_ptr, DAT_IA_ADDRESS_PTR netaddr) +DT_NetAddrFree (Per_Test_Data_t * pt_ptr, struct sockaddr *netaddr) { DT_Mdep_Free (netaddr); } boolean_t -DT_NetAddrLookupHostAddress (DAT_IA_ADDRESS_PTR to_netaddr, +DT_NetAddrLookupHostAddress (struct sockaddr * to_netaddr, char * hostname) { struct addrinfo *target; @@ -137,7 +137,7 @@ DT_NetAddrLookupHostAddress (DAT_IA_ADDR (rval >> 16) & 0xff, (rval >> 24) & 0xff); - *to_netaddr = * ((DAT_IA_ADDRESS_PTR) target->ai_addr); + *to_netaddr = * ((struct sockaddr *) target->ai_addr); return ( TRUE ); } Index: linux-kernel/dat-provider/dapl_ep_connect.c =================================================================== --- linux-kernel/dat-provider/dapl_ep_connect.c (revision 2338) +++ linux-kernel/dat-provider/dapl_ep_connect.c (working copy) @@ -43,7 +43,7 @@ * side of a connection */ u32 dapl_ep_connect(DAT_EP_HANDLE ep_handle, - DAT_IA_ADDRESS_PTR remote_ia_address, + struct sockaddr *remote_ia_address, DAT_CONN_QUAL remote_conn_qual, DAT_TIMEOUT timeout, int private_data_size, const void *private_data, DAT_QOS qos, DAT_CONNECT_FLAGS connect_flags) Index: linux-kernel/dat-provider/dapl_cr_callback.c =================================================================== --- linux-kernel/dat-provider/dapl_cr_callback.c (revision 2338) +++ linux-kernel/dat-provider/dapl_cr_callback.c (working copy) @@ -286,8 +286,8 @@ u32 dapl_connection_request(ib_cm_handle */ cr_ptr->sp_ptr = sp_ptr; /* maintain sp_ptr in case of reject */ cr_ptr->ib_cm_handle = ib_cm_handle; - cr_ptr->param.remote_ia_address_ptr = - (DAT_IA_ADDRESS_PTR) &cr_ptr->remote_ia_address; + cr_ptr->param.remote_ia_address_ptr = + (struct sockaddr *)&cr_ptr->remote_ia_address; cr_ptr->param.remote_port_qual = 0; /* Private data size will be determined by the provider layer */ @@ -349,8 +349,8 @@ u32 dapl_connection_request(ib_cm_handle dat_status = dapl_evd_post_cr_arrival_event(evd_ptr, DAT_CONNECTION_REQUEST_EVENT, sp_handle, - (DAT_IA_ADDRESS_PTR) & - sp_ptr->header.owner_ia-> + (struct sockaddr *) + &sp_ptr->header.owner_ia-> hca_ptr->hca_address, sp_ptr->conn_qual, (DAT_CR_HANDLE) cr_ptr); Index: linux-kernel/dat-provider/dapl_ep_create.c =================================================================== --- linux-kernel/dat-provider/dapl_ep_create.c (revision 2338) +++ linux-kernel/dat-provider/dapl_ep_create.c (working copy) @@ -204,9 +204,9 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand ep_ptr->param.ia_handle = ia_handle; ep_ptr->param.ep_state = DAT_EP_STATE_UNCONNECTED; ep_ptr->param.local_ia_address_ptr = - (DAT_IA_ADDRESS_PTR) & ia_ptr->hca_ptr->hca_address; + (struct sockaddr *)&ia_ptr->hca_ptr->hca_address; /* Set the remote address pointer to the end of the EP struct */ - ep_ptr->param.remote_ia_address_ptr = (DAT_IA_ADDRESS_PTR) (ep_ptr + 1); + ep_ptr->param.remote_ia_address_ptr = (struct sockaddr *) (ep_ptr + 1); ep_ptr->param.pz_handle = pz_handle; ep_ptr->param.recv_evd_handle = recv_evd_handle; Index: linux-kernel/dat-provider/dapl_adapter_util.h =================================================================== --- linux-kernel/dat-provider/dapl_adapter_util.h (revision 2338) +++ linux-kernel/dat-provider/dapl_adapter_util.h (working copy) @@ -67,7 +67,7 @@ u32 dapl_ib_qp_modify(DAPL_IA *ia_ptr, D struct dat_ep_attr *ep_attr); u32 dapl_ib_connect(DAT_EP_HANDLE ep_handle, - DAT_IA_ADDRESS_PTR remote_ia_address, + struct sockaddr *remote_ia_address, DAT_CONN_QUAL remote_conn_qual, int private_data_size, void *private_data); Index: linux-kernel/dat-provider/dapl_ep_query.c =================================================================== --- linux-kernel/dat-provider/dapl_ep_query.c (revision 2338) +++ linux-kernel/dat-provider/dapl_ep_query.c (working copy) @@ -94,7 +94,7 @@ u32 dapl_ep_query(DAT_EP_HANDLE ep_handl &ep_ptr->remote_ia_address); } ep_ptr->param.remote_ia_address_ptr = - (DAT_IA_ADDRESS_PTR) & ep_ptr->remote_ia_address; + (struct sockaddr *)&ep_ptr->remote_ia_address; *ep_param = ep_ptr->param; bail: Index: linux-kernel/dat-provider/dapl_openib_cm.c =================================================================== --- linux-kernel/dat-provider/dapl_openib_cm.c (revision 2338) +++ linux-kernel/dat-provider/dapl_openib_cm.c (working copy) @@ -393,7 +393,7 @@ error: * */ u32 dapl_ib_connect(DAT_EP_HANDLE ep_handle, - DAT_IA_ADDRESS_PTR remote_ia_address, + struct sockaddr *remote_ia_address, DAT_CONN_QUAL remote_conn_qual, int private_data_size, void *private_data) { Index: linux-kernel/dat-provider/dapl_openib_util.c =================================================================== --- linux-kernel/dat-provider/dapl_openib_util.c (revision 2338) +++ linux-kernel/dat-provider/dapl_openib_util.c (working copy) @@ -722,7 +722,7 @@ u32 dapl_ib_query_hca(DAPL_HCA *hca_ptr, VN_VENDOR_NAME, DAT_NAME_MAX_LENGTH); ia_attr->ia_address_ptr = - (DAT_IA_ADDRESS_PTR) & hca_ptr->hca_address; + (struct sockaddr *)&hca_ptr->hca_address; ia_attr->hardware_version_major = device_attr.hw_ver; ia_attr->hardware_version_minor = 0; ia_attr->firmware_version_major = device_attr.fw_ver; Index: linux-kernel/dat-provider/dapl_ep_create_with_srq.c =================================================================== --- linux-kernel/dat-provider/dapl_ep_create_with_srq.c (revision 2338) +++ linux-kernel/dat-provider/dapl_ep_create_with_srq.c (working copy) @@ -231,9 +231,9 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL ep_ptr->param.ia_handle = ia_handle; ep_ptr->param.ep_state = DAT_EP_STATE_UNCONNECTED; ep_ptr->param.local_ia_address_ptr = - (DAT_IA_ADDRESS_PTR) & ia_ptr->hca_ptr->hca_address; + (struct sockaddr *)&ia_ptr->hca_ptr->hca_address; /* Set the remote address pointer to the end of the EP struct */ - ep_ptr->param.remote_ia_address_ptr = (DAT_IA_ADDRESS_PTR) (ep_ptr + 1); + ep_ptr->param.remote_ia_address_ptr = (struct sockaddr *) (ep_ptr + 1); ep_ptr->param.pz_handle = pz_handle; ep_ptr->param.recv_evd_handle = recv_evd_handle; Index: linux-kernel/dat-provider/dapl_evd_util.c =================================================================== --- linux-kernel/dat-provider/dapl_evd_util.c (revision 2338) +++ linux-kernel/dat-provider/dapl_evd_util.c (working copy) @@ -500,7 +500,7 @@ static struct dat_event *dapl_evd_get_an u32 dapl_evd_post_cr_arrival_event(DAPL_EVD *evd_ptr, DAT_EVENT_NUMBER event_number, DAT_SP_HANDLE sp_handle, - DAT_IA_ADDRESS_PTR ia_address_ptr, + struct sockaddr *ia_address_ptr, DAT_CONN_QUAL conn_qual, DAT_CR_HANDLE cr_handle) { Index: linux-kernel/dat-provider/dapl_evd_util.h =================================================================== --- linux-kernel/dat-provider/dapl_evd_util.h (revision 2338) +++ linux-kernel/dat-provider/dapl_evd_util.h (working copy) @@ -65,7 +65,7 @@ u32 dapl_evd_event_realloc(DAPL_EVD *evd u32 dapl_evd_post_cr_arrival_event(DAPL_EVD *evd_ptr, DAT_EVENT_NUMBER event_number, DAT_SP_HANDLE sp_handle, - DAT_IA_ADDRESS_PTR ia_address_ptr, + struct sockaddr *ia_address_ptr, DAT_CONN_QUAL conn_qual, DAT_CR_HANDLE cr_handle); Index: linux-kernel/dat-provider/dapl_ep_dup_connect.c =================================================================== --- linux-kernel/dat-provider/dapl_ep_dup_connect.c (revision 2338) +++ linux-kernel/dat-provider/dapl_ep_dup_connect.c (working copy) @@ -75,7 +75,7 @@ u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep { DAPL_EP *ep_dup_ptr; u32 dat_status; - DAT_IA_ADDRESS_PTR remote_ia_address_ptr; + struct sockaddr *remote_ia_address_ptr; DAT_CONN_QUAL remote_conn_qual; ep_dup_ptr = (DAPL_EP *) ep_dup_handle; Index: linux-kernel/dat-provider/dapl.h =================================================================== --- linux-kernel/dat-provider/dapl.h (revision 2338) +++ linux-kernel/dat-provider/dapl.h (working copy) @@ -555,7 +555,7 @@ extern u32 dapl_ep_modify(DAT_EP_HANDLE const struct dat_ep_param *ep_args); extern u32 dapl_ep_connect(DAT_EP_HANDLE ep_handle, - DAT_IA_ADDRESS_PTR remote_ia_address, + struct sockaddr *remote_ia_address, DAT_CONN_QUAL remote_conn_qual, DAT_TIMEOUT timeout, int private_data_size, Index: linux-kernel/dat/dat.h =================================================================== --- linux-kernel/dat/dat.h (revision 2338) +++ linux-kernel/dat/dat.h (working copy) @@ -408,8 +408,6 @@ typedef DAT_HANDLE DAT_SRQ_HANDLE; #define DAT_HANDLE_NULL ((DAT_HANDLE)NULL) -typedef struct sockaddr *DAT_IA_ADDRESS_PTR; - typedef u64 DAT_CONN_QUAL; typedef u64 DAT_PORT_QUAL; @@ -603,9 +601,9 @@ typedef enum dat_ep_state { struct dat_ep_param { DAT_IA_HANDLE ia_handle; DAT_EP_STATE ep_state; - DAT_IA_ADDRESS_PTR local_ia_address_ptr; + struct sockaddr *local_ia_address_ptr; DAT_PORT_QUAL local_port_qual; - DAT_IA_ADDRESS_PTR remote_ia_address_ptr; + struct sockaddr *remote_ia_address_ptr; DAT_PORT_QUAL remote_port_qual; DAT_PZ_HANDLE pz_handle; DAT_EVD_HANDLE recv_evd_handle; @@ -710,7 +708,7 @@ struct dat_rsp_param { struct dat_cr_param { /* Remote IA whose Endpoint requested the connection. */ - DAT_IA_ADDRESS_PTR remote_ia_address_ptr; + struct sockaddr *remote_ia_address_ptr; /* Port qualifier of the remote Endpoint of the requested connection. */ DAT_PORT_QUAL remote_port_qual; @@ -795,7 +793,7 @@ struct dat_cr_arrival_event_data { DAT_SP_HANDLE sp_handle; /* Address of the IA on which the Connection Request arrived. */ - DAT_IA_ADDRESS_PTR local_ia_address_ptr; + struct sockaddr *local_ia_address_ptr; /* Connection Qualifier of the IA on which the Service Point received a * Connection Request. @@ -1002,7 +1000,7 @@ struct dat_ia_attr { u32 hardware_version_minor; u32 firmware_version_major; u32 firmware_version_minor; - DAT_IA_ADDRESS_PTR ia_address_ptr; + struct sockaddr *ia_address_ptr; int max_eps; int max_dto_per_ep; int max_rdma_read_per_ep_in; @@ -1149,7 +1147,7 @@ typedef u32 (*DAT_EP_QUERY_FUNC)(DAT_EP_ typedef u32 (*DAT_EP_MODIFY_FUNC)(DAT_EP_HANDLE, DAT_EP_PARAM_MASK, const struct dat_ep_param *); -typedef u32 (*DAT_EP_CONNECT_FUNC)(DAT_EP_HANDLE, DAT_IA_ADDRESS_PTR, +typedef u32 (*DAT_EP_CONNECT_FUNC)(DAT_EP_HANDLE, struct sockaddr *, DAT_CONN_QUAL, DAT_TIMEOUT, int, const void *, DAT_QOS, DAT_CONNECT_FLAGS); @@ -1496,7 +1494,7 @@ static inline u32 dat_evd_resize(DAT_EVD return DAT_CALL_PROVIDER_FUNC(evd_resize_func, evd, qlen); } -static inline u32 dat_ep_connect(DAT_EP_HANDLE ep, DAT_IA_ADDRESS_PTR ia_addr, +static inline u32 dat_ep_connect(DAT_EP_HANDLE ep, struct sockaddr *ia_addr, DAT_CONN_QUAL conn_qual, DAT_TIMEOUT timeout, int private_data_size, const void *private_data, DAT_QOS qos, From Tom.Duffy at Sun.COM Sun May 15 11:27:15 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Sun, 15 May 2005 11:27:15 -0700 Subject: [openib-general] [PATCH][0/19] kDAPL: More cleanups in dat.h Message-ID: <11161816352658@sun.com> The following set of patches further cleans up dat/dat.h. Most of them remove the use of typedef for the enums defined in dat/dat.h. Please apply in order. Signed-off-by: Tom Duffy Your friendly openib janitor. From Tom.Duffy at Sun.COM Sun May 15 11:27:15 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Sun, 15 May 2005 11:27:15 -0700 Subject: [openib-general] [PATCH][2/19] kDAPL: Remove typedefs DAT_VLEN & DAT_VADDR In-Reply-To: <11161816351137@sun.com> References: <11161816352658@sun.com> <11161816351137@sun.com> Message-ID: <11161816352496@sun.com> Signed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat/dat.h linux-kernel2/dat/dat.h --- linux-kernel/dat/dat.h 2005-05-13 11:13:02.563002000 -0700 +++ linux-kernel2/dat/dat.h 2005-05-15 10:35:47.731003000 -0700 @@ -503,9 +503,6 @@ typedef enum dat_psp_flags { typedef u32 DAT_LMR_CONTEXT; typedef u32 DAT_RMR_CONTEXT; -typedef u64 DAT_VLEN; -typedef u64 DAT_VADDR; - /* It is legal for the Consumer to specify zero for segment_length * of the dat_lmr_triplet. When 0 is specified for the * segment_length then the other two elements of the @@ -515,15 +512,15 @@ typedef u64 DAT_VADDR; struct dat_lmr_triplet { DAT_LMR_CONTEXT lmr_context; u32 pad; /* align structure on 64-bit boundry */ - DAT_VADDR virtual_address; - DAT_VLEN segment_length; + u64 virtual_address; + u64 segment_length; }; struct dat_rmr_triplet { DAT_RMR_CONTEXT rmr_context; u32 pad; /* align structure on 64-bit boundry */ - DAT_VADDR target_address; - DAT_VLEN segment_length; + u64 target_address; + u64 segment_length; }; typedef enum dat_mem_type { @@ -561,8 +558,8 @@ struct dat_rmr_param { /* Endpoint attributes */ struct dat_ep_attr { - DAT_VLEN max_message_size; - DAT_VLEN max_rdma_size; + u64 max_message_size; + u64 max_rdma_size; DAT_QOS qos; DAT_COMPLETION_FLAGS recv_completion_flags; DAT_COMPLETION_FLAGS request_completion_flags; @@ -767,7 +764,7 @@ struct dat_dto_completion_event_data { DAT_EP_HANDLE ep_handle; DAT_DTO_COOKIE user_cookie; DAT_DTO_COMPLETION_STATUS status; - DAT_VLEN transfered_length; + u64 transfered_length; }; /* RMR bind completion event data */ @@ -983,13 +980,13 @@ struct dat_lmr_param { DAT_IA_HANDLE ia_handle; DAT_MEM_TYPE mem_type; DAT_REGION_DESCRIPTION region_desc; - DAT_VLEN length; + u64 length; DAT_PZ_HANDLE pz_handle; DAT_MEM_PRIV_FLAGS mem_priv; DAT_LMR_CONTEXT lmr_context; DAT_RMR_CONTEXT rmr_context; - DAT_VLEN registered_size; - DAT_VADDR registered_address; + u64 registered_size; + u64 registered_address; }; /* General Interface Adapter attributes.*/ @@ -1009,13 +1006,13 @@ struct dat_ia_attr { int max_evd_qlen; int max_iov_segments_per_dto; int max_lmrs; - DAT_VLEN max_lmr_block_size; - DAT_VADDR max_lmr_virtual_address; + u64 max_lmr_block_size; + u64 max_lmr_virtual_address; int max_pzs; - DAT_VLEN max_message_size; - DAT_VLEN max_rdma_size; + u64 max_message_size; + u64 max_rdma_size; int max_rmrs; - DAT_VADDR max_rmr_target_address; + u64 max_rmr_target_address; int max_srqs; int max_ep_per_srq; int max_recv_per_srq; @@ -1193,11 +1190,11 @@ typedef u32 (*DAT_LMR_QUERY_FUNC)(DAT_LM typedef u32 (*DAT_LMR_SYNC_RDMA_READ_FUNC)(DAT_IA_HANDLE, const struct dat_lmr_triplet *, - DAT_VLEN); + u64); typedef u32 (*DAT_LMR_SYNC_RDMA_WRITE_FUNC)(DAT_IA_HANDLE, const struct dat_lmr_triplet *, - DAT_VLEN); + u64); typedef u32 (*DAT_RMR_CREATE_FUNC)(DAT_PZ_HANDLE, DAT_RMR_HANDLE *); @@ -1249,15 +1246,15 @@ typedef u32 (*DAT_SRQ_POST_RECV_FUNC)(DA struct dat_lmr_triplet *, DAT_DTO_COOKIE); typedef u32 (*DAT_LMR_KCREATE_FUNC)(DAT_IA_HANDLE, DAT_MEM_TYPE, - DAT_REGION_DESCRIPTION, DAT_VLEN, + DAT_REGION_DESCRIPTION, u64, DAT_PZ_HANDLE, DAT_MEM_PRIV_FLAGS, DAT_MEM_OPTIMIZE_FLAGS, DAT_LMR_HANDLE *, DAT_LMR_CONTEXT *, DAT_RMR_CONTEXT *, - DAT_VLEN *, DAT_VADDR *); + u64 *, u64 *); -typedef u32 (*DAT_IA_MEMTYPE_HINT_FUNC)(DAT_IA_HANDLE, DAT_MEM_TYPE, DAT_VLEN, - DAT_MEM_OPTIMIZE_FLAGS, DAT_VLEN *, - DAT_VADDR *); +typedef u32 (*DAT_IA_MEMTYPE_HINT_FUNC)(DAT_IA_HANDLE, DAT_MEM_TYPE, u64, + DAT_MEM_OPTIMIZE_FLAGS, u64 *, + u64 *); typedef u32 (*DAT_EVD_KCREATE_FUNC) (DAT_IA_HANDLE, int, DAT_UPCALL_POLICY, const struct dat_upcall_object *, @@ -1411,10 +1408,10 @@ static inline u32 dat_get_handle_type(DA } static inline u32 dat_ia_memtype_hint(DAT_IA_HANDLE ia, DAT_MEM_TYPE mem_type, - DAT_VLEN length, + u64 length, DAT_MEM_OPTIMIZE_FLAGS mem_optimize, - DAT_VLEN *preferred_length, - DAT_VADDR *preferred_alignment) + u64 *preferred_length, + u64 *preferred_alignment) { return DAT_CALL_PROVIDER_FUNC(ia_memtype_hint_func, ia, mem_type, length, mem_optimize, preferred_length, @@ -1635,7 +1632,7 @@ static inline u32 dat_lmr_free(DAT_LMR_H static inline u32 dat_lmr_sync_rdma_read(DAT_IA_HANDLE ia, const struct dat_lmr_triplet *iovs, - DAT_VLEN num_iovs) + u64 num_iovs) { return DAT_CALL_PROVIDER_FUNC(lmr_sync_rdma_read_func, ia, iovs, num_iovs); @@ -1643,7 +1640,7 @@ static inline u32 dat_lmr_sync_rdma_read static inline u32 dat_lmr_sync_rdma_write(DAT_IA_HANDLE ia, const struct dat_lmr_triplet *iovs, - DAT_VLEN num_iovs) + u64 num_iovs) { return DAT_CALL_PROVIDER_FUNC(lmr_sync_rdma_write_func, ia, iovs, num_iovs); @@ -1770,15 +1767,15 @@ static inline u32 dat_srq_set_lw(DAT_SRQ } static inline u32 dat_lmr_kcreate(DAT_IA_HANDLE ia, DAT_MEM_TYPE type, - DAT_REGION_DESCRIPTION region, DAT_VLEN len, + DAT_REGION_DESCRIPTION region, u64 len, DAT_PZ_HANDLE pz, DAT_MEM_PRIV_FLAGS privileges, DAT_MEM_OPTIMIZE_FLAGS optimization, DAT_LMR_HANDLE *lmr_handle, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, - DAT_VLEN *registered_length, - DAT_VADDR *registered_address) + u64 *registered_length, + u64 *registered_address) { return DAT_CALL_PROVIDER_FUNC(lmr_kcreate_func, ia, type, region, len, pz, privileges, optimization, lmr_handle, diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_adapter_util.h linux-kernel2/dat-provider/dapl_adapter_util.h --- linux-kernel/dat-provider/dapl_adapter_util.h 2005-05-13 11:10:25.781004000 -0700 +++ linux-kernel2/dat-provider/dapl_adapter_util.h 2005-05-13 11:38:08.141004000 -0700 @@ -103,10 +103,10 @@ u32 dapl_ib_pd_alloc(DAPL_IA *ia_ptr, DA u32 dapl_ib_pd_free(DAPL_PZ *pz); u32 dapl_ib_mr_register(DAPL_IA *ia_ptr, DAPL_LMR *lmr, void *virt_addr, - DAT_VLEN length, DAT_MEM_PRIV_FLAGS privileges); + u64 length, DAT_MEM_PRIV_FLAGS privileges); u32 dapl_ib_mr_register_physical(DAPL_IA *ia_ptr, DAPL_LMR *lmr, - void *phys_addr, DAT_VLEN length, + void *phys_addr, u64 length, DAT_MEM_PRIV_FLAGS privileges); u32 dapl_ib_mr_deregister(DAPL_LMR *lmr); @@ -119,8 +119,8 @@ u32 dapl_ib_mw_alloc(DAPL_RMR *rmr); u32 dapl_ib_mw_free(DAPL_RMR *rmr); u32 dapl_ib_mw_bind(DAPL_RMR *rmr, DAPL_LMR *lmr, DAPL_EP *ep, - DAPL_COOKIE *cookie, DAT_VADDR virtual_address, - DAT_VLEN length, DAT_MEM_PRIV_FLAGS mem_priv, + DAPL_COOKIE *cookie, u64 virtual_address, + u64 length, DAT_MEM_PRIV_FLAGS mem_priv, boolean_t is_signaled); u32 dapl_ib_mw_unbind(DAPL_RMR *rmr, DAPL_EP *ep, DAPL_COOKIE *cookie, diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl.h linux-kernel2/dat-provider/dapl.h --- linux-kernel/dat-provider/dapl.h 2005-05-13 11:10:25.777002000 -0700 +++ linux-kernel2/dat-provider/dapl.h 2005-05-13 11:38:08.137002000 -0700 @@ -505,9 +505,9 @@ extern u32 dapl_cr_handoff(DAT_CR_HANDLE /* EVD Functions */ extern u32 dapl_ia_memtype_hint(DAT_IA_HANDLE ia_handle, DAT_MEM_TYPE mem_type, - DAT_VLEN length, DAT_MEM_OPT mem_optimization, - DAT_VLEN *suggested_length, - DAT_VADDR *suggested_alignment); + u64 length, DAT_MEM_OPT mem_optimization, + u64 *suggested_length, + u64 *suggested_alignment); extern u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_handle, int evd_min_qlen, DAT_UPCALL_POLICY upcall_policy, @@ -624,14 +624,14 @@ extern u32 dapl_ep_set_watermark(DAT_EP_ extern u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_handle, DAT_MEM_TYPE mem_type, DAT_REGION_DESCRIPTION region_description, - DAT_VLEN length, DAT_PZ_HANDLE pz_handle, + u64 length, DAT_PZ_HANDLE pz_handle, DAT_MEM_PRIV_FLAGS privileges, DAT_MEM_OPT optimization, DAT_LMR_HANDLE *lmr_handle, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, - DAT_VLEN *registered_length, - DAT_VADDR *registered_address); + u64 *registered_length, + u64 *registered_address); extern u32 dapl_lmr_query(DAT_LMR_HANDLE, struct dat_lmr_param *); @@ -639,11 +639,11 @@ extern u32 dapl_lmr_free(DAT_LMR_HANDLE) extern u32 dapl_lmr_sync_rdma_read(DAT_IA_HANDLE ia_handle, const struct dat_lmr_triplet *local_segments, - DAT_VLEN num_segments); + u64 num_segments); extern u32 dapl_lmr_sync_rdma_write(DAT_IA_HANDLE ia_handle, const struct dat_lmr_triplet *local_segments, - DAT_VLEN num_segments); + u64 num_segments); /* RMR Functions */ diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_ia_memtype_hint.c linux-kernel2/dat-provider/dapl_ia_memtype_hint.c --- linux-kernel/dat-provider/dapl_ia_memtype_hint.c 2005-05-12 09:28:46.812024000 -0700 +++ linux-kernel2/dat-provider/dapl_ia_memtype_hint.c 2005-05-13 11:38:08.145005000 -0700 @@ -64,9 +64,9 @@ * DAT_MODEL_NOT_SUPPORTED */ u32 dapl_ia_memtype_hint(DAT_IA_HANDLE ia_handle, DAT_MEM_TYPE mem_type, - DAT_VLEN length, DAT_MEM_OPT mem_optimization, - DAT_VLEN *suggested_length, - DAT_VADDR *suggested_alignment) + u64 length, DAT_MEM_OPT mem_optimization, + u64 *suggested_length, + u64 *suggested_alignment) { DAPL_IA *ia_ptr; u32 dat_status = DAT_SUCCESS; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_lmr_kcreate.c linux-kernel2/dat-provider/dapl_lmr_kcreate.c --- linux-kernel/dat-provider/dapl_lmr_kcreate.c 2005-05-12 09:28:48.900014000 -0700 +++ linux-kernel2/dat-provider/dapl_lmr_kcreate.c 2005-05-13 12:25:34.349006000 -0700 @@ -38,13 +38,13 @@ #include "dapl_adapter_util.h" static inline u32 dapl_lmr_create_virtual(DAPL_IA *ia, void *virt_addr, - DAT_VLEN length, DAPL_PZ *pz, + u64 length, DAPL_PZ *pz, DAT_MEM_PRIV_FLAGS privileges, DAT_LMR_HANDLE *lmr_handle, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, - DAT_VLEN *registered_length, - DAT_VADDR *registered_address) + u64 *registered_length, + u64 *registered_address) { DAPL_LMR *lmr; DAT_REGION_DESCRIPTION reg_desc; @@ -98,41 +98,35 @@ static inline u32 dapl_lmr_create_virtua atomic_inc(&pz->pz_ref_count); - if (NULL != lmr_handle) { + if (lmr_handle) *lmr_handle = (DAT_LMR_HANDLE) lmr; - } - if (NULL != lmr_context) { + if (lmr_context) *lmr_context = (DAT_LMR_CONTEXT) lmr->param.lmr_context; - } - if (NULL != rmr_context) { + if (rmr_context) *rmr_context = (DAT_LMR_CONTEXT) lmr->param.rmr_context; - } - if (NULL != registered_address) { - *registered_address = (DAT_VADDR) (uintptr_t) virt_addr; - } - if (NULL != registered_length) { + if (registered_address) + *registered_address = (u64) virt_addr; + if (registered_length) *registered_length = length; - } - bail: +bail: return dat_status; } static inline u32 dapl_lmr_create_physical(DAPL_IA *ia, DAT_REGION_DESCRIPTION phys_addr, - DAT_VLEN page_count, DAPL_PZ *pz, + u64 page_count, DAPL_PZ *pz, DAT_MEM_PRIV_FLAGS privileges, DAT_LMR_HANDLE *lmr_handle, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, - DAT_VLEN *registered_length, - DAT_VADDR *registered_address) + u64 *registered_length, + u64 *registered_address) { DAPL_LMR *lmr; u32 dat_status = DAT_SUCCESS; - u64 *array; - array = (u64 *) phys_addr.for_array; + u64 *array = phys_addr.for_array; lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_PHYSICAL, @@ -145,9 +139,7 @@ static inline u32 dapl_lmr_create_physic goto bail; } - dat_status = dapl_ib_mr_register_physical(ia, - lmr, - phys_addr.for_array, + dat_status = dapl_ib_mr_register_physical(ia, lmr, phys_addr.for_array, page_count, privileges); if (DAT_SUCCESS != dat_status) { @@ -183,23 +175,18 @@ static inline u32 dapl_lmr_create_physic atomic_inc(&pz->pz_ref_count); - if (NULL != lmr_handle) { + if (lmr_handle) *lmr_handle = (DAT_LMR_HANDLE) lmr; - } - if (NULL != lmr_context) { + if (lmr_context) *lmr_context = (DAT_LMR_CONTEXT) lmr->param.lmr_context; - } - if (NULL != rmr_context) { + if (rmr_context) *rmr_context = (DAT_LMR_CONTEXT) lmr->param.rmr_context; - } - if (NULL != registered_address) { - *registered_address = (DAT_VADDR) array[0]; - } - if (NULL != registered_length) { + if (registered_address) + *registered_address = array[0]; + if (registered_length) *registered_length = page_count * PAGE_SIZE; - } - bail: +bail: return dat_status; } @@ -209,8 +196,8 @@ static inline u32 dapl_lmr_create_lmr(DA DAT_LMR_HANDLE *lmr_handle, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, - DAT_VLEN *registered_length, - DAT_VADDR *registered_address) + u64 *registered_length, + u64 *registered_address) { DAPL_LMR *lmr; DAT_REGION_DESCRIPTION reg_desc; @@ -271,24 +258,17 @@ static inline u32 dapl_lmr_create_lmr(DA atomic_inc(&pz->pz_ref_count); - if (NULL != lmr_handle) { + if (lmr_handle) *lmr_handle = (DAT_LMR_HANDLE) lmr; - } - if (NULL != lmr_context) { + if (lmr_context) *lmr_context = (DAT_LMR_CONTEXT) lmr->param.lmr_context; - } - if (NULL != rmr_context) { + if (rmr_context) *rmr_context = (DAT_LMR_CONTEXT) lmr->param.rmr_context; - } - if (NULL != registered_address) { - *registered_address = - (DAT_VADDR) (uintptr_t) original_lmr->param.region_desc. - for_va; - } - if (NULL != registered_length) { + if (registered_address) + *registered_address = (u64)original_lmr->param.region_desc.for_va; + if (registered_length) *registered_length = original_lmr->param.length; - } - bail: +bail: return dat_status; } @@ -322,10 +302,10 @@ static inline u32 dapl_lmr_create_lmr(DA * */ u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_handle, DAT_MEM_TYPE mem_type, - DAT_REGION_DESCRIPTION region_description, DAT_VLEN length, + DAT_REGION_DESCRIPTION region_description, u64 length, DAT_PZ_HANDLE pz_handle, DAT_MEM_PRIV_FLAGS privileges, DAT_MEM_OPT optimization, DAT_LMR_HANDLE *lmr_handle, - DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, DAT_VLEN *registered_length, DAT_VADDR *registered_address) + DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address) { DAPL_IA *ia; DAPL_PZ *pz; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_lmr_sync_rdma_read.c linux-kernel2/dat-provider/dapl_lmr_sync_rdma_read.c --- linux-kernel/dat-provider/dapl_lmr_sync_rdma_read.c 2005-05-12 09:28:46.987010000 -0700 +++ linux-kernel2/dat-provider/dapl_lmr_sync_rdma_read.c 2005-05-13 11:38:08.152012000 -0700 @@ -58,7 +58,7 @@ */ u32 dapl_lmr_sync_rdma_read(DAT_IA_HANDLE ia_handle, const struct dat_lmr_triplet *local_segments, - DAT_VLEN num_segments) + u64 num_segments) { DAPL_IA *ia_ptr; u32 dat_status = DAT_SUCCESS; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_lmr_sync_rdma_write.c linux-kernel2/dat-provider/dapl_lmr_sync_rdma_write.c --- linux-kernel/dat-provider/dapl_lmr_sync_rdma_write.c 2005-05-12 09:28:48.503024000 -0700 +++ linux-kernel2/dat-provider/dapl_lmr_sync_rdma_write.c 2005-05-13 11:38:08.155001000 -0700 @@ -43,7 +43,7 @@ */ u32 dapl_lmr_sync_rdma_write(DAT_IA_HANDLE ia_handle, const struct dat_lmr_triplet *local_segments, - DAT_VLEN num_segments) + u64 num_segments) { DAPL_IA *ia_ptr; u32 dat_status = DAT_SUCCESS; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_lmr_util.c linux-kernel2/dat-provider/dapl_lmr_util.c --- linux-kernel/dat-provider/dapl_lmr_util.c 2005-04-27 11:38:34.874000000 -0700 +++ linux-kernel2/dat-provider/dapl_lmr_util.c 2005-05-13 11:38:08.157005000 -0700 @@ -41,7 +41,7 @@ DAPL_LMR *dapl_lmr_alloc(DAPL_IA * ia, DAT_MEM_TYPE mem_type, DAT_REGION_DESCRIPTION region_desc, - DAT_VLEN length, + u64 length, DAT_PZ_HANDLE pz_handle, DAT_MEM_PRIV_FLAGS mem_priv) { DAPL_LMR *lmr; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_lmr_util.h linux-kernel2/dat-provider/dapl_lmr_util.h --- linux-kernel/dat-provider/dapl_lmr_util.h 2005-04-20 12:42:06.921005000 -0700 +++ linux-kernel2/dat-provider/dapl_lmr_util.h 2005-05-13 11:38:08.160008000 -0700 @@ -48,7 +48,7 @@ extern DAPL_LMR *dapl_lmr_alloc(DAPL_IA * ia, DAT_MEM_TYPE mem_type, DAT_REGION_DESCRIPTION region_desc, - DAT_VLEN length, + u64 length, DAT_PZ_HANDLE pz_handle, DAT_MEM_PRIV_FLAGS mem_priv); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_mr_util.c linux-kernel2/dat-provider/dapl_mr_util.c --- linux-kernel/dat-provider/dapl_mr_util.c 2005-04-20 12:42:07.252004000 -0700 +++ linux-kernel2/dat-provider/dapl_mr_util.c 2005-05-13 12:17:35.132016000 -0700 @@ -56,37 +56,29 @@ * */ -DAT_VADDR dapl_mr_get_address(DAT_REGION_DESCRIPTION desc, DAT_MEM_TYPE type) +u64 dapl_mr_get_address(DAT_REGION_DESCRIPTION desc, DAT_MEM_TYPE type) { + DAPL_LMR *lmr; + switch (type) { case DAT_MEM_TYPE_VIRTUAL: - { - return (DAT_VADDR) (uintptr_t) desc.for_va; - } + return (u64)desc.for_va; case DAT_MEM_TYPE_LMR: - { - DAPL_LMR *lmr; - - lmr = (DAPL_LMR *) desc.for_lmr_handle; + lmr = (DAPL_LMR *) desc.for_lmr_handle; - /* Since this function is recoursive we cannot inline it */ - return dapl_mr_get_address(lmr->param.region_desc, - lmr->param.mem_type); - } + /* Since this function is recoursive we cannot inline it */ + return dapl_mr_get_address(lmr->param.region_desc, + lmr->param.mem_type); case DAT_MEM_TYPE_PHYSICAL: - { - return desc.for_pa; - } + return desc.for_pa; default: - { - /* - * The following kDAPL memory types have not been implemented: - * DAT_MEM_TYPE_PLATFORM - * DAT_MEM_TYPE_IA - * DAT_MEM_TYPE_BYPASS - */ - dapl_os_assert(0); - return 0; - } + /* + * The following kDAPL memory types have not been implemented: + * DAT_MEM_TYPE_PLATFORM + * DAT_MEM_TYPE_IA + * DAT_MEM_TYPE_BYPASS + */ + dapl_os_assert(0); + return 0; } } diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_mr_util.h linux-kernel2/dat-provider/dapl_mr_util.h --- linux-kernel/dat-provider/dapl_mr_util.h 2005-05-11 08:28:11.969010000 -0700 +++ linux-kernel2/dat-provider/dapl_mr_util.h 2005-05-13 12:14:14.433011000 -0700 @@ -40,24 +40,7 @@ #include "dapl.h" #include "dapl_hash.h" -/********************************************************************* - * * - * Function Prototypes * - * * - *********************************************************************/ - -extern DAT_VADDR -dapl_mr_get_address(DAT_REGION_DESCRIPTION desc, DAT_MEM_TYPE type); - -static __inline__ boolean_t -dapl_mr_bounds_check(DAT_VADDR addr_a, - DAT_VLEN length_a, DAT_VADDR addr_b, DAT_VLEN length_b); - -/********************************************************************* - * * - * Inline Functions * - * * - *********************************************************************/ +extern u64 dapl_mr_get_address(DAT_REGION_DESCRIPTION desc, DAT_MEM_TYPE type); /* * dapl_mr_bounds_check @@ -67,15 +50,13 @@ dapl_mr_bounds_check(DAT_VADDR addr_a, * */ -static __inline__ boolean_t -dapl_mr_bounds_check(DAT_VADDR addr_a, - DAT_VLEN length_a, DAT_VADDR addr_b, DAT_VLEN length_b) +static inline boolean_t dapl_mr_bounds_check(u64 addr_a, u64 length_a, + u64 addr_b, u64 length_b) { - if ((addr_a <= addr_b) && (addr_b + length_b) <= (addr_a + length_a)) { + if ((addr_a <= addr_b) && (addr_b + length_b) <= (addr_a + length_a)) return TRUE; - } else { + else return FALSE; - } } #endif /* DAPL_MR_UTIL_H */ diff -Nurp -X /home/tduffy/dontdiff linux-kernel/dat-provider/dapl_openib_util.c linux-kernel2/dat-provider/dapl_openib_util.c --- linux-kernel/dat-provider/dapl_openib_util.c 2005-05-13 11:22:30.684000000 -0700 +++ linux-kernel2/dat-provider/dapl_openib_util.c 2005-05-13 11:38:08.167002000 -0700 @@ -391,14 +391,14 @@ u32 dapl_ib_pd_free(DAPL_PZ *pz) * */ u32 dapl_ib_mr_register(DAPL_IA *ia, DAPL_LMR *lmr, void *virt_addr, - DAT_VLEN length, DAT_MEM_PRIV_FLAGS privileges) + u64 length, DAT_MEM_PRIV_FLAGS privileges) { /* ITBD virtual memory registration ??? */ return (DAT_SUCCESS); } u32 dapl_ib_mr_register_physical(DAPL_IA *ia_ptr, DAPL_LMR *lmr, - void *phys_addr, DAT_VLEN length, + void *phys_addr, u64 length, DAT_MEM_PRIV_FLAGS privileges) { int ib_status; @@ -577,8 +577,8 @@ u32 dapl_ib_mw_free(DAPL_RMR *rmr) * */ u32 dapl_ib_mw_bind(DAPL_RMR *rmr, DAPL_LMR *lmr, DAPL_EP *ep, - DAPL_COOKIE *cookie, DAT_VADDR virtual_address, - DAT_VLEN length, DAT_MEM_PRIV_FLAGS mem_priv, + DAPL_COOKIE *cookie, u64 virtual_address, + u64 length, DAT_MEM_PRIV_FLAGS mem_priv, boolean_t is_signaled) { int ib_status; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/include/dapl_bpool.h linux-kernel2/test/dapltest/include/dapl_bpool.h --- linux-kernel/test/dapltest/include/dapl_bpool.h 2005-05-12 09:34:59.295003000 -0700 +++ linux-kernel2/test/dapltest/include/dapl_bpool.h 2005-05-13 11:38:08.177004000 -0700 @@ -42,9 +42,9 @@ struct Bpool_tag { int seg_size; int num_segs; /* num segments */ unsigned char *buffer_start; /* Start of buffer area */ - DAT_VLEN buffer_size; /* Size of data buffer (rounded) */ - DAT_VADDR reg_addr; /* start of registered area */ - DAT_VLEN reg_size; /* size of registered area */ + u64 buffer_size; /* Size of data buffer (rounded) */ + u64 reg_addr; /* start of registered area */ + u64 reg_size; /* size of registered area */ DAT_EP_HANDLE ep_handle; /* EP area is registered to */ DAT_LMR_HANDLE lmr_handle; /* local access */ DAT_LMR_CONTEXT lmr_context; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/include/dapl_performance_test.h linux-kernel2/test/dapltest/include/dapl_performance_test.h --- linux-kernel/test/dapltest/include/dapl_performance_test.h 2005-05-13 11:10:25.821009000 -0700 +++ linux-kernel2/test/dapltest/include/dapl_performance_test.h 2005-05-13 11:37:44.334000000 -0700 @@ -49,7 +49,7 @@ typedef struct { /* RDMA info */ DAT_RMR_CONTEXT Rdma_Context; - DAT_VADDR Rdma_Address; + u64 Rdma_Address; } Performance_Test_Op_t; typedef struct diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/include/dapl_proto.h linux-kernel2/test/dapltest/include/dapl_proto.h --- linux-kernel/test/dapltest/include/dapl_proto.h 2005-05-13 11:10:25.827001000 -0700 +++ linux-kernel2/test/dapltest/include/dapl_proto.h 2005-05-13 11:37:44.339003000 -0700 @@ -96,7 +96,7 @@ boolean_t DT_Bpool_Destroy (P Bpool * bpool_ptr); unsigned char *DT_Bpool_GetBuffer (Bpool * bpool_ptr, int index); -DAT_VADDR DT_Bpool_GetRegBuff (Bpool * bpool_ptr, int index); +u64 DT_Bpool_GetRegBuff (Bpool * bpool_ptr, int index); int DT_Bpool_GetBuffSize (Bpool * bpool_ptr, int index); struct dat_lmr_triplet *DT_Bpool_GetIOV (Bpool * bpool_ptr, int index); DAT_LMR_CONTEXT DT_Bpool_GetLMR (Bpool * bpool_ptr, int index); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/include/dapl_transaction_test.h linux-kernel2/test/dapltest/include/dapl_transaction_test.h --- linux-kernel/test/dapltest/include/dapl_transaction_test.h 2005-05-13 11:10:25.830004000 -0700 +++ linux-kernel2/test/dapltest/include/dapl_transaction_test.h 2005-05-13 11:37:44.342006000 -0700 @@ -44,7 +44,7 @@ typedef struct /* RDMA info */ DAT_RMR_CONTEXT Rdma_Context; - DAT_VADDR Rdma_Address; + u64 Rdma_Address; } Transaction_Test_Op_t; typedef struct diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_bpool.c linux-kernel2/test/dapltest/test/dapl_bpool.c --- linux-kernel/test/dapltest/test/dapl_bpool.c 2005-05-12 09:34:59.340003000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_bpool.c 2005-05-13 11:38:08.187002000 -0700 @@ -40,8 +40,8 @@ static void DTs_CreateIovec ( void **region_addr, u64 *buf_region, - DAT_VLEN buffer_size, - DAT_VLEN *bp_len) + u64 buffer_size, + u64 *bp_len) { int ii; u_int64_t ptr_addr; @@ -102,7 +102,7 @@ DT_BpoolAlloc ( int alloc_size, bpool_size; DAT_REGION_DESCRIPTION region; u32 ret = DAT_SUCCESS; - DAT_VLEN bp_len; + u64 bp_len; DAT_MEM_TYPE DT_mem_type; u64 buf_region[10]; if (pt_ptr != NULL) { @@ -385,7 +385,7 @@ DT_Bpool_GetBuffer (Bpool * bpool_ptr, i } /*****************************************************************************/ -DAT_VADDR +u64 DT_Bpool_GetRegBuff (Bpool * bpool_ptr, int index) { return ( bpool_ptr->reg_addr + index * bpool_ptr->seg_size ); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_fft_mem.c linux-kernel2/test/dapltest/test/dapl_fft_mem.c --- linux-kernel/test/dapltest/test/dapl_fft_mem.c 2005-05-12 09:28:43.415002000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_fft_mem.c 2005-05-13 11:38:08.191004000 -0700 @@ -36,13 +36,13 @@ int DT_mem_generic (Params_t *params_ptr u32 rc, expect; FFT_Connection_t conn; DAT_REGION_DESCRIPTION region; - DAT_VLEN reg_size; + u64 reg_size; DAT_LMR_HANDLE lmr_handle; DAT_LMR_CONTEXT lmr_context; - DAT_VADDR reg_addr; + u64 reg_addr; unsigned char *alloc_ptr; int res; - DAT_VLEN buffer_size; + u64 buffer_size; DAT_IA_HANDLE ia_handle; DAT_PZ_HANDLE pz_handle; DT_Tdep_Print_Head *phead; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c linux-kernel2/test/dapltest/test/dapl_fft_queryinfo.c --- linux-kernel/test/dapltest/test/dapl_fft_queryinfo.c 2005-05-13 11:01:18.714000000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_fft_queryinfo.c 2005-05-13 11:38:08.195001000 -0700 @@ -64,9 +64,9 @@ int DT_queryinfo_basic (Params_t *params DAT_RMR_HANDLE rmr_handle; struct dat_rmr_param rmr_param; DAT_REGION_DESCRIPTION region; - DAT_VLEN reg_size; - DAT_VADDR reg_addr; - DAT_VLEN buffer_size; + u64 reg_size; + u64 reg_addr; + u64 buffer_size; unsigned char *alloc_ptr; DT_Tdep_Print_Head *phead; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_limit.c linux-kernel2/test/dapltest/test/dapl_limit.c --- linux-kernel/test/dapltest/test/dapl_limit.c 2005-05-12 09:34:59.354000000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_limit.c 2005-05-13 11:38:08.203004000 -0700 @@ -885,8 +885,8 @@ limit_test ( DT_Tdep_Print_Head *phead, for (w = 0; w < cmd->width; w++) { DAT_REGION_DESCRIPTION region; - DAT_VLEN reg_size; - DAT_VADDR reg_addr; + u64 reg_size; + u64 reg_addr; hdl_sets[w].lmr_buffer = DT_Mdep_Malloc (DFLT_BUFFSZ); if (!hdl_sets[w].lmr_buffer) @@ -1045,7 +1045,7 @@ limit_test ( DT_Tdep_Print_Head *phead, struct dat_lmr_triplet *iovp = &hdlptr[w * cmd->width + i]; DAT_DTO_COOKIE cookie; - iovp->virtual_address = (DAT_VADDR) (uintptr_t) + iovp->virtual_address = (u64) (uintptr_t) hdl_sets[i].lmr_buffer; iovp->segment_length = DFLT_BUFFSZ; iovp->lmr_context = hdl_sets[i].lmr_context; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_performance_client.c linux-kernel2/test/dapltest/test/dapl_performance_client.c --- linux-kernel/test/dapltest/test/dapl_performance_client.c 2005-05-13 11:10:25.841005000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_performance_client.c 2005-05-13 11:37:44.366001000 -0700 @@ -278,7 +278,7 @@ DT_Performance_Test_Client_Phase2 ( for (i = 0; i < op->num_segs; i++) { iov[i].pad = 0U; - iov[i].virtual_address = /*(DAT_VADDR) (uintptr_t) */ + iov[i].virtual_address = /*(u64) (uintptr_t) */ DT_Bpool_GetRegBuff (op->bp, i); iov[i].segment_length = op->seg_size; iov[i].lmr_context = DT_Bpool_GetLMR (op->bp, i); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_performance_server.c linux-kernel2/test/dapltest/test/dapl_performance_server.c --- linux-kernel/test/dapltest/test/dapl_performance_server.c 2005-05-13 11:13:02.570005000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_performance_server.c 2005-05-13 11:37:44.370004000 -0700 @@ -305,7 +305,7 @@ DT_Performance_Test_Server_Exchange ( test_ptr->ep_context.op.Rdma_Context = DT_Bpool_GetRMR (test_ptr->ep_context.op.bp, 0); - test_ptr->ep_context.op.Rdma_Address = /*(DAT_VADDR) (uintptr_t)*/ + test_ptr->ep_context.op.Rdma_Address = /*(u64) (uintptr_t)*/ DT_Bpool_GetRegBuff (test_ptr->ep_context.op.bp, 0); /* diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_performance_util.c linux-kernel2/test/dapltest/test/dapl_performance_util.c --- linux-kernel/test/dapltest/test/dapl_performance_util.c 2005-05-13 11:10:25.846002000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_performance_util.c 2005-05-13 11:37:44.374000000 -0700 @@ -438,7 +438,7 @@ DT_performance_post_rdma_op ( for (j = 0; j < op->num_segs; j++) { iov[j].pad = 0U; - iov[j].virtual_address = /*(DAT_VADDR) (uintptr_t)*/ + iov[j].virtual_address = /*(u64) (uintptr_t)*/ DT_Bpool_GetRegBuff (op->bp, j); iov[j].segment_length = op->seg_size; iov[j].lmr_context = DT_Bpool_GetLMR (op->bp, j); diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_test_util.c linux-kernel2/test/dapltest/test/dapl_test_util.c --- linux-kernel/test/dapltest/test/dapl_test_util.c 2005-05-12 09:34:59.364000000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_test_util.c 2005-05-13 11:37:44.379003000 -0700 @@ -175,7 +175,7 @@ DT_post_recv_buffer (DT_Tdep_Print_Head * Prep the inputs */ iov->virtual_address = DT_Bpool_GetRegBuff(bp,index); - /* iov->virtual_address = (DAT_VADDR) (uintptr_t) buff; */ + /* iov->virtual_address = (u64) (uintptr_t) buff; */ iov->segment_length = size; iov->lmr_context = lmr_c; cookie.as_64 = (u64)0UL; @@ -220,7 +220,7 @@ DT_post_send_buffer (DT_Tdep_Print_Head * Prep the inputs */ iov->virtual_address = DT_Bpool_GetRegBuff(bp,index); - /* iov->virtual_address = (DAT_VADDR) (uintptr_t) buff; */ + /* iov->virtual_address = (u64) (uintptr_t) buff; */ iov->segment_length = size; iov->lmr_context = lmr_c; cookie.as_64 = (u64)0UL; diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_transaction_test.c linux-kernel2/test/dapltest/test/dapl_transaction_test.c --- linux-kernel/test/dapltest/test/dapl_transaction_test.c 2005-05-13 11:13:02.581001000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_transaction_test.c 2005-05-13 11:37:44.390001000 -0700 @@ -858,7 +858,7 @@ retry: test_ptr->ep_context[i].op[j].Rdma_Context = DT_Bpool_GetRMR (test_ptr->ep_context[i].op[j].bp, 0); test_ptr->ep_context[i].op[j].Rdma_Address = - /* (DAT_VADDR) (uintptr_t) */ + /* (u64) (uintptr_t) */ DT_Bpool_GetRegBuff (test_ptr->ep_context[i].op[j].bp,0); DT_Tdep_PT_Debug (3, (phead, "not-us: RDMA/RD [ va=" F64x ", ctxt=%x ]\n", @@ -894,7 +894,7 @@ retry: test_ptr->ep_context[i].op[j].Rdma_Context = DT_Bpool_GetRMR (test_ptr->ep_context[i].op[j].bp, 0); test_ptr->ep_context[i].op[j].Rdma_Address = - /* (DAT_VADDR)(uintptr_t)*/DT_Bpool_GetRegBuff (test_ptr->ep_context[i].op[j].bp,0); + /* (u64)(uintptr_t)*/DT_Bpool_GetRegBuff (test_ptr->ep_context[i].op[j].bp,0); DT_Tdep_PT_Debug (3, (phead, "not-us: RDMA/WR [ va=" F64x ", ctxt=%x ]\n", test_ptr->ep_context[i].op[j].Rdma_Address, diff -Nurp -X /home/tduffy/dontdiff linux-kernel/test/dapltest/test/dapl_transaction_util.c linux-kernel2/test/dapltest/test/dapl_transaction_util.c --- linux-kernel/test/dapltest/test/dapl_transaction_util.c 2005-05-12 09:28:43.551016000 -0700 +++ linux-kernel2/test/dapltest/test/dapl_transaction_util.c 2005-05-13 11:37:44.396002000 -0700 @@ -49,7 +49,7 @@ DT_handle_post_recv_buf (DT_Tdep_Print_H for (j = 0; j < op->num_segs; j++) { iov[j].pad = 0U; - iov[j].virtual_address = /*(DAT_VADDR) (uintptr_t) */ + iov[j].virtual_address = /*(u64) (uintptr_t) */ DT_Bpool_GetRegBuff (op->bp, j); iov[j].segment_length = op->seg_size; iov[j].lmr_context = DT_Bpool_GetLMR (op->bp, j); @@ -112,7 +112,7 @@ DT_handle_send_op (DT_Tdep_Print_Head *p { iov[j].pad = 0U; - iov[j].virtual_address = /*(DAT_VADDR) (uintptr_t)*/ + iov[j].virtual_address = /*(u64) (uintptr_t)*/ DT_Bpool_GetRegBuff (op->bp, j); iov[j].segment_length = op->seg_size; iov[j].lmr_context = DT_Bpool_GetLMR (op->bp, j); @@ -492,7 +492,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p for (j = 0; j < op->num_segs; j++) { iov[j].pad = 0U; - iov[j].virtual_address = /*(DAT_VADDR) (uintptr_t)*/ + iov[j].virtual_address = /*(u64) (uintptr_t)*/ DT_Bpool_GetRegBuff (op->bp, j); iov[j].segment_length = op->seg_size; iov[j].lmr_context = DT_Bpool_GetLMR (op->bp, j); @@ -502,7 +502,7 @@ DT_handle_rdma_op (DT_Tdep_Print_Head *p | (((uintptr_t) DT_Bpool_GetBuffer (op->bp, 0)) & 0xffffffffUL)); rmr_triplet.pad = 0U; - rmr_triplet.target_address = /*(DAT_VADDR) (uintptr_t)*/ op->Rdma_Address; + rmr_triplet.target_address = /*(u64) (uintptr_t)*/ op->Rdma_Address; rmr_triplet.segment_length = op->seg_size * op->num_segs; rmr_triplet.rmr_context = op->Rdma_Context; From Tom.Duffy at Sun.COM Sun May 15 11:27:16 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Sun, 15 May 2005 11:27:16 -0700 Subject: [openib-general] [PATCH][5/19] kDAPL: Remove typedef DAT_COMPLETION_FLAGS In-Reply-To: <1116181636449@sun.com> References: <11161816352658@sun.com> <11161816351137@sun.com> <11161816352496@sun.com> <11161816361781@sun.com> <1116181636449@sun.com> Message-ID: <11161816363220@sun.com> Signed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel4/dat/dat.h linux-kernel5/dat/dat.h --- linux-kernel4/dat/dat.h 2005-05-13 12:42:37.630017000 -0700 +++ linux-kernel5/dat/dat.h 2005-05-13 12:48:43.104001000 -0700 @@ -370,7 +370,7 @@ typedef union dat_context { typedef DAT_CONTEXT DAT_DTO_COOKIE; typedef DAT_CONTEXT DAT_RMR_COOKIE; -typedef enum dat_completion_flags { +enum dat_completion_flags { /* Completes with notification */ DAT_COMPLETION_DEFAULT_FLAG = 0x00, /* Completions suppressed if successful */ @@ -386,7 +386,7 @@ typedef enum dat_completion_flags { * dat_evd_wait. UNSIGNALLED for RECV is not allowed when EP has * this attribute. */ DAT_COMPLETION_EVD_THRESHOLD_FLAG = 0x10 -} DAT_COMPLETION_FLAGS; +}; typedef u32 DAT_TIMEOUT; /* microseconds */ @@ -561,8 +561,8 @@ struct dat_ep_attr { u64 max_message_size; u64 max_rdma_size; enum dat_qos qos; - DAT_COMPLETION_FLAGS recv_completion_flags; - DAT_COMPLETION_FLAGS request_completion_flags; + enum dat_completion_flags recv_completion_flags; + enum dat_completion_flags request_completion_flags; int max_recv_dtos; int max_request_dtos; int max_recv_iov; @@ -1065,7 +1065,7 @@ struct dat_provider_attr { DAT_MEM_TYPE lmr_mem_types_supported; DAT_IOV_OWNERSHIP iov_ownership_on_return; enum dat_qos dat_qos_supported; - DAT_COMPLETION_FLAGS completion_flags_supported; + enum dat_completion_flags completion_flags_supported; boolean_t is_thread_safe; int max_private_data_size; boolean_t supports_multipath; @@ -1157,23 +1157,23 @@ typedef u32 (*DAT_EP_DISCONNECT_FUNC)(DA typedef u32 (*DAT_EP_POST_SEND_FUNC)(DAT_EP_HANDLE, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, - DAT_COMPLETION_FLAGS); + enum dat_completion_flags); typedef u32 (*DAT_EP_POST_RECV_FUNC)(DAT_EP_HANDLE, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, - DAT_COMPLETION_FLAGS); + enum dat_completion_flags); typedef u32 (*DAT_EP_POST_RDMA_READ_FUNC)(DAT_EP_HANDLE, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, const struct dat_rmr_triplet *, - DAT_COMPLETION_FLAGS); + enum dat_completion_flags); typedef u32 (*DAT_EP_POST_RDMA_WRITE_FUNC)(DAT_EP_HANDLE, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, const struct dat_rmr_triplet *, - DAT_COMPLETION_FLAGS); + enum dat_completion_flags); typedef u32 (*DAT_EP_GET_STATUS_FUNC)(DAT_EP_HANDLE, DAT_EP_STATE *, boolean_t *, boolean_t *); @@ -1204,7 +1204,7 @@ typedef u32 (*DAT_RMR_QUERY_FUNC)(DAT_RM typedef u32 (*DAT_RMR_BIND_FUNC)(DAT_RMR_HANDLE, const struct dat_lmr_triplet *, DAT_MEM_PRIV_FLAGS, DAT_EP_HANDLE, - DAT_RMR_COOKIE, DAT_COMPLETION_FLAGS, + DAT_RMR_COOKIE, enum dat_completion_flags, DAT_RMR_CONTEXT *); typedef u32 (*DAT_RMR_FREE_FUNC)(DAT_RMR_HANDLE); @@ -1563,7 +1563,7 @@ static inline u32 dat_ep_post_rdma_read( struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, const struct dat_rmr_triplet *remote_iov, - DAT_COMPLETION_FLAGS flags) + enum dat_completion_flags flags) { return DAT_CALL_PROVIDER_FUNC(ep_post_rdma_read_func, ep, size, local_iov, cookie, remote_iov, flags); @@ -1573,7 +1573,7 @@ static inline u32 dat_ep_post_rdma_write struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, const struct dat_rmr_triplet *remote_iov, - DAT_COMPLETION_FLAGS flags) + enum dat_completion_flags flags) { return DAT_CALL_PROVIDER_FUNC(ep_post_rdma_write_func, ep, size, local_iov, cookie, remote_iov, flags); @@ -1582,7 +1582,7 @@ static inline u32 dat_ep_post_rdma_write static inline u32 dat_ep_post_recv(DAT_EP_HANDLE ep, int size, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, - DAT_COMPLETION_FLAGS flags) + enum dat_completion_flags flags) { return DAT_CALL_PROVIDER_FUNC(ep_post_recv_func, ep, size, local_iov, cookie, flags); @@ -1592,7 +1592,7 @@ static inline u32 dat_ep_post_send(DAT_E int size, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE cookie, - DAT_COMPLETION_FLAGS flags) + enum dat_completion_flags flags) { return DAT_CALL_PROVIDER_FUNC( ep_post_send_func, ep, size, local_iov, cookie, flags); @@ -1663,7 +1663,7 @@ static inline u32 dat_rmr_bind(DAT_RMR_H const struct dat_lmr_triplet *iov, DAT_MEM_PRIV_FLAGS mem_flags, DAT_EP_HANDLE ep, DAT_RMR_COOKIE cookie, - DAT_COMPLETION_FLAGS comp_flags, + enum dat_completion_flags comp_flags, DAT_RMR_CONTEXT *context) { return DAT_CALL_PROVIDER_FUNC(rmr_bind_func, rmr, iov, mem_flags, ep, diff -Nurp -X /home/tduffy/dontdiff linux-kernel4/dat-provider/dapl_ep_post_rdma_read.c linux-kernel5/dat-provider/dapl_ep_post_rdma_read.c --- linux-kernel4/dat-provider/dapl_ep_post_rdma_read.c 2005-05-13 10:10:34.053960000 -0700 +++ linux-kernel5/dat-provider/dapl_ep_post_rdma_read.c 2005-05-13 12:48:23.534001000 -0700 @@ -70,7 +70,7 @@ u32 dapl_ep_post_rdma_read(DAT_EP_HANDLE struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, - DAT_COMPLETION_FLAGS completion_flags) + enum dat_completion_flags completion_flags) { u32 dat_status; diff -Nurp -X /home/tduffy/dontdiff linux-kernel4/dat-provider/dapl_ep_post_rdma_write.c linux-kernel5/dat-provider/dapl_ep_post_rdma_write.c --- linux-kernel4/dat-provider/dapl_ep_post_rdma_write.c 2005-05-13 10:10:33.166966000 -0700 +++ linux-kernel5/dat-provider/dapl_ep_post_rdma_write.c 2005-05-13 12:48:23.536005000 -0700 @@ -70,7 +70,7 @@ u32 dapl_ep_post_rdma_write(DAT_EP_HANDL struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, - DAT_COMPLETION_FLAGS completion_flags) + enum dat_completion_flags completion_flags) { u32 dat_status; diff -Nurp -X /home/tduffy/dontdiff linux-kernel4/dat-provider/dapl_ep_post_recv.c linux-kernel5/dat-provider/dapl_ep_post_recv.c --- linux-kernel4/dat-provider/dapl_ep_post_recv.c 2005-05-13 10:10:34.508971000 -0700 +++ linux-kernel5/dat-provider/dapl_ep_post_recv.c 2005-05-13 12:48:23.539008000 -0700 @@ -69,7 +69,7 @@ u32 dapl_ep_post_recv(DAT_EP_HANDLE ep_handle, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, - DAT_COMPLETION_FLAGS completion_flags) + enum dat_completion_flags completion_flags) { DAPL_EP *ep_ptr; DAPL_COOKIE *cookie; diff -Nurp -X /home/tduffy/dontdiff linux-kernel4/dat-provider/dapl_ep_post_send.c linux-kernel5/dat-provider/dapl_ep_post_send.c --- linux-kernel4/dat-provider/dapl_ep_post_send.c 2005-05-13 10:10:33.398972000 -0700 +++ linux-kernel5/dat-provider/dapl_ep_post_send.c 2005-05-13 12:48:23.542011000 -0700 @@ -66,7 +66,7 @@ u32 dapl_ep_post_send(DAT_EP_HANDLE ep_handle, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, - DAT_COMPLETION_FLAGS completion_flags) + enum dat_completion_flags completion_flags) { struct dat_rmr_triplet remote_iov = { 0, 0, 0, 0 }; u32 dat_status; diff -Nurp -X /home/tduffy/dontdiff linux-kernel4/dat-provider/dapl_ep_util.c linux-kernel5/dat-provider/dapl_ep_util.c --- linux-kernel4/dat-provider/dapl_ep_util.c 2005-05-13 10:10:33.960972000 -0700 +++ linux-kernel5/dat-provider/dapl_ep_util.c 2005-05-13 12:48:23.546002000 -0700 @@ -257,7 +257,7 @@ void dapl_ep_default_attrs(DAPL_EP * ep_ return; } -u32 dapl_ep_check_recv_completion_flags(DAT_COMPLETION_FLAGS flags) +u32 dapl_ep_check_recv_completion_flags(enum dat_completion_flags flags) { /* @@ -276,7 +276,7 @@ u32 dapl_ep_check_recv_completion_flags( return DAT_SUCCESS; } -u32 dapl_ep_check_request_completion_flags(DAT_COMPLETION_FLAGS flags) +u32 dapl_ep_check_request_completion_flags(enum dat_completion_flags flags) { return DAT_SUCCESS; } @@ -285,7 +285,7 @@ u32 dapl_ep_post_send_req(DAT_EP_HANDLE struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, - DAT_COMPLETION_FLAGS completion_flags, + enum dat_completion_flags completion_flags, DAPL_DTO_TYPE dto_type, int op_type) { DAPL_EP *ep_ptr; diff -Nurp -X /home/tduffy/dontdiff linux-kernel4/dat-provider/dapl_ep_util.h linux-kernel5/dat-provider/dapl_ep_util.h --- linux-kernel4/dat-provider/dapl_ep_util.h 2005-05-13 10:10:34.006969000 -0700 +++ linux-kernel5/dat-provider/dapl_ep_util.h 2005-05-13 12:51:14.536002000 -0700 @@ -46,15 +46,16 @@ extern DAPL_EP *dapl_ep_alloc(DAPL_IA * extern void dapl_ep_dealloc(DAPL_EP * ep_ptr); -extern u32 dapl_ep_check_recv_completion_flags(DAT_COMPLETION_FLAGS flags); +extern u32 dapl_ep_check_recv_completion_flags(enum dat_completion_flags flags); -extern u32 dapl_ep_check_request_completion_flags(DAT_COMPLETION_FLAGS flags); +extern u32 dapl_ep_check_request_completion_flags(enum dat_completion_flags + flags); extern u32 dapl_ep_post_send_req(DAT_EP_HANDLE ep_handle, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, - DAT_COMPLETION_FLAGS completion_flags, + enum dat_completion_flags completion_flags, DAPL_DTO_TYPE dto_type, int op_type); void dapl_ep_timeout(uintptr_t arg); diff -Nurp -X /home/tduffy/dontdiff linux-kernel4/dat-provider/dapl.h linux-kernel5/dat-provider/dapl.h --- linux-kernel4/dat-provider/dapl.h 2005-05-13 12:43:32.918003000 -0700 +++ linux-kernel5/dat-provider/dapl.h 2005-05-13 12:48:23.531003000 -0700 @@ -576,26 +576,26 @@ extern u32 dapl_ep_disconnect(DAT_EP_HAN extern u32 dapl_ep_post_send(DAT_EP_HANDLE ep_handle, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, - DAT_COMPLETION_FLAGS completion_flags); + enum dat_completion_flags completion_flags); extern u32 dapl_ep_post_recv(DAT_EP_HANDLE ep_handle, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, - DAT_COMPLETION_FLAGS completion_flags); + enum dat_completion_flags completion_flags); extern u32 dapl_ep_post_rdma_read(DAT_EP_HANDLE ep_handle, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, - DAT_COMPLETION_FLAGS completion_flags); + enum dat_completion_flags completion_flags); extern u32 dapl_ep_post_rdma_write(DAT_EP_HANDLE ep_handle, int num_segments, struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie, const struct dat_rmr_triplet *remote_iov, - DAT_COMPLETION_FLAGS completion_flags); + enum dat_completion_flags completion_flags); extern u32 dapl_ep_get_status(DAT_EP_HANDLE ep_handle, DAT_EP_STATE *ep_state, boolean_t *in_dto_idle, boolean_t *out_dto_idle); @@ -657,7 +657,7 @@ extern u32 dapl_rmr_bind(DAT_RMR_HANDLE const struct dat_lmr_triplet *lmr_triplet, DAT_MEM_PRIV_FLAGS mem_priv, DAT_EP_HANDLE ep_handle, DAT_RMR_COOKIE user_cookie, - DAT_COMPLETION_FLAGS completion_flags, + enum dat_completion_flags completion_flags, DAT_RMR_CONTEXT *context); extern u32 dapl_rmr_free(DAT_RMR_HANDLE); diff -Nurp -X /home/tduffy/dontdiff linux-kernel4/dat-provider/dapl_openib_dto.h linux-kernel5/dat-provider/dapl_openib_dto.h --- linux-kernel4/dat-provider/dapl_openib_dto.h 2005-05-13 10:10:33.258972000 -0700 +++ linux-kernel5/dat-provider/dapl_openib_dto.h 2005-05-13 12:52:57.961018000 -0700 @@ -99,12 +99,12 @@ static __inline__ u32 dapl_ib_post_recv( * * Provider specific Post SEND function */ -static __inline__ u32 dapl_ib_post_send(DAPL_EP *ep_ptr, int op_type, - DAPL_COOKIE *cookie, - int num_segments, - struct dat_lmr_triplet *local_iov, - const struct dat_rmr_triplet *remote_iov, - DAT_COMPLETION_FLAGS completion_flags) +static inline u32 dapl_ib_post_send(DAPL_EP *ep_ptr, int op_type, + DAPL_COOKIE *cookie, + int num_segments, + struct dat_lmr_triplet *local_iov, + const struct dat_rmr_triplet *remote_iov, + enum dat_completion_flags completion_flags) { struct ib_send_wr send_desc = { NULL }; struct ib_send_wr *send_desc_fail; diff -Nurp -X /home/tduffy/dontdiff linux-kernel4/dat-provider/dapl_rmr_bind.c linux-kernel5/dat-provider/dapl_rmr_bind.c --- linux-kernel4/dat-provider/dapl_rmr_bind.c 2005-05-12 09:28:47.829013000 -0700 +++ linux-kernel5/dat-provider/dapl_rmr_bind.c 2005-05-13 12:55:03.339011000 -0700 @@ -51,7 +51,7 @@ static inline u32 dapl_rmr_bind_fuse(DAP DAT_MEM_PRIV_FLAGS mem_priv, DAPL_EP *ep_ptr, DAT_RMR_COOKIE user_cookie, - DAT_COMPLETION_FLAGS completion_flags, + enum dat_completion_flags completion_flags, DAT_RMR_CONTEXT *rmr_context) { DAPL_LMR *lmr; @@ -158,7 +158,7 @@ static inline u32 dapl_rmr_bind_fuse(DAP static inline u32 dapl_rmr_bind_unfuse(DAPL_RMR *rmr, DAPL_EP *ep_ptr, DAT_RMR_COOKIE user_cookie, - DAT_COMPLETION_FLAGS completion_flags) + enum dat_completion_flags cflags) { DAPL_COOKIE *cookie; u32 status = DAT_SUCCESS; @@ -189,15 +189,14 @@ static inline u32 dapl_rmr_bind_unfuse(D if (!dapl_rmr_validate_completion_flag(DAT_COMPLETION_SUPPRESS_FLAG, ep_ptr->param.ep_attr. request_completion_flags, - completion_flags) || + cflags) || !dapl_rmr_validate_completion_flag(DAT_COMPLETION_UNSIGNALLED_FLAG, ep_ptr->param.ep_attr. request_completion_flags, - completion_flags) || + cflags) || !dapl_rmr_validate_completion_flag( DAT_COMPLETION_BARRIER_FENCE_FLAG, - ep_ptr->param.ep_attr.request_completion_flags, - completion_flags)) { + ep_ptr->param.ep_attr.request_completion_flags, cflags)) { status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); goto bail1; } @@ -207,8 +206,7 @@ static inline u32 dapl_rmr_bind_unfuse(D if (DAT_SUCCESS != status) goto bail1; - is_signaled = (completion_flags & DAT_COMPLETION_UNSIGNALLED_FLAG) ? - FALSE : TRUE; + is_signaled = (cflags & DAT_COMPLETION_UNSIGNALLED_FLAG) ? FALSE : TRUE; /* * Take reference before posting to avoid race conditions with @@ -253,7 +251,7 @@ u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_han const struct dat_lmr_triplet *lmr_triplet, DAT_MEM_PRIV_FLAGS mem_priv, DAT_EP_HANDLE ep_handle, DAT_RMR_COOKIE user_cookie, - DAT_COMPLETION_FLAGS completion_flags, + enum dat_completion_flags completion_flags, DAT_RMR_CONTEXT *rmr_context) { DAPL_RMR *rmr; diff -Nurp -X /home/tduffy/dontdiff linux-kernel4/dat-provider/dapl_rmr_util.h linux-kernel5/dat-provider/dapl_rmr_util.h --- linux-kernel4/dat-provider/dapl_rmr_util.h 2005-05-11 08:28:12.678010000 -0700 +++ linux-kernel5/dat-provider/dapl_rmr_util.h 2005-05-13 12:57:13.871055000 -0700 @@ -49,27 +49,15 @@ extern DAPL_RMR *dapl_rmr_alloc(DAPL_PZ extern void dapl_rmr_dealloc(DAPL_RMR * rmr); -static __inline__ boolean_t -dapl_rmr_validate_completion_flag(DAT_COMPLETION_FLAGS mask, - DAT_COMPLETION_FLAGS allow, - DAT_COMPLETION_FLAGS request); - -/********************************************************************* - * * - * Inline Functions * - * * - *********************************************************************/ - -static __inline__ boolean_t -dapl_rmr_validate_completion_flag(DAT_COMPLETION_FLAGS mask, - DAT_COMPLETION_FLAGS allow, - DAT_COMPLETION_FLAGS request) +static inline boolean_t +dapl_rmr_validate_completion_flag(enum dat_completion_flags mask, + enum dat_completion_flags allow, + enum dat_completion_flags request) { - if ((mask & request) && !(mask & allow)) { + if ((mask & request) && !(mask & allow)) return FALSE; - } else { + else return TRUE; - } } #endif /* DAPL_RMR_UTIL_H */ From Tom.Duffy at Sun.COM Sun May 15 11:27:16 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Sun, 15 May 2005 11:27:16 -0700 Subject: [openib-general] [PATCH][4/19] kDAPL: Remove typedef DAT_HANDLE_TYPE In-Reply-To: <11161816361781@sun.com> References: <11161816352658@sun.com> <11161816351137@sun.com> <11161816352496@sun.com> <11161816361781@sun.com> Message-ID: <1116181636449@sun.com> Signed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel3/dat/dat.h linux-kernel4/dat/dat.h --- linux-kernel3/dat/dat.h 2005-05-13 12:33:26.944001000 -0700 +++ linux-kernel4/dat/dat.h 2005-05-13 12:42:37.630017000 -0700 @@ -334,7 +334,7 @@ enum dat_return_subtype { #define DAT_GET_SUBTYPE(status) ((u32)(status) & DAT_SUBTYPE_MASK) /* dat handle types */ -typedef enum dat_handle_type { +enum dat_handle_type { DAT_HANDLE_TYPE_CR, DAT_HANDLE_TYPE_EP, DAT_HANDLE_TYPE_EVD, @@ -345,7 +345,7 @@ typedef enum dat_handle_type { DAT_HANDLE_TYPE_RMR, DAT_HANDLE_TYPE_RSP, DAT_HANDLE_TYPE_SRQ -} DAT_HANDLE_TYPE; +}; #define DAT_NAME_MAX_LENGTH 256 @@ -1108,7 +1108,7 @@ typedef u32 (*DAT_SET_CONSUMER_CONTEXT_F typedef u32 (*DAT_GET_CONSUMER_CONTEXT_FUNC)(DAT_HANDLE, DAT_CONTEXT *); -typedef u32 (*DAT_GET_HANDLE_TYPE_FUNC)(DAT_HANDLE, DAT_HANDLE_TYPE *); +typedef u32 (*DAT_GET_HANDLE_TYPE_FUNC)(DAT_HANDLE, enum dat_handle_type *); typedef u32 (*DAT_CR_QUERY_FUNC)(DAT_CR_HANDLE, struct dat_cr_param *); @@ -1404,7 +1404,8 @@ static inline u32 dat_get_consumer_conte context); } -static inline u32 dat_get_handle_type(DAT_HANDLE handle, DAT_HANDLE_TYPE *type) +static inline u32 dat_get_handle_type(DAT_HANDLE handle, + enum dat_handle_type *type) { return DAT_CALL_PROVIDER_FUNC(get_handle_type_func, handle, type); } diff -Nurp -X /home/tduffy/dontdiff linux-kernel3/dat-provider/dapl_get_handle_type.c linux-kernel4/dat-provider/dapl_get_handle_type.c --- linux-kernel3/dat-provider/dapl_get_handle_type.c 2005-05-12 09:28:48.437008000 -0700 +++ linux-kernel4/dat-provider/dapl_get_handle_type.c 2005-05-13 12:45:25.580018000 -0700 @@ -56,7 +56,7 @@ * DAT_INVALID_PARAMETER */ -u32 dapl_get_handle_type(DAT_HANDLE dat_handle, DAT_HANDLE_TYPE *handle_type) +u32 dapl_get_handle_type(DAT_HANDLE dat_handle, enum dat_handle_type *type) { u32 dat_status = DAT_SUCCESS; DAPL_HEADER *header; @@ -76,8 +76,8 @@ u32 dapl_get_handle_type(DAT_HANDLE dat_ dat_status = DAT_ERROR(DAT_INVALID_HANDLE, 0); goto bail; } - *handle_type = header->handle_type; + *type = header->handle_type; - bail: +bail: return dat_status; } diff -Nurp -X /home/tduffy/dontdiff linux-kernel3/dat-provider/dapl.h linux-kernel4/dat-provider/dapl.h --- linux-kernel3/dat-provider/dapl.h 2005-05-13 12:33:26.930003000 -0700 +++ linux-kernel4/dat-provider/dapl.h 2005-05-13 12:43:32.918003000 -0700 @@ -224,7 +224,7 @@ struct dapl_hca { struct dapl_header { struct dat_provider *provider; /* required by DAT - must be first */ DAPL_MAGIC magic; /* magic number for verification */ - DAT_HANDLE_TYPE handle_type; /* struct type */ + enum dat_handle_type handle_type; /* struct type */ DAPL_IA *owner_ia; /* ia which owns this stuct */ DAPL_LLIST_ENTRY ia_list_entry; /* link entry on ia struct */ DAT_CONTEXT user_context; /* user context - opaque to DAPL */ @@ -489,7 +489,8 @@ extern u32 dapl_set_consumer_context(DAT extern u32 dapl_get_consumer_context(DAT_HANDLE handle, DAT_CONTEXT *context); -extern u32 dapl_get_handle_type(DAT_HANDLE handle, DAT_HANDLE_TYPE *context); +extern u32 dapl_get_handle_type(DAT_HANDLE handle, + enum dat_handle_type *type); /* CR Functions */ From Tom.Duffy at Sun.COM Sun May 15 11:27:16 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Sun, 15 May 2005 11:27:16 -0700 Subject: [openib-general] [PATCH][6/19] kDAPL: Remove typedefs DAT_CONNECT_FLAGS & DAT_CLOSE_FLAGS In-Reply-To: <11161816363220@sun.com> References: <11161816352658@sun.com> <11161816351137@sun.com> <11161816352496@sun.com> <11161816361781@sun.com> <1116181636449@sun.com> <11161816363220@sun.com> Message-ID: <11161816363515@sun.com> Signed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel5/dat/api.c linux-kernel6/dat/api.c --- linux-kernel5/dat/api.c 2005-05-13 11:01:19.254000000 -0700 +++ linux-kernel6/dat/api.c 2005-05-13 13:48:10.193960000 -0700 @@ -625,7 +625,7 @@ EXPORT_SYMBOL(dat_ia_openv); * Function: dat_ia_close ***********************************************************************/ -u32 dat_ia_close(DAT_IA_HANDLE ia_handle, DAT_CLOSE_FLAGS flags) +u32 dat_ia_close(DAT_IA_HANDLE ia_handle, enum dat_close_flags flags) { struct dat_provider *provider; struct dat_provider_attr provider_attr; diff -Nurp -X /home/tduffy/dontdiff linux-kernel5/dat/dat.h linux-kernel6/dat/dat.h --- linux-kernel5/dat/dat.h 2005-05-13 12:48:43.104001000 -0700 +++ linux-kernel6/dat/dat.h 2005-05-13 13:49:33.324968000 -0700 @@ -426,15 +426,15 @@ enum dat_qos { * FLAGS */ -typedef enum dat_connect_flags { +enum dat_connect_flags { DAT_CONNECT_DEFAULT_FLAG = 0x00, DAT_CONNECT_MULTIPATH_FLAG = 0x01 -} DAT_CONNECT_FLAGS; +}; -typedef enum dat_close_flags { +enum dat_close_flags { DAT_CLOSE_ABRUPT_FLAG = 0x00, DAT_CLOSE_GRACEFUL_FLAG = 0x01 -} DAT_CLOSE_FLAGS; +}; #define DAT_CLOSE_DEFAULT DAT_CLOSE_ABRUPT_FLAG @@ -1098,7 +1098,7 @@ typedef u32 (*DAT_IA_OPEN_FUNC)(const ch typedef u32 (*DAT_IA_OPENV_FUNC)(const char *, int, DAT_EVD_HANDLE *, DAT_IA_HANDLE *, u32, u32, boolean_t); -typedef u32 (*DAT_IA_CLOSE_FUNC)(DAT_IA_HANDLE, DAT_CLOSE_FLAGS); +typedef u32 (*DAT_IA_CLOSE_FUNC)(DAT_IA_HANDLE, enum dat_close_flags); typedef u32 (*DAT_IA_QUERY_FUNC)(DAT_IA_HANDLE, DAT_EVD_HANDLE *, struct dat_ia_attr *, @@ -1147,13 +1147,13 @@ typedef u32 (*DAT_EP_MODIFY_FUNC)(DAT_EP typedef u32 (*DAT_EP_CONNECT_FUNC)(DAT_EP_HANDLE, struct sockaddr *, DAT_CONN_QUAL, DAT_TIMEOUT, int, const void *, enum dat_qos, - DAT_CONNECT_FLAGS); + enum dat_connect_flags); typedef u32 (*DAT_EP_DUP_CONNECT_FUNC)(DAT_EP_HANDLE, DAT_EP_HANDLE, DAT_TIMEOUT, int, const void *, enum dat_qos); -typedef u32 (*DAT_EP_DISCONNECT_FUNC)(DAT_EP_HANDLE, DAT_CLOSE_FLAGS); +typedef u32 (*DAT_EP_DISCONNECT_FUNC)(DAT_EP_HANDLE, enum dat_close_flags); typedef u32 (*DAT_EP_POST_SEND_FUNC)(DAT_EP_HANDLE, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE, @@ -1375,7 +1375,7 @@ extern u32 dat_ia_openv(const char *name DAT_VERSION_MAJOR, DAT_VERSION_MINOR, \ DAT_THREADSAFE) -extern u32 dat_ia_close(DAT_IA_HANDLE, DAT_CLOSE_FLAGS); +extern u32 dat_ia_close(DAT_IA_HANDLE, enum dat_close_flags); extern u32 dat_strerror(u32, const char **, const char **); @@ -1498,7 +1498,7 @@ static inline u32 dat_ep_connect(DAT_EP_ DAT_CONN_QUAL conn_qual, DAT_TIMEOUT timeout, int private_data_size, const void *private_data, enum dat_qos qos, - DAT_CONNECT_FLAGS flags) + enum dat_connect_flags flags) { return DAT_CALL_PROVIDER_FUNC(ep_connect_func, ep, ia_addr, conn_qual, timeout, private_data_size, private_data, @@ -1528,7 +1528,8 @@ static inline u32 dat_ep_create_with_srq out_evd, connect_evd, srq, attr, ep); } -static inline u32 dat_ep_disconnect(DAT_EP_HANDLE ep, DAT_CLOSE_FLAGS flags) +static inline u32 dat_ep_disconnect(DAT_EP_HANDLE ep, + enum dat_close_flags flags) { return DAT_CALL_PROVIDER_FUNC(ep_disconnect_func, ep, flags); } diff -Nurp -X /home/tduffy/dontdiff linux-kernel5/dat-provider/dapl_adapter_util.h linux-kernel6/dat-provider/dapl_adapter_util.h --- linux-kernel5/dat-provider/dapl_adapter_util.h 2005-05-13 11:38:08.141004000 -0700 +++ linux-kernel6/dat-provider/dapl_adapter_util.h 2005-05-13 13:48:10.178969000 -0700 @@ -71,7 +71,7 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han DAT_CONN_QUAL remote_conn_qual, int private_data_size, void *private_data); -u32 dapl_ib_disconnect(DAPL_EP *ep_ptr, DAT_CLOSE_FLAGS close_flags); +u32 dapl_ib_disconnect(DAPL_EP *ep_ptr, enum dat_close_flags close_flags); u32 dapl_ib_setup_conn_listener(DAPL_IA *ia_ptr, u64 ServiceID, DAPL_SP *sp_ptr); diff -Nurp -X /home/tduffy/dontdiff linux-kernel5/dat-provider/dapl_ep_connect.c linux-kernel6/dat-provider/dapl_ep_connect.c --- linux-kernel5/dat-provider/dapl_ep_connect.c 2005-05-13 12:33:26.934000000 -0700 +++ linux-kernel6/dat-provider/dapl_ep_connect.c 2005-05-13 13:47:43.558966000 -0700 @@ -46,7 +46,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han struct sockaddr *remote_ia_address, DAT_CONN_QUAL remote_conn_qual, DAT_TIMEOUT timeout, int private_data_size, const void *private_data, - enum dat_qos qos, DAT_CONNECT_FLAGS connect_flags) + enum dat_qos qos, enum dat_connect_flags connect_flags) { DAPL_EP *ep_ptr; DAPL_EP alloc_ep; diff -Nurp -X /home/tduffy/dontdiff linux-kernel5/dat-provider/dapl_ep_disconnect.c linux-kernel6/dat-provider/dapl_ep_disconnect.c --- linux-kernel5/dat-provider/dapl_ep_disconnect.c 2005-05-12 09:28:48.679067000 -0700 +++ linux-kernel6/dat-provider/dapl_ep_disconnect.c 2005-05-13 13:48:10.181972000 -0700 @@ -64,7 +64,7 @@ * DAT_INVALID_PARAMETER */ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_handle, - DAT_CLOSE_FLAGS disconnect_flags) + enum dat_close_flags disconnect_flags) { DAPL_EP *ep_ptr; DAPL_EVD *evd_ptr; diff -Nurp -X /home/tduffy/dontdiff linux-kernel5/dat-provider/dapl.h linux-kernel6/dat-provider/dapl.h --- linux-kernel5/dat-provider/dapl.h 2005-05-13 12:48:23.531003000 -0700 +++ linux-kernel6/dat-provider/dapl.h 2005-05-13 13:48:10.174963000 -0700 @@ -478,7 +478,7 @@ extern u32 dapl_ia_open(const char *name DAT_EVD_HANDLE *async_evd_handle_ptr, DAT_IA_HANDLE *ia_handle_ptr); -extern u32 dapl_ia_close(DAT_IA_HANDLE, DAT_CLOSE_FLAGS); +extern u32 dapl_ia_close(DAT_IA_HANDLE, enum dat_close_flags); extern u32 dapl_ia_query(DAT_IA_HANDLE, DAT_EVD_HANDLE *, struct dat_ia_attr *, struct dat_provider_attr *); @@ -562,7 +562,7 @@ extern u32 dapl_ep_connect(DAT_EP_HANDLE int private_data_size, const void *private_data, enum dat_qos quality_of_service, - DAT_CONNECT_FLAGS connect_flags); + enum dat_connect_flags connect_flags); extern u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, DAT_EP_HANDLE ep_dup_handle, DAT_TIMEOUT timeout, @@ -571,7 +571,7 @@ extern u32 dapl_ep_dup_connect(DAT_EP_HA enum dat_qos quality_of_service); extern u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_handle, - DAT_CLOSE_FLAGS close_flags); + enum dat_close_flags close_flags); extern u32 dapl_ep_post_send(DAT_EP_HANDLE ep_handle, int num_segments, struct dat_lmr_triplet *local_iov, diff -Nurp -X /home/tduffy/dontdiff linux-kernel5/dat-provider/dapl_ia_close.c linux-kernel6/dat-provider/dapl_ia_close.c --- linux-kernel5/dat-provider/dapl_ia_close.c 2005-05-12 09:28:48.154001000 -0700 +++ linux-kernel6/dat-provider/dapl_ia_close.c 2005-05-13 13:48:10.183967000 -0700 @@ -57,7 +57,7 @@ * DAT_INSUFFICIENT_RESOURCES * DAT_INVALID_PARAMETER */ -u32 dapl_ia_close(DAT_IA_HANDLE ia_handle, DAT_CLOSE_FLAGS ia_flags) +u32 dapl_ia_close(DAT_IA_HANDLE ia_handle, enum dat_close_flags ia_flags) { DAPL_IA *ia_ptr; u32 dat_status; diff -Nurp -X /home/tduffy/dontdiff linux-kernel5/dat-provider/dapl_openib_cm.c linux-kernel6/dat-provider/dapl_openib_cm.c --- linux-kernel5/dat-provider/dapl_openib_cm.c 2005-05-13 11:10:25.802007000 -0700 +++ linux-kernel6/dat-provider/dapl_openib_cm.c 2005-05-13 13:48:10.189990000 -0700 @@ -484,7 +484,7 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han * DAT_INVALID_PARAMETER * */ -u32 dapl_ib_disconnect(DAPL_EP *ep_ptr, DAT_CLOSE_FLAGS close_flags) +u32 dapl_ib_disconnect(DAPL_EP *ep_ptr, enum dat_close_flags close_flags) { struct dapl_cm_id *conn = ep_ptr->cm_handle; int status; From Tom.Duffy at Sun.COM Sun May 15 11:27:17 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Sun, 15 May 2005 11:27:17 -0700 Subject: [openib-general] [PATCH][7/19] kDAPL: Remove typedef DAT_EVD_FLAGS In-Reply-To: <11161816363515@sun.com> References: <11161816352658@sun.com> <11161816351137@sun.com> <11161816352496@sun.com> <11161816361781@sun.com> <1116181636449@sun.com> <11161816363220@sun.com> <11161816363515@sun.com> Message-ID: <11161816371774@sun.com> Signed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel6/dat/dat.h linux-kernel7/dat/dat.h --- linux-kernel6/dat/dat.h 2005-05-13 13:49:33.324968000 -0700 +++ linux-kernel7/dat/dat.h 2005-05-13 13:53:31.679965000 -0700 @@ -438,7 +438,7 @@ enum dat_close_flags { #define DAT_CLOSE_DEFAULT DAT_CLOSE_ABRUPT_FLAG -typedef enum dat_evd_flags { +enum dat_evd_flags { DAT_EVD_SOFTWARE_FLAG = 0x001, DAT_EVD_CR_FLAG = 0x010, DAT_EVD_DTO_FLAG = 0x020, @@ -448,7 +448,7 @@ typedef enum dat_evd_flags { /* DAT events only, no software events */ DAT_EVD_DEFAULT_FLAG = 0x1F0 -} DAT_EVD_FLAGS; +}; typedef enum dat_psp_flags { DAT_PSP_CONSUMER_FLAG = 0x00, /* Consumer creates an Endpoint */ @@ -945,7 +945,7 @@ struct dat_evd_param { int evd_qlen; DAT_UPCALL_POLICY upcall_policy; struct dat_upcall_object upcall; - DAT_EVD_FLAGS evd_flags; + enum dat_evd_flags evd_flags; }; /* @@ -1048,7 +1048,7 @@ typedef enum dat_ep_creator_for_psp { * and column represents an event stream type. An entry is a 1 if the event * streams of its row and column can feed the same EVD, and 0 otherwise. The * order of the event streams in each row and column is the same as in the - * definition of DAT_EVD_FLAGS: index 0 - Software Event, 1- Connection Request, + * definition of dat_evd_flags: index 0 - Software Event, 1- Connection Request, * 2 - DTO Completion, 3 - Connection event, 4 - RMR Bind Completion, * 5 - Asynchronous event. By definition each diagonal entry is 1. * Consumer allocates an array for it and passes it IN as a pointer @@ -1260,7 +1260,7 @@ typedef u32 (*DAT_IA_MEMTYPE_HINT_FUNC)( typedef u32 (*DAT_EVD_KCREATE_FUNC) (DAT_IA_HANDLE, int, DAT_UPCALL_POLICY, const struct dat_upcall_object *, - DAT_EVD_FLAGS, DAT_EVD_HANDLE *); + enum dat_evd_flags, DAT_EVD_HANDLE *); typedef u32 (*DAT_EVD_MODIFY_UPCALL_FUNC)(DAT_EVD_HANDLE, DAT_UPCALL_POLICY, const struct dat_upcall_object *); @@ -1464,7 +1464,8 @@ static inline u32 dat_evd_free(DAT_EVD_H static inline u32 dat_evd_kcreate(DAT_IA_HANDLE ia, int qlen, DAT_UPCALL_POLICY policy, const struct dat_upcall_object *upcall, - DAT_EVD_FLAGS flags, DAT_EVD_HANDLE * evd) + enum dat_evd_flags flags, + DAT_EVD_HANDLE * evd) { return DAT_CALL_PROVIDER_FUNC(evd_kcreate_func, ia, qlen, policy, upcall, flags, evd); diff -Nurp -X /home/tduffy/dontdiff linux-kernel6/dat-provider/dapl_evd_kcreate.c linux-kernel7/dat-provider/dapl_evd_kcreate.c --- linux-kernel6/dat-provider/dapl_evd_kcreate.c 2005-05-13 10:10:33.768971000 -0700 +++ linux-kernel7/dat-provider/dapl_evd_kcreate.c 2005-05-13 13:52:08.753001000 -0700 @@ -51,7 +51,7 @@ u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_handle, int evd_min_qlen, DAT_UPCALL_POLICY upcall_policy, /* ignore */ const struct dat_upcall_object *upcall, - DAT_EVD_FLAGS evd_flags, DAT_EVD_HANDLE *evd_handle) + enum dat_evd_flags evd_flags, DAT_EVD_HANDLE *evd_handle) { DAPL_IA *ia_ptr; DAPL_EVD *evd_ptr; diff -Nurp -X /home/tduffy/dontdiff linux-kernel6/dat-provider/dapl_evd_util.c linux-kernel7/dat-provider/dapl_evd_util.c --- linux-kernel6/dat-provider/dapl_evd_util.c 2005-05-13 11:10:25.793008000 -0700 +++ linux-kernel7/dat-provider/dapl_evd_util.c 2005-05-13 13:52:08.757991000 -0700 @@ -50,7 +50,7 @@ u32 dapl_evd_event_alloc(DAPL_EVD *evd_p */ u32 dapl_evd_internal_create(DAPL_IA *ia_ptr, DAPL_CNO *cno_ptr, - int min_qlen, DAT_EVD_FLAGS evd_flags, + int min_qlen, enum dat_evd_flags evd_flags, DAPL_EVD **evd_ptr_ptr) { DAPL_EVD *evd_ptr; @@ -127,7 +127,7 @@ bail: * alloc and initialize an EVD struct */ DAPL_EVD *dapl_evd_alloc(DAPL_IA * ia_ptr, DAPL_CNO * cno_ptr, - DAT_EVD_FLAGS evd_flags, int qlen) + enum dat_evd_flags evd_flags, int qlen) { DAPL_EVD *evd_ptr; diff -Nurp -X /home/tduffy/dontdiff linux-kernel6/dat-provider/dapl_evd_util.h linux-kernel7/dat-provider/dapl_evd_util.h --- linux-kernel6/dat-provider/dapl_evd_util.h 2005-05-13 11:10:25.796005000 -0700 +++ linux-kernel7/dat-provider/dapl_evd_util.h 2005-05-13 13:52:08.761013000 -0700 @@ -41,11 +41,11 @@ #include "dapl.h" u32 dapl_evd_internal_create(DAPL_IA *ia_ptr, DAPL_CNO *cno_ptr, - int min_qlen, DAT_EVD_FLAGS evd_flags, + int min_qlen, enum dat_evd_flags evd_flags, DAPL_EVD **evd_ptr_ptr); DAPL_EVD *dapl_evd_alloc(DAPL_IA * ia_ptr, DAPL_CNO * cno_ptr, - DAT_EVD_FLAGS evd_flags, int qlen); + enum dat_evd_flags evd_flags, int qlen); u32 dapl_evd_dealloc(DAPL_EVD *evd_ptr); diff -Nurp -X /home/tduffy/dontdiff linux-kernel6/dat-provider/dapl.h linux-kernel7/dat-provider/dapl.h --- linux-kernel6/dat-provider/dapl.h 2005-05-13 13:48:10.174963000 -0700 +++ linux-kernel7/dat-provider/dapl.h 2005-05-13 13:54:33.483962000 -0700 @@ -260,7 +260,7 @@ struct dapl_cno { struct dapl_evd { DAPL_HEADER header; DAPL_EVD_STATE evd_state; - DAT_EVD_FLAGS evd_flags; + enum dat_evd_flags evd_flags; /* Derived from evd_flags; see dapls_evd_internal_create. */ boolean_t evd_producer_locking_needed; @@ -513,7 +513,7 @@ extern u32 dapl_ia_memtype_hint(DAT_IA_H extern u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_handle, int evd_min_qlen, DAT_UPCALL_POLICY upcall_policy, const struct dat_upcall_object * upcall, - DAT_EVD_FLAGS evd_flags, + enum dat_evd_flags evd_flags, DAT_EVD_HANDLE * evd_handle); extern u32 dapl_evd_kquery(DAT_EVD_HANDLE evd_handle, diff -Nurp -X /home/tduffy/dontdiff linux-kernel6/dat-provider/dapl_ia_query.c linux-kernel7/dat-provider/dapl_ia_query.c --- linux-kernel6/dat-provider/dapl_ia_query.c 2005-05-12 09:28:47.538012000 -0700 +++ linux-kernel7/dat-provider/dapl_ia_query.c 2005-05-13 13:52:08.763960000 -0700 @@ -128,7 +128,7 @@ u32 dapl_ia_query(DAT_IA_HANDLE ia_handl /* * Set up evd_stream_merging_supported options. Note there is * one bit per allowable combination, using the ordinal - * position of the DAT_EVD_FLAGS as positions in the + * position of the enum dat_evd_flags as positions in the * array. e.g. * [0][0] is DAT_EVD_SOFTWARE_FLAG | DAT_EVD_SOFTWARE_FLAG, * [0][1] is DAT_EVD_SOFTWARE_FLAG | DAT_EVD_CR_FLAG, and diff -Nurp -X /home/tduffy/dontdiff linux-kernel6/test/dapltest/include/dapl_tdep.h linux-kernel7/test/dapltest/include/dapl_tdep.h --- linux-kernel6/test/dapltest/include/dapl_tdep.h 2005-05-12 09:34:59.325004000 -0700 +++ linux-kernel7/test/dapltest/include/dapl_tdep.h 2005-05-13 13:52:08.776963000 -0700 @@ -48,7 +48,7 @@ u32 DT_Tdep_evd_create (DAT_IA_HANDLE ia_handle, int evd_min_qlen, DAT_CNO_HANDLE cno_handle, - DAT_EVD_FLAGS evd_flags, + enum dat_evd_flags evd_flags, DAT_EVD_HANDLE *evd_handle_ptr); u32 diff -Nurp -X /home/tduffy/dontdiff linux-kernel6/test/dapltest/kdapl/kdapl_tdep_evd.c linux-kernel7/test/dapltest/kdapl/kdapl_tdep_evd.c --- linux-kernel6/test/dapltest/kdapl/kdapl_tdep_evd.c 2005-05-12 09:34:59.334001000 -0700 +++ linux-kernel7/test/dapltest/kdapl/kdapl_tdep_evd.c 2005-05-13 13:52:08.779966000 -0700 @@ -98,7 +98,7 @@ u32 DT_Tdep_evd_create (DAT_IA_HANDLE ia_handle, int evd_min_qlen, DAT_CNO_HANDLE cno_handle, - DAT_EVD_FLAGS evd_flags, + enum dat_evd_flags evd_flags, DAT_EVD_HANDLE *evd_handle_ptr) { diff -Nurp -X /home/tduffy/dontdiff linux-kernel6/test/dapltest/test/dapl_limit.c linux-kernel7/test/dapltest/test/dapl_limit.c --- linux-kernel6/test/dapltest/test/dapl_limit.c 2005-05-13 11:38:08.203004000 -0700 +++ linux-kernel7/test/dapltest/test/dapl_limit.c 2005-05-13 13:52:08.788964000 -0700 @@ -433,7 +433,7 @@ limit_test ( DT_Tdep_Print_Head *phead, * The abuse is not for us this time, just prep Obj_Set. */ unsigned int w=0; - DAT_EVD_FLAGS flags = ( DAT_EVD_DTO_FLAG + enum dat_evd_flags flags = ( DAT_EVD_DTO_FLAG /* | DAT_EVD_SOFTWARE_FLAG */ | DAT_EVD_CR_FLAG | DAT_EVD_RMR_BIND_FLAG ); /* not ASYNC */ @@ -484,7 +484,7 @@ limit_test ( DT_Tdep_Print_Head *phead, unsigned int count = START_COUNT; DAT_EVD_HANDLE *hdlptr = (DAT_EVD_HANDLE *) DT_Mdep_Malloc (count * sizeof (*hdlptr)); - DAT_EVD_FLAGS flags = ( DAT_EVD_DTO_FLAG + enum dat_evd_flags flags = ( DAT_EVD_DTO_FLAG | DAT_EVD_RMR_BIND_FLAG | DAT_EVD_CR_FLAG); From Tom.Duffy at Sun.COM Sun May 15 11:27:17 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Sun, 15 May 2005 11:27:17 -0700 Subject: [openib-general] [PATCH][10/19] kDAPL: Remove typedef DAT_MEM_PRIV_FLAGS In-Reply-To: <111618163779@sun.com> References: <11161816352658@sun.com> <11161816351137@sun.com> <11161816352496@sun.com> <11161816361781@sun.com> <1116181636449@sun.com> <11161816363220@sun.com> <11161816363515@sun.com> <11161816371774@sun.com> <1116181637281@sun.com> <111618163779@sun.com> Message-ID: <11161816374043@sun.com> Signed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel9/dat/dat.h linux-kernel10/dat/dat.h --- linux-kernel9/dat/dat.h 2005-05-13 14:18:22.224001000 -0700 +++ linux-kernel10/dat/dat.h 2005-05-13 14:21:13.415001000 -0700 @@ -533,14 +533,14 @@ enum dat_mem_type { }; /* Memory privileges */ -typedef enum dat_mem_priv_flags { +enum dat_mem_priv_flags { DAT_MEM_PRIV_NONE_FLAG = 0x00, DAT_MEM_PRIV_LOCAL_READ_FLAG = 0x01, DAT_MEM_PRIV_REMOTE_READ_FLAG = 0x02, DAT_MEM_PRIV_LOCAL_WRITE_FLAG = 0x10, DAT_MEM_PRIV_REMOTE_WRITE_FLAG = 0x20, DAT_MEM_PRIV_ALL_FLAG = 0x33 -} DAT_MEM_PRIV_FLAGS; +}; #define DAT_MEM_PRIV_READ_FLAG \ (DAT_MEM_PRIV_LOCAL_READ_FLAG | DAT_MEM_PRIV_REMOTE_READ_FLAG) @@ -552,7 +552,7 @@ struct dat_rmr_param { DAT_IA_HANDLE ia_handle; DAT_PZ_HANDLE pz_handle; struct dat_lmr_triplet lmr_triplet; - DAT_MEM_PRIV_FLAGS mem_priv; + enum dat_mem_priv_flags mem_priv; DAT_RMR_CONTEXT rmr_context; }; @@ -982,7 +982,7 @@ struct dat_lmr_param { DAT_REGION_DESCRIPTION region_desc; u64 length; DAT_PZ_HANDLE pz_handle; - DAT_MEM_PRIV_FLAGS mem_priv; + enum dat_mem_priv_flags mem_priv; DAT_LMR_CONTEXT lmr_context; DAT_RMR_CONTEXT rmr_context; u64 registered_size; @@ -1203,7 +1203,7 @@ typedef u32 (*DAT_RMR_CREATE_FUNC)(DAT_P typedef u32 (*DAT_RMR_QUERY_FUNC)(DAT_RMR_HANDLE, struct dat_rmr_param *); typedef u32 (*DAT_RMR_BIND_FUNC)(DAT_RMR_HANDLE, const struct dat_lmr_triplet *, - DAT_MEM_PRIV_FLAGS, DAT_EP_HANDLE, + enum dat_mem_priv_flags, DAT_EP_HANDLE, DAT_RMR_COOKIE, enum dat_completion_flags, DAT_RMR_CONTEXT *); @@ -1249,7 +1249,7 @@ typedef u32 (*DAT_SRQ_POST_RECV_FUNC)(DA typedef u32 (*DAT_LMR_KCREATE_FUNC)(DAT_IA_HANDLE, enum dat_mem_type, DAT_REGION_DESCRIPTION, u64, - DAT_PZ_HANDLE, DAT_MEM_PRIV_FLAGS, + DAT_PZ_HANDLE, enum dat_mem_priv_flags, DAT_MEM_OPTIMIZE_FLAGS, DAT_LMR_HANDLE *, DAT_LMR_CONTEXT *, DAT_RMR_CONTEXT *, u64 *, u64 *); @@ -1664,7 +1664,7 @@ static inline u32 dat_rmr_query(DAT_RMR_ static inline u32 dat_rmr_bind(DAT_RMR_HANDLE rmr, const struct dat_lmr_triplet *iov, - DAT_MEM_PRIV_FLAGS mem_flags, + enum dat_mem_priv_flags mem_flags, DAT_EP_HANDLE ep, DAT_RMR_COOKIE cookie, enum dat_completion_flags comp_flags, DAT_RMR_CONTEXT *context) @@ -1776,7 +1776,7 @@ static inline u32 dat_srq_set_lw(DAT_SRQ static inline u32 dat_lmr_kcreate(DAT_IA_HANDLE ia, enum dat_mem_type type, DAT_REGION_DESCRIPTION region, u64 len, DAT_PZ_HANDLE pz, - DAT_MEM_PRIV_FLAGS privileges, + enum dat_mem_priv_flags privileges, DAT_MEM_OPTIMIZE_FLAGS optimization, DAT_LMR_HANDLE *lmr_handle, DAT_LMR_CONTEXT *lmr_context, diff -Nurp -X /home/tduffy/dontdiff linux-kernel9/dat-provider/dapl_adapter_util.h linux-kernel10/dat-provider/dapl_adapter_util.h --- linux-kernel9/dat-provider/dapl_adapter_util.h 2005-05-13 13:48:10.178969000 -0700 +++ linux-kernel10/dat-provider/dapl_adapter_util.h 2005-05-13 14:20:59.246005000 -0700 @@ -103,16 +103,16 @@ u32 dapl_ib_pd_alloc(DAPL_IA *ia_ptr, DA u32 dapl_ib_pd_free(DAPL_PZ *pz); u32 dapl_ib_mr_register(DAPL_IA *ia_ptr, DAPL_LMR *lmr, void *virt_addr, - u64 length, DAT_MEM_PRIV_FLAGS privileges); + u64 length, enum dat_mem_priv_flags privileges); u32 dapl_ib_mr_register_physical(DAPL_IA *ia_ptr, DAPL_LMR *lmr, void *phys_addr, u64 length, - DAT_MEM_PRIV_FLAGS privileges); + enum dat_mem_priv_flags privileges); u32 dapl_ib_mr_deregister(DAPL_LMR *lmr); u32 dapl_ib_mr_register_shared(DAPL_IA *ia_ptr, DAPL_LMR *lmr, - DAT_MEM_PRIV_FLAGS privileges); + enum dat_mem_priv_flags privileges); u32 dapl_ib_mw_alloc(DAPL_RMR *rmr); @@ -120,7 +120,7 @@ u32 dapl_ib_mw_free(DAPL_RMR *rmr); u32 dapl_ib_mw_bind(DAPL_RMR *rmr, DAPL_LMR *lmr, DAPL_EP *ep, DAPL_COOKIE *cookie, u64 virtual_address, - u64 length, DAT_MEM_PRIV_FLAGS mem_priv, + u64 length, enum dat_mem_priv_flags mem_priv, boolean_t is_signaled); u32 dapl_ib_mw_unbind(DAPL_RMR *rmr, DAPL_EP *ep, DAPL_COOKIE *cookie, diff -Nurp -X /home/tduffy/dontdiff linux-kernel9/dat-provider/dapl.h linux-kernel10/dat-provider/dapl.h --- linux-kernel9/dat-provider/dapl.h 2005-05-13 14:18:52.134000000 -0700 +++ linux-kernel10/dat-provider/dapl.h 2005-05-13 14:22:36.958045000 -0700 @@ -627,7 +627,7 @@ extern u32 dapl_ep_set_watermark(DAT_EP_ extern u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_handle, enum dat_mem_type mem_type, DAT_REGION_DESCRIPTION region_description, u64 length, DAT_PZ_HANDLE pz_handle, - DAT_MEM_PRIV_FLAGS privileges, + enum dat_mem_priv_flags privileges, DAT_MEM_OPT optimization, DAT_LMR_HANDLE *lmr_handle, DAT_LMR_CONTEXT *lmr_context, @@ -656,7 +656,8 @@ extern u32 dapl_rmr_query(DAT_RMR_HANDLE extern u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_handle, const struct dat_lmr_triplet *lmr_triplet, - DAT_MEM_PRIV_FLAGS mem_priv, DAT_EP_HANDLE ep_handle, + enum dat_mem_priv_flags mem_priv, + DAT_EP_HANDLE ep_handle, DAT_RMR_COOKIE user_cookie, enum dat_completion_flags completion_flags, DAT_RMR_CONTEXT *context); diff -Nurp -X /home/tduffy/dontdiff linux-kernel9/dat-provider/dapl_lmr_kcreate.c linux-kernel10/dat-provider/dapl_lmr_kcreate.c --- linux-kernel9/dat-provider/dapl_lmr_kcreate.c 2005-05-13 14:14:08.090970000 -0700 +++ linux-kernel10/dat-provider/dapl_lmr_kcreate.c 2005-05-13 14:23:13.154000000 -0700 @@ -39,7 +39,7 @@ static inline u32 dapl_lmr_create_virtual(DAPL_IA *ia, void *virt_addr, u64 length, DAPL_PZ *pz, - DAT_MEM_PRIV_FLAGS privileges, + enum dat_mem_priv_flags privileges, DAT_LMR_HANDLE *lmr_handle, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, @@ -117,7 +117,7 @@ bail: static inline u32 dapl_lmr_create_physical(DAPL_IA *ia, DAT_REGION_DESCRIPTION phys_addr, u64 page_count, DAPL_PZ *pz, - DAT_MEM_PRIV_FLAGS privileges, + enum dat_mem_priv_flags privileges, DAT_LMR_HANDLE *lmr_handle, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, @@ -192,7 +192,7 @@ bail: static inline u32 dapl_lmr_create_lmr(DAPL_IA *ia, DAPL_LMR *original_lmr, DAPL_PZ *pz, - DAT_MEM_PRIV_FLAGS privileges, + enum dat_mem_priv_flags privileges, DAT_LMR_HANDLE *lmr_handle, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, @@ -303,7 +303,8 @@ bail: */ u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_handle, enum dat_mem_type mem_type, DAT_REGION_DESCRIPTION region_description, u64 length, - DAT_PZ_HANDLE pz_handle, DAT_MEM_PRIV_FLAGS privileges, + DAT_PZ_HANDLE pz_handle, + enum dat_mem_priv_flags privileges, DAT_MEM_OPT optimization, DAT_LMR_HANDLE *lmr_handle, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address) { diff -Nurp -X /home/tduffy/dontdiff linux-kernel9/dat-provider/dapl_lmr_util.c linux-kernel10/dat-provider/dapl_lmr_util.c --- linux-kernel9/dat-provider/dapl_lmr_util.c 2005-05-13 14:14:08.094964000 -0700 +++ linux-kernel10/dat-provider/dapl_lmr_util.c 2005-05-13 14:23:32.165001000 -0700 @@ -42,7 +42,8 @@ DAPL_LMR *dapl_lmr_alloc(DAPL_IA * ia, enum dat_mem_type mem_type, DAT_REGION_DESCRIPTION region_desc, u64 length, - DAT_PZ_HANDLE pz_handle, DAT_MEM_PRIV_FLAGS mem_priv) + DAT_PZ_HANDLE pz_handle, + enum dat_mem_priv_flags mem_priv) { DAPL_LMR *lmr; diff -Nurp -X /home/tduffy/dontdiff linux-kernel9/dat-provider/dapl_lmr_util.h linux-kernel10/dat-provider/dapl_lmr_util.h --- linux-kernel9/dat-provider/dapl_lmr_util.h 2005-05-13 14:14:08.096974000 -0700 +++ linux-kernel10/dat-provider/dapl_lmr_util.h 2005-05-13 14:20:59.256013000 -0700 @@ -50,7 +50,7 @@ extern DAPL_LMR *dapl_lmr_alloc(DAPL_IA DAT_REGION_DESCRIPTION region_desc, u64 length, DAT_PZ_HANDLE pz_handle, - DAT_MEM_PRIV_FLAGS mem_priv); + enum dat_mem_priv_flags mem_priv); extern void dapl_lmr_dealloc(DAPL_LMR * lmr); diff -Nurp -X /home/tduffy/dontdiff linux-kernel9/dat-provider/dapl_openib_util.c linux-kernel10/dat-provider/dapl_openib_util.c --- linux-kernel9/dat-provider/dapl_openib_util.c 2005-05-13 11:38:08.167002000 -0700 +++ linux-kernel10/dat-provider/dapl_openib_util.c 2005-05-13 14:26:12.927003000 -0700 @@ -111,7 +111,7 @@ static struct ib_status_map { * Convert DAPL mem privileges to provider equivalents * * Input: - * DAT_MEM_PRIV_FLAGS + * enum dat_mem_priv_flags * * Output: * none @@ -120,21 +120,20 @@ static struct ib_status_map { * provider equivalent memory flags * */ -static __inline__ int -dapl_ib_convert_mem_privileges(DAT_MEM_PRIV_FLAGS privileges) +static inline int dapl_ib_convert_mem_privileges(enum dat_mem_priv_flags priv) { enum ib_access_flags value = 0; /* - * if (DAT_MEM_PRIV_LOCAL_READ_FLAG & privileges) do nothing + * if (DAT_MEM_PRIV_LOCAL_READ_FLAG & priv) do nothing */ - if (DAT_MEM_PRIV_LOCAL_WRITE_FLAG & privileges) + if (DAT_MEM_PRIV_LOCAL_WRITE_FLAG & priv) value |= IB_ACCESS_LOCAL_WRITE; - if (DAT_MEM_PRIV_REMOTE_READ_FLAG & privileges) + if (DAT_MEM_PRIV_REMOTE_READ_FLAG & priv) value |= IB_ACCESS_REMOTE_READ; - if (DAT_MEM_PRIV_REMOTE_WRITE_FLAG & privileges) + if (DAT_MEM_PRIV_REMOTE_WRITE_FLAG & priv) value |= IB_ACCESS_REMOTE_WRITE; return value; @@ -391,7 +390,7 @@ u32 dapl_ib_pd_free(DAPL_PZ *pz) * */ u32 dapl_ib_mr_register(DAPL_IA *ia, DAPL_LMR *lmr, void *virt_addr, - u64 length, DAT_MEM_PRIV_FLAGS privileges) + u64 length, enum dat_mem_priv_flags privileges) { /* ITBD virtual memory registration ??? */ return (DAT_SUCCESS); @@ -399,7 +398,7 @@ u32 dapl_ib_mr_register(DAPL_IA *ia, DAP u32 dapl_ib_mr_register_physical(DAPL_IA *ia_ptr, DAPL_LMR *lmr, void *phys_addr, u64 length, - DAT_MEM_PRIV_FLAGS privileges) + enum dat_mem_priv_flags privileges) { int ib_status; int i; @@ -578,7 +577,7 @@ u32 dapl_ib_mw_free(DAPL_RMR *rmr) */ u32 dapl_ib_mw_bind(DAPL_RMR *rmr, DAPL_LMR *lmr, DAPL_EP *ep, DAPL_COOKIE *cookie, u64 virtual_address, - u64 length, DAT_MEM_PRIV_FLAGS mem_priv, + u64 length, enum dat_mem_priv_flags mem_priv, boolean_t is_signaled) { int ib_status; diff -Nurp -X /home/tduffy/dontdiff linux-kernel9/dat-provider/dapl_rmr_bind.c linux-kernel10/dat-provider/dapl_rmr_bind.c --- linux-kernel9/dat-provider/dapl_rmr_bind.c 2005-05-13 12:55:03.339011000 -0700 +++ linux-kernel10/dat-provider/dapl_rmr_bind.c 2005-05-13 14:20:59.266003000 -0700 @@ -48,7 +48,7 @@ static inline u32 dapl_rmr_bind_fuse(DAPL_RMR *rmr, const struct dat_lmr_triplet *lmr_triplet, - DAT_MEM_PRIV_FLAGS mem_priv, + enum dat_mem_priv_flags mem_priv, DAPL_EP *ep_ptr, DAT_RMR_COOKIE user_cookie, enum dat_completion_flags completion_flags, @@ -249,7 +249,7 @@ static inline u32 dapl_rmr_bind_unfuse(D */ u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_handle, const struct dat_lmr_triplet *lmr_triplet, - DAT_MEM_PRIV_FLAGS mem_priv, DAT_EP_HANDLE ep_handle, + enum dat_mem_priv_flags mem_priv, DAT_EP_HANDLE ep_handle, DAT_RMR_COOKIE user_cookie, enum dat_completion_flags completion_flags, DAT_RMR_CONTEXT *rmr_context) diff -Nurp -X /home/tduffy/dontdiff linux-kernel9/test/dapltest/test/dapl_bpool.c linux-kernel10/test/dapltest/test/dapl_bpool.c --- linux-kernel9/test/dapltest/test/dapl_bpool.c 2005-05-13 14:14:08.140007000 -0700 +++ linux-kernel10/test/dapltest/test/dapl_bpool.c 2005-05-13 14:20:59.279008000 -0700 @@ -209,7 +209,7 @@ DT_BpoolAlloc ( { struct dat_lmr_triplet iov; DAT_RMR_COOKIE cookie; - DAT_MEM_PRIV_FLAGS mflags; + enum dat_mem_priv_flags mflags; struct dat_rmr_bind_completion_event_data rmr_stat; /* create the RMR */ From Tom.Duffy at Sun.COM Sun May 15 11:27:17 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Sun, 15 May 2005 11:27:17 -0700 Subject: [openib-general] [PATCH][8/19] kDAPL: Remove typedef DAT_PSP_FLAGS In-Reply-To: <11161816371774@sun.com> References: <11161816352658@sun.com> <11161816351137@sun.com> <11161816352496@sun.com> <11161816361781@sun.com> <1116181636449@sun.com> <11161816363220@sun.com> <11161816363515@sun.com> <11161816371774@sun.com> Message-ID: <1116181637281@sun.com> Signed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel7/dat/dat.h linux-kernel8/dat/dat.h --- linux-kernel7/dat/dat.h 2005-05-13 13:53:31.679965000 -0700 +++ linux-kernel8/dat/dat.h 2005-05-13 14:08:19.213960000 -0700 @@ -450,10 +450,10 @@ enum dat_evd_flags { DAT_EVD_DEFAULT_FLAG = 0x1F0 }; -typedef enum dat_psp_flags { +enum dat_psp_flags { DAT_PSP_CONSUMER_FLAG = 0x00, /* Consumer creates an Endpoint */ DAT_PSP_PROVIDER_FLAG = 0x01 /* Provider creates an Endpoint */ -} DAT_PSP_FLAGS; +}; /* * Memory Buffers @@ -684,7 +684,7 @@ struct dat_psp_param { DAT_IA_HANDLE ia_handle; DAT_CONN_QUAL conn_qual; DAT_EVD_HANDLE evd_handle; - DAT_PSP_FLAGS psp_flags; + enum dat_psp_flags psp_flags; }; /* RSP Parameters */ @@ -1210,10 +1210,10 @@ typedef u32 (*DAT_RMR_BIND_FUNC)(DAT_RMR typedef u32 (*DAT_RMR_FREE_FUNC)(DAT_RMR_HANDLE); typedef u32 (*DAT_PSP_CREATE_FUNC)(DAT_IA_HANDLE, DAT_CONN_QUAL, DAT_EVD_HANDLE, - DAT_PSP_FLAGS, DAT_PSP_HANDLE *); + enum dat_psp_flags, DAT_PSP_HANDLE *); typedef u32 (*DAT_PSP_CREATE_ANY_FUNC)(DAT_IA_HANDLE, DAT_CONN_QUAL *, - DAT_EVD_HANDLE, DAT_PSP_FLAGS, + DAT_EVD_HANDLE, enum dat_psp_flags, DAT_PSP_HANDLE *); typedef u32 (*DAT_PSP_QUERY_FUNC)(DAT_PSP_HANDLE, struct dat_psp_param *); @@ -1678,7 +1678,7 @@ static inline u32 dat_rmr_free(DAT_RMR_H } static inline u32 dat_psp_create(DAT_IA_HANDLE ia, DAT_CONN_QUAL conn_qual, - DAT_EVD_HANDLE evd, DAT_PSP_FLAGS flags, + DAT_EVD_HANDLE evd, enum dat_psp_flags flags, DAT_PSP_HANDLE *psp) { return DAT_CALL_PROVIDER_FUNC(psp_create_func, ia, conn_qual, evd, @@ -1686,7 +1686,8 @@ static inline u32 dat_psp_create(DAT_IA_ } static inline u32 dat_psp_create_any(DAT_IA_HANDLE ia, DAT_CONN_QUAL *conn_qual, - DAT_EVD_HANDLE evd, DAT_PSP_FLAGS flags, + DAT_EVD_HANDLE evd, + enum dat_psp_flags flags, DAT_PSP_HANDLE *psp) { return DAT_CALL_PROVIDER_FUNC(psp_create_any_func, ia, conn_qual, evd, diff -Nurp -X /home/tduffy/dontdiff linux-kernel7/dat-provider/dapl.h linux-kernel8/dat-provider/dapl.h --- linux-kernel7/dat-provider/dapl.h 2005-05-13 13:54:33.483962000 -0700 +++ linux-kernel8/dat-provider/dapl.h 2005-05-13 14:09:01.172975000 -0700 @@ -386,7 +386,7 @@ struct dapl_sp { DAT_IA_HANDLE ia_handle; DAT_CONN_QUAL conn_qual; DAT_EVD_HANDLE evd_handle; - DAT_PSP_FLAGS psp_flags; + enum dat_psp_flags psp_flags; DAT_EP_HANDLE ep_handle; /* maintenence fields */ @@ -665,13 +665,14 @@ extern u32 dapl_rmr_free(DAT_RMR_HANDLE) /* PSP Functions */ extern u32 dapl_psp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, - DAT_EVD_HANDLE evd_handle, DAT_PSP_FLAGS psp_flags, + DAT_EVD_HANDLE evd_handle, + enum dat_psp_flags psp_flags, DAT_PSP_HANDLE *psp_handle); extern u32 dapl_psp_create_any(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL *conn_qual, DAT_EVD_HANDLE evd_handle, - DAT_PSP_FLAGS psp_flags, + enum dat_psp_flags psp_flags, DAT_PSP_HANDLE *psp_handle); extern u32 dapl_psp_query(DAT_PSP_HANDLE, struct dat_psp_param *); diff -Nurp -X /home/tduffy/dontdiff linux-kernel7/dat-provider/dapl_psp_create_any.c linux-kernel8/dat-provider/dapl_psp_create_any.c --- linux-kernel7/dat-provider/dapl_psp_create_any.c 2005-05-12 09:28:47.294000000 -0700 +++ linux-kernel8/dat-provider/dapl_psp_create_any.c 2005-05-13 14:07:18.810968000 -0700 @@ -71,7 +71,7 @@ * DAT_MODEL_NOT_SUPPORTED */ u32 dapl_psp_create_any(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL *conn_qual, - DAT_EVD_HANDLE evd_handle, DAT_PSP_FLAGS psp_flags, + DAT_EVD_HANDLE evd_handle, enum dat_psp_flags psp_flags, DAT_PSP_HANDLE *psp_handle) { DAPL_IA *ia_ptr; diff -Nurp -X /home/tduffy/dontdiff linux-kernel7/dat-provider/dapl_psp_create.c linux-kernel8/dat-provider/dapl_psp_create.c --- linux-kernel7/dat-provider/dapl_psp_create.c 2005-05-12 09:28:47.016004000 -0700 +++ linux-kernel8/dat-provider/dapl_psp_create.c 2005-05-13 14:07:18.807965000 -0700 @@ -68,7 +68,7 @@ * DAT_MODEL_NOT_SUPPORTED */ u32 dapl_psp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, - DAT_EVD_HANDLE evd_handle, DAT_PSP_FLAGS psp_flags, + DAT_EVD_HANDLE evd_handle, enum dat_psp_flags psp_flags, DAT_PSP_HANDLE *psp_handle) { DAPL_IA *ia_ptr; From Tom.Duffy at Sun.COM Sun May 15 11:27:17 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Sun, 15 May 2005 11:27:17 -0700 Subject: [openib-general] [PATCH][9/19] kDAPL: Remove typedef DAT_MEM_TYPE In-Reply-To: <1116181637281@sun.com> References: <11161816352658@sun.com> <11161816351137@sun.com> <11161816352496@sun.com> <11161816361781@sun.com> <1116181636449@sun.com> <11161816363220@sun.com> <11161816363515@sun.com> <11161816371774@sun.com> <1116181637281@sun.com> Message-ID: <111618163779@sun.com> Signed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel8/dat/dat.h linux-kernel9/dat/dat.h --- linux-kernel8/dat/dat.h 2005-05-13 14:08:19.213960000 -0700 +++ linux-kernel9/dat/dat.h 2005-05-13 14:18:22.224001000 -0700 @@ -523,14 +523,14 @@ struct dat_rmr_triplet { u64 segment_length; }; -typedef enum dat_mem_type { +enum dat_mem_type { DAT_MEM_TYPE_VIRTUAL = 0x00, DAT_MEM_TYPE_LMR = 0x01, DAT_MEM_TYPE_PHYSICAL = 0x10, DAT_MEM_TYPE_PLATFORM = 0x20, DAT_MEM_TYPE_IA = 0x40, DAT_MEM_TYPE_BYPASS = 0x80 -} DAT_MEM_TYPE; +}; /* Memory privileges */ typedef enum dat_mem_priv_flags { @@ -978,7 +978,7 @@ typedef union dat_region_description { /* LMR Arguments */ struct dat_lmr_param { DAT_IA_HANDLE ia_handle; - DAT_MEM_TYPE mem_type; + enum dat_mem_type mem_type; DAT_REGION_DESCRIPTION region_desc; u64 length; DAT_PZ_HANDLE pz_handle; @@ -1062,7 +1062,7 @@ struct dat_provider_attr { u32 provider_version_minor; u32 dat_version_major; u32 dat_version_minor; - DAT_MEM_TYPE lmr_mem_types_supported; + enum dat_mem_type lmr_mem_types_supported; DAT_IOV_OWNERSHIP iov_ownership_on_return; enum dat_qos dat_qos_supported; enum dat_completion_flags completion_flags_supported; @@ -1247,14 +1247,14 @@ typedef u32 (*DAT_SRQ_RESIZE_FUNC)(DAT_S typedef u32 (*DAT_SRQ_POST_RECV_FUNC)(DAT_SRQ_HANDLE, int, struct dat_lmr_triplet *, DAT_DTO_COOKIE); -typedef u32 (*DAT_LMR_KCREATE_FUNC)(DAT_IA_HANDLE, DAT_MEM_TYPE, +typedef u32 (*DAT_LMR_KCREATE_FUNC)(DAT_IA_HANDLE, enum dat_mem_type, DAT_REGION_DESCRIPTION, u64, DAT_PZ_HANDLE, DAT_MEM_PRIV_FLAGS, DAT_MEM_OPTIMIZE_FLAGS, DAT_LMR_HANDLE *, DAT_LMR_CONTEXT *, DAT_RMR_CONTEXT *, u64 *, u64 *); -typedef u32 (*DAT_IA_MEMTYPE_HINT_FUNC)(DAT_IA_HANDLE, DAT_MEM_TYPE, u64, +typedef u32 (*DAT_IA_MEMTYPE_HINT_FUNC)(DAT_IA_HANDLE, enum dat_mem_type, u64, DAT_MEM_OPTIMIZE_FLAGS, u64 *, u64 *); @@ -1410,7 +1410,8 @@ static inline u32 dat_get_handle_type(DA return DAT_CALL_PROVIDER_FUNC(get_handle_type_func, handle, type); } -static inline u32 dat_ia_memtype_hint(DAT_IA_HANDLE ia, DAT_MEM_TYPE mem_type, +static inline u32 dat_ia_memtype_hint(DAT_IA_HANDLE ia, + enum dat_mem_type mem_type, u64 length, DAT_MEM_OPTIMIZE_FLAGS mem_optimize, u64 *preferred_length, @@ -1772,7 +1773,7 @@ static inline u32 dat_srq_set_lw(DAT_SRQ return DAT_CALL_PROVIDER_FUNC(srq_set_lw_func, srq, low_watermark); } -static inline u32 dat_lmr_kcreate(DAT_IA_HANDLE ia, DAT_MEM_TYPE type, +static inline u32 dat_lmr_kcreate(DAT_IA_HANDLE ia, enum dat_mem_type type, DAT_REGION_DESCRIPTION region, u64 len, DAT_PZ_HANDLE pz, DAT_MEM_PRIV_FLAGS privileges, diff -Nurp -X /home/tduffy/dontdiff linux-kernel8/dat-provider/dapl.h linux-kernel9/dat-provider/dapl.h --- linux-kernel8/dat-provider/dapl.h 2005-05-13 14:09:01.172975000 -0700 +++ linux-kernel9/dat-provider/dapl.h 2005-05-13 14:18:52.134000000 -0700 @@ -505,7 +505,8 @@ extern u32 dapl_cr_reject(DAT_CR_HANDLE extern u32 dapl_cr_handoff(DAT_CR_HANDLE cr_handle, DAT_CONN_QUAL handoff); /* EVD Functions */ -extern u32 dapl_ia_memtype_hint(DAT_IA_HANDLE ia_handle, DAT_MEM_TYPE mem_type, +extern u32 dapl_ia_memtype_hint(DAT_IA_HANDLE ia_handle, + enum dat_mem_type mem_type, u64 length, DAT_MEM_OPT mem_optimization, u64 *suggested_length, u64 *suggested_alignment); @@ -623,7 +624,7 @@ extern u32 dapl_ep_set_watermark(DAT_EP_ /* LMR functions */ -extern u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_handle, DAT_MEM_TYPE mem_type, +extern u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_handle, enum dat_mem_type mem_type, DAT_REGION_DESCRIPTION region_description, u64 length, DAT_PZ_HANDLE pz_handle, DAT_MEM_PRIV_FLAGS privileges, diff -Nurp -X /home/tduffy/dontdiff linux-kernel8/dat-provider/dapl_ia_memtype_hint.c linux-kernel9/dat-provider/dapl_ia_memtype_hint.c --- linux-kernel8/dat-provider/dapl_ia_memtype_hint.c 2005-05-13 11:38:08.145005000 -0700 +++ linux-kernel9/dat-provider/dapl_ia_memtype_hint.c 2005-05-13 14:14:08.086967000 -0700 @@ -63,7 +63,7 @@ * DAT_INVALID_HANDLE * DAT_MODEL_NOT_SUPPORTED */ -u32 dapl_ia_memtype_hint(DAT_IA_HANDLE ia_handle, DAT_MEM_TYPE mem_type, +u32 dapl_ia_memtype_hint(DAT_IA_HANDLE ia_handle, enum dat_mem_type mem_type, u64 length, DAT_MEM_OPT mem_optimization, u64 *suggested_length, u64 *suggested_alignment) diff -Nurp -X /home/tduffy/dontdiff linux-kernel8/dat-provider/dapl_lmr_kcreate.c linux-kernel9/dat-provider/dapl_lmr_kcreate.c --- linux-kernel8/dat-provider/dapl_lmr_kcreate.c 2005-05-13 12:25:34.349006000 -0700 +++ linux-kernel9/dat-provider/dapl_lmr_kcreate.c 2005-05-13 14:14:08.090970000 -0700 @@ -301,7 +301,7 @@ bail: * DAT_MODEL_NOT_SUPPORTED * */ -u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_handle, DAT_MEM_TYPE mem_type, +u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_handle, enum dat_mem_type mem_type, DAT_REGION_DESCRIPTION region_description, u64 length, DAT_PZ_HANDLE pz_handle, DAT_MEM_PRIV_FLAGS privileges, DAT_MEM_OPT optimization, DAT_LMR_HANDLE *lmr_handle, diff -Nurp -X /home/tduffy/dontdiff linux-kernel8/dat-provider/dapl_lmr_util.c linux-kernel9/dat-provider/dapl_lmr_util.c --- linux-kernel8/dat-provider/dapl_lmr_util.c 2005-05-13 11:38:08.157005000 -0700 +++ linux-kernel9/dat-provider/dapl_lmr_util.c 2005-05-13 14:14:08.094964000 -0700 @@ -39,7 +39,7 @@ #include "dapl_ia_util.h" DAPL_LMR *dapl_lmr_alloc(DAPL_IA * ia, - DAT_MEM_TYPE mem_type, + enum dat_mem_type mem_type, DAT_REGION_DESCRIPTION region_desc, u64 length, DAT_PZ_HANDLE pz_handle, DAT_MEM_PRIV_FLAGS mem_priv) diff -Nurp -X /home/tduffy/dontdiff linux-kernel8/dat-provider/dapl_lmr_util.h linux-kernel9/dat-provider/dapl_lmr_util.h --- linux-kernel8/dat-provider/dapl_lmr_util.h 2005-05-13 11:38:08.160008000 -0700 +++ linux-kernel9/dat-provider/dapl_lmr_util.h 2005-05-13 14:14:08.096974000 -0700 @@ -46,7 +46,7 @@ *********************************************************************/ extern DAPL_LMR *dapl_lmr_alloc(DAPL_IA * ia, - DAT_MEM_TYPE mem_type, + enum dat_mem_type mem_type, DAT_REGION_DESCRIPTION region_desc, u64 length, DAT_PZ_HANDLE pz_handle, diff -Nurp -X /home/tduffy/dontdiff linux-kernel8/dat-provider/dapl_mr_util.c linux-kernel9/dat-provider/dapl_mr_util.c --- linux-kernel8/dat-provider/dapl_mr_util.c 2005-05-13 12:17:35.132016000 -0700 +++ linux-kernel9/dat-provider/dapl_mr_util.c 2005-05-13 14:14:08.099977000 -0700 @@ -56,7 +56,7 @@ * */ -u64 dapl_mr_get_address(DAT_REGION_DESCRIPTION desc, DAT_MEM_TYPE type) +u64 dapl_mr_get_address(DAT_REGION_DESCRIPTION desc, enum dat_mem_type type) { DAPL_LMR *lmr; diff -Nurp -X /home/tduffy/dontdiff linux-kernel8/dat-provider/dapl_mr_util.h linux-kernel9/dat-provider/dapl_mr_util.h --- linux-kernel8/dat-provider/dapl_mr_util.h 2005-05-13 12:14:14.433011000 -0700 +++ linux-kernel9/dat-provider/dapl_mr_util.h 2005-05-13 14:19:32.745005000 -0700 @@ -40,7 +40,8 @@ #include "dapl.h" #include "dapl_hash.h" -extern u64 dapl_mr_get_address(DAT_REGION_DESCRIPTION desc, DAT_MEM_TYPE type); +extern u64 dapl_mr_get_address(DAT_REGION_DESCRIPTION desc, + enum dat_mem_type type); /* * dapl_mr_bounds_check diff -Nurp -X /home/tduffy/dontdiff linux-kernel8/test/dapltest/cmd/dapl_params.c linux-kernel9/test/dapltest/cmd/dapl_params.c --- linux-kernel8/test/dapltest/cmd/dapl_params.c 2005-05-11 08:59:09.017003000 -0700 +++ linux-kernel9/test/dapltest/cmd/dapl_params.c 2005-05-13 14:15:17.481996000 -0700 @@ -308,7 +308,7 @@ error_return: return FALSE; } -DAT_MEM_TYPE +enum dat_mem_type DT_ParseMemType (char *arg) { if (0 == strcmp (arg, "PHY")) diff -Nurp -X /home/tduffy/dontdiff linux-kernel8/test/dapltest/include/dapl_fft_cmd.h linux-kernel9/test/dapltest/include/dapl_fft_cmd.h --- linux-kernel8/test/dapltest/include/dapl_fft_cmd.h 2005-05-13 12:36:28.910012000 -0700 +++ linux-kernel9/test/dapltest/include/dapl_fft_cmd.h 2005-05-13 14:14:08.115963000 -0700 @@ -64,7 +64,7 @@ typedef struct int num_threads; /* -t */ int num_vis; /* -v */ enum dat_qos ReliabilityLevel; /* -R */ - DAT_MEM_TYPE DT_mem_type; /* -M */ + enum dat_mem_type DT_mem_type; /* -M */ } FFT_Cmd_t; #endif diff -Nurp -X /home/tduffy/dontdiff linux-kernel8/test/dapltest/include/dapl_limit_cmd.h linux-kernel9/test/dapltest/include/dapl_limit_cmd.h --- linux-kernel8/test/dapltest/include/dapl_limit_cmd.h 2005-05-13 12:36:53.685003000 -0700 +++ linux-kernel9/test/dapltest/include/dapl_limit_cmd.h 2005-05-13 14:14:08.118966000 -0700 @@ -57,7 +57,7 @@ typedef struct uint32_t debug; /* -d */ uint32_t maximum; /* -m */ uint32_t Test_List[ LIM_NUM_TESTS ]; - DAT_MEM_TYPE DT_mem_type; /* -M */ + enum dat_mem_type DT_mem_type; /* -M */ } Limit_Cmd_t; #pragma pack () diff -Nurp -X /home/tduffy/dontdiff linux-kernel8/test/dapltest/include/dapl_params.h linux-kernel9/test/dapltest/include/dapl_params.h --- linux-kernel8/test/dapltest/include/dapl_params.h 2005-05-13 12:33:26.960013000 -0700 +++ linux-kernel9/test/dapltest/include/dapl_params.h 2005-05-13 14:14:08.120970000 -0700 @@ -66,7 +66,7 @@ typedef struct void * phead; boolean_t local_is_little_endian; boolean_t debug; - DAT_MEM_TYPE mem_type; + enum dat_mem_type mem_type; Transaction_Stats2_t Client_Stats_T; Performance_Stats_t Client_Stats_P; } Params_t; diff -Nurp -X /home/tduffy/dontdiff linux-kernel8/test/dapltest/include/dapl_performance_cmd.h linux-kernel9/test/dapltest/include/dapl_performance_cmd.h --- linux-kernel8/test/dapltest/include/dapl_performance_cmd.h 2005-05-13 12:37:40.908007000 -0700 +++ linux-kernel9/test/dapltest/include/dapl_performance_cmd.h 2005-05-13 14:14:08.122977000 -0700 @@ -57,7 +57,7 @@ typedef struct { uint32_t pipeline_len; /* -p */ Performance_Cmd_Op_t op; uint32_t use_rsp; /* -r */ - DAT_MEM_TYPE DT_mem_type; /* -M */ + enum dat_mem_type DT_mem_type; /* -M */ } Performance_Cmd_t; #pragma pack () diff -Nurp -X /home/tduffy/dontdiff linux-kernel8/test/dapltest/include/dapl_proto.h linux-kernel9/test/dapltest/include/dapl_proto.h --- linux-kernel8/test/dapltest/include/dapl_proto.h 2005-05-13 12:38:11.981004000 -0700 +++ linux-kernel9/test/dapltest/include/dapl_proto.h 2005-05-13 14:14:08.128967000 -0700 @@ -535,7 +535,7 @@ const char *DT_State2Str (DAT_EP_ST enum dat_qos DT_ParseQoS (char *arg); -DAT_MEM_TYPE DT_ParseMemType (char *arg); +enum dat_mem_type DT_ParseMemType (char *arg); unsigned char *DT_AlignPtr (void * val, int align); diff -Nurp -X /home/tduffy/dontdiff linux-kernel8/test/dapltest/include/dapl_quit_cmd.h linux-kernel9/test/dapltest/include/dapl_quit_cmd.h --- linux-kernel8/test/dapltest/include/dapl_quit_cmd.h 2005-05-13 12:33:26.970006000 -0700 +++ linux-kernel9/test/dapltest/include/dapl_quit_cmd.h 2005-05-13 14:14:08.130971000 -0700 @@ -35,7 +35,7 @@ typedef struct char device_name[256]; /* -D */ uint32_t debug; /* -d */ enum dat_qos ReliabilityLevel; /* -R */ - DAT_MEM_TYPE DT_mem_type; /* -M */ + enum dat_mem_type DT_mem_type; /* -M */ } Quit_Cmd_t; #pragma pack () diff -Nurp -X /home/tduffy/dontdiff linux-kernel8/test/dapltest/include/dapl_server_cmd.h linux-kernel9/test/dapltest/include/dapl_server_cmd.h --- linux-kernel8/test/dapltest/include/dapl_server_cmd.h 2005-05-13 12:38:35.892014000 -0700 +++ linux-kernel9/test/dapltest/include/dapl_server_cmd.h 2005-05-13 14:14:08.133963000 -0700 @@ -35,7 +35,7 @@ typedef struct boolean_t debug; /* -d */ char dapl_name[256]; /* -D device name */ enum dat_qos ReliabilityLevel; /* -R */ - DAT_MEM_TYPE DT_mem_type; /* -M */ + enum dat_mem_type DT_mem_type; /* -M */ } Server_Cmd_t; #pragma pack () diff -Nurp -X /home/tduffy/dontdiff linux-kernel8/test/dapltest/include/dapl_transaction_cmd.h linux-kernel9/test/dapltest/include/dapl_transaction_cmd.h --- linux-kernel8/test/dapltest/include/dapl_transaction_cmd.h 2005-05-13 12:33:26.976005000 -0700 +++ linux-kernel9/test/dapltest/include/dapl_transaction_cmd.h 2005-05-13 14:14:08.135988000 -0700 @@ -58,7 +58,7 @@ typedef struct { enum dat_qos ReliabilityLevel; uint32_t num_ops; Transaction_Cmd_Op_t op[MAX_OPS]; - DAT_MEM_TYPE DT_mem_type; /* -M */ + enum dat_mem_type DT_mem_type; /* -M */ } Transaction_Cmd_t; #pragma pack () diff -Nurp -X /home/tduffy/dontdiff linux-kernel8/test/dapltest/test/dapl_bpool.c linux-kernel9/test/dapltest/test/dapl_bpool.c --- linux-kernel8/test/dapltest/test/dapl_bpool.c 2005-05-13 11:38:08.187002000 -0700 +++ linux-kernel9/test/dapltest/test/dapl_bpool.c 2005-05-13 14:14:08.140007000 -0700 @@ -103,7 +103,7 @@ DT_BpoolAlloc ( DAT_REGION_DESCRIPTION region; u32 ret = DAT_SUCCESS; u64 bp_len; - DAT_MEM_TYPE DT_mem_type; + enum dat_mem_type DT_mem_type; u64 buf_region[10]; if (pt_ptr != NULL) { DT_mem_type = pt_ptr->Params.mem_type; From Tom.Duffy at Sun.COM Sun May 15 11:27:18 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Sun, 15 May 2005 11:27:18 -0700 Subject: [openib-general] [PATCH][11/19] kDAPL: Remove typedef DAT_EP_STATE In-Reply-To: <11161816374043@sun.com> References: <11161816352658@sun.com> <11161816351137@sun.com> <11161816352496@sun.com> <11161816361781@sun.com> <1116181636449@sun.com> <11161816363220@sun.com> <11161816363515@sun.com> <11161816371774@sun.com> <1116181637281@sun.com> <111618163779@sun.com> <11161816374043@sun.com> Message-ID: <11161816381329@sun.com> Signed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel10/dat/dat.h linux-kernel11/dat/dat.h --- linux-kernel10/dat/dat.h 2005-05-13 14:21:13.415001000 -0700 +++ linux-kernel11/dat/dat.h 2005-05-13 14:32:57.034000000 -0700 @@ -579,7 +579,7 @@ struct dat_ep_attr { }; /* Endpoint Parameters */ -typedef enum dat_ep_state { +enum dat_ep_state { DAT_EP_STATE_UNCONNECTED, /* quiescent state */ DAT_EP_STATE_UNCONFIGURED_UNCONNECTED, DAT_EP_STATE_RESERVED, @@ -593,11 +593,11 @@ typedef enum dat_ep_state { DAT_EP_STATE_DISCONNECT_PENDING, DAT_EP_STATE_DISCONNECTED, DAT_EP_STATE_COMPLETION_PENDING -} DAT_EP_STATE; +}; struct dat_ep_param { DAT_IA_HANDLE ia_handle; - DAT_EP_STATE ep_state; + enum dat_ep_state ep_state; struct sockaddr *local_ia_address_ptr; DAT_PORT_QUAL local_port_qual; struct sockaddr *remote_ia_address_ptr; @@ -1175,7 +1175,7 @@ typedef u32 (*DAT_EP_POST_RDMA_WRITE_FUN const struct dat_rmr_triplet *, enum dat_completion_flags); -typedef u32 (*DAT_EP_GET_STATUS_FUNC)(DAT_EP_HANDLE, DAT_EP_STATE *, +typedef u32 (*DAT_EP_GET_STATUS_FUNC)(DAT_EP_HANDLE, enum dat_ep_state *, boolean_t *, boolean_t *); typedef u32 (*DAT_EP_FREE_FUNC)(DAT_EP_HANDLE); @@ -1549,7 +1549,7 @@ static inline u32 dat_ep_free(DAT_EP_HAN return DAT_CALL_PROVIDER_FUNC(ep_free_func, ep); } -static inline u32 dat_ep_get_status(DAT_EP_HANDLE ep, DAT_EP_STATE *state, +static inline u32 dat_ep_get_status(DAT_EP_HANDLE ep, enum dat_ep_state *state, boolean_t *recv_idle, boolean_t *req_idle) { return DAT_CALL_PROVIDER_FUNC(ep_get_status_func, ep, state, recv_idle, diff -Nurp -X /home/tduffy/dontdiff linux-kernel10/dat-provider/dapl_cr_accept.c linux-kernel11/dat-provider/dapl_cr_accept.c --- linux-kernel10/dat-provider/dapl_cr_accept.c 2005-05-13 10:10:33.863960000 -0700 +++ linux-kernel11/dat-provider/dapl_cr_accept.c 2005-05-13 14:31:35.115006000 -0700 @@ -45,7 +45,7 @@ u32 dapl_cr_accept(DAT_CR_HANDLE cr_hand DAPL_EP *ep_ptr; u32 dat_status; DAPL_CR *cr_ptr; - DAT_EP_STATE entry_ep_state; + enum dat_ep_state entry_ep_state; DAT_EP_HANDLE entry_ep_handle; dapl_dbg_log(DAPL_DBG_TYPE_API, diff -Nurp -X /home/tduffy/dontdiff linux-kernel10/dat-provider/dapl_cr_reject.c linux-kernel11/dat-provider/dapl_cr_reject.c --- linux-kernel10/dat-provider/dapl_cr_reject.c 2005-05-12 09:28:47.883082000 -0700 +++ linux-kernel11/dat-provider/dapl_cr_reject.c 2005-05-13 14:31:35.118009000 -0700 @@ -44,7 +44,7 @@ u32 dapl_cr_reject(DAT_CR_HANDLE cr_hand { DAPL_CR *cr_ptr; DAPL_EP *ep_ptr; - DAT_EP_STATE entry_ep_state; + enum dat_ep_state entry_ep_state; DAT_EP_HANDLE entry_ep_handle; DAPL_SP *sp_ptr; u32 dat_status; diff -Nurp -X /home/tduffy/dontdiff linux-kernel10/dat-provider/dapl_ep_get_status.c linux-kernel11/dat-provider/dapl_ep_get_status.c --- linux-kernel10/dat-provider/dapl_ep_get_status.c 2005-05-12 09:28:47.433078000 -0700 +++ linux-kernel11/dat-provider/dapl_ep_get_status.c 2005-05-13 14:31:35.121014000 -0700 @@ -59,7 +59,7 @@ * DAT_SUCCESS * DAT_INVALID_PARAMETER */ -u32 dapl_ep_get_status(DAT_EP_HANDLE ep_handle, DAT_EP_STATE *ep_state, +u32 dapl_ep_get_status(DAT_EP_HANDLE ep_handle, enum dat_ep_state *ep_state, boolean_t *in_dto_idle, boolean_t *out_dto_idle) { DAPL_EP *ep_ptr; diff -Nurp -X /home/tduffy/dontdiff linux-kernel10/dat-provider/dapl.h linux-kernel11/dat-provider/dapl.h --- linux-kernel10/dat-provider/dapl.h 2005-05-13 14:22:36.958045000 -0700 +++ linux-kernel11/dat-provider/dapl.h 2005-05-13 14:33:34.971016000 -0700 @@ -598,7 +598,8 @@ extern u32 dapl_ep_post_rdma_write(DAT_E const struct dat_rmr_triplet *remote_iov, enum dat_completion_flags completion_flags); -extern u32 dapl_ep_get_status(DAT_EP_HANDLE ep_handle, DAT_EP_STATE *ep_state, +extern u32 dapl_ep_get_status(DAT_EP_HANDLE ep_handle, + enum dat_ep_state *ep_state, boolean_t *in_dto_idle, boolean_t *out_dto_idle); extern u32 dapl_ep_free(DAT_EP_HANDLE ep_handle); diff -Nurp -X /home/tduffy/dontdiff linux-kernel10/test/dapltest/include/dapl_proto.h linux-kernel11/test/dapltest/include/dapl_proto.h --- linux-kernel10/test/dapltest/include/dapl_proto.h 2005-05-13 14:14:08.128967000 -0700 +++ linux-kernel11/test/dapltest/include/dapl_proto.h 2005-05-13 14:31:35.137014000 -0700 @@ -531,7 +531,7 @@ const char *DT_AsyncErr2Str (DAT_EV const char *DT_EventToSTr (DAT_EVENT_NUMBER event_code); -const char *DT_State2Str (DAT_EP_STATE state_code); +const char *DT_State2Str (enum dat_ep_state state_code); enum dat_qos DT_ParseQoS (char *arg); diff -Nurp -X /home/tduffy/dontdiff linux-kernel10/test/dapltest/test/dapl_cnxn.c linux-kernel11/test/dapltest/test/dapl_cnxn.c --- linux-kernel10/test/dapltest/test/dapl_cnxn.c 2005-05-12 09:28:43.330075000 -0700 +++ linux-kernel11/test/dapltest/test/dapl_cnxn.c 2005-05-13 14:31:35.140017000 -0700 @@ -31,7 +31,7 @@ int get_ep_connection_state (DT_Tdep_Print_Head *phead, DAT_EP_HANDLE ep_handle) { - DAT_EP_STATE ep_state; + enum dat_ep_state ep_state; boolean_t in_dto_idle; boolean_t out_dto_idle; u32 ret; diff -Nurp -X /home/tduffy/dontdiff linux-kernel10/test/dapltest/test/dapl_util.c linux-kernel11/test/dapltest/test/dapl_util.c --- linux-kernel10/test/dapltest/test/dapl_util.c 2005-05-12 09:34:59.380004000 -0700 +++ linux-kernel11/test/dapltest/test/dapl_util.c 2005-05-13 14:31:35.144002000 -0700 @@ -148,7 +148,7 @@ DT_EventToSTr (DAT_EVENT_NUMBER event_co * Map DAT_EP_STATE_CODE values to readable strings */ const char * -DT_State2Str (DAT_EP_STATE state_code) +DT_State2Str (enum dat_ep_state state_code) { unsigned int i; static struct From Tom.Duffy at Sun.COM Sun May 15 11:27:18 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Sun, 15 May 2005 11:27:18 -0700 Subject: [openib-general] [PATCH][12/19] kDAPL: Remove typedef DAT_EP_PARAM_MASK In-Reply-To: <11161816381329@sun.com> References: <11161816352658@sun.com> <11161816351137@sun.com> <11161816352496@sun.com> <11161816361781@sun.com> <1116181636449@sun.com> <11161816363220@sun.com> <11161816363515@sun.com> <11161816371774@sun.com> <1116181637281@sun.com> <111618163779@sun.com> <11161816374043@sun.com> <11161816381329@sun.com> Message-ID: <1116181638840@sun.com> Signed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel11/dat/dat.h linux-kernel12/dat/dat.h --- linux-kernel11/dat/dat.h 2005-05-13 14:32:57.034000000 -0700 +++ linux-kernel12/dat/dat.h 2005-05-13 14:35:17.136006000 -0700 @@ -610,7 +610,7 @@ struct dat_ep_param { struct dat_ep_attr ep_attr; }; -typedef enum dat_ep_param_mask { +enum dat_ep_param_mask { DAT_EP_FIELD_IA_HANDLE = (1 << 0), DAT_EP_FIELD_EP_STATE = (1 << 1), DAT_EP_FIELD_LOCAL_IA_ADDRESS_PTR = (1 << 2), @@ -642,7 +642,7 @@ typedef enum dat_ep_param_mask { DAT_EP_FIELD_EP_ATTR_PROVIDER_SPECIFIC_ATTR = (1 << 28), DAT_EP_FIELD_EP_ATTR_ALL = 0x3FFFF000, DAT_EP_FIELD_ALL = 0x3FFFF7FF -} DAT_EP_PARAM_MASK; +}; #define DAT_WATERMARK_INFINITE ((int)~0) #define DAT_HW_DEFAULT DAT_WATERMARK_INFINITE @@ -1141,7 +1141,7 @@ typedef u32 (*DAT_EP_CREATE_WITH_SRQ_FUN typedef u32 (*DAT_EP_QUERY_FUNC)(DAT_EP_HANDLE, struct dat_ep_param *); -typedef u32 (*DAT_EP_MODIFY_FUNC)(DAT_EP_HANDLE, DAT_EP_PARAM_MASK, +typedef u32 (*DAT_EP_MODIFY_FUNC)(DAT_EP_HANDLE, enum dat_ep_param_mask, const struct dat_ep_param *); typedef u32 (*DAT_EP_CONNECT_FUNC)(DAT_EP_HANDLE, struct sockaddr *, @@ -1556,7 +1556,7 @@ static inline u32 dat_ep_get_status(DAT_ req_idle); } -static inline u32 dat_ep_modify(DAT_EP_HANDLE ep, DAT_EP_PARAM_MASK mask, +static inline u32 dat_ep_modify(DAT_EP_HANDLE ep, enum dat_ep_param_mask mask, const struct dat_ep_param *param) { return DAT_CALL_PROVIDER_FUNC(ep_modify_func, ep, mask, param); diff -Nurp -X /home/tduffy/dontdiff linux-kernel11/dat-provider/dapl_ep_modify.c linux-kernel12/dat-provider/dapl_ep_modify.c --- linux-kernel11/dat-provider/dapl_ep_modify.c 2005-05-12 09:28:47.996001000 -0700 +++ linux-kernel12/dat-provider/dapl_ep_modify.c 2005-05-13 14:35:01.164000000 -0700 @@ -54,7 +54,7 @@ static inline u32 dapl_ep_modify_validate_parameters(DAT_EP_HANDLE ep_handle, - DAT_EP_PARAM_MASK mask, + enum dat_ep_param_mask mask, const struct dat_ep_param *ep_param, DAPL_IA ** ia_ptr, DAPL_EP ** ep_ptr, @@ -308,7 +308,7 @@ dapl_ep_modify_validate_parameters(DAT_E * DAT_INVALID_ATTRIBUTE * DAT_INVALID_STATE */ -u32 dapl_ep_modify(DAT_EP_HANDLE ep_handle, DAT_EP_PARAM_MASK mask, +u32 dapl_ep_modify(DAT_EP_HANDLE ep_handle, enum dat_ep_param_mask mask, const struct dat_ep_param *ep_param) { DAPL_IA *ia; diff -Nurp -X /home/tduffy/dontdiff linux-kernel11/dat-provider/dapl.h linux-kernel12/dat-provider/dapl.h --- linux-kernel11/dat-provider/dapl.h 2005-05-13 14:33:34.971016000 -0700 +++ linux-kernel12/dat-provider/dapl.h 2005-05-13 14:35:01.158008000 -0700 @@ -553,7 +553,7 @@ extern u32 dapl_ep_create(DAT_IA_HANDLE extern u32 dapl_ep_query(DAT_EP_HANDLE, struct dat_ep_param *); extern u32 dapl_ep_modify(DAT_EP_HANDLE ep_handle, - DAT_EP_PARAM_MASK ep_args_mask, + enum dat_ep_param_mask ep_args_mask, const struct dat_ep_param *ep_args); extern u32 dapl_ep_connect(DAT_EP_HANDLE ep_handle, From Tom.Duffy at Sun.COM Sun May 15 11:27:18 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Sun, 15 May 2005 11:27:18 -0700 Subject: [openib-general] [PATCH][13/19] kDAPL: Remove typedef DAT_SRQ_STATE In-Reply-To: <1116181638840@sun.com> References: <11161816352658@sun.com> <11161816351137@sun.com> <11161816352496@sun.com> <11161816361781@sun.com> <1116181636449@sun.com> <11161816363220@sun.com> <11161816363515@sun.com> <11161816371774@sun.com> <1116181637281@sun.com> <111618163779@sun.com> <11161816374043@sun.com> <11161816381329@sun.com> <1116181638840@sun.com> Message-ID: <11161816381177@sun.com> Signed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel12/dat/dat.h linux-kernel13/dat/dat.h --- linux-kernel12/dat/dat.h 2005-05-13 14:35:17.136006000 -0700 +++ linux-kernel13/dat/dat.h 2005-05-13 14:37:53.695001000 -0700 @@ -648,10 +648,10 @@ enum dat_ep_param_mask { #define DAT_HW_DEFAULT DAT_WATERMARK_INFINITE #define DAT_SRQ_LW_DEFAULT 0x0 -typedef enum dat_srq_state { +enum dat_srq_state { DAT_SRQ_STATE_OPERATIONAL, DAT_SRQ_STATE_ERROR -} DAT_SRQ_STATE; +}; #define DAT_VALUE_UNKNOWN (((int) ~0)-1) @@ -663,7 +663,7 @@ struct dat_srq_attr { struct dat_srq_param { DAT_IA_HANDLE ia_handle; - DAT_SRQ_STATE srq_state; + enum dat_srq_state srq_state; DAT_PZ_HANDLE pz_handle; int max_recv_dtos; int max_recv_iov; From Tom.Duffy at Sun.COM Sun May 15 11:27:18 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Sun, 15 May 2005 11:27:18 -0700 Subject: [openib-general] [PATCH][14/19] kDAPL: Remove typedef DAT_DTO_COMPLETION_STATUS & #define DAT_RMR_BIND_COMPLETION_STATUS In-Reply-To: <11161816381177@sun.com> References: <11161816352658@sun.com> <11161816351137@sun.com> <11161816352496@sun.com> <11161816361781@sun.com> <1116181636449@sun.com> <11161816363220@sun.com> <11161816363515@sun.com> <11161816371774@sun.com> <1116181637281@sun.com> <111618163779@sun.com> <11161816374043@sun.com> <11161816381329@sun.com> <1116181638840@sun.com> <11161816381177@sun.com> Message-ID: <1116181638195@sun.com> Signed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel13/dat/dat.h linux-kernel14/dat/dat.h --- linux-kernel13/dat/dat.h 2005-05-13 14:37:53.695001000 -0700 +++ linux-kernel14/dat/dat.h 2005-05-13 14:50:25.254001000 -0700 @@ -732,7 +732,7 @@ struct dat_cr_param { /* Completion status flags */ /* DTO completion status */ -typedef enum dat_dto_completion_status { +enum dat_dto_completion_status { DAT_DTO_SUCCESS = 0, DAT_DTO_ERR_FLUSHED = 1, DAT_DTO_ERR_LOCAL_LENGTH = 2, @@ -745,7 +745,7 @@ typedef enum dat_dto_completion_status { DAT_DTO_ERR_RECEIVER_NOT_READY = 9, DAT_DTO_ERR_PARTIAL_PACKET = 10, DAT_RMR_OPERATION_FAILED = 11 -} DAT_DTO_COMPLETION_STATUS; +}; /* RMR completion status */ @@ -754,7 +754,7 @@ typedef enum dat_dto_completion_status { #define DAT_RMR_BIND_FAILURE DAT_DTO_ERR_FLUSHED /* RMR completion status */ -#define DAT_RMR_BIND_COMPLETION_STATUS DAT_DTO_COMPLETION_STATUS +#define DAT_RMR_BIND_COMPLETION_STATUS enum dat_dto_completion_status /* Completion group structs (six total) */ @@ -763,7 +763,7 @@ typedef enum dat_dto_completion_status { struct dat_dto_completion_event_data { DAT_EP_HANDLE ep_handle; DAT_DTO_COOKIE user_cookie; - DAT_DTO_COMPLETION_STATUS status; + enum dat_dto_completion_status status; u64 transfered_length; }; @@ -771,7 +771,7 @@ struct dat_dto_completion_event_data { struct dat_rmr_bind_completion_event_data { DAT_RMR_HANDLE rmr_handle; DAT_RMR_COOKIE user_cookie; - DAT_RMR_BIND_COMPLETION_STATUS status; + enum dat_dto_completion_status status; }; typedef union dat_sp_handle { diff -Nurp -X /home/tduffy/dontdiff linux-kernel13/dat-provider/dapl_adapter_util.h linux-kernel14/dat-provider/dapl_adapter_util.h --- linux-kernel13/dat-provider/dapl_adapter_util.h 2005-05-13 14:20:59.246005000 -0700 +++ linux-kernel14/dat-provider/dapl_adapter_util.h 2005-05-13 15:29:23.105005000 -0700 @@ -135,7 +135,8 @@ u32 dapl_ib_completion_poll(DAPL_HCA *hc u32 dapl_ib_completion_notify(DAPL_EVD *evd_ptr, ib_notification_type_t type); -DAT_DTO_COMPLETION_STATUS dapl_ib_get_dto_status(ib_work_completion_t *cqe_ptr); +enum dat_dto_completion_status dapl_ib_get_dto_status(ib_work_completion_t + *cqe_ptr); void dapl_ib_reinit_ep(DAPL_EP * ep_ptr); diff -Nurp -X /home/tduffy/dontdiff linux-kernel13/dat-provider/dapl_evd_util.c linux-kernel14/dat-provider/dapl_evd_util.c --- linux-kernel13/dat-provider/dapl_evd_util.c 2005-05-13 13:52:08.757991000 -0700 +++ linux-kernel14/dat-provider/dapl_evd_util.c 2005-05-13 14:47:24.445001000 -0700 @@ -628,7 +628,7 @@ dapl_evd_cqe_to_event(DAPL_EVD * evd_ptr { DAPL_EP *ep_ptr; DAPL_COOKIE *cookie; - DAT_DTO_COMPLETION_STATUS dto_status; + enum dat_dto_completion_status dto_status; /* * All that can be relied on if the status is bad is the status diff -Nurp -X /home/tduffy/dontdiff linux-kernel13/dat-provider/dapl_openib_util.c linux-kernel14/dat-provider/dapl_openib_util.c --- linux-kernel13/dat-provider/dapl_openib_util.c 2005-05-13 14:26:12.927003000 -0700 +++ linux-kernel14/dat-provider/dapl_openib_util.c 2005-05-13 15:29:49.548012000 -0700 @@ -57,7 +57,7 @@ u32 dapl_ib_get_hca_ids(ib_hca_handle_t */ static struct ib_status_map { int ib_status; - DAT_DTO_COMPLETION_STATUS dat_status; + enum dat_dto_completion_status dat_status; } ib_status_map[] = { /* 0 */ { IB_WC_SUCCESS, DAT_DTO_SUCCESS}, @@ -813,7 +813,8 @@ u32 dapl_ib_completion_notify(DAPL_EVD * * Value from ib_status_map table above */ -DAT_DTO_COMPLETION_STATUS dapl_ib_get_dto_status(ib_work_completion_t * cqe_ptr) +enum dat_dto_completion_status dapl_ib_get_dto_status(ib_work_completion_t + *cqe_ptr) { u32 ib_status; int i; From Tom.Duffy at Sun.COM Sun May 15 11:27:19 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Sun, 15 May 2005 11:27:19 -0700 Subject: [openib-general] [PATCH][15/19] kDAPL: Remove typedefs DAT_IA_ASYNC_ERROR_REASON DAT_EP_ASYNC_ERROR_REASON DAT_EVD_ASYNC_ERROR_REASON DAT_SRQ_ASYNC_ERROR_REASON DAT_LMR_ASYNC_ERROR_REASON DAT_RMR_ASYNC_ERROR_REASON & DAT_PZ_ASYNC_ERROR_REASON In-Reply-To: <1116181638195@sun.com> References: <11161816352658@sun.com> <11161816351137@sun.com> <11161816352496@sun.com> <11161816361781@sun.com> <1116181636449@sun.com> <11161816363220@sun.com> <11161816363515@sun.com> <11161816371774@sun.com> <1116181637281@sun.com> <111618163779@sun.com> <11161816374043@sun.com> <11161816381329@sun.com> <1116181638840@sun.com> <11161816381177@sun.com> <1116181638195@sun.com> Message-ID: <11161816391591@sun.com> Signed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel14/dat/dat.h linux-kernel15/dat/dat.h --- linux-kernel14/dat/dat.h 2005-05-13 14:50:25.254001000 -0700 +++ linux-kernel15/dat/dat.h 2005-05-13 15:34:14.306009000 -0700 @@ -828,39 +828,39 @@ struct dat_asynch_error_event_data { }; /* The reason is object type specific and its values are defined below. */ -typedef enum ia_async_error_reason { +enum ia_async_error_reason { DAT_IA_CATASTROPHIC_ERROR, DAT_IA_OTHER_ERROR -} DAT_IA_ASYNC_ERROR_REASON; +}; -typedef enum ep_async_error_reason { +enum ep_async_error_reason { DAT_EP_TRANSFER_TO_ERROR, DAT_EP_OTHER_ERROR, DAT_SRQ_SOFT_HIGH_WATERMARK_EVENT -} DAT_EP_ASYNC_ERROR_REASON; +}; -typedef enum evd_async_error_reason { +enum evd_async_error_reason { DAT_EVD_OVERFLOW_ERROR, DAT_EVD_OTHER_ERROR -} DAT_EVD_ASYNC_ERROR_REASON; +}; -typedef enum srq_async_error_reason { +enum srq_async_error_reason { DAT_SRQ_TRANSFER_TO_ERROR, DAT_SRQ_OTHER_ERROR, DAT_SRQ_LOW_WATERMARK_EVENT -} DAT_SRQ_ASYNC_ERROR_REASON; +}; -typedef enum lmr_async_error_reason { +enum lmr_async_error_reason { DAT_LMR_OTHER_ERROR -} DAT_LMR_ASYNC_ERROR_REASON; +}; -typedef enum rmr_async_error_reason { +enum rmr_async_error_reason { DAT_RMR_OTHER_ERROR -} DAT_RMR_ASYNC_ERROR_REASON; +}; -typedef enum pz_async_error_reason { +enum pz_async_error_reason { DAT_PZ_OTHER_ERROR -} DAT_PZ_ASYNC_ERROR_REASON; +}; /* Software event data */ struct dat_software_event_data { From Tom.Duffy at Sun.COM Sun May 15 11:27:19 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Sun, 15 May 2005 11:27:19 -0700 Subject: [openib-general] [PATCH][16/19] kDAPL: Remove typedef DAT_EVENT_NUMBER In-Reply-To: <11161816391591@sun.com> References: <11161816352658@sun.com> <11161816351137@sun.com> <11161816352496@sun.com> <11161816361781@sun.com> <1116181636449@sun.com> <11161816363220@sun.com> <11161816363515@sun.com> <11161816371774@sun.com> <1116181637281@sun.com> <111618163779@sun.com> <11161816374043@sun.com> <11161816381329@sun.com> <1116181638840@sun.com> <11161816381177@sun.com> <1116181638195@sun.com> <11161816391591@sun.com> Message-ID: <1116181639894@sun.com> Signed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel15/dat/dat.h linux-kernel16/dat/dat.h --- linux-kernel15/dat/dat.h 2005-05-13 15:34:14.306009000 -0700 +++ linux-kernel16/dat/dat.h 2005-05-13 15:37:17.695015000 -0700 @@ -867,7 +867,7 @@ struct dat_software_event_data { void *pointer; }; -typedef enum dat_event_number { +enum dat_event_number { DAT_DTO_COMPLETION_EVENT = 0x00001, DAT_RMR_BIND_COMPLETION_EVENT = 0x01001, DAT_CONNECTION_REQUEST_EVENT = 0x02001, @@ -885,7 +885,7 @@ typedef enum dat_event_number { DAT_ASYNC_ERROR_TIMED_OUT = 0x08004, DAT_ASYNC_ERROR_PROVIDER_INTERNAL_ERROR = 0x08005, DAT_SOFTWARE_EVENT = 0x10001 -} DAT_EVENT_NUMBER; +}; /* Union for event Data */ @@ -901,7 +901,7 @@ typedef union dat_event_data { /* Event struct that holds all event information */ struct dat_event { - DAT_EVENT_NUMBER event_number; + enum dat_event_number event_number; DAT_EVD_HANDLE evd_handle; DAT_EVENT_DATA event_data; }; diff -Nurp -X /home/tduffy/dontdiff linux-kernel15/dat-provider/dapl_adapter_util.h linux-kernel16/dat-provider/dapl_adapter_util.h --- linux-kernel15/dat-provider/dapl_adapter_util.h 2005-05-13 15:29:23.105005000 -0700 +++ linux-kernel16/dat-provider/dapl_adapter_util.h 2005-05-13 15:36:57.956004000 -0700 @@ -145,12 +145,12 @@ void dapl_ib_disconnect_clean(DAPL_EP * const ib_cm_events_t ib_cm_event); u32 dapl_ib_get_async_event(ib_error_record_t *cause_ptr, - DAT_EVENT_NUMBER *async_event); + enum dat_event_number *async_event); -DAT_EVENT_NUMBER dapl_ib_get_dat_event(const ib_cm_events_t ib_cm_event, +enum dat_event_number dapl_ib_get_dat_event(const ib_cm_events_t ib_cm_event, boolean_t active); -ib_cm_events_t dapl_ib_get_cm_event(DAT_EVENT_NUMBER dat_event_num); +ib_cm_events_t dapl_ib_get_cm_event(enum dat_event_number dat_event_num); u32 dapl_ib_cm_remote_addr(DAT_HANDLE dat_handle, struct sockaddr_in6 *remote_ia_address); diff -Nurp -X /home/tduffy/dontdiff linux-kernel15/dat-provider/dapl_cr_callback.c linux-kernel16/dat-provider/dapl_cr_callback.c --- linux-kernel15/dat-provider/dapl_cr_callback.c 2005-05-13 11:24:33.678004000 -0700 +++ linux-kernel16/dat-provider/dapl_cr_callback.c 2005-05-13 15:36:57.961006000 -0700 @@ -49,7 +49,7 @@ u32 dapl_connection_request(ib_cm_handle const void *prd_ptr, DAPL_EVD *evd_ptr); DAPL_EP *dapl_get_sp_ep(ib_cm_handle_t ib_cm_handle, - DAPL_SP * sp_ptr, DAT_EVENT_NUMBER dat_event_num); + DAPL_SP * sp_ptr, enum dat_event_number dat_event_num); /* * The callback function registered with verbs for passive side of @@ -63,7 +63,7 @@ void dapl_cr_callback(ib_cm_handle_t ib_ DAPL_EP *ep_ptr; DAPL_EVD *evd_ptr; DAPL_SP *sp_ptr; - DAT_EVENT_NUMBER dat_event_num; + enum dat_event_number dat_event_num; u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_CM | DAPL_DBG_TYPE_CALLBACK, @@ -375,7 +375,7 @@ u32 dapl_connection_request(ib_cm_handle * the SP. */ DAPL_EP *dapl_get_sp_ep(ib_cm_handle_t ib_cm_handle, - DAPL_SP * sp_ptr, DAT_EVENT_NUMBER dat_event_num) + DAPL_SP * sp_ptr, enum dat_event_number dat_event_num) { DAPL_CR *cr_ptr; DAPL_EP *ep_ptr; diff -Nurp -X /home/tduffy/dontdiff linux-kernel15/dat-provider/dapl_evd_connection_callb.c linux-kernel16/dat-provider/dapl_evd_connection_callb.c --- linux-kernel15/dat-provider/dapl_evd_connection_callb.c 2005-05-12 09:28:47.275056000 -0700 +++ linux-kernel16/dat-provider/dapl_evd_connection_callb.c 2005-05-13 15:36:57.964000000 -0700 @@ -67,7 +67,7 @@ dapl_evd_connection_callback(ib_cm_handl { DAPL_EP *ep_ptr; DAPL_EVD *evd_ptr; - DAT_EVENT_NUMBER dat_event_num; + enum dat_event_number dat_event_num; u32 dat_status; int private_data_size; diff -Nurp -X /home/tduffy/dontdiff linux-kernel15/dat-provider/dapl_evd_qp_async_error_callb.c linux-kernel16/dat-provider/dapl_evd_qp_async_error_callb.c --- linux-kernel15/dat-provider/dapl_evd_qp_async_error_callb.c 2005-05-12 09:28:48.181057000 -0700 +++ linux-kernel16/dat-provider/dapl_evd_qp_async_error_callb.c 2005-05-13 15:36:57.967011000 -0700 @@ -59,7 +59,7 @@ dapl_evd_qp_async_error_callback(ib_erro { DAPL_EP *ep_ptr; DAPL_EVD *async_evd; - DAT_EVENT_NUMBER async_event; + enum dat_event_number async_event; u32 dat_status; ep_ptr = (DAPL_EP *) context; diff -Nurp -X /home/tduffy/dontdiff linux-kernel15/dat-provider/dapl_evd_un_async_error_callb.c linux-kernel16/dat-provider/dapl_evd_un_async_error_callb.c --- linux-kernel15/dat-provider/dapl_evd_un_async_error_callb.c 2005-05-12 09:28:48.254002000 -0700 +++ linux-kernel16/dat-provider/dapl_evd_un_async_error_callb.c 2005-05-13 15:36:57.969015000 -0700 @@ -58,7 +58,7 @@ void dapl_evd_un_async_error_callback(ib_error_record_t * cause_ptr, void *context) { DAPL_EVD *async_evd; - DAT_EVENT_NUMBER async_event; + enum dat_event_number async_event; u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK | DAPL_DBG_TYPE_EXCEPTION, diff -Nurp -X /home/tduffy/dontdiff linux-kernel15/dat-provider/dapl_evd_util.c linux-kernel16/dat-provider/dapl_evd_util.c --- linux-kernel15/dat-provider/dapl_evd_util.c 2005-05-13 14:47:24.445001000 -0700 +++ linux-kernel16/dat-provider/dapl_evd_util.c 2005-05-13 15:40:27.444000000 -0700 @@ -480,25 +480,25 @@ dapl_evd_post_overflow_event(DAPL_EVD * return; } -static struct dat_event *dapl_evd_get_and_init_event(DAPL_EVD * evd_ptr, - DAT_EVENT_NUMBER event_number) +static struct dat_event *dapl_evd_get_and_init_event(DAPL_EVD *evd_ptr, + enum dat_event_number evno) { struct dat_event *event_ptr; event_ptr = dapl_evd_get_event(evd_ptr); - if (NULL == event_ptr) { + if (!event_ptr) dapl_evd_post_overflow_event(evd_ptr->header.owner_ia-> async_error_evd, evd_ptr); - } else { + else { event_ptr->evd_handle = (DAT_EVD_HANDLE) evd_ptr; - event_ptr->event_number = event_number; + event_ptr->event_number = evno; } return event_ptr; } u32 dapl_evd_post_cr_arrival_event(DAPL_EVD *evd_ptr, - DAT_EVENT_NUMBER event_number, + enum dat_event_number event_number, DAT_SP_HANDLE sp_handle, struct sockaddr *ia_address_ptr, DAT_CONN_QUAL conn_qual, @@ -529,7 +529,7 @@ u32 dapl_evd_post_cr_arrival_event(DAPL_ } u32 dapl_evd_post_connection_event(DAPL_EVD *evd_ptr, - DAT_EVENT_NUMBER event_number, + enum dat_event_number event_number, DAT_EP_HANDLE ep_handle, int private_data_size, void *private_data) @@ -558,7 +558,7 @@ u32 dapl_evd_post_connection_event(DAPL_ } u32 dapl_evd_post_async_error_event(DAPL_EVD *evd_ptr, - DAT_EVENT_NUMBER event_number, + enum dat_event_number event_number, DAT_IA_HANDLE ia_handle) { struct dat_event *event_ptr; @@ -583,7 +583,7 @@ u32 dapl_evd_post_async_error_event(DAPL } u32 dapl_evd_post_software_event(DAPL_EVD *evd_ptr, - DAT_EVENT_NUMBER event_number, + enum dat_event_number event_number, void *pointer) { struct dat_event *event_ptr; diff -Nurp -X /home/tduffy/dontdiff linux-kernel15/dat-provider/dapl_evd_util.h linux-kernel16/dat-provider/dapl_evd_util.h --- linux-kernel15/dat-provider/dapl_evd_util.h 2005-05-13 13:52:08.761013000 -0700 +++ linux-kernel16/dat-provider/dapl_evd_util.h 2005-05-13 15:36:57.980003000 -0700 @@ -63,24 +63,24 @@ u32 dapl_evd_event_realloc(DAPL_EVD *evd */ u32 dapl_evd_post_cr_arrival_event(DAPL_EVD *evd_ptr, - DAT_EVENT_NUMBER event_number, + enum dat_event_number event_number, DAT_SP_HANDLE sp_handle, struct sockaddr *ia_address_ptr, DAT_CONN_QUAL conn_qual, DAT_CR_HANDLE cr_handle); u32 dapl_evd_post_connection_event(DAPL_EVD *evd_ptr, - DAT_EVENT_NUMBER event_number, + enum dat_event_number event_number, DAT_EP_HANDLE ep_handle, int private_data_size, void *private_data); u32 dapl_evd_post_async_error_event(DAPL_EVD *evd_ptr, - DAT_EVENT_NUMBER event_number, + enum dat_event_number event_number, DAT_IA_HANDLE ia_handle); u32 dapl_evd_post_software_event(DAPL_EVD *evd_ptr, - DAT_EVENT_NUMBER event_number, + enum dat_event_number event_number, void *pointer); /************************************* diff -Nurp -X /home/tduffy/dontdiff linux-kernel15/dat-provider/dapl_openib_util.c linux-kernel16/dat-provider/dapl_openib_util.c --- linux-kernel15/dat-provider/dapl_openib_util.c 2005-05-13 15:29:49.548012000 -0700 +++ linux-kernel16/dat-provider/dapl_openib_util.c 2005-05-13 15:50:36.083042000 -0700 @@ -859,7 +859,7 @@ enum dat_dto_completion_status dapl_ib_g */ u32 dapl_ib_get_async_event(ib_error_record_t *cause_ptr, - DAT_EVENT_NUMBER *async_event) + enum dat_event_number *async_event) { ib_async_event_type event_id; u32 dat_status = DAT_SUCCESS; @@ -955,7 +955,7 @@ u32 dapl_ib_get_hca_ids(ib_hca_handle_t static struct ib_cm_event_map { const ib_cm_events_t ib_cm_event; - DAT_EVENT_NUMBER dat_event_num; + enum dat_event_number dat_event_num; } ib_cm_event_map[DAPL_IB_EVENT_CNT] = { {IB_CME_CONNECTED, DAT_CONNECTION_EVENT_ESTABLISHED}, {IB_CME_DISCONNECTED, DAT_CONNECTION_EVENT_DISCONNECTED}, @@ -992,44 +992,41 @@ static struct ib_cm_event_map { * none * * Returns: - * DAT_EVENT_NUMBER of translated provider value + * enum dat_event_number of translated provider value */ -DAT_EVENT_NUMBER -dapl_ib_get_dat_event(const ib_cm_events_t ib_cm_event, boolean_t active) +enum dat_event_number dapl_ib_get_dat_event(const ib_cm_events_t ib_cm_event, + boolean_t active) { - DAT_EVENT_NUMBER dat_event_num; - int index; + enum dat_event_number event = 0; int i; active = active; /* keep compiler happy */ if (ib_cm_event > IB_CME_BROKEN) - return (DAT_EVENT_NUMBER) 0; + goto out; - dat_event_num = 0; - index = ib_cm_event - IB_CME_CONNECTED; + i = ib_cm_event - IB_CME_CONNECTED; - if (ib_cm_event == ib_cm_event_map[index].ib_cm_event) - dat_event_num = ib_cm_event_map[index].dat_event_num; - else { + if (ib_cm_event == ib_cm_event_map[i].ib_cm_event) + event = ib_cm_event_map[i].dat_event_num; + else /* * Do a linear walk of the list, just in case the table has been * messed up or someone changes the definitions; paranoia, but * recoverable and robust. */ - for (i = 0; i < DAPL_IB_EVENT_CNT; i++) { + for (i = 0; i < DAPL_IB_EVENT_CNT; i++) if (ib_cm_event == ib_cm_event_map[i].ib_cm_event) - dat_event_num = - ib_cm_event_map[i].dat_event_num; - } - } + event = ib_cm_event_map[i].dat_event_num; dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK, - " dapl_ib_get_dat_event:ibapi event translation: (%s) ib_event 0x%x dat_event 0x%x\n", - active ? "active" : "passive", ib_cm_event, dat_event_num); + " dapl_ib_get_dat_event:ibapi event translation: (%s) " + "ib_event 0x%x dat_event 0x%x\n", + active ? "active" : "passive", ib_cm_event, event); - return dat_event_num; +out: + return event; } /* @@ -1051,7 +1048,7 @@ dapl_ib_get_dat_event(const ib_cm_events * Returns: * ib_cm_event of translated DAPL value */ -ib_cm_events_t dapl_ib_get_cm_event(DAT_EVENT_NUMBER dat_event_num) +ib_cm_events_t dapl_ib_get_cm_event(enum dat_event_number dat_event_num) { ib_cm_events_t ib_cm_event; int i; diff -Nurp -X /home/tduffy/dontdiff linux-kernel15/test/dapltest/include/dapl_fft_util.h linux-kernel16/test/dapltest/include/dapl_fft_util.h --- linux-kernel15/test/dapltest/include/dapl_fft_util.h 2005-05-13 11:10:25.818006000 -0700 +++ linux-kernel16/test/dapltest/include/dapl_fft_util.h 2005-05-13 15:36:58.001011000 -0700 @@ -65,7 +65,7 @@ typedef struct DAT_CR_HANDLE cr_handle; Bpool *bpool; struct dat_cr_arrival_event_data cr_stat; - DAT_EVENT_NUMBER event_num; + enum dat_event_number event_num; struct sockaddr *remote_netaddr; Per_Test_Data_t *pt_ptr; boolean_t connected; diff -Nurp -X /home/tduffy/dontdiff linux-kernel15/test/dapltest/include/dapl_proto.h linux-kernel16/test/dapltest/include/dapl_proto.h --- linux-kernel15/test/dapltest/include/dapl_proto.h 2005-05-13 14:31:35.137014000 -0700 +++ linux-kernel16/test/dapltest/include/dapl_proto.h 2005-05-13 15:36:58.007005000 -0700 @@ -368,7 +368,7 @@ boolean_t DT_post_send_buffer (DT_Td boolean_t DT_conn_event_wait (DT_Tdep_Print_Head* phead, DAT_EP_HANDLE ep_handle, DAT_EVD_HANDLE evd_handle, - DAT_EVENT_NUMBER *event_number); + enum dat_event_number *event_number); boolean_t DT_disco_event_wait ( DT_Tdep_Print_Head* phead, DAT_EVD_HANDLE evd_handle, @@ -527,9 +527,9 @@ const char *DT_RetToString (u32 ret const char *DT_TransferTypeToString (DT_Transfer_Type type); -const char *DT_AsyncErr2Str (DAT_EVENT_NUMBER error_code); +const char *DT_AsyncErr2Str (enum dat_event_number error_code); -const char *DT_EventToSTr (DAT_EVENT_NUMBER event_code); +const char *DT_EventToSTr (enum dat_event_number event_code); const char *DT_State2Str (enum dat_ep_state state_code); diff -Nurp -X /home/tduffy/dontdiff linux-kernel15/test/dapltest/test/dapl_client.c linux-kernel16/test/dapltest/test/dapl_client.c --- linux-kernel15/test/dapltest/test/dapl_client.c 2005-05-13 11:10:25.837002000 -0700 +++ linux-kernel16/test/dapltest/test/dapl_client.c 2005-05-13 15:36:58.012012000 -0700 @@ -61,7 +61,7 @@ DT_cs_Client (Params_t * params_ptr, DAT_DTO_COOKIE dto_cookie; struct dat_dto_completion_event_data dto_stat; - DAT_EVENT_NUMBER event_num; + enum dat_event_number event_num; unsigned char * buffp; u32 ret; DT_Tdep_Print_Head *phead; diff -Nurp -X /home/tduffy/dontdiff linux-kernel15/test/dapltest/test/dapl_performance_client.c linux-kernel16/test/dapltest/test/dapl_performance_client.c --- linux-kernel15/test/dapltest/test/dapl_performance_client.c 2005-05-13 11:37:44.366001000 -0700 +++ linux-kernel16/test/dapltest/test/dapl_performance_client.c 2005-05-13 15:36:58.016008000 -0700 @@ -92,7 +92,7 @@ DT_Performance_Test_Client_Connect ( Performance_Test_t *test_ptr) { u32 ret; - DAT_EVENT_NUMBER event_num; + enum dat_event_number event_num; unsigned int retry_cnt = 0; /* diff -Nurp -X /home/tduffy/dontdiff linux-kernel15/test/dapltest/test/dapl_performance_server.c linux-kernel16/test/dapltest/test/dapl_performance_server.c --- linux-kernel15/test/dapltest/test/dapl_performance_server.c 2005-05-13 11:37:44.370004000 -0700 +++ linux-kernel16/test/dapltest/test/dapl_performance_server.c 2005-05-13 15:36:58.020015000 -0700 @@ -106,7 +106,7 @@ DT_Performance_Test_Server_Connect ( struct dat_cr_arrival_event_data cr_stat; DAT_CR_HANDLE cr_handle; - DAT_EVENT_NUMBER event_num; + enum dat_event_number event_num; rsp_handle = DAT_HANDLE_NULL; psp_handle = DAT_HANDLE_NULL; diff -Nurp -X /home/tduffy/dontdiff linux-kernel15/test/dapltest/test/dapl_server.c linux-kernel16/test/dapltest/test/dapl_server.c --- linux-kernel15/test/dapltest/test/dapl_server.c 2005-05-12 09:28:43.311012000 -0700 +++ linux-kernel16/test/dapltest/test/dapl_server.c 2005-05-13 15:36:58.026007000 -0700 @@ -277,7 +277,7 @@ DT_cs_Server (Params_t * params_ptr) { DAT_CR_HANDLE cr_handle; struct dat_cr_arrival_event_data cr_stat; - DAT_EVENT_NUMBER event_num; + enum dat_event_number event_num; /* Set up the Per_Test_Data */ pt_ptr = DT_Alloc_Per_Test_Data (phead); diff -Nurp -X /home/tduffy/dontdiff linux-kernel15/test/dapltest/test/dapl_test_util.c linux-kernel16/test/dapltest/test/dapl_test_util.c --- linux-kernel15/test/dapltest/test/dapl_test_util.c 2005-05-13 11:37:44.379003000 -0700 +++ linux-kernel16/test/dapltest/test/dapl_test_util.c 2005-05-13 15:36:58.032023000 -0700 @@ -308,7 +308,7 @@ boolean_t DT_conn_event_wait (DT_Tdep_Print_Head *phead, DAT_EP_HANDLE ep_handle, DAT_EVD_HANDLE evd_handle, - DAT_EVENT_NUMBER *event_number) + enum dat_event_number *event_number) { for (;;) diff -Nurp -X /home/tduffy/dontdiff linux-kernel15/test/dapltest/test/dapl_transaction_test.c linux-kernel16/test/dapltest/test/dapl_transaction_test.c --- linux-kernel15/test/dapltest/test/dapl_transaction_test.c 2005-05-13 11:37:44.390001000 -0700 +++ linux-kernel16/test/dapltest/test/dapl_transaction_test.c 2005-05-13 15:36:58.041018000 -0700 @@ -236,7 +236,7 @@ DT_Transaction_Main (void *param) Thread *thread; DAT_DTO_COOKIE dto_cookie; char *private_data_str; - DAT_EVENT_NUMBER event_num; + enum dat_event_number event_num; DT_Tdep_Print_Head *phead; int status = 0; diff -Nurp -X /home/tduffy/dontdiff linux-kernel15/test/dapltest/test/dapl_util.c linux-kernel16/test/dapltest/test/dapl_util.c --- linux-kernel15/test/dapltest/test/dapl_util.c 2005-05-13 14:31:35.144002000 -0700 +++ linux-kernel16/test/dapltest/test/dapl_util.c 2005-05-13 15:36:58.046009000 -0700 @@ -55,7 +55,7 @@ DT_RetToString (u32 ret_value) * Map DAT_ASYNC_ERROR_CODE values to readable strings */ const char * -DT_AsyncErr2Str (DAT_EVENT_NUMBER error_code) +DT_AsyncErr2Str (enum dat_event_number error_code) { unsigned int i; static struct @@ -93,14 +93,14 @@ DT_AsyncErr2Str (DAT_EVENT_NUMBER error_ } } - return ( "Invalid_DAT_EVENT_NUMBER" ); + return ( "Invalid_enum dat_event_number" ); } /* * Map DAT_EVENT_CODE values to readable strings */ const char * -DT_EventToSTr (DAT_EVENT_NUMBER event_code) +DT_EventToSTr (enum dat_event_number event_code) { unsigned int i; static struct @@ -140,7 +140,7 @@ DT_EventToSTr (DAT_EVENT_NUMBER event_co } } - return ( "Invalid_DAT_EVENT_NUMBER" ); + return ( "Invalid_enum dat_event_number" ); } From Tom.Duffy at Sun.COM Sun May 15 11:27:19 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Sun, 15 May 2005 11:27:19 -0700 Subject: [openib-general] [PATCH][17/19] kDAPL: Remove typedef DAT_UPCALL_POLICY In-Reply-To: <1116181639894@sun.com> References: <11161816352658@sun.com> <11161816351137@sun.com> <11161816352496@sun.com> <11161816361781@sun.com> <1116181636449@sun.com> <11161816363220@sun.com> <11161816363515@sun.com> <11161816371774@sun.com> <1116181637281@sun.com> <111618163779@sun.com> <11161816374043@sun.com> <11161816381329@sun.com> <1116181638840@sun.com> <11161816381177@sun.com> <1116181638195@sun.com> <11161816391591@sun.com> <1116181639894@sun.com> Message-ID: <11161816391314@sun.com> Signed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel16/dat/dat.h linux-kernel17/dat/dat.h --- linux-kernel16/dat/dat.h 2005-05-13 15:37:17.695015000 -0700 +++ linux-kernel17/dat/dat.h 2005-05-13 15:54:55.036006000 -0700 @@ -908,14 +908,15 @@ struct dat_event { /* Upcall support */ -typedef enum dat_upcall_policy { +enum dat_upcall_policy { DAT_UPCALL_DISABLE = 0, /* support no_upcalls */ DAT_UPCALL_SINGLE_INSTANCE = 1, /* support only one upcall */ DAT_UPCALL_MANY = 100, /* support multiple upcalls */ - DAT_UPCALL_TEARDOWN = -1 /* support no upcalls & return */ - /* after all in progress */ - /* UpCalls return */ -} DAT_UPCALL_POLICY; + DAT_UPCALL_TEARDOWN = -1 /* support no upcalls & return + * after all in progress + * UpCalls return. + */ +}; typedef void (*DAT_UPCALL_FUNC) (void *, const struct dat_event *, boolean_t); @@ -943,7 +944,7 @@ struct dat_provider_info { struct dat_evd_param { DAT_IA_HANDLE ia_handle; int evd_qlen; - DAT_UPCALL_POLICY upcall_policy; + enum dat_upcall_policy upcall_policy; struct dat_upcall_object upcall; enum dat_evd_flags evd_flags; }; @@ -1070,7 +1071,7 @@ struct dat_provider_attr { int max_private_data_size; boolean_t supports_multipath; DAT_EP_CREATOR_FOR_PSP ep_creator; - DAT_UPCALL_POLICY upcall_policy; + enum dat_upcall_policy upcall_policy; u32 optimal_buffer_alignment; const boolean_t evd_stream_merging_supported[6][6]; boolean_t srq_supported; @@ -1258,11 +1259,12 @@ typedef u32 (*DAT_IA_MEMTYPE_HINT_FUNC)( DAT_MEM_OPTIMIZE_FLAGS, u64 *, u64 *); -typedef u32 (*DAT_EVD_KCREATE_FUNC) (DAT_IA_HANDLE, int, DAT_UPCALL_POLICY, +typedef u32 (*DAT_EVD_KCREATE_FUNC) (DAT_IA_HANDLE, int, enum dat_upcall_policy, const struct dat_upcall_object *, enum dat_evd_flags, DAT_EVD_HANDLE *); -typedef u32 (*DAT_EVD_MODIFY_UPCALL_FUNC)(DAT_EVD_HANDLE, DAT_UPCALL_POLICY, +typedef u32 (*DAT_EVD_MODIFY_UPCALL_FUNC)(DAT_EVD_HANDLE, + enum dat_upcall_policy, const struct dat_upcall_object *); struct dat_provider { @@ -1463,7 +1465,7 @@ static inline u32 dat_evd_free(DAT_EVD_H } static inline u32 dat_evd_kcreate(DAT_IA_HANDLE ia, int qlen, - DAT_UPCALL_POLICY policy, + enum dat_upcall_policy policy, const struct dat_upcall_object *upcall, enum dat_evd_flags flags, DAT_EVD_HANDLE * evd) @@ -1473,7 +1475,7 @@ static inline u32 dat_evd_kcreate(DAT_IA } static inline u32 dat_evd_modify_upcall(DAT_EVD_HANDLE evd, - DAT_UPCALL_POLICY policy, + enum dat_upcall_policy policy, const struct dat_upcall_object *upcall) { return DAT_CALL_PROVIDER_FUNC(evd_modify_upcall_func, evd, policy, diff -Nurp -X /home/tduffy/dontdiff linux-kernel16/dat-provider/dapl_evd_kcreate.c linux-kernel17/dat-provider/dapl_evd_kcreate.c --- linux-kernel16/dat-provider/dapl_evd_kcreate.c 2005-05-13 13:52:08.753001000 -0700 +++ linux-kernel17/dat-provider/dapl_evd_kcreate.c 2005-05-13 15:52:28.074001000 -0700 @@ -49,7 +49,7 @@ */ u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_handle, int evd_min_qlen, - DAT_UPCALL_POLICY upcall_policy, /* ignore */ + enum dat_upcall_policy upcall_policy, /* ignore */ const struct dat_upcall_object *upcall, enum dat_evd_flags evd_flags, DAT_EVD_HANDLE *evd_handle) { diff -Nurp -X /home/tduffy/dontdiff linux-kernel16/dat-provider/dapl_evd_modify_upcall.c linux-kernel17/dat-provider/dapl_evd_modify_upcall.c --- linux-kernel16/dat-provider/dapl_evd_modify_upcall.c 2005-05-12 09:28:47.668011000 -0700 +++ linux-kernel17/dat-provider/dapl_evd_modify_upcall.c 2005-05-13 15:52:28.076006000 -0700 @@ -44,7 +44,7 @@ */ u32 dapl_evd_modify_upcall(DAT_EVD_HANDLE evd_handle, - DAT_UPCALL_POLICY upcall_policy, /* ignore */ + enum dat_upcall_policy upcall_policy, /* ignore */ const struct dat_upcall_object *upcall) { DAPL_EVD *evd_ptr; diff -Nurp -X /home/tduffy/dontdiff linux-kernel16/dat-provider/dapl.h linux-kernel17/dat-provider/dapl.h --- linux-kernel16/dat-provider/dapl.h 2005-05-13 14:35:01.158008000 -0700 +++ linux-kernel17/dat-provider/dapl.h 2005-05-13 15:52:28.068010000 -0700 @@ -253,7 +253,7 @@ struct dapl_ia { struct dapl_cno { DAPL_HEADER header; struct dat_upcall_object cno_upcall; - DAT_UPCALL_POLICY cno_upcall_policy; + enum dat_upcall_policy cno_upcall_policy; }; /* DAPL_EVD maps to DAT_EVD_HANDLE */ @@ -512,7 +512,7 @@ extern u32 dapl_ia_memtype_hint(DAT_IA_H u64 *suggested_alignment); extern u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_handle, int evd_min_qlen, - DAT_UPCALL_POLICY upcall_policy, + enum dat_upcall_policy upcall_policy, const struct dat_upcall_object * upcall, enum dat_evd_flags evd_flags, DAT_EVD_HANDLE * evd_handle); @@ -521,7 +521,7 @@ extern u32 dapl_evd_kquery(DAT_EVD_HANDL struct dat_evd_param *evd_args); extern u32 dapl_evd_modify_upcall(DAT_EVD_HANDLE evd_handle, - DAT_UPCALL_POLICY upcall_policy, + enum dat_upcall_policy upcall_policy, const struct dat_upcall_object * upcall); extern u32 dapl_evd_enable(DAT_EVD_HANDLE evd_handle); From Tom.Duffy at Sun.COM Sun May 15 11:27:20 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Sun, 15 May 2005 11:27:20 -0700 Subject: [openib-general] [PATCH][18/19] kDAPL: Remove typedef DAT_MEM_OPTIMIZE_FLAGS & #define DAT_MEM_OPT In-Reply-To: <11161816391314@sun.com> References: <11161816352658@sun.com> <11161816351137@sun.com> <11161816352496@sun.com> <11161816361781@sun.com> <1116181636449@sun.com> <11161816363220@sun.com> <11161816363515@sun.com> <11161816371774@sun.com> <1116181637281@sun.com> <111618163779@sun.com> <11161816374043@sun.com> <11161816381329@sun.com> <1116181638840@sun.com> <11161816381177@sun.com> <1116181638195@sun.com> <11161816391591@sun.com> <1116181639894@sun.com> <11161816391314@sun.com> Message-ID: <1116181639527@sun.com> Signed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel17/dat/dat.h linux-kernel18/dat/dat.h --- linux-kernel17/dat/dat.h 2005-05-13 15:54:55.036006000 -0700 +++ linux-kernel18/dat/dat.h 2005-05-13 16:08:45.452018000 -0700 @@ -959,14 +959,12 @@ struct dat_evd_param { */ /* memory data types */ -#define DAT_MEM_OPT DAT_MEM_OPTIMIZE_FLAGS - -typedef enum dat_mem_optimize_flags { +enum dat_mem_optimize_flags { DAT_MEM_OPTIMIZE_DONT_CARE = 0x00, DAT_MEM_OPTIMIZE_IA = 0x01, DAT_MEM_OPTIMIZE_MIN_EXPOSURE = 0x02, DAT_MEM_OPTIMIZE_EXACT_EXPOSURE = 0x04 -} DAT_MEM_OPTIMIZE_FLAGS; +}; typedef union dat_region_description { void *for_va; @@ -1251,12 +1249,13 @@ typedef u32 (*DAT_SRQ_POST_RECV_FUNC)(DA typedef u32 (*DAT_LMR_KCREATE_FUNC)(DAT_IA_HANDLE, enum dat_mem_type, DAT_REGION_DESCRIPTION, u64, DAT_PZ_HANDLE, enum dat_mem_priv_flags, - DAT_MEM_OPTIMIZE_FLAGS, DAT_LMR_HANDLE *, + enum dat_mem_optimize_flags, + DAT_LMR_HANDLE *, DAT_LMR_CONTEXT *, DAT_RMR_CONTEXT *, u64 *, u64 *); typedef u32 (*DAT_IA_MEMTYPE_HINT_FUNC)(DAT_IA_HANDLE, enum dat_mem_type, u64, - DAT_MEM_OPTIMIZE_FLAGS, u64 *, + enum dat_mem_optimize_flags, u64 *, u64 *); typedef u32 (*DAT_EVD_KCREATE_FUNC) (DAT_IA_HANDLE, int, enum dat_upcall_policy, @@ -1415,7 +1414,7 @@ static inline u32 dat_get_handle_type(DA static inline u32 dat_ia_memtype_hint(DAT_IA_HANDLE ia, enum dat_mem_type mem_type, u64 length, - DAT_MEM_OPTIMIZE_FLAGS mem_optimize, + enum dat_mem_optimize_flags mem_optimize, u64 *preferred_length, u64 *preferred_alignment) { @@ -1779,7 +1778,7 @@ static inline u32 dat_lmr_kcreate(DAT_IA DAT_REGION_DESCRIPTION region, u64 len, DAT_PZ_HANDLE pz, enum dat_mem_priv_flags privileges, - DAT_MEM_OPTIMIZE_FLAGS optimization, + enum dat_mem_optimize_flags optimization, DAT_LMR_HANDLE *lmr_handle, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, diff -Nurp -X /home/tduffy/dontdiff linux-kernel17/dat-provider/dapl.h linux-kernel18/dat-provider/dapl.h --- linux-kernel17/dat-provider/dapl.h 2005-05-13 15:52:28.068010000 -0700 +++ linux-kernel18/dat-provider/dapl.h 2005-05-13 16:09:23.703026000 -0700 @@ -507,7 +507,8 @@ extern u32 dapl_cr_handoff(DAT_CR_HANDLE /* EVD Functions */ extern u32 dapl_ia_memtype_hint(DAT_IA_HANDLE ia_handle, enum dat_mem_type mem_type, - u64 length, DAT_MEM_OPT mem_optimization, + u64 length, + enum dat_mem_optimize_flags mem_optimization, u64 *suggested_length, u64 *suggested_alignment); @@ -629,7 +630,7 @@ extern u32 dapl_lmr_kcreate(DAT_IA_HANDL DAT_REGION_DESCRIPTION region_description, u64 length, DAT_PZ_HANDLE pz_handle, enum dat_mem_priv_flags privileges, - DAT_MEM_OPT optimization, + enum dat_mem_optimize_flags optimization, DAT_LMR_HANDLE *lmr_handle, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, diff -Nurp -X /home/tduffy/dontdiff linux-kernel17/dat-provider/dapl_ia_memtype_hint.c linux-kernel18/dat-provider/dapl_ia_memtype_hint.c --- linux-kernel17/dat-provider/dapl_ia_memtype_hint.c 2005-05-13 14:14:08.086967000 -0700 +++ linux-kernel18/dat-provider/dapl_ia_memtype_hint.c 2005-05-13 16:09:42.646010000 -0700 @@ -64,7 +64,8 @@ * DAT_MODEL_NOT_SUPPORTED */ u32 dapl_ia_memtype_hint(DAT_IA_HANDLE ia_handle, enum dat_mem_type mem_type, - u64 length, DAT_MEM_OPT mem_optimization, + u64 length, + enum dat_mem_optimize_flags mem_optimization, u64 *suggested_length, u64 *suggested_alignment) { diff -Nurp -X /home/tduffy/dontdiff linux-kernel17/dat-provider/dapl_lmr_kcreate.c linux-kernel18/dat-provider/dapl_lmr_kcreate.c --- linux-kernel17/dat-provider/dapl_lmr_kcreate.c 2005-05-13 14:23:13.154000000 -0700 +++ linux-kernel18/dat-provider/dapl_lmr_kcreate.c 2005-05-13 15:58:51.348026000 -0700 @@ -305,7 +305,7 @@ u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_ha DAT_REGION_DESCRIPTION region_description, u64 length, DAT_PZ_HANDLE pz_handle, enum dat_mem_priv_flags privileges, - DAT_MEM_OPT optimization, DAT_LMR_HANDLE *lmr_handle, + enum dat_mem_optimize_flags optimization, DAT_LMR_HANDLE *lmr_handle, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address) { DAPL_IA *ia; From Tom.Duffy at Sun.COM Sun May 15 11:27:20 2005 From: Tom.Duffy at Sun.COM (Tom Duffy) Date: Sun, 15 May 2005 11:27:20 -0700 Subject: [openib-general] [PATCH][19/19] kDAPL: Remove typedefs DAT_IOV_OWNERSHIP & DAT_EP_CREATOR_FOR_PSP In-Reply-To: <1116181639527@sun.com> References: <11161816352658@sun.com> <11161816351137@sun.com> <11161816352496@sun.com> <11161816361781@sun.com> <1116181636449@sun.com> <11161816363220@sun.com> <11161816363515@sun.com> <11161816371774@sun.com> <1116181637281@sun.com> <111618163779@sun.com> <11161816374043@sun.com> <11161816381329@sun.com> <1116181638840@sun.com> <11161816381177@sun.com> <1116181638195@sun.com> <11161816391591@sun.com> <1116181639894@sun.com> <11161816391314@sun.com> <1116181639527@sun.com> Message-ID: <1116181640390@sun.com> Signed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel18/dat/dat.h linux-kernel19/dat/dat.h --- linux-kernel18/dat/dat.h 2005-05-13 16:08:45.452018000 -0700 +++ linux-kernel19/dat/dat.h 2005-05-15 09:44:20.544000000 -0700 @@ -1028,20 +1028,20 @@ struct dat_ia_attr { }; /* Provider attributes */ -typedef enum dat_iov_ownership { +enum dat_iov_ownership { /* Not a modification by the Provider; the Consumer can use anytime. */ DAT_IOV_CONSUMER = 0x0, /* Provider does not modify returned IOV DTO on completion. */ DAT_IOV_PROVIDER_NOMOD = 0x1, /* Provider can modify IOV DTO on completion; can't trust it. */ DAT_IOV_PROVIDER_MOD = 0x2 -} DAT_IOV_OWNERSHIP; +}; -typedef enum dat_ep_creator_for_psp { - DAT_PSP_CREATES_EP_NEVER, /* Provider never creates Endpoint. */ - DAT_PSP_CREATES_EP_IFASKED, /* Provider creates Endpoint if asked. */ - DAT_PSP_CREATES_EP_ALWAYS /* Provider always creates Endpoint. */ -} DAT_EP_CREATOR_FOR_PSP; +enum dat_ep_creator_for_psp { + DAT_PSP_CREATES_EP_NEVER, /* Provider never creates Endpoint. */ + DAT_PSP_CREATES_EP_IFASKED, /* Provider creates Endpoint if asked.*/ + DAT_PSP_CREATES_EP_ALWAYS /* Provider always creates Endpoint. */ +}; /* The evd_stream_merging_supported is a 2D binary matrix where each row * and column represents an event stream type. An entry is a 1 if the event @@ -1062,13 +1062,13 @@ struct dat_provider_attr { u32 dat_version_major; u32 dat_version_minor; enum dat_mem_type lmr_mem_types_supported; - DAT_IOV_OWNERSHIP iov_ownership_on_return; + enum dat_iov_ownership iov_ownership_on_return; enum dat_qos dat_qos_supported; enum dat_completion_flags completion_flags_supported; boolean_t is_thread_safe; int max_private_data_size; boolean_t supports_multipath; - DAT_EP_CREATOR_FOR_PSP ep_creator; + enum dat_ep_creator_for_psp ep_creator; enum dat_upcall_policy upcall_policy; u32 optimal_buffer_alignment; const boolean_t evd_stream_merging_supported[6][6]; From tempra at doneasy.com Sun May 15 16:49:14 2005 From: tempra at doneasy.com (Janie Dewitt) Date: Sun, 15 May 2005 20:49:14 -0300 Subject: [openib-general] Become a homeowner with low rates Message-ID: Hello, We tried contacting you awhile ago about your low interest morta(ge rate. You have qualified for the lowest rate in years... You could get over $380,000 for as little as $500 a month! Ba(d credit? Doesn't matter, low rates are fixed no matter what! To get a free, no obli,gation consultation click below: http://www.truthfu1ly.com/sign.asp Best Regards, Salvatore Eldridge to be remov(ed: http://www.truthfu1ly.com/gone.asp this process takes one week, so please be patient. we do our best to take your email/s off but you have to fill out a rem/ove or else you will continue to recieve email/s. From Dpo0BUF at msn.com Sun May 15 19:21:04 2005 From: Dpo0BUF at msn.com (Kerry Maureen) Date: Mon, 16 May 2005 04:21:04 +0200 Subject: [openib-general] Rate Quote for free! Message-ID: <2885.504008520.UAA02976@uccor.edu.ar> Hello, You have been chosen to participate in an invitation only limited time event! Are you currently paying over 3% for your mortgage? STOP! We can help you lower that today! Answer only a few questions and we can give you an approval in under 30 seconds � it�s that simple! http://g0Xu.g00d13s.com/p2.asp And stop fighting for lenders � let them fight for you! Make them work for your business by giving you the lowest rates around! $230,000 loans are available for only $340/month! WE�RE PRACTICALLY GIVING AWAY MONEY! http://1S2i.g00d13s.com/p2.asp Think your credit is too bad to get a deal like this? THINK AGAIN! We will have you saving your money in no time! Are you ready to save your money? http://UWlo.g00d13s.com/p2.asp http://WrVK.g00d13s.com/p1.asp Need to be deleted? Use the link below: sguW.this-ls-it.net/deletion.asp Regards, Kerry Maureen From tziporet at mellanox.co.il Mon May 16 01:03:09 2005 From: tziporet at mellanox.co.il (Tziporet Koren) Date: Mon, 16 May 2005 11:03:09 +0300 Subject: [openib-general] HCA Firmware Message-ID: <506C3D7B14CDD411A52C00025558DED6064BF56C@mtlex01.yok.mtl.com> Next FW release (3.3.3) will support 64K pages. If someone need it now please approach me to get a drop RC. Tziporet -----Original Message----- From: Grant Grundler [mailto:iod00d at hp.com] Sent: Tuesday, May 10, 2005 9:05 PM To: Roland Dreier Cc: openib-general-bounces at openib.org; openib-general at openib.org Subject: Re: [openib-general] HCA Firmware On Tue, May 10, 2005 at 10:22:16AM -0700, Roland Dreier wrote: > Just for completeness, for PCI-X HCAs: > . 3.3.2 is the first firmware release with working MSI-X > . No released versions of firmware will work on systems with 64K pages. > (This is a possible config for ia64 kernels at least) Yes, but default is 16K - so most folks will be using that. AFAIK, only SGI/SuSE kernels use 64K by default. grant _______________________________________________ openib-general mailing list openib-general at openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general -------------- next part -------------- An HTML attachment was scrubbed... URL: From mst at mellanox.co.il Mon May 16 06:20:18 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Mon, 16 May 2005 16:20:18 +0300 Subject: [openib-general] Re: perftest-03: put C code in .c file In-Reply-To: <20050511150543.GB10070@esmail.cup.hp.com> References: <20050511014014.GG5495@esmail.cup.hp.com> <20050511063809.GA13349@mellanox.co.il> <20050511150543.GB10070@esmail.cup.hp.com> Message-ID: <20050516132018.GH13718@mellanox.co.il> Quoting r. Grant Grundler : > Subject: Re: perftest-03: put C code in .c file > > On Wed, May 11, 2005 at 09:38:09AM +0300, Michael S. Tsirkin wrote: > > You have to svn add before svn diff -x --diff-cmd=/usr/bin/diff -x -upN > > certainly - Appended below > > > Grant, could you please resend, while also adding a Signed-off-by line? > > I've been ommitting the S-o-B line since I realized this isn't kernel code. > I'm happy to include them if you like. Yes, please do. > Can we follow Documentation/Codingstyle too? > > thanks, > grant That's my plan. -- MST - Michael S. Tsirkin From halr at voltaire.com Mon May 16 06:20:46 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 16 May 2005 09:20:46 -0400 Subject: [openib-general] [PATCH] MAD: Eliminate MAD cache leak associated with local completions Message-ID: <1116249645.4480.694.camel@localhost.localdomain> MAD: Eliminate MAD cache leak associated with local completions Signed-off-by: Hal Rosenstock Index: mad.c =================================================================== --- mad.c (revision 2331) +++ mad.c (working copy) @@ -1602,7 +1602,6 @@ INIT_LIST_HEAD(&mad_recv_wc->rmpp_list); list_add(&mad_recv_wc->recv_buf.list, &mad_recv_wc->rmpp_list); - if (mad_agent_priv->agent.rmpp_version) { mad_recv_wc = ib_process_rmpp_recv_wc(mad_agent_priv, mad_recv_wc); @@ -2054,6 +2053,8 @@ /* Empty wait list to prevent receives from finding a request */ list_splice_init(&mad_agent_priv->wait_list, &cancel_list); + /* Empty local completion list as well */ + list_splice_init(&mad_agent_priv->local_list, &cancel_list); spin_unlock_irqrestore(&mad_agent_priv->lock, flags); /* Report all cancelled requests */ @@ -2168,6 +2169,7 @@ struct ib_mad_local_private *local; struct ib_mad_agent_private *recv_mad_agent; unsigned long flags; + int recv = 0; struct ib_wc wc; struct ib_mad_send_wc mad_send_wc; @@ -2183,10 +2185,10 @@ recv_mad_agent = local->recv_mad_agent; if (!recv_mad_agent) { printk(KERN_ERR PFX "No receive MAD agent for local completion\n"); - kmem_cache_free(ib_mad_cache, local->mad_priv); goto local_send_completion; } + recv = 1; /* * Defined behavior is to complete response * before request @@ -2199,6 +2201,8 @@ local->mad_priv->header.recv_wc.mad_len = sizeof(struct ib_mad); INIT_LIST_HEAD(&local->mad_priv->header.recv_wc.rmpp_list); + list_add(&local->mad_priv->header.recv_wc.recv_buf.list, + &local->mad_priv->header.recv_wc.rmpp_list); local->mad_priv->header.recv_wc.recv_buf.grh = NULL; local->mad_priv->header.recv_wc.recv_buf.mad = &local->mad_priv->mad.mad; @@ -2229,6 +2233,8 @@ spin_lock_irqsave(&mad_agent_priv->lock, flags); list_del(&local->completion_list); atomic_dec(&mad_agent_priv->refcount); + if (!recv) + kmem_cache_free(ib_mad_cache, local->mad_priv); kfree(local); } spin_unlock_irqrestore(&mad_agent_priv->lock, flags); From halr at voltaire.com Mon May 16 06:24:05 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 16 May 2005 09:24:05 -0400 Subject: [openib-general] opensm / hot-plug problem In-Reply-To: <20050515154553.GT13718@mellanox.co.il> References: <20050515154553.GT13718@mellanox.co.il> Message-ID: <1116249762.4480.698.camel@localhost.localdomain> On Sun, 2005-05-15 at 11:45, Michael S. Tsirkin wrote: > Hello! > opensm currently keeps multiple files open under > /sys/class/infiniband/mthca%d/ports/ > In particular, this will prevent the OS from performing device > hot-plug removal. > > In my opinion, it would be better for sm to either avoid > working directly with low-level drivers, or close the sysfs > files after reading them. > The change to close the files is probably easier to implement. > > Does this make sense to people here? Right now, OpenIB OpenSM keeps the issm files open for the duration of its execution. This easily be could be changed to open and close these files whenever the issm bit needs changing. This should not be very frequent. Doesn't this narrow the window of hotswap occuring while the issm file is open ? If the OpenSM did not set (and clear) the issm capability mask, how would this be done in a manner (more) conducive with hotswap ? Would it be with an explicit local set PortInfo MAD ? -- Hal From mst at mellanox.co.il Mon May 16 07:05:00 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Mon, 16 May 2005 17:05:00 +0300 Subject: [openib-general] perftest - multiple patches merged Message-ID: <20050516140500.GI13718@mellanox.co.il> Hi! With 2353 I think I've merged all patches that I have outstanding. I plan a small cleanup of program output, hope to do it tomorrow. -- MST - Michael S. Tsirkin From jlentini at netapp.com Mon May 16 07:56:51 2005 From: jlentini at netapp.com (James Lentini) Date: Mon, 16 May 2005 10:56:51 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: change EXPORT_SYMBOL to EXPORT_SYMBOL_GPL In-Reply-To: <1116018270.20109.8.camel@duffman> References: <1115910070.25530.2.camel@duffman> <1115915608.9070.0.camel@duffman> <469958e00505121256cd8c04d@mail.gmail.com> <469958e00505131152573cd93a@mail.gmail.com> <1116018270.20109.8.camel@duffman> Message-ID: Tom, I think we should follow the example of other parts of the kernel and export our symbols without qualifications. This would keep our driver interface in synch with similar subsystems. james On Fri, 13 May 2005, Tom Duffy wrote: tduffy> On Fri, 2005-05-13 at 11:52 -0700, Caitlin Bestler wrote: tduffy> > What about kernel clients that are compiled separately from the kernel tduffy> > that want to be OS neutral and use kdat.h as published by the DAT tduffy> > Collaborative? tduffy> tduffy> Make sure this doesn't happen. tduffy> tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> Index: linux-kernel-gpl/dat/api.c tduffy> =================================================================== tduffy> --- linux-kernel-gpl/dat/api.c (revision 2338) tduffy> +++ linux-kernel-gpl/dat/api.c (working copy) tduffy> @@ -619,7 +619,7 @@ u32 dat_ia_openv(const char *name, int a tduffy> name, async_event_qlen, async_event_handle, ia_handle); tduffy> } tduffy> tduffy> -EXPORT_SYMBOL(dat_ia_openv); tduffy> +EXPORT_SYMBOL_GPL(dat_ia_openv); tduffy> tduffy> /*********************************************************************** tduffy> * Function: dat_ia_close tduffy> @@ -673,7 +673,7 @@ u32 dat_ia_close(DAT_IA_HANDLE ia_handle tduffy> return status; tduffy> } tduffy> tduffy> -EXPORT_SYMBOL(dat_ia_close); tduffy> +EXPORT_SYMBOL_GPL(dat_ia_close); tduffy> tduffy> /*********************************************************************** tduffy> * Function: dat_strerror tduffy> @@ -706,7 +706,7 @@ u32 dat_strerror(u32 value, const char * tduffy> return DAT_SUCCESS; tduffy> } tduffy> tduffy> -EXPORT_SYMBOL(dat_strerror); tduffy> +EXPORT_SYMBOL_GPL(dat_strerror); tduffy> tduffy> /*********************************************************************** tduffy> * Function: dat_registry_add_provider tduffy> @@ -731,7 +731,7 @@ u32 dat_registry_add_provider(const stru tduffy> return dat_dr_insert(provider_info, provider->ia_open_func); tduffy> } tduffy> tduffy> -EXPORT_SYMBOL(dat_registry_remove_provider); tduffy> +EXPORT_SYMBOL_GPL(dat_registry_remove_provider); tduffy> tduffy> /*********************************************************************** tduffy> * Function: dat_registry_remove_provider tduffy> @@ -750,7 +750,7 @@ u32 dat_registry_remove_provider(const s tduffy> return dat_dr_remove(provider_info); tduffy> } tduffy> tduffy> -EXPORT_SYMBOL(dat_registry_add_provider); tduffy> +EXPORT_SYMBOL_GPL(dat_registry_add_provider); tduffy> tduffy> /*********************************************************************** tduffy> * Function: dat_registry_list_providers tduffy> @@ -787,4 +787,4 @@ u32 dat_registry_list_providers(int max_ tduffy> return status; tduffy> } tduffy> tduffy> -EXPORT_SYMBOL(dat_registry_list_providers); tduffy> +EXPORT_SYMBOL_GPL(dat_registry_list_providers); tduffy> From info at mynoodle.net Mon May 16 03:25:57 2005 From: info at mynoodle.net (info at mynoodle.net) Date: 16 May 2005 19:25:57 +0900 Subject: [openib-general] $B:#F|$OFCJL$G!D(B Message-ID: <20050516102557.21776.qmail@mail.mynoodle.net> $B!!!!!!!!!!!!!!!!!!(B $B(.(,(,(/(B $B(-!@!?(-!z!yFCJL%*!<%W%s4|4VCf!y!z(B $B(1(,(,(0(B $B!!?75,3+ at _!*A4$/?7$7$$%7%9%F%`$K$h$j!"=P2q$($k3NN((B120$B!s!*(B $B!!$7$+$b$4>R2p$9$k=w at -$O3'?3:::Q$G!"??7u$J4uK>7BT7t!y!!!!!!!!!!!!!!2s2s(B $B2s2s!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!2s2s(B $B2s2s!!!!!!#A!!EPO?!"%W%m%U%#!<%k:n at .$O!!40A4L5NA!!!!!!!!2s2s(B $B2s2s!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!2s2s(B $B2s2s!!!!#B!!$46a=j$N=w at -!J:GBg#3L>!K$rR2p$r4uK>$9$kCO0h$r%W%m%U%#!<%k$KE:$($FL5NA$G$4EPO?$$$?$/$@$1$G!"$=$NCO0h$N=w at -$r%T%s%]%$%s%H$G$40FFb!*$44uK>$NJ}$O@'Hs>e5-%j%s%/$h$j$43NG'$/$@$5$$!#(B From jlentini at netapp.com Mon May 16 08:14:43 2005 From: jlentini at netapp.com (James Lentini) Date: Mon, 16 May 2005 11:14:43 -0400 (EDT) Subject: [openib-general] Re: [PATCH] [kdapl] CM: Eliminate explicit send DREQ on error in do_rep_recv In-Reply-To: <1116019630.4729.2.camel@localhost.localdomain> References: <1116019630.4729.2.camel@localhost.localdomain> Message-ID: Committed in revision 2354. On Fri, 13 May 2005, Hal Rosenstock wrote: halr> CM: Eliminate explicit send DREQ on error in do_rep_recv halr> halr> Signed-off-by: Hal Rosenstock halr> halr> Index: dapl_openib_cm.c halr> =================================================================== halr> --- dapl_openib_cm.c (revision 2348) halr> +++ dapl_openib_cm.c (working copy) halr> @@ -99,7 +99,6 @@ halr> return; halr> halr> disc: halr> - ib_send_cm_dreq(conn->cm_id, NULL, 0); /* ??? */ halr> dapl_evd_connection_callback(conn, IB_CME_LOCAL_FAILURE, NULL, halr> conn->ep); halr> } halr> halr> From halr at voltaire.com Mon May 16 08:42:47 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 16 May 2005 11:42:47 -0400 Subject: [openib-general] [PATCH] [kdapl] CM modify retry count in a couple of error messages Message-ID: <1116258167.4480.901.camel@localhost.localdomain> CM modify retry count in a couple of error messages Signed-off-by: Hal Rosenstock Index: dapl_openib_cm.c =================================================================== --- dapl_openib_cm.c (revision 2354) +++ dapl_openib_cm.c (working copy) @@ -280,7 +280,7 @@ if (rec_num <= 0) { printk(KERN_ERR "dapl_path_comp_handler: path resolution " - "failed %d retry %d!!!\n", rec_num, conn->retries); + "failed %d retry %d!!!\n", rec_num, conn->retries + 1); if (++conn->retries > OPENIB_MAX_AT_RETRY) { printk(KERN_ERR "dapl_path_comp_handler: ep_ptr 0x%p\n", conn->ep); @@ -330,7 +330,7 @@ if (rec_num <= 0) { printk(KERN_ERR "dapl_rt_comp_handler: rec num %d retry %d\n", - rec_num, conn->retries); + rec_num, conn->retries + 1); if (++conn->retries > OPENIB_MAX_AT_RETRY) { event = IB_CME_DESTINATION_UNREACHABLE; goto error; From info at uhyahya.net Mon May 16 04:28:42 2005 From: info at uhyahya.net (info at uhyahya.net) Date: 16 May 2005 20:28:42 +0900 Subject: [openib-general] $B?75,EPO?$NJ}$O(B10000$B1_J,L5NA(B Message-ID: <20050516112842.4941.qmail@mail.uhyahya.net> $B0lK|1_J,!*40A4L5NA$G;HMQ(BOK $B"#:#$J$i4V$K9g$&(BW$B%A%c%s%9!*L5NA%]%$%s%H%"%C%W3Z$7$5(B100$BG\!*"#(B $B"(EPO?=w at -2q0w$N%8%c%s%kJL?M5$%i%s%-%s%0(B $B0l0L!'5U1g!J(B189$BL>EPO?!K(B $BFs0L!'1g8r!J(B429$BL>EPO?!K(B $B;00L!'(BSM$B5U1g!J(B243$BL>EPO?!K(B $B;M0L!'%;%U%l!J(B1038$BL>EPO?!K(B $B8^0L!'ITNQ!J(B2421$BL>EPO?!K(B $BO;0L!'%F%l%(%C%A!J(B3463$BL>EPO?!K(B $B!&(B $B!&(B $B!&(B $B0lK|1_J,40A4L5NA$G$*;n$7$O(B $B"-"-"-(B http://awg.webchu.com/?summer36 $BB~:#!*?75,EPO?$7$?J}$K$OL5NA$G(B10,000$B1_J,$4MxMQ=PMh$^$9!#(B $B!ZL5NA%]%$%s%HFb$G==J,$K%Q!<%H%J!<$rC5$9;v$,=PMh$^$9!#![(B $B"((B18$B:P0J>e$NHkL)87l9g$O2<5-(BURL$B$KJV?.2<$5$$!#(B henshin_senyou at ok.kz From xma at us.ibm.com Mon May 16 09:48:52 2005 From: xma at us.ibm.com (Shirley Ma) Date: Mon, 16 May 2005 09:48:52 -0700 Subject: [openib-general] tvflash utility Message-ID: I've sucessfully upgrade my 4X Topspin HCA with Mellanox firmware 3.3.2. But this firmware doesn't work on JS20 daughter card. Does anybody use tvflash? It doesn't work on my PPC64 and ia64. I want to use this utility to read HCA on board binary image to repair the daughter card. Any help? Thanks Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From roland at topspin.com Mon May 16 09:55:35 2005 From: roland at topspin.com (Roland Dreier) Date: Mon, 16 May 2005 09:55:35 -0700 Subject: [openib-general] tvflash utility In-Reply-To: (Shirley Ma's message of "Mon, 16 May 2005 09:48:52 -0700") References: Message-ID: <52d5rrnm9k.fsf@topspin.com> Shirley> I've sucessfully upgrade my 4X Topspin HCA with Mellanox Shirley> firmware 3.3.2. But this firmware doesn't work on JS20 Shirley> daughter card. Does anybody use tvflash? It doesn't work Shirley> on my PPC64 and ia64. I want to use this utility to read Shirley> HCA on board binary image to repair the daughter Shirley> card. Any help? Firmware 3.3.2 should work on the JS20 daughtercard. You should use FW with the same memory timings as the full-height "Cougar" PCI-X card, not the half-height "Cougar Cub." Also, the tvflash from https://openib.org/svn/gen2/trunk/src/userspace/tvflash should work on ppc64 and ia64 -- I have used it to update firmware on a JS20, in fact. What goes wrong when you try? - R. From tduffy at sun.com Mon May 16 09:54:53 2005 From: tduffy at sun.com (Tom Duffy) Date: Mon, 16 May 2005 09:54:53 -0700 Subject: [openib-general] tvflash utility In-Reply-To: References: Message-ID: <1116262493.3152.0.camel@duffman> On Mon, 2005-05-16 at 09:48 -0700, Shirley Ma wrote: > > I've sucessfully upgrade my 4X Topspin HCA with Mellanox firmware > 3.3.2. But this firmware doesn't work on JS20 daughter card. > Does anybody use tvflash? It doesn't work on my PPC64 and ia64. I want > to use this utility to read HCA on board binary image to repair the > daughter card. Any help? I have used the utility on x86_64 and sparc64. Can you please explain your failure? Thanks, -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From mshefty at ichips.intel.com Mon May 16 09:58:41 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Mon, 16 May 2005 09:58:41 -0700 Subject: [openib-general] Re: [PATCH] make private data pointers const In-Reply-To: References: Message-ID: <4288D141.307@ichips.intel.com> James Lentini wrote: > > This patch affects ib_cm.h and cm.c, which appear to be maintained by > Sean, and ucm.c, which appears to be maintained by Libor. > > The patch adds the const qualifier to private data pointers passed into > the IB layer. My system rejects all of the changes. Has anyone else tried applying this patch? - Sean From roland at topspin.com Mon May 16 10:00:55 2005 From: roland at topspin.com (Roland Dreier) Date: Mon, 16 May 2005 10:00:55 -0700 Subject: [openib-general] Re: missing check in uverbs? In-Reply-To: <20050515135733.GQ13718@mellanox.co.il> (Michael S. Tsirkin's message of "Sun, 15 May 2005 16:57:33 +0300") References: <20050515135733.GQ13718@mellanox.co.il> Message-ID: <5264xjnm0o.fsf@topspin.com> Michael> Hi, Roland! I had a mix of an updated kernel level code Michael> (rev 2350) and an outdated userspace library. Running Michael> ibv_pingpong client I got the following: Yep, we shouldn't allow userspace to perform mmap() until a userspace context has been allocated. Fixed by this (already checked in): --- infiniband/core/uverbs_main.c (revision 2349) +++ infiniband/core/uverbs_main.c (working copy) @@ -389,7 +389,10 @@ static int ib_uverbs_mmap(struct file *f { struct ib_uverbs_file *file = filp->private_data; - return file->device->ib_dev->mmap(file->ucontext, vma); + if (!file->ucontext) + return -ENODEV; + else + return file->device->ib_dev->mmap(file->ucontext, vma); } static int ib_uverbs_open(struct inode *inode, struct file *filp) From halr at voltaire.com Mon May 16 09:58:05 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 16 May 2005 12:58:05 -0400 Subject: [openib-general] Running libibcm/example/simple.c Message-ID: <1116262685.4480.938.camel@localhost.localdomain> Hi again Libor, When I run the example I get the following: server: CM ID <3> Event <1> State <3> client: CM ID <2> Event <0> State <0> server: CM ID <3> Event <2> State <0> The server side looks like it first gets a REQ received event in REQ RCVD state. The client then gets a REQ error event in IDLE state. The server then gets a REP error event in IDLE state. I did change the lids and gids in cm_connect to be appropriate for my configuration. Any ideas on what is going wrong ? Thanks. Also, another nit in libibcm/example/simple.c cm_connect: ... sa.dlid = htons(0x1f9); sa.slid = htons(0x3e1); sa.dlid = 0xf901; sa.slid = 0xe103; -- Hal From xma at us.ibm.com Mon May 16 10:03:15 2005 From: xma at us.ibm.com (Shirley Ma) Date: Mon, 16 May 2005 10:03:15 -0700 Subject: [openib-general] tvflash utility In-Reply-To: <52d5rrnm9k.fsf@topspin.com> Message-ID: Thanks Roland. > You should use FW with the same memory timings as the full-height "Cougar" PCI-X card, not the half-height "Cougar Cub." I see. I tried half-height. I will try the full-height. > I have used it to update firmware on a JS20, in fact. What goes wrong when you try? It doesn't compile. pci_alloc is not in -lpci. Any library I forgot to install? The README file is not clear. Thanks Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From roland at topspin.com Mon May 16 10:07:57 2005 From: roland at topspin.com (Roland Dreier) Date: Mon, 16 May 2005 10:07:57 -0700 Subject: [openib-general] tvflash utility In-Reply-To: (Shirley Ma's message of "Mon, 16 May 2005 10:03:15 -0700") References: Message-ID: <52zmuvm74i.fsf@topspin.com> Shirley> It doesn't compile. pci_alloc is not in -lpci. Any Shirley> library I forgot to install? The README file is not Shirley> clear. -lpci should include pci_alloc() -- what version of pciutils do you have installed? One possibility is that you need to install the pciutils-devel package as well as the pciutils package. - R. From tduffy at sun.com Mon May 16 10:07:20 2005 From: tduffy at sun.com (Tom Duffy) Date: Mon, 16 May 2005 10:07:20 -0700 Subject: [openib-general] [PATCH] at: pull sparse fixes from james's branch to shahar's Message-ID: <1116263241.3152.7.camel@duffman> Hal, Can you please pull the sparse fixes from James's branch to Shahar's? svn diff -r2134:2238 gen2/users/jlentini/linux-kernel/patches/at.c Or here it is against the latest r2278 in shaharf-ibat. Signed-off-by: Tom Duffy Index: gen2/branches/shaharf-ibat/src/linux-kernel/infiniband/core/at.c =================================================================== --- gen2/branches/shaharf-ibat/src/linux-kernel/infiniband/core/at.c (revision 2278) +++ gen2/branches/shaharf-ibat/src/linux-kernel/infiniband/core/at.c (working copy) @@ -132,7 +132,7 @@ struct async { int sa_id; }; -struct async pending_reqs; /* dummy head for cyclic list */ +static struct async pending_reqs; /* dummy head for cyclic list */ struct ib_at_src { u32 ip; @@ -711,7 +711,7 @@ static void req_free(struct async *pend) pend->status = IB_AT_STATUS_INVALID; pend->type = IBAT_REQ_NONE; - pend->sa_query = 0; + pend->sa_query = NULL; } static int req_start(struct async *q, struct async *pend, @@ -727,7 +727,7 @@ static int req_start(struct async *q, st if (parent) { DEBUG("wait on parent %p", parent); - pend->next = pend->prev = 0; + pend->next = pend->prev = NULL; pend->parent = parent; pend->waiting = parent->waiting; parent->waiting = pend; @@ -735,8 +735,8 @@ static int req_start(struct async *q, st return 0; /* waiting on other request */ } - pend->waiting = 0; - pend->parent = 0; + pend->waiting = NULL; + pend->parent = NULL; DEBUG("link to pending list %p", q); pend->next = q; @@ -787,7 +787,7 @@ static void req_end(struct async *pend, if (!*rr) WARN("pending request not found in parent request!"); - pend->waiting = 0; + pend->waiting = NULL; DEBUG("child %p removed from parent %p list", pend, pend->parent); } @@ -796,10 +796,10 @@ static void req_end(struct async *pend, DEBUG("pend %p ending child req %p", pend, waiting); pend->waiting = waiting->waiting; - waiting->waiting = 0; - waiting->parent = 0; + waiting->waiting = NULL; + waiting->parent = NULL; - req_end(waiting, nrec, 0); + req_end(waiting, nrec, NULL); } if (pend->next) { @@ -886,7 +886,7 @@ static struct async *lookup_pending(stru } spin_unlock_irqrestore(&q->lock, flags); - return a == q ? 0 : a; + return a == q ? NULL : a; } static struct async *lookup_req_id(struct async *q, u64 id) @@ -901,7 +901,7 @@ static struct async *lookup_req_id(struc break; spin_unlock_irqrestore(&q->lock, flags); - return a == q ? 0 : a; + return a == q ? NULL : a; } static void flush_pending(struct async *q) @@ -912,7 +912,7 @@ static void flush_pending(struct async * DEBUG("flushing pening q %p", q); spin_lock_irqsave(&q->lock, flags); while ((a = q->next) != q) - req_end(a, -EINTR, 0); + req_end(a, -EINTR, NULL); spin_unlock_irqrestore(&q->lock, flags); } @@ -978,7 +978,7 @@ route_req_complete(struct route_req *req for (pend = req->pend.waiting; pend; pend = pend->waiting) route_req_output(req, pend->data); - req_end(&req->pend, 1, 0); + req_end(&req->pend, 1, NULL); } static void @@ -990,7 +990,7 @@ ats_route_req_complete(int status, struc DEBUG("req %p", req); - req->pend.sa_query = 0; + req->pend.sa_query = NULL; if (status) { DEBUG("status %d - check if should retry", status); @@ -1075,7 +1075,7 @@ path_req_complete(int status, struct ib_ pend->nelem = path_req_output(req, resp, 1, pend->data, pend->nelem); - req_end(&req->pend, req->pend.nelem, 0); + req_end(&req->pend, req->pend.nelem, NULL); spin_unlock_irqrestore(&pending_reqs.lock, flags); } @@ -1102,7 +1102,7 @@ static void ib_at_sweep(void *data) (req->dst_ip & 0xff000000) >> 24, jiffies, pend->start); - req_end(pend, -ETIMEDOUT, 0); + req_end(pend, -ETIMEDOUT, NULL); } } @@ -1541,7 +1541,7 @@ int ib_at_cancel(u64 req_id) /* Promote first child to be pending req */ if ((child = a->waiting)) { - child->parent = 0; + child->parent = NULL; /* link child after parent in pending list */ child->next = a->next; @@ -1549,10 +1549,10 @@ int ib_at_cancel(u64 req_id) a->next->prev = child; a->next = child; - a->waiting = 0; /* clear to avoid cancelling childs */ + a->waiting = NULL; /* clear to avoid cancelling childs */ } - req_end(a, -EINTR, 0); + req_end(a, -EINTR, NULL); spin_unlock_irqrestore(&pending_reqs.lock, flags); From itamar at mellanox.co.il Mon May 16 10:16:42 2005 From: itamar at mellanox.co.il (Itamar) Date: Mon, 16 May 2005 20:16:42 +0300 Subject: [openib-general] [patch][kdapl][trivial] fix debug prints Message-ID: <20050516171642.GA32298@mellanox.co.il> change print from mr to lmr_context add \n at end of print Signed-off-by: Itamar Rabenstein Index: dapl_ep_post_rdma_write.c =================================================================== --- dapl_ep_post_rdma_write.c (revision 2355) +++ dapl_ep_post_rdma_write.c (working copy) @@ -91,7 +91,7 @@ OP_WR_RDMA_WRITE); dapl_dbg_log(DAPL_DBG_TYPE_RTN, - "dapl_ep_post_rdma_write () returns 0x%x", dat_status); + "dapl_ep_post_rdma_write () returns 0x%x\n", dat_status); return dat_status; } Index: dapl_openib_util.c =================================================================== --- dapl_openib_util.c (revision 2355) +++ dapl_openib_util.c (working copy) @@ -433,10 +433,6 @@ ib_status); return dapl_ib_status_convert(ib_status); } - dapl_dbg_log(DAPL_DBG_TYPE_UTIL, - "dapl_ib_mr_register_physical(%p %d) got handle %p\n", - buf_list, length, mr); - #if 0 /* ib_query_mr is not implemented yet in OpenIB!!! */ ib_status = ib_query_mr(mr, &attr); @@ -453,6 +449,9 @@ lmr->param.rmr_context = mr->rkey; lmr->mr_handle = mr; + dapl_dbg_log(DAPL_DBG_TYPE_UTIL, + "dapl_ib_mr_register_physical(%p %d) got lkey 0x%x \n", + buf_list, length, lmr->param.lmr_context); return (DAT_SUCCESS); } -- Itamar From jlentini at netapp.com Mon May 16 10:17:44 2005 From: jlentini at netapp.com (James Lentini) Date: Mon, 16 May 2005 13:17:44 -0400 (EDT) Subject: [openib-general] Re: [PATCH] make private data pointers const In-Reply-To: <4288D141.307@ichips.intel.com> References: <4288D141.307@ichips.intel.com> Message-ID: Sean, Sorry about that. The message was garbled on my end. I'm not sure if it was my mail client or the mail server. I'll fix it and resend. james On Mon, 16 May 2005, Sean Hefty wrote: > James Lentini wrote: >> >> This patch affects ib_cm.h and cm.c, which appear to be maintained by Sean, >> and ucm.c, which appears to be maintained by Libor. >> >> The patch adds the const qualifier to private data pointers passed into the >> IB layer. > > My system rejects all of the changes. Has anyone else tried applying this > patch? > > - Sean > From itamar at mellanox.co.il Mon May 16 10:22:47 2005 From: itamar at mellanox.co.il (Itamar) Date: Mon, 16 May 2005 20:22:47 +0300 Subject: [openib-general] [PATCH][kdapl] delete dead code (not in Makefile) Message-ID: <20050516172247.GA32342@mellanox.co.il> delete dead code (not in Makefile) Signed-off-by: Itamar Rabenstein Index: dapl_debug.c =================================================================== --- dapl_debug.c (revision 2357) +++ dapl_debug.c (working copy) @@ -1,45 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -#include "dapl_debug.h" -#include "dapl_module.h" -#include "dapl.h" - -#ifdef DAPL_DBG - -void dapl_dbg_log(DAPL_DBG_TYPE type, const char *fmt, ...) -{ - va_list args; - - if (type & g_dapl_dbg_type) { - va_start(args, fmt); - dapl_os_vprintf(fmt, args); - va_end(args); - } -} - -#endif /* DAPL_DBG */ Index: dapl_debug.h =================================================================== --- dapl_debug.h (revision 2357) +++ dapl_debug.h (working copy) @@ -1,63 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * - * Defines common deuggging flags and data - * - * $Id$ - */ - -#ifndef DAPL_DEBUG_H -#define DAPL_DEBUG_H - -/* - * Use these bits to enable various tracing/debug options. Each bit - * represents debugging in a particular subsystem or area of the code. - */ -typedef enum { - DAPL_DBG_TYPE_ERR = 0x0001, - DAPL_DBG_TYPE_WARN = 0x0002, - DAPL_DBG_TYPE_EVD = 0x0004, - DAPL_DBG_TYPE_CM = 0x0008, - DAPL_DBG_TYPE_EP = 0x0010, - DAPL_DBG_TYPE_UTIL = 0x0020, - DAPL_DBG_TYPE_CALLBACK = 0x0040, - DAPL_DBG_TYPE_DTO_COMP_ERR = 0x0080, - DAPL_DBG_TYPE_API = 0x0100, - DAPL_DBG_TYPE_RTN = 0x0200, - DAPL_DBG_TYPE_EXCEPTION = 0x0400, - DAPL_DBG_TYPE_SRQ = 0x0800 -} DAPL_DBG_TYPE; - -#if defined(DAPL_DBG) -extern void dapl_dbg_log(DAPL_DBG_TYPE type, const char *fmt, ...); -#else /* !DAPL_DBG */ -#define dapl_dbg_log(...) -#endif /* DAPL_DBG */ - -#endif /* DAPL_DEBUG_H */ -- Itamar From roland at topspin.com Mon May 16 10:25:26 2005 From: roland at topspin.com (Roland Dreier) Date: Mon, 16 May 2005 10:25:26 -0700 Subject: [openib-general] Re: [PATCH 1/2] (fixed) mthca: max_inline_data support In-Reply-To: <20050514195703.GC10140@mellanox.co.il> (Michael S. Tsirkin's message of "Sat, 14 May 2005 22:57:03 +0300") References: <20050509141456.GH9941@mellanox.co.il> <52fyww1h90.fsf@topspin.com> <20050510153543.GG2474@mellanox.co.il> <524qd6sw1h.fsf@topspin.com> <20050514195703.GC10140@mellanox.co.il> Message-ID: <52psvrm6bd.fsf@topspin.com> Michael> Roland, roundup_pow_of_two assumes that it gets a non-0 Michael> parameter, while I think its legal to have one of Michael> max_recv_wr / max_send_wr equal 0. Is that right? Yes, good point. I forgot about the possibility of zero-length work queues. Michael> Lets keep lines <= 80, please? I'm willing to go a little over 80 columns to save vertical space -- this line was a 95-character monster so I did fix it up, but to me it's not worth breaking up a line just because it's 84 characters or so. Anyway here's what I checked in: --- infiniband/hw/mthca/mthca_qp.c (revision 2347) +++ infiniband/hw/mthca/mthca_qp.c (working copy) @@ -1211,8 +1211,10 @@ static int mthca_set_qp_size(struct mthc struct mthca_qp *qp) { if (mthca_is_memfree(dev)) { - qp->rq.max = roundup_pow_of_two(cap->max_recv_wr); - qp->sq.max = roundup_pow_of_two(cap->max_send_wr); + qp->rq.max = cap->max_recv_wr ? + roundup_pow_of_two(cap->max_recv_wr) : 0; + qp->sq.max = cap->max_send_wr ? + roundup_pow_of_two(cap->max_send_wr) : 0; } else { qp->rq.max = cap->max_recv_wr; qp->sq.max = cap->max_send_wr; @@ -1221,7 +1223,8 @@ static int mthca_set_qp_size(struct mthc qp->rq.max_gs = cap->max_recv_sge; qp->sq.max_gs = max_t(int, cap->max_send_sge, ALIGN(cap->max_inline_data + MTHCA_INLINE_HEADER_SIZE, - MTHCA_INLINE_CHUNK_SIZE) / sizeof (struct mthca_data_seg)); + MTHCA_INLINE_CHUNK_SIZE) / + sizeof (struct mthca_data_seg)); /* * For MLX transport we need 2 extra S/G entries: From halr at voltaire.com Mon May 16 10:23:32 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 16 May 2005 13:23:32 -0400 Subject: [openib-general] Re: [PATCH] at: pull sparse fixes from james's branch to shahar's In-Reply-To: <1116263241.3152.7.camel@duffman> References: <1116263241.3152.7.camel@duffman> Message-ID: <1116264212.4480.942.camel@localhost.localdomain> Hi Tom, On Mon, 2005-05-16 at 13:07, Tom Duffy wrote: > Hal, > > Can you please pull the sparse fixes from James's branch to Shahar's? I did on Thursday AM (r2310). Did you update ? Did I miss something ? -- Hal > svn diff -r2134:2238 gen2/users/jlentini/linux-kernel/patches/at.c > > Or here it is against the latest r2278 in shaharf-ibat. > > Signed-off-by: Tom Duffy > > Index: gen2/branches/shaharf-ibat/src/linux-kernel/infiniband/core/at.c > =================================================================== > --- gen2/branches/shaharf-ibat/src/linux-kernel/infiniband/core/at.c (revision 2278) > +++ gen2/branches/shaharf-ibat/src/linux-kernel/infiniband/core/at.c (working copy) > @@ -132,7 +132,7 @@ struct async { > int sa_id; > }; > > -struct async pending_reqs; /* dummy head for cyclic list */ > +static struct async pending_reqs; /* dummy head for cyclic list */ > > struct ib_at_src { > u32 ip; > @@ -711,7 +711,7 @@ static void req_free(struct async *pend) > > pend->status = IB_AT_STATUS_INVALID; > pend->type = IBAT_REQ_NONE; > - pend->sa_query = 0; > + pend->sa_query = NULL; > } > > static int req_start(struct async *q, struct async *pend, > @@ -727,7 +727,7 @@ static int req_start(struct async *q, st > > if (parent) { > DEBUG("wait on parent %p", parent); > - pend->next = pend->prev = 0; > + pend->next = pend->prev = NULL; > pend->parent = parent; > pend->waiting = parent->waiting; > parent->waiting = pend; > @@ -735,8 +735,8 @@ static int req_start(struct async *q, st > return 0; /* waiting on other request */ > } > > - pend->waiting = 0; > - pend->parent = 0; > + pend->waiting = NULL; > + pend->parent = NULL; > > DEBUG("link to pending list %p", q); > pend->next = q; > @@ -787,7 +787,7 @@ static void req_end(struct async *pend, > if (!*rr) > WARN("pending request not found in parent request!"); > > - pend->waiting = 0; > + pend->waiting = NULL; > DEBUG("child %p removed from parent %p list", > pend, pend->parent); > } > @@ -796,10 +796,10 @@ static void req_end(struct async *pend, > DEBUG("pend %p ending child req %p", pend, waiting); > pend->waiting = waiting->waiting; > > - waiting->waiting = 0; > - waiting->parent = 0; > + waiting->waiting = NULL; > + waiting->parent = NULL; > > - req_end(waiting, nrec, 0); > + req_end(waiting, nrec, NULL); > } > > if (pend->next) { > @@ -886,7 +886,7 @@ static struct async *lookup_pending(stru > } > > spin_unlock_irqrestore(&q->lock, flags); > - return a == q ? 0 : a; > + return a == q ? NULL : a; > } > > static struct async *lookup_req_id(struct async *q, u64 id) > @@ -901,7 +901,7 @@ static struct async *lookup_req_id(struc > break; > > spin_unlock_irqrestore(&q->lock, flags); > - return a == q ? 0 : a; > + return a == q ? NULL : a; > } > > static void flush_pending(struct async *q) > @@ -912,7 +912,7 @@ static void flush_pending(struct async * > DEBUG("flushing pening q %p", q); > spin_lock_irqsave(&q->lock, flags); > while ((a = q->next) != q) > - req_end(a, -EINTR, 0); > + req_end(a, -EINTR, NULL); > spin_unlock_irqrestore(&q->lock, flags); > } > > @@ -978,7 +978,7 @@ route_req_complete(struct route_req *req > for (pend = req->pend.waiting; pend; pend = pend->waiting) > route_req_output(req, pend->data); > > - req_end(&req->pend, 1, 0); > + req_end(&req->pend, 1, NULL); > } > > static void > @@ -990,7 +990,7 @@ ats_route_req_complete(int status, struc > > DEBUG("req %p", req); > > - req->pend.sa_query = 0; > + req->pend.sa_query = NULL; > > if (status) { > DEBUG("status %d - check if should retry", status); > @@ -1075,7 +1075,7 @@ path_req_complete(int status, struct ib_ > pend->nelem = path_req_output(req, resp, 1, > pend->data, pend->nelem); > > - req_end(&req->pend, req->pend.nelem, 0); > + req_end(&req->pend, req->pend.nelem, NULL); > spin_unlock_irqrestore(&pending_reqs.lock, flags); > } > > @@ -1102,7 +1102,7 @@ static void ib_at_sweep(void *data) > (req->dst_ip & 0xff000000) >> 24, > jiffies, pend->start); > > - req_end(pend, -ETIMEDOUT, 0); > + req_end(pend, -ETIMEDOUT, NULL); > } > } > > @@ -1541,7 +1541,7 @@ int ib_at_cancel(u64 req_id) > > /* Promote first child to be pending req */ > if ((child = a->waiting)) { > - child->parent = 0; > + child->parent = NULL; > > /* link child after parent in pending list */ > child->next = a->next; > @@ -1549,10 +1549,10 @@ int ib_at_cancel(u64 req_id) > a->next->prev = child; > a->next = child; > > - a->waiting = 0; /* clear to avoid cancelling childs */ > + a->waiting = NULL; /* clear to avoid cancelling childs */ > } > > - req_end(a, -EINTR, 0); > + req_end(a, -EINTR, NULL); > > spin_unlock_irqrestore(&pending_reqs.lock, flags); > > From jlentini at netapp.com Mon May 16 10:29:46 2005 From: jlentini at netapp.com (James Lentini) Date: Mon, 16 May 2005 13:29:46 -0400 (EDT) Subject: [openib-general] Re: [PATCH] make private data pointers const In-Reply-To: References: <4288D141.307@ichips.intel.com> Message-ID: I don't see any obvious configuration error in my mail client, so I'm resending the patch as an attachment. james On Mon, 16 May 2005, James Lentini wrote: > > Sean, > > Sorry about that. The message was garbled on my end. I'm not sure if it was > my mail client or the mail server. I'll fix it and resend. > > james > > On Mon, 16 May 2005, Sean Hefty wrote: > >> James Lentini wrote: >>> >>> This patch affects ib_cm.h and cm.c, which appear to be maintained by >>> Sean, and ucm.c, which appears to be maintained by Libor. >>> >>> The patch adds the const qualifier to private data pointers passed into >>> the IB layer. >> >> My system rejects all of the changes. Has anyone else tried applying this >> patch? >> >> - Sean >> > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > -------------- next part -------------- Add the const qualifier to private data pointers passed into the IB layer. Signed-off-by: James Lentini Index: include/ib_cm.h =================================================================== --- include/ib_cm.h (revision 2357) +++ include/ib_cm.h (working copy) @@ -333,7 +333,7 @@ u32 qp_num; enum ib_qp_type qp_type; u32 starting_psn; - void *private_data; + const void *private_data; u8 private_data_len; u8 peer_to_peer; u8 responder_resources; @@ -360,7 +360,7 @@ struct ib_cm_rep_param { u32 qp_num; u32 starting_psn; - void *private_data; + const void *private_data; u8 private_data_len; u8 responder_resources; u8 initiator_depth; @@ -391,7 +391,7 @@ * @private_data_len: Size of the private data buffer, in bytes. */ int ib_send_cm_rtu(struct ib_cm_id *cm_id, - void *private_data, + const void *private_data, u8 private_data_len); /** @@ -404,7 +404,7 @@ * @private_data_len: Size of the private data buffer, in bytes. */ int ib_send_cm_dreq(struct ib_cm_id *cm_id, - void *private_data, + const void *private_data, u8 private_data_len); /** @@ -416,7 +416,7 @@ * @private_data_len: Size of the private data buffer, in bytes. */ int ib_send_cm_drep(struct ib_cm_id *cm_id, - void *private_data, + const void *private_data, u8 private_data_len); /** @@ -444,7 +444,7 @@ enum ib_cm_rej_reason reason, void *ari, u8 ari_length, - void *private_data, + const void *private_data, u8 private_data_len); /** @@ -459,7 +459,7 @@ */ int ib_send_cm_mra(struct ib_cm_id *cm_id, u8 service_timeout, - void *private_data, + const void *private_data, u8 private_data_len); /** @@ -474,7 +474,7 @@ */ int ib_send_cm_lap(struct ib_cm_id *cm_id, struct ib_sa_path_rec *alternate_path, - void *private_data, + const void *private_data, u8 private_data_len); /** @@ -513,14 +513,14 @@ enum ib_cm_apr_status status, void *info, u8 info_length, - void *private_data, + const void *private_data, u8 private_data_len); struct ib_cm_sidr_req_param { struct ib_sa_path_rec *path; u64 service_id; int timeout_ms; - void *private_data; + const void *private_data; u8 private_data_len; u8 max_cm_retries; u16 pkey; @@ -540,9 +540,9 @@ u32 qp_num; u32 qkey; enum ib_cm_sidr_status status; - void *info; + const void *info; u8 info_length; - void *private_data; + const void *private_data; u8 private_data_len; }; Index: core/cm.c =================================================================== --- core/cm.c (revision 2357) +++ core/cm.c (working copy) @@ -1205,7 +1205,7 @@ static void cm_format_rtu(struct cm_rtu_msg *rtu_msg, struct cm_id_private *cm_id_priv, - void *private_data, + const void *private_data, u8 private_data_len) { cm_format_mad_hdr(&rtu_msg->hdr, CM_RTU_ATTR_ID, @@ -1236,7 +1236,7 @@ } int ib_send_cm_rtu(struct ib_cm_id *cm_id, - void *private_data, + const void *private_data, u8 private_data_len) { struct cm_id_private *cm_id_priv; @@ -1471,7 +1471,7 @@ static void cm_format_dreq(struct cm_dreq_msg *dreq_msg, struct cm_id_private *cm_id_priv, - void *private_data, + const void *private_data, u8 private_data_len) { cm_format_mad_hdr(&dreq_msg->hdr, CM_DREQ_ATTR_ID, @@ -1486,7 +1486,7 @@ } int ib_send_cm_dreq(struct ib_cm_id *cm_id, - void *private_data, + const void *private_data, u8 private_data_len) { struct cm_id_private *cm_id_priv; @@ -1531,7 +1531,7 @@ static void cm_format_drep(struct cm_drep_msg *drep_msg, struct cm_id_private *cm_id_priv, - void *private_data, + const void *private_data, u8 private_data_len) { /* todo: TID should match received DREQ */ @@ -1563,7 +1563,7 @@ } int ib_send_cm_drep(struct ib_cm_id *cm_id, - void *private_data, + const void *private_data, u8 private_data_len) { struct cm_id_private *cm_id_priv; @@ -1694,7 +1694,7 @@ enum ib_cm_rej_reason reason, void *ari, u8 ari_length, - void *private_data, + const void *private_data, u8 private_data_len) { /* todo: TID should match received REQ */ @@ -1732,7 +1732,7 @@ enum ib_cm_rej_reason reason, void *ari, u8 ari_length, - void *private_data, + const void *private_data, u8 private_data_len) { struct cm_id_private *cm_id_priv; @@ -1891,7 +1891,7 @@ static void cm_format_mra(struct cm_mra_msg *mra_msg, struct cm_id_private *cm_id_priv, u8 service_timeout, - void *private_data, + const void *private_data, u8 private_data_len) { enum cm_msg_sequence msg_sequence; @@ -1932,7 +1932,7 @@ int ib_send_cm_mra(struct ib_cm_id *cm_id, u8 service_timeout, - void *private_data, + const void *private_data, u8 private_data_len) { struct cm_id_private *cm_id_priv; @@ -2000,7 +2000,7 @@ static void cm_format_lap(struct cm_lap_msg *lap_msg, struct cm_id_private *cm_id_priv, struct ib_sa_path_rec *alternate_path, - void *private_data, + const void *private_data, u8 private_data_len) { cm_format_mad_hdr(&lap_msg->hdr, CM_LAP_ATTR_ID, @@ -2030,7 +2030,7 @@ int ib_send_cm_lap(struct ib_cm_id *cm_id, struct ib_sa_path_rec *alternate_path, - void *private_data, + const void *private_data, u8 private_data_len) { struct cm_id_private *cm_id_priv; @@ -2143,7 +2143,7 @@ enum ib_cm_apr_status status, void *info, u8 info_length, - void *private_data, + const void *private_data, u8 private_data_len) { /* todo: TID should match received LAP */ @@ -2167,7 +2167,7 @@ enum ib_cm_apr_status status, void *info, u8 info_length, - void *private_data, + const void *private_data, u8 private_data_len) { struct cm_id_private *cm_id_priv; Index: core/ucm.c =================================================================== --- core/ucm.c (revision 2357) +++ core/ucm.c (working copy) @@ -721,7 +721,7 @@ return result; } -static int ib_ucm_alloc_data(void **dest, u64 src, u32 len) +static int ib_ucm_alloc_data(const void **dest, u64 src, u32 len) { void *data; @@ -913,12 +913,12 @@ static ssize_t ib_ucm_send_private_data(struct ib_ucm_file *file, const char __user *inbuf, int in_len, int (*func)(struct ib_cm_id *cm_id, - void *private_data, + const void *private_data, u8 private_data_len)) { struct ib_ucm_private_data cmd; struct ib_ucm_context *ctx; - void *private_data = NULL; + const void *private_data = NULL; int result; if (copy_from_user(&cmd, inbuf, sizeof(cmd))) @@ -974,15 +974,15 @@ const char __user *inbuf, int in_len, int (*func)(struct ib_cm_id *cm_id, int status, - void *info, + const void *info, u8 info_len, - void *data, + const void *data, u8 data_len)) { struct ib_ucm_context *ctx; struct ib_ucm_info cmd; - void *data = NULL; - void *info = NULL; + const void *data = NULL; + const void *info = NULL; int result; if (copy_from_user(&cmd, inbuf, sizeof(cmd))) @@ -1041,7 +1041,7 @@ { struct ib_ucm_context *ctx; struct ib_ucm_mra cmd; - void *data = NULL; + const void *data = NULL; int result; if (copy_from_user(&cmd, inbuf, sizeof(cmd))) @@ -1080,7 +1080,7 @@ struct ib_ucm_context *ctx; struct ib_sa_path_rec *path = NULL; struct ib_ucm_lap cmd; - void *data = NULL; + const void *data = NULL; int result; if (copy_from_user(&cmd, inbuf, sizeof(cmd))) From tduffy at sun.com Mon May 16 10:32:05 2005 From: tduffy at sun.com (Tom Duffy) Date: Mon, 16 May 2005 10:32:05 -0700 Subject: [openib-general] Re: [PATCH] at: pull sparse fixes from james's branch to shahar's In-Reply-To: <1116264212.4480.942.camel@localhost.localdomain> References: <1116263241.3152.7.camel@duffman> <1116264212.4480.942.camel@localhost.localdomain> Message-ID: <1116264725.25812.1.camel@duffman> On Mon, 2005-05-16 at 13:23 -0400, Hal Rosenstock wrote: > I did on Thursday AM (r2310). Did you update ? Did I miss something ? My bad. You are right. I had not updated that branch of the tree. Thanks, -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From halr at voltaire.com Mon May 16 10:30:29 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 16 May 2005 13:30:29 -0400 Subject: [openib-general] trunk/src/linux-kernel/TODO Message-ID: <1116264628.4480.944.camel@localhost.localdomain> How much of this file is still relevant ? Perhaps it should be eliminated or at the least trimmed. -- Hal From halr at voltaire.com Mon May 16 10:43:43 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 16 May 2005 13:43:43 -0400 Subject: [openib-general] Re: [PATCH] at: pull sparse fixes from james's branch to shahar's In-Reply-To: <1116264725.25812.1.camel@duffman> References: <1116263241.3152.7.camel@duffman> <1116264212.4480.942.camel@localhost.localdomain> <1116264725.25812.1.camel@duffman> Message-ID: <1116265422.4480.947.camel@localhost.localdomain> On Mon, 2005-05-16 at 13:32, Tom Duffy wrote: > I had not updated that branch of the tree. BTW, this branch is currently "use at your own risk". -- Hal From tduffy at sun.com Mon May 16 10:51:50 2005 From: tduffy at sun.com (Tom Duffy) Date: Mon, 16 May 2005 10:51:50 -0700 Subject: [openib-general] Re: [PATCH] make private data pointers const In-Reply-To: References: <4288D141.307@ichips.intel.com> Message-ID: <1116265910.25812.4.camel@duffman> On Mon, 2005-05-16 at 13:29 -0400, James Lentini wrote: > I don't see any obvious configuration error in my mail client, so I'm > resending the patch as an attachment. The attachment patch applied cleanly. It seems that your first inline patch contained an extra space at the beginning on each line. It wouldn't even apply cleanly with the -l option to patch. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From mshefty at ichips.intel.com Mon May 16 11:06:48 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Mon, 16 May 2005 11:06:48 -0700 Subject: [openib-general] Re: [PATCH] make private data pointers const In-Reply-To: References: <4288D141.307@ichips.intel.com> Message-ID: <4288E138.6010602@ichips.intel.com> James Lentini wrote: > I don't see any obvious configuration error in my mail client, so I'm > resending the patch as an attachment. Thanks - committed. - Sean From mst at mellanox.co.il Mon May 16 12:16:51 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Mon, 16 May 2005 22:16:51 +0300 Subject: [openib-general] Re: [PATCH 1/2] (fixed) mthca: max_inline_data support In-Reply-To: <52psvrm6bd.fsf@topspin.com> References: <20050509141456.GH9941@mellanox.co.il> <52fyww1h90.fsf@topspin.com> <20050510153543.GG2474@mellanox.co.il> <524qd6sw1h.fsf@topspin.com> <20050514195703.GC10140@mellanox.co.il> <52psvrm6bd.fsf@topspin.com> Message-ID: <20050516191651.GC5743@mellanox.co.il> Quoting r. Roland Dreier : > Subject: Re: [PATCH 1/2] (fixed) mthca: max_inline_data support > > Michael> Roland, roundup_pow_of_two assumes that it gets a non-0 > Michael> parameter, while I think its legal to have one of > Michael> max_recv_wr / max_send_wr equal 0. Is that right? > > Yes, good point. I forgot about the possibility of zero-length work queues. > > --- infiniband/hw/mthca/mthca_qp.c (revision 2347) > +++ infiniband/hw/mthca/mthca_qp.c (working copy) > @@ -1211,8 +1211,10 @@ static int mthca_set_qp_size(struct mthc > struct mthca_qp *qp) > { > if (mthca_is_memfree(dev)) { > - qp->rq.max = roundup_pow_of_two(cap->max_recv_wr); > - qp->sq.max = roundup_pow_of_two(cap->max_send_wr); > + qp->rq.max = cap->max_recv_wr ? > + roundup_pow_of_two(cap->max_recv_wr) : 0; > + qp->sq.max = cap->max_send_wr ? > + roundup_pow_of_two(cap->max_send_wr) : 0; Another problem is that fls may return 32 and you end up with 1 << 32 which AFAIK has undefined behaviour in C. I guess kernel clients can be trusted not to, but uverbs better check this. -- MST - Michael S. Tsirkin From halr at voltaire.com Mon May 16 12:52:36 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 16 May 2005 15:52:36 -0400 Subject: [openib-general] [PATCH] [kdapl] Various AT changes Message-ID: <1116273155.4480.978.camel@localhost.localdomain> Various AT changes including: Get API in line with changes on shahar-ibat branch Integrate various changes from shahar-ibat branch which apply to the "level of functionality" originally implemented (e.g. no Service Record support) Fix a couple of minor bugs found by code inspection Better debug support Note that this does not fix the slab corruption problem :-( Signed-off-by: Hal Rosenstock Index: ib_at.h =================================================================== --- ib_at.h (revision 2331) +++ ib_at.h (working copy) @@ -30,25 +30,28 @@ * SOFTWARE. * * - * $Id:$ + * $Id$ */ #if !defined( IB_AT_H ) #define IB_AT_H +#include +#include + enum ib_at_multipathing_type { IB_AT_PATH_SAME_PORT = 0, - IB_AT_PATH_SAME_HCA = 1, /* but different ports if applicable */ - IB_AT_PATH_SAME_SYSTEM = 2, /* but different ports if applicable */ + IB_AT_PATH_SAME_HCA = 1, /* but different ports if applicable */ + IB_AT_PATH_SAME_SYSTEM = 2, /* but different ports if applicable */ IB_AT_PATH_INDEPENDENT_HCA = 3, - IB_AT_PATH_SRC_ROUTE = 4, /* application controlled multipathing */ + IB_AT_PATH_SRC_ROUTE = 4, /* application controlled multipathing */ }; enum ib_at_route_flags { - IB_AT_ROUTE_USE_DEFAULTS = 0, - IB_AT_ROUTE_FORCE_ATS, - IB_AT_ROUTE_FORCE_ARP, - IB_AT_ROUTE_FORCE_RESOLVE, + IB_AT_ROUTE_USE_DEFAULTS = 0, + IB_AT_ROUTE_FORCE_ATS = 1, + IB_AT_ROUTE_FORCE_ARP = 2, + IB_AT_ROUTE_FORCE_RESOLVE = 4, }; struct ib_at_path_attr { @@ -169,7 +172,7 @@ * See ib_at_completion structure documentation for asynchronous * operation details. */ -int ib_at_ips_by_gid(union ib_gid gid, u32 *dst_ips, int nips, +int ib_at_ips_by_gid(union ib_gid *gid, u32 *dst_ips, int nips, struct ib_at_completion *async_comp); /** @@ -208,7 +211,7 @@ * @req_id: asynchronous request ID ib_at_op_status * * Return non-negative ib_at_op_status value, - * or -EINVAL if the reqest ID is invalid. + * or -EINVAL if the request ID is invalid. */ int ib_at_status(u64 req_id); Index: at.c =================================================================== --- at.c (revision 2331) +++ at.c (working copy) @@ -30,7 +30,7 @@ * SOFTWARE. * * - * $Id:$ + * $Id$ */ #include @@ -118,7 +118,7 @@ int sa_id; }; -static struct async pending_reqs; /* dummy head for cyclic list */ +struct async pending_reqs; /* dummy head for cyclic list */ struct ib_at_src { u32 ip; @@ -158,7 +158,6 @@ static void path_req_complete(int stat, struct ib_sa_path_rec *ret, void *ctx); static int resolve_path(struct path_req *req); - static int resolve_ip(struct ib_at_src *src, u32 dst_ip, u32 src_ip, int tos, union ib_gid *dgid) { @@ -254,7 +253,7 @@ src->dev = priv->ca; src->port = priv->port; src->pkey = cpu_to_be16(priv->pkey); - memcpy(&src->gid, (ipoib_dev->dev_addr + 4), sizeof(src->gid)); + memcpy(&src->gid, ipoib_dev->dev_addr + 4, sizeof(src->gid)); if (!dgid) return 0; @@ -264,7 +263,7 @@ * the IB device which was found. */ if (rt->u.dst.neighbour->dev->flags & IFF_LOOPBACK) { - memcpy(dgid, (ipoib_dev->dev_addr + 4), + memcpy(dgid, ipoib_dev->dev_addr + 4, sizeof(union ib_gid)); return 1; @@ -272,7 +271,7 @@ if ((NUD_CONNECTED|NUD_DELAY|NUD_PROBE) & rt->u.dst.neighbour->nud_state) { - memcpy(dgid, (rt->u.dst.neighbour->ha + 4), + memcpy(dgid, rt->u.dst.neighbour->ha + 4, sizeof(union ib_gid)); return 1; @@ -285,9 +284,17 @@ static u64 alloc_req_id(void) { - static u64 req_id = 1; + static u64 req_id = 0; + u64 new_id; + unsigned long flags; - return ++req_id; + spin_lock_irqsave(&pending_reqs.lock, flags); + new_id = ++req_id; + if (!new_id) + new_id = ++req_id; + spin_unlock_irqrestore(&pending_reqs.lock, flags); + + return new_id; } static void req_init(struct async *pend, void *data, int nelem, int type, @@ -361,7 +368,7 @@ { struct async *pend = v; - DEBUG("complete req %p\n", pend); + DEBUG("complete pend %p", pend); pend->comp.fn(pend->comp.req_id, pend->comp.context, pend->nelem); @@ -373,20 +380,23 @@ struct async **rr, *waiting; unsigned long flags = 0; - DEBUG("pend %p nrec %d", pend, nrec); + DEBUG("pend %p nrec %d async %p", pend, nrec, q); if (pend->status != IB_AT_STATUS_PENDING) - WARN("pend %p already completed??", pend); + WARN("pend %p already completed? status %d", pend, pend->status); pend->status = nrec < 0 ? IB_AT_STATUS_ERROR : IB_AT_STATUS_COMPLETED; - if (pend->sa_query) + if (pend->sa_query) { ib_sa_cancel_query(pend->sa_id, pend->sa_query); + pend->sa_query = NULL; + } if (q) spin_lock_irqsave(&q->lock, flags); if (pend->parent) { + DEBUG("pend->parent %p", pend->parent); for (rr = &pend->parent->waiting; *rr; rr = &(*rr)->waiting) if (*rr == pend) { *rr = (*rr)->waiting; @@ -476,11 +486,13 @@ unsigned long flags; struct async *a; - DEBUG("lookup in q %p req %p", q, new); + DEBUG("lookup in q %p pending %p", q, new); spin_lock_irqsave(&q->lock, flags); - for (a = q->next; a != q; a = a->next) + for (a = q->next; a != q; a = a->next) { + DEBUG("%d %d", a->type, type); if (a->type == type && same_fn(a, new)) break; + } spin_unlock_irqrestore(&q->lock, flags); return a == q ? NULL : a; @@ -574,13 +586,14 @@ DEBUG("req %p", req); if (req->pend.parent) { - WARN("path_req_complete for child req %p???", req); + WARN("for child req %p???", req); return; } if (status) { - DEBUG("timed out - check if should retry"); - if (jiffies - req->pend.start < IB_AT_REQ_TIMEOUT) + DEBUG("status %d - check if should retry", status); + if (status == -ETIMEDOUT && + jiffies - req->pend.start < IB_AT_REQ_TIMEOUT) resolve_path(req); else req_end(&req->pend, -ETIMEDOUT, &pending_reqs); @@ -605,6 +618,7 @@ { struct async *pend, *next; struct route_req *req; + struct path_req *preq; unsigned long flags; DEBUG("start sweeping"); @@ -613,18 +627,36 @@ for (pend = pending_reqs.next; pend != &pending_reqs; pend = next) { next = pend->next; - req = container_of(pend, struct route_req, pend); + switch (pend->type) { + case IBAT_REQ_ARP: + case IBAT_REQ_ATS: + req = container_of(pend, struct route_req, pend); - DEBUG("examining route req %p pend %p", req, pend); - if (jiffies > (pend->start + IB_AT_REQ_TIMEOUT)) { - DEBUG("req delete <%d.%d.%d.%d> <%lu:%lu>", - (req->dst_ip & 0x000000ff), - (req->dst_ip & 0x0000ff00) >> 8, - (req->dst_ip & 0x00ff0000) >> 16, - (req->dst_ip & 0xff000000) >> 24, - jiffies, pend->start); + DEBUG("examining route req %p pend %p", req, pend); + if (jiffies > pend->start + IB_AT_REQ_TIMEOUT) { + DEBUG("req delete <%d.%d.%d.%d> <%lu:%lu>", + (req->dst_ip & 0x000000ff), + (req->dst_ip & 0x0000ff00) >> 8, + (req->dst_ip & 0x00ff0000) >> 16, + (req->dst_ip & 0xff000000) >> 24, + jiffies, pend->start); - req_end(pend, -ETIMEDOUT, NULL); + req_end(pend, -ETIMEDOUT, NULL); + } + break; + case IBAT_REQ_PATHREC: + preq = container_of(pend, struct path_req, pend); + + DEBUG("examining path req %p pend %p", preq, pend); + if (jiffies > pend->start + IB_AT_REQ_TIMEOUT) { + DEBUG("req delete path <%lu:%lu>", + jiffies, pend->start); + + req_end(pend, -ETIMEDOUT, NULL); + } + break; + default: + WARN("unknown async req type %d", pend->type); } } @@ -651,7 +683,7 @@ if (req->pend.type == IBAT_REQ_ATS) { WARN("ATS - not yet"); - return 0; + return -1; /* 0 when supported */ } WARN("bad req %p type %d", req, req->pend.type); @@ -666,32 +698,31 @@ .dgid = req->rt.dgid, .sgid = req->rt.sgid, }; - int r; if (req->pend.type != IBAT_REQ_PATHREC) { WARN("bad req %p type %d", req, req->pend.type); return -1; } - r = ib_sa_path_rec_get(req->rt.out_dev, - req->rt.out_port, - &rec, - (IB_SA_PATH_REC_DGID | - IB_SA_PATH_REC_SGID | - IB_SA_PATH_REC_PKEY | - IB_SA_PATH_REC_NUMB_PATH), - req->pend.timeout_ms, - GFP_KERNEL, - path_req_complete, - req, - &req->pend.sa_query); + req->pend.sa_id = ib_sa_path_rec_get(req->rt.out_dev, + req->rt.out_port, + &rec, + (IB_SA_PATH_REC_DGID | + IB_SA_PATH_REC_SGID | + IB_SA_PATH_REC_PKEY | + IB_SA_PATH_REC_NUMB_PATH), + req->pend.timeout_ms, + GFP_KERNEL, + path_req_complete, + req, + &req->pend.sa_query); - if (r < 0) - return r; + if (req->pend.sa_id < 0) { + WARN("ib_sa_path_rec_get %d", req->pend.sa_id); + return req->pend.sa_id; + } req->pend.timeout_ms <<= 1; /* exponential backoff */ - req->pend.sa_id = r; - return 0; } @@ -716,10 +747,12 @@ spin_lock_irqsave(&q->lock, flags); for (a = q->next; a != q; a = a->next) { + DEBUG("a %p", a); if (a->type != IBAT_REQ_ARP) continue; req = container_of(a, struct route_req, pend); + DEBUG("req %p", req); if (arp->op == __constant_htons(ARPOP_REPLY)) { if (arp->dst_ip == req->dst_ip) @@ -751,7 +784,6 @@ * queue IB arp packet onto work queue. */ DEBUG("recv IB ARP - queue work"); - work = kmalloc(sizeof(*work), GFP_ATOMIC); if (!work) goto done; @@ -763,7 +795,6 @@ done: kfree_skb(skb); - return 0; } @@ -796,17 +827,20 @@ r = resolve_ip(&rreq->src, dst_ip, src_ip, tos, &rreq->dgid); if (r < 0) { + DEBUG("resolve_ip r < 0 free req %p", rreq); kmem_cache_free(route_req_cache, rreq); return r; } if (r > 0) { route_req_output(rreq, ib_route); + DEBUG("resolve_ip r > 0 free req %p", rreq); kmem_cache_free(route_req_cache, rreq); return 1; } if (!async_comp) { + DEBUG("!async_comp free req %p", rreq); kmem_cache_free(route_req_cache, rreq); return -EWOULDBLOCK; } @@ -855,6 +889,7 @@ */ if (!async_comp) { + DEBUG("!async_comp free req %p", preq); kmem_cache_free(path_req_cache, preq); return -EWOULDBLOCK; } @@ -871,7 +906,7 @@ } EXPORT_SYMBOL(ib_at_paths_by_route); -int ib_at_ips_by_gid(union ib_gid gid, u32 *dst_ips, int nips, +int ib_at_ips_by_gid(union ib_gid *gid, u32 *dst_ips, int nips, struct ib_at_completion *async_comp) { return -1; /* FIXME: not implemented yet */ @@ -910,7 +945,7 @@ a->next->prev = child; a->next = child; - a->waiting = NULL; /* clear to avoid cancelling childs */ + a->waiting = NULL; /* clear to avoid cancelling children */ } req_end(a, -EINTR, NULL); @@ -950,7 +985,16 @@ DEBUG("IB AT services init"); - route_req_cache = kmem_cache_create("ib_at_route_reqs", + /* + * init pending lists' dummies. + */ + pending_reqs.next = pending_reqs.prev = &pending_reqs; + spin_lock_init(&pending_reqs.lock); + + /* + * Init memory pools + */ + route_req_cache = kmem_cache_create("ib_at_routes", sizeof(struct route_req), 0, SLAB_HWCACHE_ALIGN, NULL, NULL); @@ -960,7 +1004,7 @@ goto err_route; } - path_req_cache = kmem_cache_create("ib_at_path_reqs", + path_req_cache = kmem_cache_create("ib_at_paths", sizeof(struct path_req), 0, SLAB_HWCACHE_ALIGN, NULL, NULL); @@ -970,6 +1014,9 @@ goto err_path; } + /* + * Init ib at worker thread and queue + */ ib_at_wq = create_workqueue("ib_at_wq"); if (!ib_at_wq) { WARN("Failed to allocate IB AT wait queue."); @@ -979,6 +1026,7 @@ INIT_WORK(&ib_at_timer, ib_at_sweep, NULL); queue_delayed_work(ib_at_wq, &ib_at_timer, IB_AT_SWEEP_INTERVAL); + /* * install device for receiving ARP packets in parallel to the normal * Linux ARP, this will be the SDP notifier that an ARP request has From jcarr at linuxmachines.com Mon May 16 15:26:57 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Mon, 16 May 2005 15:26:57 -0700 Subject: [openib-general] How about ib_send_page() ? Message-ID: <42891E31.9000908@linuxmachines.com> It seems to me it would be useful to have a simple ib_send_page() function. This is essentially what I'm going to end up writing for what I need IB to do. If there is anyone else that has similar needs or interests I'd be happy to work with them. The CM works well enough to allow me to initiate connections between HCA's. I'm now going to spend some time trying to write something to kmalloc pages and test the different ways to initiate IB communication between the hosts. Hopefully I can get something smart enough to perform better than IPoIB does. (side note: it would seem IPoIB could be re-written to dramatically improve it's performance). So, in the end there would be things like: ibnode = look_for_node(); // init_node_connection(ibnode); // sets up PD,WR,QP,CQ's etc. ib_send_page( ibnode, page ); Enjoy, Jeff From iod00d at hp.com Mon May 16 15:31:44 2005 From: iod00d at hp.com (Grant Grundler) Date: Mon, 16 May 2005 15:31:44 -0700 Subject: [openib-general] tvflash utility In-Reply-To: <52zmuvm74i.fsf@topspin.com> References: <52zmuvm74i.fsf@topspin.com> Message-ID: <20050516223144.GA30200@esmail.cup.hp.com> On Mon, May 16, 2005 at 10:07:57AM -0700, Roland Dreier wrote: > Shirley> It doesn't compile. pci_alloc is not in -lpci. Any > Shirley> library I forgot to install? The README file is not > Shirley> clear. > > -lpci should include pci_alloc() -- what version of pciutils do you > have installed? One possibility is that you need to install the > pciutils-devel package as well as the pciutils package. tvflash builds fine for me on ia64 (Debian). I have (debian) pciutils-dev installed. I don't think there are any other unusual dependencies though. grant grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/tvflash$ make clean test -z "src/tvflash" || rm -f src/tvflash rm -f *.o grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/tvflash$ make make all-am make[1]: Entering directory `/home/src/openib_gen2/src/userspace/tvflash' if gcc -DHAVE_CONFIG_H -I. -I. -I. -Wall -g -O2 -MT src_tvflash-tvflash.o -MD -MP -MF ".deps/src_tvflash-tvflash.Tpo" -c -o src_tvflash-tvflash.o `test -f 'src/tvflash.c' || echo './'`src/tvflash.c; \ then mv -f ".deps/src_tvflash-tvflash.Tpo" ".deps/src_tvflash-tvflash.Po"; else rm -f ".deps/src_tvflash-tvflash.Tpo"; exit 1; fi gcc -g -O2 -o src/tvflash src_tvflash-tvflash.o -lpci make[1]: Leaving directory `/home/src/openib_gen2/src/userspace/tvflash' grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/tvflash$ dpkg -l pci\* Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Installed/Config-files/Unpacked/Failed-config/Half-installed |/ Err?=(none)/Hold/Reinst-required/X=both-problems (Status,Err: uppercase=bad) ||/ Name Version Description +++-==============-==============-============================================ ii pciutils 2.1.11-15 Linux PCI Utilities ii pciutils-dev 2.1.11-15 Linux PCI Utilities (development files) grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/tvflash$ From xma at us.ibm.com Mon May 16 15:30:49 2005 From: xma at us.ibm.com (Shirley Ma) Date: Mon, 16 May 2005 15:30:49 -0700 Subject: [openib-general] autom4te.cache in libmthca seems wrong? Message-ID: The output of autogen.sh in libmthca is as below: --------------------------------------------------------- + aclocal -I config Can't locate object method "path" via package "Request" at /usr/share/autoconf/Autom4te/C4che.pm line 69, line 112. aclocal: autom4te failed with exit status: 1 + libtoolize --force --copy Putting files in AC_CONFIG_AUX_DIR, `config'. + autoheader Can't locate object method "path" via package "Request" at /usr/share/autoconf/Autom4te/C4che.pm line 69, line 112. autoheader: /usr/bin/autom4te failed with exit status: 1 + automake --foreign --add-missing --copy Can't locate object method "path" via package "Request" at /usr/share/autoconf/Autom4te/C4che.pm line 69, line 112. automake: autoconf failed with exit status: 1 + autoconf Can't locate object method "path" via package "Request" at /usr/share/autoconf/Autom4te/C4che.pm line 69, line 112. 0 The autoconf version I used was autoconf-2.59-75.1. After I replace with libibverbs/autom4te.cache, the autogen.sh works. Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From libor at topspin.com Mon May 16 15:33:16 2005 From: libor at topspin.com (Libor Michalek) Date: Mon, 16 May 2005 15:33:16 -0700 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <42891E31.9000908@linuxmachines.com>; from jcarr@linuxmachines.com on Mon, May 16, 2005 at 03:26:57PM -0700 References: <42891E31.9000908@linuxmachines.com> Message-ID: <20050516153316.A32217@topspin.com> On Mon, May 16, 2005 at 03:26:57PM -0700, Jeff Carr wrote: > It seems to me it would be useful to have a simple ib_send_page() function. > > This is essentially what I'm going to end up writing for what I need IB > to do. If there is anyone else that has similar needs or interests I'd > be happy to work with them. > > The CM works well enough to allow me to initiate connections between > HCA's. I'm now going to spend some time trying to write something to > kmalloc pages and test the different ways to initiate IB communication > between the hosts. Hopefully I can get something smart enough to perform > better than IPoIB does. FYI, this is what SDP in regular sockets mode is doing. The internal buffers are allocated using __get_free_page and then userspace data is copied to/from the buffer which are transmitted/received by posting send/receive on the allocated pages. -Libor From xma at us.ibm.com Mon May 16 15:35:08 2005 From: xma at us.ibm.com (Shirley Ma) Date: Mon, 16 May 2005 15:35:08 -0700 Subject: [openib-general] tvflash utility In-Reply-To: <20050516223144.GA30200@esmail.cup.hp.com> Message-ID: Hi, Grant, > I have (debian) pciutils-dev installed. I did a fully installation, still missed this package. I need to install this package. JS20 daughtercard has been successfully loaded with 3.3.2 cougar firmware (fw-23108-a1-rel.mlx + MTPB23108_128MB.brd). Thanks Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 Grant Grundler 05/16/2005 03:31 PM To Roland Dreier cc Shirley Ma/Beaverton/IBM at IBMUS, openib-general at openib.org Subject Re: [openib-general] tvflash utility On Mon, May 16, 2005 at 10:07:57AM -0700, Roland Dreier wrote: > Shirley> It doesn't compile. pci_alloc is not in -lpci. Any > Shirley> library I forgot to install? The README file is not > Shirley> clear. > > -lpci should include pci_alloc() -- what version of pciutils do you > have installed? One possibility is that you need to install the > pciutils-devel package as well as the pciutils package. tvflash builds fine for me on ia64 (Debian). I have (debian) pciutils-dev installed. I don't think there are any other unusual dependencies though. grant grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/tvflash$ make clean test -z "src/tvflash" || rm -f src/tvflash rm -f *.o grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/tvflash$ make make all-am make[1]: Entering directory `/home/src/openib_gen2/src/userspace/tvflash' if gcc -DHAVE_CONFIG_H -I. -I. -I. -Wall -g -O2 -MT src_tvflash-tvflash.o -MD -MP -MF ".deps/src_tvflash-tvflash.Tpo" -c -o src_tvflash-tvflash.o `test -f 'src/tvflash.c' || echo './'`src/tvflash.c; \ then mv -f ".deps/src_tvflash-tvflash.Tpo" ".deps/src_tvflash-tvflash.Po"; else rm -f ".deps/src_tvflash-tvflash.Tpo"; exit 1; fi gcc -g -O2 -o src/tvflash src_tvflash-tvflash.o -lpci make[1]: Leaving directory `/home/src/openib_gen2/src/userspace/tvflash' grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/tvflash$ dpkg -l pci\* Desired=Unknown/Install/Remove/Purge/Hold | Status=Not/Installed/Config-files/Unpacked/Failed-config/Half-installed |/ Err?=(none)/Hold/Reinst-required/X=both-problems (Status,Err: uppercase=bad) ||/ Name Version Description +++-==============-==============-============================================ ii pciutils 2.1.11-15 Linux PCI Utilities ii pciutils-dev 2.1.11-15 Linux PCI Utilities (development files) grundler at gsyprf3:/usr/src/openib_gen2/src/userspace/tvflash$ -------------- next part -------------- An HTML attachment was scrubbed... URL: From roland at topspin.com Mon May 16 15:47:04 2005 From: roland at topspin.com (Roland Dreier) Date: Mon, 16 May 2005 15:47:04 -0700 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <42891E31.9000908@linuxmachines.com> (Jeff Carr's message of "Mon, 16 May 2005 15:26:57 -0700") References: <42891E31.9000908@linuxmachines.com> Message-ID: <52u0l2lrfb.fsf@topspin.com> Jeff> (side note: it would seem IPoIB could be re-written to Jeff> dramatically improve it's performance). Out of curiousity, what would the rewrite change to obtain better performance? Thanks, Roland From roland at topspin.com Mon May 16 15:51:08 2005 From: roland at topspin.com (Roland Dreier) Date: Mon, 16 May 2005 15:51:08 -0700 Subject: [openib-general] autom4te.cache in libmthca seems wrong? In-Reply-To: (Shirley Ma's message of "Mon, 16 May 2005 15:30:49 -0700") References: Message-ID: <52mzqulr8j.fsf@topspin.com> autom4te.cache is not checked into subversion, so I'm not sure where your bad copy of autom4te.cache came from: $ svn ls https://openib.org/svn/gen2/trunk/src/userspace/libmthca|grep autom4te|wc 0 0 0 $ svn ls https://openib.org/svn/gen2/trunk/src/userspace/libmthca AUTHORS COPYING Makefile.am autogen.sh config/ configure.in debian/ libmthca.spec.in src/ - R. From jcarr at linuxmachines.com Mon May 16 15:54:18 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Mon, 16 May 2005 15:54:18 -0700 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <20050516153316.A32217@topspin.com> References: <42891E31.9000908@linuxmachines.com> <20050516153316.A32217@topspin.com> Message-ID: <4289249A.2070609@linuxmachines.com> Libor Michalek wrote: > On Mon, May 16, 2005 at 03:26:57PM -0700, Jeff Carr wrote: > >>It seems to me it would be useful to have a simple ib_send_page() function. >> >>This is essentially what I'm going to end up writing for what I need IB >>to do. If there is anyone else that has similar needs or interests I'd >>be happy to work with them. >> >>The CM works well enough to allow me to initiate connections between >>HCA's. I'm now going to spend some time trying to write something to >>kmalloc pages and test the different ways to initiate IB communication >>between the hosts. Hopefully I can get something smart enough to perform >>better than IPoIB does. > > > FYI, this is what SDP in regular sockets mode is doing. The internal > buffers are allocated using __get_free_page and then userspace data > is copied to/from the buffer which are transmitted/received by posting > send/receive on the allocated pages. OK. Well I would rather make something generic. Besides, wasn't there some MS patent issue? The last thread on the subject that I read kinda made it sound like you were going to look into the issue and respond. Maybe I missed the response; there's a lot of mail in the archives for this list. In any case, I'd rather just make something more generic that perhaps can be used with NBD. Jeff From ardavis at ichips.intel.com Mon May 16 17:07:44 2005 From: ardavis at ichips.intel.com (Arlin Davis) Date: Mon, 16 May 2005 17:07:44 -0700 Subject: [openib-general] ibv_reg_mr limits? Message-ID: <428935D0.1010508@ichips.intel.com> Is there a memory registration limit with ibv_reg_mr? My pingpong test returns error when trying to register 100 MB of memory on my server with 1GB of memory. -arlin From iod00d at hp.com Mon May 16 17:26:04 2005 From: iod00d at hp.com (Grant Grundler) Date: Mon, 16 May 2005 17:26:04 -0700 Subject: [openib-general] autom4te.cache in libmthca seems wrong? In-Reply-To: <52mzqulr8j.fsf@topspin.com> References: <52mzqulr8j.fsf@topspin.com> Message-ID: <20050517002604.GD30200@esmail.cup.hp.com> On Mon, May 16, 2005 at 03:51:08PM -0700, Roland Dreier wrote: > autom4te.cache is not checked into subversion, so I'm not sure where > your bad copy of autom4te.cache came from: It's generated by autogen.sh. Delete the autom4te.cache and re-run autogen.sh. I have autoconf v2.59a-3 (debian "testing") installed and that works fine. grant From iod00d at hp.com Mon May 16 17:43:16 2005 From: iod00d at hp.com (Grant Grundler) Date: Mon, 16 May 2005 17:43:16 -0700 Subject: [openib-general] ibv_reg_mr limits? In-Reply-To: <428935D0.1010508@ichips.intel.com> References: <428935D0.1010508@ichips.intel.com> Message-ID: <20050517004316.GF30200@esmail.cup.hp.com> On Mon, May 16, 2005 at 05:07:44PM -0700, Arlin Davis wrote: > Is there a memory registration limit with ibv_reg_mr? > > My pingpong test returns error when trying to register 100 MB of memory > on my server with 1GB of memory. what does "ulimit -a" say? grant From jcarr at linuxmachines.com Mon May 16 18:00:49 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Mon, 16 May 2005 18:00:49 -0700 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <52u0l2lrfb.fsf@topspin.com> References: <42891E31.9000908@linuxmachines.com> <52u0l2lrfb.fsf@topspin.com> Message-ID: <42894241.1010908@linuxmachines.com> Roland Dreier wrote: > Jeff> (side note: it would seem IPoIB could be re-written to > Jeff> dramatically improve it's performance). > > Out of curiousity, what would the rewrite change to obtain better > performance? I'm just speculating that it could be rewritten to improve performance. There were many complaints in the past about the speed of IPoIB; I thought that was still the case. Maybe the performance has increased and I have my systems misconfigured. In any case between my two hosts with netperf: root at delta:~# netperf -H foxtrot -- -s 105472 -r 105472 TCP STREAM TEST to foxtrot Recv Send Send Socket Socket Message Elapsed Size Size Size Time Throughput bytes bytes bytes secs. 10^6bits/sec 87380 210944 210944 10.00 2335.21 with vmstat on the server showing: -----memory---------- ---swap-- -----io---- --system-- ----cpu---- swpd free buff cache si so bi bo in cs us sy id wa 0 1115720 440988 49836 0 0 0 0 43943 8251 0 94 6 0 0 1115720 440988 49836 0 0 0 0 43725 8384 0 94 6 0 So the CPU is throttled at around 100-200k packets/sec with 43k interrupts / sec. Perhaps the HCA can be configured to respond/trigger events/generate interrupts less often under higher load. Some ethernet chips have this kind of functionality (interrupt after x packets or after n time). Jeff From iod00d at hp.com Mon May 16 18:47:28 2005 From: iod00d at hp.com (Grant Grundler) Date: Mon, 16 May 2005 18:47:28 -0700 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <42894241.1010908@linuxmachines.com> References: <42891E31.9000908@linuxmachines.com> <52u0l2lrfb.fsf@topspin.com> <42894241.1010908@linuxmachines.com> Message-ID: <20050517014728.GI30200@esmail.cup.hp.com> On Mon, May 16, 2005 at 06:00:49PM -0700, Jeff Carr wrote: > with vmstat on the server showing: > > -----memory---------- ---swap-- -----io---- --system-- ----cpu---- > swpd free buff cache si so bi bo in cs us sy id wa > 0 1115720 440988 49836 0 0 0 0 43943 8251 0 94 6 0 > 0 1115720 440988 49836 0 0 0 0 43725 8384 0 94 6 0 vmstat doesn't tell you where the time is being spent. Get a profile or try out the beta Pentium M or AMD64 perfmon support that was recently posted: http://www.hpl.hp.com/hosted/linux/mail-archives/perfmon/2005-May/000389.html perfmon homepage doesn't reflect the beta version but gives a good overview: http://www.hpl.hp.com/research/linux/perfmon/ You also want to explore "netperf -C" option that' available with netperf 2.4.0-rc1 (See www.netperf.orf). I've posted results here before about binding netperf/netserver processes to different CPUs. > So the CPU is throttled at around 100-200k packets/sec with 43k > interrupts / sec. Perhaps the HCA can be configured to respond/trigger > events/generate interrupts less often under higher load. Some ethernet > chips have this kind of functionality (interrupt after x packets or > after n time). NAPI obsoletes interrupt coalescing to a large degree on regular NICs. But IPoIB can't really implement NAPI since it's sending work to a shared HCA. And any form of interrupt coalescing would interfere with any latency sensitive work as well (if present). grant From jcarr at linuxmachines.com Mon May 16 18:55:38 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Mon, 16 May 2005 18:55:38 -0700 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <52u0l2lrfb.fsf@topspin.com> References: <42891E31.9000908@linuxmachines.com> <52u0l2lrfb.fsf@topspin.com> Message-ID: <42894F1A.3060706@linuxmachines.com> Roland Dreier wrote: > Jeff> (side note: it would seem IPoIB could be re-written to > Jeff> dramatically improve it's performance). > > Out of curiousity, what would the rewrite change to obtain better > performance? Could (or would it help if) the MTU was increased to something much larger than 2044? Kernel NFS transfer of a cached 1GB file (some, but not lots, of overhead for NFS itself) hitting 1.2Gbits/sec: # dd if=1GB_file of=/dev/null bs=1M 1000+0 records in 1000+0 records out 1048576000 bytes transferred in 6.661947 seconds (157397829 bytes/sec) Server 30% idle, client 0%. client ifconfig start / end: RX packets:1248583 errors:0 dropped:0 overruns:0 frame:0 TX packets:4083901 errors:0 dropped:0 overruns:0 carrier:0 RX packets:1792589 errors:0 dropped:0 overruns:0 frame:0 TX packets:4115907 errors:0 dropped:0 overruns:0 carrier:0 server ifconfig start / end; RX packets:4083900 errors:0 dropped:0 overruns:0 frame:0 TX packets:1248584 errors:0 dropped:0 overruns:0 carrier:0 RX packets:4115906 errors:0 dropped:0 overruns:0 frame:0 TX packets:1792590 errors:0 dropped:0 overruns:0 carrier:0 So, about 500k server TX packets (2K/packet) & client 32k TX packets. This transfer generates about 100k interrupts. So, this test does almost as many packets/sec as netperf & half the interrupts at about half the speed of netperf. So, perhaps a bigger MTU and/or bigger chunking of transmits per interrupt would improve the performance. Then again, a raw dd of a memory cached 1GB file only hits ~1GB/sec. The kernel takes quite a lot of processing time for this. Still, one might hope smartly configured NFS or NBD could hit 500MB/sec between machines when going over IB purely between RAM. Jeff From jcarr at linuxmachines.com Mon May 16 19:20:07 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Mon, 16 May 2005 19:20:07 -0700 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <20050517014728.GI30200@esmail.cup.hp.com> References: <42891E31.9000908@linuxmachines.com> <52u0l2lrfb.fsf@topspin.com> <42894241.1010908@linuxmachines.com> <20050517014728.GI30200@esmail.cup.hp.com> Message-ID: <428954D7.3060003@linuxmachines.com> Grant Grundler wrote: > vmstat doesn't tell you where the time is being spent. > > Get a profile or try out the beta Pentium M or AMD64 perfmon support Yes, I would but I didn't think I could because I have Xeon's. > You also want to explore "netperf -C" option that' available with > netperf 2.4.0-rc1 (See www.netperf.orf). I've posted results here > before about binding netperf/netserver processes to different CPUs. I think I remember this thread. I'll look back for it again. It does look like the -C option is in 2.3.4 though: root at delta:~# netperf -C -H foxtrot -- -s 105472 -r 105472 TCP STREAM TEST to foxtrot Recv Send Send Utilization Service Demand Socket Socket Message Elapsed Send Recv Send Recv Size Size Size Time Throughput local remote local remote bytes bytes bytes secs. 10^6bits/s % U % T us/KB us/KB 87380 210944 210944 10.00 2330.71 -1.00 89.49 -1.000 3.146 Still, I'll look back for the thread on the subject. > NAPI obsoletes interrupt coalescing to a large degree on regular NICs. ? > But IPoIB can't really implement NAPI since it's sending work to > a shared HCA. > And any form of interrupt coalescing would interfere with > any latency sensitive work as well (if present). Surely. It would have to be configureable for people (like me) that wanted it. I don't care about latency, just about throughput. What's the point of having a fast network if you can't send *lots* of data over it. :) Jeff From xma at us.ibm.com Mon May 16 19:50:35 2005 From: xma at us.ibm.com (Shirley Ma) Date: Mon, 16 May 2005 19:50:35 -0700 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <428954D7.3060003@linuxmachines.com> Message-ID: >> But IPoIB can't really implement NAPI since it's sending work to >> a shared HCA. >> And any form of interrupt coalescing would interfere with >> any latency sensitive work as well (if present). >Surely. It would have to be configureable for people (like me) that wanted it. Configurable feature is not a bad idea for people who are more interested in IPoIB throughput. As I recalled Roland had an experimental of NAPI, it didn't gain too much performance in the previous stack. Thanks Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From iod00d at hp.com Mon May 16 21:32:50 2005 From: iod00d at hp.com (Grant Grundler) Date: Mon, 16 May 2005 21:32:50 -0700 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <428954D7.3060003@linuxmachines.com> References: <42891E31.9000908@linuxmachines.com> <52u0l2lrfb.fsf@topspin.com> <42894241.1010908@linuxmachines.com> <20050517014728.GI30200@esmail.cup.hp.com> <428954D7.3060003@linuxmachines.com> Message-ID: <20050517043250.GK30200@esmail.cup.hp.com> On Mon, May 16, 2005 at 07:20:07PM -0700, Jeff Carr wrote: > >You also want to explore "netperf -C" option that' available with > >netperf 2.4.0-rc1 (See www.netperf.orf). I've posted results here > >before about binding netperf/netserver processes to different CPUs. > > I think I remember this thread. I'll look back for it again. > > It does look like the -C option is in 2.3.4 though: Sorry - I meant "-T". And that seems to have been added to 2.3pl2 as well. > >NAPI obsoletes interrupt coalescing to a large degree on regular NICs. > > ? 42. :^) If it's NAPI that means nothing, here's probably the best summary: http://lwn.net/Articles/30098/ > > >But IPoIB can't really implement NAPI since it's sending work to > >a shared HCA. > > > And any form of interrupt coalescing would interfere with > >any latency sensitive work as well (if present). > > Surely. It would have to be configureable for people (like me) that > wanted it. I don't care about latency, just about throughput. NAPI, TOE, and lots of other things help ethernet with throughput. None of those help with latency and one has to spend alot of time tuning to get things right, much less as a tradeoff. I'm blissfully ignorant of any tuning that goes on for openib.org. > What's the > point of having a fast network if you can't send *lots* of data over it. :) One can. Using SDP, netperf TCP_STREAM measured 650 MB/s using the regular PCI-X card. I wonder if that could be faster with bigger socket buffers but don't know (IIRC, IB socket buffers are fixed at 256KB). Anyway, that's with about 60% CPU (1.5 Ghz ia64). I haven't checked to see what libibverbs can do with "native" mode (if one is willing/able to modify the application.) ISTR Michael Tsirkin had another streaming test in mind to measure peak bandwidth. But I'm not quite done with rdma_lat yet anyway... grant From mst at mellanox.co.il Mon May 16 22:51:49 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Tue, 17 May 2005 08:51:49 +0300 Subject: [openib-general] Re: ibv_reg_mr limits? In-Reply-To: <20050517004316.GF30200@esmail.cup.hp.com> References: <428935D0.1010508@ichips.intel.com> <20050517004316.GF30200@esmail.cup.hp.com> Message-ID: <20050517055149.GB14268@mellanox.co.il> Quoting r. Grant Grundler : > Subject: Re: ibv_reg_mr limits? > > On Mon, May 16, 2005 at 05:07:44PM -0700, Arlin Davis wrote: > > Is there a memory registration limit with ibv_reg_mr? > > > > My pingpong test returns error when trying to register 100 MB of memory > > on my server with 1GB of memory. > > what does "ulimit -a" say? > > grant It doesnt seem like we do a hidden mlock anymore, so no ulimit shall be involved. -- MST - Michael S. Tsirkin From mst at mellanox.co.il Mon May 16 23:15:01 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Tue, 17 May 2005 09:15:01 +0300 Subject: [openib-general] Re: tvflash utility In-Reply-To: References: Message-ID: <20050517061501.GC14268@mellanox.co.il> Quoting r. Shirley Ma : > Subject: tvflash utility > > > I've sucessfully upgrade my 4X Topspin HCA with Mellanox firmware 3.3.2. But > this firmware doesn't work on JS20 daughter card. > Does anybody use tvflash? It doesn't work on my PPC64 and ia64. I want to use > this utility to read HCA on board binary image to repair the daughter card. Any > help? > > Thanks > Shirley Ma > IBM Linux Technology Center > 15300 SW Koll Parkway > Beaverton, OR 97006-6063 > Phone(Fax): (503) 578-7638 > If you already have the firmware binary, you can use mstflint. It works fine on topspin cards. -- MST - Michael S. Tsirkin From mst at mellanox.co.il Mon May 16 23:15:41 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Tue, 17 May 2005 09:15:41 +0300 Subject: [openib-general] Re: How about ib_send_page() ? In-Reply-To: <52u0l2lrfb.fsf@topspin.com> References: <42891E31.9000908@linuxmachines.com> <52u0l2lrfb.fsf@topspin.com> Message-ID: <20050517061541.GD14268@mellanox.co.il> Quoting r. Roland Dreier : > Subject: Re: How about ib_send_page() ? > > Jeff> (side note: it would seem IPoIB could be re-written to > Jeff> dramatically improve it's performance). > > Out of curiousity, what would the rewrite change to obtain better > performance? > > Thanks, > Roland I have some ideas worth investigating: 1. I think an option to enable zero copy would help for the send side. This means we have to set NETIF_F_NO_CSUM so it likely cant be done with current hardware without breaking IPoverIB spec compliance, but we dont need a spec just to write code. 2. I dont think NAPI was ever tried on gen2 code. Worth a try? Might help more in combination with (1). 3. There's a draft for connected mode. That would increase the MTU and so help the receive side as well. -- MST - Michael S. Tsirkin From halr at voltaire.com Tue May 17 04:10:35 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 17 May 2005 07:10:35 -0400 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <42894F1A.3060706@linuxmachines.com> References: <42891E31.9000908@linuxmachines.com> <52u0l2lrfb.fsf@topspin.com> <42894F1A.3060706@linuxmachines.com> Message-ID: <1116328235.4480.1006.camel@localhost.localdomain> On Mon, 2005-05-16 at 21:55, Jeff Carr wrote: > Could (or would it help if) the MTU was increased to something much > larger than 2044? As the packet rate is one limiting factor, this has been discussed before. With UD, the IB limit is 4K (-4 for CRC) but the device limit is 2K (-4), which is the default MTU for IPoIB. hence 2044 for normal IPoIB. This is one of the main reasons for IPoIB connected mode, which is not currently supported by OpenIB IPoIB. Given that there are now 2 ULPs which use the CM, this wouldn't be hard to support. (It's more a matter of someone having some time to do this). Note that IPoIB connected mode is currently an internet draft. -- Hal From eitan at mellanox.co.il Tue May 17 07:53:43 2005 From: eitan at mellanox.co.il (Eitan Zahavi) Date: Tue, 17 May 2005 17:53:43 +0300 Subject: [openib-general] RE: OpenSM Routing Scalability Proposal Message-ID: <506C3D7B14CDD411A52C00025558DED607C3017C@mtlex01.yok.mtl.com> Hi All, This is an updated proposal document for your comments. The main change is in describing the need for preserving enough data to enable incremental routing algorithm. So the actual proposal is to implement the algorithm described in section 4.3. EZ <> Eitan Zahavi Design Technology Director Mellanox Technologies LTD Tel:+972-4-9097208 Fax:+972-4-9593245 P.O. Box 586 Yokneam 20692 ISRAEL -----Original Message----- From: Eitan Zahavi Sent: Thursday, May 05, 2005 6:46 PM To: 'openib-general at openib.org' Subject: OpenSM Routing Scalability Proposal Hi All, Here are some of my thoughts/proposals regarding faster and low memory profile OpenSM routing solution. Any feedback is welcome. I am not comfortable with the complexity order numbers and will highly appreciate your double check. The "persistency" area is also in its initial stages. << File: OsmRoutingScalability.pdf >> Thanks Eitan Zahavi Design Technology Director Mellanox Technologies LTD Tel:+972-4-9097208 Fax:+972-4-9593245 P.O. Box 586 Yokneam 20692 ISRAEL -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: OpenSM Routing.pdf Type: application/octet-stream Size: 64577 bytes Desc: not available URL: From iod00d at hp.com Tue May 17 07:59:39 2005 From: iod00d at hp.com (Grant Grundler) Date: Tue, 17 May 2005 07:59:39 -0700 Subject: [openib-general] perftest - multiple patches merged In-Reply-To: <20050516140500.GI13718@mellanox.co.il> References: <20050516140500.GI13718@mellanox.co.il> Message-ID: <20050517145939.GA1078@esmail.cup.hp.com> On Mon, May 16, 2005 at 05:05:00PM +0300, Michael S. Tsirkin wrote: > Hi! > With 2353 I think I've merged all patches that I have outstanding. > I plan a small cleanup of program output, hope to do it tomorrow. Yup! I went over it yesterday and most of the bits are in. thanks, grant From halr at voltaire.com Tue May 17 07:56:38 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 17 May 2005 10:56:38 -0400 Subject: [openib-general] User Level Events - request for support In-Reply-To: <506C3D7B14CDD411A52C00025558DED6047EEFA3@mtlex01.yok.mtl.com> References: <506C3D7B14CDD411A52C00025558DED6047EEFA3@mtlex01.yok.mtl.com> Message-ID: <1116341798.4478.6.camel@localhost.localdomain> On Thu, 2005-03-17 at 04:12, Eitan Zahavi wrote: > I would like to propose for gen2 stack to have a user level API > supporting registration for notifications on unaffiliated asynchronous > events. > > As an example in cases when the local IB port link goes down an SM > that runs on top of this port needs to be notified and start a sweep > when the port is back up. Missing such event, in user land, prevents > the SM from knowing about the change. > > Currently gen1 and gen2 OpenSM is not registered to get these events. > The SM will then fail to reconfigure the subnet in cases like: > > 1. The SM cable connects to a switch and the user changes the > switch port the SM is connected to. In this case the SM might be in > the middle of a sweep and do not even notice (due to the short time > the change takes) that there was a change. Traps coming from the > switch are forwarded to the old port the SM was connected to and are > dropped (as the port is down). As OpenSM does not know there was any > change in the subnet topology, it will not perform a heavy sweep. > > 2. The switch connected to the SM is rebooted. If the reboot > happens so fast that it falls between two light sweeps, OpenSM will > not be able to know the switch was reset (as all SMPs are DR). > > Although one can write special case code to handle these cases, due to > the asynchronous nature of things there are many races that can not be > resolved without a simple "port up/down" event that the SM should > register to. > > Hope this provides enough reason behind my request for a user-level > event notification mechanism. The OpenSM vendor layer should be enhanced with an additional API for a local port state changed event (and take a flag for port down and port up). OpenSM could then take this event and handle it generically. This could be implemented for gen2 (OpenIB) and gen1 if the events can be generated from VAPI or whatever. -- Hal From roland at topspin.com Tue May 17 08:07:32 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 17 May 2005 08:07:32 -0700 Subject: [openib-general] ibv_reg_mr limits? References: <428935D0.1010508@ichips.intel.com> Message-ID: <5264xhlwln.fsf@topspin.com> Arlin> Is there a memory registration limit with ibv_reg_mr? My Arlin> pingpong test returns error when trying to register 100 MB Arlin> of memory on my server with 1GB of memory. mthca (in the kernel) currently implements memory registration in a fairly stupid way that requires it to allocate a contiguous chunk of memory to hold the entire page table of the memory region being registered. This can be fixed but right now I would expect the limit on a single registration to be ~64 MB (or less if memory is fragmented). - R. From eitan at mellanox.co.il Tue May 17 08:20:31 2005 From: eitan at mellanox.co.il (Eitan Zahavi) Date: Tue, 17 May 2005 18:20:31 +0300 Subject: [openib-general] User Level Events - request for support Message-ID: <506C3D7B14CDD411A52C00025558DED607C3017E@mtlex01.yok.mtl.com> > Hal Wrote: > The OpenSM vendor layer should be enhanced with an additional API for a > local port state changed event (and take a flag for port down and port > up). > > OpenSM could then take this event and handle it generically. This could > be implemented for gen2 (OpenIB) and gen1 if the events can be generated > from VAPI or whatever. > Yes. I agree. We should have osm_vendor_api.h define some callback registration function for the local port up/down event. I will implement the OpenSM code for firing a sweep one the port is up. EZ Eitan Zahavi Design Technology Director Mellanox Technologies LTD Tel:+972-4-9097208 Fax:+972-4-9593245 P.O. Box 586 Yokneam 20692 ISRAEL -------------- next part -------------- An HTML attachment was scrubbed... URL: From iod00d at hp.com Tue May 17 08:51:50 2005 From: iod00d at hp.com (Grant Grundler) Date: Tue, 17 May 2005 08:51:50 -0700 Subject: [openib-general] perftest-06 patch - output routines Message-ID: <20050517155150.GB1078@esmail.cup.hp.com> Michael, The basic measurement is in integer (cycles_t) units. I don't like to convert to floating point until we have to. The following patch makes delta[] an array of cycles_t and doesn't convert to floating point until the values are printed as microseconds. This patch completely restructures the output: o replace report-all with report-histogram. If someone really want's "all", use -UH (or add an alias "-A"). Normal use case is someone just wants one or the other. "-H" is soft-of-shorthand for "rdma_lat | sort -k2 -n" o shortened the summary to 1 line of output (min/median/max) o print_mhz() and print_cycles() are seperate...a bit more twiddling and they could be unified again. But it's pretty clear what's going on and merging them back together again makes it less clear. o some additional white space cleanups. thanks, grant Index: perftest/rdma_lat.c =================================================================== --- perftest/rdma_lat.c (revision 2359) +++ perftest/rdma_lat.c (working copy) @@ -1,6 +1,7 @@ /* * Copyright (c) 2005 Topspin Communications. All rights reserved. * Copyright (c) 2005 Mellanox Technologies Ltd. All rights reserved. + * Copyright (c) 2005 Hewlett Packard, Inc (Grant Grundler) * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -57,22 +58,14 @@ #include "get_clock.h" -static int double_compar(const void * aptr, const void * bptr) -{ - const double *a = aptr; - const double *b = bptr; - - if (*a < *b) return -1; - if (*a > *b) return 1; - else return 0; -} - -enum { - PINGPONG_RDMA_WRID = 3, -}; +#define PINGPONG_RDMA_WRID 3 static int page_size; +static int report_unsorted; +static int report_histogram; +static int report_cycles; /* report delta's in cycles, not microsec's */ + struct pingpong_context { struct ibv_context *context; @@ -462,6 +455,107 @@ static void usage(const char *argv0) printf(" -U, --report-unsorted (implies -A) print out unsorted results (default sorted)\n"); } +static inline cycles_t get_median(int iters, cycles_t delta[]) +{ + /* + * When there is an + * odd number of samples, the median is the middle number. + * even number of samples, the median is the mean of the + * two middle numbers. + * + * Reminder: iters is the number of exchanges, not number of samples. + */ + if ((iters - 1) & 1) + return delta[iters / 2]; + else + return (delta[iters / 2] + delta[iters / 2 - 1]) / 2; +} + +static int cycles_cmp(const void * aptr, const void * bptr) +{ + const cycles_t *a = aptr; + const cycles_t *b = bptr; + if (*a < *b) return -1; + if (*a > *b) return 1; + return 0; +} + +#define dump_delta(h, f, d, iters, delta) \ +{ \ + unsigned int i; \ + printf("#, usec\n"); \ + for(i = 0; i < (iters)-1; ++i) \ + printf(f, i+1, delta[i] / (d)); \ + printf("\n\n"); \ +} + +static void print_hz(unsigned int iters, cycles_t *tstamp) +{ + double mhz = 2 * get_cpu_mhz(); + cycles_t median; + cycles_t *delta = malloc(iters * sizeof *delta); + unsigned int i; + + if (!delta) { + perror("malloc"); + exit (0); + } + + for (i = 0; i < iters; ++i) + delta[i] = tstamp[i + 1] - tstamp[i]; + + if (report_unsorted) + dump_delta("#, usec\n", "%d, %f\n", mhz, iters, delta); + + qsort(delta, iters - 1, sizeof *delta, cycles_cmp); + + if (report_histogram) + dump_delta("#, usec\n", "%d, %f\n", mhz, iters, delta); + + median = get_median(iters, delta); + + printf("Latency min/median/max: %f/%f/%f\n", + delta[0]/mhz, median/mhz, delta[iters-2]/mhz); + + free(delta); +} + + +static void print_cycles(unsigned int iters, cycles_t *tstamp) +{ + cycles_t median; + cycles_t *delta = malloc(iters * sizeof *delta); + unsigned int i; + + if (!delta) { + perror("malloc"); + exit (0); + } + + for (i = 0; i < iters ; ++i) + delta[i] = tstamp[i + 1] - tstamp[i]; + + if (report_unsorted) + dump_delta("#, cycles\n", "%d, %lu\n", 2, iters, delta); + + qsort(delta, (iters - 1), sizeof *delta, cycles_cmp); + + /* Definition of histogram: + * http://www.itl.nist.gov/div898/handbook/eda/section3/histogra.htm + */ + if (report_histogram) + dump_delta("#, cycles\n", "%d, %lu\n", 2, iters, delta); + + median = get_median(iters, delta); + + printf("Latency min/median/max: %lu/%lu/%lu cycles\n", + (unsigned long) delta[0]/2, + (unsigned long) median/2, + (unsigned long) delta[iters-2]/2); + + free(delta); +} + int main(int argc, char *argv[]) { struct dlist *dev_list; @@ -477,9 +571,6 @@ int main(int argc, char *argv[]) int rx_depth = 1; int tx_depth = 50; int iters = 1000; - int report_all = 0; - int report_unsorted = 0; - int report_cpu_cycles = 0; int scnt, rcnt, ccnt; int client_first_post; int sockfd; @@ -489,12 +580,6 @@ int main(int argc, char *argv[]) volatile char *post_buf; cycles_t *tstamp; - double median; - double *delta; - int i; - - double mhz; - const char* units; /* Parameter parsing. */ while (1) { @@ -508,12 +593,12 @@ int main(int argc, char *argv[]) { .name = "iters", .has_arg = 1, .val = 'n' }, { .name = "tx-depth", .has_arg = 1, .val = 't' }, { .name = "report-cycles", .has_arg = 0, .val = 'C' }, - { .name = "report-all", .has_arg = 0, .val = 'A' }, + { .name = "report-histogram", .has_arg = 0, .val = 'H' }, { .name = "report-unsorted",.has_arg = 0, .val = 'U' }, { 0 } }; - c = getopt_long(argc, argv, "p:d:i:s:n:t:CAU", long_options, NULL); + c = getopt_long(argc, argv, "p:d:i:s:n:t:CHU", long_options, NULL); if (c == -1) break; @@ -555,17 +640,16 @@ int main(int argc, char *argv[]) break; - case 'A': - report_all = 1; + case 'C': + report_cycles = 1; break; - case 'C': - report_cpu_cycles = 1; + case 'H': + report_histogram = 1; break; case 'U': report_unsorted = 1; - report_all = 1; break; default: @@ -586,14 +670,6 @@ int main(int argc, char *argv[]) srand48(getpid() * time(NULL)); - if (report_cpu_cycles) { - mhz = 1; - units = "usec"; - } else { - mhz = get_cpu_mhz(); - units = "clocks"; - } - page_size = sysconf(_SC_PAGESIZE); dev_list = ibv_get_devices(); @@ -640,15 +716,13 @@ int main(int argc, char *argv[]) if (servername) { sockfd = pp_client_connect(servername, port); - } else { - sockfd = pp_server_connect(port); - } - if (sockfd < 0) - return 1; - - if (servername) { + if (sockfd < 0) + return 1; rem_dest = pp_client_exch_dest(sockfd, &my_dest); } else { + sockfd = pp_server_connect(port); + if (sockfd < 0) + return 1; rem_dest = pp_server_exch_dest(sockfd, &my_dest); } @@ -751,54 +825,11 @@ int main(int argc, char *argv[]) } } - /* Done with the test. Report results. */ - - delta = malloc(iters * sizeof *delta); - if (!delta) { - perror("malloc"); - return 1; - } - - for (i = 0; i < iters - 1; ++i) { - delta[i]=(tstamp[i + 1] - tstamp[i]) / mhz; - } - - if (report_unsorted) { - printf("#, %s\n", units); - for(i = 0; i < iters - 1; ++i) { - printf("%d, %f\n", i, delta[i] / 2); - } - - printf("\n\n"); - } - - qsort(delta, iters - 1, sizeof *delta, double_compar); - - if (report_all && ! report_unsorted) { - printf("#, %s\n", units); - for(i = 0; i < iters - 1; ++i) { - printf("%d, %f\n", i, delta[i] / 2); - } - - printf("\n\n"); - } - - /* When there is an odd number of numbers, the median is simply - * the middle number. - * When there is an even number of numbers, the median is the mean - * of the two middle numbers. - * - * Reminder: iters is the number of exchanges, not number of samples. - */ - - if ((iters - 1) % 2) - median = delta[iters / 2]; - else - median = (delta[iters / 2] + delta[iters / 2 + 1]) / 2; - - printf("Latency minimum: %f %s\n", delta[0] / 2, units); - printf("Latency statistical median: %f %s\n", median / 2, units); - printf("Latency maximum: %f %s\n", delta[iters - 2] / 2, units); + if (report_cycles) + print_cycles(iters, tstamp); + else + print_hz(iters, tstamp); + free(tstamp); return 0; } From blist at aon.at Tue May 17 09:26:38 2005 From: blist at aon.at (Bernhard Fischer) Date: Tue, 17 May 2005 18:26:38 +0200 Subject: [openib-general] ibv_reg_mr limits? In-Reply-To: <5264xhlwln.fsf@topspin.com> References: <428935D0.1010508@ichips.intel.com> <5264xhlwln.fsf@topspin.com> Message-ID: <20050517162638.GA7553@aon.at> On Tue, May 17, 2005 at 08:07:32AM -0700, Roland Dreier wrote: > Arlin> Is there a memory registration limit with ibv_reg_mr? My > Arlin> pingpong test returns error when trying to register 100 MB > Arlin> of memory on my server with 1GB of memory. > >mthca (in the kernel) currently implements memory registration in a >fairly stupid way that requires it to allocate a contiguous chunk of Any timeframe on when this will be changed? I'm right now seeing a bunch of "rank XYZ reg mr failed!" in my test-applet which are my fault, but i wonder if -- after having fixed my test-applet -- i should go and boot with mem=384M and make mthca handle the rest of the mem on its own, as a temporary workaround >memory to hold the entire page table of the memory region being >registered. This can be fixed but right now I would expect the limit >on a single registration to be ~64 MB (or less if memory is fragmented). > > - R. >_______________________________________________ >openib-general mailing list >openib-general at openib.org >http://openib.org/mailman/listinfo/openib-general > >To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From xma at us.ibm.com Tue May 17 09:54:37 2005 From: xma at us.ibm.com (Shirley Ma) Date: Tue, 17 May 2005 09:54:37 -0700 Subject: [openib-general] Gen2 test suites Message-ID: There are some test suites under Gen2. But it's not trivial to set up without understanding the stack well. It would be hard for people(distros/customers) to evaluate this stack. I am thinking if we have a test frame for Gen2 stack, we could set up a nightly build/regression test. The life would be much easy both for developers and and customers. Please point out if my suggestion sounds impossible. Or maybe someone has already been working on this. Thanks Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From xma at us.ibm.com Tue May 17 10:04:42 2005 From: xma at us.ibm.com (Shirley Ma) Date: Tue, 17 May 2005 10:04:42 -0700 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <1116328235.4480.1006.camel@localhost.localdomain> Message-ID: > Note that IPoIB connected mode is currently an internet draft. Since there is an IP issue of SDP, it's better to implement IPoIB connected mode soon. We can ask for when this draft comes to RFC. Thanks Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From roland at topspin.com Tue May 17 10:05:22 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 17 May 2005 10:05:22 -0700 Subject: [openib-general] Gen2 test suites In-Reply-To: (Shirley Ma's message of "Tue, 17 May 2005 09:54:37 -0700") References: Message-ID: <52vf5hkckt.fsf@topspin.com> Shirley> There are some test suites under Gen2. But it's not Shirley> trivial to set up without understanding the stack Shirley> well. It would be hard for people(distros/customers) to Shirley> evaluate this stack. I am thinking if we have a test Shirley> frame for Gen2 stack, we could set up a nightly Shirley> build/regression test. The life would be much easy both Shirley> for developers and and customers. Please point out if my Shirley> suggestion sounds impossible. Or maybe someone has Shirley> already been working on this. I think an automated test suite would be extremely useful, but of course it would be a lot of work to implement and maintain. - R. From roland at topspin.com Tue May 17 10:06:09 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 17 May 2005 10:06:09 -0700 Subject: [openib-general] ibv_reg_mr limits? In-Reply-To: <20050517162638.GA7553@aon.at> (Bernhard Fischer's message of "Tue, 17 May 2005 18:26:38 +0200") References: <428935D0.1010508@ichips.intel.com> <5264xhlwln.fsf@topspin.com> <20050517162638.GA7553@aon.at> Message-ID: <52r7g5kcji.fsf@topspin.com> Bernhard> Any timeframe on when this will be changed? I hope to fix this up soonish but there are a lot of small things like this to clean up. - R. From roland at topspin.com Tue May 17 10:08:58 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 17 May 2005 10:08:58 -0700 Subject: [openib-general] Re: [PATCH 1/2] (fixed) mthca: max_inline_data support In-Reply-To: <20050516191651.GC5743@mellanox.co.il> (Michael S. Tsirkin's message of "Mon, 16 May 2005 22:16:51 +0300") References: <20050509141456.GH9941@mellanox.co.il> <52fyww1h90.fsf@topspin.com> <20050510153543.GG2474@mellanox.co.il> <524qd6sw1h.fsf@topspin.com> <20050514195703.GC10140@mellanox.co.il> <52psvrm6bd.fsf@topspin.com> <20050516191651.GC5743@mellanox.co.il> Message-ID: <52mzqtkcet.fsf@topspin.com> Michael> Another problem is that fls may return 32 and you end up Michael> with 1 << 32 which AFAIK has undefined behaviour in C. I Michael> guess kernel clients can be trusted not to, but uverbs Michael> better check this. Good point. Maybe something like this would be a good idea? - R. --- infiniband/hw/mthca/mthca_qp.c (revision 2360) +++ infiniband/hw/mthca/mthca_qp.c (working copy) @@ -1210,6 +1210,11 @@ static int mthca_alloc_qp_common(struct static int mthca_set_qp_size(struct mthca_dev *dev, struct ib_qp_cap *cap, struct mthca_qp *qp) { + /* Sanity check QP size before proceeding */ + if (cap->max_send_wr > 65536 || cap->max_recv_wr > 65536 || + cap->max_send_sge > 64 || cap->max_recv_sge > 64) + return -EINVAL; + if (mthca_is_memfree(dev)) { qp->rq.max = cap->max_recv_wr ? roundup_pow_of_two(cap->max_recv_wr) : 0; From ftillier at infiniconsys.com Tue May 17 10:14:52 2005 From: ftillier at infiniconsys.com (Fab Tillier) Date: Tue, 17 May 2005 10:14:52 -0700 Subject: [openib-general] User Level Events - request for support In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C3017E@mtlex01.yok.mtl.com> Message-ID: <000001c55b03$f365ed20$9b5aa8c0@infiniconsys.com> > From: Eitan Zahavi [mailto:eitan at mellanox.co.il] > Sent: Tuesday, May 17, 2005 8:21 AM > > > Hal Wrote: > > The OpenSM vendor layer should be enhanced with an additional API for a > > local port state changed event (and take a flag for port down and port > > up). > > > > OpenSM could then take this event and handle it generically. This could > > be implemented for gen2 (OpenIB) and gen1 if the events can be generated > > from VAPI or whatever. > > > > Yes. I agree. We should have osm_vendor_api.h define some callback > registration function for the local port up/down event. I will implement > the OpenSM code for firing a sweep one the port is up. Just to be clear, you need to trap when the port goes to INIT, not ACTIVE. Also, while the SM cares about DOWN/INIT, other ULPs tend to care about DOWN/ACTIVE. I suggest the mechanisms for these events include the port state, and fire for any state change (DOWN, INIT, ARMED, ACTIVE). - Fab From sean.hefty at intel.com Tue May 17 10:16:35 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Tue, 17 May 2005 10:16:35 -0700 Subject: [openib-general] [RFC] [PATCH] [MAD] new call - ib_modify_mad() Message-ID: The following patch adds a new call to the MAD layer, ib_modify_mad(). This call will allow the user to adjust the timeout value of a sent MAD. It will eventually be used to support the CM MRA message, so as to avoid duplicating the MAD layer timeout/retry mechanism. ib_cancel_mad() was simplified by calling ib_modify_mad, and could be eliminated. However, I think it makes sense to keep ib_cancel_mad. Signed-off-by: Sean Hefty Index: include/ib_mad.h =================================================================== --- include/ib_mad.h (revision 2355) +++ include/ib_mad.h (working copy) @@ -484,8 +484,18 @@ void ib_free_recv_mad(struct ib_mad_recv * MADs will be returned to the user through the corresponding * ib_mad_send_handler. */ -void ib_cancel_mad(struct ib_mad_agent *mad_agent, - u64 wr_id); +void ib_cancel_mad(struct ib_mad_agent *mad_agent, u64 wr_id); + +/** + * ib_modify_mad - Modifies an outstanding send MAD operation. + * @mad_agent: Specifies the registration associated with sent MAD. + * @wr_id: Indicates the work request identifier of the MAD to modify. + * @timeout_ms: New timeout value for sent MAD. + * + * This call will reset the timeout value for a sent MAD to the specified + * value. + */ +int ib_modify_mad(struct ib_mad_agent *mad_agent, u64 wr_id, u32 timeout_ms); /** * ib_redirect_mad_qp - Registers a QP for MAD services. Index: core/mad.c =================================================================== --- core/mad.c (revision 2355) +++ core/mad.c (working copy) @@ -63,7 +63,6 @@ static int ib_mad_post_receive_mads(stru struct ib_mad_private *mad); static void cancel_mads(struct ib_mad_agent_private *mad_agent_priv); static void timeout_sends(void *data); -static void cancel_sends(void *data); static void local_completions(void *data); static int add_nonoui_reg_req(struct ib_mad_reg_req *mad_reg_req, struct ib_mad_agent_private *agent_priv, @@ -345,8 +344,6 @@ struct ib_mad_agent *ib_register_mad_age INIT_LIST_HEAD(&mad_agent_priv->local_list); INIT_WORK(&mad_agent_priv->local_work, local_completions, mad_agent_priv); - INIT_LIST_HEAD(&mad_agent_priv->canceled_list); - INIT_WORK(&mad_agent_priv->canceled_work, cancel_sends, mad_agent_priv); atomic_set(&mad_agent_priv->refcount, 1); init_waitqueue_head(&mad_agent_priv->wait); @@ -2094,40 +2091,7 @@ find_send_by_wr_id(struct ib_mad_agent_p return NULL; } -void cancel_sends(void *data) -{ - struct ib_mad_agent_private *mad_agent_priv; - struct ib_mad_send_wr_private *mad_send_wr; - struct ib_mad_send_wc mad_send_wc; - unsigned long flags; - - mad_agent_priv = (struct ib_mad_agent_private *)data; - - mad_send_wc.status = IB_WC_WR_FLUSH_ERR; - mad_send_wc.vendor_err = 0; - - spin_lock_irqsave(&mad_agent_priv->lock, flags); - while (!list_empty(&mad_agent_priv->canceled_list)) { - mad_send_wr = list_entry(mad_agent_priv->canceled_list.next, - struct ib_mad_send_wr_private, - agent_list); - - list_del(&mad_send_wr->agent_list); - spin_unlock_irqrestore(&mad_agent_priv->lock, flags); - - mad_send_wc.wr_id = mad_send_wr->wr_id; - mad_agent_priv->agent.send_handler(&mad_agent_priv->agent, - &mad_send_wc); - - kfree(mad_send_wr); - atomic_dec(&mad_agent_priv->refcount); - spin_lock_irqsave(&mad_agent_priv->lock, flags); - } - spin_unlock_irqrestore(&mad_agent_priv->lock, flags); -} - -void ib_cancel_mad(struct ib_mad_agent *mad_agent, - u64 wr_id) +int ib_modify_mad(struct ib_mad_agent *mad_agent, u64 wr_id, u32 timeout_ms) { struct ib_mad_agent_private *mad_agent_priv; struct ib_mad_send_wr_private *mad_send_wr; @@ -2137,29 +2101,30 @@ void ib_cancel_mad(struct ib_mad_agent * agent); spin_lock_irqsave(&mad_agent_priv->lock, flags); mad_send_wr = find_send_by_wr_id(mad_agent_priv, wr_id); - if (!mad_send_wr) { + if (!mad_send_wr || mad_send_wr->status != IB_WC_SUCCESS) { spin_unlock_irqrestore(&mad_agent_priv->lock, flags); - goto out; + return -EINVAL; } - if (mad_send_wr->status == IB_WC_SUCCESS) - mad_send_wr->refcount -= (mad_send_wr->timeout > 0); - - if (mad_send_wr->refcount != 0) { + if (!timeout_ms) { mad_send_wr->status = IB_WC_WR_FLUSH_ERR; - spin_unlock_irqrestore(&mad_agent_priv->lock, flags); - goto out; + mad_send_wr->refcount -= (mad_send_wr->timeout > 0); } - list_del(&mad_send_wr->agent_list); - list_add_tail(&mad_send_wr->agent_list, &mad_agent_priv->canceled_list); - adjust_timeout(mad_agent_priv); + mad_send_wr->send_wr.wr.ud.timeout_ms = timeout_ms; + if (!mad_send_wr->timeout || mad_send_wr->refcount > 1) + mad_send_wr->timeout = msecs_to_jiffies(timeout_ms); + else + ib_reset_mad_timeout(mad_send_wr, timeout_ms); + spin_unlock_irqrestore(&mad_agent_priv->lock, flags); + return 0; +} +EXPORT_SYMBOL(ib_modify_mad); - queue_work(mad_agent_priv->qp_info->port_priv->wq, - &mad_agent_priv->canceled_work); -out: - return; +void ib_cancel_mad(struct ib_mad_agent *mad_agent, u64 wr_id) +{ + ib_modify_mad(mad_agent, wr_id, 0); } EXPORT_SYMBOL(ib_cancel_mad); @@ -2282,8 +2247,6 @@ static void timeout_sends(void *data) unsigned long flags, delay; mad_agent_priv = (struct ib_mad_agent_private *)data; - - mad_send_wc.status = IB_WC_RESP_TIMEOUT_ERR; mad_send_wc.vendor_err = 0; spin_lock_irqsave(&mad_agent_priv->lock, flags); @@ -2308,6 +2271,10 @@ static void timeout_sends(void *data) spin_unlock_irqrestore(&mad_agent_priv->lock, flags); + if (mad_send_wr->status == IB_WC_SUCCESS) + mad_send_wc.status = IB_WC_RESP_TIMEOUT_ERR; + else + mad_send_wc.status = mad_send_wr->status; mad_send_wc.wr_id = mad_send_wr->wr_id; mad_agent_priv->agent.send_handler(&mad_agent_priv->agent, &mad_send_wc); Index: core/mad_priv.h =================================================================== --- core/mad_priv.h (revision 2355) +++ core/mad_priv.h (working copy) @@ -97,8 +97,6 @@ struct ib_mad_agent_private { unsigned long timeout; struct list_head local_list; struct work_struct local_work; - struct list_head canceled_list; - struct work_struct canceled_work; struct list_head rmpp_list; atomic_t refcount; From libor at topspin.com Tue May 17 10:27:34 2005 From: libor at topspin.com (Libor Michalek) Date: Tue, 17 May 2005 10:27:34 -0700 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <4289249A.2070609@linuxmachines.com>; from jcarr@linuxmachines.com on Mon, May 16, 2005 at 03:54:18PM -0700 References: <42891E31.9000908@linuxmachines.com> <20050516153316.A32217@topspin.com> <4289249A.2070609@linuxmachines.com> Message-ID: <20050517102734.A645@topspin.com> On Mon, May 16, 2005 at 03:54:18PM -0700, Jeff Carr wrote: > Libor Michalek wrote: > > On Mon, May 16, 2005 at 03:26:57PM -0700, Jeff Carr wrote: > > > >>It seems to me it would be useful to have a simple ib_send_page() function. > >> > >>This is essentially what I'm going to end up writing for what I need IB > >>to do. If there is anyone else that has similar needs or interests I'd > >>be happy to work with them. > >> > >>The CM works well enough to allow me to initiate connections between > >>HCA's. I'm now going to spend some time trying to write something to > >>kmalloc pages and test the different ways to initiate IB communication > >>between the hosts. Hopefully I can get something smart enough to perform > >>better than IPoIB does. > > > > FYI, this is what SDP in regular sockets mode is doing. The internal > > buffers are allocated using __get_free_page and then userspace data > > is copied to/from the buffer which are transmitted/received by posting > > send/receive on the allocated pages. > > OK. Well I would rather make something generic. Besides, wasn't there > some MS patent issue? The last thread on the subject that I read kinda > made it sound like you were going to look into the issue and respond. > Maybe I missed the response; there's a lot of mail in the archives for > this list. Sorry, I discussed it at the openib.org workshop. The short answer is that after going through the code we didn't feel that the license was necessary. The question is whether there is any code which relies on MS patents, aside from trying to prove a negative, it does not appear to be the case. > In any case, I'd rather just make something more generic that perhaps > can be used with NBD. I wasn't trying to discourage you, just let you know what existing code is doing, either as a solution, or as an example. -Libor From halr at voltaire.com Tue May 17 10:26:17 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 17 May 2005 13:26:17 -0400 Subject: [openib-general] User Level Events - request for support In-Reply-To: <000001c55b03$f365ed20$9b5aa8c0@infiniconsys.com> References: <000001c55b03$f365ed20$9b5aa8c0@infiniconsys.com> Message-ID: <1116350777.4478.105.camel@localhost.localdomain> On Tue, 2005-05-17 at 13:14, Fab Tillier wrote: > > From: Eitan Zahavi [mailto:eitan at mellanox.co.il] > > Sent: Tuesday, May 17, 2005 8:21 AM > > > > > Hal Wrote: > > > The OpenSM vendor layer should be enhanced with an additional API for a > > > local port state changed event (and take a flag for port down and port > > > up). > > > > > > OpenSM could then take this event and handle it generically. This could > > > be implemented for gen2 (OpenIB) and gen1 if the events can be generated > > > from VAPI or whatever. > > > > > > > Yes. I agree. We should have osm_vendor_api.h define some callback > > registration function for the local port up/down event. I will implement > > the OpenSM code for firing a sweep one the port is up. > > Just to be clear, you need to trap when the port goes to INIT, not ACTIVE. > Also, while the SM cares about DOWN/INIT, other ULPs tend to care about > DOWN/ACTIVE. I suggest the mechanisms for these events include the port > state, and fire for any state change (DOWN, INIT, ARMED, ACTIVE). Are you referring to the link state SM trap which is applicable to switch ports ? If so, we are talking about different things. ULP handling of these events is separate and the "domain" of the ULP. I was referring to the addition of events needed for proper SM operation on its own local port, not on other ports in the subnet. -- Hal From xma at us.ibm.com Tue May 17 10:35:43 2005 From: xma at us.ibm.com (Shirley Ma) Date: Tue, 17 May 2005 10:35:43 -0700 Subject: [openib-general] Gen2 test suites In-Reply-To: <52vf5hkckt.fsf@topspin.com> Message-ID: >I think an automated test suite would be extremely useful, but of > course it would be a lot of work to implement and maintain. We would have more resouce this year, we could work on this with community's help if we think it's important. Thanks Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From mshefty at ichips.intel.com Tue May 17 10:43:20 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Tue, 17 May 2005 10:43:20 -0700 Subject: [openib-general] Gen2 test suites In-Reply-To: References: Message-ID: <428A2D38.7060905@ichips.intel.com> Shirley Ma wrote: >>I think an automated test suite would be extremely useful, but of >>course it would be a lot of work to implement and maintain. > > We would have more resouce this year, we could work on this with > community's help if we think it's important. I think having this would be great. - Sean From ftillier at infiniconsys.com Tue May 17 10:59:47 2005 From: ftillier at infiniconsys.com (Fab Tillier) Date: Tue, 17 May 2005 10:59:47 -0700 Subject: [openib-general] User Level Events - request for support In-Reply-To: <1116350777.4478.105.camel@localhost.localdomain> Message-ID: <000101c55b0a$37608480$9b5aa8c0@infiniconsys.com> > From: Hal Rosenstock [mailto:halr at voltaire.com] > Sent: Tuesday, May 17, 2005 10:26 AM > > On Tue, 2005-05-17 at 13:14, Fab Tillier wrote: > > > > Just to be clear, you need to trap when the port goes to INIT, not > > ACTIVE. Also, while the SM cares about DOWN/INIT, other ULPs tend to > > care about DOWN/ACTIVE. I suggest the mechanisms for these events > > include the port state, and fire for any state change (DOWN, INIT, ARMED, > > ACTIVE). > > Are you referring to the link state SM trap which is applicable to > switch ports ? If so, we are talking about different things. ULP > handling of these events is separate and the "domain" of the ULP. I was > referring to the addition of events needed for proper SM operation on > its own local port, not on other ports in the subnet. No, I'm referring to local port state. Maybe the HCA's event notification isn't quite as sophisticated as I envisioned. The IB spec defines two states for the physical link state - UP and DOWN. For the logical link state, the IB spec defines DOWN, INIT, ARMED, ACTIVE, and ACTIVE_DEFER. The PRM states that the port state change event has a subtype to indicate the down and active *logical* states only. If that's the case, how does the SM detect that the link is connected again, but not configured? Are there additional subtypes for the other logical states that the PRM left out? The logical state moves from DOWN to INIT by HW when the physical state goes from DOWN to UP. Once in INIT, the SM can configure the local port and start sweeping. INIT is what the SM cares about - if it waits for ACTIVE, it will wait until the next sweep since the state change to ARM and ACTIVE are performed by the SM. Hopefully that makes more sense. - Fab From tduffy at sun.com Tue May 17 11:08:17 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 17 May 2005 11:08:17 -0700 Subject: [openib-general] [PATCH] kDAPL: remove bogus arch check Message-ID: <1116353297.9611.9.camel@duffman> This removes the check for supported architectures (DAPL should build on all archs). Signed-off-by: Tom Duffy Index: linux-kernel-sparc64/dat-provider/dapl_util.h =================================================================== --- linux-kernel-sparc64/dat-provider/dapl_util.h (revision 2362) +++ linux-kernel-sparc64/dat-provider/dapl_util.h (working copy) @@ -32,10 +32,6 @@ #ifndef DAPL_UTIL_H #define DAPL_UTIL_H -#if !defined (__i386__) && !defined (__ia64__) && !defined(__PPC__) && !defined (__x86_64__) -#error UNDEFINED ARCH -#endif - #include #include From kashyapv at us.ibm.com Tue May 17 10:46:48 2005 From: kashyapv at us.ibm.com (Vivek Kashyap) Date: Tue, 17 May 2005 10:46:48 -0700 (PDT) Subject: Fw: [openib-general] How about ib_send_page() ? In-Reply-To: Message-ID: An implementation can work with the current draft. The delta to an implementation over an existing IPoIB-UD is not much. That should also help uncover or clarify any hidden issues not well covered in the draft. I've received a few comments - more on details - that I'll incorporate in the next version of the draft. Aiming to post it by the next IETF cut off. With IPoIB-connected mode, existing applications can utilise a higher MTU (the MTU/buffer allocations are negotiable) among connected mode nodes while continuing to interoperate with IPoIB over UD implementations. To be an RFC, we will need another round of review on ietf mailing list. One aspect that will help is a look into the interaction with TCP (see section 6.1 in draft-ietf-ipoib-connected-mode-00.txt). Vivek PS. I'm not subscribed to this list so please cc me at kashyapv at us.ibm.com on any relevant responses. > ----- Forwarded by Shirley Ma/Beaverton/IBM on 05/17/2005 11:01 AM ----- > > Shirley > Ma/Beaverton/IBM@ > IBMUS To > Sent by: Hal Rosenstock > openib-general-bo cc > unces at openib.org openib-general-bounces at openib.org, > openib-general at openib.org > Subject > 05/17/2005 10:04 Re: [openib-general] How about > AM ib_send_page() ? > > > > > > > > > > > > > Note that IPoIB connected mode is currently an internet draft. > Since there is an IP issue of SDP, it's better to implement IPoIB connected > mode soon. We can ask for when this draft comes to RFC. > > Thanks > Shirley Ma > IBM Linux Technology Center > 15300 SW Koll Parkway > Beaverton, OR 97006-6063 > Phone(Fax): (503) 578-7638_______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general From tduffy at sun.com Tue May 17 11:46:26 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 17 May 2005 11:46:26 -0700 Subject: [openib-general] [PATCH] kdapltest: get kdapltest kernel part compiling under sparc64 Message-ID: <1116355586.24548.2.camel@duffman> This patch allows the kernel part of kdapltest to compile on sparc64. I still have not tested any of the dapl stuff on sparc64. Signed-off-by: Tom Duffy Index: linux-kernel-sparc64/test/dapltest/mdep/linux/dapl_mdep_kernel.h =================================================================== --- linux-kernel-sparc64/test/dapltest/mdep/linux/dapl_mdep_kernel.h (revision 2362) +++ linux-kernel-sparc64/test/dapltest/mdep/linux/dapl_mdep_kernel.h (working copy) @@ -127,6 +127,8 @@ DT_Mdep_GetTimeStamp ( void ) unsigned long x; x = get_cycles (); return x; +#elif defined(__sparc_v9__) + return get_cycles(); #else #error "Non-Pentium Linux - unimplemented" #endif From halr at voltaire.com Tue May 17 11:58:50 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 17 May 2005 14:58:50 -0400 Subject: [openib-general] User Level Events - request for support In-Reply-To: <000101c55b0a$37608480$9b5aa8c0@infiniconsys.com> References: <000101c55b0a$37608480$9b5aa8c0@infiniconsys.com> Message-ID: <1116356330.4478.157.camel@localhost.localdomain> On Tue, 2005-05-17 at 13:59, Fab Tillier wrote: > > From: Hal Rosenstock [mailto:halr at voltaire.com] > > Sent: Tuesday, May 17, 2005 10:26 AM > > > > On Tue, 2005-05-17 at 13:14, Fab Tillier wrote: > > > > > > Just to be clear, you need to trap when the port goes to INIT, not > > > ACTIVE. Also, while the SM cares about DOWN/INIT, other ULPs tend to > > > care about DOWN/ACTIVE. I suggest the mechanisms for these events > > > include the port state, and fire for any state change (DOWN, INIT, > ARMED, > > > ACTIVE). > > > > Are you referring to the link state SM trap which is applicable to > > switch ports ? If so, we are talking about different things. ULP > > handling of these events is separate and the "domain" of the ULP. I was > > referring to the addition of events needed for proper SM operation on > > its own local port, not on other ports in the subnet. > > No, I'm referring to local port state. Maybe the HCA's event notification > isn't quite as sophisticated as I envisioned. The IB spec defines two > states for the physical link state - UP and DOWN. For the logical link > state, the IB spec defines DOWN, INIT, ARMED, ACTIVE, and ACTIVE_DEFER. Nit: I don't think that ACTIVE_DEFER is a "true" logical link state (although it is part of the link state machine). > The PRM states that the port state change event has a subtype to indicate the > down and active *logical* states only. If that's the case, how does the SM > detect that the link is connected again, but not configured? Are there > additional subtypes for the other logical states that the PRM left out? If that is what the firmware does, it would need to poll the link state (aka port state) once it got the DOWN event :-( There are several ways to accomplish this (e.g. local MADs, querying mthca, etc.). > The logical state moves from DOWN to INIT by HW when the physical > state goes from DOWN to UP. > Once in INIT, the SM can configure the local port and > start sweeping. INIT is what the SM cares about Right. INIT (PortState) after DOWN (PortPhysicalState). > - if it waits for ACTIVE, > it will wait until the next sweep since the state change to ARM and ACTIVE > are performed by the SM. > Hopefully that makes more sense. Thanks. -- Hal From jlentini at netapp.com Tue May 17 12:14:04 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 17 May 2005 15:14:04 -0400 (EDT) Subject: [openib-general] [PATCH][0/19] kDAPL: More cleanups in dat.h In-Reply-To: <11161816352658@sun.com> References: <11161816352658@sun.com> Message-ID: Thanks Tom. Committed in revision 2363. On Sun, 15 May 2005, Tom Duffy wrote: tduffy> The following set of patches further cleans up dat/dat.h. tduffy> tduffy> Most of them remove the use of typedef for the enums defined in dat/dat.h. tduffy> tduffy> Please apply in order. tduffy> tduffy> Signed-off-by: Tom Duffy tduffy> Your friendly openib janitor. tduffy> tduffy> _______________________________________________ tduffy> openib-general mailing list tduffy> openib-general at openib.org tduffy> http://openib.org/mailman/listinfo/openib-general tduffy> tduffy> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general tduffy> From jlentini at netapp.com Tue May 17 12:20:08 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 17 May 2005 15:20:08 -0400 (EDT) Subject: [openib-general] Re: [PATCH] [kdapl] CM modify retry count in a couple of error messages In-Reply-To: <1116258167.4480.901.camel@localhost.localdomain> References: <1116258167.4480.901.camel@localhost.localdomain> Message-ID: Committed in revision 2365. On Mon, 16 May 2005, Hal Rosenstock wrote: halr> CM modify retry count in a couple of error messages halr> halr> Signed-off-by: Hal Rosenstock halr> halr> Index: dapl_openib_cm.c halr> =================================================================== halr> --- dapl_openib_cm.c (revision 2354) halr> +++ dapl_openib_cm.c (working copy) halr> @@ -280,7 +280,7 @@ halr> halr> if (rec_num <= 0) { halr> printk(KERN_ERR "dapl_path_comp_handler: path resolution " halr> - "failed %d retry %d!!!\n", rec_num, conn->retries); halr> + "failed %d retry %d!!!\n", rec_num, conn->retries + 1); halr> if (++conn->retries > OPENIB_MAX_AT_RETRY) { halr> printk(KERN_ERR "dapl_path_comp_handler: ep_ptr 0x%p\n", halr> conn->ep); halr> @@ -330,7 +330,7 @@ halr> halr> if (rec_num <= 0) { halr> printk(KERN_ERR "dapl_rt_comp_handler: rec num %d retry %d\n", halr> - rec_num, conn->retries); halr> + rec_num, conn->retries + 1); halr> if (++conn->retries > OPENIB_MAX_AT_RETRY) { halr> event = IB_CME_DESTINATION_UNREACHABLE; halr> goto error; halr> halr> From iod00d at hp.com Tue May 17 12:24:49 2005 From: iod00d at hp.com (Grant Grundler) Date: Tue, 17 May 2005 12:24:49 -0700 Subject: [openib-general] Re: [PATCH 1/2] (fixed) mthca: max_inline_data support In-Reply-To: <52mzqtkcet.fsf@topspin.com> References: <20050509141456.GH9941@mellanox.co.il> <52fyww1h90.fsf@topspin.com> <20050510153543.GG2474@mellanox.co.il> <524qd6sw1h.fsf@topspin.com> <20050514195703.GC10140@mellanox.co.il> <52psvrm6bd.fsf@topspin.com> <20050516191651.GC5743@mellanox.co.il> <52mzqtkcet.fsf@topspin.com> Message-ID: <20050517192449.GE1078@esmail.cup.hp.com> On Tue, May 17, 2005 at 10:08:58AM -0700, Roland Dreier wrote: > Michael> Another problem is that fls may return 32 and you end up > Michael> with 1 << 32 which AFAIK has undefined behaviour in C. AFAIK, this is only true for ILP32 C compilers. LP64 C compiler must allow >32 values. ... > - R. > > --- infiniband/hw/mthca/mthca_qp.c (revision 2360) > +++ infiniband/hw/mthca/mthca_qp.c (working copy) > @@ -1210,6 +1210,11 @@ static int mthca_alloc_qp_common(struct > static int mthca_set_qp_size(struct mthca_dev *dev, struct ib_qp_cap *cap, > struct mthca_qp *qp) > { > + /* Sanity check QP size before proceeding */ > + if (cap->max_send_wr > 65536 || cap->max_recv_wr > 65536 || > + cap->max_send_sge > 64 || cap->max_recv_sge > 64) > + return -EINVAL; This doesn't seem right if it's going to work for both 32 and 64-bit binaries. sizeof(long) should be in there somewhere. Unless the above is a generic "safe" limit. I'm looking at mthca_alloc_wqe_buf() and not quite knowing what the real limits are here. grant From jlentini at netapp.com Tue May 17 12:25:48 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 17 May 2005 15:25:48 -0400 (EDT) Subject: [openib-general] Re: [patch][kdapl][trivial] fix debug prints In-Reply-To: <20050516171642.GA32298@mellanox.co.il> References: <20050516171642.GA32298@mellanox.co.il> Message-ID: Committed in revision 2366. On Mon, 16 May 2005, Itamar wrote: itamar> itamar> change print from mr to lmr_context itamar> add \n at end of print itamar> itamar> Signed-off-by: Itamar Rabenstein itamar> itamar> Index: dapl_ep_post_rdma_write.c itamar> =================================================================== itamar> --- dapl_ep_post_rdma_write.c (revision 2355) itamar> +++ dapl_ep_post_rdma_write.c (working copy) itamar> @@ -91,7 +91,7 @@ itamar> OP_WR_RDMA_WRITE); itamar> itamar> dapl_dbg_log(DAPL_DBG_TYPE_RTN, itamar> - "dapl_ep_post_rdma_write () returns 0x%x", dat_status); itamar> + "dapl_ep_post_rdma_write () returns 0x%x\n", dat_status); itamar> itamar> return dat_status; itamar> } itamar> Index: dapl_openib_util.c itamar> =================================================================== itamar> --- dapl_openib_util.c (revision 2355) itamar> +++ dapl_openib_util.c (working copy) itamar> @@ -433,10 +433,6 @@ itamar> ib_status); itamar> return dapl_ib_status_convert(ib_status); itamar> } itamar> - dapl_dbg_log(DAPL_DBG_TYPE_UTIL, itamar> - "dapl_ib_mr_register_physical(%p %d) got handle %p\n", itamar> - buf_list, length, mr); itamar> - itamar> #if 0 itamar> /* ib_query_mr is not implemented yet in OpenIB!!! */ itamar> ib_status = ib_query_mr(mr, &attr); itamar> @@ -453,6 +449,9 @@ itamar> lmr->param.rmr_context = mr->rkey; itamar> lmr->mr_handle = mr; itamar> itamar> + dapl_dbg_log(DAPL_DBG_TYPE_UTIL, itamar> + "dapl_ib_mr_register_physical(%p %d) got lkey 0x%x \n", itamar> + buf_list, length, lmr->param.lmr_context); itamar> return (DAT_SUCCESS); itamar> } itamar> itamar> -- itamar> Itamar itamar> From jlentini at netapp.com Tue May 17 12:31:50 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 17 May 2005 15:31:50 -0400 (EDT) Subject: [openib-general] Re: [PATCH][kdapl] delete dead code (not in Makefile) In-Reply-To: <20050516172247.GA32342@mellanox.co.il> References: <20050516172247.GA32342@mellanox.co.il> Message-ID: Committed in revision 2367. On Mon, 16 May 2005, Itamar wrote: itamar> itamar> delete dead code (not in Makefile) itamar> itamar> Signed-off-by: Itamar Rabenstein itamar> itamar> itamar> Index: dapl_debug.c itamar> =================================================================== itamar> --- dapl_debug.c (revision 2357) itamar> +++ dapl_debug.c (working copy) itamar> @@ -1,45 +0,0 @@ itamar> -/* itamar> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. itamar> - * itamar> - * This Software is licensed under one of the following licenses: itamar> - * itamar> - * 1) under the terms of the "Common Public License 1.0" a copy of which is itamar> - * available from the Open Source Initiative, see itamar> - * http://www.opensource.org/licenses/cpl.php. itamar> - * itamar> - * 2) under the terms of the "The BSD License" a copy of which is itamar> - * available from the Open Source Initiative, see itamar> - * http://www.opensource.org/licenses/bsd-license.php. itamar> - * itamar> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a itamar> - * copy of which is available from the Open Source Initiative, see itamar> - * http://www.opensource.org/licenses/gpl-license.php. itamar> - * itamar> - * Licensee has the right to choose one of the above licenses. itamar> - * itamar> - * Redistributions of source code must retain the above copyright itamar> - * notice and one of the license notices. itamar> - * itamar> - * Redistributions in binary form must reproduce both the above copyright itamar> - * notice, one of the license notices in the documentation itamar> - * and/or other materials provided with the distribution. itamar> - */ itamar> - itamar> -#include "dapl_debug.h" itamar> -#include "dapl_module.h" itamar> -#include "dapl.h" itamar> - itamar> -#ifdef DAPL_DBG itamar> - itamar> -void dapl_dbg_log(DAPL_DBG_TYPE type, const char *fmt, ...) itamar> -{ itamar> - va_list args; itamar> - itamar> - if (type & g_dapl_dbg_type) { itamar> - va_start(args, fmt); itamar> - dapl_os_vprintf(fmt, args); itamar> - va_end(args); itamar> - } itamar> -} itamar> - itamar> -#endif /* DAPL_DBG */ itamar> Index: dapl_debug.h itamar> =================================================================== itamar> --- dapl_debug.h (revision 2357) itamar> +++ dapl_debug.h (working copy) itamar> @@ -1,63 +0,0 @@ itamar> -/* itamar> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. itamar> - * itamar> - * This Software is licensed under one of the following licenses: itamar> - * itamar> - * 1) under the terms of the "Common Public License 1.0" a copy of which is itamar> - * available from the Open Source Initiative, see itamar> - * http://www.opensource.org/licenses/cpl.php. itamar> - * itamar> - * 2) under the terms of the "The BSD License" a copy of which is itamar> - * available from the Open Source Initiative, see itamar> - * http://www.opensource.org/licenses/bsd-license.php. itamar> - * itamar> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a itamar> - * copy of which is available from the Open Source Initiative, see itamar> - * http://www.opensource.org/licenses/gpl-license.php. itamar> - * itamar> - * Licensee has the right to choose one of the above licenses. itamar> - * itamar> - * Redistributions of source code must retain the above copyright itamar> - * notice and one of the license notices. itamar> - * itamar> - * Redistributions in binary form must reproduce both the above copyright itamar> - * notice, one of the license notices in the documentation itamar> - * and/or other materials provided with the distribution. itamar> - */ itamar> - itamar> -/* itamar> - * itamar> - * Defines common deuggging flags and data itamar> - * itamar> - * $Id$ itamar> - */ itamar> - itamar> -#ifndef DAPL_DEBUG_H itamar> -#define DAPL_DEBUG_H itamar> - itamar> -/* itamar> - * Use these bits to enable various tracing/debug options. Each bit itamar> - * represents debugging in a particular subsystem or area of the code. itamar> - */ itamar> -typedef enum { itamar> - DAPL_DBG_TYPE_ERR = 0x0001, itamar> - DAPL_DBG_TYPE_WARN = 0x0002, itamar> - DAPL_DBG_TYPE_EVD = 0x0004, itamar> - DAPL_DBG_TYPE_CM = 0x0008, itamar> - DAPL_DBG_TYPE_EP = 0x0010, itamar> - DAPL_DBG_TYPE_UTIL = 0x0020, itamar> - DAPL_DBG_TYPE_CALLBACK = 0x0040, itamar> - DAPL_DBG_TYPE_DTO_COMP_ERR = 0x0080, itamar> - DAPL_DBG_TYPE_API = 0x0100, itamar> - DAPL_DBG_TYPE_RTN = 0x0200, itamar> - DAPL_DBG_TYPE_EXCEPTION = 0x0400, itamar> - DAPL_DBG_TYPE_SRQ = 0x0800 itamar> -} DAPL_DBG_TYPE; itamar> - itamar> -#if defined(DAPL_DBG) itamar> -extern void dapl_dbg_log(DAPL_DBG_TYPE type, const char *fmt, ...); itamar> -#else /* !DAPL_DBG */ itamar> -#define dapl_dbg_log(...) itamar> -#endif /* DAPL_DBG */ itamar> - itamar> -#endif /* DAPL_DEBUG_H */ itamar> -- itamar> Itamar itamar> From iod00d at hp.com Tue May 17 13:07:49 2005 From: iod00d at hp.com (Grant Grundler) Date: Tue, 17 May 2005 13:07:49 -0700 Subject: [openib-general] perftest-06 patch - output routines In-Reply-To: <20050517155150.GB1078@esmail.cup.hp.com> References: <20050517155150.GB1078@esmail.cup.hp.com> Message-ID: <20050517200749.GG1078@esmail.cup.hp.com> On Tue, May 17, 2005 at 08:51:50AM -0700, Grant Grundler wrote: ... > The following patch makes delta[] an array of cycles_t > and doesn't convert to floating point until the values > are printed as microseconds. ... Sorry, just noticed I forgot: Signed-off-by: Grant Grundler thanks, grant From jlentini at netapp.com Tue May 17 13:20:10 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 17 May 2005 16:20:10 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: remove bogus arch check In-Reply-To: <1116353297.9611.9.camel@duffman> References: <1116353297.9611.9.camel@duffman> Message-ID: Committed in revision 2368. On Tue, 17 May 2005, Tom Duffy wrote: tduffy> This removes the check for supported architectures (DAPL should build on tduffy> all archs). tduffy> tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> Index: linux-kernel-sparc64/dat-provider/dapl_util.h tduffy> =================================================================== tduffy> --- linux-kernel-sparc64/dat-provider/dapl_util.h (revision 2362) tduffy> +++ linux-kernel-sparc64/dat-provider/dapl_util.h (working copy) tduffy> @@ -32,10 +32,6 @@ tduffy> #ifndef DAPL_UTIL_H tduffy> #define DAPL_UTIL_H tduffy> tduffy> -#if !defined (__i386__) && !defined (__ia64__) && !defined(__PPC__) && !defined (__x86_64__) tduffy> -#error UNDEFINED ARCH tduffy> -#endif tduffy> - tduffy> #include tduffy> tduffy> #include tduffy> From blist at aon.at Tue May 17 13:22:34 2005 From: blist at aon.at (Bernhard Fischer) Date: Tue, 17 May 2005 22:22:34 +0200 Subject: [openib-general] Gen2 test suites In-Reply-To: <428A2D38.7060905@ichips.intel.com> References: <428A2D38.7060905@ichips.intel.com> Message-ID: <20050517202234.GT15688@aon.at> On Tue, May 17, 2005 at 10:43:20AM -0700, Sean Hefty wrote: >Shirley Ma wrote: >>>I think an automated test suite would be extremely useful, but of >>>course it would be a lot of work to implement and maintain. >> >>We would have more resouce this year, we could work on this with >>community's help if we think it's important. > >I think having this would be great. [Community help or a testsuite? ;) Late here, so please excuse me] Let me just point anyone interrested to qmtest. YMMV. From jlentini at netapp.com Tue May 17 13:32:31 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 17 May 2005 16:32:31 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kdapltest: get kdapltest kernel part compiling under sparc64 In-Reply-To: <1116355586.24548.2.camel@duffman> References: <1116355586.24548.2.camel@duffman> Message-ID: I committed a change for this in revision 2369. Instead of adding a case for __sparc_v9__, I implemented the function with get_cycles() in all cases since all but one case used this function (and the one that didn't probably should have). Let me know if you see any problems with this. james On Tue, 17 May 2005, Tom Duffy wrote: tduffy> This patch allows the kernel part of kdapltest to compile on sparc64. I tduffy> still have not tested any of the dapl stuff on sparc64. tduffy> tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> Index: linux-kernel-sparc64/test/dapltest/mdep/linux/dapl_mdep_kernel.h tduffy> =================================================================== tduffy> --- linux-kernel-sparc64/test/dapltest/mdep/linux/dapl_mdep_kernel.h (revision 2362) tduffy> +++ linux-kernel-sparc64/test/dapltest/mdep/linux/dapl_mdep_kernel.h (working copy) tduffy> @@ -127,6 +127,8 @@ DT_Mdep_GetTimeStamp ( void ) tduffy> unsigned long x; tduffy> x = get_cycles (); tduffy> return x; tduffy> +#elif defined(__sparc_v9__) tduffy> + return get_cycles(); tduffy> #else tduffy> #error "Non-Pentium Linux - unimplemented" tduffy> #endif tduffy> From libor at topspin.com Tue May 17 13:44:44 2005 From: libor at topspin.com (Libor Michalek) Date: Tue, 17 May 2005 13:44:44 -0700 Subject: [openib-general] Re: Running libibcm/example/simple.c In-Reply-To: <1116262685.4480.938.camel@localhost.localdomain>; from halr@voltaire.com on Mon, May 16, 2005 at 12:58:05PM -0400 References: <1116262685.4480.938.camel@localhost.localdomain> Message-ID: <20050517134444.B645@topspin.com> On Mon, May 16, 2005 at 12:58:05PM -0400, Hal Rosenstock wrote: > Hi again Libor, > > When I run the example I get the following: > server: > CM ID <3> Event <1> State <3> > client: > CM ID <2> Event <0> State <0> > server: > CM ID <3> Event <2> State <0> > > The server side looks like it first gets a REQ received event in REQ > RCVD state. The client then gets a REQ error event in IDLE state. The > server then gets a REP error event in IDLE state. Which type of system is this on? I haven't tried it on 64bit yet, but I'll do that now. Do you get any errors in the log file for either host? If you run the client without a server running do you get the following: CM ID <2> Event <11> State <0> > Also, another nit in libibcm/example/simple.c > cm_connect: > ... > sa.dlid = htons(0x1f9); > sa.slid = htons(0x3e1); > > sa.dlid = 0xf901; > sa.slid = 0xe103; I got rid of the second pair. -Libor From halr at voltaire.com Tue May 17 13:48:25 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 17 May 2005 16:48:25 -0400 Subject: [openib-general] Re: Running libibcm/example/simple.c In-Reply-To: <20050517134444.B645@topspin.com> References: <1116262685.4480.938.camel@localhost.localdomain> <20050517134444.B645@topspin.com> Message-ID: <1116362905.8968.12.camel@localhost.localdomain> On Tue, 2005-05-17 at 16:44, Libor Michalek wrote: > On Mon, May 16, 2005 at 12:58:05PM -0400, Hal Rosenstock wrote: > > Hi again Libor, > > > > When I run the example I get the following: > > server: > > CM ID <3> Event <1> State <3> > > client: > > CM ID <2> Event <0> State <0> > > server: > > CM ID <3> Event <2> State <0> > > > > The server side looks like it first gets a REQ received event in REQ > > RCVD state. The client then gets a REQ error event in IDLE state. The > > server then gets a REP error event in IDLE state. > > Which type of system is this on? Plain old x86 (i386). > I haven't tried it on 64bit yet, but > I'll do that now. > > Do you get any errors in the log file for either host? No. > If you run the > client without a server running do you get the following: > > CM ID <2> Event <11> State <0> (When I ran the server also), I ran it server/client on same machine. -- Hal > > Also, another nit in libibcm/example/simple.c > > cm_connect: > > ... > > sa.dlid = htons(0x1f9); > > sa.slid = htons(0x3e1); > > > > sa.dlid = 0xf901; > > sa.slid = 0xe103; > > I got rid of the second pair. > > -Libor From libor at topspin.com Tue May 17 13:58:37 2005 From: libor at topspin.com (Libor Michalek) Date: Tue, 17 May 2005 13:58:37 -0700 Subject: [openib-general] Re: Re: [PATCH] (repost) sdp: replace mlock with get_user_pages In-Reply-To: <20050514191930.GB10140@mellanox.co.il>; from mst@mellanox.co.il on Sat, May 14, 2005 at 10:19:30PM +0300 References: <20050505110158.GH30653@mellanox.co.il> <20050506175415.I26823@topspin.com> <20050510143232.GF2474@mellanox.co.il> <20050512151051.C23734@topspin.com> <52fywst4z3.fsf@topspin.com> <20050513135145.GA320@mellanox.co.il> <20050513100149.B24123@topspin.com> <20050514191930.GB10140@mellanox.co.il> Message-ID: <20050517135837.C645@topspin.com> On Sat, May 14, 2005 at 10:19:30PM +0300, Michael S. Tsirkin wrote: > Quoting r. Libor Michalek : > > > > > > Another alternative would be to always complete aios asynchronously, > > > > which should preserve the order. I guess this would hurt latency for > > > > small ios... > > > > > > To avoid hurting latency, lets count the number of outstanding > > > asynchronous AIOs, and if there are asynchronous AIOs complete > > > all of them asynchronously. > > > > > > Does this make sense? > > > > Yes, except that the current iocb code does not reference individual > > sockets anywhere, and do_iocb_complete would have to be the function > > which decremented the per connection counter of outstanding AIOs. > > But do_iocb_complete is always called with socket locked, is it not? > If so its not hard to add that counting. iocb_complete() is always called with the socket locked, but by the time do_iocb_complete() executes it's unknown what the socket is doing since it's not referenced anywhere in sdp_iocb. -Libor From libor at topspin.com Tue May 17 14:04:02 2005 From: libor at topspin.com (Libor Michalek) Date: Tue, 17 May 2005 14:04:02 -0700 Subject: [openib-general] Re: [PATCH] (repost) sdp: replace mlock with get_user_pages In-Reply-To: <20050515080503.GB13718@mellanox.co.il>; from mst@mellanox.co.il on Sun, May 15, 2005 at 11:05:03AM +0300 References: <20050505110158.GH30653@mellanox.co.il> <20050506175415.I26823@topspin.com> <20050510143232.GF2474@mellanox.co.il> <20050512151051.C23734@topspin.com> <20050515080503.GB13718@mellanox.co.il> Message-ID: <20050517140402.D645@topspin.com> On Sun, May 15, 2005 at 11:05:03AM +0300, Michael S. Tsirkin wrote: > Quoting r. Libor Michalek : > > + /* > > + * valid result can be 0 or 1 for complete so > > + * we ignore the value. > > + */ > > + (void)aio_complete(iocb->req, value, 0); > > > > if (in_atomic() || irqs_disabled()) { > > INIT_WORK(&iocb->completion, do_iocb_complete, (void *)iocb); > > By the way, as was already discussed here, in_atomic() may not > catch all cases where its illegal to sleep. > If the iocb functions get passed the socket parameter, > the iocb can use the conn->lock.users flag to figure out that its > OK to sleep. The conn->lock.users flag is not appropriate since it can be set and it's still OK to sleep. For example it is set in numerous places where we call kmalloc with the GFP_KERNEL flag. The conn->lock.users flag is only modified inside a spinlock with IRQs disabled, but it can be either value outside of that spinlock. -Libor From jlentini at netapp.com Tue May 17 14:07:30 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 17 May 2005 17:07:30 -0400 (EDT) Subject: [openib-general] Re: [PATCH] [kdapl] Various AT changes In-Reply-To: <1116273155.4480.978.camel@localhost.localdomain> References: <1116273155.4480.978.camel@localhost.localdomain> Message-ID: Committed in revision 2371. On Mon, 16 May 2005, Hal Rosenstock wrote: halr> Various AT changes including: halr> Get API in line with changes on shahar-ibat branch halr> Integrate various changes from shahar-ibat branch which apply to the halr> "level of functionality" originally implemented (e.g. no Service Record halr> support) halr> Fix a couple of minor bugs found by code inspection halr> Better debug support halr> halr> Note that this does not fix the slab corruption problem :-( halr> halr> Signed-off-by: Hal Rosenstock halr> halr> Index: ib_at.h halr> =================================================================== halr> --- ib_at.h (revision 2331) halr> +++ ib_at.h (working copy) halr> @@ -30,25 +30,28 @@ halr> * SOFTWARE. halr> * halr> * halr> - * $Id:$ halr> + * $Id$ halr> */ halr> halr> #if !defined( IB_AT_H ) halr> #define IB_AT_H halr> halr> +#include halr> +#include halr> + halr> enum ib_at_multipathing_type { halr> IB_AT_PATH_SAME_PORT = 0, halr> - IB_AT_PATH_SAME_HCA = 1, /* but different ports if applicable */ halr> - IB_AT_PATH_SAME_SYSTEM = 2, /* but different ports if applicable */ halr> + IB_AT_PATH_SAME_HCA = 1, /* but different ports if applicable */ halr> + IB_AT_PATH_SAME_SYSTEM = 2, /* but different ports if applicable */ halr> IB_AT_PATH_INDEPENDENT_HCA = 3, halr> - IB_AT_PATH_SRC_ROUTE = 4, /* application controlled multipathing */ halr> + IB_AT_PATH_SRC_ROUTE = 4, /* application controlled multipathing */ halr> }; halr> halr> enum ib_at_route_flags { halr> - IB_AT_ROUTE_USE_DEFAULTS = 0, halr> - IB_AT_ROUTE_FORCE_ATS, halr> - IB_AT_ROUTE_FORCE_ARP, halr> - IB_AT_ROUTE_FORCE_RESOLVE, halr> + IB_AT_ROUTE_USE_DEFAULTS = 0, halr> + IB_AT_ROUTE_FORCE_ATS = 1, halr> + IB_AT_ROUTE_FORCE_ARP = 2, halr> + IB_AT_ROUTE_FORCE_RESOLVE = 4, halr> }; halr> halr> struct ib_at_path_attr { halr> @@ -169,7 +172,7 @@ halr> * See ib_at_completion structure documentation for asynchronous halr> * operation details. halr> */ halr> -int ib_at_ips_by_gid(union ib_gid gid, u32 *dst_ips, int nips, halr> +int ib_at_ips_by_gid(union ib_gid *gid, u32 *dst_ips, int nips, halr> struct ib_at_completion *async_comp); halr> halr> /** halr> @@ -208,7 +211,7 @@ halr> * @req_id: asynchronous request ID ib_at_op_status halr> * halr> * Return non-negative ib_at_op_status value, halr> - * or -EINVAL if the reqest ID is invalid. halr> + * or -EINVAL if the request ID is invalid. halr> */ halr> int ib_at_status(u64 req_id); halr> halr> Index: at.c halr> =================================================================== halr> --- at.c (revision 2331) halr> +++ at.c (working copy) halr> @@ -30,7 +30,7 @@ halr> * SOFTWARE. halr> * halr> * halr> - * $Id:$ halr> + * $Id$ halr> */ halr> halr> #include halr> @@ -118,7 +118,7 @@ halr> int sa_id; halr> }; halr> halr> -static struct async pending_reqs; /* dummy head for cyclic list */ halr> +struct async pending_reqs; /* dummy head for cyclic list */ halr> halr> struct ib_at_src { halr> u32 ip; halr> @@ -158,7 +158,6 @@ halr> static void path_req_complete(int stat, struct ib_sa_path_rec *ret, void *ctx); halr> static int resolve_path(struct path_req *req); halr> halr> - halr> static int resolve_ip(struct ib_at_src *src, u32 dst_ip, u32 src_ip, halr> int tos, union ib_gid *dgid) halr> { halr> @@ -254,7 +253,7 @@ halr> src->dev = priv->ca; halr> src->port = priv->port; halr> src->pkey = cpu_to_be16(priv->pkey); halr> - memcpy(&src->gid, (ipoib_dev->dev_addr + 4), sizeof(src->gid)); halr> + memcpy(&src->gid, ipoib_dev->dev_addr + 4, sizeof(src->gid)); halr> halr> if (!dgid) halr> return 0; halr> @@ -264,7 +263,7 @@ halr> * the IB device which was found. halr> */ halr> if (rt->u.dst.neighbour->dev->flags & IFF_LOOPBACK) { halr> - memcpy(dgid, (ipoib_dev->dev_addr + 4), halr> + memcpy(dgid, ipoib_dev->dev_addr + 4, halr> sizeof(union ib_gid)); halr> halr> return 1; halr> @@ -272,7 +271,7 @@ halr> halr> if ((NUD_CONNECTED|NUD_DELAY|NUD_PROBE) & halr> rt->u.dst.neighbour->nud_state) { halr> - memcpy(dgid, (rt->u.dst.neighbour->ha + 4), halr> + memcpy(dgid, rt->u.dst.neighbour->ha + 4, halr> sizeof(union ib_gid)); halr> halr> return 1; halr> @@ -285,9 +284,17 @@ halr> halr> static u64 alloc_req_id(void) halr> { halr> - static u64 req_id = 1; halr> + static u64 req_id = 0; halr> + u64 new_id; halr> + unsigned long flags; halr> halr> - return ++req_id; halr> + spin_lock_irqsave(&pending_reqs.lock, flags); halr> + new_id = ++req_id; halr> + if (!new_id) halr> + new_id = ++req_id; halr> + spin_unlock_irqrestore(&pending_reqs.lock, flags); halr> + halr> + return new_id; halr> } halr> halr> static void req_init(struct async *pend, void *data, int nelem, int type, halr> @@ -361,7 +368,7 @@ halr> { halr> struct async *pend = v; halr> halr> - DEBUG("complete req %p\n", pend); halr> + DEBUG("complete pend %p", pend); halr> halr> pend->comp.fn(pend->comp.req_id, pend->comp.context, pend->nelem); halr> halr> @@ -373,20 +380,23 @@ halr> struct async **rr, *waiting; halr> unsigned long flags = 0; halr> halr> - DEBUG("pend %p nrec %d", pend, nrec); halr> + DEBUG("pend %p nrec %d async %p", pend, nrec, q); halr> halr> if (pend->status != IB_AT_STATUS_PENDING) halr> - WARN("pend %p already completed??", pend); halr> + WARN("pend %p already completed? status %d", pend, pend->status); halr> halr> pend->status = nrec < 0 ? IB_AT_STATUS_ERROR : IB_AT_STATUS_COMPLETED; halr> halr> - if (pend->sa_query) halr> + if (pend->sa_query) { halr> ib_sa_cancel_query(pend->sa_id, pend->sa_query); halr> + pend->sa_query = NULL; halr> + } halr> halr> if (q) halr> spin_lock_irqsave(&q->lock, flags); halr> halr> if (pend->parent) { halr> + DEBUG("pend->parent %p", pend->parent); halr> for (rr = &pend->parent->waiting; *rr; rr = &(*rr)->waiting) halr> if (*rr == pend) { halr> *rr = (*rr)->waiting; halr> @@ -476,11 +486,13 @@ halr> unsigned long flags; halr> struct async *a; halr> halr> - DEBUG("lookup in q %p req %p", q, new); halr> + DEBUG("lookup in q %p pending %p", q, new); halr> spin_lock_irqsave(&q->lock, flags); halr> - for (a = q->next; a != q; a = a->next) halr> + for (a = q->next; a != q; a = a->next) { halr> + DEBUG("%d %d", a->type, type); halr> if (a->type == type && same_fn(a, new)) halr> break; halr> + } halr> halr> spin_unlock_irqrestore(&q->lock, flags); halr> return a == q ? NULL : a; halr> @@ -574,13 +586,14 @@ halr> DEBUG("req %p", req); halr> halr> if (req->pend.parent) { halr> - WARN("path_req_complete for child req %p???", req); halr> + WARN("for child req %p???", req); halr> return; halr> } halr> halr> if (status) { halr> - DEBUG("timed out - check if should retry"); halr> - if (jiffies - req->pend.start < IB_AT_REQ_TIMEOUT) halr> + DEBUG("status %d - check if should retry", status); halr> + if (status == -ETIMEDOUT && halr> + jiffies - req->pend.start < IB_AT_REQ_TIMEOUT) halr> resolve_path(req); halr> else halr> req_end(&req->pend, -ETIMEDOUT, &pending_reqs); halr> @@ -605,6 +618,7 @@ halr> { halr> struct async *pend, *next; halr> struct route_req *req; halr> + struct path_req *preq; halr> unsigned long flags; halr> halr> DEBUG("start sweeping"); halr> @@ -613,18 +627,36 @@ halr> for (pend = pending_reqs.next; pend != &pending_reqs; pend = next) { halr> next = pend->next; halr> halr> - req = container_of(pend, struct route_req, pend); halr> + switch (pend->type) { halr> + case IBAT_REQ_ARP: halr> + case IBAT_REQ_ATS: halr> + req = container_of(pend, struct route_req, pend); halr> halr> - DEBUG("examining route req %p pend %p", req, pend); halr> - if (jiffies > (pend->start + IB_AT_REQ_TIMEOUT)) { halr> - DEBUG("req delete <%d.%d.%d.%d> <%lu:%lu>", halr> - (req->dst_ip & 0x000000ff), halr> - (req->dst_ip & 0x0000ff00) >> 8, halr> - (req->dst_ip & 0x00ff0000) >> 16, halr> - (req->dst_ip & 0xff000000) >> 24, halr> - jiffies, pend->start); halr> + DEBUG("examining route req %p pend %p", req, pend); halr> + if (jiffies > pend->start + IB_AT_REQ_TIMEOUT) { halr> + DEBUG("req delete <%d.%d.%d.%d> <%lu:%lu>", halr> + (req->dst_ip & 0x000000ff), halr> + (req->dst_ip & 0x0000ff00) >> 8, halr> + (req->dst_ip & 0x00ff0000) >> 16, halr> + (req->dst_ip & 0xff000000) >> 24, halr> + jiffies, pend->start); halr> halr> - req_end(pend, -ETIMEDOUT, NULL); halr> + req_end(pend, -ETIMEDOUT, NULL); halr> + } halr> + break; halr> + case IBAT_REQ_PATHREC: halr> + preq = container_of(pend, struct path_req, pend); halr> + halr> + DEBUG("examining path req %p pend %p", preq, pend); halr> + if (jiffies > pend->start + IB_AT_REQ_TIMEOUT) { halr> + DEBUG("req delete path <%lu:%lu>", halr> + jiffies, pend->start); halr> + halr> + req_end(pend, -ETIMEDOUT, NULL); halr> + } halr> + break; halr> + default: halr> + WARN("unknown async req type %d", pend->type); halr> } halr> } halr> halr> @@ -651,7 +683,7 @@ halr> halr> if (req->pend.type == IBAT_REQ_ATS) { halr> WARN("ATS - not yet"); halr> - return 0; halr> + return -1; /* 0 when supported */ halr> } halr> halr> WARN("bad req %p type %d", req, req->pend.type); halr> @@ -666,32 +698,31 @@ halr> .dgid = req->rt.dgid, halr> .sgid = req->rt.sgid, halr> }; halr> - int r; halr> halr> if (req->pend.type != IBAT_REQ_PATHREC) { halr> WARN("bad req %p type %d", req, req->pend.type); halr> return -1; halr> } halr> halr> - r = ib_sa_path_rec_get(req->rt.out_dev, halr> - req->rt.out_port, halr> - &rec, halr> - (IB_SA_PATH_REC_DGID | halr> - IB_SA_PATH_REC_SGID | halr> - IB_SA_PATH_REC_PKEY | halr> - IB_SA_PATH_REC_NUMB_PATH), halr> - req->pend.timeout_ms, halr> - GFP_KERNEL, halr> - path_req_complete, halr> - req, halr> - &req->pend.sa_query); halr> + req->pend.sa_id = ib_sa_path_rec_get(req->rt.out_dev, halr> + req->rt.out_port, halr> + &rec, halr> + (IB_SA_PATH_REC_DGID | halr> + IB_SA_PATH_REC_SGID | halr> + IB_SA_PATH_REC_PKEY | halr> + IB_SA_PATH_REC_NUMB_PATH), halr> + req->pend.timeout_ms, halr> + GFP_KERNEL, halr> + path_req_complete, halr> + req, halr> + &req->pend.sa_query); halr> halr> - if (r < 0) halr> - return r; halr> + if (req->pend.sa_id < 0) { halr> + WARN("ib_sa_path_rec_get %d", req->pend.sa_id); halr> + return req->pend.sa_id; halr> + } halr> halr> req->pend.timeout_ms <<= 1; /* exponential backoff */ halr> - req->pend.sa_id = r; halr> - halr> return 0; halr> } halr> halr> @@ -716,10 +747,12 @@ halr> halr> spin_lock_irqsave(&q->lock, flags); halr> for (a = q->next; a != q; a = a->next) { halr> + DEBUG("a %p", a); halr> if (a->type != IBAT_REQ_ARP) halr> continue; halr> halr> req = container_of(a, struct route_req, pend); halr> + DEBUG("req %p", req); halr> halr> if (arp->op == __constant_htons(ARPOP_REPLY)) { halr> if (arp->dst_ip == req->dst_ip) halr> @@ -751,7 +784,6 @@ halr> * queue IB arp packet onto work queue. halr> */ halr> DEBUG("recv IB ARP - queue work"); halr> - halr> work = kmalloc(sizeof(*work), GFP_ATOMIC); halr> if (!work) halr> goto done; halr> @@ -763,7 +795,6 @@ halr> halr> done: halr> kfree_skb(skb); halr> - halr> return 0; halr> } halr> halr> @@ -796,17 +827,20 @@ halr> halr> r = resolve_ip(&rreq->src, dst_ip, src_ip, tos, &rreq->dgid); halr> if (r < 0) { halr> + DEBUG("resolve_ip r < 0 free req %p", rreq); halr> kmem_cache_free(route_req_cache, rreq); halr> return r; halr> } halr> halr> if (r > 0) { halr> route_req_output(rreq, ib_route); halr> + DEBUG("resolve_ip r > 0 free req %p", rreq); halr> kmem_cache_free(route_req_cache, rreq); halr> return 1; halr> } halr> halr> if (!async_comp) { halr> + DEBUG("!async_comp free req %p", rreq); halr> kmem_cache_free(route_req_cache, rreq); halr> return -EWOULDBLOCK; halr> } halr> @@ -855,6 +889,7 @@ halr> */ halr> halr> if (!async_comp) { halr> + DEBUG("!async_comp free req %p", preq); halr> kmem_cache_free(path_req_cache, preq); halr> return -EWOULDBLOCK; halr> } halr> @@ -871,7 +906,7 @@ halr> } halr> EXPORT_SYMBOL(ib_at_paths_by_route); halr> halr> -int ib_at_ips_by_gid(union ib_gid gid, u32 *dst_ips, int nips, halr> +int ib_at_ips_by_gid(union ib_gid *gid, u32 *dst_ips, int nips, halr> struct ib_at_completion *async_comp) halr> { halr> return -1; /* FIXME: not implemented yet */ halr> @@ -910,7 +945,7 @@ halr> a->next->prev = child; halr> a->next = child; halr> halr> - a->waiting = NULL; /* clear to avoid cancelling childs */ halr> + a->waiting = NULL; /* clear to avoid cancelling children */ halr> } halr> halr> req_end(a, -EINTR, NULL); halr> @@ -950,7 +985,16 @@ halr> halr> DEBUG("IB AT services init"); halr> halr> - route_req_cache = kmem_cache_create("ib_at_route_reqs", halr> + /* halr> + * init pending lists' dummies. halr> + */ halr> + pending_reqs.next = pending_reqs.prev = &pending_reqs; halr> + spin_lock_init(&pending_reqs.lock); halr> + halr> + /* halr> + * Init memory pools halr> + */ halr> + route_req_cache = kmem_cache_create("ib_at_routes", halr> sizeof(struct route_req), halr> 0, SLAB_HWCACHE_ALIGN, halr> NULL, NULL); halr> @@ -960,7 +1004,7 @@ halr> goto err_route; halr> } halr> halr> - path_req_cache = kmem_cache_create("ib_at_path_reqs", halr> + path_req_cache = kmem_cache_create("ib_at_paths", halr> sizeof(struct path_req), halr> 0, SLAB_HWCACHE_ALIGN, halr> NULL, NULL); halr> @@ -970,6 +1014,9 @@ halr> goto err_path; halr> } halr> halr> + /* halr> + * Init ib at worker thread and queue halr> + */ halr> ib_at_wq = create_workqueue("ib_at_wq"); halr> if (!ib_at_wq) { halr> WARN("Failed to allocate IB AT wait queue."); halr> @@ -979,6 +1026,7 @@ halr> halr> INIT_WORK(&ib_at_timer, ib_at_sweep, NULL); halr> queue_delayed_work(ib_at_wq, &ib_at_timer, IB_AT_SWEEP_INTERVAL); halr> + halr> /* halr> * install device for receiving ARP packets in parallel to the normal halr> * Linux ARP, this will be the SDP notifier that an ARP request has halr> halr> halr> From halr at voltaire.com Tue May 17 14:10:58 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 17 May 2005 17:10:58 -0400 Subject: [openib-general] [PATCH] [kdapl] More minor AT changes Message-ID: <1116364258.8968.17.camel@localhost.localdomain> More minor AT changes Integrate a couple more changes from shahar-ibat branch Signed-off-by: Hal Rosenstock Index: at.c =================================================================== --- at.c (revision 2371) +++ at.c (working copy) @@ -454,8 +454,8 @@ struct path_req *pa = container_of(a, struct path_req, pend); struct path_req *pb = container_of(b, struct path_req, pend); - return !memcmp(&pa->rt.sgid, &pa->rt.sgid, sizeof(pa->rt.sgid)) && - !memcmp(&pa->rt.sgid, &pa->rt.sgid, sizeof(pa->rt.sgid)) && + return !memcmp(&pa->rt.sgid, &pb->rt.sgid, sizeof(pa->rt.sgid)) && + !memcmp(&pa->rt.dgid, &pb->rt.dgid, sizeof(pa->rt.dgid)) && pa->rt.out_dev == pb->rt.out_dev && pa->rt.out_port == pb->rt.out_port && pa->rt.attr.pkey == pb->rt.attr.pkey && @@ -590,18 +590,18 @@ return; } + req->pend.sa_query = NULL; + if (status) { DEBUG("status %d - check if should retry", status); if (status == -ETIMEDOUT && jiffies - req->pend.start < IB_AT_REQ_TIMEOUT) resolve_path(req); else - req_end(&req->pend, -ETIMEDOUT, &pending_reqs); + req_end(&req->pend, status, &pending_reqs); return; } - req->pend.sa_query = NULL; - req->pend.nelem = path_req_output(req, resp, 1, req->pend.data, req->pend.nelem); @@ -737,13 +737,13 @@ arp = (struct ib_arp *)skb->nh.raw; - DEBUG("Process IB ARP ip <%d.%d.%d.%d> gid <0x%016llx%016llx>", - (arp->src_ip & 0x000000ff), - (arp->src_ip & 0x0000ff00) >> 8, - (arp->src_ip & 0x00ff0000) >> 16, - (arp->src_ip & 0xff000000) >> 24, - be64_to_cpu(arp->src_gid.global.subnet_prefix), - be64_to_cpu(arp->src_gid.global.interface_id)); + WARN("Process IB ARP ip <%d.%d.%d.%d> gid <0x%016llx%016llx>", + (arp->src_ip & 0x000000ff), + (arp->src_ip & 0x0000ff00) >> 8, + (arp->src_ip & 0x00ff0000) >> 16, + (arp->src_ip & 0xff000000) >> 24, + be64_to_cpu(arp->src_gid.global.subnet_prefix), + be64_to_cpu(arp->src_gid.global.interface_id)); spin_lock_irqsave(&q->lock, flags); for (a = q->next; a != q; a = a->next) { @@ -811,6 +811,9 @@ struct async *parent; int r; + if (flags & IB_AT_ROUTE_FORCE_ATS) + return -EINVAL; /* for now */ + if (!ib_route || !dst_ip) return -EINVAL; @@ -875,6 +878,7 @@ /* TODO: cache lookup if (in_cache) { + DEBUG("!in_cache free req %p", preq); kmem_cache_free(path_req_cache, preq); return path_req_output(preq, cached_arr, n, path_arr, npath); } @@ -883,6 +887,7 @@ /* TODO: resolve outdev if not given r = resolve_outdev(&preq->rt); if (r < 0) { + DEBUG("resolve_outdev r < 0 free req %p", preq); kmem_cache_free(path_req_cache, preq); return r; } @@ -1029,7 +1034,7 @@ /* * install device for receiving ARP packets in parallel to the normal - * Linux ARP, this will be the SDP notifier that an ARP request has + * Linux ARP, this will be the notifier that an ARP request has * completed. */ dev_add_pack(&ib_at_arp_type); From libor at topspin.com Tue May 17 14:18:57 2005 From: libor at topspin.com (Libor Michalek) Date: Tue, 17 May 2005 14:18:57 -0700 Subject: [openib-general] Re: Running libibcm/example/simple.c In-Reply-To: <1116362905.8968.12.camel@localhost.localdomain>; from halr@voltaire.com on Tue, May 17, 2005 at 04:48:25PM -0400 References: <1116262685.4480.938.camel@localhost.localdomain> <20050517134444.B645@topspin.com> <1116362905.8968.12.camel@localhost.localdomain> Message-ID: <20050517141857.E645@topspin.com> On Tue, May 17, 2005 at 04:48:25PM -0400, Hal Rosenstock wrote: > On Tue, 2005-05-17 at 16:44, Libor Michalek wrote: > > On Mon, May 16, 2005 at 12:58:05PM -0400, Hal Rosenstock wrote: > > > Hi again Libor, > > > > > > When I run the example I get the following: > > > server: > > > CM ID <3> Event <1> State <3> > > > client: > > > CM ID <2> Event <0> State <0> > > > server: > > > CM ID <3> Event <2> State <0> > > > If you run the > > client without a server running do you get the following: > > > > CM ID <2> Event <11> State <0> > > (When I ran the server also), I ran it server/client on same machine. That is what's exposing the problem. Since the test is using a fake QPN, a check in the kernel CM see's it as a collision. This patch fixes the issue. -Libor Index: examples/simple.c =================================================================== --- examples/simple.c (revision 2370) +++ examples/simple.c (working copy) @@ -183,7 +183,7 @@ int main(int argc, char **argv) cm_id = event->cm_id; - rep.qp_num = event->param.req_rcvd.remote_qpn; + rep.qp_num = event->param.req_rcvd.remote_qpn + 1; rep.starting_psn = event->param.req_rcvd.starting_psn; rep.private_data = NULL; From jlentini at netapp.com Tue May 17 14:30:55 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 17 May 2005 17:30:55 -0400 (EDT) Subject: [openib-general] Re: [PATCH] [kdapl] More minor AT changes In-Reply-To: <1116364258.8968.17.camel@localhost.localdomain> References: <1116364258.8968.17.camel@localhost.localdomain> Message-ID: Committed in revision 2373. On Tue, 17 May 2005, Hal Rosenstock wrote: halr> More minor AT changes halr> Integrate a couple more changes from shahar-ibat branch halr> halr> Signed-off-by: Hal Rosenstock halr> halr> Index: at.c halr> =================================================================== halr> --- at.c (revision 2371) halr> +++ at.c (working copy) halr> @@ -454,8 +454,8 @@ halr> struct path_req *pa = container_of(a, struct path_req, pend); halr> struct path_req *pb = container_of(b, struct path_req, pend); halr> halr> - return !memcmp(&pa->rt.sgid, &pa->rt.sgid, sizeof(pa->rt.sgid)) && halr> - !memcmp(&pa->rt.sgid, &pa->rt.sgid, sizeof(pa->rt.sgid)) && halr> + return !memcmp(&pa->rt.sgid, &pb->rt.sgid, sizeof(pa->rt.sgid)) && halr> + !memcmp(&pa->rt.dgid, &pb->rt.dgid, sizeof(pa->rt.dgid)) && halr> pa->rt.out_dev == pb->rt.out_dev && halr> pa->rt.out_port == pb->rt.out_port && halr> pa->rt.attr.pkey == pb->rt.attr.pkey && halr> @@ -590,18 +590,18 @@ halr> return; halr> } halr> halr> + req->pend.sa_query = NULL; halr> + halr> if (status) { halr> DEBUG("status %d - check if should retry", status); halr> if (status == -ETIMEDOUT && halr> jiffies - req->pend.start < IB_AT_REQ_TIMEOUT) halr> resolve_path(req); halr> else halr> - req_end(&req->pend, -ETIMEDOUT, &pending_reqs); halr> + req_end(&req->pend, status, &pending_reqs); halr> return; halr> } halr> halr> - req->pend.sa_query = NULL; halr> - halr> req->pend.nelem = path_req_output(req, resp, 1, halr> req->pend.data, req->pend.nelem); halr> halr> @@ -737,13 +737,13 @@ halr> halr> arp = (struct ib_arp *)skb->nh.raw; halr> halr> - DEBUG("Process IB ARP ip <%d.%d.%d.%d> gid <0x%016llx%016llx>", halr> - (arp->src_ip & 0x000000ff), halr> - (arp->src_ip & 0x0000ff00) >> 8, halr> - (arp->src_ip & 0x00ff0000) >> 16, halr> - (arp->src_ip & 0xff000000) >> 24, halr> - be64_to_cpu(arp->src_gid.global.subnet_prefix), halr> - be64_to_cpu(arp->src_gid.global.interface_id)); halr> + WARN("Process IB ARP ip <%d.%d.%d.%d> gid <0x%016llx%016llx>", halr> + (arp->src_ip & 0x000000ff), halr> + (arp->src_ip & 0x0000ff00) >> 8, halr> + (arp->src_ip & 0x00ff0000) >> 16, halr> + (arp->src_ip & 0xff000000) >> 24, halr> + be64_to_cpu(arp->src_gid.global.subnet_prefix), halr> + be64_to_cpu(arp->src_gid.global.interface_id)); halr> halr> spin_lock_irqsave(&q->lock, flags); halr> for (a = q->next; a != q; a = a->next) { halr> @@ -811,6 +811,9 @@ halr> struct async *parent; halr> int r; halr> halr> + if (flags & IB_AT_ROUTE_FORCE_ATS) halr> + return -EINVAL; /* for now */ halr> + halr> if (!ib_route || !dst_ip) halr> return -EINVAL; halr> halr> @@ -875,6 +878,7 @@ halr> halr> /* TODO: cache lookup halr> if (in_cache) { halr> + DEBUG("!in_cache free req %p", preq); halr> kmem_cache_free(path_req_cache, preq); halr> return path_req_output(preq, cached_arr, n, path_arr, npath); halr> } halr> @@ -883,6 +887,7 @@ halr> /* TODO: resolve outdev if not given halr> r = resolve_outdev(&preq->rt); halr> if (r < 0) { halr> + DEBUG("resolve_outdev r < 0 free req %p", preq); halr> kmem_cache_free(path_req_cache, preq); halr> return r; halr> } halr> @@ -1029,7 +1034,7 @@ halr> halr> /* halr> * install device for receiving ARP packets in parallel to the normal halr> - * Linux ARP, this will be the SDP notifier that an ARP request has halr> + * Linux ARP, this will be the notifier that an ARP request has halr> * completed. halr> */ halr> dev_add_pack(&ib_at_arp_type); halr> halr> halr> From mst at mellanox.co.il Tue May 17 15:10:54 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 18 May 2005 01:10:54 +0300 Subject: [openib-general] Re: Re: [PATCH 1/2] (fixed) mthca: max_inline_data support In-Reply-To: <20050517192449.GE1078@esmail.cup.hp.com> References: <20050509141456.GH9941@mellanox.co.il> <52fyww1h90.fsf@topspin.com> <20050510153543.GG2474@mellanox.co.il> <524qd6sw1h.fsf@topspin.com> <20050514195703.GC10140@mellanox.co.il> <52psvrm6bd.fsf@topspin.com> <20050516191651.GC5743@mellanox.co.il> <52mzqtkcet.fsf@topspin.com> <20050517192449.GE1078@esmail.cup.hp.com> Message-ID: <20050517221054.GA17134@mellanox.co.il> Quoting r. Grant Grundler : > Subject: Re: Re: [PATCH 1/2] (fixed) mthca: max_inline_data support > > On Tue, May 17, 2005 at 10:08:58AM -0700, Roland Dreier wrote: > > Michael> Another problem is that fls may return 32 and you end up > > Michael> with 1 << 32 which AFAIK has undefined behaviour in C. > > AFAIK, this is only true for ILP32 C compilers. > LP64 C compiler must allow >32 values. > IFAIK it undefined if int is 32 bit. It seems that's the case with gcc even on 64 bit systems. At least on x86_64 with gcc 3.3.3, int is 32 bit, so: ~>cat x.c #include #include #include int foo(int arg, int shift) { return arg << shift; } int main(int argc, char **argv) { printf("1<<32 = %ld\n", foo(1, 32)); return 0; } ~>gcc x.c ~>./a.out 1<<32 = 1 -- MST - Michael S. Tsirkin From mst at mellanox.co.il Tue May 17 15:14:07 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 18 May 2005 01:14:07 +0300 Subject: [openib-general] Re: [PATCH] (repost) sdp: replace mlock with get_user_pages In-Reply-To: <20050517140402.D645@topspin.com> References: <20050505110158.GH30653@mellanox.co.il> <20050506175415.I26823@topspin.com> <20050510143232.GF2474@mellanox.co.il> <20050512151051.C23734@topspin.com> <20050515080503.GB13718@mellanox.co.il> <20050517140402.D645@topspin.com> Message-ID: <20050517221407.GB17134@mellanox.co.il> Quoting r. Libor Michalek : > Subject: Re: [PATCH] (repost) sdp: replace mlock with get_user_pages > > On Sun, May 15, 2005 at 11:05:03AM +0300, Michael S. Tsirkin wrote: > > Quoting r. Libor Michalek : > > > > + /* > > > + * valid result can be 0 or 1 for complete so > > > + * we ignore the value. > > > + */ > > > + (void)aio_complete(iocb->req, value, 0); > > > > > > if (in_atomic() || irqs_disabled()) { > > > INIT_WORK(&iocb->completion, do_iocb_complete, (void *)iocb); > > > > By the way, as was already discussed here, in_atomic() may not > > catch all cases where its illegal to sleep. > > If the iocb functions get passed the socket parameter, > > the iocb can use the conn->lock.users flag to figure out that its > > OK to sleep. > > The conn->lock.users flag is not appropriate since it can be set and > it's still OK to sleep. For example it is set in numerous places where > we call kmalloc with the GFP_KERNEL flag. The conn->lock.users flag is > only modified inside a spinlock with IRQs disabled, but it can be either > value outside of that spinlock. > > -Libor > SDP_CONN_LOCK_IRQ does not set the users flag. If users is set this means that the socket was locked with sdp_conn_lock, which in turn means we are not in interrupt. Isnt that right? -- MST - Michael S. Tsirkin From bjordan.ics at gmail.com Tue May 17 15:20:39 2005 From: bjordan.ics at gmail.com (William Jordan) Date: Tue, 17 May 2005 18:20:39 -0400 Subject: [openib-general] [PATCH] CM cmpost utility fixes In-Reply-To: <78d18e20505171513e71fa39@mail.gmail.com> References: <78d18e20505171513e71fa39@mail.gmail.com> Message-ID: <78d18e205051715204b101435@mail.gmail.com> Sean, You mentioned that the cm_id leak was fixed, but I don't see a patch for it, so I'm submitting one. Fixed leak of cm_ids in server side of cmpost. Fixed an Oops if number of connections on server and client side don't match. Signed-off-by: Bill Jordan Index: cmpost.c =================================================================== --- cmpost.c (revision 2369) +++ cmpost.c (working copy) @@ -207,7 +207,7 @@ static int modify_to_rts(struct cmtest_n return 0; } -static void req_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event) +static int req_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event) { struct cmtest_node *node; struct ib_cm_req_event_param *req; @@ -215,6 +215,10 @@ static void req_handler(struct ib_cm_id int ret; down(&test.sem); + if (test.conn_index == connections) { + up(&test.sem); + goto fail1; + } node = &test.nodes[ test.conn_index++]; up(&test.sem); @@ -223,11 +227,11 @@ static void req_handler(struct ib_cm_id ret = modify_to_rtr(node); if (ret) - goto fail; + goto fail2; ret = post_recvs(node); if (ret) - goto fail; + goto fail2; req = &event->param.req_rcvd; memset(&rep, 0, sizeof rep); @@ -243,15 +247,18 @@ static void req_handler(struct ib_cm_id ret = ib_send_cm_rep(cm_id, &rep); if (ret) { printk("cmpost: failed to send CM REP: %d\n", ret); - goto fail; + goto fail2; } - return; -fail: - printk("cmpost: failing connection request\n"); - ib_send_cm_rej(cm_id, IB_CM_REJ_UNSUPPORTED, NULL, 0, NULL, 0); + return 0; +fail2: + node->cm_id = NULL; atomic_dec(&test.disconnects_left); if (atomic_dec_and_test(&test.connects_left)) wake_up(&test.wait); +fail1: + printk("cmpost: failing connection request\n"); + ib_send_cm_rej(cm_id, IB_CM_REJ_UNSUPPORTED, NULL, 0, NULL, 0); + return -1; } static void rep_handler(struct cmtest_node *node, struct ib_cm_event *event) @@ -310,10 +317,11 @@ fail: static int cm_handler(struct ib_cm_id *cm_id, struct ib_cm_event *event) { struct cmtest_node *node = cm_id->context; + int ret = 0; switch (event->event) { case IB_CM_REQ_RECEIVED: - req_handler(cm_id, event); + ret = req_handler(cm_id, event); break; case IB_CM_REP_RECEIVED: rep_handler(node, event); @@ -349,7 +357,7 @@ static int cm_handler(struct ib_cm_id *c default: break; } - return 0; + return ret; } static void comp_handler(struct ib_cq *cq, void *cq_context) @@ -406,6 +414,8 @@ static int init_node(struct cmtest_node printk("cmpost: failed to create cm_id: %d\n", ret); return ret; } + } else { + node->cm_id = NULL; } cqe = message_count ? message_count * 2 : 2; @@ -438,7 +448,7 @@ static void destroy_node(struct cmtest_n { ib_destroy_qp(node->qp); ib_destroy_cq(node->cq); - if (!is_server) + if (node->cm_id) ib_destroy_cm_id(node->cm_id); } -- Bill Jordan InfiniCon Systems From xma at us.ibm.com Tue May 17 15:26:41 2005 From: xma at us.ibm.com (Shirley Ma) Date: Tue, 17 May 2005 15:26:41 -0700 Subject: [openib-general] Gen2 test suites In-Reply-To: <20050517202234.GT15688@aon.at> Message-ID: > Community help or a testsuite? ;) Of course testsuites :-). I am setting up two blaze servers for nightly build and regression test. There are some tests and utilities in Gen2 stack already. If you have any nice opensouce test suites, I am glad to integrate them. Please give me any advice on what kind of tests you are interested in. Thanks Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From mshefty at ichips.intel.com Tue May 17 15:32:31 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Tue, 17 May 2005 15:32:31 -0700 Subject: [openib-general] [PATCH] CM cmpost utility fixes In-Reply-To: <78d18e205051715204b101435@mail.gmail.com> References: <78d18e20505171513e71fa39@mail.gmail.com> <78d18e205051715204b101435@mail.gmail.com> Message-ID: <428A70FF.6030605@ichips.intel.com> William Jordan wrote: > Sean, > You mentioned that the cm_id leak was fixed, but I don't see a patch > for it, so I'm > submitting one. I don't see where I fixed it either. Maybe I only fixed the listen_id, oh well, thanks for the patch. > Fixed leak of cm_ids in server side of cmpost. Fixed an Oops if number > of connections > on server and client side don't match. Thanks - I've been wanting to do this, but just haven't had the time yet. I hope to commit this within the next couple of days. - Sean From mst at mellanox.co.il Tue May 17 15:33:34 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 18 May 2005 01:33:34 +0300 Subject: [openib-general] Re: Re: [PATCH] (repost) sdp: replace mlock with get_user_pages In-Reply-To: <20050517135837.C645@topspin.com> References: <20050505110158.GH30653@mellanox.co.il> <20050506175415.I26823@topspin.com> <20050510143232.GF2474@mellanox.co.il> <20050512151051.C23734@topspin.com> <52fywst4z3.fsf@topspin.com> <20050513135145.GA320@mellanox.co.il> <20050513100149.B24123@topspin.com> <20050514191930.GB10140@mellanox.co.il> <20050517135837.C645@topspin.com> Message-ID: <20050517223334.GC17134@mellanox.co.il> Quoting r. Libor Michalek : > Subject: Re: Re: [PATCH] (repost) sdp: replace mlock with get_user_pages > > On Sat, May 14, 2005 at 10:19:30PM +0300, Michael S. Tsirkin wrote: > > Quoting r. Libor Michalek : > > > > > > > > Another alternative would be to always complete aios asynchronously, > > > > > which should preserve the order. I guess this would hurt latency for > > > > > small ios... > > > > > > > > To avoid hurting latency, lets count the number of outstanding > > > > asynchronous AIOs, and if there are asynchronous AIOs complete > > > > all of them asynchronously. > > > > > > > > Does this make sense? > > > > > > Yes, except that the current iocb code does not reference individual > > > sockets anywhere, and do_iocb_complete would have to be the function > > > which decremented the per connection counter of outstanding AIOs. > > > > But do_iocb_complete is always called with socket locked, is it not? > > If so its not hard to add that counting. > > iocb_complete() is always called with the socket locked, but by the > time do_iocb_complete() executes it's unknown what the socket is doing > since it's not referenced anywhere in sdp_iocb. > > -Libor > Wait, thats exactly the case we were trying to handle - spawning a thread to complete the iocb. So if we add a conn pointer to the iocb, we can increment a counter before spawning a thread to complete the iocb and decrement after the iocb is complete. If counter > 0, spawn a thread to complete synchronous io. -- MST - Michael S. Tsirkin From iod00d at hp.com Tue May 17 15:39:38 2005 From: iod00d at hp.com (Grant Grundler) Date: Tue, 17 May 2005 15:39:38 -0700 Subject: [openib-general] Re: Re: [PATCH 1/2] (fixed) mthca: max_inline_data support In-Reply-To: <20050517221054.GA17134@mellanox.co.il> References: <20050509141456.GH9941@mellanox.co.il> <52fyww1h90.fsf@topspin.com> <20050510153543.GG2474@mellanox.co.il> <524qd6sw1h.fsf@topspin.com> <20050514195703.GC10140@mellanox.co.il> <52psvrm6bd.fsf@topspin.com> <20050516191651.GC5743@mellanox.co.il> <52mzqtkcet.fsf@topspin.com> <20050517192449.GE1078@esmail.cup.hp.com> <20050517221054.GA17134@mellanox.co.il> Message-ID: <20050517223938.GM1078@esmail.cup.hp.com> On Wed, May 18, 2005 at 01:10:54AM +0300, Michael S. Tsirkin wrote: > IFAIK it undefined if int is 32 bit. Doh - of course you are right. I was thinking "long", not "int". If the value can legitimately be bigger than an 32-bits, then perhaps the code should be using "unsigned long" for the math? sorry for the noise, grant From mst at mellanox.co.il Tue May 17 15:39:26 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 18 May 2005 01:39:26 +0300 Subject: [openib-general] Re: User Level Events - request for support In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C3017E@mtlex01.yok.mtl.com> References: <506C3D7B14CDD411A52C00025558DED607C3017E@mtlex01.yok.mtl.com> Message-ID: <20050517223926.GD17134@mellanox.co.il> Quoting r. Eitan Zahavi : > Subject: RE: User Level Events - request for support > > > Hal Wrote: > > The OpenSM vendor layer should be enhanced with an additional API for a > > local port state changed event (and take a flag for port down and port > > up). > > > > OpenSM could then take this event and handle it generically. This could > > be implemented for gen2 (OpenIB) and gen1 if the events can be generated > > from VAPI or whatever. > > > > Yes. I agree. We should have osm_vendor_api.h define some callback registration > function for the local port up/down event. I will implement the OpenSM code for > firing a sweep one the port is up. > > EZ > > Eitan Zahavi > Design Technology Director > Mellanox Technologies LTD > Tel:+972-4-9097208 > Fax:+972-4-9593245 > P.O. Box 586 Yokneam 20692 ISRAEL IMHO a function that blocks till there's an event would be a saner API since it matches what kernel can provide. opensm can always create a thread if it wants to get a callback asynchronously. -- MST - Michael S. Tsirkin From halr at voltaire.com Tue May 17 15:41:02 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 17 May 2005 18:41:02 -0400 Subject: [openib-general] Re: Running libibcm/example/simple.c In-Reply-To: <20050517141857.E645@topspin.com> References: <1116262685.4480.938.camel@localhost.localdomain> <20050517134444.B645@topspin.com> <1116362905.8968.12.camel@localhost.localdomain> <20050517141857.E645@topspin.com> Message-ID: <1116369662.8968.38.camel@localhost.localdomain> On Tue, 2005-05-17 at 17:18, Libor Michalek wrote: > > (When I ran the server also), I ran it server/client on same machine. > > That is what's exposing the problem. Since the test is using a fake QPN, > a check in the kernel CM see's it as a collision. This patch fixes the issue. Yup, that fixed it. Thanks. -- Hal From libor at topspin.com Tue May 17 16:04:14 2005 From: libor at topspin.com (Libor Michalek) Date: Tue, 17 May 2005 16:04:14 -0700 Subject: [openib-general] Re: [PATCH] (repost) sdp: replace mlock with get_user_pages In-Reply-To: <20050517221407.GB17134@mellanox.co.il>; from mst@mellanox.co.il on Wed, May 18, 2005 at 01:14:07AM +0300 References: <20050505110158.GH30653@mellanox.co.il> <20050506175415.I26823@topspin.com> <20050510143232.GF2474@mellanox.co.il> <20050512151051.C23734@topspin.com> <20050515080503.GB13718@mellanox.co.il> <20050517140402.D645@topspin.com> <20050517221407.GB17134@mellanox.co.il> Message-ID: <20050517160414.F645@topspin.com> On Wed, May 18, 2005 at 01:14:07AM +0300, Michael S. Tsirkin wrote: > Quoting r. Libor Michalek : > > Subject: Re: [PATCH] (repost) sdp: replace mlock with get_user_pages > > > > On Sun, May 15, 2005 at 11:05:03AM +0300, Michael S. Tsirkin wrote: > > > Quoting r. Libor Michalek : > > > > > > + /* > > > > + * valid result can be 0 or 1 for complete so > > > > + * we ignore the value. > > > > + */ > > > > + (void)aio_complete(iocb->req, value, 0); > > > > > > > > if (in_atomic() || irqs_disabled()) { > > > > INIT_WORK(&iocb->completion, do_iocb_complete, (void *)iocb); > > > > > > By the way, as was already discussed here, in_atomic() may not > > > catch all cases where its illegal to sleep. > > > If the iocb functions get passed the socket parameter, > > > the iocb can use the conn->lock.users flag to figure out that its > > > OK to sleep. > > > > The conn->lock.users flag is not appropriate since it can be set and > > it's still OK to sleep. For example it is set in numerous places where > > we call kmalloc with the GFP_KERNEL flag. The conn->lock.users flag is > > only modified inside a spinlock with IRQs disabled, but it can be either > > value outside of that spinlock. > > > > -Libor > > > > SDP_CONN_LOCK_IRQ does not set the users flag. > > If users is set this means that the socket was locked with > sdp_conn_lock, which in turn means we are not in interrupt. > Isnt that right? Now I see what you're driving at. Currently, the flag is still set when we process the backlog in sdp_conn_unlock() and then sdp_conn_internal_unlock() which is handled under the spinlock, but there's no reason I can see not to clear it ahead of sdp_conn_internal_unlock() instead of after, since it's all under the same spinlock. Same goes for sdp_conn_relock() it would need to be cleared/reset before/after the drain all within the same spinlock. At that point I see no reason not to use it to determine if it is safe to sleep. Either that or make a second flag to avoid overloading the "users" flag's meaning. However, I don't think having this knowledge in iocb_complete() fixes the race condition I described. In the race I saw the iocb_complete() was called correctly inside of sdp_conn_unlock() and then a following aio write finished synchronously. -Libor From jcarr at linuxmachines.com Tue May 17 18:32:38 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Tue, 17 May 2005 18:32:38 -0700 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <20050517043250.GK30200@esmail.cup.hp.com> References: <42891E31.9000908@linuxmachines.com> <52u0l2lrfb.fsf@topspin.com> <42894241.1010908@linuxmachines.com> <20050517014728.GI30200@esmail.cup.hp.com> <428954D7.3060003@linuxmachines.com> <20050517043250.GK30200@esmail.cup.hp.com> Message-ID: <428A9B36.4040901@linuxmachines.com> Grant Grundler wrote: > If it's NAPI that means nothing, here's probably the best summary: > http://lwn.net/Articles/30098/ Cool; I see now. >>>But IPoIB can't really implement NAPI since it's sending work to >>>a shared HCA. Hmm. I'm not knowledgeable to know why; I'll have to take your word for it. I'm not sure yet all the conditions that the HCA can generate interrupts. But if I sit back and look at the logic of this arguement then it seemed like: Hey. is there was a way to not generate so many interrupts? That's handled by NAPI OK. That looks interesting. But, we can't do NAPI because we can't just disable interrupts. Darn. But wait, why can't we just not generate interrupts in the first place then? Isn't that what the midas touch of netdev->poll() really is? e1000 has: quit_polling: netif_rx_complete(netdev); e1000_irq_enable(adapter); Maybe IB can mimic the concept here by acting intellegently for us? Have disable_rx_and_rxnobuff_ints() only disable interrupts for the IPoIB ULP? Anyway, my knowledge here still sucks so I'm probably so far off base I'm not even on the field. Either way it's fun digging around here. > NAPI, TOE, and lots of other things help ethernet with throughput. Hmmm, TOE is interesting. > One can. Using SDP, netperf TCP_STREAM measured 650 MB/s using the > regular PCI-X card. Yes, I have the same speed results using perf_main(). The perf_main() test isn't that interesting I think though. It really just transfers the exact same memory window across two nodes. (at least as far as I can tell that is what it does). Anyway, I'm just noticing that this simple dd test from memory doesn't go much over 1GB/sec. So this is an interesting non-IB problem. root at jcarr:/# dd if=/dev/shm/test of=/dev/null bs=4K 196608+0 records in 196608+0 records out 805306368 bytes transferred in 0.628504 seconds (1281306571 bytes/sec) > I wonder if that could be faster with bigger socket > buffers but don't know (IIRC, IB socket buffers are fixed at 256KB). > Anyway, that's with about 60% CPU (1.5 Ghz ia64). I've got a 3.2Ghz Xeon. 2GB ram. > I haven't checked to see what libibverbs can do with "native" mode > (if one is willing/able to modify the application.) 42 > ISTR Michael Tsirkin had another streaming test in mind to measure > peak bandwidth. But I'm not quite done with rdma_lat yet anyway... Enjoy, Jeff From iod00d at hp.com Tue May 17 18:38:21 2005 From: iod00d at hp.com (Grant Grundler) Date: Tue, 17 May 2005 18:38:21 -0700 Subject: [openib-general] diff-perftest-07 replace pp_get_local_lid() Message-ID: <20050518013821.GO1078@esmail.cup.hp.com> Hi Michael, Following patch to rdma_lat.c: o replaces pp_get_local_lid with code from ibv_pingpong. This calls into libibverbs instead of fishing around in /sys FS. o makes two minor white space fix-ups. Signed-off-by: Grant Grundler I'd like to slowly restructure main() into multiple distinct parts: 1) parameter parsing/setting 2) global data init (e.g. srand()) 3) setup connection 3) negotiate test+parameters with server/client 4) run test (maybe several iterations with different params) 5) exit/cleanup I'm thinking about how to keep the server running and iterating. The goal is to be able to run a sequence of tests just from the client side. Or is this a waste of time? Should I rather be looking at fixing up netperf to support IB? thanks, grant From iod00d at hp.com Tue May 17 19:08:16 2005 From: iod00d at hp.com (Grant Grundler) Date: Tue, 17 May 2005 19:08:16 -0700 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <428A9B36.4040901@linuxmachines.com> References: <42891E31.9000908@linuxmachines.com> <52u0l2lrfb.fsf@topspin.com> <42894241.1010908@linuxmachines.com> <20050517014728.GI30200@esmail.cup.hp.com> <428954D7.3060003@linuxmachines.com> <20050517043250.GK30200@esmail.cup.hp.com> <428A9B36.4040901@linuxmachines.com> Message-ID: <20050518020816.GP1078@esmail.cup.hp.com> On Tue, May 17, 2005 at 06:32:38PM -0700, Jeff Carr wrote: > >>>But IPoIB can't really implement NAPI since it's sending work to > >>>a shared HCA. > > Hmm. I'm not knowledgeable to know why; I'll have to take your word for > it. I'm not sure yet all the conditions that the HCA can generate > interrupts. Wellll..Looks like I'm wrong. Previous email on this thread suggested it's possible by people who know alot more about it than I do. But I'm still concerned it's going to affect latency. > > But if I sit back and look at the logic of this arguement then it seemed > like: > > Hey. is there was a way to not generate so many interrupts? > That's handled by NAPI > OK. That looks interesting. Right - but that's the generic "this is how linux deals with this" argument. > But, we can't do NAPI because we can't just disable interrupts. Sorry - seems like I'm assuming too much about the capabilities of the HCAs. > Darn. > But wait, why can't we just not generate interrupts in the first place then? > > Isn't that what the midas touch of netdev->poll() really is? e1000 has: > quit_polling: netif_rx_complete(netdev); > e1000_irq_enable(adapter); I'm more familiar with tg3 driver. The disadvantage to NAPI in the tg3 implementation is it *alays* disables interrupts on the card before calling netif_rx_schedule(). Then it lets the OS decide when to actually process those packets *already* received in a safer context....then once tg3 decides it's done all the work it re-eables the interrupts. Just like e1000 does above. There are some workloads where the PCI bus utilization is "suboptimal" because the enable/disable of interrupts interfers with the DMA flows and costs excessive overhead. > Maybe IB can mimic the concept here by acting intellegently for us? > Have disable_rx_and_rxnobuff_ints() only disable interrupts for the > IPoIB ULP? Anyway, my knowledge here still sucks so I'm probably so far > off base I'm not even on the field. Either way it's fun digging around here. Based on previous comments, I'm hoping that's the case. But I don't know either. > >One can. Using SDP, netperf TCP_STREAM measured 650 MB/s using the > >regular PCI-X card. > > Yes, I have the same speed results using perf_main(). > > The perf_main() test isn't that interesting I think though. It really > just transfers the exact same memory window across two nodes. (at least > as far as I can tell that is what it does). > > Anyway, I'm just noticing that this simple dd test from memory doesn't > go much over 1GB/sec. So this is an interesting non-IB problem. > > root at jcarr:/# dd if=/dev/shm/test of=/dev/null bs=4K > 196608+0 records in > 196608+0 records out > 805306368 bytes transferred in 0.628504 seconds (1281306571 bytes/sec) Yeah. Sounds like there is. Should be able to do several GB/s like that. I suppose it's possibly an issue with the Memory controller too. ZX1 interleaves accesses across 4 DIMMs to get the memory bandwidth. Might check to make sure your box is "optimally" configured too. grundler at iota$ dd if=/dev/shm/test of=/dev/null bs=4K dd: opening `/dev/shm/test': No such file or directory Sorry - what do I need to do to create /dev/shm/test? I should probably "cheat" and use 16KB block since that is the native page size on ia64. thanks, grant From roland at topspin.com Tue May 17 19:48:32 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 17 May 2005 19:48:32 -0700 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <428A9B36.4040901@linuxmachines.com> (Jeff Carr's message of "Tue, 17 May 2005 18:32:38 -0700") References: <42891E31.9000908@linuxmachines.com> <52u0l2lrfb.fsf@topspin.com> <42894241.1010908@linuxmachines.com> <20050517014728.GI30200@esmail.cup.hp.com> <428954D7.3060003@linuxmachines.com> <20050517043250.GK30200@esmail.cup.hp.com> <428A9B36.4040901@linuxmachines.com> Message-ID: <528y2di70f.fsf@topspin.com> First of all, let me say that to me, IPoIB performance tuning isn't really that interesting. IPoIB is very easy to set up and there's a wide variety of tools that spit out all sorts of numbers, so it's definitely a very accessible area of research, but in the end there are probably better ways to use IB hardware. With that said, I should emphasize that I don't want to discourage anyone from working on whatever strikes their fancy, and I'd certainly be happy to merge patches that improve our performance. The most interesting optimization available is implementing the IPoIB connected mode draft, although I don't think it's as easy as Vivek indicated -- for example, I'm not sure how to deal with having different MTUs depending on the destination. Jeff> Hey. is there was a way to not generate so many interrupts? Jeff> That's handled by NAPI OK. That looks interesting. But, we Jeff> can't do NAPI because we can't just disable interrupts. Jeff> Darn. But wait, why can't we just not generate interrupts Jeff> in the first place then? NAPI isn't really a throughput optimization. It pretty much only helps with small packet workloads. Also, it helps the RX path much more than the TX path, which means it doesn't help much in a symmetric test like netperf or NPtcp. If you look at the table at the beginning of Documentation/networking/NAPI_HOWTO.txt, you can see that for e1000 with 1024 byte packets, the NIC generates 872K RX interrupts for 1M RX packets -- NAPI doesn't kick in totally until the packet size is down to 256 bytes. It is possible although somewhat ugly to implement NAPI for IPoIB, and I actually had hacked up such a patch a while ago. However it didn't end up helping so I never pursued it. When I profile a system running IPoIB throughput tests, half or more of the CPU time is going to skb_copy_and_csum() and other parts of the core kernel's network stack, so our ability to optimize IPoIB is somewhat limited. I've already dealt with all of the easy optimization targets from profiling in IPoIB and mthca -- with MSI-X, we do zero MMIO reads, and Michael Tsirkin has heavily tuned our locking. I'd be curious to know what kind of TCP throughput a modern 10 gigE NIC gets with an MTU of 2044. In some sense that is probably an upper bound for IPoIB throughput. - R. From jlentini at netapp.com Tue May 17 20:25:50 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 17 May 2005 23:25:50 -0400 (EDT) Subject: [openib-general] [PATCH][0/19] kDAPL: More cleanups in dat.h In-Reply-To: <11161816352658@sun.com> References: <11161816352658@sun.com> Message-ID: Tom, I just noticed that one of these patches removed the definition and use of uintptr_t. This results in a few warning messages during compilation. DAPL was using the uintptr_t type when casting to an integer type from a pointer type. Is there another type that the kernel uses for this purpose? I know that long is typically used, but according to Harbison and Stelle this is not guaranteed to be correct. james On Sun, 15 May 2005, Tom Duffy wrote: tduffy> The following set of patches further cleans up dat/dat.h. tduffy> tduffy> Most of them remove the use of typedef for the enums defined in dat/dat.h. tduffy> tduffy> Please apply in order. tduffy> tduffy> Signed-off-by: Tom Duffy tduffy> Your friendly openib janitor. tduffy> tduffy> _______________________________________________ tduffy> openib-general mailing list tduffy> openib-general at openib.org tduffy> http://openib.org/mailman/listinfo/openib-general tduffy> tduffy> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general tduffy> From jlentini at netapp.com Tue May 17 20:28:08 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 17 May 2005 23:28:08 -0400 (EDT) Subject: [openib-general] [DAPL] updated locking Message-ID: I've just checked in an update (revision 2374) to use spin_lock_irqsave and spin_lock_irqrestore in DAPL. Please let me know if you encounter any problems. james From libor at topspin.com Tue May 17 20:29:29 2005 From: libor at topspin.com (Libor Michalek) Date: Tue, 17 May 2005 20:29:29 -0700 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <20050518020816.GP1078@esmail.cup.hp.com>; from iod00d@hp.com on Tue, May 17, 2005 at 07:08:16PM -0700 References: <42891E31.9000908@linuxmachines.com> <52u0l2lrfb.fsf@topspin.com> <42894241.1010908@linuxmachines.com> <20050517014728.GI30200@esmail.cup.hp.com> <428954D7.3060003@linuxmachines.com> <20050517043250.GK30200@esmail.cup.hp.com> <428A9B36.4040901@linuxmachines.com> <20050518020816.GP1078@esmail.cup.hp.com> Message-ID: <20050517202929.A1383@topspin.com> On Tue, May 17, 2005 at 07:08:16PM -0700, Grant Grundler wrote: > On Tue, May 17, 2005 at 06:32:38PM -0700, Jeff Carr wrote: > > >>>But IPoIB can't really implement NAPI since it's sending work to > > >>>a shared HCA. > > > > Hmm. I'm not knowledgeable to know why; I'll have to take your word for > > it. I'm not sure yet all the conditions that the HCA can generate > > interrupts. > > Wellll..Looks like I'm wrong. Previous email on this thread > suggested it's possible by people who know alot more about > it than I do. But I'm still concerned it's going to affect > latency. We do something very NAPI like in SDP. Basically if userspace has the socket lock and an interrupt for one of that sockets CQs is received we mark the socket as having received a notification and do not reenable notifications. As part of userspace unlocking the socket the flags are checked to see if a CQ event has been received, if so then the CQ is polled until it is empty and then notifications are on it are reenabled. When notifications are not enabled the HCA will not generate interrupts for events on that CQ. -Libor From roland at topspin.com Tue May 17 20:31:02 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 17 May 2005 20:31:02 -0700 Subject: [openib-general] [PATCH][0/19] kDAPL: More cleanups in dat.h In-Reply-To: (James Lentini's message of "Tue, 17 May 2005 23:25:50 -0400 (EDT)") References: <11161816352658@sun.com> Message-ID: <521x85i51l.fsf@topspin.com> James> DAPL was using the uintptr_t type when casting to an James> integer type from a pointer type. Is there another type James> that the kernel uses for this purpose? I know that long is James> typically used, but according to Harbison and Stelle this James> is not guaranteed to be correct. The assumption that pointers can be safely cast to and from unsigned long is used all over the Linux kernel. In other words, use "unsigned long" instead of "uintptr_t". - R. From felix at chelsio.com Tue May 17 21:29:32 2005 From: felix at chelsio.com (Felix Marti) Date: Tue, 17 May 2005 21:29:32 -0700 Subject: [openib-general] How about ib_send_page() ? Message-ID: <8A71B368A89016469F72CD08050AD3340911E7@maui.asicdesigners.com> -----Original Message----- From: openib-general-bounces at openib.org [mailto:openib-general-bounces at openib.org] On Behalf Of Roland Dreier Sent: Tuesday, May 17, 2005 7:49 PM To: Jeff Carr Cc: openib-general at openib.org Subject: Re: [openib-general] How about ib_send_page() ? First of all, let me say that to me, IPoIB performance tuning isn't really that interesting. IPoIB is very easy to set up and there's a wide variety of tools that spit out all sorts of numbers, so it's definitely a very accessible area of research, but in the end there are probably better ways to use IB hardware. With that said, I should emphasize that I don't want to discourage anyone from working on whatever strikes their fancy, and I'd certainly be happy to merge patches that improve our performance. The most interesting optimization available is implementing the IPoIB connected mode draft, although I don't think it's as easy as Vivek indicated -- for example, I'm not sure how to deal with having different MTUs depending on the destination. Jeff> Hey. is there was a way to not generate so many interrupts? Jeff> That's handled by NAPI OK. That looks interesting. But, we Jeff> can't do NAPI because we can't just disable interrupts. Jeff> Darn. But wait, why can't we just not generate interrupts Jeff> in the first place then? NAPI isn't really a throughput optimization. It pretty much only helps with small packet workloads. Also, it helps the RX path much more than the TX path, which means it doesn't help much in a symmetric test like netperf or NPtcp. If you look at the table at the beginning of Documentation/networking/NAPI_HOWTO.txt, you can see that for e1000 with 1024 byte packets, the NIC generates 872K RX interrupts for 1M RX packets -- NAPI doesn't kick in totally until the packet size is down to 256 bytes. It is possible although somewhat ugly to implement NAPI for IPoIB, and I actually had hacked up such a patch a while ago. However it didn't end up helping so I never pursued it. When I profile a system running IPoIB throughput tests, half or more of the CPU time is going to skb_copy_and_csum() and other parts of the core kernel's network stack, so our ability to optimize IPoIB is somewhat limited. I've already dealt with all of the easy optimization targets from profiling in IPoIB and mthca -- with MSI-X, we do zero MMIO reads, and Michael Tsirkin has heavily tuned our locking. I'd be curious to know what kind of TCP throughput a modern 10 gigE NIC gets with an MTU of 2044. In some sense that is probably an upper bound for IPoIB throughput. >>>> I get just above 5G on RX (goodput, as reported by netperf) on a single opteron 248 (100%) using standard ethernet MTU (1500). TX performance is higher (close to 7G), but it is probably not the kind of comparison that you're interested in, since TSO removes the dependency on the wire MTU. Similarly, the TOE completely shields the host from the wire MTU (in addition of removing ACK traffic) and I'm getting 7.5G RX and TX with about 50% CPU utilization (user/kernel space memory copy!) These numbers are without NAPI. felix <<< - R. _______________________________________________ openib-general mailing list openib-general at openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From ascensio at doramail.com Tue May 17 22:33:56 2005 From: ascensio at doramail.com (Helen Curtis) Date: Wed, 18 May 2005 08:33:56 +0300 Subject: [openib-general] Your account #8M7078 Message-ID: <114341.0437.ascensio@doramail.com> Hello, We tried contacting you awhile ago about your low interest morta(ge rate. You have qualified for the lowest rate in years... You could get over $380,000 for as little as $500 a month! Ba(d credit? Doesn't matter, low rates are fixed no matter what! To get a free, no obli,gation consultation click below: http://www.truthfu1ly.com/sign.asp Best Regards, Leola Britt to be remov(ed: http://www.truthfu1ly.com/gone.asp this process takes one week, so please be patient. we do our best to take your email/s off but you have to fill out a rem/ove or else you will continue to recieve email/s. From mlleinin at hpcn.ca.sandia.gov Wed May 18 00:08:09 2005 From: mlleinin at hpcn.ca.sandia.gov (Matt Leininger) Date: Wed, 18 May 2005 00:08:09 -0700 Subject: [openib-general] Gen2 test suites In-Reply-To: References: Message-ID: <1116400089.3796.268.camel@localhost> On Tue, 2005-05-17 at 15:26 -0700, Shirley Ma wrote: > > > Community help or a testsuite? ;) > Of course testsuites :-). > > I am setting up two blaze servers for nightly build and regression > test. There are some tests and utilities in Gen2 stack already. If you > have any nice opensouce test suites, I am glad to integrate them. > > Please give me any advice on what kind of tests you are interested in. > Sandia, LANL, and LLNL are very interested in an automated test suite for OpenIB. Sandia and LANL have four IB clusters that could be used for running the test suite. These mid-sized cluster testbeds range between 128-256 nodes [ 138 (x86), 128 (EM64T), 128 (x86), and 256 (Opteron) ]. I have a few PPC970 nodes we could use as well. An automated way to do nightly functionality and performance testing of OpenIB is a great way to provide feedback to developers and other folks in the OpenIB community. What are you using the run the tests? Your own scripts? Something else? Thanks, - Matt From spoole at lanl.gov Wed May 18 00:25:15 2005 From: spoole at lanl.gov (Stephen Poole) Date: Wed, 18 May 2005 01:25:15 -0600 Subject: [openib-general] Gen2 test suites In-Reply-To: <1116400089.3796.268.camel@localhost> References: <1116400089.3796.268.camel@localhost> Message-ID: We have been working on the test harness that we discussed at Sonoma. It is not ready for prime time, but it is a large effort within LANL. It is to be used for other regression test suites. As soon as it is "cleaned up" and ready for prime time, it will be made available. Currently the POC for this for IB is Alfred Torrez (atorrez at lanl.gov). This is a multi-divison project, so it will still be a while before it can be released. Just a heads-up. Steve... At 12:08 AM -0700 5/18/05, Matt Leininger wrote: >On Tue, 2005-05-17 at 15:26 -0700, Shirley Ma wrote: >> >> > Community help or a testsuite? ;) >> Of course testsuites :-). >> >> I am setting up two blaze servers for nightly build and regression >> test. There are some tests and utilities in Gen2 stack already. If you >> have any nice opensouce test suites, I am glad to integrate them. >> >> Please give me any advice on what kind of tests you are interested in. >> > Sandia, LANL, and LLNL are very interested in an automated test suite >for OpenIB. Sandia and LANL have four IB clusters that could be used >for running the test suite. These mid-sized cluster testbeds range >between 128-256 nodes [ 138 (x86), 128 (EM64T), 128 (x86), and 256 >(Opteron) ]. I have a few PPC970 nodes we could use as well. An >automated way to do nightly functionality and performance testing of >OpenIB is a great way to provide feedback to developers and other folks >in the OpenIB community. > > What are you using the run the tests? Your own scripts? Something >else? > > Thanks, > > > - Matt -- Steve Poole (spoole at lanl.gov) Office: 505.665.9662 Los Alamos National Laboratory Cell: 505.699.3807 CCN - Special Projects / Advanced Development Fax: 505.665.7793 P.O. Box 1663, MS B255 Los Alamos, NM. 87545 03149801S From Ouissem.Benfredj at int-evry.fr Wed May 18 00:34:04 2005 From: Ouissem.Benfredj at int-evry.fr (Ouissem BEN FREDJ) Date: Wed, 18 May 2005 09:34:04 +0200 Subject: [openib-general] Vapi Install Message-ID: <428AEFEC.601@int-evry.fr> Hello, Some questions about IB packages. How can I install just the Infiniband HCA Driver and the VAPI package? How can I do that using the "IB Gold" software (http://www.mellanox.com) Is the "Infiniband Subnet Manager" mandatory? Can I configure statically the subnet? PS: Im using 2.4.21-20.EL kernel with x86_64 architecture. Thank you in advance, Ouissem Ben Fredj From eitan at mellanox.co.il Wed May 18 00:43:44 2005 From: eitan at mellanox.co.il (Eitan Zahavi) Date: Wed, 18 May 2005 10:43:44 +0300 Subject: [openib-general] RE: User Level Events - request for support Message-ID: <506C3D7B14CDD411A52C00025558DED607C30183@mtlex01.yok.mtl.com> To isolate the implementation of OpenSM from the specific access layer implementation the thread etc would be in the vendor layer. The code in OpenSM core will use a simpler API: provide a callback and context. Eitan Zahavi Design Technology Director Mellanox Technologies LTD Tel:+972-4-9097208 Fax:+972-4-9593245 P.O. Box 586 Yokneam 20692 ISRAEL > -----Original Message----- > From: Michael S. Tsirkin [mailto:mst at mellanox.co.il] > Sent: Wednesday, May 18, 2005 1:39 AM > To: Eitan Zahavi > Cc: 'Hal Rosenstock'; Liran Sorani; Yael Kalka; openib-general at openib.org > Subject: Re: User Level Events - request for support > > Quoting r. Eitan Zahavi : > > Subject: RE: User Level Events - request for support > > > > > Hal Wrote: > > > The OpenSM vendor layer should be enhanced with an additional API for a > > > local port state changed event (and take a flag for port down and port > > > up). > > > > > > OpenSM could then take this event and handle it generically. This could > > > be implemented for gen2 (OpenIB) and gen1 if the events can be generated > > > from VAPI or whatever. > > > > > > > Yes. I agree. We should have osm_vendor_api.h define some callback registration > > function for the local port up/down event. I will implement the OpenSM code for > > firing a sweep one the port is up. > > > > EZ > > > > Eitan Zahavi > > Design Technology Director > > Mellanox Technologies LTD > > Tel:+972-4-9097208 > > Fax:+972-4-9593245 > > P.O. Box 586 Yokneam 20692 ISRAEL > > IMHO a function that blocks till there's an event would be a saner API > since it matches what kernel can provide. > opensm can always create a thread if it wants to get a callback > asynchronously. > > -- > MST - Michael S. Tsirkin -------------- next part -------------- An HTML attachment was scrubbed... URL: From mst at mellanox.co.il Wed May 18 01:20:23 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 18 May 2005 11:20:23 +0300 Subject: [openib-general] Re: [PATCH][0/19] kDAPL: More cleanups in dat.h In-Reply-To: <521x85i51l.fsf@topspin.com> References: <11161816352658@sun.com> <521x85i51l.fsf@topspin.com> Message-ID: <20050518082023.GC25713@mellanox.co.il> Quoting r. Roland Dreier : > Subject: Re: [PATCH][0/19] kDAPL: More cleanups in dat.h > > James> DAPL was using the uintptr_t type when casting to an > James> integer type from a pointer type. Is there another type > James> that the kernel uses for this purpose? I know that long is > James> typically used, but according to Harbison and Stelle this > James> is not guaranteed to be correct. AFAIK that appeared in the C99 standard. You need #include to get these types though. > The assumption that pointers can be safely cast to and from unsigned > long is used all over the Linux kernel. In other words, use "unsigned > long" instead of "uintptr_t". > > - R. Right. Its OK because kernel requires gcc. -- MST - Michael S. Tsirkin From itamar at mellanox.co.il Wed May 18 04:27:30 2005 From: itamar at mellanox.co.il (Itamar) Date: Wed, 18 May 2005 14:27:30 +0300 Subject: [openib-general] [PATCH][kdapl]add missing kfree Message-ID: <20050518112730.GA25981@mellanox.co.il> Index: dapl_openib_util.c =================================================================== --- dapl_openib_util.c (revision 2374) +++ dapl_openib_util.c (working copy) @@ -432,6 +432,7 @@ ib_status); return dapl_ib_status_convert(ib_status); } + kfree(buf_list); #if 0 /* ib_query_mr is not implemented yet in OpenIB!!! */ ib_status = ib_query_mr(mr, &attr); -- Itamar From itamar at mellanox.co.il Wed May 18 04:50:31 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Wed, 18 May 2005 14:50:31 +0300 Subject: [openib-general] RE: [PATCH][kdapl]add missing kfree Message-ID: <91DB792C7985D411BEC300B40080D29CC35B3E@mtvex01.mtv.mtl.com> sorry there is still bug in this patch i will resend it with the real fix. Itamar > -----Original Message----- > From: Itamar [mailto:itamar at mellanox.co.il] > Sent: Wednesday, May 18, 2005 2:28 PM > To: James.Lentini at netapp.com; openib-general at openib.org > Cc: Itamar > Subject: [PATCH][kdapl]add missing kfree > > > > Index: dapl_openib_util.c > =================================================================== > --- dapl_openib_util.c (revision 2374) > +++ dapl_openib_util.c (working copy) > @@ -432,6 +432,7 @@ > ib_status); > return dapl_ib_status_convert(ib_status); > } > + kfree(buf_list); > #if 0 > /* ib_query_mr is not implemented yet in OpenIB!!! */ > ib_status = ib_query_mr(mr, &attr); > -- > Itamar > From itamar at mellanox.co.il Wed May 18 04:32:29 2005 From: itamar at mellanox.co.il (Itamar) Date: Wed, 18 May 2005 14:32:29 +0300 Subject: [openib-general] [PATCH][kdapl] resend add missing kfree Message-ID: <20050518113229.GA26005@mellanox.co.il> adding missing kfree Signed-off-by: Itamar Rabenstein Index: dapl_openib_util.c =================================================================== --- dapl_openib_util.c (revision 2374) +++ dapl_openib_util.c (working copy) @@ -425,6 +425,7 @@ acl |= IB_ACCESS_MW_BIND; mr = ib_reg_phys_mr((struct ib_pd *)((DAPL_PZ *) lmr->param.pz_handle)-> pd_handle, buf_list, length, acl, &iova); + kfree(buf_list); if (IS_ERR(mr)) { ib_status = PTR_ERR(mr); dapl_dbg_log(DAPL_DBG_TYPE_ERR, -- Itamar From itamar at mellanox.co.il Wed May 18 04:39:48 2005 From: itamar at mellanox.co.il (Itamar) Date: Wed, 18 May 2005 14:39:48 +0300 Subject: [openib-general] [PATCH][kdapl] fix spin_lock_irqsave/spin_unlock_irqrestore implementation Message-ID: <20050518113948.GA26274@mellanox.co.il> when spin_lock_irqsave and spin_unlock_irqrestore are not called in the same function need to pass the flags form the save the store Signed-off-by: Itamar Rabenstein Index: dapl_evd_util.c =================================================================== --- dapl_evd_util.c (revision 2374) +++ dapl_evd_util.c (working copy) @@ -378,20 +378,19 @@ * that the lock is held. */ -static struct dat_event *dapl_evd_get_event(DAPL_EVD * evd_ptr) +static struct dat_event *dapl_evd_get_event(DAPL_EVD * evd_ptr, unsigned long *flags) { struct dat_event *event; - unsigned long flags; if (evd_ptr->evd_producer_locking_needed) { - spin_lock_irqsave(&evd_ptr->header.lock, flags); + spin_lock_irqsave(&evd_ptr->header.lock, *flags); } event = (struct dat_event *) dapl_rbuf_remove(&evd_ptr->free_event_queue); /* Release the lock if it was taken and the call failed. */ if (!event && evd_ptr->evd_producer_locking_needed) { - spin_unlock_irqrestore(&evd_ptr->header.lock, flags); + spin_unlock_irqrestore(&evd_ptr->header.lock, *flags); } return event; @@ -406,10 +405,10 @@ */ static void dapl_evd_post_event(DAPL_EVD *evd_ptr, - const struct dat_event *event_ptr) + const struct dat_event *event_ptr, + unsigned long flags) { u32 dat_status; - unsigned long flags; DAPL_CNO * cno_to_trigger = NULL; dapl_dbg_log(DAPL_DBG_TYPE_EVD, @@ -459,7 +458,7 @@ DAPL_EVD * overflow_evd_ptr) { struct dat_event *overflow_event; - + unsigned long flags; /* The overflow_evd_ptr mght be the same as evd. * In that case we've got a catastrophic overflow. */ @@ -469,7 +468,7 @@ return; } - overflow_event = dapl_evd_get_event(overflow_evd_ptr); + overflow_event = dapl_evd_get_event(overflow_evd_ptr, &flags); if (!overflow_event) { /* this is not good */ overflow_evd_ptr->catastrophic_overflow = TRUE; @@ -477,17 +476,18 @@ return; } dapl_evd_format_overflow_event(overflow_evd_ptr, overflow_event); - dapl_evd_post_event(overflow_evd_ptr, overflow_event); + dapl_evd_post_event(overflow_evd_ptr, overflow_event, flags); return; } static struct dat_event *dapl_evd_get_and_init_event(DAPL_EVD *evd_ptr, - enum dat_event_number evno) + enum dat_event_number evno, + unsigned long *flags) { struct dat_event *event_ptr; - event_ptr = dapl_evd_get_event(evd_ptr); + event_ptr = dapl_evd_get_event(evd_ptr, flags); if (!event_ptr) dapl_evd_post_overflow_event(evd_ptr->header.owner_ia-> async_error_evd, evd_ptr); @@ -507,7 +507,8 @@ DAT_CR_HANDLE cr_handle) { struct dat_event *event_ptr; - event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number); + unsigned long flags; + event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number, &flags); /* * Note event lock may be held on successful return * to be released by dapl_evd_post_event(), if provider side locking @@ -525,7 +526,7 @@ event_ptr->event_data.cr_arrival_event_data.conn_qual = conn_qual; event_ptr->event_data.cr_arrival_event_data.cr_handle = cr_handle; - dapl_evd_post_event(evd_ptr, event_ptr); + dapl_evd_post_event(evd_ptr, event_ptr, flags); return DAT_SUCCESS; } @@ -537,7 +538,8 @@ void *private_data) { struct dat_event *event_ptr; - event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number); + unsigned long flags; + event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number, &flags); /* * Note event lock may be held on successful return * to be released by dapl_evd_post_event(), if provider side locking @@ -554,7 +556,7 @@ = private_data_size; event_ptr->event_data.connect_event_data.private_data = private_data; - dapl_evd_post_event(evd_ptr, event_ptr); + dapl_evd_post_event(evd_ptr, event_ptr, flags); return DAT_SUCCESS; } @@ -564,7 +566,8 @@ DAT_IA_HANDLE ia_handle) { struct dat_event *event_ptr; - event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number); + unsigned long flags; + event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number, &flags); /* * Note event lock may be held on successful return * to be released by dapl_evd_post_event(), if provider side locking @@ -579,7 +582,7 @@ event_ptr->event_data.asynch_error_event_data.dat_handle = (DAT_HANDLE) ia_handle; - dapl_evd_post_event(evd_ptr, event_ptr); + dapl_evd_post_event(evd_ptr, event_ptr, flags); return DAT_SUCCESS; } @@ -589,7 +592,8 @@ void *pointer) { struct dat_event *event_ptr; - event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number); + unsigned long flags; + event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number, &flags); /* * Note event lock may be held on successful return * to be released by dapl_evd_post_event(), if provider side locking @@ -603,7 +607,7 @@ event_ptr->event_data.software_event_data.pointer = pointer; - dapl_evd_post_event(evd_ptr, event_ptr); + dapl_evd_post_event(evd_ptr, event_ptr, flags); return DAT_SUCCESS; } -- Itamar From halr at voltaire.com Wed May 18 06:15:41 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 18 May 2005 09:15:41 -0400 Subject: [openib-general] [PATCH] [kdapl] ib_at.h: Fix commentary typo Message-ID: <1116422141.8968.190.camel@localhost.localdomain> ib_at.h: Fix commentary typo Signed-off-by: Hal Rosenstock Index: ib_at.h =================================================================== --- ib_at.h (revision 2373) +++ ib_at.h (working copy) @@ -149,7 +149,7 @@ * other failover schemes, bandwidth aggregation or source based routing. * Note that multipathing request is meaningless unless npath is greater than 1. * - * Returned ib_route structure includes the recommended pkey and tos_tag for + * Returned ib_route structure includes the recommended pkey and qos_tag for * this route. * * See ib_at_completion structure documentation for asynchronous operation From mst at mellanox.co.il Wed May 18 06:51:30 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 18 May 2005 16:51:30 +0300 Subject: [openib-general] Re: diff-perftest-07 replace pp_get_local_lid() In-Reply-To: <20050518013821.GO1078@esmail.cup.hp.com> References: <20050518013821.GO1078@esmail.cup.hp.com> Message-ID: <20050518135130.GE25713@mellanox.co.il> Quoting r. Grant Grundler : > Subject: diff-perftest-07 replace pp_get_local_lid() > > Hi Michael, > > Following patch to rdma_lat.c: > o replaces pp_get_local_lid with code from ibv_pingpong. > This calls into libibverbs instead of fishing around in /sys FS. > > o makes two minor white space fix-ups. > > Signed-off-by: Grant Grundler No patch was attached :( > > I'd like to slowly restructure main() into multiple distinct parts: > 1) parameter parsing/setting > 2) global data init (e.g. srand()) > 3) setup connection > 3) negotiate test+parameters with server/client > 4) run test (maybe several iterations with different params) > 5) exit/cleanup Good, but lets pass data around in function parameters instead of in global data section. > I'm thinking about how to keep the server running and iterating. > The goal is to be able to run a sequence of tests just > from the client side. I just do ssh server $PWD/rdma_lat & ( sleep 1 && $PWD/rdma_lat) > Or is this a waste of time? > Should I rather be looking at fixing up netperf to support IB? > > thanks, > grant > That may be kind of hard, given that uverbs API is completely different from socket API. -- MST - Michael S. Tsirkin From mst at mellanox.co.il Wed May 18 06:51:49 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 18 May 2005 16:51:49 +0300 Subject: [openib-general] Re: perftest-06 patch - output routines In-Reply-To: <20050517200749.GG1078@esmail.cup.hp.com> References: <20050517155150.GB1078@esmail.cup.hp.com> <20050517200749.GG1078@esmail.cup.hp.com> Message-ID: <20050518135149.GF25713@mellanox.co.il> Quoting r. Grant Grundler : > Subject: Re: perftest-06 patch - output routines > > On Tue, May 17, 2005 at 08:51:50AM -0700, Grant Grundler wrote: > ... > > The following patch makes delta[] an array of cycles_t > > and doesn't convert to floating point until the values > > are printed as microseconds. > ... > > Sorry, just noticed I forgot: > Signed-off-by: Grant Grundler > > thanks, > grant > Thanks, I merged that. -- MST - Michael S. Tsirkin From jlentini at netapp.com Wed May 18 07:12:06 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 18 May 2005 10:12:06 -0400 (EDT) Subject: [openib-general] Re: [PATCH][0/19] kDAPL: More cleanups in dat.h In-Reply-To: <20050518082023.GC25713@mellanox.co.il> References: <11161816352658@sun.com> <521x85i51l.fsf@topspin.com> <20050518082023.GC25713@mellanox.co.il> Message-ID: On Wed, 18 May 2005, Michael S. Tsirkin wrote: > Quoting r. Roland Dreier : >> Subject: Re: [PATCH][0/19] kDAPL: More cleanups in dat.h >> >> James> DAPL was using the uintptr_t type when casting to an >> James> integer type from a pointer type. Is there another type >> James> that the kernel uses for this purpose? I know that long is >> James> typically used, but according to Harbison and Stelle this >> James> is not guaranteed to be correct. > > AFAIK that appeared in the C99 standard. > You need #include to get these types though. The problem is stdint.h is a user level include file. The linux kernel sources don't contain a common definition of uintptr_t. There is a #define for uintptr_t in drivers/net/sk98lin/h/skdrv1st.h in 2.6.11, but obviously that was not intended for general use. I'll go with unsigned long. >> The assumption that pointers can be safely cast to and from unsigned >> long is used all over the Linux kernel. In other words, use "unsigned >> long" instead of "uintptr_t". >> >> - R. > > Right. Its OK because kernel requires gcc. > > -- > MST - Michael S. Tsirkin > From jlentini at netapp.com Wed May 18 07:16:04 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 18 May 2005 10:16:04 -0400 (EDT) Subject: [openib-general] Re: [PATCH][kdapl] resend add missing kfree In-Reply-To: <20050518113229.GA26005@mellanox.co.il> References: <20050518113229.GA26005@mellanox.co.il> Message-ID: Committed in revision 2378. On Wed, 18 May 2005, Itamar wrote: itamar> adding missing kfree itamar> itamar> Signed-off-by: Itamar Rabenstein itamar> itamar> Index: dapl_openib_util.c itamar> =================================================================== itamar> --- dapl_openib_util.c (revision 2374) itamar> +++ dapl_openib_util.c (working copy) itamar> @@ -425,6 +425,7 @@ itamar> acl |= IB_ACCESS_MW_BIND; itamar> mr = ib_reg_phys_mr((struct ib_pd *)((DAPL_PZ *) lmr->param.pz_handle)-> itamar> pd_handle, buf_list, length, acl, &iova); itamar> + kfree(buf_list); itamar> if (IS_ERR(mr)) { itamar> ib_status = PTR_ERR(mr); itamar> dapl_dbg_log(DAPL_DBG_TYPE_ERR, itamar> itamar> -- itamar> Itamar itamar> From jlentini at netapp.com Wed May 18 07:20:05 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 18 May 2005 10:20:05 -0400 (EDT) Subject: [openib-general] Re: [PATCH] [kdapl] ib_at.h: Fix commentary typo In-Reply-To: <1116422141.8968.190.camel@localhost.localdomain> References: <1116422141.8968.190.camel@localhost.localdomain> Message-ID: Committed in revision 2379. On Wed, 18 May 2005, Hal Rosenstock wrote: halr> ib_at.h: Fix commentary typo halr> halr> Signed-off-by: Hal Rosenstock halr> halr> Index: ib_at.h halr> =================================================================== halr> --- ib_at.h (revision 2373) halr> +++ ib_at.h (working copy) halr> @@ -149,7 +149,7 @@ halr> * other failover schemes, bandwidth aggregation or source based routing. halr> * Note that multipathing request is meaningless unless npath is greater than 1. halr> * halr> - * Returned ib_route structure includes the recommended pkey and tos_tag for halr> + * Returned ib_route structure includes the recommended pkey and qos_tag for halr> * this route. halr> * halr> * See ib_at_completion structure documentation for asynchronous operation halr> halr> From mst at mellanox.co.il Wed May 18 08:46:43 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 18 May 2005 18:46:43 +0300 Subject: [openib-general] perftest updated Message-ID: <20050518154643.GL25713@mellanox.co.il> Hi! I have made some further cleanups: - summary format cleaned up somewhat, for better usability and so I can grep for the values I want more easily - report options now isolated in a structure of their own thats the way I plan to go down with other variables - report now done in the report function, unifying mhz/cycles format -- MST - Michael S. Tsirkin From jlentini at netapp.com Wed May 18 09:42:07 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 18 May 2005 12:42:07 -0400 (EDT) Subject: [openib-general] Re: [PATCH][kdapl] fix spin_lock_irqsave/spin_unlock_irqrestore implementation In-Reply-To: <20050518113948.GA26274@mellanox.co.il> References: <20050518113948.GA26274@mellanox.co.il> Message-ID: Wow, I didn't realize that we did this in the code. Can we simplify this? I'd rather not have the code taking a lock in one function and releasing it in another. This code came from the generic source forge implementation. Do we need to do "evd producer locking" for OpenIB? If not we can remove these locks. james On Wed, 18 May 2005, Itamar wrote: itamar> when spin_lock_irqsave and spin_unlock_irqrestore are not called in the same function itamar> need to pass the flags form the save the store itamar> itamar> Signed-off-by: Itamar Rabenstein itamar> itamar> Index: dapl_evd_util.c itamar> =================================================================== itamar> --- dapl_evd_util.c (revision 2374) itamar> +++ dapl_evd_util.c (working copy) itamar> @@ -378,20 +378,19 @@ itamar> * that the lock is held. itamar> */ itamar> itamar> -static struct dat_event *dapl_evd_get_event(DAPL_EVD * evd_ptr) itamar> +static struct dat_event *dapl_evd_get_event(DAPL_EVD * evd_ptr, unsigned long *flags) itamar> { itamar> struct dat_event *event; itamar> - unsigned long flags; itamar> itamar> if (evd_ptr->evd_producer_locking_needed) { itamar> - spin_lock_irqsave(&evd_ptr->header.lock, flags); itamar> + spin_lock_irqsave(&evd_ptr->header.lock, *flags); itamar> } itamar> itamar> event = (struct dat_event *) dapl_rbuf_remove(&evd_ptr->free_event_queue); itamar> itamar> /* Release the lock if it was taken and the call failed. */ itamar> if (!event && evd_ptr->evd_producer_locking_needed) { itamar> - spin_unlock_irqrestore(&evd_ptr->header.lock, flags); itamar> + spin_unlock_irqrestore(&evd_ptr->header.lock, *flags); itamar> } itamar> itamar> return event; itamar> @@ -406,10 +405,10 @@ itamar> */ itamar> itamar> static void dapl_evd_post_event(DAPL_EVD *evd_ptr, itamar> - const struct dat_event *event_ptr) itamar> + const struct dat_event *event_ptr, itamar> + unsigned long flags) itamar> { itamar> u32 dat_status; itamar> - unsigned long flags; itamar> DAPL_CNO * cno_to_trigger = NULL; itamar> itamar> dapl_dbg_log(DAPL_DBG_TYPE_EVD, itamar> @@ -459,7 +458,7 @@ itamar> DAPL_EVD * overflow_evd_ptr) itamar> { itamar> struct dat_event *overflow_event; itamar> - itamar> + unsigned long flags; itamar> /* The overflow_evd_ptr mght be the same as evd. itamar> * In that case we've got a catastrophic overflow. itamar> */ itamar> @@ -469,7 +468,7 @@ itamar> return; itamar> } itamar> itamar> - overflow_event = dapl_evd_get_event(overflow_evd_ptr); itamar> + overflow_event = dapl_evd_get_event(overflow_evd_ptr, &flags); itamar> if (!overflow_event) { itamar> /* this is not good */ itamar> overflow_evd_ptr->catastrophic_overflow = TRUE; itamar> @@ -477,17 +476,18 @@ itamar> return; itamar> } itamar> dapl_evd_format_overflow_event(overflow_evd_ptr, overflow_event); itamar> - dapl_evd_post_event(overflow_evd_ptr, overflow_event); itamar> + dapl_evd_post_event(overflow_evd_ptr, overflow_event, flags); itamar> itamar> return; itamar> } itamar> itamar> static struct dat_event *dapl_evd_get_and_init_event(DAPL_EVD *evd_ptr, itamar> - enum dat_event_number evno) itamar> + enum dat_event_number evno, itamar> + unsigned long *flags) itamar> { itamar> struct dat_event *event_ptr; itamar> itamar> - event_ptr = dapl_evd_get_event(evd_ptr); itamar> + event_ptr = dapl_evd_get_event(evd_ptr, flags); itamar> if (!event_ptr) itamar> dapl_evd_post_overflow_event(evd_ptr->header.owner_ia-> itamar> async_error_evd, evd_ptr); itamar> @@ -507,7 +507,8 @@ itamar> DAT_CR_HANDLE cr_handle) itamar> { itamar> struct dat_event *event_ptr; itamar> - event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number); itamar> + unsigned long flags; itamar> + event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number, &flags); itamar> /* itamar> * Note event lock may be held on successful return itamar> * to be released by dapl_evd_post_event(), if provider side locking itamar> @@ -525,7 +526,7 @@ itamar> event_ptr->event_data.cr_arrival_event_data.conn_qual = conn_qual; itamar> event_ptr->event_data.cr_arrival_event_data.cr_handle = cr_handle; itamar> itamar> - dapl_evd_post_event(evd_ptr, event_ptr); itamar> + dapl_evd_post_event(evd_ptr, event_ptr, flags); itamar> itamar> return DAT_SUCCESS; itamar> } itamar> @@ -537,7 +538,8 @@ itamar> void *private_data) itamar> { itamar> struct dat_event *event_ptr; itamar> - event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number); itamar> + unsigned long flags; itamar> + event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number, &flags); itamar> /* itamar> * Note event lock may be held on successful return itamar> * to be released by dapl_evd_post_event(), if provider side locking itamar> @@ -554,7 +556,7 @@ itamar> = private_data_size; itamar> event_ptr->event_data.connect_event_data.private_data = private_data; itamar> itamar> - dapl_evd_post_event(evd_ptr, event_ptr); itamar> + dapl_evd_post_event(evd_ptr, event_ptr, flags); itamar> itamar> return DAT_SUCCESS; itamar> } itamar> @@ -564,7 +566,8 @@ itamar> DAT_IA_HANDLE ia_handle) itamar> { itamar> struct dat_event *event_ptr; itamar> - event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number); itamar> + unsigned long flags; itamar> + event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number, &flags); itamar> /* itamar> * Note event lock may be held on successful return itamar> * to be released by dapl_evd_post_event(), if provider side locking itamar> @@ -579,7 +582,7 @@ itamar> event_ptr->event_data.asynch_error_event_data.dat_handle = itamar> (DAT_HANDLE) ia_handle; itamar> itamar> - dapl_evd_post_event(evd_ptr, event_ptr); itamar> + dapl_evd_post_event(evd_ptr, event_ptr, flags); itamar> itamar> return DAT_SUCCESS; itamar> } itamar> @@ -589,7 +592,8 @@ itamar> void *pointer) itamar> { itamar> struct dat_event *event_ptr; itamar> - event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number); itamar> + unsigned long flags; itamar> + event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number, &flags); itamar> /* itamar> * Note event lock may be held on successful return itamar> * to be released by dapl_evd_post_event(), if provider side locking itamar> @@ -603,7 +607,7 @@ itamar> itamar> event_ptr->event_data.software_event_data.pointer = pointer; itamar> itamar> - dapl_evd_post_event(evd_ptr, event_ptr); itamar> + dapl_evd_post_event(evd_ptr, event_ptr, flags); itamar> itamar> return DAT_SUCCESS; itamar> } itamar> itamar> -- itamar> Itamar itamar> From blist at aon.at Wed May 18 10:05:02 2005 From: blist at aon.at (Bernhard Fischer) Date: Wed, 18 May 2005 19:05:02 +0200 Subject: [openib-general] [PATCH][RFC][0/4] InfiniBand userspace verbs implementation In-Reply-To: <200544159.Ahk9l0puXy39U6u6@topspin.com> References: <200544159.Ahk9l0puXy39U6u6@topspin.com> Message-ID: <20050518170502.GB7553@aon.at> On Mon, Apr 04, 2005 at 03:09:00PM -0700, Roland Dreier wrote: >Here is an initial implementation of InfiniBand userspace verbs. I >plan to commit this code to the OpenIB repository shortly, and submit >it for inclusion during the 2.6.13 cycle, so I am posting it early for >comments. [snip] >Please take a look at this code if you have a chance. I would >appreciate high-level criticism of the design and implementation as >well as nitpicky complaints about coding style and typos. Can we eventually have checks for NULL before dereferencing members? I'd be happy if they would be wrapped in #ifdef DEBUG or the like, so common use would not suffer the additional penalty those checks imply. e.g.: ibv_modify_qp(NULL,...) etc, etc i.e. ibv_() and also the API exposed by mthca. Something like ibverbs:verbs.c:145: Error: ibv_modify_qp called with qp==NULL Not pretty, agree, but would be convenient for testing new users of those libraries. Just a suggestion. > >In particular, the memory pinning code in in uverbs_mem.c could stand >a looking over. In addition, a sanity check of the write()-based >scheme for passing commands into the kernel in uverbs_main.c and >uverbs_cmd.c is probably worthwhile. From iod00d at hp.com Wed May 18 11:06:20 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 18 May 2005 11:06:20 -0700 Subject: [openib-general] Re: diff-perftest-07 replace pp_get_local_lid() In-Reply-To: <20050518135130.GE25713@mellanox.co.il> References: <20050518013821.GO1078@esmail.cup.hp.com> <20050518135130.GE25713@mellanox.co.il> Message-ID: <20050518180620.GA6004@esmail.cup.hp.com> On Wed, May 18, 2005 at 04:51:30PM +0300, Michael S. Tsirkin wrote: > Quoting r. Grant Grundler : > > Following patch to rdma_lat.c: > > o replaces pp_get_local_lid with code from ibv_pingpong. > > This calls into libibverbs instead of fishing around in /sys FS. > > > > o makes two minor white space fix-ups. > > > > Signed-off-by: Grant Grundler > > No patch was attached :( Doh! sorry...appended now > > I'd like to slowly restructure main() into multiple distinct parts: > > 1) parameter parsing/setting > > 2) global data init (e.g. srand()) > > 3) setup connection > > 3) negotiate test+parameters with server/client > > 4) run test (maybe several iterations with different params) > > 5) exit/cleanup > > Good, but lets pass data around in function parameters > instead of in global data section. Certainly. There should be very little/no global data when I'm done. srand/errno type stuff should be about it. > > I'm thinking about how to keep the server running and iterating. > > The goal is to be able to run a sequence of tests just > > from the client side. > > I just do ssh server $PWD/rdma_lat & ( sleep 1 && $PWD/rdma_lat) hrm...that would work too though it's not as intuitive. > > Or is this a waste of time? > > Should I rather be looking at fixing up netperf to support IB? > > That may be kind of hard, given that uverbs API is completely different > from socket API. Agreed - but that's not the part I'm trying to leverage. I want: o statistical infrastructure. o automatically run multiple iterations until confidence level is reached o CPU utilization (aka "Service Demand") o bind task to a particular CPU I can add those things to rmda_lat but it seems like it's a wash to add IB support to netperf. And since netperf.org is pretty well known and no-one has a clue about rdma_lat..it seems like a more obvious place to do the work. I guess that's a political argument for doing the work in netperf. I'll think about it more and talk to the netperf maintainer later this week. thanks, grant Signed-off-by: Grant Grundler --- rdma_lat.c-06 2005-05-17 08:50:35.000000000 -0700 +++ rdma_lat.c 2005-05-17 13:19:56.000000000 -0700 @@ -91,31 +91,17 @@ unsigned long long vaddr; }; -/* - * pp_get_local_lid() uses a pretty bogus method for finding the LID - * of a local port. Please don't copy this into your app (or if you - * do, please rip it out soon). - */ -static uint16_t pp_get_local_lid(struct ibv_device *dev, int port) -{ - char path[256]; - char val[16]; - char *name; - if (sysfs_get_mnt_path(path, sizeof path)) { - fprintf(stderr, "Couldn't find sysfs mount.\n"); - return 0; - } +/* shamelessly taken from src/userspace/libibverbs/examples/pingpong.c */ - asprintf(&name, "%s/class/infiniband/%s/ports/%d/lid", path, - ibv_get_device_name(dev), port); +static uint16_t pp_get_local_lid(struct pingpong_context *ctx, int port) +{ + struct ibv_port_attr attr; - if (sysfs_read_attribute_value(name, val, sizeof val)) { - fprintf(stderr, "Couldn't read LID at %s\n", name); + if (ibv_query_port(ctx->context, port, &attr)) return 0; - } - return strtol(val, NULL, 0); + return attr.lid; } static int pp_client_connect(const char *servername, int port) @@ -698,7 +684,7 @@ /* Create connection between client and server. * We do it by exchanging data over a TCP socket connection. */ - my_dest.lid = pp_get_local_lid(ib_dev, ib_port); + my_dest.lid = pp_get_local_lid(ctx, ib_port); my_dest.qpn = ctx->qp->qp_num; my_dest.psn = lrand48() & 0xffffff; if (!my_dest.lid) { @@ -758,10 +744,7 @@ scnt = 0; rcnt = 0; ccnt = 0; - if (servername) - client_first_post = 1; - else - client_first_post = 0; + client_first_post = (servername != 0); poll_buf = ctx->poll_buf; post_buf = ctx->post_buf; @@ -815,7 +798,7 @@ } if (wc.status != IBV_WC_SUCCESS) { fprintf(stderr, "Completion wth error at %s:\n", - servername?"client":"server"); + servername ? "client" : "server"); fprintf(stderr, "Failed status %d: wr_id %d\n", wc.status, (int) wc.wr_id); fprintf(stderr, "scnt=%d, rcnt=%d, ccnt=%d\n", From iod00d at hp.com Wed May 18 11:19:53 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 18 May 2005 11:19:53 -0700 Subject: [openib-general] Re: [PATCH][0/19] kDAPL: More cleanups in dat.h In-Reply-To: <20050518082023.GC25713@mellanox.co.il> References: <11161816352658@sun.com> <521x85i51l.fsf@topspin.com> <20050518082023.GC25713@mellanox.co.il> Message-ID: <20050518181953.GB6004@esmail.cup.hp.com> On Wed, May 18, 2005 at 11:20:23AM +0300, Michael S. Tsirkin wrote: > > The assumption that pointers can be safely cast to and from unsigned > > long is used all over the Linux kernel. In other words, use "unsigned > > long" instead of "uintptr_t". > > > > - R. > > Right. Its OK because kernel requires gcc. Well, the kernel requires gcc can be used. Ie no instrinsics that gcc doesn't understand. ia64 kernel can be built using icc (Intel) compiler as well. grant From blist at aon.at Wed May 18 11:24:48 2005 From: blist at aon.at (Bernhard Fischer) Date: Wed, 18 May 2005 20:24:48 +0200 Subject: [openib-general] Re: diff-perftest-07 replace pp_get_local_lid() In-Reply-To: <20050518180620.GA6004@esmail.cup.hp.com> References: <20050518013821.GO1078@esmail.cup.hp.com> <20050518135130.GE25713@mellanox.co.il> <20050518180620.GA6004@esmail.cup.hp.com> Message-ID: <20050518182448.GC7553@aon.at> On Wed, May 18, 2005 at 11:06:20AM -0700, Grant Grundler wrote: >On Wed, May 18, 2005 at 04:51:30PM +0300, Michael S. Tsirkin wrote: >> > I'm thinking about how to keep the server running and iterating. >> > The goal is to be able to run a sequence of tests just >> > from the client side. >> >> I just do ssh server $PWD/rdma_lat & ( sleep 1 && $PWD/rdma_lat) > >hrm...that would work too though it's not as intuitive. Would only work partly.. try: ssh server $PWD/rdma_lat & ( sleep 1 && $PWD/rdma_lat -s 0 or -1) and your server process will just hang. I know.. don't do that then (checking if size<1 may just paper over missing checks in verbs. i don't know). From iod00d at hp.com Wed May 18 11:29:23 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 18 May 2005 11:29:23 -0700 Subject: [openib-general] Gen2 test suites In-Reply-To: References: <1116400089.3796.268.camel@localhost> Message-ID: <20050518182923.GC6004@esmail.cup.hp.com> On Wed, May 18, 2005 at 01:25:15AM -0600, Stephen Poole wrote: > We have been working on the test harness that we discussed at Sonoma. > It is not ready for prime time, but it is a large effort within LANL. > It is to be used for other regression test suites. As soon as it is > "cleaned up" and ready for prime time, it will be made available. Opensource is about making the developement process visible - not the end product. That's where "commit early, commit often" phrase comes from. > Currently the POC for this for IB is Alfred Torrez > (atorrez at lanl.gov). This is a multi-divison project, so it will still > be a while before it can be released. Just a heads-up. Please consider using openib.org SVN or some other public repository to do the developement. It will make it easier for other potential partners to grab your very latest bits and give them a whirl. And if they don't work, report bugs (hopefully with patches). I hope this isn't just about developing a test framework - enough of those exist under GPL or other open source license. e.g. The one Bernhard Fischer pointed out is here: http://www.codesourcery.com/qmtest/ But there are lots of other ones from OSDL and on sourceforge. grant From xma at us.ibm.com Wed May 18 11:27:49 2005 From: xma at us.ibm.com (Shirley Ma) Date: Wed, 18 May 2005 18:27:49 +0000 Subject: [openib-general] Gen2 test suites In-Reply-To: <1116400089.3796.268.camel@localhost> Message-ID: >Sandia, LANL, and LLNL are very interested in an automated test suite > for OpenIB. Sandia and LANL have four IB clusters that could be used > for running the test suite. These mid-sized cluster testbeds range > between 128-256 nodes [ 138 (x86), 128 (EM64T), 128 (x86), and 256 > (Opteron) ]. I have a few PPC970 nodes we could use as well. An > automated way to do nightly functionality and performance testing of > OpenIB is a great way to provide feedback to developers and other folks > in the OpenIB community. Great! > What are you using the run the tests? Your own scripts? Something >else? First step is to use my own scripts to do the nightly build. I will do more research to see whether any available automation infrastructure could help. Appreciate any sugguestions. Thanks Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From iod00d at hp.com Wed May 18 11:33:31 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 18 May 2005 11:33:31 -0700 Subject: [openib-general] Vapi Install In-Reply-To: <428AEFEC.601@int-evry.fr> References: <428AEFEC.601@int-evry.fr> Message-ID: <20050518183331.GD6004@esmail.cup.hp.com> On Wed, May 18, 2005 at 09:34:04AM +0200, Ouissem BEN FREDJ wrote: > Hello, > > Some questions about IB packages. > > How can I install just the Infiniband HCA Driver and the VAPI package? > How can I do that using the "IB Gold" software (http://www.mellanox.com) > > Is the "Infiniband Subnet Manager" mandatory? Can I configure statically > the subnet? > > PS: Im using 2.4.21-20.EL kernel with x86_64 architecture. You might start with the FAQ: http://www.openib.org/docs/openib_faq.txt Sounds like you want to contact whichever vendor you purchased the cards from for support. grant From blist at aon.at Wed May 18 11:35:47 2005 From: blist at aon.at (Bernhard Fischer) Date: Wed, 18 May 2005 20:35:47 +0200 Subject: [openib-general] Re: [openib-commits] r2384 - gen2/trunk/src/userspace/management/libibmad/src In-Reply-To: <20050518181445.9B3742283D9@openib.ca.sandia.gov> References: <20050518181445.9B3742283D9@openib.ca.sandia.gov> Message-ID: <20050518183546.GD7553@aon.at> On Wed, May 18, 2005 at 11:14:45AM -0700, halr at openib.org wrote: >Author: halr >Date: 2005-05-18 11:14:44 -0700 (Wed, 18 May 2005) >New Revision: 2384 > >Modified: > gen2/trunk/src/userspace/management/libibmad/src/sa.c Hal, I think this typo does also occur in gen2/trunk/src/userspace/management/osm/include/iba/ib_types.h Thank you, >Log: >Fix typo in #define >-#define IB_PR_COMPMASK_PFTLIFETIME (1ull<<21) >+#define IB_PR_COMPMASK_PKTLIFETIME (1ull<<21) From mst at mellanox.co.il Wed May 18 11:44:36 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 18 May 2005 21:44:36 +0300 Subject: [openib-general] Re: Re: diff-perftest-07 replace pp_get_local_lid() In-Reply-To: <20050518182448.GC7553@aon.at> References: <20050518013821.GO1078@esmail.cup.hp.com> <20050518135130.GE25713@mellanox.co.il> <20050518180620.GA6004@esmail.cup.hp.com> <20050518182448.GC7553@aon.at> Message-ID: <20050518184436.GA27680@mellanox.co.il> Quoting r. Bernhard Fischer : > Subject: Re: Re: diff-perftest-07 replace pp_get_local_lid() > > On Wed, May 18, 2005 at 11:06:20AM -0700, Grant Grundler wrote: > >On Wed, May 18, 2005 at 04:51:30PM +0300, Michael S. Tsirkin wrote: > > >> > I'm thinking about how to keep the server running and iterating. > >> > The goal is to be able to run a sequence of tests just > >> > from the client side. > >> > >> I just do ssh server $PWD/rdma_lat & ( sleep 1 && $PWD/rdma_lat) > > > >hrm...that would work too though it's not as intuitive. > > Would only work partly.. Sorry, you need ssh server $PWD/rdma_lat & ( sleep 1 && $PWD/rdma_lat server) > try: > ssh server $PWD/rdma_lat & ( sleep 1 && $PWD/rdma_lat -s 0 or -1) > and your server process will just hang. > I know.. don't do that then (checking if size<1 may just paper over > missing checks in verbs. i don't know). > Size 0 doesnt make sense for this test since I'm polling on data: no data = nothing to poll upon. -s 1 should work - I'll check tomorrow. Please remember you must pass same flags to server and client. -- MST - Michael S. Tsirkin From blist at aon.at Wed May 18 11:56:31 2005 From: blist at aon.at (Bernhard Fischer) Date: Wed, 18 May 2005 20:56:31 +0200 Subject: [openib-general] Re: Re: diff-perftest-07 replace pp_get_local_lid() In-Reply-To: <20050518184436.GA27680@mellanox.co.il> References: <20050518013821.GO1078@esmail.cup.hp.com> <20050518135130.GE25713@mellanox.co.il> <20050518180620.GA6004@esmail.cup.hp.com> <20050518182448.GC7553@aon.at> <20050518184436.GA27680@mellanox.co.il> Message-ID: <20050518185631.GE7553@aon.at> On Wed, May 18, 2005 at 09:44:36PM +0300, Michael S. Tsirkin wrote: >Quoting r. Bernhard Fischer : >> Subject: Re: Re: diff-perftest-07 replace pp_get_local_lid() >> >> On Wed, May 18, 2005 at 11:06:20AM -0700, Grant Grundler wrote: >> >hrm...that would work too though it's not as intuitive. >> >> Would only work partly.. > >Sorry, you need > > ssh server $PWD/rdma_lat & ( sleep 1 && $PWD/rdma_lat server) > >> try: >> ssh server $PWD/rdma_lat & ( sleep 1 && $PWD/rdma_lat -s 0 or -1) >> and your server process will just hang. >> I know.. don't do that then (checking if size<1 may just paper over >> missing checks in verbs. i don't know). >> > >Size 0 doesnt make sense for this test since I'm polling on data: >no data = nothing to poll upon. If size 0 and -1 (not 1 as you seem to have read) are illegal (they obviously are for malloc) you should check if(size <1){usage();exit(1);} Same for the other arguments. I'll send a patchlet if noone did fix that by next week. >-s 1 should work - I'll check tomorrow. -s 1 did work for me last time i looked. In fact, back then only 1 worked, i got errors for bigger values i tried (42,128,256,512,1k,2k,4k,16k,32k,64k,128k,42M etc). From halr at voltaire.com Wed May 18 12:02:00 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 18 May 2005 15:02:00 -0400 Subject: [openib-general] Re: [openib-commits] r2384 - gen2/trunk/src/userspace/management/libibmad/src In-Reply-To: <20050518183546.GD7553@aon.at> References: <20050518181445.9B3742283D9@openib.ca.sandia.gov> <20050518183546.GD7553@aon.at> Message-ID: <1116442920.4479.3.camel@localhost.localdomain> On Wed, 2005-05-18 at 14:35, Bernhard Fischer wrote: > On Wed, May 18, 2005 at 11:14:45AM -0700, halr at openib.org wrote: > >Author: halr > >Date: 2005-05-18 11:14:44 -0700 (Wed, 18 May 2005) > >New Revision: 2384 > > > >Modified: > > gen2/trunk/src/userspace/management/libibmad/src/sa.c > > Hal, > I think this typo does also occur in > gen2/trunk/src/userspace/management/osm/include/iba/ib_types.h > > Thank you, > >Log: > >Fix typo in #define > > >-#define IB_PR_COMPMASK_PFTLIFETIME (1ull<<21) > >+#define IB_PR_COMPMASK_PKTLIFETIME (1ull<<21) Thanks. I just changed that one too. -- Hal From mst at mellanox.co.il Wed May 18 12:07:52 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 18 May 2005 22:07:52 +0300 Subject: [openib-general] Re: diff-perftest-07 replace pp_get_local_lid() In-Reply-To: <20050518180620.GA6004@esmail.cup.hp.com> References: <20050518013821.GO1078@esmail.cup.hp.com> <20050518135130.GE25713@mellanox.co.il> <20050518180620.GA6004@esmail.cup.hp.com> Message-ID: <20050518190752.GA27421@mellanox.co.il> Quoting r. Grant Grundler : > Subject: Re: diff-perftest-07 replace pp_get_local_lid() > > On Wed, May 18, 2005 at 04:51:30PM +0300, Michael S. Tsirkin wrote: > > Quoting r. Grant Grundler : > > > Following patch to rdma_lat.c: > > > o replaces pp_get_local_lid with code from ibv_pingpong. > > > This calls into libibverbs instead of fishing around in /sys FS. > > > > > > o makes two minor white space fix-ups. > > > > > > Signed-off-by: Grant Grundler > > > > No patch was attached :( > > Doh! sorry...appended now > > > > I'd like to slowly restructure main() into multiple distinct parts: > > > 1) parameter parsing/setting > > > 2) global data init (e.g. srand()) > > > 3) setup connection > > > 3) negotiate test+parameters with server/client > > > 4) run test (maybe several iterations with different params) > > > 5) exit/cleanup > > > > Good, but lets pass data around in function parameters > > instead of in global data section. > > Certainly. There should be very little/no global data > when I'm done. srand/errno type stuff should be about it. > > > > I'm thinking about how to keep the server running and iterating. > > > The goal is to be able to run a sequence of tests just > > > from the client side. > > > > I just do ssh server $PWD/rdma_lat & ( sleep 1 && $PWD/rdma_lat) > > hrm...that would work too though it's not as intuitive. > > > > Or is this a waste of time? > > > Should I rather be looking at fixing up netperf to support IB? > > > > That may be kind of hard, given that uverbs API is completely different > > from socket API. > > Agreed - but that's not the part I'm trying to leverage. > I want: > o statistical infrastructure. > o automatically run multiple iterations until confidence level is reached > o CPU utilization (aka "Service Demand") > o bind task to a particular CPU > > I can add those things to rmda_lat but it seems like it's a wash > to add IB support to netperf. > > > And since netperf.org is pretty well > known and no-one has a clue about rdma_lat..it seems like a more > obvious place to do the work. I guess that's a political > argument for doing the work in netperf. I'll think about it more > and talk to the netperf maintainer later this week. > > thanks, > grant Fair enough - the more the merrier. I'd like to re-iterate what I am trying to do currently: - show how to utilize uverbs efficiently - get a benchmark useful e.g. for hardware tuning and/or testing For both these goals its very important for the test to be simple and readable- netperf sure is not it. I started with latency and am looking at bandwidth now, it seems these may be part of the same test - but CPU utilization may be harder. -- MST - Michael S. Tsirkin From mshefty at ichips.intel.com Wed May 18 12:22:36 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Wed, 18 May 2005 12:22:36 -0700 Subject: [openib-general] CM private data Message-ID: <428B95FC.9000803@ichips.intel.com> Do any applications make use of the private data in these CM message: RTU, MRA, or DREP? I'm doubtful of the RTU or DREP, but not as sure of the MRA. Since no replies are generated in response to these messages, the CM does not keep them after their sends complete. However, it may need to resend the messages. For example, it will resend the RTU if a duplicate REP is received. If no applications are using the private data, I will not worry about storing the private data for the retransmissions at this time. - Sean From itamar at mellanox.co.il Wed May 18 12:44:36 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Wed, 18 May 2005 22:44:36 +0300 Subject: [openib-general] RE: [PATCH][kdapl] fix spin_lock_irqsave/spin_unlock_irqrestore i mplementation Message-ID: <91DB792C7985D411BEC300B40080D29CC35B46@mtvex01.mtv.mtl.com> "evd producer locking" is something that we need in openib kdapl as it was in Source Forge implementation. If the evd gets more than one event stream we need a lock to prevent corruption of the evd, when 2 streams will try to add event at the same time The reason for the lock being taken in one function and release in the othere can be found directly in the remark before the functions /* * Event posting code follows. */ /* * These next two functions (dapl_evd_get_event and dapl_evd_post_event) * are a pair. They are always called together, from one of the functions * at the end of this file (dapl_evd_post_*_event). * * Note that if producer side locking is enabled, the first one takes the * EVD lock and the second releases it. */ Itamar > -----Original Message----- > From: James Lentini [mailto:jlentini at netapp.com] > Sent: Wednesday, May 18, 2005 7:42 PM > To: Itamar > Cc: openib-general > Subject: Re: [PATCH][kdapl] fix > spin_lock_irqsave/spin_unlock_irqrestore > implementation > > > > Wow, I didn't realize that we did this in the code. > > Can we simplify this? I'd rather not have the code taking a lock in > one function and releasing it in another. This code came from the > generic source forge implementation. Do we need to do > "evd producer locking" for OpenIB? If not we can remove these locks. > > james > > On Wed, 18 May 2005, Itamar wrote: > > itamar> when spin_lock_irqsave and spin_unlock_irqrestore are > not called in the same function > itamar> need to pass the flags form the save the store > itamar> > itamar> Signed-off-by: Itamar Rabenstein > itamar> > itamar> Index: dapl_evd_util.c > itamar> > =================================================================== > itamar> --- dapl_evd_util.c (revision 2374) > itamar> +++ dapl_evd_util.c (working copy) > itamar> @@ -378,20 +378,19 @@ > itamar> * that the lock is held. > itamar> */ > itamar> > itamar> -static struct dat_event *dapl_evd_get_event(DAPL_EVD > * evd_ptr) > itamar> +static struct dat_event *dapl_evd_get_event(DAPL_EVD > * evd_ptr, unsigned long *flags) > itamar> { > itamar> struct dat_event *event; > itamar> - unsigned long flags; > itamar> > itamar> if (evd_ptr->evd_producer_locking_needed) { > itamar> - spin_lock_irqsave(&evd_ptr->header.lock, flags); > itamar> + > spin_lock_irqsave(&evd_ptr->header.lock, *flags); > itamar> } > itamar> > itamar> event = (struct dat_event *) > dapl_rbuf_remove(&evd_ptr->free_event_queue); > itamar> > itamar> /* Release the lock if it was taken and the > call failed. */ > itamar> if (!event && evd_ptr->evd_producer_locking_needed) { > itamar> - > spin_unlock_irqrestore(&evd_ptr->header.lock, flags); > itamar> + > spin_unlock_irqrestore(&evd_ptr->header.lock, *flags); > itamar> } > itamar> > itamar> return event; > itamar> @@ -406,10 +405,10 @@ > itamar> */ > itamar> > itamar> static void dapl_evd_post_event(DAPL_EVD *evd_ptr, > itamar> - const struct dat_event > *event_ptr) > itamar> + const struct dat_event > *event_ptr, > itamar> + unsigned long flags) > itamar> { > itamar> u32 dat_status; > itamar> - unsigned long flags; > itamar> DAPL_CNO * cno_to_trigger = NULL; > itamar> > itamar> dapl_dbg_log(DAPL_DBG_TYPE_EVD, > itamar> @@ -459,7 +458,7 @@ > itamar> DAPL_EVD * overflow_evd_ptr) > itamar> { > itamar> struct dat_event *overflow_event; > itamar> - > itamar> + unsigned long flags; > itamar> /* The overflow_evd_ptr mght be the same as evd. > itamar> * In that case we've got a catastrophic overflow. > itamar> */ > itamar> @@ -469,7 +468,7 @@ > itamar> return; > itamar> } > itamar> > itamar> - overflow_event = dapl_evd_get_event(overflow_evd_ptr); > itamar> + overflow_event = > dapl_evd_get_event(overflow_evd_ptr, &flags); > itamar> if (!overflow_event) { > itamar> /* this is not good */ > itamar> overflow_evd_ptr->catastrophic_overflow = TRUE; > itamar> @@ -477,17 +476,18 @@ > itamar> return; > itamar> } > itamar> > dapl_evd_format_overflow_event(overflow_evd_ptr, overflow_event); > itamar> - dapl_evd_post_event(overflow_evd_ptr, overflow_event); > itamar> + dapl_evd_post_event(overflow_evd_ptr, > overflow_event, flags); > itamar> > itamar> return; > itamar> } > itamar> > itamar> static struct dat_event > *dapl_evd_get_and_init_event(DAPL_EVD *evd_ptr, > itamar> - enum dat_event_number evno) > itamar> + enum dat_event_number evno, > itamar> + > unsigned long *flags) > itamar> { > itamar> struct dat_event *event_ptr; > itamar> > itamar> - event_ptr = dapl_evd_get_event(evd_ptr); > itamar> + event_ptr = dapl_evd_get_event(evd_ptr, flags); > itamar> if (!event_ptr) > itamar> > dapl_evd_post_overflow_event(evd_ptr->header.owner_ia-> > itamar> async_error_evd, evd_ptr); > itamar> @@ -507,7 +507,8 @@ > itamar> DAT_CR_HANDLE cr_handle) > itamar> { > itamar> struct dat_event *event_ptr; > itamar> - event_ptr = > dapl_evd_get_and_init_event(evd_ptr, event_number); > itamar> + unsigned long flags; > itamar> + event_ptr = > dapl_evd_get_and_init_event(evd_ptr, event_number, &flags); > itamar> /* > itamar> * Note event lock may be held on successful return > itamar> * to be released by dapl_evd_post_event(), if > provider side locking > itamar> @@ -525,7 +526,7 @@ > itamar> > event_ptr->event_data.cr_arrival_event_data.conn_qual = conn_qual; > itamar> > event_ptr->event_data.cr_arrival_event_data.cr_handle = cr_handle; > itamar> > itamar> - dapl_evd_post_event(evd_ptr, event_ptr); > itamar> + dapl_evd_post_event(evd_ptr, event_ptr, flags); > itamar> > itamar> return DAT_SUCCESS; > itamar> } > itamar> @@ -537,7 +538,8 @@ > itamar> void *private_data) > itamar> { > itamar> struct dat_event *event_ptr; > itamar> - event_ptr = > dapl_evd_get_and_init_event(evd_ptr, event_number); > itamar> + unsigned long flags; > itamar> + event_ptr = > dapl_evd_get_and_init_event(evd_ptr, event_number, &flags); > itamar> /* > itamar> * Note event lock may be held on successful return > itamar> * to be released by dapl_evd_post_event(), if > provider side locking > itamar> @@ -554,7 +556,7 @@ > itamar> = private_data_size; > itamar> > event_ptr->event_data.connect_event_data.private_data = private_data; > itamar> > itamar> - dapl_evd_post_event(evd_ptr, event_ptr); > itamar> + dapl_evd_post_event(evd_ptr, event_ptr, flags); > itamar> > itamar> return DAT_SUCCESS; > itamar> } > itamar> @@ -564,7 +566,8 @@ > itamar> DAT_IA_HANDLE ia_handle) > itamar> { > itamar> struct dat_event *event_ptr; > itamar> - event_ptr = > dapl_evd_get_and_init_event(evd_ptr, event_number); > itamar> + unsigned long flags; > itamar> + event_ptr = > dapl_evd_get_and_init_event(evd_ptr, event_number, &flags); > itamar> /* > itamar> * Note event lock may be held on successful return > itamar> * to be released by dapl_evd_post_event(), if > provider side locking > itamar> @@ -579,7 +582,7 @@ > itamar> > event_ptr->event_data.asynch_error_event_data.dat_handle = > itamar> (DAT_HANDLE) ia_handle; > itamar> > itamar> - dapl_evd_post_event(evd_ptr, event_ptr); > itamar> + dapl_evd_post_event(evd_ptr, event_ptr, flags); > itamar> > itamar> return DAT_SUCCESS; > itamar> } > itamar> @@ -589,7 +592,8 @@ > itamar> void *pointer) > itamar> { > itamar> struct dat_event *event_ptr; > itamar> - event_ptr = > dapl_evd_get_and_init_event(evd_ptr, event_number); > itamar> + unsigned long flags; > itamar> + event_ptr = > dapl_evd_get_and_init_event(evd_ptr, event_number, &flags); > itamar> /* > itamar> * Note event lock may be held on successful return > itamar> * to be released by dapl_evd_post_event(), if > provider side locking > itamar> @@ -603,7 +607,7 @@ > itamar> > itamar> > event_ptr->event_data.software_event_data.pointer = pointer; > itamar> > itamar> - dapl_evd_post_event(evd_ptr, event_ptr); > itamar> + dapl_evd_post_event(evd_ptr, event_ptr, flags); > itamar> > itamar> return DAT_SUCCESS; > itamar> } > itamar> > itamar> -- > itamar> Itamar > itamar> > From ftillier at silverstorm.com Wed May 18 12:27:33 2005 From: ftillier at silverstorm.com (Fab Tillier) Date: Wed, 18 May 2005 12:27:33 -0700 Subject: [openib-general] CM private data In-Reply-To: <428B95FC.9000803@ichips.intel.com> Message-ID: <000201c55bdf$a4c0f1b0$9b5aa8c0@infiniconsys.com> > From: Sean Hefty [mailto:mshefty at ichips.intel.com] > Sent: Wednesday, May 18, 2005 12:23 PM > > Do any applications make use of the private data in these CM message: RTU, > MRA, or DREP? I'm doubtful of the RTU or DREP, but not as sure of the MRA. > > Since no replies are generated in response to these messages, the CM does > not keep them after their sends complete. However, it may need to resend > the messages. For example, it will resend the RTU if a duplicate REP is > received. > > If no applications are using the private data, I will not worry about > storing the private data for the retransmissions at this time. If you're not going to store them, I would suggest removing the private data for the associated calls (RTU, MRA, DREP). That way it becomes very clear to applications wishing to use the private data that they can't. - Fab From mshefty at ichips.intel.com Wed May 18 12:32:06 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Wed, 18 May 2005 12:32:06 -0700 Subject: [openib-general] CM private data In-Reply-To: <000201c55bdf$a4c0f1b0$9b5aa8c0@infiniconsys.com> References: <000201c55bdf$a4c0f1b0$9b5aa8c0@infiniconsys.com> Message-ID: <428B9836.1050808@ichips.intel.com> Fab Tillier wrote: > If you're not going to store them, I would suggest removing the private data > for the associated calls (RTU, MRA, DREP). That way it becomes very clear > to applications wishing to use the private data that they can't. That makes it more difficult to add in the support later, since it involves an API change and would effect all clients. As long as no one is using the private data yet, then the implementation of storing the private data can just be deferred until it is needed. - Sean From jlentini at netapp.com Wed May 18 12:49:12 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 18 May 2005 15:49:12 -0400 (EDT) Subject: [openib-general] CM private data In-Reply-To: <428B9836.1050808@ichips.intel.com> References: <000201c55bdf$a4c0f1b0$9b5aa8c0@infiniconsys.com> <428B9836.1050808@ichips.intel.com> Message-ID: On Wed, 18 May 2005, Sean Hefty wrote: > Fab Tillier wrote: >> If you're not going to store them, I would suggest removing the private >> data >> for the associated calls (RTU, MRA, DREP). That way it becomes very clear >> to applications wishing to use the private data that they can't. > > That makes it more difficult to add in the support later, since it involves > an API change and would effect all clients. As long as no one is using the > private data yet, then the implementation of storing the private data can > just be deferred until it is needed. If future CM users aren't aware of this, they are bound use this feature and encounter hard to find bugs. Did you consider using asserts to detect when clients uses unimplemented parts of the API? james From jcarr at linuxmachines.com Wed May 18 13:00:13 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Wed, 18 May 2005 13:00:13 -0700 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <20050518020816.GP1078@esmail.cup.hp.com> References: <42891E31.9000908@linuxmachines.com> <52u0l2lrfb.fsf@topspin.com> <42894241.1010908@linuxmachines.com> <20050517014728.GI30200@esmail.cup.hp.com> <428954D7.3060003@linuxmachines.com> <20050517043250.GK30200@esmail.cup.hp.com> <428A9B36.4040901@linuxmachines.com> <20050518020816.GP1078@esmail.cup.hp.com> Message-ID: <428B9ECD.5040201@linuxmachines.com> Grant Grundler wrote: >>root at jcarr:/# dd if=/dev/shm/test of=/dev/null bs=4K >>196608+0 records in >>196608+0 records out >>805306368 bytes transferred in 0.628504 seconds (1281306571 bytes/sec) > > > Yeah. Sounds like there is. Should be able to do several GB/s like that. > > I suppose it's possibly an issue with the Memory controller too. > ZX1 interleaves accesses across 4 DIMMs to get the memory bandwidth. > Might check to make sure your box is "optimally" configured too. > > grundler at iota$ dd if=/dev/shm/test of=/dev/null bs=4K > dd: opening `/dev/shm/test': No such file or directory I just did dd if=/dev/zero of=test bs=1M count=768 I assume that the speed would be the same if there was non-zero data :) I only have 2GB of ram. > Sorry - what do I need to do to create /dev/shm/test? > > I should probably "cheat" and use 16KB block since that > is the native page size on ia64. "cheating" in that sense is good. 16KB blocks will probably increase the performance a lot. I was even thinking I would try to do that on my Xeon system. (I don't know if that's possible.) Jeff From jcarr at linuxmachines.com Wed May 18 13:38:48 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Wed, 18 May 2005 13:38:48 -0700 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <20050518020816.GP1078@esmail.cup.hp.com> References: <42891E31.9000908@linuxmachines.com> <52u0l2lrfb.fsf@topspin.com> <42894241.1010908@linuxmachines.com> <20050517014728.GI30200@esmail.cup.hp.com> <428954D7.3060003@linuxmachines.com> <20050517043250.GK30200@esmail.cup.hp.com> <428A9B36.4040901@linuxmachines.com> <20050518020816.GP1078@esmail.cup.hp.com> Message-ID: <428BA7D8.8070108@linuxmachines.com> Grant Grundler wrote: > Wellll..Looks like I'm wrong. Previous email on this thread > suggested it's possible by people who know alot more about > it than I do. But I'm still concerned it's going to affect > latency. Maybe that's one reason why NAPI was made a compile time option? Anyway, it might be just as easy to implement IB netdev->poll() also as a compile time option. >>Isn't that what the midas touch of netdev->poll() really is? e1000 has: >>quit_polling: netif_rx_complete(netdev); >> e1000_irq_enable(adapter); > > > I'm more familiar with tg3 driver. The disadvantage to NAPI in the > tg3 implementation is it *alays* disables interrupts on the card before > calling netif_rx_schedule(). Then it lets the OS decide when to > actually process those packets *already* received in a safer > context....then once tg3 decides it's done all the work > it re-eables the interrupts. Just like e1000 does above. > > There are some workloads where the PCI bus utilization is "suboptimal" > because the enable/disable of interrupts interfers with the DMA flows > and costs excessive overhead. Maybe. I'm not a PCI expert and have only used a PCI bus analyzer a few times. I don't understand how disabling the interrupts would interfere with DMA or add overhead. That condition strikes me as being chip dependent. So, perhaps the tg3 does that, but an IB HCA would be different. (?) Jeff From roland at topspin.com Wed May 18 13:42:21 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 18 May 2005 13:42:21 -0700 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <8A71B368A89016469F72CD08050AD3340911E7@maui.asicdesigners.com> (Felix Marti's message of "Tue, 17 May 2005 21:29:32 -0700") References: <8A71B368A89016469F72CD08050AD3340911E7@maui.asicdesigners.com> Message-ID: <52k6lwfeqa.fsf@topspin.com> Felix> I get just above 5G on RX (goodput, as reported by netperf) Felix> on a single opteron 248 (100%) using standard ethernet MTU Felix> (1500). Felix> TX performance is higher (close to 7G), but it is probably Felix> not the kind of comparison that you're interested in, since Felix> TSO removes the dependency on the wire MTU. Felix> Similarly, the TOE completely shields the host from the Felix> wire MTU (in addition of removing ACK traffic) and I'm Felix> getting 7.5G RX and TX with about 50% CPU utilization Felix> (user/kernel space memory copy!) Felix> These numbers are without NAPI. Thanks very much for posting these numbers. It's interesting to me that you are using a single CPU for benchmarking. How do the numbers compare if you run on an SMP system? Obviously there is more CPU available but there are SMP losses due to cacheline pingpong, "lock" prefix and other locking overhead, etc. Thanks, Roland From tduffy at sun.com Wed May 18 14:03:46 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 18 May 2005 14:03:46 -0700 Subject: [openib-general] [PATCH] kDAPL: add some clarification to a few debug printks Message-ID: <1116450226.11460.2.camel@duffman> I was running into some issues and noticed that these printks were not very clear about where they were coming from. So, adding a little more info. Signed-off-by: Tom Duffy Index: linux-kernel/dat-provider/dapl_openib_qp.c =================================================================== --- linux-kernel/dat-provider/dapl_openib_qp.c (revision 2382) +++ linux-kernel/dat-provider/dapl_openib_qp.c (working copy) @@ -108,7 +108,8 @@ u32 dapl_ib_qp_alloc(DAPL_IA *ia_ptr, DA ep_ptr->qp_handle = ib_create_qp(ib_pd_handle, &qp_attr); if (IS_ERR(ep_ptr->qp_handle)) { ib_status = PTR_ERR(ep_ptr->qp_handle); - dapl_dbg_log(DAPL_DBG_TYPE_ERR, " failed code = %d\n", + dapl_dbg_log(DAPL_DBG_TYPE_ERR, + " dapl_ib_qp_alloc: ib_create_qp failed = %d\n", ib_status); return dapl_ib_status_convert(ib_status); } @@ -197,8 +198,9 @@ ib_cq_handle_t dapl_get_dto_cq(DAPL_IA * ib_status = PTR_ERR(ia_ptr->hca_ptr->ib_trans. null_ib_cq_handle); - dapl_dbg_log(DAPL_DBG_TYPE_ERR, " failed code = %d\n", - ib_status); + dapl_dbg_log(DAPL_DBG_TYPE_ERR, + " dapl_get_dto_cq: ib_create_cq failed " + "= %d\n", ib_status); ia_ptr->hca_ptr->ib_trans.null_ib_cq_handle = IB_INVALID_HANDLE; } From mshefty at ichips.intel.com Wed May 18 14:29:33 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Wed, 18 May 2005 14:29:33 -0700 Subject: [openib-general] CM private data In-Reply-To: References: <000201c55bdf$a4c0f1b0$9b5aa8c0@infiniconsys.com> <428B9836.1050808@ichips.intel.com> Message-ID: <428BB3BD.7050702@ichips.intel.com> James Lentini wrote: > If future CM users aren't aware of this, they are bound use this feature > and encounter hard to find bugs. > > Did you consider using asserts to detect when clients uses unimplemented > parts of the API? I did consider adding a failure if private data is sent in the MRA, DREP, or RTU. But I first want to know if any current applications are using private data for those messages. There is a subtle issue with the current retransmission mechanism that I need to fix that brought this to my attention, and if private data is needed, then I will just fix that now as well. - Sean From tduffy at sun.com Wed May 18 14:31:14 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 18 May 2005 14:31:14 -0700 Subject: [openib-general] [PATCH][TRIVIAL] cm: remove compile warning Message-ID: <1116451874.11460.9.camel@duffman> This patch removes the compile warning: CC [M] drivers/infiniband/core/cm.o /build1/tduffy/openib-work/linux-2.6.12-rc4-openib/drivers/infiniband/core/cm.c: In function ‘cm_issue_rej’: /build1/tduffy/openib-work/linux-2.6.12-rc4-openib/drivers/infiniband/core/cm.c:887: warning: ‘msg’ may be used uninitialized in this function Signed-off-by: Tom Duffy Index: drivers/infiniband/core/cm.c =================================================================== --- drivers/infiniband/core/cm.c (revision 2397) +++ drivers/infiniband/core/cm.c (working copy) @@ -884,7 +884,7 @@ static int cm_issue_rej(struct cm_port * enum cm_msg_response msg_rejected, void *ari, u8 ari_length) { - struct ib_mad_send_buf *msg; + struct ib_mad_send_buf *msg = NULL; struct ib_send_wr *bad_send_wr; struct cm_rej_msg *rej_msg, *rcv_msg; int ret; From mshefty at ichips.intel.com Wed May 18 14:35:52 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Wed, 18 May 2005 14:35:52 -0700 Subject: [openib-general] [PATCH][TRIVIAL] cm: remove compile warning In-Reply-To: <1116451874.11460.9.camel@duffman> References: <1116451874.11460.9.camel@duffman> Message-ID: <428BB538.3030002@ichips.intel.com> Tom Duffy wrote: > This patch removes the compile warning: > > CC [M] drivers/infiniband/core/cm.o > /build1/tduffy/openib-work/linux-2.6.12-rc4-openib/drivers/infiniband/core/cm.c: In function ‘cm_issue_rej’: > /build1/tduffy/openib-work/linux-2.6.12-rc4-openib/drivers/infiniband/core/cm.c:887: warning: ‘msg’ may be used uninitialized in this function > > Signed-off-by: Tom Duffy Thanks, Tom! Applied. - Sean From jcarr at linuxmachines.com Wed May 18 14:41:12 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Wed, 18 May 2005 14:41:12 -0700 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <528y2di70f.fsf@topspin.com> References: <42891E31.9000908@linuxmachines.com> <52u0l2lrfb.fsf@topspin.com> <42894241.1010908@linuxmachines.com> <20050517014728.GI30200@esmail.cup.hp.com> <428954D7.3060003@linuxmachines.com> <20050517043250.GK30200@esmail.cup.hp.com> <428A9B36.4040901@linuxmachines.com> <528y2di70f.fsf@topspin.com> Message-ID: <428BB678.6000505@linuxmachines.com> Roland Dreier wrote: > The most interesting optimization available is implementing the IPoIB > connected mode draft, although I don't think it's as easy as Vivek > indicated -- for example, I'm not sure how to deal with having > different MTUs depending on the destination. Thank you for that reference. I'll read that now. > NAPI isn't really a throughput optimization. It pretty much only > helps with small packet workloads. I'm not so sure. Maybe 2 or 4K == small in the IB case. > Also, it helps the RX path much > more than the TX path, which means it doesn't help much in a symmetric > test like netperf or NPtcp. If you look at the table at the beginning > of Documentation/networking/NAPI_HOWTO.txt, you can see that for e1000 > with 1024 byte packets, the NIC generates 872K RX interrupts for 1M RX > packets -- NAPI doesn't kick in totally until the packet size is down > to 256 bytes. Yes, I'd be willing to guess they did these tests on a fast enough CPU so that by the time they got to the 512B test the CPU was able to handle things. By 1KB, the number of packets they could send got bound by the 1gb cable; the cpu was idle enough that NAPI stopped being triggered and was able to handle 80k interrupts/sec. I don't want to just generate noise here. I just was curious as to the causes of IPoIB performance being CPU bound. > It is possible although somewhat ugly to implement NAPI for IPoIB, and > I actually had hacked up such a patch a while ago. However it didn't > end up helping so I never pursued it. Bummer. > When I profile a system running IPoIB throughput tests, half or more > of the CPU time is going to skb_copy_and_csum() and other parts of the > core kernel's network stack, so our ability to optimize IPoIB is > somewhat limited. I've already dealt with all of the easy > optimization targets from profiling in IPoIB and mthca -- with MSI-X, > we do zero MMIO reads, and Michael Tsirkin has heavily tuned our locking. Oh ya, that would be a problem if IPoIB is generating and verifying a software checksum for each packet. IMHO, from my novice view of IB so far, I don't see why IB didn't build on top of ethernet more. That's one thing I still don't get. Jeff From iod00d at hp.com Wed May 18 14:41:25 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 18 May 2005 14:41:25 -0700 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <428B9ECD.5040201@linuxmachines.com> References: <42891E31.9000908@linuxmachines.com> <52u0l2lrfb.fsf@topspin.com> <42894241.1010908@linuxmachines.com> <20050517014728.GI30200@esmail.cup.hp.com> <428954D7.3060003@linuxmachines.com> <20050517043250.GK30200@esmail.cup.hp.com> <428A9B36.4040901@linuxmachines.com> <20050518020816.GP1078@esmail.cup.hp.com> <428B9ECD.5040201@linuxmachines.com> Message-ID: <20050518214125.GA6704@esmail.cup.hp.com> On Wed, May 18, 2005 at 01:00:13PM -0700, Jeff Carr wrote: > Grant Grundler wrote: > >grundler at iota$ dd if=/dev/shm/test of=/dev/null bs=4K > >dd: opening `/dev/shm/test': No such file or directory > > I just did dd if=/dev/zero of=test bs=1M count=768 I realized later last night that /dev/shm is tmpfs and that I had to create the file myself like you suggested above. > I assume that the speed would be the same if there was non-zero data :) Maybe. I don't know. I've rerun this to get some idea of what is going on: grundler at gsyprf3:~$ dd if=/dev/shm/ggg of=/dev/null bs=4k 255568+0 records in 255568+0 records out 1046806528 bytes transferred in 0.580582 seconds (1803029894 bytes/sec) grundler at gsyprf3:~$ dd if=/dev/shm/ggg of=/dev/null bs=16k 63892+0 records in 63892+0 records out 1046806528 bytes transferred in 0.316682 seconds (3305546209 bytes/sec) grundler at gsyprf3:~$ dd if=/dev/shm/ggg of=/dev/null bs=64k 15973+0 records in 15973+0 records out 1046806528 bytes transferred in 0.275389 seconds (3801192840 bytes/sec) 4K -> 1.8 GB/s 16k -> 3.3 GB/s 64k -> 3.8 GB/s This seems reasonable. IIRC the ZX1 chipset has 6GB/s backplane but one CPU can only drive ~4GB/s. > I only have 2GB of ram. I don't think that's important as long at you have enough available to /dev/shm. More important is how memory is interleaved across the DIMMs and how well the memcpy routines are optimized. You need to physically verify DIMM placement and linux should have highly optimized routines for most architectures today. > >I should probably "cheat" and use 16KB block since that > >is the native page size on ia64. > > "cheating" in that sense is good. 16KB blocks will probably increase the > performance a lot. I was even thinking I would try to do that on my Xeon > system. (I don't know if that's possible.) I don't see why not. It ovbiously helps on the IA64 box. We want to measure the copy speed, not the syscall speed, right? :^) BTW, can you remind me again why this was important to rdma_lat test? It was just to prove the VM/memcopy wasn't the bottleneck, right? grant From iod00d at hp.com Wed May 18 14:51:45 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 18 May 2005 14:51:45 -0700 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <428BA7D8.8070108@linuxmachines.com> References: <42891E31.9000908@linuxmachines.com> <52u0l2lrfb.fsf@topspin.com> <42894241.1010908@linuxmachines.com> <20050517014728.GI30200@esmail.cup.hp.com> <428954D7.3060003@linuxmachines.com> <20050517043250.GK30200@esmail.cup.hp.com> <428A9B36.4040901@linuxmachines.com> <20050518020816.GP1078@esmail.cup.hp.com> <428BA7D8.8070108@linuxmachines.com> Message-ID: <20050518215145.GB6704@esmail.cup.hp.com> On Wed, May 18, 2005 at 01:38:48PM -0700, Jeff Carr wrote: > Maybe. I'm not a PCI expert and have only used a PCI bus analyzer a few > times. I don't understand how disabling the interrupts would interfere > with DMA or add overhead. PCI ordering rules force the PCI Bus controller to service MMIO reads (used to flush MMIO writes) and it may be before a PCI DMA read request has been fully satisfied. Ie disconnect and retry to get the rest of the data. If the driver only did MMIO writes, some chipsets allow DMA reads to bypass MMIO writes (both are going in the same direction) until the PCI device is satisfied or their latency timer has expired. MMIO reads typically only use 4 bytes of 8 byte wide PCI bus and similarly for any system the MMIO read might cross. Ie transaction setup overhead and poor bus width utization are reasons why people want to design cards that don't need MMIO reads. Add the CPU stall time (1000+ cycles per MMIO read) and it's very compelling. MMIO Writes are less of a problem since the only real downside is PCI bus utilization - not so much a problem with DMA flows. > That condition strikes me as being chip dependent. > So, perhaps the tg3 does that, but an IB HCA would be different. (?) Yes. Libor just clarified in previous mail the mechanism IB can use. I don't know how updates to those fields are communicated to the kernel/HCA but I hope it doesn't involve MMIO reads. grant From iod00d at hp.com Wed May 18 15:08:27 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 18 May 2005 15:08:27 -0700 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <528y2di70f.fsf@topspin.com> References: <42891E31.9000908@linuxmachines.com> <52u0l2lrfb.fsf@topspin.com> <42894241.1010908@linuxmachines.com> <20050517014728.GI30200@esmail.cup.hp.com> <428954D7.3060003@linuxmachines.com> <20050517043250.GK30200@esmail.cup.hp.com> <428A9B36.4040901@linuxmachines.com> <528y2di70f.fsf@topspin.com> Message-ID: <20050518220827.GC6704@esmail.cup.hp.com> On Tue, May 17, 2005 at 07:48:32PM -0700, Roland Dreier wrote: ... > NAPI isn't really a throughput optimization. It pretty much only > helps with small packet workloads. In general, yes, I agree. NAPI prevents the NIC from saturating the CPU with interrupts from small packets. It's also one defense against DoS attacks. But one side effect is RX descriptor cachelines don't pingpong as much between IO controller and CPU. NAPI also helps throughput by improving PCI bus utilization. This isn't an issue for GigE unless one has a GigE link on PCI 33Mhz/32-bit PCI bus...in this case RX is obviously heavily favored over TX. (workaround was to reduce the RX descriptor ring size). At least that's what Jamal Hadi, Robert Olsson, and I concluded in a problem we were trying to sort out last year. I don't know where IB and 10GigE are WRT such problems. But I expect people closer to the HW have bus traces and are aware of similar issues. > Also, it helps the RX path much > more than the TX path, which means it doesn't help much in a symmetric > test like netperf or NPtcp. Is netperf TCP_STREAM test symmetric? I know netperf can run symetric workloads (e.g. TCP_RR, 1 byte message) but my understand was web server workloads typically are asymetric (heavily loading TX). And most of the scripts provided with netperf try asymetric parameters. ... > When I profile a system running IPoIB throughput tests, half or more > of the CPU time is going to skb_copy_and_csum() and other parts of the > core kernel's network stack, so our ability to optimize IPoIB is > somewhat limited. I've already dealt with all of the easy > optimization targets from profiling in IPoIB and mthca -- with MSI-X, > we do zero MMIO reads, and Michael Tsirkin has heavily tuned our locking. Ok. thanks, grant From sean.hefty at intel.com Wed May 18 15:30:33 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Wed, 18 May 2005 15:30:33 -0700 Subject: [openib-general] [PATCH] [CM] add MRA handling to CM Message-ID: The following patch adds MRA receive handling to the CM, and fixes a bug where the state checking was reversed when sending the MRA. Note: if the MRA is lost, it will not be resent. That support will come in a separate patch. Signed-off-by: Sean Hefty Index: cm.c =================================================================== --- cm.c (revision 2402) +++ cm.c (working copy) @@ -1712,7 +1712,12 @@ static void cm_format_rej(struct cm_rej_ rej_msg->local_comm_id = 0; cm_rej_set_msg_rejected(rej_msg, CM_MSG_RESPONSE_REQ); break; + case IB_CM_MRA_REQ_SENT: + rej_msg->local_comm_id = cm_id_priv->id.local_id; + cm_rej_set_msg_rejected(rej_msg, CM_MSG_RESPONSE_REQ); + break; case IB_CM_REP_RCVD: + case IB_CM_MRA_REP_SENT: rej_msg->local_comm_id = cm_id_priv->id.local_id; cm_rej_set_msg_rejected(rej_msg, CM_MSG_RESPONSE_REP); break; @@ -1826,7 +1831,9 @@ static struct cm_id_private * cm_acquire cm_id_priv = NULL; } spin_unlock_irqrestore(&cm.lock, flags); - } else + } else if (cm_rej_get_msg_rejected(rej_msg) == CM_MSG_RESPONSE_REQ) + cm_id_priv = cm_acquire_id(rej_msg->remote_comm_id, 0); + else cm_id_priv = cm_acquire_id(rej_msg->remote_comm_id, remote_id); return cm_id_priv; @@ -1849,14 +1856,14 @@ static int cm_rej_handler(struct cm_work spin_lock_irqsave(&cm_id_priv->lock, flags); switch (cm_id_priv->id.state) { case IB_CM_REQ_SENT: + case IB_CM_MRA_REQ_RCVD: case IB_CM_REP_SENT: + case IB_CM_MRA_REP_RCVD: ib_cancel_mad(cm_id_priv->av.port->mad_agent, (unsigned long) cm_id_priv->msg); /* fall through */ case IB_CM_REQ_RCVD: case IB_CM_MRA_REQ_SENT: - case IB_CM_MRA_REQ_RCVD: - case IB_CM_MRA_REP_RCVD: if (rej_msg->reason == IB_CM_REJ_STALE_CONN) cm_enter_timewait(cm_id_priv); else @@ -1957,10 +1964,10 @@ int ib_send_cm_mra(struct ib_cm_id *cm_i service_timeout, private_data, private_data_len); spin_lock_irqsave(&cm_id_priv->lock, flags); - if (cm_id_priv->id.state != IB_CM_REQ_RCVD && - cm_id_priv->id.state != IB_CM_REP_RCVD && - (cm_id_priv->id.state != IB_CM_ESTABLISHED || - cm_id_priv->id.lap_state != IB_CM_LAP_RCVD)) { + if (cm_id_priv->id.state == IB_CM_REQ_RCVD || + cm_id_priv->id.state == IB_CM_REP_RCVD || + (cm_id_priv->id.state == IB_CM_ESTABLISHED && + cm_id_priv->id.lap_state == IB_CM_LAP_RCVD)) { ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent, &msg->send_wr, &bad_send_wr); @@ -1993,11 +2000,80 @@ out: } EXPORT_SYMBOL(ib_send_cm_mra); +static struct cm_id_private * cm_acquire_mraed_id(struct cm_mra_msg *mra_msg) +{ + switch (cm_mra_get_msg_mraed(mra_msg)) { + case CM_MSG_RESPONSE_REQ: + return cm_acquire_id(mra_msg->remote_comm_id, 0); + case CM_MSG_RESPONSE_REP: + case CM_MSG_RESPONSE_OTHER: + return cm_acquire_id(mra_msg->remote_comm_id, + mra_msg->local_comm_id); + default: + return NULL; + } +} + static int cm_mra_handler(struct cm_work *work) { - /* todo: write MRA handler */ - /* todo: add timeout mechanism separate from retries for - receiver of MRA */ + struct cm_id_private *cm_id_priv; + struct cm_mra_msg *mra_msg; + unsigned long flags; + int timeout, ret; + + mra_msg = (struct cm_mra_msg *)work->mad_recv_wc->recv_buf.mad; + cm_id_priv = cm_acquire_mraed_id(mra_msg); + if (!cm_id_priv) + return -EINVAL; + + work->cm_event.private_data = &mra_msg->private_data; + work->cm_event.param.mra_rcvd.service_timeout = + cm_mra_get_service_timeout(mra_msg); + timeout = cm_convert_to_ms(cm_mra_get_service_timeout(mra_msg)); + + spin_lock_irqsave(&cm_id_priv->lock, flags); + switch (cm_id_priv->id.state) { + case IB_CM_REQ_SENT: + if (cm_mra_get_msg_mraed(mra_msg) != CM_MSG_RESPONSE_REQ || + ib_modify_mad(cm_id_priv->av.port->mad_agent, + (unsigned long) cm_id_priv->msg, timeout)) + goto out; + cm_id_priv->id.state = IB_CM_MRA_REQ_RCVD; + break; + case IB_CM_REP_SENT: + if (cm_mra_get_msg_mraed(mra_msg) != CM_MSG_RESPONSE_REP || + ib_modify_mad(cm_id_priv->av.port->mad_agent, + (unsigned long) cm_id_priv->msg, timeout)) + goto out; + cm_id_priv->id.state = IB_CM_MRA_REP_RCVD; + break; + case IB_CM_ESTABLISHED: + if (cm_mra_get_msg_mraed(mra_msg) != CM_MSG_RESPONSE_OTHER || + cm_id_priv->id.lap_state != IB_CM_LAP_SENT || + ib_modify_mad(cm_id_priv->av.port->mad_agent, + (unsigned long) cm_id_priv->msg, timeout)) + goto out; + cm_id_priv->id.lap_state = IB_CM_MRA_LAP_RCVD; + break; + default: + goto out; + } + + cm_id_priv->msg->context[1] = (void *) (unsigned long) + cm_id_priv->id.state; + ret = atomic_inc_and_test(&cm_id_priv->work_count); + if (!ret) + list_add_tail(&work->list, &cm_id_priv->work_list); + spin_unlock_irqrestore(&cm_id_priv->lock, flags); + + if (ret) + cm_process_work(cm_id_priv, work); + else + cm_deref_id(cm_id_priv); + return 0; +out: + spin_unlock_irqrestore(&cm_id_priv->lock, flags); + cm_deref_id(cm_id_priv); return -EINVAL; } @@ -2557,19 +2633,21 @@ static void cm_process_send_error(struct memset(&cm_event, 0, sizeof cm_event); cm_id_priv = msg->context[0]; - state = (enum ib_cm_state) (unsigned long) msg->context[1]; /* Discard old sends or ones without a response. */ spin_lock_irqsave(&cm_id_priv->lock, flags); + state = (enum ib_cm_state) (unsigned long) msg->context[1]; if (msg != cm_id_priv->msg || state != cm_id_priv->id.state) goto discard; switch (state) { case IB_CM_REQ_SENT: + case IB_CM_MRA_REQ_RCVD: cm_reset_to_idle(cm_id_priv); cm_event.event = IB_CM_REQ_ERROR; break; case IB_CM_REP_SENT: + case IB_CM_MRA_REP_RCVD: cm_reset_to_idle(cm_id_priv); cm_event.event = IB_CM_REP_ERROR; break; From mshefty at ichips.intel.com Wed May 18 15:35:53 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Wed, 18 May 2005 15:35:53 -0700 Subject: [openib-general] [PATCH] CM cmpost utility fixes In-Reply-To: <78d18e205051715204b101435@mail.gmail.com> References: <78d18e20505171513e71fa39@mail.gmail.com> <78d18e205051715204b101435@mail.gmail.com> Message-ID: <428BC349.2070906@ichips.intel.com> William Jordan wrote: > Sean, > You mentioned that the cm_id leak was fixed, but I don't see a patch > for it, so I'm > submitting one. > > Fixed leak of cm_ids in server side of cmpost. Fixed an Oops if number > of connections > on server and client side don't match. > > Signed-off-by: Bill Jordan When I try to apply this patch, most of it is rejected. - Sean From iod00d at hp.com Wed May 18 15:43:06 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 18 May 2005 15:43:06 -0700 Subject: [openib-general] Re: diff-perftest-07 replace pp_get_local_lid() In-Reply-To: <20050518190752.GA27421@mellanox.co.il> References: <20050518013821.GO1078@esmail.cup.hp.com> <20050518135130.GE25713@mellanox.co.il> <20050518180620.GA6004@esmail.cup.hp.com> <20050518190752.GA27421@mellanox.co.il> Message-ID: <20050518224306.GD6704@esmail.cup.hp.com> On Wed, May 18, 2005 at 10:07:52PM +0300, Michael S. Tsirkin wrote: ... > > obvious place to do the work. I guess that's a political > > argument for doing the work in netperf. I'll think about it more > > and talk to the netperf maintainer later this week. > > Fair enough - the more the merrier. Rick Jones thinks it should be straight forward to add netperf_ib.c. I'll work with him to update his silly HPUX box to a more silly parisc-linux box so we can run subversion. Then I can start "borrowing" code from rmda_lat to do the same things using netperf infrastructure. > I'd like to re-iterate what I am trying to do currently: > - show how to utilize uverbs efficiently > - get a benchmark useful e.g. for hardware tuning and/or testing > > For both these goals its very important for the test to be simple > and readable - netperf sure is not it. Yes, I agree with both the goals and rdma_lat covers those pretty well already. I think netperf is more interesting for more complex stuff since it has slightly different goals and serves a different purpose. (e.g. binding tasks to CPUs and reporting statistics) I'll finish up cleaning up rmda_lat and then use it as an example (like you intended) to add IB functionality into netperf. > I started with latency and am looking at bandwidth now, it seems these > may be part of the same test If the two tests can share code, I'd rather see two example tests and parts of rdma_lat.c split off into a seperate file. I expect all the socket handling and context setup stuff could be shared. The more bells and whistles you add to rdma_lat, the further it gets away from your primary goal (simple, efficient example). >- but CPU utilization may be harder. Isn't the use of get_cycles() a sufficient form of CPU utilization for the two goals above? Or is there more kernel overhead we aren't catching somehow? grant From roland at topspin.com Wed May 18 16:04:14 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 18 May 2005 16:04:14 -0700 Subject: [openib-general] Re: [RFC] [PATCH] user_mad: Support RMPP on send side In-Reply-To: <1115845653.4490.188.camel@localhost.localdomain> (Hal Rosenstock's message of "11 May 2005 17:07:33 -0400") References: <1115845653.4490.188.camel@localhost.localdomain> Message-ID: <5264xgf85t.fsf@topspin.com> This looks OK to check in with one small comment on the following: - if (copy_to_user(buf, &packet->mad, sizeof packet->mad)) + if (copy_to_user(buf, &packet->mad, + min(count, packet->length + + sizeof (struct ib_user_mad)))) ret = -EFAULT; else - ret = sizeof packet->mad; + ret = count; This code will truncate a received MAD that is bigger than the buffer passed into read(), but return the full size of the packet. I don't think read() is allowed to do this: the return value can be at most the count value passed in by the user. I think we have two options: truncate and return the actual amount of data read to the user, or return an error if the user's buffer is too small. - R. From mshefty at ichips.intel.com Wed May 18 16:08:10 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Wed, 18 May 2005 16:08:10 -0700 Subject: [openib-general] [PATCH] CM cmpost utility fixes In-Reply-To: <428BC349.2070906@ichips.intel.com> References: <78d18e20505171513e71fa39@mail.gmail.com> <78d18e205051715204b101435@mail.gmail.com> <428BC349.2070906@ichips.intel.com> Message-ID: <428BCADA.1040605@ichips.intel.com> Sean Hefty wrote: > William Jordan wrote: > >> Sean, >> You mentioned that the cm_id leak was fixed, but I don't see a patch >> for it, so I'm >> submitting one. >> >> Fixed leak of cm_ids in server side of cmpost. Fixed an Oops if number >> of connections >> on server and client side don't match. >> >> Signed-off-by: Bill Jordan > > > When I try to apply this patch, most of it is rejected. Don't worry about this... I'm just going to merge in the changes by hand. Thanks. - Sean From caitlin.bestler at gmail.com Wed May 18 16:15:30 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Wed, 18 May 2005 16:15:30 -0700 Subject: [openib-general] CM private data In-Reply-To: <428B95FC.9000803@ichips.intel.com> References: <428B95FC.9000803@ichips.intel.com> Message-ID: <469958e005051816153ec9759f@mail.gmail.com> I believe that an IT-API it_ep_accept() supplies private data that it expects to be delivered to its peer when the three-way handshake option is selected. DAT only exposes a two-way handshake, so there it never requires private data on the RTU. I don't know if any IT-API applications actually require the three-way handshake. On 5/18/05, Sean Hefty wrote: > Do any applications make use of the private data in these CM message: RTU, > MRA, or DREP? I'm doubtful of the RTU or DREP, but not as sure of the MRA. > > Since no replies are generated in response to these messages, the CM does > not keep them after their sends complete. However, it may need to resend > the messages. For example, it will resend the RTU if a duplicate REP is > received. > > If no applications are using the private data, I will not worry about > storing the private data for the retransmissions at this time. > > - Sean > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From tduffy at sun.com Wed May 18 16:49:32 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 18 May 2005 16:49:32 -0700 Subject: [openib-general] [PATCH] kDAPL: return is not a function Message-ID: <1116460172.8031.2.camel@duffman> return is not a function Signed-off-by: Tom Duffy Index: linux-kernel-return/dat-provider/dapl_cookie.c =================================================================== --- linux-kernel-return/dat-provider/dapl_cookie.c (revision 2382) +++ linux-kernel-return/dat-provider/dapl_cookie.c (working copy) @@ -134,9 +134,9 @@ u32 dapl_cb_create(DAPL_COOKIE_BUFFER *b buffer->pool[i].ep = ep; } - return (DAT_SUCCESS); + return DAT_SUCCESS; } else { - return (DAT_INSUFFICIENT_RESOURCES); + return DAT_INSUFFICIENT_RESOURCES; } } Index: linux-kernel-return/dat-provider/dapl_ring_buffer_util.c =================================================================== --- linux-kernel-return/dat-provider/dapl_ring_buffer_util.c (revision 2382) +++ linux-kernel-return/dat-provider/dapl_ring_buffer_util.c (working copy) @@ -237,7 +237,7 @@ void *dapl_rbuf_remove(DAPL_RING_BUFFER if (val == pos) { pos = (pos + 1) & rbuf->lim; /* verify in range */ - return (rbuf->base[pos]); + return rbuf->base[pos]; } } Index: linux-kernel-return/dat-provider/dapl_lmr_util.c =================================================================== --- linux-kernel-return/dat-provider/dapl_lmr_util.c (revision 2382) +++ linux-kernel-return/dat-provider/dapl_lmr_util.c (working copy) @@ -50,7 +50,7 @@ DAPL_LMR *dapl_lmr_alloc(DAPL_IA * ia, /* Allocate LMR */ lmr = (DAPL_LMR *) kmalloc(sizeof(DAPL_LMR), GFP_ATOMIC); if (NULL == lmr) { - return (NULL); + return NULL; } /* zero the structure */ @@ -80,7 +80,7 @@ DAPL_LMR *dapl_lmr_alloc(DAPL_IA * ia, lmr->param.mem_priv = mem_priv; atomic_set(&lmr->lmr_ref_count, 0); - return (lmr); + return lmr; } void dapl_lmr_dealloc(DAPL_LMR * lmr) Index: linux-kernel-return/dat-provider/dapl_hca_util.c =================================================================== --- linux-kernel-return/dat-provider/dapl_hca_util.c (revision 2382) +++ linux-kernel-return/dat-provider/dapl_hca_util.c (working copy) @@ -83,7 +83,7 @@ DAPL_HCA *dapl_hca_alloc(char *name, str } } - return (hca_ptr); + return hca_ptr; } /* Index: linux-kernel-return/dat-provider/dapl_cr_util.c =================================================================== --- linux-kernel-return/dat-provider/dapl_cr_util.c (revision 2382) +++ linux-kernel-return/dat-provider/dapl_cr_util.c (working copy) @@ -44,7 +44,7 @@ DAPL_CR *dapl_cr_alloc(DAPL_IA * ia_ptr) /* Allocate EP */ cr_ptr = (DAPL_CR *) kmalloc(sizeof(DAPL_CR), GFP_ATOMIC); if (cr_ptr == NULL) { - return (NULL); + return NULL; } /* zero the structure */ @@ -62,7 +62,7 @@ DAPL_CR *dapl_cr_alloc(DAPL_IA * ia_ptr) dapl_llist_init_entry(&cr_ptr->header.ia_list_entry); spin_lock_init(&cr_ptr->header.lock); - return (cr_ptr); + return cr_ptr; } /* Index: linux-kernel-return/dat-provider/dapl_ia_util.c =================================================================== --- linux-kernel-return/dat-provider/dapl_ia_util.c (revision 2382) +++ linux-kernel-return/dat-provider/dapl_ia_util.c (working copy) @@ -65,7 +65,7 @@ DAPL_IA *dapl_ia_alloc(struct dat_provid /* Allocate IA */ ia_ptr = (DAPL_IA *) kmalloc(sizeof(DAPL_IA), GFP_ATOMIC); if (ia_ptr == NULL) { - return (NULL); + return NULL; } /* zero the structure */ @@ -100,7 +100,7 @@ DAPL_IA *dapl_ia_alloc(struct dat_provid dapl_hca_link_ia(hca_ptr, ia_ptr); - return (ia_ptr); + return ia_ptr; } /* Index: linux-kernel-return/dat-provider/dapl_pz_util.c =================================================================== --- linux-kernel-return/dat-provider/dapl_pz_util.c (revision 2382) +++ linux-kernel-return/dat-provider/dapl_pz_util.c (working copy) @@ -59,7 +59,7 @@ DAPL_PZ *dapl_pz_alloc(DAPL_IA * ia) /* Allocate PZ */ pz = (DAPL_PZ *) kmalloc(sizeof(DAPL_PZ), GFP_ATOMIC); if (NULL == pz) { - return (NULL); + return NULL; } /* zero the structure */ @@ -83,7 +83,7 @@ DAPL_PZ *dapl_pz_alloc(DAPL_IA * ia) */ atomic_set(&pz->pz_ref_count, 0); - return (pz); + return pz; } /* Index: linux-kernel-return/dat-provider/dapl_openib_cm.c =================================================================== --- linux-kernel-return/dat-provider/dapl_openib_cm.c (revision 2382) +++ linux-kernel-return/dat-provider/dapl_openib_cm.c (working copy) @@ -844,7 +844,7 @@ u32 dapl_ib_cm_remote_addr(DAT_HANDLE da rc = ib_cm_get_remote_gid(cr_ptr->ib_cm_handle, (u8 *) & remote_gid); if (rc < 0) - return (rc); + return rc; if (!memcmp(&ia_ptr->hca_ptr->ib_trans.gid, &remote_gid, 16)) { memcpy(remote_ia_address, &ia_ptr->hca_ptr->hca_address, Index: linux-kernel-return/dat-provider/dapl_openib_util.c =================================================================== --- linux-kernel-return/dat-provider/dapl_openib_util.c (revision 2382) +++ linux-kernel-return/dat-provider/dapl_openib_util.c (working copy) @@ -334,7 +334,7 @@ u32 dapl_ib_pd_alloc(DAPL_IA *ia, DAPL_P } pz->pd_handle = (ib_pd_handle_t) pd; - return (DAT_SUCCESS); + return DAT_SUCCESS; } /* @@ -367,7 +367,7 @@ u32 dapl_ib_pd_free(DAPL_PZ *pz) } pz->pd_handle = IB_INVALID_HANDLE; - return (DAT_SUCCESS); + return DAT_SUCCESS; } /* @@ -393,7 +393,7 @@ u32 dapl_ib_mr_register(DAPL_IA *ia, DAP u64 length, enum dat_mem_priv_flags privileges) { /* ITBD virtual memory registration ??? */ - return (DAT_SUCCESS); + return DAT_SUCCESS; } u32 dapl_ib_mr_register_physical(DAPL_IA *ia_ptr, DAPL_LMR *lmr, @@ -452,7 +452,7 @@ u32 dapl_ib_mr_register_physical(DAPL_IA dapl_dbg_log(DAPL_DBG_TYPE_UTIL, "dapl_ib_mr_register_physical(%p %d) got lkey 0x%x \n", buf_list, length, lmr->param.lmr_context); - return (DAT_SUCCESS); + return DAT_SUCCESS; } /* @@ -486,7 +486,7 @@ u32 dapl_ib_mr_deregister(DAPL_LMR *lmr) lmr->param.lmr_context = 0; lmr->mr_handle = IB_INVALID_HANDLE; - return (DAT_SUCCESS); + return DAT_SUCCESS; } /* @@ -523,7 +523,7 @@ u32 dapl_ib_mw_alloc(DAPL_RMR *rmr) rmr->param.rmr_context = mw->rkey; dapl_dbg_log(DAPL_DBG_TYPE_UTIL, "created mw_handle = %p, r_key = 0x%x\n", mw, mw->rkey); - return (DAT_SUCCESS); + return DAT_SUCCESS; } /* @@ -556,7 +556,7 @@ u32 dapl_ib_mw_free(DAPL_RMR *rmr) rmr->param.rmr_context = 0; rmr->mw_handle = IB_INVALID_HANDLE; - return (DAT_SUCCESS); + return DAT_SUCCESS; } /* @@ -601,13 +601,13 @@ u32 dapl_ib_mw_bind(DAPL_RMR *rmr, DAPL_ if (ib_status < 0) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, "ib_bind_mw error code return = %d\n", ib_status); - return (dapl_ib_status_convert(ib_status)); + return dapl_ib_status_convert(ib_status); } rmr->param.rmr_context = (DAT_RMR_CONTEXT) rmr->mw_handle->rkey; dapl_dbg_log(DAPL_DBG_TYPE_UTIL, " new_rkey = 0x%x\n", rmr->mw_handle->rkey); - return (DAT_SUCCESS); + return DAT_SUCCESS; } /* @@ -643,13 +643,13 @@ u32 dapl_ib_mw_unbind(DAPL_RMR *rmr, DAP if (ib_status < 0) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, " ib_bind_mw error code return = %d\n", ib_status); - return (dapl_ib_status_convert(ib_status)); + return dapl_ib_status_convert(ib_status); } rmr->param.rmr_context = (DAT_RMR_CONTEXT) rmr->mw_handle->rkey; dapl_dbg_log(DAPL_DBG_TYPE_UTIL, " new_rkey = 0x%x\n", rmr->mw_handle->rkey); - return (DAT_SUCCESS); + return DAT_SUCCESS; } /* @@ -709,7 +709,7 @@ u32 dapl_ib_query_hca(DAPL_HCA *hca_ptr, dapl_dbg_log(DAPL_DBG_TYPE_ERR, " ib_query_device error code return = %d\n", ib_status); - return (dapl_ib_status_convert(ib_status)); + return dapl_ib_status_convert(ib_status); } /* ITBD need to review the attr */ Index: linux-kernel-return/dat-provider/dapl_rmr_util.c =================================================================== --- linux-kernel-return/dat-provider/dapl_rmr_util.c (revision 2382) +++ linux-kernel-return/dat-provider/dapl_rmr_util.c (working copy) @@ -35,7 +35,7 @@ DAPL_RMR *dapl_rmr_alloc(DAPL_PZ * pz) /* Allocate LMR */ rmr = (DAPL_RMR *) kmalloc(sizeof(DAPL_RMR), GFP_ATOMIC); if (NULL == rmr) { - return (NULL); + return NULL; } /* zero the structure */ @@ -68,7 +68,7 @@ DAPL_RMR *dapl_rmr_alloc(DAPL_PZ * pz) rmr->pz = pz; rmr->lmr = NULL; - return (rmr); + return rmr; } void dapl_rmr_dealloc(DAPL_RMR * rmr) Index: linux-kernel-return/dat-provider/dapl_sp_util.c =================================================================== --- linux-kernel-return/dat-provider/dapl_sp_util.c (revision 2382) +++ linux-kernel-return/dat-provider/dapl_sp_util.c (working copy) @@ -66,7 +66,7 @@ DAPL_SP *dapl_sp_alloc(DAPL_IA * ia_ptr, /* Allocate EP */ sp_ptr = (DAPL_SP *) kmalloc(sizeof(DAPL_SP), GFP_ATOMIC); if (sp_ptr == NULL) { - return (NULL); + return NULL; } /* zero the structure */ @@ -94,7 +94,7 @@ DAPL_SP *dapl_sp_alloc(DAPL_IA * ia_ptr, */ dapl_llist_init_head(&sp_ptr->cr_list_head); - return (sp_ptr); + return sp_ptr; } /* Index: linux-kernel-return/dat-provider/dapl_llist.c =================================================================== --- linux-kernel-return/dat-provider/dapl_llist.c (revision 2382) +++ linux-kernel-return/dat-provider/dapl_llist.c (working copy) @@ -198,7 +198,7 @@ void *dapl_llist_remove_head(DAPL_LLIST_ first->flink = NULL; first->blink = NULL; first->list_head = NULL; - return (first->data); + return first->data; } /* @@ -224,7 +224,7 @@ void *dapl_llist_remove_tail(DAPL_LLIST_ last->blink = NULL; last->list_head = NULL; - return (last->data); + return last->data; } /* @@ -274,7 +274,7 @@ void *dapl_llist_remove_entry(DAPL_LLIST entry->flink = NULL; entry->blink = NULL; - return (entry->data); + return entry->data; } /* @@ -287,7 +287,7 @@ void *dapl_llist_peek_head(DAPL_LLIST_HE dapl_os_assert(!dapl_llist_is_empty(head)); first = *head; - return (first->data); + return first->data; } /* @@ -310,7 +310,7 @@ void *dapl_llist_next_entry(DAPL_LLIST_H return NULL; } } - return (next->data); + return next->data; } #ifdef DAPL_DBG From jcarr at linuxmachines.com Wed May 18 16:54:10 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Wed, 18 May 2005 16:54:10 -0700 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <20050518214125.GA6704@esmail.cup.hp.com> References: <42891E31.9000908@linuxmachines.com> <52u0l2lrfb.fsf@topspin.com> <42894241.1010908@linuxmachines.com> <20050517014728.GI30200@esmail.cup.hp.com> <428954D7.3060003@linuxmachines.com> <20050517043250.GK30200@esmail.cup.hp.com> <428A9B36.4040901@linuxmachines.com> <20050518020816.GP1078@esmail.cup.hp.com> <428B9ECD.5040201@linuxmachines.com> <20050518214125.GA6704@esmail.cup.hp.com> Message-ID: <428BD5A2.1070308@linuxmachines.com> Grant Grundler wrote: > 4K -> 1.8 GB/s > 16k -> 3.3 GB/s > 64k -> 3.8 GB/s > > This seems reasonable. > IIRC the ZX1 chipset has 6GB/s backplane but one CPU can only drive ~4GB/s. I have a E7501. Thanks for running this test. I'd not looked so closely at this before or been up to the wall against it where it matters. 1K -> .5 GB/s 4K -> 1.2 GB/s 16K -> 1.7 GB/s 32K -> 1.8 GB/s 64K -> 1.9 GB/s 128K -> 1.9 GB/s 256K -> 1.8 GB/s 512K -> 1.7 GB/s 1M -> 1.2 GB/s 2M -> .7 GB/s > I don't see why not. > It ovbiously helps on the IA64 box. > We want to measure the copy speed, not the syscall speed, right? :^) Maybe someday I'll define PAGE_SHIFT to 14 and see if it boots. ia64 does something with KERNEL_STACK_SIZE_ORDER too. Anyway, this is OT to IB. > BTW, can you remind me again why this was important to rdma_lat test? I didn't bring it up for that purpose; I was just speaking in general IB terms. > It was just to prove the VM/memcopy wasn't the bottleneck, right? Ya, I was just investigating these details after I noticed that raw memory copy numbers were not *too* far away from how far IB is supposed to be. Jeff From mshefty at ichips.intel.com Wed May 18 17:04:48 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Wed, 18 May 2005 17:04:48 -0700 Subject: [openib-general] CM private data In-Reply-To: <469958e005051816153ec9759f@mail.gmail.com> References: <428B95FC.9000803@ichips.intel.com> <469958e005051816153ec9759f@mail.gmail.com> Message-ID: <428BD820.1080908@ichips.intel.com> Caitlin Bestler wrote: > I believe that an IT-API it_ep_accept() supplies private > data that it expects to be delivered to its peer when the > three-way handshake option is selected. > > DAT only exposes a two-way handshake, so there it never > requires private data on the RTU. > > I don't know if any IT-API applications actually require the > three-way handshake. There may also be some lustre related implementations that make use of the private data in the RTU, so it sounds like this may be needed. - Sean From jcarr at linuxmachines.com Wed May 18 17:11:18 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Wed, 18 May 2005 17:11:18 -0700 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <20050517102734.A645@topspin.com> References: <42891E31.9000908@linuxmachines.com> <20050516153316.A32217@topspin.com> <4289249A.2070609@linuxmachines.com> <20050517102734.A645@topspin.com> Message-ID: <428BD9A6.5060808@linuxmachines.com> Libor Michalek wrote: >>OK. Well I would rather make something generic. Besides, wasn't there >>some MS patent issue? The last thread on the subject that I read kinda >>made it sound like you were going to look into the issue and respond. >>Maybe I missed the response; there's a lot of mail in the archives for >>this list. > > Sorry, I discussed it at the openib.org workshop. The short answer is > that after going through the code we didn't feel that the license was > necessary. The question is whether there is any code which relies on MS > patents, aside from trying to prove a negative, it does not appear to > be the case. Cool. I tried to find this supposed patent or patent application but couldn't. If you happen to know what number it is I would be interesting to read it also. It never found someone post the actual number in the archives. >>In any case, I'd rather just make something more generic that perhaps >>can be used with NBD. > > > I wasn't trying to discourage you, just let you know what existing code > is doing, either as a solution, or as an example. I'll take a look at it then. I guess I figured if I was going to end up looking at some patent infected mindset/solution then I might as well go about trying to do what I'm trying to do and maybe it will be a novel approach or based of some other older idea that isn't patented. Yes, I hate software patents that much. Anyway, thanks for the heads up. Jeff From iod00d at hp.com Wed May 18 17:56:37 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 18 May 2005 17:56:37 -0700 Subject: [openib-general] [PATCH] rewrite perftest/README Message-ID: <20050519005637.GD7557@esmail.cup.hp.com> Michael, Here's a complete rewrite of the README file. Should make it easier for people to understand o how to build o run o interpret results I'd still like to add abit more about statisical significance of the sample size of 1000 but am just "refreshing my memory" (ok, lame excuse, I've forgotten everything from 20years ago :^) on how to do that. Maybe you can craft something based on your experience plus the observations below? Ditching the last two (extreme) readings from the "server" side of the histogram: o standard deviation 86 cycles o "arithmetic mean" (average) 7135 cycles o median 7126 cycles o min 6906 cycles (sorted sample #1) o max 7490 cycles (sorted sample #997) (For the record, #998 is 8798 and #999 is 50305, we are clearly measuring something else here too) (1.5Ghz IA64, PCI-X, 2.6.11, forgot which SVN they are running) The median value is *very* reproducible on this configuration. +- 1 cycle consistently over 5 runs of rdma_lat. thanks, grant Signed-off-by: Grant Grundler Index: perftest/README =================================================================== --- perftest/README (revision 2404) +++ perftest/README (working copy) @@ -1,33 +1,84 @@ -This directory includes gen2 uverbs microbenchmarks - -useful as usage examples and for performance tuning. +This directory includes gen2 uverbs microbenchmarks. -Testing methodology: -- CPU clock instruction is used to get CPU clock without context switch. -- Median (as opposed to average) result is reported. The median is less - sensitive to extreme scores. An option to report the full result - distribution for alternative statistical analysis is provided. +The tests are intend as: + 1) simple, efficient usage examples. + This code is GPL. Copy it literally only if you are + releasing GPL products. Otherwise use it as a guide only. -Architectures supported: -- i686, x86_64 + 2) a useful benchmark + e.g. for HW or SW tuning and/or functional testing. + Please post results/observations to openib-general email list. + See www.openib.org "Contact Us" link for details. -Tests in this directory: + +Testing methodology +------------------- + +- uses CPU cycle counter to get time stamps without context switch. + Some arches do NOT implement get_cycle(). e.g. Intel 80486 or older PPC. + +- measures round-trip time but reports half of that as one-way latency. + ie. May not be sufficiently accurate for asymetrical configurations. + +- Min/Median/Max result is always reported. + The median (vs average) is less sensitive to extreme scores. + Typically the "Max" value is the first value measured. + +- larger samples only marginally help. The default (1000) is pretty good. + Note that an array of cycles_t (typically unsigned long) is allocated + once to collect samples and again to store the difference between them. + Really big sample sizes (e.g. 1 million) might see other problems + with the program. + +- "-H" option will dump the histogram for additional statistical analysis. + See xgraph, ygraph, r-base (http://www.r-project.org/), pspp, or other + statistical math programs. + +Architectures tested: i686, x86_64, ia64 + + + +Test Descriptions +----------------- rdma_lat.c - latency test with RDMA write transactions -Building: + +Build Tests +----------- "make" to build all tests -Running: + Debian: build-dep on linux-kernel-headers (for asm/timex.h file) + build-dep on libibverbs-dev + depends on libibverbs1 + + +Run Tests +--------- + +Prerequisites: + kernel 2.6 + ib_uverbs (kernel module) matches libibverbs + ("match" means binary compatible, but ideally same SVN rev) + Debian: dpkg -i libibverbs1_0.1.0-1_ia64.deb + +Server: ./ +Client: ./ + + o IMPORTANT: The SAME OPTIONS must be passed to both server and client. + o "--help" will list the current . + o is IPv4 or IPv6 address assigned to the IPoIB port. -Server: - ./ -Client: - ./ +First Config IPoIB on both client and server with something like: + modprobe ib_mthca msi_x=1 # assumes platform supports MSI-X + modprobe ib_ipoib + ifconfig ib0 10.0.1.30 netmask 255.255.255.0 + modprobe ib_uverbs -Run - ./ --help +Then (e.g.) "rdma_lat -UC | less" on the server side. -For a list of options. Same options must be passed to both server and client. +Run modprobe/ifconfig again on the client side (with a different IB). +Lastly "rmda_lat -UC 10.0.1.31" on the client. -MST +rmda_lat will exit on both server and client after printing results. From iod00d at hp.com Wed May 18 17:57:42 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 18 May 2005 17:57:42 -0700 Subject: [openib-general] perftest updated In-Reply-To: <20050518154643.GL25713@mellanox.co.il> References: <20050518154643.GL25713@mellanox.co.il> Message-ID: <20050519005742.GE7557@esmail.cup.hp.com> On Wed, May 18, 2005 at 06:46:43PM +0300, Michael S. Tsirkin wrote: > > Hi! > I have made some further cleanups: > - summary format cleaned up somewhat, for better usability > and so I can grep for the values I want more easily ah ok. I didn't realize you had that use. > - report options now isolated in a structure of their own > thats the way I plan to go down with other variables ok > - report now done in the report function, unifying mhz/cycles format Yes - this looks better than the original code. thanks, grant From felix at chelsio.com Wed May 18 20:00:15 2005 From: felix at chelsio.com (Felix Marti) Date: Wed, 18 May 2005 20:00:15 -0700 Subject: [openib-general] How about ib_send_page() ? Message-ID: <8A71B368A89016469F72CD08050AD334067778@maui.asicdesigners.com> Hi Roland, define SMP :) at these rates, system architecture comes into place, i.e. on opteron (numa) platforms i get 7.5G (with toe) when rx/tx on cpu0 (on the tyan motherboard, the pci-x tunnel is connected to cpu0). when i run the same test on cpu1, goodput degrades to 7.3G. similarly, running netperf rr tests, i get just below 10us user space latency when running on cpu0 and about 12us when running cpu1. xeon platforms don't show this difference (smp), but throughput and latency are always a bit worse. yes, cacheline pingponging can really affect performance, but we try to avoid it as much as we can in the driver. once i have a bit more time, i'm gonna do more performance work and benchmarks on MP, which includes are more detailed look at napi. felix ________________________________ From: Roland Dreier [mailto:roland at topspin.com] Sent: Wed 5/18/2005 1:42 PM To: Felix Marti Cc: Jeff Carr; openib-general at openib.org Subject: Re: [openib-general] How about ib_send_page() ? Felix> I get just above 5G on RX (goodput, as reported by netperf) Felix> on a single opteron 248 (100%) using standard ethernet MTU Felix> (1500). Felix> TX performance is higher (close to 7G), but it is probably Felix> not the kind of comparison that you're interested in, since Felix> TSO removes the dependency on the wire MTU. Felix> Similarly, the TOE completely shields the host from the Felix> wire MTU (in addition of removing ACK traffic) and I'm Felix> getting 7.5G RX and TX with about 50% CPU utilization Felix> (user/kernel space memory copy!) Felix> These numbers are without NAPI. Thanks very much for posting these numbers. It's interesting to me that you are using a single CPU for benchmarking. How do the numbers compare if you run on an SMP system? Obviously there is more CPU available but there are SMP losses due to cacheline pingpong, "lock" prefix and other locking overhead, etc. Thanks, Roland From support at 4allmemory.com Wed May 18 20:18:58 2005 From: support at 4allmemory.com (support at 4allmemory.com) Date: Wed, 18 May 2005 22:18:58 -0500 Subject: [openib-general] Thank you for your email inquiry Message-ID: <585519671_62999648@web01.waytechnology.com> *** THIS IS AN AUTO-RESPONDER / AN AUTOMATED REPLY FROM WWW.4ALLMEMORY.COM *** Thank you for your email. We are dedicated to responding to our customers in a timely manner. We handle email messages in the order in which they are received, and we reply to most messages within 24 business hours. Have you checked out our Customer Service page? You will find order tracking and shipping information, as well as answers to some of the most commonly asked questions about our products and company at: http://www.4allmemory.com/service Our normal operating hours are: Monday - Friday: 8:00am - 8:00pm CST If your matter is urgent, please call our office at (800)566-3727. We appreciate your business! Customer Support Department www.4AllMemory.Com 800-566-3727 636-825-6555 636-225-7475 Fax Providing Outstanding Customer Service Everyday! From halr at voltaire.com Wed May 18 21:16:59 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 19 May 2005 00:16:59 -0400 Subject: [openib-general] Re: [RFC] [PATCH] user_mad: Support RMPP on send side In-Reply-To: <5264xgf85t.fsf@topspin.com> References: <1115845653.4490.188.camel@localhost.localdomain> <5264xgf85t.fsf@topspin.com> Message-ID: <1116476016.4479.128.camel@localhost.localdomain> On Wed, 2005-05-18 at 19:04, Roland Dreier wrote: > This looks OK to check in with one small comment on the following: > > - if (copy_to_user(buf, &packet->mad, sizeof packet->mad)) > + if (copy_to_user(buf, &packet->mad, > + min(count, packet->length + > + sizeof (struct ib_user_mad)))) > ret = -EFAULT; > else > - ret = sizeof packet->mad; > + ret = count; > > This code will truncate a received MAD that is bigger than the buffer > passed into read(), but return the full size of the packet. I don't > think read() is allowed to do this: the return value can be at most > the count value passed in by the user. Is this really true ? Sean's idea for the receive side for RMPP was to return the buffer size needed if the buffer supplied was too small. > I think we have two options: truncate and return the actual amount of > data read to the user, or return an error if the user's buffer is too > small. Now that you've pointed this out, is it really useful to truncate the buffer ? It seems like an error if the user buffer is too small for the packet received. Can a user do anything useful with a partial packet ? -- Hal From paul.baxter at dsl.pipex.com Wed May 18 22:32:32 2005 From: paul.baxter at dsl.pipex.com (Paul Baxter) Date: Thu, 19 May 2005 06:32:32 +0100 Subject: [openib-general] [PATCH] rewrite perftest/README References: <20050519005637.GD7557@esmail.cup.hp.com> Message-ID: <004801c55c34$29e893d0$8000000a@blorp> Grant, Michael Great work! I wanted to point out a recent thread on comp.arch discussing the merits of using the geometric mean instead of the arithmetic mean as a basis for sampling the timing of a population.My statistics is a bit rusty, but calculating and presenting GM might be useful a well? http://groups.google.co.uk/group/comp.arch/browse_thread/thread/3f5a9ed1d79ed726/416e58b5e48c1715?q=geometric&rnum=6&hl=en#416e58b5e48c1715 [comp.arch, 'SPEC use of Geometric Mean', May 13th, 2005, John Mashey ] From mst at mellanox.co.il Wed May 18 22:51:26 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Thu, 19 May 2005 08:51:26 +0300 Subject: [openib-general] Re: diff-perftest-07 replace pp_get_local_lid() In-Reply-To: <20050518135130.GE25713@mellanox.co.il> References: <20050518013821.GO1078@esmail.cup.hp.com> <20050518135130.GE25713@mellanox.co.il> Message-ID: <20050519055126.GA4079@mellanox.co.il> Quoting r. Michael S. Tsirkin : > Subject: Re: diff-perftest-07 replace pp_get_local_lid() > > Should I rather be looking at fixing up netperf to support IB? > > > > thanks, > > grant > > > > That may be kind of hard, given that uverbs API is completely different > from socket API. Wait, isnt this what SDP is doing? -- MST - Michael S. Tsirkin From mst at mellanox.co.il Thu May 19 00:20:07 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Thu, 19 May 2005 10:20:07 +0300 Subject: [openib-general] Re: [PATCH] rewrite perftest/README In-Reply-To: <20050519005637.GD7557@esmail.cup.hp.com> References: <20050519005637.GD7557@esmail.cup.hp.com> Message-ID: <20050519072007.GO25713@mellanox.co.il> Quoting r. Grant Grundler : > Subject: [PATCH] rewrite perftest/README > > Michael, > Here's a complete rewrite of the README file. > Should make it easier for people to understand > o how to build > o run > o interpret results > > > I'd still like to add abit more about statisical significance > of the sample size of 1000 but am just "refreshing my memory" > (ok, lame excuse, I've forgotten everything from 20years ago :^) > on how to do that. Maybe you can craft something based on your > experience plus the observations below? > > Ditching the last two (extreme) readings from the "server" side > of the histogram: > o standard deviation 86 cycles > o "arithmetic mean" (average) 7135 cycles > o median 7126 cycles > o min 6906 cycles (sorted sample #1) > o max 7490 cycles (sorted sample #997) > > (For the record, #998 is 8798 and #999 is 50305, we are clearly > measuring something else here too) > > (1.5Ghz IA64, PCI-X, 2.6.11, forgot which SVN they are running) > > The median value is *very* reproducible on this configuration. > +- 1 cycle consistently over 5 runs of rdma_lat. > > > thanks, > grant > > Signed-off-by: Grant Grundler Thanks, checked in. -- MST - Michael S. Tsirkin From mst at mellanox.co.il Thu May 19 00:21:40 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Thu, 19 May 2005 10:21:40 +0300 Subject: [openib-general] Re: diff-perftest-07 replace pp_get_local_lid() In-Reply-To: <20050518013821.GO1078@esmail.cup.hp.com> References: <20050518013821.GO1078@esmail.cup.hp.com> Message-ID: <20050519072140.GP25713@mellanox.co.il> Quoting r. Grant Grundler : > Subject: diff-perftest-07 replace pp_get_local_lid() > > Hi Michael, > > Following patch to rdma_lat.c: > o replaces pp_get_local_lid with code from ibv_pingpong. > This calls into libibverbs instead of fishing around in /sys FS. > > o makes two minor white space fix-ups. > > Signed-off-by: Grant Grundler > > > I'd like to slowly restructure main() into multiple distinct parts: > 1) parameter parsing/setting > 2) global data init (e.g. srand()) > 3) setup connection > 3) negotiate test+parameters with server/client > 4) run test (maybe several iterations with different params) > 5) exit/cleanup > > I'm thinking about how to keep the server running and iterating. > The goal is to be able to run a sequence of tests just > from the client side. > > Or is this a waste of time? > Should I rather be looking at fixing up netperf to support IB? > > thanks, > grant > thanks, checked in. -- MST - Michael S. Tsirkin From mst at mellanox.co.il Thu May 19 00:23:47 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Thu, 19 May 2005 10:23:47 +0300 Subject: [openib-general] Re: [RFC] [PATCH] user_mad: Support RMPP on send side In-Reply-To: <5264xgf85t.fsf@topspin.com> References: <1115845653.4490.188.camel@localhost.localdomain> <5264xgf85t.fsf@topspin.com> Message-ID: <20050519072347.GQ25713@mellanox.co.il> Quoting r. Roland Dreier : > Subject: Re: [RFC] [PATCH] user_mad: Support RMPP on send side > > This looks OK to check in with one small comment on the following: > > - if (copy_to_user(buf, &packet->mad, sizeof packet->mad)) > + if (copy_to_user(buf, &packet->mad, > + min(count, packet->length + > + sizeof (struct ib_user_mad)))) > ret = -EFAULT; > else > - ret = sizeof packet->mad; > + ret = count; > > This code will truncate a received MAD that is bigger than the buffer > passed into read(), but return the full size of the packet. I don't > think read() is allowed to do this: the return value can be at most > the count value passed in by the user. > > I think we have two options: truncate and return the actual amount of > data read to the user, or return an error if the user's buffer is too > small. > > - R. > If you truncate, how will the user know the MAD was truncated? -- MST - Michael S. Tsirkin From mst at mellanox.co.il Thu May 19 01:12:38 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Thu, 19 May 2005 11:12:38 +0300 Subject: [openib-general] sdp_link.c info_list locking question Message-ID: <20050519081238.GA4717@mellanox.co.il> Libor, I'm looking at sdp_link.c, and I dont see any lock protecting the info_list linked list. What prevents sdp_path_info_lookup from being called while sdp_path_info_create or sdp_path_info_destroy is in progress? Thanks, -- MST - Michael S. Tsirkin From mst at mellanox.co.il Thu May 19 01:52:08 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Thu, 19 May 2005 11:52:08 +0300 Subject: [openib-general] registering read-only memory Message-ID: <20050519085208.GC4717@mellanox.co.il> Roland, the following code snippet: const char foo[]="Michael Tsirkin"; ibv_reg_mr(ctx->pd, foo, strlen(foo), 0); exposes two problems with ibv_reg_mr: 1. Compiling this code I get a warning: warning: passing arg 2 of `ibv_reg_mr' discards qualifiers from pointer target type. Same if foo is declared volatile. I suggest changing ibv_reg_mr to accept const volatile void * as a second parameter. const is OK since ibv_reg_mr never actually writes to the buffer. volatile is needed by some applications to prevent the compiler from assuming it can re-order accesses to this buffer. Patch attached (below). 2. ibv_reg_mr fails. Why is that? System details: >gcc --version gcc (GCC) 3.3.3 (SuSE Linux) >uname -a Linux swlab156 2.6.11-openib #29 SMP Mon Apr 18 16:17:51 IDT 2005 x86_64 x86_64 x86_64 GNU/Linux Make ibv_reg_mr accept buffer as volatile const *. Signed-off-by: Michael S. Tsirkin Index: libibverbs/include/infiniband/verbs.h =================================================================== --- libibverbs/include/infiniband/verbs.h (revision 2408) +++ libibverbs/include/infiniband/verbs.h (working copy) @@ -499,7 +499,7 @@ extern int ibv_dealloc_pd(struct ibv_pd /** * ibv_reg_mr - Register a memory region */ -extern struct ibv_mr *ibv_reg_mr(struct ibv_pd *pd, void *addr, +extern struct ibv_mr *ibv_reg_mr(struct ibv_pd *pd, volatile const void *addr, size_t length, enum ibv_access_flags access); /** Index: libibverbs/src/verbs.c =================================================================== --- libibverbs/src/verbs.c (revision 2408) +++ libibverbs/src/verbs.c (working copy) @@ -64,7 +64,7 @@ int ibv_dealloc_pd(struct ibv_pd *pd) return pd->context->ops.dealloc_pd(pd); } -struct ibv_mr *ibv_reg_mr(struct ibv_pd *pd, void *addr, +struct ibv_mr *ibv_reg_mr(struct ibv_pd *pd, volatile const void *addr, size_t length, enum ibv_access_flags access) { struct ibv_mr *mr; Index: libmthca/src/mthca.h =================================================================== --- libmthca/src/mthca.h (revision 2408) +++ libmthca/src/mthca.h (working copy) @@ -260,7 +260,7 @@ extern int mthca_query_port(struct ibv_c extern struct ibv_pd *mthca_alloc_pd(struct ibv_context *context); extern int mthca_free_pd(struct ibv_pd *pd); -extern struct ibv_mr *mthca_reg_mr(struct ibv_pd *pd, void *addr, +extern struct ibv_mr *mthca_reg_mr(struct ibv_pd *pd, volatile const void *addr, size_t length, enum ibv_access_flags access); extern int mthca_dereg_mr(struct ibv_mr *mr); Index: libmthca/src/verbs.c =================================================================== --- libmthca/src/verbs.c (revision 2408) +++ libmthca/src/verbs.c (working copy) @@ -93,7 +93,8 @@ int mthca_free_pd(struct ibv_pd *pd) return 0; } -static struct ibv_mr *__mthca_reg_mr(struct ibv_pd *pd, void *addr, +static struct ibv_mr *__mthca_reg_mr(struct ibv_pd *pd, + volatile const void *addr, size_t length, uint64_t hca_va, enum ibv_access_flags access) { @@ -113,7 +114,7 @@ static struct ibv_mr *__mthca_reg_mr(str return mr; } -struct ibv_mr *mthca_reg_mr(struct ibv_pd *pd, void *addr, +struct ibv_mr *mthca_reg_mr(struct ibv_pd *pd, volatile const void *addr, size_t length, enum ibv_access_flags access) { return __mthca_reg_mr(pd, addr, length, (uintptr_t) addr, access); -- MST - Michael S. Tsirkin From ogerlitz at voltaire.com Thu May 19 02:33:25 2005 From: ogerlitz at voltaire.com (Or Gerlitz) Date: Thu, 19 May 2005 12:33:25 +0300 Subject: [openib-general] CM private data Message-ID: Caitlin> I believe that an IT-API it_ep_accept() supplies private data that it expects to be delivered to its peer when the three-way handshake option is selected. Both IT-API it_ep_accept() and DAT dat_cr_accept() cause the provider at the passive side to send a --REP--, so how --RTU-- is related to the _accept calls? I think you ment to say that exposing two-way handshake means that the consumer can not supply private data for the RTU as he is not aware of it (the RTU) being sent? Or. -----Original Message----- From: openib-general-bounces at openib.org [mailto:openib-general-bounces at openib.org] On Behalf Of Caitlin Bestler Sent: Thursday, May 19, 2005 2:16 AM To: Sean Hefty Cc: openib-general Subject: Re: [openib-general] CM private data I believe that an IT-API it_ep_accept() supplies private data that it expects to be delivered to its peer when the three-way handshake option is selected. DAT only exposes a two-way handshake, so there it never requires private data on the RTU. I don't know if any IT-API applications actually require the three-way handshake. On 5/18/05, Sean Hefty wrote: > Do any applications make use of the private data in these CM message: > RTU, MRA, or DREP? I'm doubtful of the RTU or DREP, but not as sure of the MRA. > > Since no replies are generated in response to these messages, the CM > does not keep them after their sends complete. However, it may need > to resend the messages. For example, it will resend the RTU if a > duplicate REP is received. > > If no applications are using the private data, I will not worry about > storing the private data for the retransmissions at this time. > > - Sean > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > _______________________________________________ openib-general mailing list openib-general at openib.org http://openib.org/mailman/listinfo/openib-general To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From halr at voltaire.com Thu May 19 03:43:10 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 19 May 2005 06:43:10 -0400 Subject: [openib-general] Re: [RFC] [PATCH] user_mad: Support RMPP on send side In-Reply-To: <20050519072347.GQ25713@mellanox.co.il> References: <1115845653.4490.188.camel@localhost.localdomain> <5264xgf85t.fsf@topspin.com> <20050519072347.GQ25713@mellanox.co.il> Message-ID: <1116499390.4479.132.camel@localhost.localdomain> On Thu, 2005-05-19 at 03:23, Michael S. Tsirkin wrote: > Quoting r. Roland Dreier : > > Subject: Re: [RFC] [PATCH] user_mad: Support RMPP on send side > > > > This looks OK to check in with one small comment on the following: > > > > - if (copy_to_user(buf, &packet->mad, sizeof packet->mad)) > > + if (copy_to_user(buf, &packet->mad, > > + min(count, packet->length + > > + sizeof (struct ib_user_mad)))) > > ret = -EFAULT; > > else > > - ret = sizeof packet->mad; > > + ret = count; > > > > This code will truncate a received MAD that is bigger than the buffer > > passed into read(), but return the full size of the packet. I don't > > think read() is allowed to do this: the return value can be at most > > the count value passed in by the user. > > > > I think we have two options: truncate and return the actual amount of > > data read to the user, or return an error if the user's buffer is too > > small. > > > > - R. > > > > If you truncate, how will the user know the MAD was truncated? The length returned. Roland is saying to fix it is one option (rather than ret = count). I'm now questioning whether that is useful behavior. Would a subsequent read get the next part of the packet and the user piece them together ? If not, then this should be an error for this case (which is the second option). I am leaning towards that unless someone has an objection to that behavior. -- Hal From caitlin.bestler at gmail.com Thu May 19 06:21:51 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Thu, 19 May 2005 06:21:51 -0700 Subject: [openib-general] CM private data In-Reply-To: References: Message-ID: <469958e0050519062150c3bb62@mail.gmail.com> With a two-way handshake only the passive side accepts the connection request (it_ep_accept() or dat_cr_accept()). IT-API defines an optional *three-way* handshake where the active side must *also* call it_ep_accept() before the final connection establishment can proceed. This does not map to iWARP/MPA in any standard way, so rather than defining a protocol to wrap the first handshake private data, DAT decided to stick with only two-way handshaking. There has been no screaming, so we can presume that *most* applications find two-way handshaking acceptable. However an IT-API provider is free to say that it supports three-way handshaking, and I believe it is implied (if not required) that InfiniBand providers do so. What I do not know is if any actual applications have ever made use of this capability, or if it is only a defensive encoding of a protocol option into an API that prefers not to avoid removal of options that are available at the wire level. DAT emphasized providing a clean API for transport neutral services, and so simply said "use two-way handshaking". In any event, if no support is going to be provided for private data on the second it_ep_accept at the verb layer then that should be explicitly documented, and I'd suggest sending a 'heads up' to the IT-API authors, On 5/19/05, Or Gerlitz wrote: > Caitlin> I believe that an IT-API it_ep_accept() supplies private data > that it expects to be delivered to its peer when the three-way handshake > option is selected. > > Both IT-API it_ep_accept() and DAT dat_cr_accept() cause the provider at > the passive side to send a --REP--, > so how --RTU-- is related to the _accept calls? > > I think you ment to say that exposing two-way handshake means that the > consumer can not supply private data for > the RTU as he is not aware of it (the RTU) being sent? > > Or. > > -----Original Message----- > From: openib-general-bounces at openib.org > [mailto:openib-general-bounces at openib.org] On Behalf Of Caitlin Bestler > Sent: Thursday, May 19, 2005 2:16 AM > To: Sean Hefty > Cc: openib-general > Subject: Re: [openib-general] CM private data > > I believe that an IT-API it_ep_accept() supplies private data that it > expects to be delivered to its peer when the three-way handshake option > is selected. > > DAT only exposes a two-way handshake, so there it never requires private > data on the RTU. > > I don't know if any IT-API applications actually require the three-way > handshake. > > > On 5/18/05, Sean Hefty wrote: > > Do any applications make use of the private data in these CM message: > > RTU, MRA, or DREP? I'm doubtful of the RTU or DREP, but not as sure > of the MRA. > > > > Since no replies are generated in response to these messages, the CM > > does not keep them after their sends complete. However, it may need > > to resend the messages. For example, it will resend the RTU if a > > duplicate REP is received. > > > > If no applications are using the private data, I will not worry about > > storing the private data for the retransmissions at this time. > > > > - Sean > > _______________________________________________ > > openib-general mailing list > > openib-general at openib.org > > http://openib.org/mailman/listinfo/openib-general > > > > To unsubscribe, please visit > > http://openib.org/mailman/listinfo/openib-general > > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From mst at mellanox.co.il Thu May 19 06:24:16 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Thu, 19 May 2005 16:24:16 +0300 Subject: [openib-general] [PATCH] management: resource leak fixes Message-ID: <20050519132416.GK4717@mellanox.co.il> Management libraries leak resources (memory, file/directory handles). Also a trailing whitespace fix in one place in libibcommon. Signed-off-by: Michael S. Tsirkin Index: libibumad/src/umad.c =================================================================== --- libibumad/src/umad.c (revision 2411) +++ libibumad/src/umad.c (working copy) @@ -357,8 +357,10 @@ get_ca(char *ca_name, umad_ca_t *ca) if (!(dir = opendir(dir_name))) return -ENOENT; - if ((r = scandir(dir_name, &namelist, 0, alphasort)) < 0) - return -EIO; + if ((r = scandir(dir_name, &namelist, 0, alphasort)) < 0) { + ret = errno < 0 ? errno : -EIO; + goto error; + } ret = 0; ca->numports = 0; @@ -388,6 +390,7 @@ get_ca(char *ca_name, umad_ca_t *ca) free(namelist[i]); free(namelist); + closedir(dir); put_ca(ca); return 0; @@ -395,7 +398,8 @@ clean: for (i = 0; i < r; i++) free(namelist[i]); free(namelist); - +error: + closedir(dir); release_ca(ca); return ret; Index: libibcommon/src/sysfs.c =================================================================== --- libibcommon/src/sysfs.c (revision 2411) +++ libibcommon/src/sysfs.c (working copy) @@ -56,6 +56,7 @@ #include #include #include +#include #include "common.h" @@ -86,14 +87,20 @@ sys_read_string(char *dir_name, char *fi if ((fd = open(path, O_RDONLY)) < 0) return ret_code(); - if ((r = read(fd, str, max_len)) < 0) + if ((r = read(fd, str, max_len)) < 0) { + int e = errno; + close(fd); + errno = e; return ret_code(); + } str[(r < max_len) ? r : max_len - 1] = 0; if ((s = strrchr(str, '\n'))) *s = 0; - return 0; + + close(fd); + return 0; } int -- MST - Michael S. Tsirkin From parks at lanl.gov Thu May 19 07:25:08 2005 From: parks at lanl.gov (Parks Fields) Date: Thu, 19 May 2005 08:25:08 -0600 Subject: [openib-general] 2.6.12 question Message-ID: <6.2.1.2.2.20050519082246.039c1430@ccn-mail.lanl.gov> Hello Is there a branch for 2.6.12? We have noticed that we get different performance when built on 2.6.11-5 vs 2.6.12-rc4. thanks From mst at mellanox.co.il Thu May 19 07:43:34 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Thu, 19 May 2005 17:43:34 +0300 Subject: [openib-general] Re: 2.6.12 question In-Reply-To: <6.2.1.2.2.20050519082246.039c1430@ccn-mail.lanl.gov> References: <6.2.1.2.2.20050519082246.039c1430@ccn-mail.lanl.gov> Message-ID: <20050519144334.GM4717@mellanox.co.il> Quoting r. Parks Fields : > Subject: 2.6.12 question > > Hello > > Is there a branch for 2.6.12? > We have noticed that we get different performance when built on 2.6.11-5 vs > 2.6.12-rc4. > > thanks > 1. Could you elaborate please? 2. I suggest you make sure the kernels are built with all the same options, especially regarding security, networking and cpu type options. -- MST - Michael S. Tsirkin From halr at voltaire.com Thu May 19 07:42:30 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 19 May 2005 10:42:30 -0400 Subject: [openib-general] Re: [PATCH] management: resource leak fixes In-Reply-To: <20050519132416.GK4717@mellanox.co.il> References: <20050519132416.GK4717@mellanox.co.il> Message-ID: <1116513573.4479.224.camel@localhost.localdomain> On Thu, 2005-05-19 at 09:24, Michael S. Tsirkin wrote: > Management libraries leak resources (memory, file/directory handles). > Also a trailing whitespace fix in one place in libibcommon. > > Signed-off-by: Michael S. Tsirkin Thanks. Applied. -- Hal From parks at lanl.gov Thu May 19 07:55:14 2005 From: parks at lanl.gov (Parks Fields) Date: Thu, 19 May 2005 08:55:14 -0600 Subject: [openib-general] Re: 2.6.12 question In-Reply-To: <20050519144334.GM4717@mellanox.co.il> References: <6.2.1.2.2.20050519082246.039c1430@ccn-mail.lanl.gov> <20050519144334.GM4717@mellanox.co.il> Message-ID: <6.2.1.2.2.20050519085408.039d72e0@ccn-mail.lanl.gov> > >2. I suggest you make sure the kernels are built with all the same options, > especially regarding security, networking and cpu type options. This is on the same box, but I have to check the .config files. From roland at topspin.com Thu May 19 08:04:01 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 19 May 2005 08:04:01 -0700 Subject: [openib-general] 2.6.12 question In-Reply-To: <6.2.1.2.2.20050519082246.039c1430@ccn-mail.lanl.gov> (Parks Fields's message of "Thu, 19 May 2005 08:25:08 -0600") References: <6.2.1.2.2.20050519082246.039c1430@ccn-mail.lanl.gov> Message-ID: <52psvnb6la.fsf@topspin.com> Parks> Is there a branch for 2.6.12? We have noticed that we get Parks> different performance when built on 2.6.11-5 vs 2.6.12-rc4. Obviously the drivers/infiniband already in 2.6.12-rc4 should work fine. Current svn should also work -- I think the only change required is for SDP, and Tom Duffy posted a patch. - R. From roland at topspin.com Thu May 19 08:05:43 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 19 May 2005 08:05:43 -0700 Subject: [openib-general] Re: registering read-only memory In-Reply-To: <20050519085208.GC4717@mellanox.co.il> (Michael S. Tsirkin's message of "Thu, 19 May 2005 11:52:08 +0300") References: <20050519085208.GC4717@mellanox.co.il> Message-ID: <52ll6bb6ig.fsf@topspin.com> Michael> 2. ibv_reg_mr fails. Why is that? How does it fail? R. From mst at mellanox.co.il Thu May 19 08:49:38 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Thu, 19 May 2005 18:49:38 +0300 Subject: [openib-general] Re: registering read-only memory In-Reply-To: <52ll6bb6ig.fsf@topspin.com> References: <20050519085208.GC4717@mellanox.co.il> <52ll6bb6ig.fsf@topspin.com> Message-ID: <20050519154938.GA5854@mellanox.co.il> Quoting r. Roland Dreier : > Subject: Re: registering read-only memory > > Michael> 2. ibv_reg_mr fails. Why is that? > > How does it fail? > > R. > Returns NULL, naturally. -- MST - Michael S. Tsirkin From mshefty at ichips.intel.com Thu May 19 09:06:34 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Thu, 19 May 2005 09:06:34 -0700 Subject: [openib-general] CM private data In-Reply-To: <469958e0050519062150c3bb62@mail.gmail.com> References: <469958e0050519062150c3bb62@mail.gmail.com> Message-ID: <428CB98A.3010703@ichips.intel.com> Caitlin Bestler wrote: > In any event, if no support is going to be provided for > private data on the second it_ep_accept at the verb > layer then that should be explicitly documented, and > I'd suggest sending a 'heads up' to the IT-API authors, To clarify, I was only trying to determine when to implement this, not if. Based on the feedback, I will try to fix this as part of my next set of changes to the CM. Thanks, Sean From libor at topspin.com Thu May 19 09:12:02 2005 From: libor at topspin.com (Libor Michalek) Date: Thu, 19 May 2005 09:12:02 -0700 Subject: [openib-general] Re: sdp_link.c info_list locking question In-Reply-To: <20050519081238.GA4717@mellanox.co.il>; from mst@mellanox.co.il on Thu, May 19, 2005 at 11:12:38AM +0300 References: <20050519081238.GA4717@mellanox.co.il> Message-ID: <20050519091202.A9076@topspin.com> On Thu, May 19, 2005 at 11:12:38AM +0300, Michael S. Tsirkin wrote: > Libor, I'm looking at sdp_link.c, and I dont see any lock > protecting the info_list linked list. > What prevents sdp_path_info_lookup from being called while > sdp_path_info_create or sdp_path_info_destroy is in progress? Michael, you are correct that the locking is missing, there's actually a line about it in the TODO file. When I moved the sdp code to gen2 it was easier to rewrite sdp_link because of all the other junk that was in the old version, but to get something done quickly I left out locking. I figured eventually the IP to PathRecord service would be broken into a seperate module, like the old ip2pr, since there is the possibility for multiple consumers. However, that has yet to happen, so if someone wants to add the locking, feel free. -Libor From mst at mellanox.co.il Thu May 19 09:20:37 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Thu, 19 May 2005 19:20:37 +0300 Subject: [openib-general] Re: sdp_link.c info_list locking question In-Reply-To: <20050519091202.A9076@topspin.com> References: <20050519081238.GA4717@mellanox.co.il> <20050519091202.A9076@topspin.com> Message-ID: <20050519162037.GB5946@mellanox.co.il> Quoting r. Libor Michalek : > Subject: Re: sdp_link.c info_list locking question > > On Thu, May 19, 2005 at 11:12:38AM +0300, Michael S. Tsirkin wrote: > > Libor, I'm looking at sdp_link.c, and I dont see any lock > > protecting the info_list linked list. > > What prevents sdp_path_info_lookup from being called while > > sdp_path_info_create or sdp_path_info_destroy is in progress? > > Michael, you are correct that the locking is missing, there's > actually a line about it in the TODO file. When I moved the > sdp code to gen2 it was easier to rewrite sdp_link because of > all the other junk that was in the old version, but to get > something done quickly I left out locking. I figured eventually > the IP to PathRecord service would be broken into a seperate > module, like the old ip2pr, since there is the possibility for > multiple consumers. However, that has yet to happen, so if > someone wants to add the locking, feel free. > > > -Libor > OK. I might look at this next week. First step would be to switch to standard list macros, though. -- MST - Michael S. Tsirkin From caitlin.bestler at gmail.com Thu May 19 09:24:58 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Thu, 19 May 2005 09:24:58 -0700 Subject: [openib-general] CM private data In-Reply-To: <428CB98A.3010703@ichips.intel.com> References: <469958e0050519062150c3bb62@mail.gmail.com> <428CB98A.3010703@ichips.intel.com> Message-ID: <469958e00505190924593b0382@mail.gmail.com> In that case, I'll point out that no application can rely upon the final data being delivered because the first data can easily beat the RTU. When that happens the IT-API layer declares the connection established, and provides no private data. So being predictably unreliable for one implementation stage is certainly something you can get away with. Even when you add support it might be quite acceptable to send the private data *only* on the first try, or to require the IT-API layer to do the retries. I was mostly responding to the suggestion that the private data be eliminated from the API if nobody had any use for it. On 5/19/05, Sean Hefty wrote: > Caitlin Bestler wrote: > > In any event, if no support is going to be provided for > > private data on the second it_ep_accept at the verb > > layer then that should be explicitly documented, and > > I'd suggest sending a 'heads up' to the IT-API authors, > > To clarify, I was only trying to determine when to implement this, not if. > Based on the feedback, I will try to fix this as part of my next set of > changes to the CM. > > Thanks, > Sean > From ftillier at silverstorm.com Thu May 19 09:34:16 2005 From: ftillier at silverstorm.com (Fab Tillier) Date: Thu, 19 May 2005 09:34:16 -0700 Subject: [openib-general] CM private data In-Reply-To: <469958e00505190924593b0382@mail.gmail.com> Message-ID: <000401c55c90$9a01ea70$9b5aa8c0@infiniconsys.com> > From: Caitlin Bestler [mailto:caitlin.bestler at gmail.com] > Sent: Thursday, May 19, 2005 9:25 AM > > So being predictably unreliable for one implementation > stage is certainly something you can get away with. > Even when you add support it might be quite acceptable > to send the private data *only* on the first try, or to > require the IT-API layer to do the retries. Only sending the user's private data on the first try requires the user to support connection establishment with: - no private data (no RTU received) - zero private data (RTU retry) - remote peer's private data (first RTU) The receipt of private data does not mean that private data is actually what the remote side sent, and also requires users to never use all zero private data since that would make the second and third case above indistinguishable. So if the CM is going to expose the private data, it needs to put that private data in all retries. I'm still for hiding the RTU private data. I think it's useless because it's unreliable - anything exchanged via private data in the RTU must also be exchanged by other means in case the connection is established before the RTU is received. Any ULPs that depend on the RTU private data are setting themselves up for potential failures. Just my opinion, though... - Fab From mshefty at ichips.intel.com Thu May 19 09:39:53 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Thu, 19 May 2005 09:39:53 -0700 Subject: [openib-general] CM private data In-Reply-To: <000401c55c90$9a01ea70$9b5aa8c0@infiniconsys.com> References: <000401c55c90$9a01ea70$9b5aa8c0@infiniconsys.com> Message-ID: <428CC159.1080206@ichips.intel.com> Fab Tillier wrote: > I'm still for hiding the RTU private data. I think it's useless because > it's unreliable - anything exchanged via private data in the RTU must also > be exchanged by other means in case the connection is established before the > RTU is received. Any ULPs that depend on the RTU private data are setting > themselves up for potential failures. This depends on the implementation. If the server side of a connection initiates the data transfer, it cannot do so until an RTU is received. - Sean From halr at voltaire.com Thu May 19 09:38:28 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 19 May 2005 12:38:28 -0400 Subject: [openib-general] user_mad::ib_umad_read question Message-ID: <1116520708.4479.19.camel@hal.voltaire.com> Hi, In ib_umad_read, there is currently (or soon to be something like) the following: ... packet = list_entry(file->recv_list.next, struct ib_umad_packet, list); list_del(&packet->list); spin_unlock_irq(&file->recv_lock); if (copy_to_user(buf, &packet->mad, min(count, packet->length + sizeof (struct ib_user_mad)))) ret = -EFAULT; else ret = count; kfree(packet); return ret; Should the packet be thrown away because copy_to_user() fails ? Shouldn't it be placed back at the head of the list ? Unfortunately, that would mean holding the recv lock longer (through the duration of copy_to_user). -- Hal From kashyapv at us.ibm.com Thu May 19 09:41:54 2005 From: kashyapv at us.ibm.com (Vivek Kashyap) Date: Thu, 19 May 2005 09:41:54 -0700 (PDT) Subject: [openib-general] How about ib_send_page() ? In-Reply-To: Message-ID: > > The most interesting optimization available is implementing the IPoIB > connected mode draft, although I don't think it's as easy as Vivek > indicated -- for example, I'm not sure how to deal with having > different MTUs depending on the destination. The draft does allow for a negotiation per connection for the implementations that wish to take advantage of it. However, an implementation can by default choose to use a 'connected-mode MTU' e.g. 32K always. It can then, for every connection choose to, negotiate to this value and if it is not workable fall back to the UD mode and deny the connection mode. The ARP entries hold the connected mode flags thereby keeping track of the mode to use per destination. I'd be more than happy to discuss other implementation issues. As I noted earlier it will also help refine the draft. Vivek P.S. cc replies to kashyapv at us.ibm.com. From mshefty at ichips.intel.com Thu May 19 09:46:44 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Thu, 19 May 2005 09:46:44 -0700 Subject: [openib-general] user_mad::ib_umad_read question In-Reply-To: <1116520708.4479.19.camel@hal.voltaire.com> References: <1116520708.4479.19.camel@hal.voltaire.com> Message-ID: <428CC2F4.8040403@ichips.intel.com> Hal Rosenstock wrote: > In ib_umad_read, there is currently (or soon to be something like) the > following: > ... > packet = list_entry(file->recv_list.next, struct ib_umad_packet, list); > list_del(&packet->list); > > spin_unlock_irq(&file->recv_lock); > > if (copy_to_user(buf, &packet->mad, > min(count, packet->length + > sizeof (struct ib_user_mad)))) > ret = -EFAULT; > else > ret = count; > > kfree(packet); > return ret; > > Should the packet be thrown away because copy_to_user() fails ? > Shouldn't it be placed back at the head of the list ? Unfortunately, > that would mean holding the recv lock longer (through the duration of > copy_to_user). In the failure case, could you just re-acquire the lock and insert the packet at the head of the list? Even if this results in reading MADs out of order, I wouldn't think that this would matter. - Sean From mst at mellanox.co.il Thu May 19 09:48:10 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Thu, 19 May 2005 19:48:10 +0300 Subject: [openib-general] Re: user_mad::ib_umad_read question In-Reply-To: <1116520708.4479.19.camel@hal.voltaire.com> References: <1116520708.4479.19.camel@hal.voltaire.com> Message-ID: <20050519164810.GC5946@mellanox.co.il> Quoting r. Hal Rosenstock : > Subject: user_mad::ib_umad_read question > > Hi, > > In ib_umad_read, there is currently (or soon to be something like) the > following: > ... > packet = list_entry(file->recv_list.next, struct ib_umad_packet, list); > list_del(&packet->list); > > spin_unlock_irq(&file->recv_lock); > > if (copy_to_user(buf, &packet->mad, > min(count, packet->length + > sizeof (struct ib_user_mad)))) > ret = -EFAULT; > else > ret = count; > > kfree(packet); > return ret; > > Should the packet be thrown away because copy_to_user() fails ? > Shouldn't it be placed back at the head of the list ? Unfortunately, > that would mean holding the recv lock longer (through the duration of > copy_to_user). > > -- Hal copy_to_user might sleep so you cant call it under a spinlock. Since the user is only hurting himself by passing an illegal address, I'd think it doesnt hurt to drop the mad. -- MST - Michael S. Tsirkin From libor at topspin.com Thu May 19 09:48:35 2005 From: libor at topspin.com (Libor Michalek) Date: Thu, 19 May 2005 09:48:35 -0700 Subject: [openib-general] CM private data In-Reply-To: <000401c55c90$9a01ea70$9b5aa8c0@infiniconsys.com>; from ftillier@silverstorm.com on Thu, May 19, 2005 at 09:34:16AM -0700 References: <469958e00505190924593b0382@mail.gmail.com> <000401c55c90$9a01ea70$9b5aa8c0@infiniconsys.com> Message-ID: <20050519094835.B9076@topspin.com> On Thu, May 19, 2005 at 09:34:16AM -0700, Fab Tillier wrote: > > From: Caitlin Bestler [mailto:caitlin.bestler at gmail.com] > > Sent: Thursday, May 19, 2005 9:25 AM > > > > So being predictably unreliable for one implementation > > stage is certainly something you can get away with. > > Even when you add support it might be quite acceptable > > to send the private data *only* on the first try, or to > > require the IT-API layer to do the retries. > > I'm still for hiding the RTU private data. I think it's useless because > it's unreliable - anything exchanged via private data in the RTU must also > be exchanged by other means in case the connection is established before the > RTU is received. Any ULPs that depend on the RTU private data are setting > themselves up for potential failures. I agree for exactly the reason you give, I can't think of a legitimate use for RTU private data. I'd get rid of it entirely, from the code as well as the spec, which is why I think it would be a waste of someones time to add the correct support for it. -Libor From ftillier at silverstorm.com Thu May 19 09:49:54 2005 From: ftillier at silverstorm.com (Fab Tillier) Date: Thu, 19 May 2005 09:49:54 -0700 Subject: [openib-general] CM private data In-Reply-To: <428CC159.1080206@ichips.intel.com> Message-ID: <000501c55c92$c8fdbd20$9b5aa8c0@infiniconsys.com> > From: Sean Hefty [mailto:mshefty at ichips.intel.com] > Sent: Thursday, May 19, 2005 9:40 AM > > Fab Tillier wrote: > > I'm still for hiding the RTU private data. I think it's useless because > > it's unreliable - anything exchanged via private data in the RTU must > > also be exchanged by other means in case the connection is established > > before the RTU is received. Any ULPs that depend on the RTU private > > data are setting themselves up for potential failures. > > This depends on the implementation. If the server side of a connection > initiates the data transfer, it cannot do so until an RTU is received. Good point. So this gets back to the CM providing the user's RTU private data even in retries. - Fab From halr at voltaire.com Thu May 19 09:58:52 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 19 May 2005 12:58:52 -0400 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: References: Message-ID: <1116521932.4479.27.camel@hal.voltaire.com> Hi Vivek, On Thu, 2005-05-19 at 12:41, Vivek Kashyap wrote: > > > > > > The most interesting optimization available is implementing the IPoIB > > connected mode draft, although I don't think it's as easy as Vivek > > indicated -- for example, I'm not sure how to deal with having > > different MTUs depending on the destination. > > > > The draft does allow for a negotiation per connection for the implementations > that wish to take advantage of it. However, an implementation can by default > choose to use a 'connected-mode MTU' e.g. 32K always. It can then, for every > connection choose to, negotiate to this value and if it is not workable fall > back to the UD mode and deny the connection mode. The ARP entries hold the > connected mode flags thereby keeping track of the mode to use per destination. Sounds like there should be an "agreement" on a default connected mode MTU or else this will drop down to UD. I have a couple of clarification questions on 5.1 Per-Connection MTU: 1. I presume the Receive MTU is in the first 2 bytes of the private data in the CM messages. Is that correct ? 2. Also, CM REQ is mentioned for the requester receive MTU. Wouldn't CM REP carry the granted receive MTU which is constrained to be the requested MTU or less ? So 2 things on this: The I-D says "The private data field MUST carry" the receive MTU. Does that include RTUs ? Thanks. -- Hal From jlentini at netapp.com Thu May 19 10:03:08 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 19 May 2005 13:03:08 -0400 (EDT) Subject: [openib-general] [PATCH] remove redundant check in mthca_provider.c Message-ID: Remove redundant check of pd->uobject in mthca_provider.c Signed-off-by: James Lentini -------------- next part -------------- Index: infiniband/hw/mthca/mthca_provider.c =================================================================== --- infiniband/hw/mthca/mthca_provider.c (revision 2404) +++ infiniband/hw/mthca/mthca_provider.c (working copy) @@ -478,9 +478,7 @@ kfree(qp); return ERR_PTR(err); } - } - if (pd->uobject) { qp->mr.ibmr.lkey = ucmd.lkey; qp->sq.db_index = ucmd.sq_db_index; qp->rq.db_index = ucmd.rq_db_index; From iod00d at hp.com Thu May 19 10:28:24 2005 From: iod00d at hp.com (Grant Grundler) Date: Thu, 19 May 2005 10:28:24 -0700 Subject: [openib-general] Re: diff-perftest-07 replace pp_get_local_lid() In-Reply-To: <20050519055126.GA4079@mellanox.co.il> References: <20050518013821.GO1078@esmail.cup.hp.com> <20050518135130.GE25713@mellanox.co.il> <20050519055126.GA4079@mellanox.co.il> Message-ID: <20050519172824.GD10318@esmail.cup.hp.com> On Thu, May 19, 2005 at 08:51:26AM +0300, Michael S. Tsirkin wrote: > Quoting r. Michael S. Tsirkin : > > Subject: Re: diff-perftest-07 replace pp_get_local_lid() > > > Should I rather be looking at fixing up netperf to support IB? > > > > > > thanks, > > > grant > > > > > > > That may be kind of hard, given that uverbs API is completely different > > from socket API. > > Wait, isnt this what SDP is doing? SDP is a thin, kernel "translation layer" from IP sockets to verbs, right? The idea I'm chasing is for netperf to use uverbs directly like rdma_lat does. grant From roland at topspin.com Thu May 19 10:31:47 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 19 May 2005 10:31:47 -0700 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: (Vivek Kashyap's message of "Thu, 19 May 2005 09:41:54 -0700 (PDT)") References: Message-ID: <521x83azr0.fsf@topspin.com> Vivek> The draft does allow for a negotiation per connection for Vivek> the implementations that wish to take advantage of Vivek> it. However, an implementation can by default choose to use Vivek> a 'connected-mode MTU' e.g. 32K always. It can then, for Vivek> every connection choose to, negotiate to this value and if Vivek> it is not workable fall back to the UD mode and deny the Vivek> connection mode. The ARP entries hold the connected mode Vivek> flags thereby keeping track of the mode to use per Vivek> destination. But this means that the MTU of the link will be different for UD destinations (including multicast) and RC destinations, right? Or am I missing something? - R. From iod00d at hp.com Thu May 19 11:21:02 2005 From: iod00d at hp.com (Grant Grundler) Date: Thu, 19 May 2005 11:21:02 -0700 Subject: [openib-general] [PATCH] rewrite perftest/README In-Reply-To: <004801c55c34$29e893d0$8000000a@blorp> References: <20050519005637.GD7557@esmail.cup.hp.com> <004801c55c34$29e893d0$8000000a@blorp> Message-ID: <20050519182102.GF10318@esmail.cup.hp.com> On Thu, May 19, 2005 at 06:32:32AM +0100, Paul Baxter wrote: > Grant, Michael > > Great work! > > I wanted to point out a recent thread on comp.arch discussing the merits of > using the geometric mean instead of the arithmetic mean as a basis for > sampling the timing of a population. Paul, thanks for pointing out the thread - I'm learning alot! :^) > My statistics is a bit rusty, but > calculating and presenting GM might be useful a well? After reading the original posting, I think the answer is NO. Particularly, two comments in John Mashey's original posting: | The GM is the correct mean for combining benchmark results intended | to be a sample from a larger population of programs and intended to | predict the performance distribution of other benchmarks, which after | all, is what people want for generalized performance comparisons. | This is true if the population follows a *lognormal* [described later] | distribution ... and it turns out, many do so. rdma_lat is one benchmark - not a "combination of benchmark results". If perftest ends up with more than two benchmarks, we can try to look weighted geometric mean to get one number from running the whole mess. But TBH, I have no interest or need to do that. Sounds like statistical wanking off to me. I'm much more interested in comparing differences of individual rdma_lat runs and sort out why the results are different. | If you know your workload, your benchamrks *are* the population, | and you use algebra. We know pretty well what we are measuring with rmda_lat. Well, Michael does. I only have a dangerously vague understanding. :^) I know we have two sets of data: "warmup" and "runtime". The first two or three measurements are related to warmup. John Mashey's posting suggested keeping separate "groups of data" apart. Ie. it was probably correct to not include "warmup" values in the std deviation and arithmetic mean (avg). Conclusion: The median value still looks like the right thing to report. > http://groups.google.co.uk/group/comp.arch/browse_thread/thread/3f5a9ed1d79ed726/416e58b5e48c1715?q=geometric&rnum=6&hl=en#416e58b5e48c1715 > > [comp.arch, 'SPEC use of Geometric Mean', May 13th, 2005, John Mashey ] thanks again, grant From iod00d at hp.com Thu May 19 11:25:33 2005 From: iod00d at hp.com (Grant Grundler) Date: Thu, 19 May 2005 11:25:33 -0700 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <8A71B368A89016469F72CD08050AD334067778@maui.asicdesigners.com> References: <8A71B368A89016469F72CD08050AD334067778@maui.asicdesigners.com> Message-ID: <20050519182533.GG10318@esmail.cup.hp.com> On Wed, May 18, 2005 at 08:00:15PM -0700, Felix Marti wrote: > Hi Roland, > > define SMP :) Anytime a CPU is cache coherent with another CPU. > at these rates, system architecture comes into place, Definitely. The architecture puts boundaries on how coherency can be implemented...and thus available memory bus bandwidth. grant From trimmer at silverstorm.com Thu May 19 11:56:23 2005 From: trimmer at silverstorm.com (Rimmer, Todd) Date: Thu, 19 May 2005 14:56:23 -0400 Subject: [openib-general] CM private data Message-ID: <5D78D28F88822E4D8702BB9EEF1A436773DD74@mercury.infiniconsys.com> All three of these have mechanisms whereby the message can be skipped, in which case applications should not depend on the private data (IB spec mentions that apps should not depend on them). For example, A inbound receive while in RTR state after having sent a REP can be treated as an RTU, in which case later arrival of the RTU is to be discarded by the CM. Todd Rimmer > -----Original Message----- > From: Tillier, Fabian > Sent: Wednesday, May 18, 2005 3:28 PM > To: 'Sean Hefty'; openib-general > Subject: RE: [openib-general] CM private data > > > > From: Sean Hefty [mailto:mshefty at ichips.intel.com] > > Sent: Wednesday, May 18, 2005 12:23 PM > > > > Do any applications make use of the private data in these > CM message: RTU, > > MRA, or DREP? I'm doubtful of the RTU or DREP, but not as > sure of the > MRA. > > > > Since no replies are generated in response to these > messages, the CM does > > not keep them after their sends complete. However, it may > need to resend > > the messages. For example, it will resend the RTU if a > duplicate REP is > > received. > > > > If no applications are using the private data, I will not > worry about > > storing the private data for the retransmissions at this time. > > If you're not going to store them, I would suggest removing > the private data > for the associated calls (RTU, MRA, DREP). That way it > becomes very clear > to applications wishing to use the private data that they can't. > > - Fab > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From paul.baxter at dsl.pipex.com Thu May 19 12:29:05 2005 From: paul.baxter at dsl.pipex.com (Paul Baxter) Date: Thu, 19 May 2005 20:29:05 +0100 Subject: [openib-general] Infinipath support in OpenIB References: <20050519005637.GD7557@esmail.cup.hp.com> <004801c55c34$29e893d0$8000000a@blorp> <20050519182102.GF10318@esmail.cup.hp.com> Message-ID: <00b601c55ca9$06325f00$8000000a@blorp> I read with interest the PR blurb at http://supercomputingonline.com/article.php?sid=8740 regarding InfiniPath's very low latencies and high throughput for MPI even at very modest message sizes. The article states: 'InfiniPath will also support the OpenIB software stack providing full InfiniBand compliance.' Is this something the OpenIB software developers are working on and something that can be commented on in public, or should I speculate/ask Pathscale? Paul From tduffy at sun.com Thu May 19 12:29:55 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 19 May 2005 12:29:55 -0700 Subject: [openib-general] OOPS: ib_mad crashery on bootup Message-ID: <1116530995.980.9.camel@duffman> This is from latest bits, r2414. Bringing up interface ib0: stack segment: 0000 [1] SMP CPU 0 Modules linked in: ext3 jbd dm_mod video container button battery ac ohci_hcd tpm_nsc tpm i2c_amd756 i2c_core ib_mthca ib_ipoib ib_sa ib_mad ib_core tg3 floppy xfs exportfs mptscsih mptbase sd_mod scsi_mod Pid: 1253, comm: ib_mad1 Not tainted 2.6.12-rc4openib RIP: 0010:[] {:ib_mad:ib_mad_send_done_handler+31} RSP: 0018:ffff81007ed71d48 EFLAGS: 00010296 RAX: 0100000000001508 RBX: ffff81007f346298 RCX: 0000000000000000 RDX: ffffc20000052001 RSI: ffff81007ed71dc8 RDI: ffff81007f3461c0 RBP: 0100000000001500 R08: ffff81007ed70000 R09: ffff81003fa69070 R10: ffff81003fc4d258 R11: 0000000000000008 R12: ffff81003f33c4b0 R13: ffff81003f33c4b0 R14: ffff81007ed71dc8 R15: ffffffff880f89a0 FS: 00002aaaaaad4e00(0000) GS:ffffffff80489000(0000) knlGS:0000000000000000 CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b CR2: 0000003a2428ebf0 CR3: 0000000000101000 CR4: 00000000000006e0 Process ib_mad1 (pid: 1253, threadinfo ffff81007ed70000, task ffff81003fa69070) Stack: 0000000080000001 0000000000000000 ffff81003fa69070 ffff81007f346298 ffff81007f3462a0 ffff81003fd8c778 ffff81007f3461c0 ffff81007ed71dc8 ffffffff880f89a0 ffffffff880f8f1b Call Trace:{:ib_mad:ib_mad_completion_handler+0} {:ib_mad:ib_mad_completion_handler+1403} {cache_free_debugcheck+715} {__wake_up+67} {:ib_mad:ib_mad_completion_handler+0} {worker_thread+476} {default_wake_function+0} {keventd_create_kthread+0} {worker_thread+0} {keventd_create_kthread+0} {kthread+217} {schedule_tail+64} {child_rip+8} {keventd_create_kthread+0} {kthread+0} {child_rip+0} Code: 4c 8b 7d 20 48 89 04 24 48 89 ef 31 db e8 bf 3c 25 f8 49 8b RIP {:ib_mad:ib_mad_send_done_handler+31} RSP Error, some other host already uses address 192.168.0.233. -- I wish we lived in the America of yesteryear that only exists in the minds of us Republicans. -- Ned Flanders -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From tduffy at sun.com Thu May 19 12:34:08 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 19 May 2005 12:34:08 -0700 Subject: [openib-general] kdapltest regression? failing now... Message-ID: <1116531248.980.10.camel@duffman> I am not sure when this started, but after updating to top of trunk*, I can no longer get kdapltest to work properly. Both ipoib and sdp are working. Both server and client are returning an error: DAT_INVALID_HANDLE. This is coming from ib_create_qp(). With debugging turned on: [root at flopteron2 ~]# ./kdapltest -T S -D mthca0a -d kDAPL: dapl_ia_open (mthca0a, 8, ffff81000b806308, ffff81000b8062d8) kDAPL: dapl_ia_open () returns 0x0 kDAPL: dapl_pz_create (ffff81001ba165c8, ffff81000b8062e0) kDAPL: dapl_evd_kcreate (ffff81001ba165c8, 8, 1, upcall, 0x20, ffff81000b8062e8) kDAPL: dapl_evd_kcreate (ffff81001ba165c8, 8, 1, upcall, 0xa0, ffff81000b8062f0) kDAPL: dapl_evd_kcreate (ffff81001ba165c8, 8, 1, upcall, 0x10, ffff81000b806300) kDAPL: dapl_evd_kcreate (ffff81001ba165c8, 8, 1, upcall, 0x40, ffff81000b8062f8) kDAPL: dapl_ep_create (ffff81001ba165c8, ffff81001b9442c8, ffff81001ba164b0, ffff81001ba166e0, ffff81001ba22050, 0000000000000000, ffff81000b806318) kDAPL: dapl_ib_qp_alloc: ib_create_qp failed = -22 kDAPL: dapl_evd_free (ffff81001ba22050) kDAPL: dapl_evd_free () returns 0x0 kDAPL: dapl_evd_free (ffff81001ba22168) kDAPL: dapl_evd_free () returns 0x0 kDAPL: dapl_evd_free (ffff81001ba166e0) kDAPL: dapl_evd_free () returns 0x0 kDAPL: dapl_evd_free (ffff81001ba164b0) kDAPL: dapl_evd_free () returns 0x0 kDAPL: dapl_pz_free (ffff81001b9442c8) kDAPL: dapl_ia_query (ffff81001ba165c8, 0000000000000000, 0000000000000000, ffff81001bba7b28) kDAPL: dapl_ia_query () returns 0x0 kDAPL: dapl_ia_close (ffff81001ba165c8, 1) kDAPL: dapl_evd_free (ffff81001ba167f8) kDAPL: dapl_evd_free () returns 0x0 Server_Cmd.debug: 1 Server_Cmd.dapl_name: mthca0a DT_cs_Server: IA mthca0a opened DT_cs_Server: PZ created DT_cs_Server: dat_ep_create error: DAT_INVALID_HANDLE DT_cs_Server: Waiting for clients to all go away... DT_cs_Server: Cleaning up ... DT_cs_Server: IA mthca0a closed DT_cs_Server (mthca0a): Exiting. TEST INSTANCE 0 TEST return code = 1 Also, the ib_at module prints this out now when you ping (after running kdapltest)... ib_at: ib_at_arp_work: Process IB ARP ip <192.168.0.26> gid <0xfe800000000000000002c9010a99e031> -tduffy * running x86_64 SMP, 2.6.12-rc4, gcc 4.0.0-6, OpenIB r2414, opensm r2414 2 machines back-2-back -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From jlentini at netapp.com Thu May 19 12:40:18 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 19 May 2005 15:40:18 -0400 (EDT) Subject: [openib-general] kdapltest regression? failing now... In-Reply-To: <1116531248.980.10.camel@duffman> References: <1116531248.980.10.camel@duffman> Message-ID: I'm looking into this Tom. The following code was added to hw/mthca/mthca_qp.c on Friday (starting on line 1233): if ((qp->transport == MLX && qp->sq.max_gs + 2 > dev->limits.max_sg) || qp->sq.max_gs > dev->limits.max_sg || qp->rq.max_gs > dev->limits.max_sg) return -EINVAL; If anyone knows what we have set incorrectly, please let me know. Thanks, james On Thu, 19 May 2005, Tom Duffy wrote: tduffy> I am not sure when this started, but after updating to top of trunk*, I tduffy> can no longer get kdapltest to work properly. Both ipoib and sdp are tduffy> working. tduffy> tduffy> Both server and client are returning an error: DAT_INVALID_HANDLE. This tduffy> is coming from ib_create_qp(). With debugging turned on: tduffy> tduffy> [root at flopteron2 ~]# ./kdapltest -T S -D mthca0a -d tduffy> kDAPL: dapl_ia_open (mthca0a, 8, ffff81000b806308, ffff81000b8062d8) tduffy> kDAPL: dapl_ia_open () returns 0x0 tduffy> kDAPL: dapl_pz_create (ffff81001ba165c8, ffff81000b8062e0) tduffy> kDAPL: dapl_evd_kcreate (ffff81001ba165c8, 8, 1, upcall, 0x20, ffff81000b8062e8) tduffy> kDAPL: dapl_evd_kcreate (ffff81001ba165c8, 8, 1, upcall, 0xa0, ffff81000b8062f0) tduffy> kDAPL: dapl_evd_kcreate (ffff81001ba165c8, 8, 1, upcall, 0x10, ffff81000b806300) tduffy> kDAPL: dapl_evd_kcreate (ffff81001ba165c8, 8, 1, upcall, 0x40, ffff81000b8062f8) tduffy> kDAPL: dapl_ep_create (ffff81001ba165c8, ffff81001b9442c8, ffff81001ba164b0, ffff81001ba166e0, ffff81001ba22050, 0000000000000000, ffff81000b806318) tduffy> kDAPL: dapl_ib_qp_alloc: ib_create_qp failed = -22 tduffy> kDAPL: dapl_evd_free (ffff81001ba22050) tduffy> kDAPL: dapl_evd_free () returns 0x0 tduffy> kDAPL: dapl_evd_free (ffff81001ba22168) tduffy> kDAPL: dapl_evd_free () returns 0x0 tduffy> kDAPL: dapl_evd_free (ffff81001ba166e0) tduffy> kDAPL: dapl_evd_free () returns 0x0 tduffy> kDAPL: dapl_evd_free (ffff81001ba164b0) tduffy> kDAPL: dapl_evd_free () returns 0x0 tduffy> kDAPL: dapl_pz_free (ffff81001b9442c8) tduffy> kDAPL: dapl_ia_query (ffff81001ba165c8, 0000000000000000, 0000000000000000, ffff81001bba7b28) tduffy> kDAPL: dapl_ia_query () returns 0x0 tduffy> kDAPL: dapl_ia_close (ffff81001ba165c8, 1) tduffy> kDAPL: dapl_evd_free (ffff81001ba167f8) tduffy> kDAPL: dapl_evd_free () returns 0x0 tduffy> Server_Cmd.debug: 1 tduffy> Server_Cmd.dapl_name: mthca0a tduffy> DT_cs_Server: IA mthca0a opened tduffy> DT_cs_Server: PZ created tduffy> DT_cs_Server: dat_ep_create error: DAT_INVALID_HANDLE tduffy> DT_cs_Server: Waiting for clients to all go away... tduffy> DT_cs_Server: Cleaning up ... tduffy> DT_cs_Server: IA mthca0a closed tduffy> DT_cs_Server (mthca0a): Exiting. tduffy> TEST INSTANCE 0 tduffy> TEST return code = 1 tduffy> tduffy> Also, the ib_at module prints this out now when you ping (after running tduffy> kdapltest)... tduffy> tduffy> ib_at: ib_at_arp_work: Process IB ARP ip <192.168.0.26> gid <0xfe800000000000000002c9010a99e031> tduffy> tduffy> -tduffy tduffy> tduffy> * running x86_64 SMP, 2.6.12-rc4, gcc 4.0.0-6, OpenIB r2414, opensm r2414 2 machines back-2-back tduffy> From johann at pathscale.com Thu May 19 12:43:59 2005 From: johann at pathscale.com (Johann George) Date: Thu, 19 May 2005 12:43:59 -0700 Subject: [openib-general] Infinipath support in OpenIB In-Reply-To: <00b601c55ca9$06325f00$8000000a@blorp> References: <20050519005637.GD7557@esmail.cup.hp.com> <004801c55c34$29e893d0$8000000a@blorp> <20050519182102.GF10318@esmail.cup.hp.com> <00b601c55ca9$06325f00$8000000a@blorp> Message-ID: <20050519194359.GA9315@cuprite.internal.keyresearch.com> Paul: > The article states: 'InfiniPath will also support the OpenIB software stack > providing full InfiniBand compliance.' > > Is this something the OpenIB software developers are working on and > something that can be commented on in public, or should I speculate/ask > Pathscale? Thanks for asking. We are actively working on supporting OpenIB with the help of the OpenIB community and hope to be able to submit code that supports InfiniPath in the near future. Johann From mshefty at ichips.intel.com Thu May 19 12:44:13 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Thu, 19 May 2005 12:44:13 -0700 Subject: [openib-general] Infinipath support in OpenIB In-Reply-To: <00b601c55ca9$06325f00$8000000a@blorp> References: <20050519005637.GD7557@esmail.cup.hp.com> <004801c55c34$29e893d0$8000000a@blorp> <20050519182102.GF10318@esmail.cup.hp.com> <00b601c55ca9$06325f00$8000000a@blorp> Message-ID: <428CEC8D.2090904@ichips.intel.com> Paul Baxter wrote: > I read with interest the PR blurb at > http://supercomputingonline.com/article.php?sid=8740 > regarding InfiniPath's very low latencies and high throughput for MPI > even at very modest message sizes. > > The article states: 'InfiniPath will also support the OpenIB software > stack providing full InfiniBand compliance.' > > Is this something the OpenIB software developers are working on and > something that can be commented on in public, or should I speculate/ask > Pathscale? I would think that as long as they provide an HCA driver that registers with the core layer and exposes the proper APIs, they would be able to run with the openib stack. I do not know if their code is open source however. - Sean From tduffy at sun.com Thu May 19 12:46:04 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 19 May 2005 12:46:04 -0700 Subject: [openib-general] Infinipath support in OpenIB In-Reply-To: <00b601c55ca9$06325f00$8000000a@blorp> References: <20050519005637.GD7557@esmail.cup.hp.com> <004801c55c34$29e893d0$8000000a@blorp> <20050519182102.GF10318@esmail.cup.hp.com> <00b601c55ca9$06325f00$8000000a@blorp> Message-ID: <1116531964.980.13.camel@duffman> On Thu, 2005-05-19 at 20:29 +0100, Paul Baxter wrote: > I read with interest the PR blurb at > http://supercomputingonline.com/article.php?sid=8740 > regarding InfiniPath's very low latencies and high throughput for MPI even > at very modest message sizes. > > The article states: 'InfiniPath will also support the OpenIB software stack > providing full InfiniBand compliance.' Also interesting that OpenIB is suddenly an "industry standard." ;-) -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From jlentini at netapp.com Thu May 19 12:51:47 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 19 May 2005 15:51:47 -0400 (EDT) Subject: [openib-general] kdapltest regression? failing now... In-Reply-To: References: <1116531248.980.10.camel@duffman> Message-ID: For what it's worth, this is the check that we are "failing": qp->sq.max_gs > dev->limits.max_sg ( qp->sq.max_gs + 2 > dev->limits.max_sg is also true but qp->transport == MLX is not). On Thu, 19 May 2005, James Lentini wrote: > > I'm looking into this Tom. > > The following code was added to hw/mthca/mthca_qp.c on Friday > (starting on line 1233): > > > if ((qp->transport == MLX && qp->sq.max_gs + 2 > dev->limits.max_sg) || > qp->sq.max_gs > dev->limits.max_sg || qp->rq.max_gs > dev->limits.max_sg) > return -EINVAL; > > If anyone knows what we have set incorrectly, please let me know. > > Thanks, > james > > On Thu, 19 May 2005, Tom Duffy wrote: > > tduffy> I am not sure when this started, but after updating to top of trunk*, I > tduffy> can no longer get kdapltest to work properly. Both ipoib and sdp are > tduffy> working. > tduffy> > tduffy> Both server and client are returning an error: DAT_INVALID_HANDLE. This > tduffy> is coming from ib_create_qp(). With debugging turned on: > tduffy> > tduffy> [root at flopteron2 ~]# ./kdapltest -T S -D mthca0a -d > tduffy> kDAPL: dapl_ia_open (mthca0a, 8, ffff81000b806308, ffff81000b8062d8) > tduffy> kDAPL: dapl_ia_open () returns 0x0 > tduffy> kDAPL: dapl_pz_create (ffff81001ba165c8, ffff81000b8062e0) > tduffy> kDAPL: dapl_evd_kcreate (ffff81001ba165c8, 8, 1, upcall, 0x20, ffff81000b8062e8) > tduffy> kDAPL: dapl_evd_kcreate (ffff81001ba165c8, 8, 1, upcall, 0xa0, ffff81000b8062f0) > tduffy> kDAPL: dapl_evd_kcreate (ffff81001ba165c8, 8, 1, upcall, 0x10, ffff81000b806300) > tduffy> kDAPL: dapl_evd_kcreate (ffff81001ba165c8, 8, 1, upcall, 0x40, ffff81000b8062f8) > tduffy> kDAPL: dapl_ep_create (ffff81001ba165c8, ffff81001b9442c8, ffff81001ba164b0, ffff81001ba166e0, ffff81001ba22050, 0000000000000000, ffff81000b806318) > tduffy> kDAPL: dapl_ib_qp_alloc: ib_create_qp failed = -22 > tduffy> kDAPL: dapl_evd_free (ffff81001ba22050) > tduffy> kDAPL: dapl_evd_free () returns 0x0 > tduffy> kDAPL: dapl_evd_free (ffff81001ba22168) > tduffy> kDAPL: dapl_evd_free () returns 0x0 > tduffy> kDAPL: dapl_evd_free (ffff81001ba166e0) > tduffy> kDAPL: dapl_evd_free () returns 0x0 > tduffy> kDAPL: dapl_evd_free (ffff81001ba164b0) > tduffy> kDAPL: dapl_evd_free () returns 0x0 > tduffy> kDAPL: dapl_pz_free (ffff81001b9442c8) > tduffy> kDAPL: dapl_ia_query (ffff81001ba165c8, 0000000000000000, 0000000000000000, ffff81001bba7b28) > tduffy> kDAPL: dapl_ia_query () returns 0x0 > tduffy> kDAPL: dapl_ia_close (ffff81001ba165c8, 1) > tduffy> kDAPL: dapl_evd_free (ffff81001ba167f8) > tduffy> kDAPL: dapl_evd_free () returns 0x0 > tduffy> Server_Cmd.debug: 1 > tduffy> Server_Cmd.dapl_name: mthca0a > tduffy> DT_cs_Server: IA mthca0a opened > tduffy> DT_cs_Server: PZ created > tduffy> DT_cs_Server: dat_ep_create error: DAT_INVALID_HANDLE > tduffy> DT_cs_Server: Waiting for clients to all go away... > tduffy> DT_cs_Server: Cleaning up ... > tduffy> DT_cs_Server: IA mthca0a closed > tduffy> DT_cs_Server (mthca0a): Exiting. > tduffy> TEST INSTANCE 0 > tduffy> TEST return code = 1 > tduffy> > tduffy> Also, the ib_at module prints this out now when you ping (after running > tduffy> kdapltest)... > tduffy> > tduffy> ib_at: ib_at_arp_work: Process IB ARP ip <192.168.0.26> gid <0xfe800000000000000002c9010a99e031> > tduffy> > tduffy> -tduffy > tduffy> > tduffy> * running x86_64 SMP, 2.6.12-rc4, gcc 4.0.0-6, OpenIB r2414, opensm r2414 2 machines back-2-back > tduffy> > From tduffy at sun.com Thu May 19 13:09:13 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 19 May 2005 13:09:13 -0700 Subject: [openib-general] RE: [PATCH][kdapl] fix spin_lock_irqsave/spin_unlock_irqrestore i mplementation In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35B46@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35B46@mtvex01.mtv.mtl.com> Message-ID: <1116533353.980.18.camel@duffman> On Wed, 2005-05-18 at 22:44 +0300, Itamar Rabenstein wrote: > "evd producer locking" is something that we need in openib kdapl > as it was in Source Forge implementation. This is not a good excuse. OpenIB kdapl is a totally different beast from the sourceforge implementation and will differ over time. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From Mark.Seger at hp.com Thu May 19 13:11:48 2005 From: Mark.Seger at hp.com (Mark Seger) Date: Thu, 19 May 2005 16:11:48 -0400 Subject: [openib-general] performance counters in /sys Message-ID: <428CF304.4000201@hp.com> I've looked through the archives for more details on this topic and while there were some interesting discussions during Sept 2004, none of them really seemed to touch on this specific topic - programmatic access to performance counters. If this has already been discussed elsewhere a pointer would be appreciated. The current location for the IB counters is: /sys/class/infiniband/mthca0/ports/1/counters and within that directory there, there are 16 counters. If I want to programmatically access all the counters for all the hcas, this means I have to go to 2 directories (one per port) per hca and then open/read 16 files in each. If I want to do that with any kind of frequency, say once every second or so, that's a lot of extra work that I'd like to cut down on. I believe there are probably several scenarios of how people may want to look at the data and for a human interface would think something like /proc/meminfo style would at least reduce the number of reads by a factor of 16, which is certainly better. However, this means that programmatically I still need to match a line in the file with a particular item of data that I'm interested in and that means comparing each line's label to known values which does take some extra overhead since I potentially have to do over 100 compares for each set of 16 counters. Better yet, a format such as /proc/net/dev shows one line per network interface followed by all the counters, space separated for human consumption. This can be argued to be better but as the number of counters grow, scaling becomes an issue because when cat'ing the file, you get into word wrap and readability becomes difficult if not impossible. Yet a more compact format is that of /proc/pid/stat in which all the fields are simply separated by a single space with no attempt to read the output by anything other than a program. Personally that would be my vote. And finally there's the notion of having a version number associated with the data in case it changes so the consumer will know what to do with the new format as is done with /proc/slabinfo. So based on all that I'd ask the question of what people would think of adding a new structure, perhaps under the directory /sys/class/infiniband (maintaining the old for those who feel it is more suitable for human consumption) that looks something like: Vx.y hca0 1 off ctr1 ctr2 ctr3 ... hca0 2 on ctr1 ctr2 ctr3 ... hca1 1 on ctr1 ctr2 ctr3 ... hca1 2 off ctr1 ctr2 ctr3 ... in which 2 ports are on and 2 off, followed by the 16 counters associated with them The only other thing that could be useful would be an extra field for the protocol, such that for a given interface/port, I could see the traffic counters for each type of protocol that one might choose to support, such as mpi, portals, etc. personally, I think one size (programmatic vs human readable) doesn't fit all although one could argue the need for both and I certainly wouldn't object. however, if I had to pick only one I'd pick programmatic since counters can be extremely difficult to interpret when they're continually increasing and I for one can't do subtraction of large numbers that quickly in my head. 8-) comments? opinions? -mark From halr at voltaire.com Thu May 19 13:22:23 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 19 May 2005 16:22:23 -0400 Subject: [openib-general] performance counters in /sys In-Reply-To: <428CF304.4000201@hp.com> References: <428CF304.4000201@hp.com> Message-ID: <1116534143.4479.179.camel@hal.voltaire.com> On Thu, 2005-05-19 at 16:11, Mark Seger wrote: > I've looked through the archives for more details on this topic and > while there were some interesting discussions during Sept 2004, none of > them really seemed to touch on this specific topic - programmatic access > to performance counters. If this has already been discussed elsewhere a > pointer would be appreciated. > > The current location for the IB counters is: > > /sys/class/infiniband/mthca0/ports/1/counters > > and within that directory there, there are 16 counters. These represent the IB mandatory port counters. That's the driver version of this. There are other ways to get at this. There is a perfquery tool which queries the PMA via a PerfGet(PortCounters): PortSelect:......................1 CounterSelect:...................0x0000 SymbolErrors:....................0 LinkRecovers:....................0 LinkDowned:......................0 RcvErrors:.......................0 RcvRemotePhysErrors:.............0 RcvSwRelayErrors:................0 XmtDiscards:.....................0 XmtConstraintErrors:.............0 RcvConstraintErrors:.............0 LinkIntegrityErrors:.............0 ExcBufOverrunErrors:.............0 Vl15Dropped:.....................0 XmtBytes:........................0 RcvBytes:........................0 XmtPkts:.........................0 RcvBytes:........................0 > If I want to > programmatically access all the counters for all the hcas, There may be other counters beyond these mandatory ones. > this means I > have to go to 2 directories (one per port) per hca and then open/read 16 > files in each. If I want to do that with any kind of frequency, say > once every second or so, that's a lot of extra work that I'd like to cut > down on. > > I believe there are probably several scenarios of how people may want to > look at the data and for a human interface would think something like > /proc/meminfo style would at least reduce the number of reads by a > factor of 16, which is certainly better. However, this means that > programmatically I still need to match a line in the file with a > particular item of data that I'm interested in and that means comparing > each line's label to known values which does take some extra overhead > since I potentially have to do over 100 compares for each set of 16 > counters. > > Better yet, a format such as /proc/net/dev shows one line per network > interface followed by all the counters, space separated for human > consumption. This can be argued to be better but as the number of > counters grow, scaling becomes an issue because when cat'ing the file, > you get into word wrap and readability becomes difficult if not impossible. > > Yet a more compact format is that of /proc/pid/stat in which all the > fields are simply separated by a single space with no attempt to read > the output by anything other than a program. Personally that would be > my vote. > > And finally there's the notion of having a version number associated > with the data in case it changes so the consumer will know what to do > with the new format as is done with /proc/slabinfo. > > So based on all that I'd ask the question of what people would think of > adding a new structure, perhaps under the directory > /sys/class/infiniband (maintaining the old for those who feel it is more > suitable for human consumption) that looks something like: > > Vx.y > hca0 1 off ctr1 ctr2 ctr3 ... > hca0 2 on ctr1 ctr2 ctr3 ... > hca1 1 on ctr1 ctr2 ctr3 ... > hca1 2 off ctr1 ctr2 ctr3 ... > > in which 2 ports are on and 2 off, followed by the 16 counters > associated with them > > The only other thing that could be useful would be an extra field for > the protocol, such that for a given interface/port, I could see the > traffic counters for each type of protocol that one might choose to > support, such as mpi, portals, etc. There are no hardware counters for these. These would need to be filled in somehow by software. > personally, I think one size (programmatic vs human readable) doesn't > fit all although one could argue the need for both and I certainly > wouldn't object. however, if I had to pick only one I'd pick > programmatic since counters can be extremely difficult to interpret when > they're continually increasing and I for one can't do subtraction of > large numbers that quickly in my head. 8-) > > comments? opinions? > > -mark > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From jlentini at netapp.com Thu May 19 13:30:53 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 19 May 2005 16:30:53 -0400 (EDT) Subject: [openib-general] kdapltest regression? failing now... In-Reply-To: References: <1116531248.980.10.camel@duffman> Message-ID: I think I figure this out. DAPL was assuming a particular maximum scatter gather list size. I'm going to change it to query for this value. Hopefully I'll have a fix shortly. james On Thu, 19 May 2005, James Lentini wrote: > > For what it's worth, this is the check that we are "failing": > > qp->sq.max_gs > dev->limits.max_sg > > ( qp->sq.max_gs + 2 > dev->limits.max_sg is also true but > qp->transport == MLX is not). > > On Thu, 19 May 2005, James Lentini wrote: > >> >> I'm looking into this Tom. >> >> The following code was added to hw/mthca/mthca_qp.c on Friday >> (starting on line 1233): >> >> >> if ((qp->transport == MLX && qp->sq.max_gs + 2 > dev->limits.max_sg) || >> qp->sq.max_gs > dev->limits.max_sg || qp->rq.max_gs > >> dev->limits.max_sg) >> return -EINVAL; >> >> If anyone knows what we have set incorrectly, please let me know. >> >> Thanks, >> james >> >> On Thu, 19 May 2005, Tom Duffy wrote: >> >> tduffy> I am not sure when this started, but after updating to top of >> trunk*, I >> tduffy> can no longer get kdapltest to work properly. Both ipoib and sdp >> are >> tduffy> working. >> tduffy> >> tduffy> Both server and client are returning an error: DAT_INVALID_HANDLE. >> This >> tduffy> is coming from ib_create_qp(). With debugging turned on: >> tduffy> >> tduffy> [root at flopteron2 ~]# ./kdapltest -T S -D mthca0a -d >> tduffy> kDAPL: dapl_ia_open (mthca0a, 8, ffff81000b806308, >> ffff81000b8062d8) >> tduffy> kDAPL: dapl_ia_open () returns 0x0 >> tduffy> kDAPL: dapl_pz_create (ffff81001ba165c8, ffff81000b8062e0) >> tduffy> kDAPL: dapl_evd_kcreate (ffff81001ba165c8, 8, 1, upcall, 0x20, >> ffff81000b8062e8) >> tduffy> kDAPL: dapl_evd_kcreate (ffff81001ba165c8, 8, 1, upcall, 0xa0, >> ffff81000b8062f0) >> tduffy> kDAPL: dapl_evd_kcreate (ffff81001ba165c8, 8, 1, upcall, 0x10, >> ffff81000b806300) >> tduffy> kDAPL: dapl_evd_kcreate (ffff81001ba165c8, 8, 1, upcall, 0x40, >> ffff81000b8062f8) >> tduffy> kDAPL: dapl_ep_create (ffff81001ba165c8, ffff81001b9442c8, >> ffff81001ba164b0, ffff81001ba166e0, ffff81001ba22050, 0000000000000000, >> ffff81000b806318) >> tduffy> kDAPL: dapl_ib_qp_alloc: ib_create_qp failed = -22 >> tduffy> kDAPL: dapl_evd_free (ffff81001ba22050) >> tduffy> kDAPL: dapl_evd_free () returns 0x0 >> tduffy> kDAPL: dapl_evd_free (ffff81001ba22168) >> tduffy> kDAPL: dapl_evd_free () returns 0x0 >> tduffy> kDAPL: dapl_evd_free (ffff81001ba166e0) >> tduffy> kDAPL: dapl_evd_free () returns 0x0 >> tduffy> kDAPL: dapl_evd_free (ffff81001ba164b0) >> tduffy> kDAPL: dapl_evd_free () returns 0x0 >> tduffy> kDAPL: dapl_pz_free (ffff81001b9442c8) >> tduffy> kDAPL: dapl_ia_query (ffff81001ba165c8, 0000000000000000, >> 0000000000000000, ffff81001bba7b28) >> tduffy> kDAPL: dapl_ia_query () returns 0x0 >> tduffy> kDAPL: dapl_ia_close (ffff81001ba165c8, 1) >> tduffy> kDAPL: dapl_evd_free (ffff81001ba167f8) >> tduffy> kDAPL: dapl_evd_free () returns 0x0 >> tduffy> Server_Cmd.debug: 1 >> tduffy> Server_Cmd.dapl_name: mthca0a >> tduffy> DT_cs_Server: IA mthca0a opened >> tduffy> DT_cs_Server: PZ created >> tduffy> DT_cs_Server: dat_ep_create error: DAT_INVALID_HANDLE >> tduffy> DT_cs_Server: Waiting for clients to all go away... >> tduffy> DT_cs_Server: Cleaning up ... >> tduffy> DT_cs_Server: IA mthca0a closed >> tduffy> DT_cs_Server (mthca0a): Exiting. >> tduffy> TEST INSTANCE 0 >> tduffy> TEST return code = 1 >> tduffy> >> tduffy> Also, the ib_at module prints this out now when you ping (after >> running >> tduffy> kdapltest)... >> tduffy> >> tduffy> ib_at: ib_at_arp_work: Process IB ARP ip <192.168.0.26> gid >> <0xfe800000000000000002c9010a99e031> >> tduffy> >> tduffy> -tduffy >> tduffy> >> tduffy> * running x86_64 SMP, 2.6.12-rc4, gcc 4.0.0-6, OpenIB r2414, opensm >> r2414 2 machines back-2-back >> tduffy> >> > From itamar at mellanox.co.il Thu May 19 14:04:31 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Fri, 20 May 2005 00:04:31 +0300 Subject: [openib-general] RE: [PATCH][kdapl] fix spin_lock_irqsave/spi n_unlock_irqrestore i mplementation Message-ID: <91DB792C7985D411BEC300B40080D29CC35B6D@mtvex01.mtv.mtl.com> I am not saying that we need it in OpenIb code because it was in SF code I said that in OpenIb implementation we need it the same as in SF we need it. we must have this lock because for example : if the same evd will be a CM evd for 2 ep's each one on different Thread both can try to add an event to the evd in the same time there for we need to lock the evd when we take an empty event from one list (empty list) and to unlock it after we add the event to the second list (waiting events list) the lock is in one function and the unlock in the second function. so we need in out OpenIb code and also SF code need it (;-) Itamar > -----Original Message----- > From: Tom Duffy [mailto:tduffy at sun.com] > Sent: Thursday, May 19, 2005 11:09 PM > To: Itamar Rabenstein > Cc: James Lentini; openib-general > Subject: Re: [openib-general] RE: [PATCH][kdapl] fix > spin_lock_irqsave/spin_unlock_irqrestore i mplementation > > > On Wed, 2005-05-18 at 22:44 +0300, Itamar Rabenstein wrote: > > "evd producer locking" is something that we need in openib kdapl > > as it was in Source Forge implementation. > > This is not a good excuse. OpenIB kdapl is a totally different beast > from the sourceforge implementation and will differ over time. > > -tduffy > From yaronh at voltaire.com Thu May 19 13:45:32 2005 From: yaronh at voltaire.com (Yaron Haviv) Date: Thu, 19 May 2005 23:45:32 +0300 Subject: [openib-general] performance counters in /sys Message-ID: <35EA21F54A45CB47B879F21A91F4862F581222@taurus.voltaire.com> > -----Original Message----- > From: openib-general-bounces at openib.org [mailto:openib-general- > bounces at openib.org] On Behalf Of Hal Rosenstock > Sent: Thursday, May 19, 2005 11:22 PM > > On Thu, 2005-05-19 at 16:11, Mark Seger wrote: > > > > The only other thing that could be useful would be an extra field for > > the protocol, such that for a given interface/port, I could see the > > traffic counters for each type of protocol that one might choose to > > support, such as mpi, portals, etc. > > There are no hardware counters for these. These would need to be filled > in somehow by software. > Mark/Hal, I believe you can use the per VL counters for that (IB allows counting traffic on a specific VL) By matching ULPs to VLs (e.g. through the ib_at lib we suggested) You can get both congestion isolation per traffic type as well as the ability to count traffic per ULP (note that up to 8 VLs are supported in the Mellanox chips) Yaron From halr at voltaire.com Thu May 19 14:34:16 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 19 May 2005 17:34:16 -0400 Subject: [openib-general] [PATCH] user_mad: Support RMPP on send side Message-ID: <1116538456.4479.236.camel@hal.voltaire.com> user_mad: Support RMPP on send side Note that this change will need a coordinated change to OpenSM and some userspace/management libraries which will be done as soon as possible once this patch is accepted. Receive side support for RMPP will be added separately. Signed-off-by: Hal Rosenstock Index: infiniband/include/ib_user_mad.h =================================================================== --- infiniband/include/ib_user_mad.h (revision 2413) +++ infiniband/include/ib_user_mad.h (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Voltaire, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -42,7 +43,7 @@ * Increment this value if any changes that break userspace ABI * compatibility are made. */ -#define IB_USER_MAD_ABI_VERSION 2 +#define IB_USER_MAD_ABI_VERSION 3 /* * Make sure that all structs defined in this file remain laid out so @@ -51,8 +52,7 @@ */ /** - * ib_user_mad - MAD packet - * @data - Contents of MAD + * ib_user_mad_hdr - MAD packet header * @id - ID of agent MAD received with/to be sent with * @status - 0 on successful receive, ETIMEDOUT if no response * received (transaction ID in data[] will be set to TID of original @@ -72,8 +72,7 @@ * * All multi-byte quantities are stored in network (big endian) byte order. */ -struct ib_user_mad { - __u8 data[256]; +struct ib_user_mad_hdr { __u32 id; __u32 status; __u32 timeout_ms; @@ -91,6 +90,17 @@ }; /** + * ib_user_mad - MAD packet + * @hdr - MAD packet header + * @data - Contents of MAD + * + */ +struct ib_user_mad { + struct ib_user_mad_hdr hdr; + __u8 data[0]; +}; + +/** * ib_user_mad_reg_req - MAD registration request * @id - Set by the kernel; used to identify agent in future requests. * @qpn - Queue pair number; must be 0 or 1. @@ -103,6 +113,8 @@ * management class to receive. * @oui: Indicates IEEE OUI when mgmt_class is a vendor class * in the range from 0x30 to 0x4f. Otherwise not used. + * @rmpp_version: If set, indicates the RMPP version used. + * */ struct ib_user_mad_reg_req { __u32 id; @@ -111,6 +123,7 @@ __u8 mgmt_class; __u8 mgmt_class_version; __u8 oui[3]; + __u8 rmpp_version; }; #define IB_IOCTL_MAGIC 0x1b Index: infiniband/core/user_mad.c =================================================================== --- infiniband/core/user_mad.c (revision 2413) +++ infiniband/core/user_mad.c (working copy) @@ -1,5 +1,6 @@ /* * Copyright (c) 2004 Topspin Communications. All rights reserved. + * Copyright (c) 2005 Voltaire, Inc. All rights reserved. * * This software is available to you under a choice of one of two * licenses. You may choose to be licensed under the terms of the GNU @@ -94,10 +95,12 @@ }; struct ib_umad_packet { - struct ib_user_mad mad; struct ib_ah *ah; + struct ib_mad_send_buf *msg; struct list_head list; + int length; DECLARE_PCI_UNMAP_ADDR(mapping) + struct ib_user_mad mad; }; static const dev_t base_dev = MKDEV(IB_UMAD_MAJOR, IB_UMAD_MINOR_BASE); @@ -114,10 +117,10 @@ int ret = 1; down_read(&file->agent_mutex); - for (packet->mad.id = 0; - packet->mad.id < IB_UMAD_MAX_AGENTS; - packet->mad.id++) - if (agent == file->agent[packet->mad.id]) { + for (packet->mad.hdr.id = 0; + packet->mad.hdr.id < IB_UMAD_MAX_AGENTS; + packet->mad.hdr.id++) + if (agent == file->agent[packet->mad.hdr.id]) { spin_lock_irq(&file->recv_lock); list_add_tail(&packet->list, &file->recv_list); spin_unlock_irq(&file->recv_lock); @@ -138,14 +141,11 @@ struct ib_umad_packet *packet = (void *) (unsigned long) send_wc->wr_id; - dma_unmap_single(agent->device->dma_device, - pci_unmap_addr(packet, mapping), - sizeof packet->mad.data, - DMA_TO_DEVICE); - ib_destroy_ah(packet->ah); + ib_free_send_mad(packet->msg); + ib_destroy_ah(packet->msg->send_wr.wr.ud.ah); if (send_wc->status == IB_WC_RESP_TIMEOUT_ERR) { - packet->mad.status = ETIMEDOUT; + packet->mad.hdr.status = ETIMEDOUT; if (!queue_packet(file, agent, packet)) return; @@ -159,30 +159,34 @@ { struct ib_umad_file *file = agent->context; struct ib_umad_packet *packet; + int length; + if (mad_recv_wc->wc->status != IB_WC_SUCCESS) goto out; - packet = kmalloc(sizeof *packet, GFP_KERNEL); + length = 256; /* until RMPP is supported */ + packet = kmalloc(sizeof *packet + length, GFP_KERNEL); if (!packet) goto out; - memset(packet, 0, sizeof *packet); + memset(packet, 0, sizeof *packet + length); + packet->length = length; - memcpy(packet->mad.data, mad_recv_wc->recv_buf.mad, sizeof packet->mad.data); - packet->mad.status = 0; - packet->mad.qpn = cpu_to_be32(mad_recv_wc->wc->src_qp); - packet->mad.lid = cpu_to_be16(mad_recv_wc->wc->slid); - packet->mad.sl = mad_recv_wc->wc->sl; - packet->mad.path_bits = mad_recv_wc->wc->dlid_path_bits; - packet->mad.grh_present = !!(mad_recv_wc->wc->wc_flags & IB_WC_GRH); - if (packet->mad.grh_present) { + memcpy(packet->mad.data, mad_recv_wc->recv_buf.mad, length); + packet->mad.hdr.status = 0; + packet->mad.hdr.qpn = cpu_to_be32(mad_recv_wc->wc->src_qp); + packet->mad.hdr.lid = cpu_to_be16(mad_recv_wc->wc->slid); + packet->mad.hdr.sl = mad_recv_wc->wc->sl; + packet->mad.hdr.path_bits = mad_recv_wc->wc->dlid_path_bits; + packet->mad.hdr.grh_present = !!(mad_recv_wc->wc->wc_flags & IB_WC_GRH); + if (packet->mad.hdr.grh_present) { /* XXX parse GRH */ - packet->mad.gid_index = 0; - packet->mad.hop_limit = 0; - packet->mad.traffic_class = 0; - memset(packet->mad.gid, 0, 16); - packet->mad.flow_label = 0; + packet->mad.hdr.gid_index = 0; + packet->mad.hdr.hop_limit = 0; + packet->mad.hdr.traffic_class = 0; + memset(packet->mad.hdr.gid, 0, 16); + packet->mad.hdr.flow_label = 0; } if (queue_packet(file, agent, packet)) @@ -199,7 +203,7 @@ struct ib_umad_packet *packet; ssize_t ret; - if (count < sizeof (struct ib_user_mad)) + if (count < sizeof (struct ib_user_mad) + 256) /* until RMPP supported */ return -EINVAL; spin_lock_irq(&file->recv_lock); @@ -222,12 +226,20 @@ spin_unlock_irq(&file->recv_lock); - if (copy_to_user(buf, &packet->mad, sizeof packet->mad)) + if (count < packet->length + sizeof (struct ib_user_mad)) + ret = -EINVAL; + else if (copy_to_user(buf, &packet->mad, + packet->length + sizeof (struct ib_user_mad))) ret = -EFAULT; else - ret = sizeof packet->mad; - - kfree(packet); + ret = packet->length + sizeof (struct ib_user_mad); + if (ret < 0) { + /* Requeue packet */ + spin_lock_irq(&file->recv_lock); + list_add(&packet->list, &file->recv_list); + spin_unlock_irq(&file->recv_lock); + } else + kfree(packet); return ret; } @@ -238,107 +250,155 @@ struct ib_umad_packet *packet; struct ib_mad_agent *agent; struct ib_ah_attr ah_attr; - struct ib_sge gather_list; - struct ib_send_wr *bad_wr, wr = { - .opcode = IB_WR_SEND, - .sg_list = &gather_list, - .num_sge = 1, - .send_flags = IB_SEND_SIGNALED, - }; + struct ib_send_wr *bad_wr; + struct ib_rmpp_mad *rmpp_mad; u8 method; u64 *tid; - int ret; + int ret, length, hdr_len, data_len, rmpp_hdr_size; + int rmpp_active = 0; if (count < sizeof (struct ib_user_mad)) return -EINVAL; - packet = kmalloc(sizeof *packet, GFP_KERNEL); + length = count - sizeof (struct ib_user_mad); + packet = kmalloc(sizeof *packet + sizeof(struct ib_mad_hdr) + + sizeof(struct ib_rmpp_hdr), GFP_KERNEL); if (!packet) return -ENOMEM; - if (copy_from_user(&packet->mad, buf, sizeof packet->mad)) { - kfree(packet); - return -EFAULT; + if (copy_from_user(&packet->mad, buf, + sizeof (struct ib_user_mad) + + sizeof(struct ib_mad_hdr) + + sizeof(struct ib_rmpp_hdr))) { + ret = -EFAULT; + goto err; } - if (packet->mad.id < 0 || packet->mad.id >= IB_UMAD_MAX_AGENTS) { + if (packet->mad.hdr.id < 0 || + packet->mad.hdr.id >= IB_UMAD_MAX_AGENTS) { ret = -EINVAL; goto err; } + packet->length = length; + down_read(&file->agent_mutex); - agent = file->agent[packet->mad.id]; + agent = file->agent[packet->mad.hdr.id]; if (!agent) { ret = -EINVAL; goto err_up; } + memset(&ah_attr, 0, sizeof ah_attr); + ah_attr.dlid = be16_to_cpu(packet->mad.hdr.lid); + ah_attr.sl = packet->mad.hdr.sl; + ah_attr.src_path_bits = packet->mad.hdr.path_bits; + ah_attr.port_num = file->port->port_num; + if (packet->mad.hdr.grh_present) { + ah_attr.ah_flags = IB_AH_GRH; + memcpy(ah_attr.grh.dgid.raw, packet->mad.hdr.gid, 16); + ah_attr.grh.flow_label = packet->mad.hdr.flow_label; + ah_attr.grh.hop_limit = packet->mad.hdr.hop_limit; + ah_attr.grh.traffic_class = packet->mad.hdr.traffic_class; + } + + packet->ah = ib_create_ah(agent->qp->pd, &ah_attr); + if (IS_ERR(packet->ah)) { + ret = PTR_ERR(packet->ah); + goto err_up; + } + + rmpp_mad = (struct ib_rmpp_mad *) packet->mad.data; + if (ib_get_rmpp_flags(&rmpp_mad->rmpp_hdr) & IB_MGMT_RMPP_FLAG_ACTIVE) { + /* RMPP active */ + if (!agent->rmpp_version) { + ret = -EINVAL; + goto err_ah; + } + /* Validate that management class can support RMPP */ + if (rmpp_mad->mad_hdr.mgmt_class == IB_MGMT_CLASS_SUBN_ADM) { + hdr_len = offsetof(struct ib_sa_mad, data); + data_len = length; + } else if ((rmpp_mad->mad_hdr.mgmt_class >= IB_MGMT_CLASS_VENDOR_RANGE2_START) && + (rmpp_mad->mad_hdr.mgmt_class <= IB_MGMT_CLASS_VENDOR_RANGE2_END)) { + hdr_len = offsetof(struct ib_vendor_mad, data); + data_len = length - hdr_len; + } else { + ret = -EINVAL; + goto err_ah; + } + rmpp_active = 1; + } else { + if (length > sizeof(struct ib_mad)) { + ret = -EINVAL; + goto err_ah; + } + hdr_len = offsetof(struct ib_mad, data); + data_len = length - hdr_len; + } + + packet->msg = ib_create_send_mad(agent, + be32_to_cpu(packet->mad.hdr.qpn), + 0, packet->ah, rmpp_active, + hdr_len, data_len, + GFP_KERNEL); + if (IS_ERR(packet->msg)) { + ret = PTR_ERR(packet->msg); + goto err_ah; + } + + packet->msg->send_wr.wr.ud.retries = 0; + packet->msg->send_wr.wr.ud.timeout_ms = packet->mad.hdr.timeout_ms; + /* Override send WR WRID created by ib_create_send_mad */ + packet->msg->send_wr.wr_id = (unsigned long) packet; + + if (!rmpp_active) { + /* Copy message from user into send buffer */ + copy_from_user(packet->msg->mad, + buf + sizeof(struct ib_user_mad), length); + } else { + rmpp_hdr_size = sizeof(struct ib_mad_hdr) + + sizeof(struct ib_rmpp_hdr); + /* Only copy MAD headers (RMPP header in place) */ + memcpy(packet->msg->mad, packet->mad.data, + sizeof(struct ib_mad_hdr)); + /* Now, copy rest of message from user into send buffer */ + copy_from_user(((struct ib_rmpp_mad *)packet->msg->mad)->data, + buf + sizeof(struct ib_user_mad) + rmpp_hdr_size, + length - rmpp_hdr_size); + } + /* * If userspace is generating a request that will generate a * response, we need to make sure the high-order part of the * transaction ID matches the agent being used to send the * MAD. */ - method = ((struct ib_mad_hdr *) packet->mad.data)->method; + method = packet->msg->mad->mad_hdr.method; if (!(method & IB_MGMT_METHOD_RESP) && method != IB_MGMT_METHOD_TRAP_REPRESS && method != IB_MGMT_METHOD_SEND) { - tid = &((struct ib_mad_hdr *) packet->mad.data)->tid; + tid = &packet->msg->mad->mad_hdr.tid; *tid = cpu_to_be64(((u64) agent->hi_tid) << 32 | (be64_to_cpup(tid) & 0xffffffff)); } - memset(&ah_attr, 0, sizeof ah_attr); - ah_attr.dlid = be16_to_cpu(packet->mad.lid); - ah_attr.sl = packet->mad.sl; - ah_attr.src_path_bits = packet->mad.path_bits; - ah_attr.port_num = file->port->port_num; - if (packet->mad.grh_present) { - ah_attr.ah_flags = IB_AH_GRH; - memcpy(ah_attr.grh.dgid.raw, packet->mad.gid, 16); - ah_attr.grh.flow_label = packet->mad.flow_label; - ah_attr.grh.hop_limit = packet->mad.hop_limit; - ah_attr.grh.traffic_class = packet->mad.traffic_class; - } + ret = ib_post_send_mad(agent, &packet->msg->send_wr, &bad_wr); + if (ret) + goto err_msg; - packet->ah = ib_create_ah(agent->qp->pd, &ah_attr); - if (IS_ERR(packet->ah)) { - ret = PTR_ERR(packet->ah); - goto err_up; - } + up_read(&file->agent_mutex); - gather_list.addr = dma_map_single(agent->device->dma_device, - packet->mad.data, - sizeof packet->mad.data, - DMA_TO_DEVICE); - gather_list.length = sizeof packet->mad.data; - gather_list.lkey = file->mr[packet->mad.id]->lkey; - pci_unmap_addr_set(packet, mapping, gather_list.addr); + return sizeof (struct ib_user_mad_hdr) + packet->length; - wr.wr.ud.mad_hdr = (struct ib_mad_hdr *) packet->mad.data; - wr.wr.ud.ah = packet->ah; - wr.wr.ud.remote_qpn = be32_to_cpu(packet->mad.qpn); - wr.wr.ud.remote_qkey = be32_to_cpu(packet->mad.qkey); - wr.wr.ud.timeout_ms = packet->mad.timeout_ms; - wr.wr.ud.retries = 0; +err_msg: + ib_free_send_mad(packet->msg); - wr.wr_id = (unsigned long) packet; +err_ah: + ib_destroy_ah(packet->ah); - ret = ib_post_send_mad(agent, &wr, &bad_wr); - if (ret) { - dma_unmap_single(agent->device->dma_device, - pci_unmap_addr(packet, mapping), - sizeof packet->mad.data, - DMA_TO_DEVICE); - goto err_up; - } - - up_read(&file->agent_mutex); - - return sizeof packet->mad; - err_up: up_read(&file->agent_mutex); @@ -400,7 +460,8 @@ agent = ib_register_mad_agent(file->port->ib_dev, file->port->port_num, ureq.qpn ? IB_QPT_GSI : IB_QPT_SMI, ureq.mgmt_class ? &req : NULL, - 0, send_handler, recv_handler, file); + ureq.rmpp_version, + send_handler, recv_handler, file); if (IS_ERR(agent)) { ret = PTR_ERR(agent); goto out; Index: docs/user_mad.txt =================================================================== --- docs/user_mad.txt (revision 2413) +++ docs/user_mad.txt (working copy) @@ -28,12 +28,14 @@ Receiving MADs - MADs are received using read(). The buffer passed to read() must be - large enough to hold at least one struct ib_user_mad. For example: + MADs are received using read(). The receive side does not currently + support RMPP so the buffer passed to read() must be at least one + struct ib_user_mad + 256 bytes. For example: - struct ib_user_mad mad; - ret = read(fd, &mad, sizeof mad); - if (ret != sizeof mad) + struct ib_user_mad *mad; + mad = malloc(sizeof *mad + 256); + ret = read(fd, mad, sizeof *mad + 256); + if (ret != sizeof mad + 256) perror("read"); In addition to the actual MAD contents, the other struct ib_user_mad @@ -50,18 +52,21 @@ MADs are sent using write(). The agent ID for sending should be filled into the id field of the MAD, the destination LID should be - filled into the lid field, and so on. For example: + filled into the lid field, and so on. The send side does support + RMPP so arbitrary length MAD can be sent. For example: - struct ib_user_mad mad; + struct ib_user_mad *mad; - /* fill in mad.data */ + mad = malloc(sizeof *mad + mad_length); - mad.id = my_agent; /* req.id from agent registration */ - mad.lid = my_dest; /* in network byte order... */ + /* fill in mad->data */ + + mad->hdr.id = my_agent; /* req.id from agent registration */ + mad->hdr.lid = my_dest; /* in network byte order... */ /* etc. */ - ret = write(fd, &mad, sizeof mad); - if (ret != sizeof mad) + ret = write(fd, &mad, sizeof *mad + mad_length); + if (ret != sizeof *mad + mad_length) perror("write"); Setting IsSM Capability Bit From halr at voltaire.com Thu May 19 14:42:00 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 19 May 2005 17:42:00 -0400 Subject: [openib-general] Re: [RFC] [PATCH] user_mad: Support RMPP on send side In-Reply-To: <5264xgf85t.fsf@topspin.com> References: <1115845653.4490.188.camel@localhost.localdomain> <5264xgf85t.fsf@topspin.com> Message-ID: <1116538920.4479.247.camel@hal.voltaire.com> On Wed, 2005-05-18 at 19:04, Roland Dreier wrote: > This looks OK to check in with one small comment on the following: > > - if (copy_to_user(buf, &packet->mad, sizeof packet->mad)) > + if (copy_to_user(buf, &packet->mad, > + min(count, packet->length + > + sizeof (struct ib_user_mad)))) > ret = -EFAULT; > else > - ret = sizeof packet->mad; > + ret = count; > > This code will truncate a received MAD that is bigger than the buffer > passed into read(), but return the full size of the packet. I don't > think read() is allowed to do this: the return value can be at most > the count value passed in by the user. > > I think we have two options: truncate and return the actual amount of > data read to the user, or return an error if the user's buffer is too > small. OK; I just reissued the patch with the real length transferred. Also in the case of too short buffer or a copy_to_user error, the packet is requeued. -- Hal From halr at voltaire.com Thu May 19 14:48:21 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 19 May 2005 17:48:21 -0400 Subject: [openib-general] performance counters in /sys In-Reply-To: <35EA21F54A45CB47B879F21A91F4862F581222@taurus.voltaire.com> References: <35EA21F54A45CB47B879F21A91F4862F581222@taurus.voltaire.com> Message-ID: <1116539301.4479.255.camel@hal.voltaire.com> On Thu, 2005-05-19 at 16:45, Yaron Haviv wrote: > I believe you can use the per VL counters for that > (IB allows counting traffic on a specific VL) > By matching ULPs to VLs (e.g. through the ib_at lib we suggested) > You can get both congestion isolation per traffic type as well as the > ability to count traffic per ULP > (note that up to 8 VLs are supported in the Mellanox chips) PortXmitDataVL[n], PortRcvDataVL[n], PortXmitPktVL[n], and PortRcvPktVL[n] are all IB optional. Do the Mellanox HCAs support these counters ? -- Hal From Diego at Mellanox.com Thu May 19 15:01:08 2005 From: Diego at Mellanox.com (Diego Crupnicoff) Date: Thu, 19 May 2005 15:01:08 -0700 Subject: [openib-general] performance counters in /sys Message-ID: <25AE7F432672D511B8DC00B0D0DF11DA04D5738B@MTIEX01> > -----Original Message----- > From: Hal Rosenstock [mailto:halr at voltaire.com] > Sent: Thursday, May 19, 2005 6:48 PM > To: Yaron Haviv > Cc: Mark Seger; openib-general at openib.org > Subject: RE: [openib-general] performance counters in /sys > > > On Thu, 2005-05-19 at 16:45, Yaron Haviv wrote: > > I believe you can use the per VL counters for that > > (IB allows counting traffic on a specific VL) > > By matching ULPs to VLs (e.g. through the ib_at lib we suggested) > > You can get both congestion isolation per traffic type as > well as the > > ability to count traffic per ULP > > (note that up to 8 VLs are supported in the Mellanox chips) > > PortXmitDataVL[n], PortRcvDataVL[n], PortXmitPktVL[n], and > PortRcvPktVL[n] are all IB optional. Do the Mellanox HCAs > support these counters ? > > -- Hal No -------------- next part -------------- An HTML attachment was scrubbed... URL: From halr at voltaire.com Thu May 19 16:09:37 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 19 May 2005 19:09:37 -0400 Subject: [openib-general] [PATCH] at: Change normal message from WARN to DEBUG Message-ID: <1116544177.4479.259.camel@hal.voltaire.com> at: Change normal message from WARN to DEBUG Signed-off-by: Hal Rosenstock Index: at.c =================================================================== --- at.c (revision 2379) +++ at.c (working copy) @@ -737,13 +737,13 @@ arp = (struct ib_arp *)skb->nh.raw; - WARN("Process IB ARP ip <%d.%d.%d.%d> gid <0x%016llx%016llx>", - (arp->src_ip & 0x000000ff), - (arp->src_ip & 0x0000ff00) >> 8, - (arp->src_ip & 0x00ff0000) >> 16, - (arp->src_ip & 0xff000000) >> 24, - be64_to_cpu(arp->src_gid.global.subnet_prefix), - be64_to_cpu(arp->src_gid.global.interface_id)); + DEBUG("Process IB ARP ip <%d.%d.%d.%d> gid <0x%016llx%016llx>", + (arp->src_ip & 0x000000ff), + (arp->src_ip & 0x0000ff00) >> 8, + (arp->src_ip & 0x00ff0000) >> 16, + (arp->src_ip & 0xff000000) >> 24, + be64_to_cpu(arp->src_gid.global.subnet_prefix), + be64_to_cpu(arp->src_gid.global.interface_id)); spin_lock_irqsave(&q->lock, flags); for (a = q->next; a != q; a = a->next) { From lindahl at pathscale.com Thu May 19 16:17:17 2005 From: lindahl at pathscale.com (Greg Lindahl) Date: Thu, 19 May 2005 16:17:17 -0700 Subject: [openib-general] Infinipath support in OpenIB In-Reply-To: <1116531964.980.13.camel@duffman> References: <20050519005637.GD7557@esmail.cup.hp.com> <004801c55c34$29e893d0$8000000a@blorp> <20050519182102.GF10318@esmail.cup.hp.com> <00b601c55ca9$06325f00$8000000a@blorp> <1116531964.980.13.camel@duffman> Message-ID: <20050519231717.GA4228@greglaptop.internal.keyresearch.com> On Thu, May 19, 2005 at 12:46:04PM -0700, Tom Duffy wrote: > > The article states: 'InfiniPath will also support the OpenIB software stack > > providing full InfiniBand compliance.' > > Also interesting that OpenIB is suddenly an "industry standard." ;-) Tom, There's a reason for that -- when we asked customers, they asked us to support OpenIB in particular. So congratulations, not only is OpenIB a part of the IB ecosystem, it's one that everyone seems to want to support. We didn't want to invent a new flavor of verbs or DAPL anyway. No point to it, no benefit to anyone. -- greg From jlentini at netapp.com Thu May 19 18:43:44 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 19 May 2005 21:43:44 -0400 (EDT) Subject: [openib-general] kdapltest regression? failing now... In-Reply-To: References: <1116531248.980.10.camel@duffman> Message-ID: I commited a fix for this in revision 2420. The problem turned out to be that DAPL wasn't initializing the max_inline_data value of the QP attr's cap structure. Let me know if you still have any problems. There is a patch in the pipeline that will remove the IBAT printout you mentioned. james On Thu, 19 May 2005, James Lentini wrote: > > I think I figure this out. DAPL was assuming a particular maximum scatter > gather list size. I'm going to change it to query for this value. Hopefully > I'll have a fix shortly. > > james > > On Thu, 19 May 2005, James Lentini wrote: > >> >> For what it's worth, this is the check that we are "failing": >> >> qp->sq.max_gs > dev->limits.max_sg >> >> ( qp->sq.max_gs + 2 > dev->limits.max_sg is also true but >> qp->transport == MLX is not). >> >> On Thu, 19 May 2005, James Lentini wrote: >> >>> >>> I'm looking into this Tom. >>> >>> The following code was added to hw/mthca/mthca_qp.c on Friday >>> (starting on line 1233): >>> >>> >>> if ((qp->transport == MLX && qp->sq.max_gs + 2 > dev->limits.max_sg) || >>> qp->sq.max_gs > dev->limits.max_sg || qp->rq.max_gs > >>> dev->limits.max_sg) >>> return -EINVAL; >>> >>> If anyone knows what we have set incorrectly, please let me know. >>> >>> Thanks, >>> james >>> >>> On Thu, 19 May 2005, Tom Duffy wrote: >>> >>> tduffy> I am not sure when this started, but after updating to top of >>> trunk*, I >>> tduffy> can no longer get kdapltest to work properly. Both ipoib and sdp >>> are >>> tduffy> working. >>> tduffy> >>> tduffy> Both server and client are returning an error: DAT_INVALID_HANDLE. >>> This >>> tduffy> is coming from ib_create_qp(). With debugging turned on: >>> tduffy> >>> tduffy> [root at flopteron2 ~]# ./kdapltest -T S -D mthca0a -d >>> tduffy> kDAPL: dapl_ia_open (mthca0a, 8, ffff81000b806308, >>> ffff81000b8062d8) >>> tduffy> kDAPL: dapl_ia_open () returns 0x0 >>> tduffy> kDAPL: dapl_pz_create (ffff81001ba165c8, ffff81000b8062e0) >>> tduffy> kDAPL: dapl_evd_kcreate (ffff81001ba165c8, 8, 1, upcall, 0x20, >>> ffff81000b8062e8) >>> tduffy> kDAPL: dapl_evd_kcreate (ffff81001ba165c8, 8, 1, upcall, 0xa0, >>> ffff81000b8062f0) >>> tduffy> kDAPL: dapl_evd_kcreate (ffff81001ba165c8, 8, 1, upcall, 0x10, >>> ffff81000b806300) >>> tduffy> kDAPL: dapl_evd_kcreate (ffff81001ba165c8, 8, 1, upcall, 0x40, >>> ffff81000b8062f8) >>> tduffy> kDAPL: dapl_ep_create (ffff81001ba165c8, ffff81001b9442c8, >>> ffff81001ba164b0, ffff81001ba166e0, ffff81001ba22050, 0000000000000000, >>> ffff81000b806318) >>> tduffy> kDAPL: dapl_ib_qp_alloc: ib_create_qp failed = -22 >>> tduffy> kDAPL: dapl_evd_free (ffff81001ba22050) >>> tduffy> kDAPL: dapl_evd_free () returns 0x0 >>> tduffy> kDAPL: dapl_evd_free (ffff81001ba22168) >>> tduffy> kDAPL: dapl_evd_free () returns 0x0 >>> tduffy> kDAPL: dapl_evd_free (ffff81001ba166e0) >>> tduffy> kDAPL: dapl_evd_free () returns 0x0 >>> tduffy> kDAPL: dapl_evd_free (ffff81001ba164b0) >>> tduffy> kDAPL: dapl_evd_free () returns 0x0 >>> tduffy> kDAPL: dapl_pz_free (ffff81001b9442c8) >>> tduffy> kDAPL: dapl_ia_query (ffff81001ba165c8, 0000000000000000, >>> 0000000000000000, ffff81001bba7b28) >>> tduffy> kDAPL: dapl_ia_query () returns 0x0 >>> tduffy> kDAPL: dapl_ia_close (ffff81001ba165c8, 1) >>> tduffy> kDAPL: dapl_evd_free (ffff81001ba167f8) >>> tduffy> kDAPL: dapl_evd_free () returns 0x0 >>> tduffy> Server_Cmd.debug: 1 >>> tduffy> Server_Cmd.dapl_name: mthca0a >>> tduffy> DT_cs_Server: IA mthca0a opened >>> tduffy> DT_cs_Server: PZ created >>> tduffy> DT_cs_Server: dat_ep_create error: DAT_INVALID_HANDLE >>> tduffy> DT_cs_Server: Waiting for clients to all go away... >>> tduffy> DT_cs_Server: Cleaning up ... >>> tduffy> DT_cs_Server: IA mthca0a closed >>> tduffy> DT_cs_Server (mthca0a): Exiting. >>> tduffy> TEST INSTANCE 0 >>> tduffy> TEST return code = 1 >>> tduffy> >>> tduffy> Also, the ib_at module prints this out now when you ping (after >>> running >>> tduffy> kdapltest)... >>> tduffy> >>> tduffy> ib_at: ib_at_arp_work: Process IB ARP ip <192.168.0.26> gid >>> <0xfe800000000000000002c9010a99e031> >>> tduffy> >>> tduffy> -tduffy >>> tduffy> >>> tduffy> * running x86_64 SMP, 2.6.12-rc4, gcc 4.0.0-6, OpenIB r2414, >>> opensm r2414 2 machines back-2-back >>> tduffy> >>> >> > From jlentini at netapp.com Thu May 19 18:45:52 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 19 May 2005 21:45:52 -0400 (EDT) Subject: [openib-general] [kDAPL] module parameter names Message-ID: With revision 2420, I made the dat and ib_dat_provider module debug parameter names consistent. They are now both dbg_mask. From jlentini at netapp.com Thu May 19 19:14:22 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 19 May 2005 22:14:22 -0400 (EDT) Subject: [openib-general] Re: [PATCH][kdapl] fix spin_lock_irqsave/spin_unlock_irqrestore implementation In-Reply-To: <20050518113948.GA26274@mellanox.co.il> References: <20050518113948.GA26274@mellanox.co.il> Message-ID: Itamar, Thank you for pointing this out. Long term I think it will be better to keep the flags with the spin lock so that these "scoping" issues don't crop up and force us to pass the flags around. The fix for this is in revision 2420. james On Wed, 18 May 2005, Itamar wrote: itamar> when spin_lock_irqsave and spin_unlock_irqrestore are not called in the same function itamar> need to pass the flags form the save the store itamar> itamar> Signed-off-by: Itamar Rabenstein itamar> itamar> Index: dapl_evd_util.c itamar> =================================================================== itamar> --- dapl_evd_util.c (revision 2374) itamar> +++ dapl_evd_util.c (working copy) itamar> @@ -378,20 +378,19 @@ itamar> * that the lock is held. itamar> */ itamar> itamar> -static struct dat_event *dapl_evd_get_event(DAPL_EVD * evd_ptr) itamar> +static struct dat_event *dapl_evd_get_event(DAPL_EVD * evd_ptr, unsigned long *flags) itamar> { itamar> struct dat_event *event; itamar> - unsigned long flags; itamar> itamar> if (evd_ptr->evd_producer_locking_needed) { itamar> - spin_lock_irqsave(&evd_ptr->header.lock, flags); itamar> + spin_lock_irqsave(&evd_ptr->header.lock, *flags); itamar> } itamar> itamar> event = (struct dat_event *) dapl_rbuf_remove(&evd_ptr->free_event_queue); itamar> itamar> /* Release the lock if it was taken and the call failed. */ itamar> if (!event && evd_ptr->evd_producer_locking_needed) { itamar> - spin_unlock_irqrestore(&evd_ptr->header.lock, flags); itamar> + spin_unlock_irqrestore(&evd_ptr->header.lock, *flags); itamar> } itamar> itamar> return event; itamar> @@ -406,10 +405,10 @@ itamar> */ itamar> itamar> static void dapl_evd_post_event(DAPL_EVD *evd_ptr, itamar> - const struct dat_event *event_ptr) itamar> + const struct dat_event *event_ptr, itamar> + unsigned long flags) itamar> { itamar> u32 dat_status; itamar> - unsigned long flags; itamar> DAPL_CNO * cno_to_trigger = NULL; itamar> itamar> dapl_dbg_log(DAPL_DBG_TYPE_EVD, itamar> @@ -459,7 +458,7 @@ itamar> DAPL_EVD * overflow_evd_ptr) itamar> { itamar> struct dat_event *overflow_event; itamar> - itamar> + unsigned long flags; itamar> /* The overflow_evd_ptr mght be the same as evd. itamar> * In that case we've got a catastrophic overflow. itamar> */ itamar> @@ -469,7 +468,7 @@ itamar> return; itamar> } itamar> itamar> - overflow_event = dapl_evd_get_event(overflow_evd_ptr); itamar> + overflow_event = dapl_evd_get_event(overflow_evd_ptr, &flags); itamar> if (!overflow_event) { itamar> /* this is not good */ itamar> overflow_evd_ptr->catastrophic_overflow = TRUE; itamar> @@ -477,17 +476,18 @@ itamar> return; itamar> } itamar> dapl_evd_format_overflow_event(overflow_evd_ptr, overflow_event); itamar> - dapl_evd_post_event(overflow_evd_ptr, overflow_event); itamar> + dapl_evd_post_event(overflow_evd_ptr, overflow_event, flags); itamar> itamar> return; itamar> } itamar> itamar> static struct dat_event *dapl_evd_get_and_init_event(DAPL_EVD *evd_ptr, itamar> - enum dat_event_number evno) itamar> + enum dat_event_number evno, itamar> + unsigned long *flags) itamar> { itamar> struct dat_event *event_ptr; itamar> itamar> - event_ptr = dapl_evd_get_event(evd_ptr); itamar> + event_ptr = dapl_evd_get_event(evd_ptr, flags); itamar> if (!event_ptr) itamar> dapl_evd_post_overflow_event(evd_ptr->header.owner_ia-> itamar> async_error_evd, evd_ptr); itamar> @@ -507,7 +507,8 @@ itamar> DAT_CR_HANDLE cr_handle) itamar> { itamar> struct dat_event *event_ptr; itamar> - event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number); itamar> + unsigned long flags; itamar> + event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number, &flags); itamar> /* itamar> * Note event lock may be held on successful return itamar> * to be released by dapl_evd_post_event(), if provider side locking itamar> @@ -525,7 +526,7 @@ itamar> event_ptr->event_data.cr_arrival_event_data.conn_qual = conn_qual; itamar> event_ptr->event_data.cr_arrival_event_data.cr_handle = cr_handle; itamar> itamar> - dapl_evd_post_event(evd_ptr, event_ptr); itamar> + dapl_evd_post_event(evd_ptr, event_ptr, flags); itamar> itamar> return DAT_SUCCESS; itamar> } itamar> @@ -537,7 +538,8 @@ itamar> void *private_data) itamar> { itamar> struct dat_event *event_ptr; itamar> - event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number); itamar> + unsigned long flags; itamar> + event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number, &flags); itamar> /* itamar> * Note event lock may be held on successful return itamar> * to be released by dapl_evd_post_event(), if provider side locking itamar> @@ -554,7 +556,7 @@ itamar> = private_data_size; itamar> event_ptr->event_data.connect_event_data.private_data = private_data; itamar> itamar> - dapl_evd_post_event(evd_ptr, event_ptr); itamar> + dapl_evd_post_event(evd_ptr, event_ptr, flags); itamar> itamar> return DAT_SUCCESS; itamar> } itamar> @@ -564,7 +566,8 @@ itamar> DAT_IA_HANDLE ia_handle) itamar> { itamar> struct dat_event *event_ptr; itamar> - event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number); itamar> + unsigned long flags; itamar> + event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number, &flags); itamar> /* itamar> * Note event lock may be held on successful return itamar> * to be released by dapl_evd_post_event(), if provider side locking itamar> @@ -579,7 +582,7 @@ itamar> event_ptr->event_data.asynch_error_event_data.dat_handle = itamar> (DAT_HANDLE) ia_handle; itamar> itamar> - dapl_evd_post_event(evd_ptr, event_ptr); itamar> + dapl_evd_post_event(evd_ptr, event_ptr, flags); itamar> itamar> return DAT_SUCCESS; itamar> } itamar> @@ -589,7 +592,8 @@ itamar> void *pointer) itamar> { itamar> struct dat_event *event_ptr; itamar> - event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number); itamar> + unsigned long flags; itamar> + event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number, &flags); itamar> /* itamar> * Note event lock may be held on successful return itamar> * to be released by dapl_evd_post_event(), if provider side locking itamar> @@ -603,7 +607,7 @@ itamar> itamar> event_ptr->event_data.software_event_data.pointer = pointer; itamar> itamar> - dapl_evd_post_event(evd_ptr, event_ptr); itamar> + dapl_evd_post_event(evd_ptr, event_ptr, flags); itamar> itamar> return DAT_SUCCESS; itamar> } itamar> itamar> -- itamar> Itamar itamar> From jlentini at netapp.com Thu May 19 19:21:17 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 19 May 2005 22:21:17 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: add some clarification to a few debug printks In-Reply-To: <1116450226.11460.2.camel@duffman> References: <1116450226.11460.2.camel@duffman> Message-ID: Commited in revision 2421. On Wed, 18 May 2005, Tom Duffy wrote: tduffy> I was running into some issues and noticed that these printks were not tduffy> very clear about where they were coming from. So, adding a little more tduffy> info. tduffy> tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_openib_qp.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_openib_qp.c (revision 2382) tduffy> +++ linux-kernel/dat-provider/dapl_openib_qp.c (working copy) tduffy> @@ -108,7 +108,8 @@ u32 dapl_ib_qp_alloc(DAPL_IA *ia_ptr, DA tduffy> ep_ptr->qp_handle = ib_create_qp(ib_pd_handle, &qp_attr); tduffy> if (IS_ERR(ep_ptr->qp_handle)) { tduffy> ib_status = PTR_ERR(ep_ptr->qp_handle); tduffy> - dapl_dbg_log(DAPL_DBG_TYPE_ERR, " failed code = %d\n", tduffy> + dapl_dbg_log(DAPL_DBG_TYPE_ERR, tduffy> + " dapl_ib_qp_alloc: ib_create_qp failed = %d\n", tduffy> ib_status); tduffy> return dapl_ib_status_convert(ib_status); tduffy> } tduffy> @@ -197,8 +198,9 @@ ib_cq_handle_t dapl_get_dto_cq(DAPL_IA * tduffy> ib_status = tduffy> PTR_ERR(ia_ptr->hca_ptr->ib_trans. tduffy> null_ib_cq_handle); tduffy> - dapl_dbg_log(DAPL_DBG_TYPE_ERR, " failed code = %d\n", tduffy> - ib_status); tduffy> + dapl_dbg_log(DAPL_DBG_TYPE_ERR, tduffy> + " dapl_get_dto_cq: ib_create_cq failed " tduffy> + "= %d\n", ib_status); tduffy> ia_ptr->hca_ptr->ib_trans.null_ib_cq_handle = tduffy> IB_INVALID_HANDLE; tduffy> } tduffy> From jlentini at netapp.com Thu May 19 19:36:26 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 19 May 2005 22:36:26 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: return is not a function In-Reply-To: <1116460172.8031.2.camel@duffman> References: <1116460172.8031.2.camel@duffman> Message-ID: Committed in revision 2422. On Wed, 18 May 2005, Tom Duffy wrote: tduffy> return is not a function tduffy> tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> Index: linux-kernel-return/dat-provider/dapl_cookie.c tduffy> =================================================================== tduffy> --- linux-kernel-return/dat-provider/dapl_cookie.c (revision 2382) tduffy> +++ linux-kernel-return/dat-provider/dapl_cookie.c (working copy) tduffy> @@ -134,9 +134,9 @@ u32 dapl_cb_create(DAPL_COOKIE_BUFFER *b tduffy> buffer->pool[i].ep = ep; tduffy> } tduffy> tduffy> - return (DAT_SUCCESS); tduffy> + return DAT_SUCCESS; tduffy> } else { tduffy> - return (DAT_INSUFFICIENT_RESOURCES); tduffy> + return DAT_INSUFFICIENT_RESOURCES; tduffy> } tduffy> } tduffy> tduffy> Index: linux-kernel-return/dat-provider/dapl_ring_buffer_util.c tduffy> =================================================================== tduffy> --- linux-kernel-return/dat-provider/dapl_ring_buffer_util.c (revision 2382) tduffy> +++ linux-kernel-return/dat-provider/dapl_ring_buffer_util.c (working copy) tduffy> @@ -237,7 +237,7 @@ void *dapl_rbuf_remove(DAPL_RING_BUFFER tduffy> if (val == pos) { tduffy> pos = (pos + 1) & rbuf->lim; /* verify in range */ tduffy> tduffy> - return (rbuf->base[pos]); tduffy> + return rbuf->base[pos]; tduffy> } tduffy> } tduffy> tduffy> Index: linux-kernel-return/dat-provider/dapl_lmr_util.c tduffy> =================================================================== tduffy> --- linux-kernel-return/dat-provider/dapl_lmr_util.c (revision 2382) tduffy> +++ linux-kernel-return/dat-provider/dapl_lmr_util.c (working copy) tduffy> @@ -50,7 +50,7 @@ DAPL_LMR *dapl_lmr_alloc(DAPL_IA * ia, tduffy> /* Allocate LMR */ tduffy> lmr = (DAPL_LMR *) kmalloc(sizeof(DAPL_LMR), GFP_ATOMIC); tduffy> if (NULL == lmr) { tduffy> - return (NULL); tduffy> + return NULL; tduffy> } tduffy> tduffy> /* zero the structure */ tduffy> @@ -80,7 +80,7 @@ DAPL_LMR *dapl_lmr_alloc(DAPL_IA * ia, tduffy> lmr->param.mem_priv = mem_priv; tduffy> atomic_set(&lmr->lmr_ref_count, 0); tduffy> tduffy> - return (lmr); tduffy> + return lmr; tduffy> } tduffy> tduffy> void dapl_lmr_dealloc(DAPL_LMR * lmr) tduffy> Index: linux-kernel-return/dat-provider/dapl_hca_util.c tduffy> =================================================================== tduffy> --- linux-kernel-return/dat-provider/dapl_hca_util.c (revision 2382) tduffy> +++ linux-kernel-return/dat-provider/dapl_hca_util.c (working copy) tduffy> @@ -83,7 +83,7 @@ DAPL_HCA *dapl_hca_alloc(char *name, str tduffy> } tduffy> } tduffy> tduffy> - return (hca_ptr); tduffy> + return hca_ptr; tduffy> } tduffy> tduffy> /* tduffy> Index: linux-kernel-return/dat-provider/dapl_cr_util.c tduffy> =================================================================== tduffy> --- linux-kernel-return/dat-provider/dapl_cr_util.c (revision 2382) tduffy> +++ linux-kernel-return/dat-provider/dapl_cr_util.c (working copy) tduffy> @@ -44,7 +44,7 @@ DAPL_CR *dapl_cr_alloc(DAPL_IA * ia_ptr) tduffy> /* Allocate EP */ tduffy> cr_ptr = (DAPL_CR *) kmalloc(sizeof(DAPL_CR), GFP_ATOMIC); tduffy> if (cr_ptr == NULL) { tduffy> - return (NULL); tduffy> + return NULL; tduffy> } tduffy> tduffy> /* zero the structure */ tduffy> @@ -62,7 +62,7 @@ DAPL_CR *dapl_cr_alloc(DAPL_IA * ia_ptr) tduffy> dapl_llist_init_entry(&cr_ptr->header.ia_list_entry); tduffy> spin_lock_init(&cr_ptr->header.lock); tduffy> tduffy> - return (cr_ptr); tduffy> + return cr_ptr; tduffy> } tduffy> tduffy> /* tduffy> Index: linux-kernel-return/dat-provider/dapl_ia_util.c tduffy> =================================================================== tduffy> --- linux-kernel-return/dat-provider/dapl_ia_util.c (revision 2382) tduffy> +++ linux-kernel-return/dat-provider/dapl_ia_util.c (working copy) tduffy> @@ -65,7 +65,7 @@ DAPL_IA *dapl_ia_alloc(struct dat_provid tduffy> /* Allocate IA */ tduffy> ia_ptr = (DAPL_IA *) kmalloc(sizeof(DAPL_IA), GFP_ATOMIC); tduffy> if (ia_ptr == NULL) { tduffy> - return (NULL); tduffy> + return NULL; tduffy> } tduffy> tduffy> /* zero the structure */ tduffy> @@ -100,7 +100,7 @@ DAPL_IA *dapl_ia_alloc(struct dat_provid tduffy> tduffy> dapl_hca_link_ia(hca_ptr, ia_ptr); tduffy> tduffy> - return (ia_ptr); tduffy> + return ia_ptr; tduffy> } tduffy> tduffy> /* tduffy> Index: linux-kernel-return/dat-provider/dapl_pz_util.c tduffy> =================================================================== tduffy> --- linux-kernel-return/dat-provider/dapl_pz_util.c (revision 2382) tduffy> +++ linux-kernel-return/dat-provider/dapl_pz_util.c (working copy) tduffy> @@ -59,7 +59,7 @@ DAPL_PZ *dapl_pz_alloc(DAPL_IA * ia) tduffy> /* Allocate PZ */ tduffy> pz = (DAPL_PZ *) kmalloc(sizeof(DAPL_PZ), GFP_ATOMIC); tduffy> if (NULL == pz) { tduffy> - return (NULL); tduffy> + return NULL; tduffy> } tduffy> tduffy> /* zero the structure */ tduffy> @@ -83,7 +83,7 @@ DAPL_PZ *dapl_pz_alloc(DAPL_IA * ia) tduffy> */ tduffy> atomic_set(&pz->pz_ref_count, 0); tduffy> tduffy> - return (pz); tduffy> + return pz; tduffy> } tduffy> tduffy> /* tduffy> Index: linux-kernel-return/dat-provider/dapl_openib_cm.c tduffy> =================================================================== tduffy> --- linux-kernel-return/dat-provider/dapl_openib_cm.c (revision 2382) tduffy> +++ linux-kernel-return/dat-provider/dapl_openib_cm.c (working copy) tduffy> @@ -844,7 +844,7 @@ u32 dapl_ib_cm_remote_addr(DAT_HANDLE da tduffy> tduffy> rc = ib_cm_get_remote_gid(cr_ptr->ib_cm_handle, (u8 *) & remote_gid); tduffy> if (rc < 0) tduffy> - return (rc); tduffy> + return rc; tduffy> tduffy> if (!memcmp(&ia_ptr->hca_ptr->ib_trans.gid, &remote_gid, 16)) { tduffy> memcpy(remote_ia_address, &ia_ptr->hca_ptr->hca_address, tduffy> Index: linux-kernel-return/dat-provider/dapl_openib_util.c tduffy> =================================================================== tduffy> --- linux-kernel-return/dat-provider/dapl_openib_util.c (revision 2382) tduffy> +++ linux-kernel-return/dat-provider/dapl_openib_util.c (working copy) tduffy> @@ -334,7 +334,7 @@ u32 dapl_ib_pd_alloc(DAPL_IA *ia, DAPL_P tduffy> } tduffy> tduffy> pz->pd_handle = (ib_pd_handle_t) pd; tduffy> - return (DAT_SUCCESS); tduffy> + return DAT_SUCCESS; tduffy> } tduffy> tduffy> /* tduffy> @@ -367,7 +367,7 @@ u32 dapl_ib_pd_free(DAPL_PZ *pz) tduffy> } tduffy> tduffy> pz->pd_handle = IB_INVALID_HANDLE; tduffy> - return (DAT_SUCCESS); tduffy> + return DAT_SUCCESS; tduffy> } tduffy> tduffy> /* tduffy> @@ -393,7 +393,7 @@ u32 dapl_ib_mr_register(DAPL_IA *ia, DAP tduffy> u64 length, enum dat_mem_priv_flags privileges) tduffy> { tduffy> /* ITBD virtual memory registration ??? */ tduffy> - return (DAT_SUCCESS); tduffy> + return DAT_SUCCESS; tduffy> } tduffy> tduffy> u32 dapl_ib_mr_register_physical(DAPL_IA *ia_ptr, DAPL_LMR *lmr, tduffy> @@ -452,7 +452,7 @@ u32 dapl_ib_mr_register_physical(DAPL_IA tduffy> dapl_dbg_log(DAPL_DBG_TYPE_UTIL, tduffy> "dapl_ib_mr_register_physical(%p %d) got lkey 0x%x \n", tduffy> buf_list, length, lmr->param.lmr_context); tduffy> - return (DAT_SUCCESS); tduffy> + return DAT_SUCCESS; tduffy> } tduffy> tduffy> /* tduffy> @@ -486,7 +486,7 @@ u32 dapl_ib_mr_deregister(DAPL_LMR *lmr) tduffy> lmr->param.lmr_context = 0; tduffy> lmr->mr_handle = IB_INVALID_HANDLE; tduffy> tduffy> - return (DAT_SUCCESS); tduffy> + return DAT_SUCCESS; tduffy> } tduffy> tduffy> /* tduffy> @@ -523,7 +523,7 @@ u32 dapl_ib_mw_alloc(DAPL_RMR *rmr) tduffy> rmr->param.rmr_context = mw->rkey; tduffy> dapl_dbg_log(DAPL_DBG_TYPE_UTIL, tduffy> "created mw_handle = %p, r_key = 0x%x\n", mw, mw->rkey); tduffy> - return (DAT_SUCCESS); tduffy> + return DAT_SUCCESS; tduffy> } tduffy> tduffy> /* tduffy> @@ -556,7 +556,7 @@ u32 dapl_ib_mw_free(DAPL_RMR *rmr) tduffy> tduffy> rmr->param.rmr_context = 0; tduffy> rmr->mw_handle = IB_INVALID_HANDLE; tduffy> - return (DAT_SUCCESS); tduffy> + return DAT_SUCCESS; tduffy> } tduffy> tduffy> /* tduffy> @@ -601,13 +601,13 @@ u32 dapl_ib_mw_bind(DAPL_RMR *rmr, DAPL_ tduffy> if (ib_status < 0) { tduffy> dapl_dbg_log(DAPL_DBG_TYPE_ERR, tduffy> "ib_bind_mw error code return = %d\n", ib_status); tduffy> - return (dapl_ib_status_convert(ib_status)); tduffy> + return dapl_ib_status_convert(ib_status); tduffy> } tduffy> tduffy> rmr->param.rmr_context = (DAT_RMR_CONTEXT) rmr->mw_handle->rkey; tduffy> dapl_dbg_log(DAPL_DBG_TYPE_UTIL, " new_rkey = 0x%x\n", tduffy> rmr->mw_handle->rkey); tduffy> - return (DAT_SUCCESS); tduffy> + return DAT_SUCCESS; tduffy> } tduffy> tduffy> /* tduffy> @@ -643,13 +643,13 @@ u32 dapl_ib_mw_unbind(DAPL_RMR *rmr, DAP tduffy> if (ib_status < 0) { tduffy> dapl_dbg_log(DAPL_DBG_TYPE_ERR, tduffy> " ib_bind_mw error code return = %d\n", ib_status); tduffy> - return (dapl_ib_status_convert(ib_status)); tduffy> + return dapl_ib_status_convert(ib_status); tduffy> } tduffy> tduffy> rmr->param.rmr_context = (DAT_RMR_CONTEXT) rmr->mw_handle->rkey; tduffy> dapl_dbg_log(DAPL_DBG_TYPE_UTIL, " new_rkey = 0x%x\n", tduffy> rmr->mw_handle->rkey); tduffy> - return (DAT_SUCCESS); tduffy> + return DAT_SUCCESS; tduffy> } tduffy> tduffy> /* tduffy> @@ -709,7 +709,7 @@ u32 dapl_ib_query_hca(DAPL_HCA *hca_ptr, tduffy> dapl_dbg_log(DAPL_DBG_TYPE_ERR, tduffy> " ib_query_device error code return = %d\n", tduffy> ib_status); tduffy> - return (dapl_ib_status_convert(ib_status)); tduffy> + return dapl_ib_status_convert(ib_status); tduffy> } tduffy> tduffy> /* ITBD need to review the attr */ tduffy> Index: linux-kernel-return/dat-provider/dapl_rmr_util.c tduffy> =================================================================== tduffy> --- linux-kernel-return/dat-provider/dapl_rmr_util.c (revision 2382) tduffy> +++ linux-kernel-return/dat-provider/dapl_rmr_util.c (working copy) tduffy> @@ -35,7 +35,7 @@ DAPL_RMR *dapl_rmr_alloc(DAPL_PZ * pz) tduffy> /* Allocate LMR */ tduffy> rmr = (DAPL_RMR *) kmalloc(sizeof(DAPL_RMR), GFP_ATOMIC); tduffy> if (NULL == rmr) { tduffy> - return (NULL); tduffy> + return NULL; tduffy> } tduffy> tduffy> /* zero the structure */ tduffy> @@ -68,7 +68,7 @@ DAPL_RMR *dapl_rmr_alloc(DAPL_PZ * pz) tduffy> rmr->pz = pz; tduffy> rmr->lmr = NULL; tduffy> tduffy> - return (rmr); tduffy> + return rmr; tduffy> } tduffy> tduffy> void dapl_rmr_dealloc(DAPL_RMR * rmr) tduffy> Index: linux-kernel-return/dat-provider/dapl_sp_util.c tduffy> =================================================================== tduffy> --- linux-kernel-return/dat-provider/dapl_sp_util.c (revision 2382) tduffy> +++ linux-kernel-return/dat-provider/dapl_sp_util.c (working copy) tduffy> @@ -66,7 +66,7 @@ DAPL_SP *dapl_sp_alloc(DAPL_IA * ia_ptr, tduffy> /* Allocate EP */ tduffy> sp_ptr = (DAPL_SP *) kmalloc(sizeof(DAPL_SP), GFP_ATOMIC); tduffy> if (sp_ptr == NULL) { tduffy> - return (NULL); tduffy> + return NULL; tduffy> } tduffy> tduffy> /* zero the structure */ tduffy> @@ -94,7 +94,7 @@ DAPL_SP *dapl_sp_alloc(DAPL_IA * ia_ptr, tduffy> */ tduffy> dapl_llist_init_head(&sp_ptr->cr_list_head); tduffy> tduffy> - return (sp_ptr); tduffy> + return sp_ptr; tduffy> } tduffy> tduffy> /* tduffy> Index: linux-kernel-return/dat-provider/dapl_llist.c tduffy> =================================================================== tduffy> --- linux-kernel-return/dat-provider/dapl_llist.c (revision 2382) tduffy> +++ linux-kernel-return/dat-provider/dapl_llist.c (working copy) tduffy> @@ -198,7 +198,7 @@ void *dapl_llist_remove_head(DAPL_LLIST_ tduffy> first->flink = NULL; tduffy> first->blink = NULL; tduffy> first->list_head = NULL; tduffy> - return (first->data); tduffy> + return first->data; tduffy> } tduffy> tduffy> /* tduffy> @@ -224,7 +224,7 @@ void *dapl_llist_remove_tail(DAPL_LLIST_ tduffy> last->blink = NULL; tduffy> last->list_head = NULL; tduffy> tduffy> - return (last->data); tduffy> + return last->data; tduffy> } tduffy> tduffy> /* tduffy> @@ -274,7 +274,7 @@ void *dapl_llist_remove_entry(DAPL_LLIST tduffy> entry->flink = NULL; tduffy> entry->blink = NULL; tduffy> tduffy> - return (entry->data); tduffy> + return entry->data; tduffy> } tduffy> tduffy> /* tduffy> @@ -287,7 +287,7 @@ void *dapl_llist_peek_head(DAPL_LLIST_HE tduffy> tduffy> dapl_os_assert(!dapl_llist_is_empty(head)); tduffy> first = *head; tduffy> - return (first->data); tduffy> + return first->data; tduffy> } tduffy> tduffy> /* tduffy> @@ -310,7 +310,7 @@ void *dapl_llist_next_entry(DAPL_LLIST_H tduffy> return NULL; tduffy> } tduffy> } tduffy> - return (next->data); tduffy> + return next->data; tduffy> } tduffy> tduffy> #ifdef DAPL_DBG tduffy> From jlentini at netapp.com Thu May 19 19:42:58 2005 From: jlentini at netapp.com (James Lentini) Date: Thu, 19 May 2005 22:42:58 -0400 (EDT) Subject: [openib-general] Re: [PATCH] at: Change normal message from WARN to DEBUG In-Reply-To: <1116544177.4479.259.camel@hal.voltaire.com> References: <1116544177.4479.259.camel@hal.voltaire.com> Message-ID: Committed in revision 2423. On Thu, 19 May 2005, Hal Rosenstock wrote: halr> at: Change normal message from WARN to DEBUG halr> halr> Signed-off-by: Hal Rosenstock halr> halr> Index: at.c halr> =================================================================== halr> --- at.c (revision 2379) halr> +++ at.c (working copy) halr> @@ -737,13 +737,13 @@ halr> halr> arp = (struct ib_arp *)skb->nh.raw; halr> halr> - WARN("Process IB ARP ip <%d.%d.%d.%d> gid <0x%016llx%016llx>", halr> - (arp->src_ip & 0x000000ff), halr> - (arp->src_ip & 0x0000ff00) >> 8, halr> - (arp->src_ip & 0x00ff0000) >> 16, halr> - (arp->src_ip & 0xff000000) >> 24, halr> - be64_to_cpu(arp->src_gid.global.subnet_prefix), halr> - be64_to_cpu(arp->src_gid.global.interface_id)); halr> + DEBUG("Process IB ARP ip <%d.%d.%d.%d> gid <0x%016llx%016llx>", halr> + (arp->src_ip & 0x000000ff), halr> + (arp->src_ip & 0x0000ff00) >> 8, halr> + (arp->src_ip & 0x00ff0000) >> 16, halr> + (arp->src_ip & 0xff000000) >> 24, halr> + be64_to_cpu(arp->src_gid.global.subnet_prefix), halr> + be64_to_cpu(arp->src_gid.global.interface_id)); halr> halr> spin_lock_irqsave(&q->lock, flags); halr> for (a = q->next; a != q; a = a->next) { halr> halr> From xma at us.ibm.com Thu May 19 21:30:14 2005 From: xma at us.ibm.com (Shirley Ma) Date: Thu, 19 May 2005 21:30:14 -0700 Subject: [openib-general] OOPS: ib_mad crashery on bootup In-Reply-To: <1116530995.980.9.camel@duffman> Message-ID: > Error, some other host already uses address 192.168.0.233. I hit a simliar problem a while ago. But I couldn't reproduct it since then. Is this reproducible if you configure duplicate IP addresses? Thanks Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From tduffy at sun.com Thu May 19 22:06:27 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 19 May 2005 22:06:27 -0700 Subject: [openib-general] OOPS: ib_mad crashery on bootup In-Reply-To: References: Message-ID: <1116565587.10540.3.camel@duffman> On Thu, 2005-05-19 at 21:30 -0700, Shirley Ma wrote: > > > Error, some other host already uses address 192.168.0.233. > > I hit a simliar problem a while ago. But I couldn't reproduct it since > then. > Is this reproducible if you configure duplicate IP addresses? I don't think this particular error is related to the crash I observed. I see this on normal bootup of FC3 w/ ib0 configured. The crash I saw happened once, but upon reboot was not reproducible. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From tduffy at sun.com Thu May 19 22:15:37 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 19 May 2005 22:15:37 -0700 Subject: [openib-general] kdapltest regression? failing now... In-Reply-To: References: <1116531248.980.10.camel@duffman> Message-ID: <1116566137.10540.6.camel@duffman> On Thu, 2005-05-19 at 21:43 -0400, James Lentini wrote: > I commited a fix for this in revision 2420. The problem turned out to > be that DAPL wasn't initializing the max_inline_data value of the QP > attr's cap structure. > > Let me know if you still have any problems. Good job. All is well now. Tested working. Thanks, -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From Mark.Seger at hp.com Fri May 20 04:25:58 2005 From: Mark.Seger at hp.com (Mark Seger) Date: Fri, 20 May 2005 07:25:58 -0400 Subject: [openib-general] performance counters in /sys In-Reply-To: <1116539301.4479.255.camel@hal.voltaire.com> References: <35EA21F54A45CB47B879F21A91F4862F581222@taurus.voltaire.com> <1116539301.4479.255.camel@hal.voltaire.com> Message-ID: <428DC946.5080207@hp.com> Hal Rosenstock wrote: >On Thu, 2005-05-19 at 16:45, Yaron Haviv wrote: > > >>I believe you can use the per VL counters for that >>(IB allows counting traffic on a specific VL) >>By matching ULPs to VLs (e.g. through the ib_at lib we suggested) >>You can get both congestion isolation per traffic type as well as the >>ability to count traffic per ULP >>(note that up to 8 VLs are supported in the Mellanox chips) >> >> > >PortXmitDataVL[n], PortRcvDataVL[n], PortXmitPktVL[n], and >PortRcvPktVL[n] are all IB optional. Do the Mellanox HCAs support these >counters ? > >-- Hal > I didn't realize some counters are optional. That kind of blows my notion of a fixed format, space separated output. 8-( Perhaps a better way to focus this conversation is to see what the general feeling is towards statistics that are more oriented towards consumption by programs rather than humans and then discuss the details of format if some consensus can be reached. I'm also wondering if data is to be combined for mulitple hcas/ports it may make more sense to put it into /proc where those types of structures appear, but that level of discussion should probably also be deferred. -mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From halr at voltaire.com Fri May 20 04:36:25 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 20 May 2005 07:36:25 -0400 Subject: [openib-general] performance counters in /sys In-Reply-To: <428DC946.5080207@hp.com> References: <35EA21F54A45CB47B879F21A91F4862F581222@taurus.voltaire.com> <1116539301.4479.255.camel@hal.voltaire.com> <428DC946.5080207@hp.com> Message-ID: <1116588985.4479.790.camel@hal.voltaire.com> On Fri, 2005-05-20 at 07:25, Mark Seger wrote: > I didn't realize some counters are optional. That kind of blows my > notion of a fixed format, space separated output. 8-( The various hardware counters are also allowed to have different widths. -- Hal From halr at voltaire.com Fri May 20 06:23:00 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 20 May 2005 09:23:00 -0400 Subject: [openib-general] Re: [RFC] [PATCH] user_mad: Support RMPP on send side In-Reply-To: <5264xgf85t.fsf@topspin.com> References: <1115845653.4490.188.camel@localhost.localdomain> <5264xgf85t.fsf@topspin.com> Message-ID: <1116595380.4478.65.camel@hal.voltaire.com> On Wed, 2005-05-18 at 19:04, Roland Dreier wrote: > This looks OK to check in with one small comment on the following: > > - if (copy_to_user(buf, &packet->mad, sizeof packet->mad)) > + if (copy_to_user(buf, &packet->mad, > + min(count, packet->length + > + sizeof (struct ib_user_mad)))) > ret = -EFAULT; > else > - ret = sizeof packet->mad; > + ret = count; > > This code will truncate a received MAD that is bigger than the buffer > passed into read(), but return the full size of the packet. I don't > think read() is allowed to do this: the return value can be at most > the count value passed in by the user. > > I think we have two options: truncate and return the actual amount of > data read to the user, or return an error if the user's buffer is too > small. The man page for read states: read() attempts to read up to count bytes from file descriptor fd into the buffer starting at buf. RETURN VALUE On success, the number of bytes read is returned (zero indicates end of file), and the file position is advanced by this number. For RMPP reads (next set of changes to this), it would eliminate an additional call (ioctl) if on a read that a return length larger than the count supplied indicates that the read did not occur and the count (buffer size) to be used for the next read to get the entire packet. It appears to me that there is nothing that enforces the man page behavior above in Linux so this is a "convention". Is this something we can take advantage of or do we need to require the additional call to get the buffer length ? -- Hal From parks at lanl.gov Fri May 20 06:55:32 2005 From: parks at lanl.gov (Parks Fields) Date: Fri, 20 May 2005 07:55:32 -0600 Subject: [openib-general] 2.6.12 question In-Reply-To: <52psvnb6la.fsf@topspin.com> References: <6.2.1.2.2.20050519082246.039c1430@ccn-mail.lanl.gov> <52psvnb6la.fsf@topspin.com> Message-ID: <6.2.1.2.2.20050520075305.0399f0e0@ccn-mail.lanl.gov> Ok, My ba. Not sure what was really wrong, all the IB stuff was there but several other modules missing. After a rebuild of the kernel things are back to normal. Thanks again parks At 09:04 AM 5/19/2005, Roland Dreier wrote: > Parks> Is there a branch for 2.6.12? We have noticed that we get > Parks> different performance when built on 2.6.11-5 vs 2.6.12-rc4. > >Obviously the drivers/infiniband already in 2.6.12-rc4 should work >fine. Current svn should also work -- I think the only change >required is for SDP, and Tom Duffy posted a patch. > > - R. From bernhard at aon.at Fri May 20 07:35:58 2005 From: bernhard at aon.at (Bernhard Fischer) Date: Fri, 20 May 2005 16:35:58 +0200 Subject: diff-perftest-08 [was: Re: [openib-general] Re: Re: diff-perftest-07 replace pp_get_local_lid()] In-Reply-To: <20050518185631.GE7553@aon.at> References: <20050518013821.GO1078@esmail.cup.hp.com> <20050518135130.GE25713@mellanox.co.il> <20050518180620.GA6004@esmail.cup.hp.com> <20050518182448.GC7553@aon.at> <20050518184436.GA27680@mellanox.co.il> <20050518185631.GE7553@aon.at> Message-ID: <20050520143558.GY15688@aon.at> On Wed, May 18, 2005 at 08:56:31PM +0200, Bernhard Fischer wrote: >On Wed, May 18, 2005 at 09:44:36PM +0300, Michael S. Tsirkin wrote: >>Quoting r. Bernhard Fischer : >>> Subject: Re: Re: diff-perftest-07 replace pp_get_local_lid() >>Size 0 doesnt make sense for this test since I'm polling on data: >>no data = nothing to poll upon. > >If size 0 and -1 (not 1 as you seem to have read) are illegal (they >obviously are for malloc) you should check if(size <1){usage();exit(1);} >Same for the other arguments. I'll send a patchlet if noone did fix >that by next week. > >>-s 1 should work - I'll check tomorrow. Looking at the code, 1 is the default and not 4096 as advertised (see attached patch to reflect default of 1 in usage). o Change usage() to match the actual default of size. o sparse warning amout non-ANSI function decl of get_cpu_mhz(), whitespace in printf while at it. o Add printing Size, TX-depth and Iters to print_report(). Without a short description of the workload, it is bit hard to interpret the results if one cannot see the command-line used to run the tests. o Check arguments -s and -t o Trim some whitespace o Remove odd 'write(sockfd,"foo"...' (What was that good for?) Signed-off-by: Bernhard Fischer > >-s 1 did work for me last time i looked. >In fact, back then only 1 worked, i got errors for bigger values i tried >(42,128,256,512,1k,2k,4k,16k,32k,64k,128k,42M etc). The problem seems to be that i (wrongly) believed in the default size of 4k, so assumed i could run the server without args and pass -s 2 to the client. This is a wrong assertion as size really defaults to 1. I can specify -s 940 as the maximum, using 941 gives me "cannot create QP" in rdma_lat.c. Is this expected? What are the contraints and limits for size here? Quick unrelated question: Is there any limit on the number of cqe's i can pass to mthca_create_cq? I initially tried 40960 or 4096 but that did not work. I'm now using 64 but would like to use more than those in the future.. -------------- next part -------------- diff -X excl -rduNp gen2.2423.oorig/trunk/src/userspace/perftest/get_clock.c gen2.2423/trunk/src/userspace/perftest/get_clock.c --- gen2.2423.oorig/trunk/src/userspace/perftest/get_clock.c 2005-05-17 14:38:30.000000000 +0000 +++ gen2.2423/trunk/src/userspace/perftest/get_clock.c 2005-05-20 11:47:06.000000000 +0000 @@ -35,7 +35,7 @@ #include #include -double get_cpu_mhz() +double get_cpu_mhz(void) { FILE* f; char buf[256]; @@ -55,7 +55,7 @@ double get_cpu_mhz() continue; } if (mhz != m) { - fprintf(stderr,"Conflicting CPU frequency values " + fprintf(stderr,"Conflicting CPU frequency values" " detected: %lf != %lf\n", mhz, m); return 0.0; diff -X excl -rduNp gen2.2423.oorig/trunk/src/userspace/perftest/rdma_lat.c gen2.2423/trunk/src/userspace/perftest/rdma_lat.c --- gen2.2423.oorig/trunk/src/userspace/perftest/rdma_lat.c 2005-05-20 07:06:11.000000000 +0000 +++ gen2.2423/trunk/src/userspace/perftest/rdma_lat.c 2005-05-20 12:43:51.000000000 +0000 @@ -434,7 +434,7 @@ static void usage(const char *argv0) printf(" -p, --port= listen on/connect to port (default 18515)\n"); printf(" -d, --ib-dev= use IB device (default first device found)\n"); printf(" -i, --ib-port= use port of IB device (default 1)\n"); - printf(" -s, --size= size of message to exchange (default 4096)\n"); + printf(" -s, --size= size of message to exchange (default 1)\n"); printf(" -t, --tx-depth= size of tx queue (default 50)\n"); printf(" -n, --iters= number of exchanges (at least 2, default 1000)\n"); printf(" -C, --report-cycles report times in cpu cycle units (default microseconds)\n"); @@ -468,7 +468,8 @@ static int cycles_compare(const void * a } static void print_report(struct report_options * options, - unsigned int iters, cycles_t *tstamp) + unsigned int iters, cycles_t *tstamp, + struct pingpong_context *ctx) { double cycles_to_units; cycles_t median; @@ -507,6 +508,9 @@ static void print_report(struct report_o printf("%d, %g\n", i + 1, delta[i] / cycles_to_units / 2); } + printf("Size: %d, TX-depth: %d, Exchanges: %d\n", + ctx->size, ctx->tx_depth, iters); + median = get_median(iters, delta); printf("Latency typical: %g %s\n", median / cycles_to_units / 2, units); @@ -519,8 +523,8 @@ static void print_report(struct report_o int main(int argc, char *argv[]) { - struct dlist *dev_list; - struct ibv_device *ib_dev; + struct dlist *dev_list; + struct ibv_device *ib_dev; struct pingpong_context *ctx; struct pingpong_dest my_dest; struct pingpong_dest *rem_dest; @@ -536,7 +540,7 @@ int main(int argc, char *argv[]) int client_first_post; int sockfd; struct ibv_qp *qp; - struct ibv_send_wr *wr; + struct ibv_send_wr *wr; volatile char *poll_buf; volatile char *post_buf; struct report_options report = {}; @@ -555,7 +559,7 @@ int main(int argc, char *argv[]) { .name = "iters", .has_arg = 1, .val = 'n' }, { .name = "tx-depth", .has_arg = 1, .val = 't' }, { .name = "report-cycles", .has_arg = 0, .val = 'C' }, - { .name = "report-histogram", .has_arg = 0, .val = 'H' }, + { .name = "report-histogram",.has_arg = 0, .val = 'H' }, { .name = "report-unsorted",.has_arg = 0, .val = 'U' }, { 0 } }; @@ -587,10 +591,12 @@ int main(int argc, char *argv[]) case 's': size = strtol(optarg, NULL, 0); + if (size < 1) { usage(argv[0]); return 1; } break; case 't': tx_depth = strtol(optarg, NULL, 0); + if (tx_depth < 1) { usage(argv[0]); return 1; } break; case 'n': @@ -674,7 +680,6 @@ int main(int argc, char *argv[]) "RKey %#08x VAddr %#016Lx\n", my_dest.lid, my_dest.qpn, my_dest.psn, my_dest.rkey, my_dest.vaddr); - if (servername) { sockfd = pp_client_connect(servername, port); @@ -707,7 +712,6 @@ int main(int argc, char *argv[]) rem_dest = pp_server_exch_dest(sockfd, &my_dest); } - write(sockfd, "done", sizeof "done"); close(sockfd); wr = &ctx->wr; @@ -784,7 +788,7 @@ int main(int argc, char *argv[]) } } - print_report(&report, iters, tstamp); + print_report(&report, iters, tstamp, ctx); free(tstamp); return 0; From halr at voltaire.com Fri May 20 07:49:40 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 20 May 2005 10:49:40 -0400 Subject: [openib-general] Possible minor change to ucm.c Message-ID: <1116600579.4479.19.camel@hal.voltaire.com> Hi Libor, A possible minor change to ucm.c::ib_ucm_event_handler line 440 would be: uevent->cm_id = (id == IB_UCM_CM_ID_INVALID ? cm_id : NULL); rather than: uevent->cm_id = ((event->event == IB_CM_REQ_RECEIVED || event->event == IB_CM_SIDR_REQ_RECEIVED ) ? cm_id : NULL); -- Hal From blist at aon.at Fri May 20 07:59:44 2005 From: blist at aon.at (Bernhard Fischer) Date: Fri, 20 May 2005 16:59:44 +0200 Subject: [openib-general] perftest on i386 In-Reply-To: <20050520143558.GY15688@aon.at> References: <20050518013821.GO1078@esmail.cup.hp.com> <20050518135130.GE25713@mellanox.co.il> <20050518180620.GA6004@esmail.cup.hp.com> <20050518182448.GC7553@aon.at> <20050518184436.GA27680@mellanox.co.il> <20050518185631.GE7553@aon.at> <20050520143558.GY15688@aon.at> Message-ID: <20050520145944.GZ15688@aon.at> Hi, On i386 UP, i see odd results during runs. The machine is basically idle. The worst latency is always somewhere around exchange 12-80. Is this expected and if not, does somebody have an idea what may cause this? Thank you, $ ./rdma_lat -i 2 -U -s 1 tst09 -t 1 | egrep -v ", 3.(1|2|3)" local address: LID 0x02, QPN 0x1a0406, PSN 0x5379e1 RKey 0x1040436 VAddr 0x00000008051001 remote address: LID 0x01, QPN 0x1d0406, PSN 0xd7aed6, RKey 0x188043c VAddr 0x00000008051001 #, usec 1, 12.8368 2, 4.94677 12, 7.0644 13, 87.8933 26, 64.8241 109, 7.10019 166, 5.88006 193, 3.47097 250, 11.6861 305, 11.9567 388, 6.57047 445, 5.39661 529, 6.8158 536, 3.46172 586, 5.7806 671, 4.3199 728, 5.78413 813, 4.36494 870, 38.4934 902, 3.40266 944, 6.93949 Size: 1, TX-depth: 1, Exchanges: 1000 Latency typical: 3.26619 usec Latency best : 3.17244 usec Latency worst : 87.8933 usec From iod00d at hp.com Fri May 20 08:52:21 2005 From: iod00d at hp.com (Grant Grundler) Date: Fri, 20 May 2005 08:52:21 -0700 Subject: [openib-general] perftest on i386 In-Reply-To: <20050520145944.GZ15688@aon.at> References: <20050518013821.GO1078@esmail.cup.hp.com> <20050518135130.GE25713@mellanox.co.il> <20050518180620.GA6004@esmail.cup.hp.com> <20050518182448.GC7553@aon.at> <20050518184436.GA27680@mellanox.co.il> <20050518185631.GE7553@aon.at> <20050520143558.GY15688@aon.at> <20050520145944.GZ15688@aon.at> Message-ID: <20050520155221.GA14411@esmail.cup.hp.com> On Fri, May 20, 2005 at 04:59:44PM +0200, Bernhard Fischer wrote: > Hi, > > On i386 UP, i see odd results during runs. The machine is basically > idle. The runs are really short if you are using the default (-n 1000). I don't expect them to be visible to something like top or vmstat. > The worst latency is always somewhere around exchange 12-80. > Is this expected and if not, does somebody have an idea what may cause > this? It probably depends on which x86 chipset you are using and what else might be going on. > Thank you, > > $ ./rdma_lat -i 2 -U -s 1 tst09 -t 1 | egrep -v ", 3.(1|2|3)" > local address: LID 0x02, QPN 0x1a0406, PSN 0x5379e1 RKey 0x1040436 > VAddr 0x00000008051001 > remote address: LID 0x01, QPN 0x1d0406, PSN 0xd7aed6, RKey 0x188043c > VAddr 0x00000008051001 > #, usec > 1, 12.8368 Interesting that this is only ~4x normal latency. > 2, 4.94677 > 12, 7.0644 > 13, 87.8933 > 26, 64.8241 Could it be we are crossing a cacheline boundary someplace every 13 or 14 exchanges? Any simple statistical method to determine if the long delays are cyclical? grant From iod00d at hp.com Fri May 20 09:11:46 2005 From: iod00d at hp.com (Grant Grundler) Date: Fri, 20 May 2005 09:11:46 -0700 Subject: [openib-general] performance counters in /sys In-Reply-To: <428DC946.5080207@hp.com> References: <35EA21F54A45CB47B879F21A91F4862F581222@taurus.voltaire.com> <1116539301.4479.255.camel@hal.voltaire.com> <428DC946.5080207@hp.com> Message-ID: <20050520161146.GB14411@esmail.cup.hp.com> On Fri, May 20, 2005 at 07:25:58AM -0400, Mark Seger wrote: > I didn't realize some counters are optional. That kind of blows my > notion of a fixed format, space separated output. 8-( Not entirely. One could fill unimplemented values with -1 or 0. > Perhaps a better way to focus this conversation is to see what the > general feeling is towards statistics that are more oriented towards > consumption by programs rather than humans and then discuss the details > of format if some consensus can be reached. I'm also wondering if data > is to be combined for mulitple hcas/ports it may make more sense to put > it into /proc where those types of structures appear, but that level of > discussion should probably also be deferred. I think the general trend is to move away from /proc. For monitoring performance, a single /sys file that lists all devices is the lightest weight and is acceptable according to Documentation/filesystems/sysfs.txt: | Attributes should be ASCII text files, preferably with only one value | per file. It is noted that it may not be efficient to contain only | value per file, so it is socially acceptable to express an array of | values of the same type. Hrm...this means we'd still need 16+ arrays...not much of an improvement. Any other thoughts? grant From mshefty at ichips.intel.com Fri May 20 09:14:22 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Fri, 20 May 2005 09:14:22 -0700 Subject: [openib-general] OOPS: ib_mad crashery on bootup In-Reply-To: <1116565587.10540.3.camel@duffman> References: <1116565587.10540.3.camel@duffman> Message-ID: <428E0CDE.2050403@ichips.intel.com> Tom Duffy wrote: > On Thu, 2005-05-19 at 21:30 -0700, Shirley Ma wrote: > >>>Error, some other host already uses address 192.168.0.233. >> >>I hit a simliar problem a while ago. But I couldn't reproduct it since >>then. >>Is this reproducible if you configure duplicate IP addresses? > > > I don't think this particular error is related to the crash I observed. > I see this on normal bootup of FC3 w/ ib0 configured. > > The crash I saw happened once, but upon reboot was not reproducible. I have not been able to reproduce this. It may be related to a race condition in the MAD layer, but I don't see where at this point. - Sean From rminnich at lanl.gov Fri May 20 09:20:29 2005 From: rminnich at lanl.gov (Ronald G. Minnich) Date: Fri, 20 May 2005 10:20:29 -0600 (MDT) Subject: [openib-general] performance counters in /sys In-Reply-To: <20050520161146.GB14411@esmail.cup.hp.com> References: <35EA21F54A45CB47B879F21A91F4862F581222@taurus.voltaire.com> <1116539301.4479.255.camel@hal.voltaire.com> <428DC946.5080207@hp.com> <20050520161146.GB14411@esmail.cup.hp.com> Message-ID: On Fri, 20 May 2005, Grant Grundler wrote: > Not entirely. One could fill unimplemented values with -1 or 0. or use s-expressions a la supermon. That's worked the best for us in widely varying environments. These fixed-format tables of the type that /proc delivers are painful. > For monitoring performance, a single /sys file that lists all devices > is the lightest weight and is acceptable according to > Documentation/filesystems/sysfs.txt: > > | Attributes should be ASCII text files, preferably with only one value > | per file. It is noted that it may not be efficient to contain only > | value per file, so it is socially acceptable to express an array of > | values of the same type. yes, that is the accepted thing nowadays. And, boy, is it SLOW. SLOW, SLOW, SLOW. Open file, read ONE value, close file. Takes a long time. So we've seen that for a number of values you really can't sample at interesting rates. Of course, you can speed it up a bit -- keep your files open, right? open once, read, fseek 0, read, ... Except then, if you need to unload the module, well, you can't: you have the /sys files open and the module can't really be unloaded. Of course, if you only need values very 10 seconds or so, /sys is ok. But if you're look at 100 or 1khz, monitoring, which we do sometimes here, /sys is entirely inadequate. /sys is wonderful, and all, but it's not for high speed monitoring. If I understood what to look for from the driver better (I lack knowledge in this IB area :-) I could write a supermon kernel module for it. Again, this only matters if you need performance, I'm not sure what people need here. ron From iod00d at hp.com Fri May 20 09:48:43 2005 From: iod00d at hp.com (Grant Grundler) Date: Fri, 20 May 2005 09:48:43 -0700 Subject: [openib-general] performance counters in /sys In-Reply-To: References: <35EA21F54A45CB47B879F21A91F4862F581222@taurus.voltaire.com> <1116539301.4479.255.camel@hal.voltaire.com> <428DC946.5080207@hp.com> <20050520161146.GB14411@esmail.cup.hp.com> Message-ID: <20050520164843.GC14411@esmail.cup.hp.com> On Fri, May 20, 2005 at 10:20:29AM -0600, Ronald G. Minnich wrote: > yes, that is the accepted thing nowadays. And, boy, is it SLOW. SLOW, > SLOW, SLOW. > Open file, read ONE value, close file. Takes a long time. So we've seen > that for a number of values you really can't sample at interesting rates. Exactly. That's Mark's gripe too. > Of course, you can speed it up a bit -- keep your files open, right? > open once, read, fseek 0, read, ... > Except then, if you need to unload the module, well, you can't: you have > the /sys files open and the module can't really be unloaded. That's good, IMHO. Unloading the HCA driver will disable the device. On ia64 platforms (and probably PPC too) will lead to hard fail on the next MMIO read since the device won't respond to MMIO space accesses. "lsof /sys/class/infiniband" would help track down the monitoring tool. > If I understood what to look for from the driver better (I lack knowledge > in this IB area :-) I could write a supermon kernel module for it. The location/format of existing IB perf counters is advertised today. See code in drivers/infiniband/core/sysfs.c that deals with: static struct attribute_group pma_group = { .name = "counters", .attrs = pma_attrs }; > Again, this only matters if you need performance, I'm not sure what > people need here. I talked with Mark Seger yesterday and he's looking for once per second or subsecond reads - not Khz sample rates. One answer might be to ignore convention and add an "pma_all_attr". For one or two cards, this should be sufficient. I think the right answer for Khz sample rates would be to add an interface to a user space lib (e.g. libibverbs) that exports something like "ethtool -d" uses to dump the NIC registers. thanks, grant From xma at us.ibm.com Fri May 20 09:54:18 2005 From: xma at us.ibm.com (Shirley Ma) Date: Fri, 20 May 2005 09:54:18 -0700 Subject: [openib-general] OOPS: ib_mad crashery on bootup In-Reply-To: <428E0CDE.2050403@ichips.intel.com> Message-ID: > Code: 4c 8b 7d 20 48 89 04 24 48 89 ef 31 db e8 bf 3c 25 f8 49 8b > RIP {:ib_mad:ib_mad_send_done_handler+31} RSP objdump could help where is the crash. Thanks Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From xma at us.ibm.com Fri May 20 10:01:58 2005 From: xma at us.ibm.com (Shirley Ma) Date: Fri, 20 May 2005 10:01:58 -0700 Subject: [openib-general] performance counters in /sys In-Reply-To: <20050520164843.GC14411@esmail.cup.hp.com> Message-ID: On Fri, May 20, 2005 at 10:20:29AM -0600, Ronald G. Minnich wrote: > yes, that is the accepted thing nowadays. And, boy, is it SLOW. SLOW, > SLOW, SLOW. > Open file, read ONE value, close file. Takes a long time. So we've seen > that for a number of values you really can't sample at interesting rates. I haven't looked the performance counters yet. netlink is another way to collect info. Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From tduffy at sun.com Fri May 20 10:27:02 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 20 May 2005 10:27:02 -0700 Subject: [openib-general] OOPS: ib_mad crashery on bootup In-Reply-To: References: Message-ID: <1116610022.10540.17.camel@duffman> On Fri, 2005-05-20 at 09:54 -0700, Shirley Ma wrote: > > > Code: 4c 8b 7d 20 48 89 04 24 48 89 ef 31 db e8 bf 3c 25 f8 49 8b > > RIP {:ib_mad:ib_mad_send_done_handler+31} RSP > > > objdump could help where is the crash. How do you mean? [root at flopteron2 ~]# objdump -S /lib/modules/2.6.12-rc4openib/kernel/drivers/infiniband/core/ib_mad.ko <-- snip --> 0000000000001320 : 1320: 41 57 push %r15 1322: 41 56 push %r14 1324: 49 89 f6 mov %rsi,%r14 1327: 41 55 push %r13 1329: 41 54 push %r12 132b: 55 push %rbp 132c: 53 push %rbx 132d: 48 83 ec 18 sub $0x18,%rsp 1331: 4c 8b 2e mov (%rsi),%r13 1334: 49 8b 6d 10 mov 0x10(%r13),%rbp 1338: 4d 89 ec mov %r13,%r12 133b: 48 8d 45 08 lea 0x8(%rbp),%rax 133f: 4c 8b 7d 20 mov 0x20(%rbp),%r15 1343: 48 89 04 24 mov %rax,(%rsp) 1347: 48 89 ef mov %rbp,%rdi 134a: 31 db xor %ebx,%ebx 134c: e8 00 00 00 00 callq 1351 1351: 49 8b 14 24 mov (%r12),%rdx 1355: 48 89 c6 mov %rax,%rsi 1358: 49 8b 44 24 08 mov 0x8(%r12),%rax 135d: 48 89 42 08 mov %rax,0x8(%rdx) 1361: 48 89 10 mov %rdx,(%rax) 1364: 49 c7 44 24 08 00 02 movq $0x200200,0x8(%r12) 136b: 20 00 136d: 49 c7 04 24 00 01 10 movq $0x100100,(%r12) 1374: 00 1375: 8b 55 18 mov 0x18(%rbp),%edx 1378: 8b 4d 1c mov 0x1c(%rbp),%ecx 137b: 8d 42 ff lea 0xffffffffffffffff(%rdx),%eax 137e: 39 ca cmp %ecx,%edx 1380: 89 45 18 mov %eax,0x18(%rbp) 1383: 7e 33 jle 13b8 1385: 49 8b 5f 60 mov 0x60(%r15),%rbx 1389: 48 8b 53 08 mov 0x8(%rbx),%rdx 138d: 48 8b 03 mov (%rbx),%rax 1390: 49 89 dc mov %rbx,%r12 1393: 48 89 50 08 mov %rdx,0x8(%rax) 1397: 48 89 02 mov %rax,(%rdx) 139a: 48 c7 43 08 00 02 20 movq $0x200200,0x8(%rbx) 13a1: 00 13a2: 48 8b 14 24 mov (%rsp),%rdx 13a6: 48 8b 42 08 mov 0x8(%rdx),%rax 13aa: 48 89 13 mov %rdx,(%rbx) 13ad: 48 89 5a 08 mov %rbx,0x8(%rdx) 13b1: 48 89 18 mov %rbx,(%rax) 13b4: 48 89 43 08 mov %rax,0x8(%rbx) 13b8: 48 89 ef mov %rbp,%rdi 13bb: e8 00 00 00 00 callq 13c0 13c0: 49 8b 85 a0 00 00 00 mov 0xa0(%r13),%rax 13c7: 49 89 06 mov %rax,(%r14) 13ca: 41 8b 87 84 00 00 00 mov 0x84(%r15),%eax 13d1: 85 c0 test %eax,%eax 13d3: 74 1b je 13f0 13d5: 49 8d 75 30 lea 0x30(%r13),%rsi 13d9: b9 04 00 00 00 mov $0x4,%ecx 13de: 4c 89 f2 mov %r14,%rdx 13e1: 4c 89 ff mov %r15,%rdi 13e4: e8 77 ec ff ff callq 60 13e9: 66 data16 13ea: 66 data16 13eb: 66 data16 13ec: 90 nop 13ed: 66 data16 13ee: 66 data16 13ef: 90 nop 13f0: 4c 89 f6 mov %r14,%rsi 13f3: 4c 89 ef mov %r13,%rdi 13f6: e8 00 00 00 00 callq 13fb 13fb: 48 85 db test %rbx,%rbx 13fe: 74 3a je 143a 1400: 49 8b 7f 08 mov 0x8(%r15),%rdi 1404: 48 8d 73 30 lea 0x30(%rbx),%rsi 1408: 48 8d 54 24 10 lea 0x10(%rsp),%rdx 140d: 48 8b 07 mov (%rdi),%rax 1410: ff 90 60 01 00 00 callq *0x160(%rax) 1416: 85 c0 test %eax,%eax 1418: 74 20 je 143a 141a: 89 c6 mov %eax,%esi 141c: 48 c7 c7 00 00 00 00 mov $0x0,%rdi 1423: 31 c0 xor %eax,%eax 1425: 49 89 dd mov %rbx,%r13 1428: e8 00 00 00 00 callq 142d 142d: 41 c7 46 08 02 00 00 movl $0x2,0x8(%r14) 1434: 00 1435: e9 0d ff ff ff jmpq 1347 143a: 48 83 c4 18 add $0x18,%rsp 143e: 5b pop %rbx 143f: 5d pop %rbp 1440: 41 5c pop %r12 1442: 41 5d pop %r13 1444: 41 5e pop %r14 1446: 41 5f pop %r15 1448: c3 retq 1449: 66 data16 144a: 66 data16 144b: 66 data16 144c: 90 nop 144d: 66 data16 144e: 66 data16 144f: 90 nop -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From tduffy at sun.com Fri May 20 10:51:19 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 20 May 2005 10:51:19 -0700 Subject: [openib-general] [PATCH] kDAPL & at: sizeof is not a function Message-ID: <1116611479.10540.23.camel@duffman> sizeof is not a function. Also it is generally better to take the sizeof the actual object your are memcpy()ing or kmalloc()ing in case the type changes. Signed-off-by: Tom Duffy Index: linux-kernel-sizeof/dat-provider/dapl_cookie.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_cookie.c (revision 2423) +++ linux-kernel-sizeof/dat-provider/dapl_cookie.c (working copy) @@ -123,7 +123,7 @@ u32 dapl_cb_create(DAPL_COOKIE_BUFFER *b */ size++; - buffer->pool = kmalloc(size * sizeof(DAPL_COOKIE), GFP_ATOMIC); + buffer->pool = kmalloc(size * sizeof *buffer->pool, GFP_ATOMIC); if (NULL != buffer->pool) { buffer->pool_size = size; atomic_set(&buffer->head, 0); Index: linux-kernel-sizeof/dat-provider/dapl_ep_connect.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_ep_connect.c (revision 2423) +++ linux-kernel-sizeof/dat-provider/dapl_ep_connect.c (working copy) @@ -281,7 +281,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han DAT_EP_STATE_ACTIVE_CONNECTION_PENDING && timeout != DAT_TIMEOUT_INFINITE) { ep_ptr->cxn_timer = (DAPL_OS_TIMER *) - kmalloc(sizeof(DAPL_OS_TIMER), GFP_ATOMIC); + kmalloc(sizeof *ep_ptr->cxn_timer, GFP_ATOMIC); dapl_timer_set(ep_ptr->cxn_timer, dapl_ep_timeout, ep_ptr, timeout); Index: linux-kernel-sizeof/dat-provider/dapl_ring_buffer_util.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_ring_buffer_util.c (revision 2423) +++ linux-kernel-sizeof/dat-provider/dapl_ring_buffer_util.c (working copy) @@ -72,7 +72,7 @@ u32 dapl_rbuf_alloc(DAPL_RING_BUFFER *rb rsize <<= 1; } - rbuf->base = (void *) kmalloc(rsize * sizeof(void *), GFP_ATOMIC); + rbuf->base = (void *) kmalloc(rsize * sizeof *rbuf->base, GFP_ATOMIC); if (rbuf->base != NULL) { rbuf->lim = rsize - 1; atomic_set(&rbuf->head, 0); Index: linux-kernel-sizeof/dat-provider/dapl_lmr_util.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_lmr_util.c (revision 2423) +++ linux-kernel-sizeof/dat-provider/dapl_lmr_util.c (working copy) @@ -48,13 +48,13 @@ DAPL_LMR *dapl_lmr_alloc(DAPL_IA * ia, DAPL_LMR *lmr; /* Allocate LMR */ - lmr = (DAPL_LMR *) kmalloc(sizeof(DAPL_LMR), GFP_ATOMIC); + lmr = (DAPL_LMR *) kmalloc(sizeof *lmr, GFP_ATOMIC); if (NULL == lmr) { return NULL; } /* zero the structure */ - memset(lmr, 0, sizeof(DAPL_LMR)); + memset(lmr, 0, sizeof *lmr); /* * initialize the header Index: linux-kernel-sizeof/dat-provider/dapl_hca_util.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_hca_util.c (revision 2423) +++ linux-kernel-sizeof/dat-provider/dapl_hca_util.c (working copy) @@ -60,9 +60,9 @@ DAPL_HCA *dapl_hca_alloc(char *name, str { DAPL_HCA *hca_ptr; - hca_ptr = kmalloc(sizeof(DAPL_HCA), GFP_ATOMIC); + hca_ptr = kmalloc(sizeof *hca_ptr, GFP_ATOMIC); if (NULL != hca_ptr) { - memset(hca_ptr, 0, sizeof(DAPL_HCA)); + memset(hca_ptr, 0, sizeof *hca_ptr); if (DAT_SUCCESS == dapl_hash_create(DAPL_HASH_TABLE_DEFAULT_CAPACITY, Index: linux-kernel-sizeof/dat-provider/dapl_cr_util.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_cr_util.c (revision 2423) +++ linux-kernel-sizeof/dat-provider/dapl_cr_util.c (working copy) @@ -42,13 +42,13 @@ DAPL_CR *dapl_cr_alloc(DAPL_IA * ia_ptr) DAPL_CR *cr_ptr; /* Allocate EP */ - cr_ptr = (DAPL_CR *) kmalloc(sizeof(DAPL_CR), GFP_ATOMIC); + cr_ptr = (DAPL_CR *) kmalloc(sizeof *cr_ptr, GFP_ATOMIC); if (cr_ptr == NULL) { return NULL; } /* zero the structure */ - memset(cr_ptr, 0, sizeof(DAPL_CR)); + memset(cr_ptr, 0, sizeof *cr_ptr); /* * initialize the header Index: linux-kernel-sizeof/dat-provider/dapl_srq_util.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_srq_util.c (revision 2423) +++ linux-kernel-sizeof/dat-provider/dapl_srq_util.c (working copy) @@ -59,14 +59,14 @@ DAPL_SRQ *dapl_srq_alloc(DAPL_IA * ia_pt DAPL_SRQ *srq_ptr; /* Allocate SRQ */ - srq_ptr = (DAPL_SRQ *) kmalloc(sizeof(DAPL_SRQ), GFP_ATOMIC); + srq_ptr = (DAPL_SRQ *) kmalloc(sizeof *srq_ptr, GFP_ATOMIC); if (srq_ptr == NULL) { goto bail; } /* zero the structure */ - memset(srq_ptr, 0, sizeof(DAPL_SRQ)); + memset(srq_ptr, 0, sizeof *srq_ptr); /* * initialize the header Index: linux-kernel-sizeof/dat-provider/dapl_pz_util.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_pz_util.c (revision 2423) +++ linux-kernel-sizeof/dat-provider/dapl_pz_util.c (working copy) @@ -57,13 +57,13 @@ DAPL_PZ *dapl_pz_alloc(DAPL_IA * ia) DAPL_PZ *pz; /* Allocate PZ */ - pz = (DAPL_PZ *) kmalloc(sizeof(DAPL_PZ), GFP_ATOMIC); + pz = (DAPL_PZ *) kmalloc(sizeof *pz, GFP_ATOMIC); if (NULL == pz) { return NULL; } /* zero the structure */ - memset(pz, 0, sizeof(DAPL_PZ)); + memset(pz, 0, sizeof *pz); /* * initialize the header Index: linux-kernel-sizeof/dat-provider/dapl_ia_util.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_ia_util.c (revision 2423) +++ linux-kernel-sizeof/dat-provider/dapl_ia_util.c (working copy) @@ -63,13 +63,13 @@ DAPL_IA *dapl_ia_alloc(struct dat_provid DAPL_IA *ia_ptr; /* Allocate IA */ - ia_ptr = (DAPL_IA *) kmalloc(sizeof(DAPL_IA), GFP_ATOMIC); + ia_ptr = (DAPL_IA *) kmalloc(sizeof *ia_ptr, GFP_ATOMIC); if (ia_ptr == NULL) { return NULL; } /* zero the structure */ - memset(ia_ptr, 0, sizeof(DAPL_IA)); + memset(ia_ptr, 0, sizeof *ia_ptr); /* * initialize the header Index: linux-kernel-sizeof/dat-provider/dapl_provider.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_provider.c (revision 2423) +++ linux-kernel-sizeof/dat-provider/dapl_provider.c (working copy) @@ -150,27 +150,25 @@ u32 dapl_provider_list_create(void) /* create the head node */ g_dapl_provider_list.head = - kmalloc(sizeof(DAPL_PROVIDER_LIST_NODE), GFP_ATOMIC); + kmalloc(sizeof *g_dapl_provider_list.head, GFP_ATOMIC); if (NULL == g_dapl_provider_list.head) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto bail; } - memset(g_dapl_provider_list.head, 0, - sizeof(DAPL_PROVIDER_LIST_NODE)); + memset(g_dapl_provider_list.head, 0, sizeof *g_dapl_provider_list.head); /* create the tail node */ g_dapl_provider_list.tail = - kmalloc(sizeof(DAPL_PROVIDER_LIST_NODE), GFP_ATOMIC); + kmalloc(sizeof *g_dapl_provider_list.tail, GFP_ATOMIC); if (NULL == g_dapl_provider_list.tail) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto bail; } - memset(g_dapl_provider_list.tail, 0, - sizeof(DAPL_PROVIDER_LIST_NODE)); + memset(g_dapl_provider_list.tail, 0, sizeof *g_dapl_provider_list.tail); g_dapl_provider_list.head->next = g_dapl_provider_list.tail; g_dapl_provider_list.tail->prev = g_dapl_provider_list.head; @@ -215,7 +213,7 @@ u32 dapl_provider_list_insert(const char u32 status = DAT_SUCCESS; unsigned int len; - cur_node = kmalloc(sizeof(DAPL_PROVIDER_LIST_NODE), GFP_ATOMIC); + cur_node = kmalloc(sizeof *cur_node, GFP_ATOMIC); if (NULL == cur_node) { status = Index: linux-kernel-sizeof/dat-provider/dapl_openib_cm.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_openib_cm.c (revision 2423) +++ linux-kernel-sizeof/dat-provider/dapl_openib_cm.c (working copy) @@ -427,7 +427,7 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han ia_ptr = conn->ep->header.owner_ia; /* Setup QP/CM parameters */ - memset(&conn->param, 0, sizeof(conn->param)); + memset(&conn->param, 0, sizeof conn->param); conn->param.qp_type = IB_QPT_RC; conn->param.qp_num = conn->ep->qp_handle->qp_num; conn->param.private_data = private_data; @@ -441,7 +441,7 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han conn->param.local_cm_response_timeout = DAPL_OPENIB_CM_RESPONSE_TIMEOUT; conn->param.max_cm_retries = DAPL_OPENIB_MAX_CM_RETRIES; - memcpy(&conn->service_id, &remote_conn_qual, sizeof(u64)); + memcpy(&conn->service_id, &remote_conn_qual, sizeof conn->service_id); conn->remote_ia_address = remote_ia_address; conn->dapl_comp.fn = &dapl_rt_comp_handler; @@ -771,7 +771,7 @@ u32 dapl_ib_accept_connection(DAT_CR_HAN ep_ptr->cm_handle = conn; conn->ep = ep_ptr; - memset(&passive_params, 0, sizeof(passive_params)); + memset(&passive_params, 0, sizeof passive_params); passive_params.private_data = prd_ptr; passive_params.private_data_len = private_data_size; passive_params.qp_num = ep_ptr->qp_handle->qp_num; @@ -848,7 +848,7 @@ u32 dapl_ib_cm_remote_addr(DAT_HANDLE da if (!memcmp(&ia_ptr->hca_ptr->ib_trans.gid, &remote_gid, 16)) { memcpy(remote_ia_address, &ia_ptr->hca_ptr->hca_address, - sizeof(struct sockaddr)); + sizeof *remote_ia_address); return DAT_SUCCESS; } Index: linux-kernel-sizeof/dat-provider/dapl_openib_util.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_openib_util.c (revision 2423) +++ linux-kernel-sizeof/dat-provider/dapl_openib_util.c (working copy) @@ -164,7 +164,7 @@ u32 dapl_ib_open_hca(char *name, DAPL_HC { u32 dat_ret; - memset(&hca_ptr->ib_trans.gid.global, 0, sizeof(union ib_gid)); + memset(&hca_ptr->ib_trans.gid.global, 0, sizeof hca_ptr->ib_trans.gid); hca_ptr->ib_trans.lid = 0; dat_ret = dapl_ib_get_hca_ids(hca_ptr->ib_hca_handle, @@ -410,7 +410,7 @@ u32 dapl_ib_mr_register_physical(DAPL_IA array = (u64 *) phys_addr; buf_list = (struct ib_phys_buf *) - kmalloc(length * sizeof(struct ib_phys_buf), GFP_ATOMIC); + kmalloc(length * sizeof *buf_list, GFP_ATOMIC); if (NULL == buf_list) return DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); Index: linux-kernel-sizeof/dat-provider/dapl_ep_modify.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_ep_modify.c (revision 2423) +++ linux-kernel-sizeof/dat-provider/dapl_ep_modify.c (working copy) @@ -385,8 +385,7 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand alloc_ep.send_iov_num = ep_attr1.max_request_iov; alloc_ep.send_iov = kmalloc(ep_attr1.max_request_iov - * sizeof(ib_data_segment_t), - GFP_ATOMIC); + * sizeof *alloc_ep.send_iov, GFP_ATOMIC); if (alloc_ep.recv_iov == NULL) { dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); @@ -396,8 +395,7 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand alloc_ep.recv_iov_num = ep_attr1.max_recv_iov; alloc_ep.recv_iov = kmalloc(ep_attr1.max_recv_iov - * sizeof(ib_data_segment_t), - GFP_ATOMIC); + * sizeof *alloc_ep.recv_iov, GFP_ATOMIC); if (alloc_ep.recv_iov == NULL) { dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); Index: linux-kernel-sizeof/dat-provider/dapl_openib_qp.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_openib_qp.c (revision 2423) +++ linux-kernel-sizeof/dat-provider/dapl_openib_qp.c (working copy) @@ -305,7 +305,7 @@ int dapl_modify_qp_state_to_init(ib_qp_h int qp_attr_mask; int ib_status; - memset(&qp_attr, 0, sizeof(qp_attr)); + memset(&qp_attr, 0, sizeof qp_attr); qp_attr.qp_state = IB_QPS_INIT; qp_attr_mask = IB_QP_STATE; qp_attr.qp_access_flags = @@ -328,7 +328,7 @@ int dapl_modify_qp_state_to_error(ib_qp_ int qp_attr_mask; int ib_status; - memset(&qp_attr, 0, sizeof(qp_attr)); + memset(&qp_attr, 0, sizeof qp_attr); qp_attr.qp_state = IB_QPS_ERR; qp_attr_mask = IB_QP_STATE; ib_status = ib_modify_qp(qp, &qp_attr, qp_attr_mask); @@ -342,7 +342,7 @@ int dapl_modify_qp_state_to_reset(ib_qp_ int qp_attr_mask; int ib_status; - memset(&qp_attr, 0, sizeof(qp_attr)); + memset(&qp_attr, 0, sizeof qp_attr); qp_attr.qp_state = IB_QPS_RESET; qp_attr_mask = IB_QP_STATE; ib_status = ib_modify_qp(qp, &qp_attr, qp_attr_mask); @@ -356,7 +356,7 @@ int dapl_modify_qp_state_to_rtr(struct i int qp_attr_mask; int ib_status; - memset(&qp_attr, 0, sizeof(qp_attr)); + memset(&qp_attr, 0, sizeof qp_attr); qp_attr.qp_state = IB_QPS_RTR; ib_status = ib_cm_init_qp_attr(cm_id, &qp_attr, &qp_attr_mask); @@ -376,7 +376,7 @@ int dapl_modify_qp_state_to_rts(struct i int qp_attr_mask; int ib_status; - memset(&qp_attr, 0, sizeof(qp_attr)); + memset(&qp_attr, 0, sizeof qp_attr); qp_attr.qp_state = IB_QPS_RTS; ib_status = ib_cm_init_qp_attr(cm_id, &qp_attr, &qp_attr_mask); Index: linux-kernel-sizeof/dat-provider/dapl_cno_util.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_cno_util.c (revision 2423) +++ linux-kernel-sizeof/dat-provider/dapl_cno_util.c (working copy) @@ -42,12 +42,12 @@ DAPL_CNO *dapl_cno_alloc(DAPL_IA *ia_ptr { DAPL_CNO *cno_ptr; - cno_ptr = kmalloc(sizeof(*cno_ptr), GFP_ATOMIC); + cno_ptr = kmalloc(sizeof *cno_ptr, GFP_ATOMIC); if (!cno_ptr) return NULL; /* zero the structure */ - memset(cno_ptr, 0, sizeof(*cno_ptr)); + memset(cno_ptr, 0, sizeof *cno_ptr); /* * Initialize the header. Index: linux-kernel-sizeof/dat-provider/dapl_rmr_util.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_rmr_util.c (revision 2423) +++ linux-kernel-sizeof/dat-provider/dapl_rmr_util.c (working copy) @@ -33,13 +33,13 @@ DAPL_RMR *dapl_rmr_alloc(DAPL_PZ * pz) DAPL_RMR *rmr; /* Allocate LMR */ - rmr = (DAPL_RMR *) kmalloc(sizeof(DAPL_RMR), GFP_ATOMIC); + rmr = (DAPL_RMR *) kmalloc(sizeof *rmr, GFP_ATOMIC); if (NULL == rmr) { return NULL; } /* zero the structure */ - memset(rmr, 0, sizeof(DAPL_RMR)); + memset(rmr, 0, sizeof *rmr); /* * initialize the header Index: linux-kernel-sizeof/dat-provider/dapl_ep_util.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_ep_util.c (revision 2423) +++ linux-kernel-sizeof/dat-provider/dapl_ep_util.c (working copy) @@ -60,13 +60,13 @@ DAPL_EP *dapl_ep_alloc(DAPL_IA * ia_ptr, DAPL_EP *ep_ptr; /* Allocate EP */ - ep_ptr = (DAPL_EP *) kmalloc(sizeof(DAPL_EP) + sizeof(struct sockaddr), + ep_ptr = (DAPL_EP *) kmalloc(sizeof *ep_ptr + sizeof (struct sockaddr), GFP_ATOMIC); if (ep_ptr == NULL) goto bail; /* zero the structure */ - memset(ep_ptr, 0, sizeof(DAPL_EP) + sizeof(struct sockaddr)); + memset(ep_ptr, 0, sizeof *ep_ptr + sizeof (struct sockaddr)); /* * initialize the header @@ -125,7 +125,7 @@ DAPL_EP *dapl_ep_alloc(DAPL_IA * ia_ptr, ep_ptr->send_iov_num = ep_ptr->param.ep_attr.max_request_iov; ep_ptr->recv_iov = kmalloc(ep_ptr->recv_iov_num * - sizeof(ib_data_segment_t), GFP_ATOMIC); + sizeof *ep_ptr->recv_iov, GFP_ATOMIC); if (NULL == ep_ptr->recv_iov) { dapl_ep_dealloc(ep_ptr); @@ -134,7 +134,7 @@ DAPL_EP *dapl_ep_alloc(DAPL_IA * ia_ptr, } ep_ptr->send_iov = kmalloc(ep_ptr->send_iov_num * - sizeof(ib_data_segment_t), GFP_ATOMIC); + sizeof *ep_ptr->send_iov, GFP_ATOMIC); if (NULL == ep_ptr->send_iov) { dapl_ep_dealloc(ep_ptr); Index: linux-kernel-sizeof/dat-provider/dapl_evd_util.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_evd_util.c (revision 2423) +++ linux-kernel-sizeof/dat-provider/dapl_evd_util.c (working copy) @@ -132,13 +132,13 @@ DAPL_EVD *dapl_evd_alloc(DAPL_IA * ia_pt DAPL_EVD *evd_ptr; /* Allocate EVD */ - evd_ptr = (DAPL_EVD *) kmalloc(sizeof(DAPL_EVD), GFP_ATOMIC); + evd_ptr = (DAPL_EVD *) kmalloc(sizeof *evd_ptr, GFP_ATOMIC); if (!evd_ptr) { goto bail; } /* zero the structure */ - memset(evd_ptr, 0, sizeof(DAPL_EVD)); + memset(evd_ptr, 0, sizeof *evd_ptr); /* * initialize the header Index: linux-kernel-sizeof/dat-provider/dapl_util.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_util.c (revision 2423) +++ linux-kernel-sizeof/dat-provider/dapl_util.c (working copy) @@ -167,7 +167,7 @@ void dapl_dbg_log(DAPL_DBG_TYPE type, co if (type & g_dapl_dbg_mask) { va_start(args, fmt); - vsnprintf(buf, sizeof(buf), fmt, args); + vsnprintf(buf, sizeof buf, fmt, args); printk(KERN_ALERT "kDAPL: %s", buf); va_end(args); } Index: linux-kernel-sizeof/dat-provider/dapl_sp_util.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_sp_util.c (revision 2423) +++ linux-kernel-sizeof/dat-provider/dapl_sp_util.c (working copy) @@ -64,12 +64,12 @@ DAPL_SP *dapl_sp_alloc(DAPL_IA * ia_ptr, DAPL_SP *sp_ptr; /* Allocate EP */ - sp_ptr = (DAPL_SP *) kmalloc(sizeof(DAPL_SP), GFP_ATOMIC); + sp_ptr = (DAPL_SP *) kmalloc(sizeof *sp_ptr, GFP_ATOMIC); if (sp_ptr == NULL) return NULL; /* zero the structure */ - memset(sp_ptr, 0, sizeof(DAPL_SP)); + memset(sp_ptr, 0, sizeof *sp_ptr); /* * initialize the header Index: linux-kernel-sizeof/dat-provider/dapl_hash.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_hash.c (revision 2423) +++ linux-kernel-sizeof/dat-provider/dapl_hash.c (working copy) @@ -214,7 +214,7 @@ dapl_hash_add(DAPL_HASH_TABLEP p_table, status = TRUE; } else { DAPL_HASH_ELEM *newelement = (DAPL_HASH_ELEM *) - kmalloc(sizeof(DAPL_HASH_ELEM), GFP_ATOMIC); + kmalloc(sizeof *newelement, GFP_ATOMIC); /* * Add an element to the end of the chain */ @@ -324,14 +324,14 @@ dapl_hash_delete_element(DAPL_HASH_ELEM u32 dapl_hash_create(int table_size, DAPL_HASH_TABLE **pp_table) { DAPL_HASH_TABLE *p_table; - int table_length = table_size * sizeof(DAPL_HASH_ELEM); + int table_length = table_size * sizeof (struct DAPL_HASH_ELEM); u32 dat_status = DAT_SUCCESS; int i; dapl_os_assert(pp_table); /* Allocate hash table */ - p_table = kmalloc(sizeof(DAPL_HASH_TABLE), GFP_ATOMIC); + p_table = kmalloc(sizeof *p_table, GFP_ATOMIC); if (NULL == p_table) { dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); @@ -339,7 +339,7 @@ u32 dapl_hash_create(int table_size, DAP } /* Init hash table, allocate and init and buckets */ - memset(p_table, 0, sizeof(DAPL_HASH_TABLE)); + memset(p_table, 0, sizeof *p_table); p_table->tbl_size = table_size; p_table->table = (DAPL_HASH_ELEM *) kmalloc(table_length, GFP_ATOMIC); if (NULL == p_table->table) { Index: linux-kernel-sizeof/patches/at.c =================================================================== --- linux-kernel-sizeof/patches/at.c (revision 2423) +++ linux-kernel-sizeof/patches/at.c (working copy) @@ -253,7 +253,7 @@ static int resolve_ip(struct ib_at_src * src->dev = priv->ca; src->port = priv->port; src->pkey = cpu_to_be16(priv->pkey); - memcpy(&src->gid, ipoib_dev->dev_addr + 4, sizeof(src->gid)); + memcpy(&src->gid, ipoib_dev->dev_addr + 4, sizeof src->gid); if (!dgid) return 0; @@ -263,17 +263,13 @@ static int resolve_ip(struct ib_at_src * * the IB device which was found. */ if (rt->u.dst.neighbour->dev->flags & IFF_LOOPBACK) { - memcpy(dgid, ipoib_dev->dev_addr + 4, - sizeof(union ib_gid)); - + memcpy(dgid, ipoib_dev->dev_addr + 4, sizeof *dgid); return 1; } if ((NUD_CONNECTED|NUD_DELAY|NUD_PROBE) & rt->u.dst.neighbour->nud_state) { - memcpy(dgid, rt->u.dst.neighbour->ha + 4, - sizeof(union ib_gid)); - + memcpy(dgid, rt->u.dst.neighbour->ha + 4, sizeof *dgid); return 1; } @@ -300,7 +296,7 @@ static u64 alloc_req_id(void) static void req_init(struct async *pend, void *data, int nelem, int type, struct ib_at_completion *async_comp) { - memset(pend, 0, sizeof(*pend)); + memset(pend, 0, sizeof *pend); pend->id = async_comp->req_id = alloc_req_id(); pend->status = IB_AT_STATUS_INVALID; @@ -454,8 +450,8 @@ static int same_path_req(struct async *a struct path_req *pa = container_of(a, struct path_req, pend); struct path_req *pb = container_of(b, struct path_req, pend); - return !memcmp(&pa->rt.sgid, &pb->rt.sgid, sizeof(pa->rt.sgid)) && - !memcmp(&pa->rt.dgid, &pb->rt.dgid, sizeof(pa->rt.dgid)) && + return !memcmp(&pa->rt.sgid, &pb->rt.sgid, sizeof pa->rt.sgid) && + !memcmp(&pa->rt.dgid, &pb->rt.dgid, sizeof pa->rt.dgid) && pa->rt.out_dev == pb->rt.out_dev && pa->rt.out_port == pb->rt.out_port && pa->rt.attr.pkey == pb->rt.attr.pkey && @@ -784,7 +780,7 @@ static int ib_at_arp_recv(struct sk_buff * queue IB arp packet onto work queue. */ DEBUG("recv IB ARP - queue work"); - work = kmalloc(sizeof(*work), GFP_ATOMIC); + work = kmalloc(sizeof *work, GFP_ATOMIC); if (!work) goto done; @@ -1000,7 +996,7 @@ static int ib_at_init(void) * Init memory pools */ route_req_cache = kmem_cache_create("ib_at_routes", - sizeof(struct route_req), + sizeof (struct route_req), 0, SLAB_HWCACHE_ALIGN, NULL, NULL); if (!route_req_cache) { @@ -1010,7 +1006,7 @@ static int ib_at_init(void) } path_req_cache = kmem_cache_create("ib_at_paths", - sizeof(struct path_req), + sizeof (struct path_req), 0, SLAB_HWCACHE_ALIGN, NULL, NULL); if (!path_req_cache) { Index: linux-kernel-sizeof/dat/dictionary.c =================================================================== --- linux-kernel-sizeof/dat/dictionary.c (revision 2423) +++ linux-kernel-sizeof/dat/dictionary.c (working copy) @@ -96,7 +96,7 @@ u32 dat_dictionary_entry_create(struct d BUG_ON(NULL == p_entry); - entry = kmalloc(sizeof(*entry), GFP_ATOMIC); + entry = kmalloc(sizeof *entry, GFP_ATOMIC); if (NULL == entry) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); Index: linux-kernel-sizeof/dat/dr.c =================================================================== --- linux-kernel-sizeof/dat/dr.c (revision 2423) +++ linux-kernel-sizeof/dat/dr.c (working copy) @@ -84,7 +84,7 @@ u32 dat_dr_insert(const struct dat_provi unsigned long flags; u32 status; - data = kmalloc(sizeof(*data), GFP_ATOMIC); + data = kmalloc(sizeof *data, GFP_ATOMIC); if (NULL == data) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); @@ -241,7 +241,7 @@ u32 dat_dr_list(int max_to_return, int * goto bail; } - array = kmalloc(array_size * sizeof(*array), GFP_ATOMIC); + array = kmalloc(array_size * sizeof *array, GFP_ATOMIC); if (NULL == array) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, Index: linux-kernel-sizeof/dat/core.c =================================================================== --- linux-kernel-sizeof/dat/core.c (revision 2423) +++ linux-kernel-sizeof/dat/core.c (working copy) @@ -66,7 +66,7 @@ void dat_dbg_print(DAT_DBG_TYPE type, co va_list args; va_start(args, fmt); - vsnprintf(buf, sizeof(buf), fmt, args); + vsnprintf(buf, sizeof buf, fmt, args); printk(KERN_ALERT "DAT: %s", buf); va_end(args); } From mshefty at ichips.intel.com Fri May 20 10:58:50 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Fri, 20 May 2005 10:58:50 -0700 Subject: [openib-general] [PATCH] kDAPL & at: sizeof is not a function In-Reply-To: <1116611479.10540.23.camel@duffman> References: <1116611479.10540.23.camel@duffman> Message-ID: <428E255A.3020600@ichips.intel.com> Tom Duffy wrote: > - rbuf->base = (void *) kmalloc(rsize * sizeof(void *), GFP_ATOMIC); > + rbuf->base = (void *) kmalloc(rsize * sizeof *rbuf->base, GFP_ATOMIC); Should this be sizeof rbuf->base? Also, do we need any of the casts before the kmallocs? - Sean From tduffy at sun.com Fri May 20 11:04:53 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 20 May 2005 11:04:53 -0700 Subject: [openib-general] [PATCH] kDAPL & at: sizeof is not a function In-Reply-To: <428E255A.3020600@ichips.intel.com> References: <1116611479.10540.23.camel@duffman> <428E255A.3020600@ichips.intel.com> Message-ID: <1116612293.10540.36.camel@duffman> On Fri, 2005-05-20 at 10:58 -0700, Sean Hefty wrote: > Tom Duffy wrote: > > - rbuf->base = (void *) kmalloc(rsize * sizeof(void *), GFP_ATOMIC); > > + rbuf->base = (void *) kmalloc(rsize * sizeof *rbuf->base, GFP_ATOMIC); > > Should this be sizeof rbuf->base? rbuf->base is a void **. I guess it really doesn't matter since they are the same size. > Also, do we need any of the casts before the kmallocs? Nope. Another patch to get rid of them is forthcoming. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From xma at us.ibm.com Fri May 20 11:33:12 2005 From: xma at us.ibm.com (Shirley Ma) Date: Fri, 20 May 2005 11:33:12 -0700 Subject: [openib-general] OOPS: ib_mad crashery on bootup In-Reply-To: <1116610022.10540.17.camel@duffman> Message-ID: How about output of objdump -S /lib/modules/2.6.12-rc4openib/kernel/drivers/infiniband/core/mad.o? Thanks Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From xma at us.ibm.com Fri May 20 11:39:29 2005 From: xma at us.ibm.com (Shirley Ma) Date: Fri, 20 May 2005 11:39:29 -0700 Subject: [openib-general] OOPS: ib_mad crashery on bootup In-Reply-To: Message-ID: I meant the source code tree not the lib. objdump -S ...2.6.12-rc4openib/kernel/drivers/infiniband/core/mad.o? Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From halr at voltaire.com Fri May 20 11:37:29 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 20 May 2005 14:37:29 -0400 Subject: [openib-general] OOPS: ib_mad crashery on bootup In-Reply-To: <1116610022.10540.17.camel@duffman> References: <1116610022.10540.17.camel@duffman> Message-ID: <1116614249.4479.176.camel@hal.voltaire.com> On Fri, 2005-05-20 at 13:27, Tom Duffy wrote: > On Fri, 2005-05-20 at 09:54 -0700, Shirley Ma wrote: > > > > > Code: 4c 8b 7d 20 48 89 04 24 48 89 ef 31 db e8 bf 3c 25 f8 49 8b > > > RIP {:ib_mad:ib_mad_send_done_handler+31} ib_mad_send_done_handler+31 looks like the spin_lock_irqsave call > RSP > > > > > > objdump could help where is the crash. > > How do you mean? > > [root at flopteron2 ~]# objdump -S /lib/modules/2.6.12-rc4openib/kernel/drivers/infiniband/core/ib_mad.ko > > <-- snip --> > > 0000000000001320 : > 1320: 41 57 push %r15 > 1322: 41 56 push %r14 > 1324: 49 89 f6 mov %rsi,%r14 > 1327: 41 55 push %r13 > 1329: 41 54 push %r12 > 132b: 55 push %rbp > 132c: 53 push %rbx > 132d: 48 83 ec 18 sub $0x18,%rsp > 1331: 4c 8b 2e mov (%rsi),%r13 > 1334: 49 8b 6d 10 mov 0x10(%r13),%rbp > 1338: 4d 89 ec mov %r13,%r12 > 133b: 48 8d 45 08 lea 0x8(%rbp),%rax > 133f: 4c 8b 7d 20 mov 0x20(%rbp),%r15 > 1343: 48 89 04 24 mov %rax,(%rsp) > 1347: 48 89 ef mov %rbp,%rdi > 134a: 31 db xor %ebx,%ebx > 134c: e8 00 00 00 00 callq 1351 > 1351: 49 8b 14 24 mov (%r12),%rdx > 1355: 48 89 c6 mov %rax,%rsi > 1358: 49 8b 44 24 08 mov 0x8(%r12),%rax > 135d: 48 89 42 08 mov %rax,0x8(%rdx) > 1361: 48 89 10 mov %rdx,(%rax) > 1364: 49 c7 44 24 08 00 02 movq $0x200200,0x8(%r12) > 136b: 20 00 > 136d: 49 c7 04 24 00 01 10 movq $0x100100,(%r12) > 1374: 00 > 1375: 8b 55 18 mov 0x18(%rbp),%edx > 1378: 8b 4d 1c mov 0x1c(%rbp),%ecx > 137b: 8d 42 ff lea 0xffffffffffffffff(%rdx),%eax > 137e: 39 ca cmp %ecx,%edx > 1380: 89 45 18 mov %eax,0x18(%rbp) > 1383: 7e 33 jle 13b8 > 1385: 49 8b 5f 60 mov 0x60(%r15),%rbx > 1389: 48 8b 53 08 mov 0x8(%rbx),%rdx > 138d: 48 8b 03 mov (%rbx),%rax > 1390: 49 89 dc mov %rbx,%r12 > 1393: 48 89 50 08 mov %rdx,0x8(%rax) > 1397: 48 89 02 mov %rax,(%rdx) > 139a: 48 c7 43 08 00 02 20 movq $0x200200,0x8(%rbx) > 13a1: 00 > 13a2: 48 8b 14 24 mov (%rsp),%rdx > 13a6: 48 8b 42 08 mov 0x8(%rdx),%rax > 13aa: 48 89 13 mov %rdx,(%rbx) > 13ad: 48 89 5a 08 mov %rbx,0x8(%rdx) > 13b1: 48 89 18 mov %rbx,(%rax) > 13b4: 48 89 43 08 mov %rax,0x8(%rbx) > 13b8: 48 89 ef mov %rbp,%rdi > 13bb: e8 00 00 00 00 callq 13c0 > 13c0: 49 8b 85 a0 00 00 00 mov 0xa0(%r13),%rax > 13c7: 49 89 06 mov %rax,(%r14) > 13ca: 41 8b 87 84 00 00 00 mov 0x84(%r15),%eax > 13d1: 85 c0 test %eax,%eax > 13d3: 74 1b je 13f0 > 13d5: 49 8d 75 30 lea 0x30(%r13),%rsi > 13d9: b9 04 00 00 00 mov $0x4,%ecx > 13de: 4c 89 f2 mov %r14,%rdx > 13e1: 4c 89 ff mov %r15,%rdi > 13e4: e8 77 ec ff ff callq 60 > 13e9: 66 data16 > 13ea: 66 data16 > 13eb: 66 data16 > 13ec: 90 nop > 13ed: 66 data16 > 13ee: 66 data16 > 13ef: 90 nop > 13f0: 4c 89 f6 mov %r14,%rsi > 13f3: 4c 89 ef mov %r13,%rdi > 13f6: e8 00 00 00 00 callq 13fb > 13fb: 48 85 db test %rbx,%rbx > 13fe: 74 3a je 143a > 1400: 49 8b 7f 08 mov 0x8(%r15),%rdi > 1404: 48 8d 73 30 lea 0x30(%rbx),%rsi > 1408: 48 8d 54 24 10 lea 0x10(%rsp),%rdx > 140d: 48 8b 07 mov (%rdi),%rax > 1410: ff 90 60 01 00 00 callq *0x160(%rax) > 1416: 85 c0 test %eax,%eax > 1418: 74 20 je 143a > 141a: 89 c6 mov %eax,%esi > 141c: 48 c7 c7 00 00 00 00 mov $0x0,%rdi > 1423: 31 c0 xor %eax,%eax > 1425: 49 89 dd mov %rbx,%r13 > 1428: e8 00 00 00 00 callq 142d > 142d: 41 c7 46 08 02 00 00 movl $0x2,0x8(%r14) > 1434: 00 > 1435: e9 0d ff ff ff jmpq 1347 > 143a: 48 83 c4 18 add $0x18,%rsp > 143e: 5b pop %rbx > 143f: 5d pop %rbp > 1440: 41 5c pop %r12 > 1442: 41 5d pop %r13 > 1444: 41 5e pop %r14 > 1446: 41 5f pop %r15 > 1448: c3 retq > 1449: 66 data16 > 144a: 66 data16 > 144b: 66 data16 > 144c: 90 nop > 144d: 66 data16 > 144e: 66 data16 > 144f: 90 nop > > -tduffy > > > ______________________________________________________________________ > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From tduffy at sun.com Fri May 20 11:54:01 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 20 May 2005 11:54:01 -0700 Subject: [openib-general] OOPS: ib_mad crashery on bootup In-Reply-To: References: Message-ID: <1116615241.10540.52.camel@duffman> On Fri, 2005-05-20 at 11:33 -0700, Shirley Ma wrote: > > How about output of > > objdump > -S /lib/modules/2.6.12-rc4openib/kernel/drivers/infiniband/core/mad.o? That path is incorrect, but anyways, I don't see any difference when looking at: objdump -S /build1/tduffy/openib-work/build/openib/rc/drivers/infiniband/core/mad.o <-- snip --> 0000000000001320 : 1320: 41 57 push %r15 1322: 41 56 push %r14 1324: 49 89 f6 mov %rsi,%r14 1327: 41 55 push %r13 1329: 41 54 push %r12 132b: 55 push %rbp 132c: 53 push %rbx 132d: 48 83 ec 18 sub $0x18,%rsp 1331: 4c 8b 2e mov (%rsi),%r13 1334: 49 8b 6d 10 mov 0x10(%r13),%rbp 1338: 4d 89 ec mov %r13,%r12 133b: 48 8d 45 08 lea 0x8(%rbp),%rax 133f: 4c 8b 7d 20 mov 0x20(%rbp),%r15 1343: 48 89 04 24 mov %rax,(%rsp) 1347: 48 89 ef mov %rbp,%rdi 134a: 31 db xor %ebx,%ebx 134c: e8 00 00 00 00 callq 1351 Is there another part of the dump that you are interested in? Here is the relevant mad.s: [linux-2.6.12-rc4-openib]$ make O=/build1/tduffy/openib-work/build/openib/rc/ drivers/infiniband/core/mad.s <-- snip --> .type ib_mad_send_done_handler, @function ib_mad_send_done_handler: pushq %r15 pushq %r14 movq %rsi, %r14 pushq %r13 pushq %r12 pushq %rbp pushq %rbx subq $24, %rsp movq (%rsi), %r13 movq 16(%r13), %rbp movq %r13, %r12 leaq 8(%rbp), %rax movq 32(%rbp), %r15 movq %rax, (%rsp) .p2align 4,,7 .L286: movq %rbp, %rdi xorl %ebx, %ebx call _spin_lock_irqsave movq (%r12), %rdx movq %rax, %rsi movq 8(%r12), %rax movq %rax, 8(%rdx) movq %rdx, (%rax) movq $2097664, 8(%r12) movq $1048832, (%r12) movl 24(%rbp), %edx movl 28(%rbp), %ecx leal -1(%rdx), %eax cmpl %ecx, %edx movl %eax, 24(%rbp) jle .L289 movq 96(%r15), %rbx movq 8(%rbx), %rdx movq (%rbx), %rax movq %rbx, %r12 movq %rdx, 8(%rax) movq %rax, (%rdx) movq $2097664, 8(%rbx) movq (%rsp), %rdx movq 8(%rdx), %rax movq %rdx, (%rbx) movq %rbx, 8(%rdx) movq %rbx, (%rax) movq %rax, 8(%rbx) .L289: movq %rbp, %rdi call _spin_unlock_irqrestore movq 160(%r13), %rax movq %rax, (%r14) movl 132(%r15), %eax testl %eax, %eax je .L290 leaq 48(%r13), %rsi movl $4, %ecx movq %r14, %rdx movq %r15, %rdi call snoop_send .p2align 4,,7 .L290: movq %r14, %rsi movq %r13, %rdi call ib_mad_complete_send_wr testq %rbx, %rbx je .L295 movq 8(%r15), %rdi leaq 48(%rbx), %rsi leaq 16(%rsp), %rdx movq (%rdi), %rax call *352(%rax) testl %eax, %eax je .L295 movl %eax, %esi movq $.LC17, %rdi xorl %eax, %eax movq %rbx, %r13 call printk movl $2, 8(%r14) jmp .L286 .L295: addq $24, %rsp popq %rbx popq %rbp popq %r12 popq %r13 popq %r14 popq %r15 ret .size ib_mad_send_done_handler, .-ib_mad_send_done_handler .p2align 4,,15 Oh, and here it is with CONFIG_DEBUG_INFO=y 0000000000001320 : 1320: 41 57 push %r15 1322: 41 56 push %r14 1324: 49 89 f6 mov %rsi,%r14 1327: 41 55 push %r13 1329: 41 54 push %r12 132b: 55 push %rbp 132c: 53 push %rbx 132d: 48 83 ec 18 sub $0x18,%rsp 1331: 4c 8b 2e mov (%rsi),%r13 1334: 49 8b 6d 10 mov 0x10(%r13),%rbp 1338: 4d 89 ec mov %r13,%r12 133b: 48 8d 45 08 lea 0x8(%rbp),%rax 133f: 4c 8b 7d 20 mov 0x20(%rbp),%r15 1343: 48 89 04 24 mov %rax,(%rsp) 1347: 48 89 ef mov %rbp,%rdi 134a: 31 db xor %ebx,%ebx 134c: e8 00 00 00 00 callq 1351 * in an undefined state. */ static inline void list_del(struct list_head *entry) { __list_del(entry->prev, entry->next); 1351: 49 8b 14 24 mov (%r12),%rdx 1355: 48 89 c6 mov %rax,%rsi * in an undefined state. */ static inline void list_del(struct list_head *entry) { __list_del(entry->prev, entry->next); 1358: 49 8b 44 24 08 mov 0x8(%r12),%rax 135d: 48 89 42 08 mov %rax,0x8(%rdx) 1361: 48 89 10 mov %rdx,(%rax) entry->next = LIST_POISON1; entry->prev = LIST_POISON2; 1364: 49 c7 44 24 08 00 02 movq $0x200200,0x8(%r12) 136b: 20 00 136d: 49 c7 04 24 00 01 10 movq $0x100100,(%r12) 1374: 00 1375: 8b 55 18 mov 0x18(%rbp),%edx 1378: 8b 4d 1c mov 0x1c(%rbp),%ecx 137b: 8d 42 ff lea 0xffffffffffffffff(%rdx),%eax 137e: 39 ca cmp %ecx,%edx 1380: 89 45 18 mov %eax,0x18(%rbp) 1383: 7e 33 jle 13b8 1385: 49 8b 5f 60 mov 0x60(%r15),%rbx * in an undefined state. */ static inline void list_del(struct list_head *entry) { __list_del(entry->prev, entry->next); 1389: 48 8b 53 08 mov 0x8(%rbx),%rdx 138d: 48 8b 03 mov (%rbx),%rax 1390: 49 89 dc mov %rbx,%r12 * the prev/next entries already! */ static inline void __list_del(struct list_head * prev, struct list_head * next) { next->prev = prev; 1393: 48 89 50 08 mov %rdx,0x8(%rax) prev->next = next; 1397: 48 89 02 mov %rax,(%rdx) } /** * list_del - deletes entry from list. * @entry: the element to delete from the list. * Note: list_empty on entry does not return true after this, the entry is * in an undefined state. */ static inline void list_del(struct list_head *entry) { __list_del(entry->prev, entry->next); entry->next = LIST_POISON1; entry->prev = LIST_POISON2; 139a: 48 c7 43 08 00 02 20 movq $0x200200,0x8(%rbx) 13a1: 00 13a2: 48 8b 14 24 mov (%rsp),%rdx 13a6: 48 8b 42 08 mov 0x8(%rdx),%rax 13aa: 48 89 13 mov %rdx,(%rbx) 13ad: 48 89 5a 08 mov %rbx,0x8(%rdx) 13b1: 48 89 18 mov %rbx,(%rax) 13b4: 48 89 43 08 mov %rax,0x8(%rbx) 13b8: 48 89 ef mov %rbp,%rdi 13bb: e8 00 00 00 00 callq 13c0 13c0: 49 8b 85 a0 00 00 00 mov 0xa0(%r13),%rax 13c7: 49 89 06 mov %rax,(%r14) 13ca: 41 8b 87 84 00 00 00 mov 0x84(%r15),%eax 13d1: 85 c0 test %eax,%eax 13d3: 74 1b je 13f0 13d5: 49 8d 75 30 lea 0x30(%r13),%rsi 13d9: b9 04 00 00 00 mov $0x4,%ecx 13de: 4c 89 f2 mov %r14,%rdx 13e1: 4c 89 ff mov %r15,%rdi 13e4: e8 77 ec ff ff callq 60 13e9: 66 data16 13ea: 66 data16 13eb: 66 data16 13ec: 90 nop 13ed: 66 data16 13ee: 66 data16 13ef: 90 nop 13f0: 4c 89 f6 mov %r14,%rsi 13f3: 4c 89 ef mov %r13,%rdi 13f6: e8 00 00 00 00 callq 13fb 13fb: 48 85 db test %rbx,%rbx 13fe: 74 3a je 143a 1400: 49 8b 7f 08 mov 0x8(%r15),%rdi static inline int ib_post_send(struct ib_qp *qp, struct ib_send_wr *send_wr, struct ib_send_wr **bad_send_wr) { return qp->device->post_send(qp, send_wr, bad_send_wr); 1404: 48 8d 73 30 lea 0x30(%rbx),%rsi 1408: 48 8d 54 24 10 lea 0x10(%rsp),%rdx 140d: 48 8b 07 mov (%rdi),%rax 1410: ff 90 60 01 00 00 callq *0x160(%rax) 1416: 85 c0 test %eax,%eax 1418: 74 20 je 143a 141a: 89 c6 mov %eax,%esi 141c: 48 c7 c7 00 00 00 00 mov $0x0,%rdi 1423: 31 c0 xor %eax,%eax 1425: 49 89 dd mov %rbx,%r13 1428: e8 00 00 00 00 callq 142d 142d: 41 c7 46 08 02 00 00 movl $0x2,0x8(%r14) 1434: 00 1435: e9 0d ff ff ff jmpq 1347 143a: 48 83 c4 18 add $0x18,%rsp 143e: 5b pop %rbx 143f: 5d pop %rbp 1440: 41 5c pop %r12 1442: 41 5d pop %r13 1444: 41 5e pop %r14 1446: 41 5f pop %r15 1448: c3 retq 1449: 66 data16 144a: 66 data16 144b: 66 data16 144c: 90 nop 144d: 66 data16 144e: 66 data16 144f: 90 nop -tduffy -- I wish we lived in the America of yesteryear that only exists in the minds of us Republicans. -- Ned Flanders -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From mshefty at ichips.intel.com Fri May 20 12:07:19 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Fri, 20 May 2005 12:07:19 -0700 Subject: [openib-general] OOPS: ib_mad crashery on bootup In-Reply-To: <1116614249.4479.176.camel@hal.voltaire.com> References: <1116610022.10540.17.camel@duffman> <1116614249.4479.176.camel@hal.voltaire.com> Message-ID: <428E3567.1060705@ichips.intel.com> Hal Rosenstock wrote: > On Fri, 2005-05-20 at 13:27, Tom Duffy wrote: > >>On Fri, 2005-05-20 at 09:54 -0700, Shirley Ma wrote: >> >>>>Code: 4c 8b 7d 20 48 89 04 24 48 89 ef 31 db e8 bf 3c 25 f8 49 8b >>>>RIP {:ib_mad:ib_mad_send_done_handler+31} > > > ib_mad_send_done_handler+31 looks like the spin_lock_irqsave call Possibly the MAD or work completion or stack has been corrupted? I've tried to reproduce this running MAD stress tests over just the core, mad, and mt modules, but haven't been successful yet. - Sean From tduffy at sun.com Fri May 20 12:28:41 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 20 May 2005 12:28:41 -0700 Subject: [openib-general] [PATCH] kDAPL no need to cast kmalloc()'s return In-Reply-To: <428E255A.3020600@ichips.intel.com> References: <1116611479.10540.23.camel@duffman> <428E255A.3020600@ichips.intel.com> Message-ID: <1116617321.27740.5.camel@duffman> On Fri, 2005-05-20 at 10:58 -0700, Sean Hefty wrote: > Also, do we need any of the casts before the kmallocs? This patch removes the cast before kmalloc(). It also makes sure to check the return of kmalloc() -- was missing in one spot in dapl_ep_connect.c. Make the NULL check (relatively) consistent across the code. Signed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat/dictionary.c linux-kernel-kmallc/dat/dictionary.c --- linux-kernel-sizeof/dat/dictionary.c 2005-05-20 09:55:56.230003000 -0700 +++ linux-kernel-kmallc/dat/dictionary.c 2005-05-20 12:07:59.344001000 -0700 @@ -97,7 +97,7 @@ u32 dat_dictionary_entry_create(struct d BUG_ON(NULL == p_entry); entry = kmalloc(sizeof *entry, GFP_ATOMIC); - if (NULL == entry) { + if (!entry) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto bail; diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat/dr.c linux-kernel-kmallc/dat/dr.c --- linux-kernel-sizeof/dat/dr.c 2005-05-20 09:56:11.726007000 -0700 +++ linux-kernel-kmallc/dat/dr.c 2005-05-20 12:08:31.954000000 -0700 @@ -85,7 +85,7 @@ u32 dat_dr_insert(const struct dat_provi u32 status; data = kmalloc(sizeof *data, GFP_ATOMIC); - if (NULL == data) { + if (!data) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto bail; @@ -242,7 +242,7 @@ u32 dat_dr_list(int max_to_return, int * } array = kmalloc(array_size * sizeof *array, GFP_ATOMIC); - if (NULL == array) { + if (!array) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_cookie.c linux-kernel-kmallc/dat-provider/dapl_cookie.c --- linux-kernel-sizeof/dat-provider/dapl_cookie.c 2005-05-20 10:47:10.741014000 -0700 +++ linux-kernel-kmallc/dat-provider/dapl_cookie.c 2005-05-20 12:08:49.614000000 -0700 @@ -124,7 +124,7 @@ u32 dapl_cb_create(DAPL_COOKIE_BUFFER *b size++; buffer->pool = kmalloc(size * sizeof *buffer->pool, GFP_ATOMIC); - if (NULL != buffer->pool) { + if (buffer->pool) { buffer->pool_size = size; atomic_set(&buffer->head, 0); atomic_set(&buffer->tail, 0); @@ -135,9 +135,8 @@ u32 dapl_cb_create(DAPL_COOKIE_BUFFER *b } return DAT_SUCCESS; - } else { + } else return DAT_INSUFFICIENT_RESOURCES; - } } /* diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_cr_util.c linux-kernel-kmallc/dat-provider/dapl_cr_util.c --- linux-kernel-sizeof/dat-provider/dapl_cr_util.c 2005-05-20 09:39:00.781014000 -0700 +++ linux-kernel-kmallc/dat-provider/dapl_cr_util.c 2005-05-20 12:03:14.960015000 -0700 @@ -42,10 +42,9 @@ DAPL_CR *dapl_cr_alloc(DAPL_IA * ia_ptr) DAPL_CR *cr_ptr; /* Allocate EP */ - cr_ptr = (DAPL_CR *) kmalloc(sizeof *cr_ptr, GFP_ATOMIC); - if (cr_ptr == NULL) { + cr_ptr = kmalloc(sizeof *cr_ptr, GFP_ATOMIC); + if (!cr_ptr) return NULL; - } /* zero the structure */ memset(cr_ptr, 0, sizeof *cr_ptr); diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_ep_connect.c linux-kernel-kmallc/dat-provider/dapl_ep_connect.c --- linux-kernel-sizeof/dat-provider/dapl_ep_connect.c 2005-05-20 09:40:17.942011000 -0700 +++ linux-kernel-kmallc/dat-provider/dapl_ep_connect.c 2005-05-20 12:13:33.164001000 -0700 @@ -280,8 +280,13 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han if (ep_ptr->param.ep_state == DAT_EP_STATE_ACTIVE_CONNECTION_PENDING && timeout != DAT_TIMEOUT_INFINITE) { - ep_ptr->cxn_timer = (DAPL_OS_TIMER *) - kmalloc(sizeof *ep_ptr->cxn_timer, GFP_ATOMIC); + ep_ptr->cxn_timer = kmalloc(sizeof *ep_ptr->cxn_timer, + GFP_ATOMIC); + if (!ep_ptr->cxn_timer) { + dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, + DAT_RESOURCE_MEMORY); + goto bail; + } dapl_timer_set(ep_ptr->cxn_timer, dapl_ep_timeout, ep_ptr, timeout); diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_ep_modify.c linux-kernel-kmallc/dat-provider/dapl_ep_modify.c --- linux-kernel-sizeof/dat-provider/dapl_ep_modify.c 2005-05-20 10:08:28.558003000 -0700 +++ linux-kernel-kmallc/dat-provider/dapl_ep_modify.c 2005-05-20 12:14:07.358003000 -0700 @@ -386,7 +386,7 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand alloc_ep.send_iov_num = ep_attr1.max_request_iov; alloc_ep.send_iov = kmalloc(ep_attr1.max_request_iov * sizeof *alloc_ep.send_iov, GFP_ATOMIC); - if (alloc_ep.recv_iov == NULL) { + if (!alloc_ep.recv_iov) { dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto bail; @@ -396,7 +396,7 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand alloc_ep.recv_iov_num = ep_attr1.max_recv_iov; alloc_ep.recv_iov = kmalloc(ep_attr1.max_recv_iov * sizeof *alloc_ep.recv_iov, GFP_ATOMIC); - if (alloc_ep.recv_iov == NULL) { + if (!alloc_ep.recv_iov) { dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto bail; diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_ep_util.c linux-kernel-kmallc/dat-provider/dapl_ep_util.c --- linux-kernel-sizeof/dat-provider/dapl_ep_util.c 2005-05-20 10:08:04.002013000 -0700 +++ linux-kernel-kmallc/dat-provider/dapl_ep_util.c 2005-05-20 12:15:02.171014000 -0700 @@ -60,9 +60,8 @@ DAPL_EP *dapl_ep_alloc(DAPL_IA * ia_ptr, DAPL_EP *ep_ptr; /* Allocate EP */ - ep_ptr = (DAPL_EP *) kmalloc(sizeof *ep_ptr + sizeof (struct sockaddr), - GFP_ATOMIC); - if (ep_ptr == NULL) + ep_ptr = kmalloc(sizeof *ep_ptr + sizeof (struct sockaddr), GFP_ATOMIC); + if (!ep_ptr) goto bail; /* zero the structure */ @@ -126,8 +125,7 @@ DAPL_EP *dapl_ep_alloc(DAPL_IA * ia_ptr, ep_ptr->recv_iov = kmalloc(ep_ptr->recv_iov_num * sizeof *ep_ptr->recv_iov, GFP_ATOMIC); - - if (NULL == ep_ptr->recv_iov) { + if (!ep_ptr->recv_iov) { dapl_ep_dealloc(ep_ptr); ep_ptr = NULL; goto bail; @@ -135,8 +133,7 @@ DAPL_EP *dapl_ep_alloc(DAPL_IA * ia_ptr, ep_ptr->send_iov = kmalloc(ep_ptr->send_iov_num * sizeof *ep_ptr->send_iov, GFP_ATOMIC); - - if (NULL == ep_ptr->send_iov) { + if (!ep_ptr->send_iov) { dapl_ep_dealloc(ep_ptr); ep_ptr = NULL; goto bail; diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_evd_util.c linux-kernel-kmallc/dat-provider/dapl_evd_util.c --- linux-kernel-sizeof/dat-provider/dapl_evd_util.c 2005-05-20 09:42:44.127003000 -0700 +++ linux-kernel-kmallc/dat-provider/dapl_evd_util.c 2005-05-20 12:15:18.594000000 -0700 @@ -132,10 +132,9 @@ DAPL_EVD *dapl_evd_alloc(DAPL_IA * ia_pt DAPL_EVD *evd_ptr; /* Allocate EVD */ - evd_ptr = (DAPL_EVD *) kmalloc(sizeof *evd_ptr, GFP_ATOMIC); - if (!evd_ptr) { + evd_ptr = kmalloc(sizeof *evd_ptr, GFP_ATOMIC); + if (!evd_ptr) goto bail; - } /* zero the structure */ memset(evd_ptr, 0, sizeof *evd_ptr); @@ -178,10 +177,8 @@ u32 dapl_evd_event_alloc(DAPL_EVD *evd_p u32 dat_status = DAT_SUCCESS; /* Allocate EVENTs */ - event_ptr = - (struct dat_event *)kmalloc(evd_ptr->qlen * sizeof *event_ptr, - GFP_ATOMIC); - if (event_ptr == NULL) { + event_ptr = kmalloc(evd_ptr->qlen * sizeof *event_ptr, GFP_ATOMIC); + if (!event_ptr) { dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto bail; diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_hash.c linux-kernel-kmallc/dat-provider/dapl_hash.c --- linux-kernel-sizeof/dat-provider/dapl_hash.c 2005-05-20 10:10:06.017003000 -0700 +++ linux-kernel-kmallc/dat-provider/dapl_hash.c 2005-05-20 12:15:55.837003000 -0700 @@ -213,8 +213,8 @@ dapl_hash_add(DAPL_HASH_TABLEP p_table, p_table->num_entries++; status = TRUE; } else { - DAPL_HASH_ELEM *newelement = (DAPL_HASH_ELEM *) - kmalloc(sizeof *newelement, GFP_ATOMIC); + DAPL_HASH_ELEM *newelement = kmalloc(sizeof *newelement, + GFP_ATOMIC); /* * Add an element to the end of the chain */ @@ -232,10 +232,8 @@ dapl_hash_add(DAPL_HASH_TABLEP p_table, lastelement->next_element = newelement; p_table->num_entries++; status = TRUE; - } else { - /* allocation failed - should not happen */ + } else status = FALSE; - } } /* @@ -332,7 +330,7 @@ u32 dapl_hash_create(int table_size, DAP /* Allocate hash table */ p_table = kmalloc(sizeof *p_table, GFP_ATOMIC); - if (NULL == p_table) { + if (!p_table) { dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto bail; @@ -341,8 +339,8 @@ u32 dapl_hash_create(int table_size, DAP /* Init hash table, allocate and init and buckets */ memset(p_table, 0, sizeof *p_table); p_table->tbl_size = table_size; - p_table->table = (DAPL_HASH_ELEM *) kmalloc(table_length, GFP_ATOMIC); - if (NULL == p_table->table) { + p_table->table = kmalloc(table_length, GFP_ATOMIC); + if (!p_table->table) { kfree(p_table); dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_hca_util.c linux-kernel-kmallc/dat-provider/dapl_hca_util.c --- linux-kernel-sizeof/dat-provider/dapl_hca_util.c 2005-05-20 09:44:53.376002000 -0700 +++ linux-kernel-kmallc/dat-provider/dapl_hca_util.c 2005-05-20 12:16:15.827004000 -0700 @@ -61,7 +61,7 @@ DAPL_HCA *dapl_hca_alloc(char *name, str DAPL_HCA *hca_ptr; hca_ptr = kmalloc(sizeof *hca_ptr, GFP_ATOMIC); - if (NULL != hca_ptr) { + if (hca_ptr) { memset(hca_ptr, 0, sizeof *hca_ptr); if (DAT_SUCCESS == diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_ia_util.c linux-kernel-kmallc/dat-provider/dapl_ia_util.c --- linux-kernel-sizeof/dat-provider/dapl_ia_util.c 2005-05-20 09:45:22.314001000 -0700 +++ linux-kernel-kmallc/dat-provider/dapl_ia_util.c 2005-05-20 12:04:44.308003000 -0700 @@ -63,10 +63,9 @@ DAPL_IA *dapl_ia_alloc(struct dat_provid DAPL_IA *ia_ptr; /* Allocate IA */ - ia_ptr = (DAPL_IA *) kmalloc(sizeof *ia_ptr, GFP_ATOMIC); - if (ia_ptr == NULL) { + ia_ptr = kmalloc(sizeof *ia_ptr, GFP_ATOMIC); + if (!ia_ptr) return NULL; - } /* zero the structure */ memset(ia_ptr, 0, sizeof *ia_ptr); diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_lmr_util.c linux-kernel-kmallc/dat-provider/dapl_lmr_util.c --- linux-kernel-sizeof/dat-provider/dapl_lmr_util.c 2005-05-20 09:45:43.253013000 -0700 +++ linux-kernel-kmallc/dat-provider/dapl_lmr_util.c 2005-05-20 12:05:02.135002000 -0700 @@ -48,10 +48,9 @@ DAPL_LMR *dapl_lmr_alloc(DAPL_IA * ia, DAPL_LMR *lmr; /* Allocate LMR */ - lmr = (DAPL_LMR *) kmalloc(sizeof *lmr, GFP_ATOMIC); - if (NULL == lmr) { + lmr = kmalloc(sizeof *lmr, GFP_ATOMIC); + if (!lmr) return NULL; - } /* zero the structure */ memset(lmr, 0, sizeof *lmr); diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_openib_util.c linux-kernel-kmallc/dat-provider/dapl_openib_util.c --- linux-kernel-sizeof/dat-provider/dapl_openib_util.c 2005-05-20 10:02:29.919003000 -0700 +++ linux-kernel-kmallc/dat-provider/dapl_openib_util.c 2005-05-20 12:05:19.439003000 -0700 @@ -409,9 +409,8 @@ u32 dapl_ib_mr_register_physical(DAPL_IA u64 *array; array = (u64 *) phys_addr; - buf_list = (struct ib_phys_buf *) - kmalloc(length * sizeof *buf_list, GFP_ATOMIC); - if (NULL == buf_list) + buf_list = kmalloc(length * sizeof *buf_list, GFP_ATOMIC); + if (!buf_list) return DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_provider.c linux-kernel-kmallc/dat-provider/dapl_provider.c --- linux-kernel-sizeof/dat-provider/dapl_provider.c 2005-05-20 09:53:27.437003000 -0700 +++ linux-kernel-kmallc/dat-provider/dapl_provider.c 2005-05-20 12:06:06.106003000 -0700 @@ -151,7 +151,7 @@ u32 dapl_provider_list_create(void) /* create the head node */ g_dapl_provider_list.head = kmalloc(sizeof *g_dapl_provider_list.head, GFP_ATOMIC); - if (NULL == g_dapl_provider_list.head) { + if (!g_dapl_provider_list.head) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto bail; @@ -162,7 +162,7 @@ u32 dapl_provider_list_create(void) /* create the tail node */ g_dapl_provider_list.tail = kmalloc(sizeof *g_dapl_provider_list.tail, GFP_ATOMIC); - if (NULL == g_dapl_provider_list.tail) { + if (!g_dapl_provider_list.tail) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto bail; @@ -214,8 +214,7 @@ u32 dapl_provider_list_insert(const char unsigned int len; cur_node = kmalloc(sizeof *cur_node, GFP_ATOMIC); - - if (NULL == cur_node) { + if (!cur_node) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto bail; diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_pz_util.c linux-kernel-kmallc/dat-provider/dapl_pz_util.c --- linux-kernel-sizeof/dat-provider/dapl_pz_util.c 2005-05-20 09:53:45.600010000 -0700 +++ linux-kernel-kmallc/dat-provider/dapl_pz_util.c 2005-05-20 12:06:18.950008000 -0700 @@ -57,10 +57,9 @@ DAPL_PZ *dapl_pz_alloc(DAPL_IA * ia) DAPL_PZ *pz; /* Allocate PZ */ - pz = (DAPL_PZ *) kmalloc(sizeof *pz, GFP_ATOMIC); - if (NULL == pz) { + pz = kmalloc(sizeof *pz, GFP_ATOMIC); + if (!pz) return NULL; - } /* zero the structure */ memset(pz, 0, sizeof *pz); diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_ring_buffer_util.c linux-kernel-kmallc/dat-provider/dapl_ring_buffer_util.c --- linux-kernel-sizeof/dat-provider/dapl_ring_buffer_util.c 2005-05-20 10:10:58.091013000 -0700 +++ linux-kernel-kmallc/dat-provider/dapl_ring_buffer_util.c 2005-05-20 12:06:48.434000000 -0700 @@ -72,14 +72,13 @@ u32 dapl_rbuf_alloc(DAPL_RING_BUFFER *rb rsize <<= 1; } - rbuf->base = (void *) kmalloc(rsize * sizeof *rbuf->base, GFP_ATOMIC); - if (rbuf->base != NULL) { + rbuf->base = kmalloc(rsize * sizeof *rbuf->base, GFP_ATOMIC); + if (rbuf->base) { rbuf->lim = rsize - 1; atomic_set(&rbuf->head, 0); atomic_set(&rbuf->tail, 0); - } else { + } else return DAT_INSUFFICIENT_RESOURCES | DAT_RESOURCE_MEMORY; - } return DAT_SUCCESS; } diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_rmr_util.c linux-kernel-kmallc/dat-provider/dapl_rmr_util.c --- linux-kernel-sizeof/dat-provider/dapl_rmr_util.c 2005-05-20 09:54:37.797008000 -0700 +++ linux-kernel-kmallc/dat-provider/dapl_rmr_util.c 2005-05-20 12:07:00.373015000 -0700 @@ -33,10 +33,9 @@ DAPL_RMR *dapl_rmr_alloc(DAPL_PZ * pz) DAPL_RMR *rmr; /* Allocate LMR */ - rmr = (DAPL_RMR *) kmalloc(sizeof *rmr, GFP_ATOMIC); - if (NULL == rmr) { + rmr = kmalloc(sizeof *rmr, GFP_ATOMIC); + if (!rmr) return NULL; - } /* zero the structure */ memset(rmr, 0, sizeof *rmr); diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_sp_util.c linux-kernel-kmallc/dat-provider/dapl_sp_util.c --- linux-kernel-sizeof/dat-provider/dapl_sp_util.c 2005-05-20 09:54:54.673012000 -0700 +++ linux-kernel-kmallc/dat-provider/dapl_sp_util.c 2005-05-20 12:07:08.389004000 -0700 @@ -64,8 +64,8 @@ DAPL_SP *dapl_sp_alloc(DAPL_IA * ia_ptr, DAPL_SP *sp_ptr; /* Allocate EP */ - sp_ptr = (DAPL_SP *) kmalloc(sizeof *sp_ptr, GFP_ATOMIC); - if (sp_ptr == NULL) + sp_ptr = kmalloc(sizeof *sp_ptr, GFP_ATOMIC); + if (!sp_ptr) return NULL; /* zero the structure */ diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_srq_util.c linux-kernel-kmallc/dat-provider/dapl_srq_util.c --- linux-kernel-sizeof/dat-provider/dapl_srq_util.c 2005-05-20 09:55:16.196002000 -0700 +++ linux-kernel-kmallc/dat-provider/dapl_srq_util.c 2005-05-20 12:07:15.455003000 -0700 @@ -59,11 +59,9 @@ DAPL_SRQ *dapl_srq_alloc(DAPL_IA * ia_pt DAPL_SRQ *srq_ptr; /* Allocate SRQ */ - srq_ptr = (DAPL_SRQ *) kmalloc(sizeof *srq_ptr, GFP_ATOMIC); - - if (srq_ptr == NULL) { + srq_ptr = kmalloc(sizeof *srq_ptr, GFP_ATOMIC); + if (!srq_ptr) goto bail; - } /* zero the structure */ memset(srq_ptr, 0, sizeof *srq_ptr); diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_util.h linux-kernel-kmallc/dat-provider/dapl_util.h --- linux-kernel-sizeof/dat-provider/dapl_util.h 2005-05-19 18:38:07.276001000 -0700 +++ linux-kernel-kmallc/dat-provider/dapl_util.h 2005-05-20 12:07:49.323013000 -0700 @@ -147,7 +147,7 @@ static __inline__ void *dapl_os_realloc( void *newptr; newptr = kmalloc(size, GFP_ATOMIC); - if (newptr != NULL) { + if (newptr) { /* copy the mem array to the new */ memcpy(newptr, ptr, size); kfree(ptr); @@ -162,9 +162,8 @@ static __inline__ void *dapl_os_realloc( static __inline__ char *dapl_os_strdup(const char *str) { char *ns = kmalloc(strlen(str) + 1, GFP_KERNEL); - if (ns) { + if (ns) strcpy(ns, str); - } return ns; } From jcarr at linuxmachines.com Fri May 20 13:02:48 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Fri, 20 May 2005 13:02:48 -0700 Subject: [openib-general] in need of a simple ulp In-Reply-To: <427FDA9D.7030905@ichips.intel.com> References: <426FCF0A.3070806@linuxmachines.com> <426FD098.9090300@ichips.intel.com> <427AC218.9070307@linuxmachines.com> <427AC385.5070308@ichips.intel.com> <427BA0AD.6010105@linuxmachines.com> <427FDA9D.7030905@ichips.intel.com> Message-ID: <428E4268.1040101@linuxmachines.com> Sean Hefty wrote: > Jeff Carr wrote: >> May 5 16:31:50 localhost kernel: ib_mthca 0000:09:00.0: 1a0084/0: >> error CQE -> QPN 1a0406, WQE @ 00000042 >> May 5 16:31:50 localhost kernel: [ 0] 001a0406 >> May 5 16:31:50 localhost kernel: [ 4] 00001aed >> May 5 16:31:50 localhost kernel: [ 8] 00000004 >> May 5 16:31:50 localhost kernel: [ c] 00003800 >> May 5 16:31:50 localhost kernel: [10] 128a0000 >> May 5 16:31:50 localhost kernel: [14] 00000000 >> May 5 16:31:50 localhost kernel: [18] 00000042 >> May 5 16:31:50 localhost kernel: [1c] ff000000 >> >> if you up the message_count to 0x1000. I'm guessing this is just some >> normal overrun error though. > > > It's taken me a while to look at this, but I think that this is a real > error. There must also be some limit to how many cqe's you can allocate with ib_post_recv(). (?) > Cmpost is setting the CQ size too small, which can lead to the CQ > overrun. The number of cqe's should have been message_count * 2, rather > than just message_count. Message_count is fine on the client side, > which receives all messages before sending. But on the server side, > receives could begin coming in before all sends are done. OK. Wow. That makes cqe's and ib_post_recv() even more confusing then. There must be some way to delete/free these? They don't get re-used I take it? Surely it wasn't intended that ib_post_recv() be initially run for each transfer expected in the lifetime of the connection. :) There must also be some information about what is known about these cqe's. How do we know if one of them was used for a transfer from the server to the client or from the client letting the server know the transfer was recieved? I know that this isn't a CM question; but this question is best asked against this code simplicity. (Simplicity is good) Jeff From mshefty at ichips.intel.com Fri May 20 13:40:42 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Fri, 20 May 2005 13:40:42 -0700 Subject: [openib-general] in need of a simple ulp In-Reply-To: <428E4268.1040101@linuxmachines.com> References: <426FCF0A.3070806@linuxmachines.com> <426FD098.9090300@ichips.intel.com> <427AC218.9070307@linuxmachines.com> <427AC385.5070308@ichips.intel.com> <427BA0AD.6010105@linuxmachines.com> <427FDA9D.7030905@ichips.intel.com> <428E4268.1040101@linuxmachines.com> Message-ID: <428E4B4A.2090705@ichips.intel.com> Jeff Carr wrote: > There must also be some limit to how many cqe's you can allocate with > ib_post_recv(). (?) Each receive posted will generate a cqe. That limit is controlled by the message_count variable. > There must be some way to delete/free these? They don't get re-used I > take it? Surely it wasn't intended that ib_post_recv() be initially run > for each transfer expected in the lifetime of the connection. :) There needs to be one cqe reserved for each possible completion that can be outstanding at one time. If you know that the number of completions will be high initially, and then drop to a smaller value, there is a verb that can be used to resize the CQ. (I don't believe it is implemented in the openib stack at this time.) > There must also be some information about what is known about these > cqe's. How do we know if one of them was used for a transfer from the > server to the client or from the client letting the server know the > transfer was recieved? In this case you won't know what's on a cqe until you poll the completion. An alternative way of structuring the code is to create two CQs, and tie one to the send queue, the other to the receive queue. - Sean From tduffy at sun.com Fri May 20 13:57:47 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 20 May 2005 13:57:47 -0700 Subject: [openib-general] RE: [PATCH][kdapl] fix spin_lock_irqsave/spi n_unlock_irqrestore i mplementation In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35B6D@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35B6D@mtvex01.mtv.mtl.com> Message-ID: <1116622667.27740.7.camel@duffman> On Fri, 2005-05-20 at 00:04 +0300, Itamar Rabenstein wrote: > I am not saying that we need it in OpenIb code because it was in SF code > I said that in OpenIb implementation we need it the same as in SF we need > it. > we must have this lock because for example : > if the same evd will be a CM evd for 2 ep's > each one on different Thread both can try to add an event to the evd in the > same time > there for we need to lock the evd when we take an empty event from one list > (empty list) > and to unlock it after we add the event to the second list (waiting events > list) > the lock is in one function and the unlock in the second function. > > so we need in out OpenIb code and also SF code need it (;-) Fair enough. Thanks for the clarification. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From blist at aon.at Fri May 20 14:33:17 2005 From: blist at aon.at (Bernhard Fischer) Date: Fri, 20 May 2005 23:33:17 +0200 Subject: [openib-general] in need of a simple ulp In-Reply-To: <428E4B4A.2090705@ichips.intel.com> References: <426FCF0A.3070806@linuxmachines.com> <426FD098.9090300@ichips.intel.com> <427AC218.9070307@linuxmachines.com> <427AC385.5070308@ichips.intel.com> <427BA0AD.6010105@linuxmachines.com> <427FDA9D.7030905@ichips.intel.com> <428E4268.1040101@linuxmachines.com> <428E4B4A.2090705@ichips.intel.com> Message-ID: <20050520213317.GC15688@aon.at> On Fri, May 20, 2005 at 01:40:42PM -0700, Sean Hefty wrote: >Jeff Carr wrote: >>There must also be some limit to how many cqe's you can allocate with >>ib_post_recv(). (?) > >Each receive posted will generate a cqe. That limit is controlled by the >message_count variable. > >>There must be some way to delete/free these? They don't get re-used I >>take it? Surely it wasn't intended that ib_post_recv() be initially run >>for each transfer expected in the lifetime of the connection. :) > >There needs to be one cqe reserved for each possible completion that can be >outstanding at one time. If you know that the number of completions will >be high initially, and then drop to a smaller value, there is a verb that >can be used to resize the CQ. (I don't believe it is implemented in the >openib stack at this time.) > >>There must also be some information about what is known about these >>cqe's. How do we know if one of them was used for a transfer from the >>server to the client or from the client letting the server know the >>transfer was recieved? > >In this case you won't know what's on a cqe until you poll the completion. >An alternative way of structuring the code is to create two CQs, and tie >one to the send queue, the other to the receive queue. hmz, yes, see "Quick unrelated question:" in http://openib.org/pipermail/openib-general/2005-May/006796.html I do have ...s_cq and r_cq. For two nodes, using cqe=64 does obviously work, but when i've got, say, a setup where 786 nodes each have r and s? This currently w{on't,ill in no way way} work (for me)!? Help!! Also, Sean, any thought on the qp limit question referenced there? TIA From blist at aon.at Fri May 20 14:59:57 2005 From: blist at aon.at (Bernhard Fischer) Date: Fri, 20 May 2005 23:59:57 +0200 Subject: [openib-general] [PATCH] auto*() && ./configure -!ac_default_prefix Message-ID: <20050520215957.GD15688@aon.at> Hi, [bt\nGPL throughout, on my side,.. thusly] * trunk/src/userspace/lib{mthca,ibcm}/configure.in: if prefix != ac_default_prefix use prefix for checks. fixup for e.g. SuSE when giving a non standart --prefix; Should be fixed upstream, non-debian will need a while to pick those up, though; therefore, please consider.. :-/ PS: the ../ib/..... defaults are not covered by this but w{ould,ill} need seperate, explicit treatement. Not happy with this, but crucial reality, it seems. PPS: anyone ever tested to install chain into --prefix=/scratch/infiniband after a rm -rf /usr/local/* ? Drats. -------------- next part -------------- diff -X excl -rduNp gen2.2423.oorig/trunk/src/userspace/libibcm/configure.in gen2.2423/trunk/src/userspace/libibcm/configure.in --- gen2.2423.oorig/trunk/src/userspace/libibcm/configure.in 2005-05-04 12:20:03.000000000 +0000 +++ gen2.2423/trunk/src/userspace/libibcm/configure.in 2005-05-20 10:18:12.000000000 +0000 @@ -12,6 +12,14 @@ AM_PROG_LIBTOOL dnl Checks for programs AC_PROG_CC +dnl if prefix != ac_default_prefix use prefix for checks +dnl fixup for e.g. SuSE when giving a non standard --prefix +if test "x$prefix" != "xNONE" -a "$prefix" != "$ac_default_prefix" +then + CPPFLAGS="$CPPFLAGS -I$prefix/include" + LDFLAGS="$LDFLAGS -L$prefix/lib" +fi + dnl Checks for libraries dnl Checks for header files. diff -X excl -rduNp gen2.2423.oorig/trunk/src/userspace/libmthca/configure.in gen2.2423/trunk/src/userspace/libmthca/configure.in --- gen2.2423.oorig/trunk/src/userspace/libmthca/configure.in 2005-05-04 12:20:05.000000000 +0000 +++ gen2.2423/trunk/src/userspace/libmthca/configure.in 2005-05-20 10:12:57.000000000 +0000 @@ -11,6 +11,14 @@ AM_PROG_LIBTOOL dnl Checks for programs AC_PROG_CC +dnl if prefix != ac_default_prefix use prefix for checks +dnl fixup for e.g. SuSE when giving a non standard --prefix +if test "x$prefix" != "xNONE" -a "$prefix" != "$ac_default_prefix" +then + CPPFLAGS="$CPPFLAGS -I$prefix/include" + LDFLAGS="$LDFLAGS -L$prefix/lib" +fi + dnl Checks for libraries dnl Checks for header files. From xma at us.ibm.com Fri May 20 15:00:57 2005 From: xma at us.ibm.com (Shirley Ma) Date: Fri, 20 May 2005 15:00:57 -0700 Subject: [openib-general] OOPS: ib_mad crashery on bootup In-Reply-To: <1116615241.10540.52.camel@duffman> Message-ID: I don't have anything else to say rather than the crash is in list_del(&mad_list->list), which you've shown already. It's better to have crash dump tool installed. One suggestion is stressing interface up/down might hit this problem. Thanks Shirley Ma IBM Linux Technology Center 15300 SW Koll Parkway Beaverton, OR 97006-6063 Phone(Fax): (503) 578-7638 -------------- next part -------------- An HTML attachment was scrubbed... URL: From tduffy at sun.com Fri May 20 15:10:08 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 20 May 2005 15:10:08 -0700 Subject: [openib-general] Re: [openib-commits] r2426 - gen2/users/jlentini/linux-kernel/dat-provider In-Reply-To: <20050520220508.CB9A12283D9@openib.ca.sandia.gov> References: <20050520220508.CB9A12283D9@openib.ca.sandia.gov> Message-ID: <1116627008.3627.5.camel@duffman> On Fri, 2005-05-20 at 15:05 -0700, jlentini at openib.org wrote: > Initial cleanup of DAPL "verbs" abstraction layer. > Signed-off-by: James Lentini Quick fix to allow build: Signed-off-by: Tom Duffy Index: linux-kernel-clean/dat-provider/dapl_evd_util.c =================================================================== --- linux-kernel-clean/dat-provider/dapl_evd_util.c (revision 2426) +++ linux-kernel-clean/dat-provider/dapl_evd_util.c (working copy) @@ -502,7 +502,6 @@ u32 dapl_evd_post_cr_arrival_event(DAPL_ if (event_ptr == NULL) return DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); - } event_ptr->event_data.cr_arrival_event_data.sp_handle = sp_handle; event_ptr->event_data.cr_arrival_event_data.local_ia_address_ptr = From jlentini at netapp.com Fri May 20 15:16:15 2005 From: jlentini at netapp.com (James Lentini) Date: Fri, 20 May 2005 18:16:15 -0400 (EDT) Subject: [openib-general] Re: [openib-commits] r2426 - gen2/users/jlentini/linux-kernel/dat-provider In-Reply-To: <1116627008.3627.5.camel@duffman> References: <20050520220508.CB9A12283D9@openib.ca.sandia.gov> <1116627008.3627.5.camel@duffman> Message-ID: Sorry. Committed in revision 2427. On Fri, 20 May 2005, Tom Duffy wrote: tduffy> On Fri, 2005-05-20 at 15:05 -0700, jlentini at openib.org wrote: tduffy> > Initial cleanup of DAPL "verbs" abstraction layer. tduffy> > Signed-off-by: James Lentini tduffy> tduffy> Quick fix to allow build: tduffy> tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> Index: linux-kernel-clean/dat-provider/dapl_evd_util.c tduffy> =================================================================== tduffy> --- linux-kernel-clean/dat-provider/dapl_evd_util.c (revision 2426) tduffy> +++ linux-kernel-clean/dat-provider/dapl_evd_util.c (working copy) tduffy> @@ -502,7 +502,6 @@ u32 dapl_evd_post_cr_arrival_event(DAPL_ tduffy> if (event_ptr == NULL) tduffy> return DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, tduffy> DAT_RESOURCE_MEMORY); tduffy> - } tduffy> tduffy> event_ptr->event_data.cr_arrival_event_data.sp_handle = sp_handle; tduffy> event_ptr->event_data.cr_arrival_event_data.local_ia_address_ptr = tduffy> From tduffy at sun.com Fri May 20 15:19:51 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 20 May 2005 15:19:51 -0700 Subject: [openib-general] [PATCHv2] kDAPL: sizeof is not a function Message-ID: <1116627591.3627.9.camel@duffman> This is a repost rebased off of r2427. Signed-off-by: Tom Duffy Index: linux-kernel-sizeof/dat-provider/dapl_cookie.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_cookie.c (revision 2427) +++ linux-kernel-sizeof/dat-provider/dapl_cookie.c (working copy) @@ -123,7 +123,7 @@ u32 dapl_cb_create(DAPL_COOKIE_BUFFER *b */ size++; - buffer->pool = kmalloc(size * sizeof(DAPL_COOKIE), GFP_ATOMIC); + buffer->pool = kmalloc(size * sizeof *buffer->pool, GFP_ATOMIC); if (NULL != buffer->pool) { buffer->pool_size = size; atomic_set(&buffer->head, 0); Index: linux-kernel-sizeof/dat-provider/dapl_ep_connect.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_ep_connect.c (revision 2427) +++ linux-kernel-sizeof/dat-provider/dapl_ep_connect.c (working copy) @@ -281,8 +281,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han DAT_EP_STATE_ACTIVE_CONNECTION_PENDING && timeout != DAT_TIMEOUT_INFINITE) { ep_ptr->cxn_timer = (struct dapl_timer_entry *) - kmalloc(sizeof(struct dapl_timer_entry), - GFP_ATOMIC); + kmalloc(sizeof *ep_ptr->cxn_timer, GFP_ATOMIC); dapl_timer_set(ep_ptr->cxn_timer, dapl_ep_timeout, ep_ptr, timeout); Index: linux-kernel-sizeof/dat-provider/dapl_ring_buffer_util.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_ring_buffer_util.c (revision 2427) +++ linux-kernel-sizeof/dat-provider/dapl_ring_buffer_util.c (working copy) @@ -72,7 +72,7 @@ u32 dapl_rbuf_alloc(DAPL_RING_BUFFER *rb rsize <<= 1; } - rbuf->base = (void *) kmalloc(rsize * sizeof(void *), GFP_ATOMIC); + rbuf->base = (void *) kmalloc(rsize * sizeof *rbuf->base, GFP_ATOMIC); if (rbuf->base != NULL) { rbuf->lim = rsize - 1; atomic_set(&rbuf->head, 0); Index: linux-kernel-sizeof/dat-provider/dapl_lmr_util.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_lmr_util.c (revision 2427) +++ linux-kernel-sizeof/dat-provider/dapl_lmr_util.c (working copy) @@ -48,13 +48,13 @@ DAPL_LMR *dapl_lmr_alloc(DAPL_IA * ia, DAPL_LMR *lmr; /* Allocate LMR */ - lmr = (DAPL_LMR *) kmalloc(sizeof(DAPL_LMR), GFP_ATOMIC); + lmr = (DAPL_LMR *) kmalloc(sizeof *lmr, GFP_ATOMIC); if (NULL == lmr) { return NULL; } /* zero the structure */ - memset(lmr, 0, sizeof(DAPL_LMR)); + memset(lmr, 0, sizeof *lmr); /* * initialize the header Index: linux-kernel-sizeof/dat-provider/dapl_hca_util.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_hca_util.c (revision 2427) +++ linux-kernel-sizeof/dat-provider/dapl_hca_util.c (working copy) @@ -60,9 +60,9 @@ DAPL_HCA *dapl_hca_alloc(char *name, str { DAPL_HCA *hca_ptr; - hca_ptr = kmalloc(sizeof(DAPL_HCA), GFP_ATOMIC); + hca_ptr = kmalloc(sizeof *hca_ptr, GFP_ATOMIC); if (NULL != hca_ptr) { - memset(hca_ptr, 0, sizeof(DAPL_HCA)); + memset(hca_ptr, 0, sizeof *hca_ptr); if (DAT_SUCCESS == dapl_hash_create(DAPL_HASH_TABLE_DEFAULT_CAPACITY, Index: linux-kernel-sizeof/dat-provider/dapl_cr_util.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_cr_util.c (revision 2427) +++ linux-kernel-sizeof/dat-provider/dapl_cr_util.c (working copy) @@ -42,13 +42,13 @@ DAPL_CR *dapl_cr_alloc(DAPL_IA * ia_ptr) DAPL_CR *cr_ptr; /* Allocate EP */ - cr_ptr = (DAPL_CR *) kmalloc(sizeof(DAPL_CR), GFP_ATOMIC); + cr_ptr = (DAPL_CR *) kmalloc(sizeof *cr_ptr, GFP_ATOMIC); if (cr_ptr == NULL) { return NULL; } /* zero the structure */ - memset(cr_ptr, 0, sizeof(DAPL_CR)); + memset(cr_ptr, 0, sizeof *cr_ptr); /* * initialize the header Index: linux-kernel-sizeof/dat-provider/dapl_srq_util.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_srq_util.c (revision 2427) +++ linux-kernel-sizeof/dat-provider/dapl_srq_util.c (working copy) @@ -59,14 +59,14 @@ DAPL_SRQ *dapl_srq_alloc(DAPL_IA * ia_pt DAPL_SRQ *srq_ptr; /* Allocate SRQ */ - srq_ptr = (DAPL_SRQ *) kmalloc(sizeof(DAPL_SRQ), GFP_ATOMIC); + srq_ptr = (DAPL_SRQ *) kmalloc(sizeof *srq_ptr, GFP_ATOMIC); if (srq_ptr == NULL) { goto bail; } /* zero the structure */ - memset(srq_ptr, 0, sizeof(DAPL_SRQ)); + memset(srq_ptr, 0, sizeof *srq_ptr); /* * initialize the header Index: linux-kernel-sizeof/dat-provider/dapl_pz_util.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_pz_util.c (revision 2427) +++ linux-kernel-sizeof/dat-provider/dapl_pz_util.c (working copy) @@ -57,12 +57,12 @@ DAPL_PZ *dapl_pz_alloc(DAPL_IA * ia) DAPL_PZ *pz; /* Allocate PZ */ - pz = (DAPL_PZ *) kmalloc(sizeof(DAPL_PZ), GFP_ATOMIC); + pz = (DAPL_PZ *) kmalloc(sizeof *pz, GFP_ATOMIC); if (NULL == pz) return NULL; /* zero the structure */ - memset(pz, 0, sizeof(DAPL_PZ)); + memset(pz, 0, sizeof *pz); /* * initialize the header Index: linux-kernel-sizeof/dat-provider/dapl_ia_util.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_ia_util.c (revision 2427) +++ linux-kernel-sizeof/dat-provider/dapl_ia_util.c (working copy) @@ -63,13 +63,13 @@ DAPL_IA *dapl_ia_alloc(struct dat_provid DAPL_IA *ia_ptr; /* Allocate IA */ - ia_ptr = (DAPL_IA *) kmalloc(sizeof(DAPL_IA), GFP_ATOMIC); + ia_ptr = (DAPL_IA *) kmalloc(sizeof *ia_ptr, GFP_ATOMIC); if (ia_ptr == NULL) { return NULL; } /* zero the structure */ - memset(ia_ptr, 0, sizeof(DAPL_IA)); + memset(ia_ptr, 0, sizeof *ia_ptr); /* * initialize the header Index: linux-kernel-sizeof/dat-provider/dapl_provider.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_provider.c (revision 2427) +++ linux-kernel-sizeof/dat-provider/dapl_provider.c (working copy) @@ -150,27 +150,25 @@ u32 dapl_provider_list_create(void) /* create the head node */ g_dapl_provider_list.head = - kmalloc(sizeof(DAPL_PROVIDER_LIST_NODE), GFP_ATOMIC); + kmalloc(sizeof *g_dapl_provider_list.head, GFP_ATOMIC); if (NULL == g_dapl_provider_list.head) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto bail; } - memset(g_dapl_provider_list.head, 0, - sizeof(DAPL_PROVIDER_LIST_NODE)); + memset(g_dapl_provider_list.head, 0, sizeof *g_dapl_provider_list.head); /* create the tail node */ g_dapl_provider_list.tail = - kmalloc(sizeof(DAPL_PROVIDER_LIST_NODE), GFP_ATOMIC); + kmalloc(sizeof *g_dapl_provider_list.tail, GFP_ATOMIC); if (NULL == g_dapl_provider_list.tail) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto bail; } - memset(g_dapl_provider_list.tail, 0, - sizeof(DAPL_PROVIDER_LIST_NODE)); + memset(g_dapl_provider_list.tail, 0, sizeof *g_dapl_provider_list.tail); g_dapl_provider_list.head->next = g_dapl_provider_list.tail; g_dapl_provider_list.tail->prev = g_dapl_provider_list.head; @@ -215,7 +213,7 @@ u32 dapl_provider_list_insert(const char u32 status = DAT_SUCCESS; unsigned int len; - cur_node = kmalloc(sizeof(DAPL_PROVIDER_LIST_NODE), GFP_ATOMIC); + cur_node = kmalloc(sizeof *cur_node, GFP_ATOMIC); if (NULL == cur_node) { status = Index: linux-kernel-sizeof/dat-provider/dapl_openib_cm.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_openib_cm.c (revision 2427) +++ linux-kernel-sizeof/dat-provider/dapl_openib_cm.c (working copy) @@ -431,7 +431,7 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han ia_ptr = conn->ep->header.owner_ia; /* Setup QP/CM parameters */ - memset(&conn->param, 0, sizeof(conn->param)); + memset(&conn->param, 0, sizeof conn->param); conn->param.qp_type = IB_QPT_RC; conn->param.qp_num = conn->ep->qp_handle->qp_num; conn->param.private_data = private_data; @@ -445,7 +445,7 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han conn->param.local_cm_response_timeout = DAPL_OPENIB_CM_RESPONSE_TIMEOUT; conn->param.max_cm_retries = DAPL_OPENIB_MAX_CM_RETRIES; - memcpy(&conn->service_id, &remote_conn_qual, sizeof(u64)); + memcpy(&conn->service_id, &remote_conn_qual, sizeof conn->service_id); conn->remote_ia_address = remote_ia_address; conn->dapl_comp.fn = &dapl_rt_comp_handler; @@ -770,7 +770,7 @@ u32 dapl_ib_accept_connection(DAT_CR_HAN ep_ptr->cm_handle = conn; conn->ep = ep_ptr; - memset(&passive_params, 0, sizeof(passive_params)); + memset(&passive_params, 0, sizeof passive_params); passive_params.private_data = prd_ptr; passive_params.private_data_len = private_data_size; passive_params.qp_num = ep_ptr->qp_handle->qp_num; @@ -847,7 +847,7 @@ u32 dapl_ib_cm_remote_addr(DAT_HANDLE da if (!memcmp(&ia_ptr->hca_ptr->ib_trans.gid, &remote_gid, 16)) { memcpy(remote_ia_address, &ia_ptr->hca_ptr->hca_address, - sizeof(struct sockaddr)); + sizeof *remote_ia_address); return DAT_SUCCESS; } Index: linux-kernel-sizeof/dat-provider/dapl_openib_util.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_openib_util.c (revision 2427) +++ linux-kernel-sizeof/dat-provider/dapl_openib_util.c (working copy) @@ -163,7 +163,7 @@ u32 dapl_ib_open_hca(char *name, DAPL_HC { u32 dat_ret; - memset(&hca_ptr->ib_trans.gid.global, 0, sizeof(union ib_gid)); + memset(&hca_ptr->ib_trans.gid.global, 0, sizeof hca_ptr->ib_trans.gid); hca_ptr->ib_trans.lid = 0; dat_ret = dapl_ib_get_hca_ids(hca_ptr->ib_hca_handle, @@ -409,7 +409,7 @@ u32 dapl_ib_mr_register_physical(DAPL_IA array = (u64 *) phys_addr; buf_list = (struct ib_phys_buf *) - kmalloc(length * sizeof(struct ib_phys_buf), GFP_ATOMIC); + kmalloc(length * sizeof *buf_list, GFP_ATOMIC); if (NULL == buf_list) return DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); Index: linux-kernel-sizeof/dat-provider/dapl_ep_modify.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_ep_modify.c (revision 2427) +++ linux-kernel-sizeof/dat-provider/dapl_ep_modify.c (working copy) @@ -385,8 +385,7 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand alloc_ep.send_iov_num = ep_attr1.max_request_iov; alloc_ep.send_iov = kmalloc(ep_attr1.max_request_iov - * sizeof(struct ib_sge), - GFP_ATOMIC); + * sizeof *alloc_ep.send_iov, GFP_ATOMIC); if (alloc_ep.recv_iov == NULL) { dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); @@ -396,8 +395,7 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand alloc_ep.recv_iov_num = ep_attr1.max_recv_iov; alloc_ep.recv_iov = kmalloc(ep_attr1.max_recv_iov - * sizeof(struct ib_sge), - GFP_ATOMIC); + * sizeof *alloc_ep.recv_iov, GFP_ATOMIC); if (alloc_ep.recv_iov == NULL) { dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); Index: linux-kernel-sizeof/dat-provider/dapl_openib_qp.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_openib_qp.c (revision 2427) +++ linux-kernel-sizeof/dat-provider/dapl_openib_qp.c (working copy) @@ -305,7 +305,7 @@ int dapl_modify_qp_state_to_init(struct int qp_attr_mask; int ib_status; - memset(&qp_attr, 0, sizeof(qp_attr)); + memset(&qp_attr, 0, sizeof qp_attr); qp_attr.qp_state = IB_QPS_INIT; qp_attr_mask = IB_QP_STATE; qp_attr.qp_access_flags = @@ -328,7 +328,7 @@ int dapl_modify_qp_state_to_error(struct int qp_attr_mask; int ib_status; - memset(&qp_attr, 0, sizeof(qp_attr)); + memset(&qp_attr, 0, sizeof qp_attr); qp_attr.qp_state = IB_QPS_ERR; qp_attr_mask = IB_QP_STATE; ib_status = ib_modify_qp(qp, &qp_attr, qp_attr_mask); @@ -342,7 +342,7 @@ int dapl_modify_qp_state_to_reset(struct int qp_attr_mask; int ib_status; - memset(&qp_attr, 0, sizeof(qp_attr)); + memset(&qp_attr, 0, sizeof qp_attr); qp_attr.qp_state = IB_QPS_RESET; qp_attr_mask = IB_QP_STATE; ib_status = ib_modify_qp(qp, &qp_attr, qp_attr_mask); @@ -356,7 +356,7 @@ int dapl_modify_qp_state_to_rtr(struct i int qp_attr_mask; int ib_status; - memset(&qp_attr, 0, sizeof(qp_attr)); + memset(&qp_attr, 0, sizeof qp_attr); qp_attr.qp_state = IB_QPS_RTR; ib_status = ib_cm_init_qp_attr(cm_id, &qp_attr, &qp_attr_mask); @@ -376,7 +376,7 @@ int dapl_modify_qp_state_to_rts(struct i int qp_attr_mask; int ib_status; - memset(&qp_attr, 0, sizeof(qp_attr)); + memset(&qp_attr, 0, sizeof qp_attr); qp_attr.qp_state = IB_QPS_RTS; ib_status = ib_cm_init_qp_attr(cm_id, &qp_attr, &qp_attr_mask); Index: linux-kernel-sizeof/dat-provider/dapl_cno_util.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_cno_util.c (revision 2427) +++ linux-kernel-sizeof/dat-provider/dapl_cno_util.c (working copy) @@ -42,12 +42,12 @@ DAPL_CNO *dapl_cno_alloc(DAPL_IA *ia_ptr { DAPL_CNO *cno_ptr; - cno_ptr = kmalloc(sizeof(*cno_ptr), GFP_ATOMIC); + cno_ptr = kmalloc(sizeof *cno_ptr, GFP_ATOMIC); if (!cno_ptr) return NULL; /* zero the structure */ - memset(cno_ptr, 0, sizeof(*cno_ptr)); + memset(cno_ptr, 0, sizeof *cno_ptr); /* * Initialize the header. Index: linux-kernel-sizeof/dat-provider/dapl_rmr_util.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_rmr_util.c (revision 2427) +++ linux-kernel-sizeof/dat-provider/dapl_rmr_util.c (working copy) @@ -33,13 +33,13 @@ DAPL_RMR *dapl_rmr_alloc(DAPL_PZ * pz) DAPL_RMR *rmr; /* Allocate LMR */ - rmr = (DAPL_RMR *) kmalloc(sizeof(DAPL_RMR), GFP_ATOMIC); + rmr = (DAPL_RMR *) kmalloc(sizeof *rmr, GFP_ATOMIC); if (NULL == rmr) { return NULL; } /* zero the structure */ - memset(rmr, 0, sizeof(DAPL_RMR)); + memset(rmr, 0, sizeof *rmr); /* * initialize the header Index: linux-kernel-sizeof/dat-provider/dapl_ep_util.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_ep_util.c (revision 2427) +++ linux-kernel-sizeof/dat-provider/dapl_ep_util.c (working copy) @@ -60,13 +60,13 @@ DAPL_EP *dapl_ep_alloc(DAPL_IA * ia_ptr, DAPL_EP *ep_ptr; /* Allocate EP */ - ep_ptr = (DAPL_EP *) kmalloc(sizeof(DAPL_EP) + sizeof(struct sockaddr), + ep_ptr = (DAPL_EP *) kmalloc(sizeof *ep_ptr + sizeof (struct sockaddr), GFP_ATOMIC); if (ep_ptr == NULL) goto bail; /* zero the structure */ - memset(ep_ptr, 0, sizeof(DAPL_EP) + sizeof(struct sockaddr)); + memset(ep_ptr, 0, sizeof *ep_ptr + sizeof (struct sockaddr)); /* * initialize the header @@ -125,7 +125,7 @@ DAPL_EP *dapl_ep_alloc(DAPL_IA * ia_ptr, ep_ptr->send_iov_num = ep_ptr->param.ep_attr.max_request_iov; ep_ptr->recv_iov = kmalloc(ep_ptr->recv_iov_num * - sizeof(struct ib_sge), GFP_ATOMIC); + sizeof *ep_ptr->recv_iov, GFP_ATOMIC); if (NULL == ep_ptr->recv_iov) { dapl_ep_dealloc(ep_ptr); @@ -134,7 +134,7 @@ DAPL_EP *dapl_ep_alloc(DAPL_IA * ia_ptr, } ep_ptr->send_iov = kmalloc(ep_ptr->send_iov_num * - sizeof(struct ib_sge), GFP_ATOMIC); + sizeof *ep_ptr->send_iov, GFP_ATOMIC); if (NULL == ep_ptr->send_iov) { dapl_ep_dealloc(ep_ptr); Index: linux-kernel-sizeof/dat-provider/dapl_evd_util.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_evd_util.c (revision 2427) +++ linux-kernel-sizeof/dat-provider/dapl_evd_util.c (working copy) @@ -123,13 +123,13 @@ DAPL_EVD *dapl_evd_alloc(DAPL_IA * ia_pt DAPL_EVD *evd_ptr; /* Allocate EVD */ - evd_ptr = (DAPL_EVD *) kmalloc(sizeof(DAPL_EVD), GFP_ATOMIC); + evd_ptr = (DAPL_EVD *) kmalloc(sizeof *evd_ptr, GFP_ATOMIC); if (!evd_ptr) { goto bail; } /* zero the structure */ - memset(evd_ptr, 0, sizeof(DAPL_EVD)); + memset(evd_ptr, 0, sizeof *evd_ptr); /* * initialize the header Index: linux-kernel-sizeof/dat-provider/dapl_util.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_util.c (revision 2427) +++ linux-kernel-sizeof/dat-provider/dapl_util.c (working copy) @@ -167,7 +167,7 @@ void dapl_dbg_log(DAPL_DBG_TYPE type, co if (type & g_dapl_dbg_mask) { va_start(args, fmt); - vsnprintf(buf, sizeof(buf), fmt, args); + vsnprintf(buf, sizeof buf, fmt, args); printk(KERN_ALERT "kDAPL: %s", buf); va_end(args); } Index: linux-kernel-sizeof/dat-provider/dapl_sp_util.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_sp_util.c (revision 2427) +++ linux-kernel-sizeof/dat-provider/dapl_sp_util.c (working copy) @@ -64,12 +64,12 @@ DAPL_SP *dapl_sp_alloc(DAPL_IA * ia_ptr, DAPL_SP *sp_ptr; /* Allocate EP */ - sp_ptr = (DAPL_SP *) kmalloc(sizeof(DAPL_SP), GFP_ATOMIC); + sp_ptr = (DAPL_SP *) kmalloc(sizeof *sp_ptr, GFP_ATOMIC); if (sp_ptr == NULL) return NULL; /* zero the structure */ - memset(sp_ptr, 0, sizeof(DAPL_SP)); + memset(sp_ptr, 0, sizeof *sp_ptr); /* * initialize the header Index: linux-kernel-sizeof/dat-provider/dapl_hash.c =================================================================== --- linux-kernel-sizeof/dat-provider/dapl_hash.c (revision 2427) +++ linux-kernel-sizeof/dat-provider/dapl_hash.c (working copy) @@ -214,7 +214,7 @@ dapl_hash_add(DAPL_HASH_TABLEP p_table, status = TRUE; } else { DAPL_HASH_ELEM *newelement = (DAPL_HASH_ELEM *) - kmalloc(sizeof(DAPL_HASH_ELEM), GFP_ATOMIC); + kmalloc(sizeof *newelement, GFP_ATOMIC); /* * Add an element to the end of the chain */ @@ -324,14 +324,14 @@ dapl_hash_delete_element(DAPL_HASH_ELEM u32 dapl_hash_create(int table_size, DAPL_HASH_TABLE **pp_table) { DAPL_HASH_TABLE *p_table; - int table_length = table_size * sizeof(DAPL_HASH_ELEM); + int table_length = table_size * sizeof (struct DAPL_HASH_ELEM); u32 dat_status = DAT_SUCCESS; int i; dapl_os_assert(pp_table); /* Allocate hash table */ - p_table = kmalloc(sizeof(DAPL_HASH_TABLE), GFP_ATOMIC); + p_table = kmalloc(sizeof *p_table, GFP_ATOMIC); if (NULL == p_table) { dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); @@ -339,7 +339,7 @@ u32 dapl_hash_create(int table_size, DAP } /* Init hash table, allocate and init and buckets */ - memset(p_table, 0, sizeof(DAPL_HASH_TABLE)); + memset(p_table, 0, sizeof *p_table); p_table->tbl_size = table_size; p_table->table = (DAPL_HASH_ELEM *) kmalloc(table_length, GFP_ATOMIC); if (NULL == p_table->table) { Index: linux-kernel-sizeof/patches/at.c =================================================================== --- linux-kernel-sizeof/patches/at.c (revision 2427) +++ linux-kernel-sizeof/patches/at.c (working copy) @@ -253,7 +253,7 @@ static int resolve_ip(struct ib_at_src * src->dev = priv->ca; src->port = priv->port; src->pkey = cpu_to_be16(priv->pkey); - memcpy(&src->gid, ipoib_dev->dev_addr + 4, sizeof(src->gid)); + memcpy(&src->gid, ipoib_dev->dev_addr + 4, sizeof src->gid); if (!dgid) return 0; @@ -263,17 +263,13 @@ static int resolve_ip(struct ib_at_src * * the IB device which was found. */ if (rt->u.dst.neighbour->dev->flags & IFF_LOOPBACK) { - memcpy(dgid, ipoib_dev->dev_addr + 4, - sizeof(union ib_gid)); - + memcpy(dgid, ipoib_dev->dev_addr + 4, sizeof *dgid); return 1; } if ((NUD_CONNECTED|NUD_DELAY|NUD_PROBE) & rt->u.dst.neighbour->nud_state) { - memcpy(dgid, rt->u.dst.neighbour->ha + 4, - sizeof(union ib_gid)); - + memcpy(dgid, rt->u.dst.neighbour->ha + 4, sizeof *dgid); return 1; } @@ -300,7 +296,7 @@ static u64 alloc_req_id(void) static void req_init(struct async *pend, void *data, int nelem, int type, struct ib_at_completion *async_comp) { - memset(pend, 0, sizeof(*pend)); + memset(pend, 0, sizeof *pend); pend->id = async_comp->req_id = alloc_req_id(); pend->status = IB_AT_STATUS_INVALID; @@ -454,8 +450,8 @@ static int same_path_req(struct async *a struct path_req *pa = container_of(a, struct path_req, pend); struct path_req *pb = container_of(b, struct path_req, pend); - return !memcmp(&pa->rt.sgid, &pb->rt.sgid, sizeof(pa->rt.sgid)) && - !memcmp(&pa->rt.dgid, &pb->rt.dgid, sizeof(pa->rt.dgid)) && + return !memcmp(&pa->rt.sgid, &pb->rt.sgid, sizeof pa->rt.sgid) && + !memcmp(&pa->rt.dgid, &pb->rt.dgid, sizeof pa->rt.dgid) && pa->rt.out_dev == pb->rt.out_dev && pa->rt.out_port == pb->rt.out_port && pa->rt.attr.pkey == pb->rt.attr.pkey && @@ -784,7 +780,7 @@ static int ib_at_arp_recv(struct sk_buff * queue IB arp packet onto work queue. */ DEBUG("recv IB ARP - queue work"); - work = kmalloc(sizeof(*work), GFP_ATOMIC); + work = kmalloc(sizeof *work, GFP_ATOMIC); if (!work) goto done; @@ -1000,7 +996,7 @@ static int ib_at_init(void) * Init memory pools */ route_req_cache = kmem_cache_create("ib_at_routes", - sizeof(struct route_req), + sizeof (struct route_req), 0, SLAB_HWCACHE_ALIGN, NULL, NULL); if (!route_req_cache) { @@ -1010,7 +1006,7 @@ static int ib_at_init(void) } path_req_cache = kmem_cache_create("ib_at_paths", - sizeof(struct path_req), + sizeof (struct path_req), 0, SLAB_HWCACHE_ALIGN, NULL, NULL); if (!path_req_cache) { Index: linux-kernel-sizeof/dat/dictionary.c =================================================================== --- linux-kernel-sizeof/dat/dictionary.c (revision 2427) +++ linux-kernel-sizeof/dat/dictionary.c (working copy) @@ -96,7 +96,7 @@ u32 dat_dictionary_entry_create(struct d BUG_ON(NULL == p_entry); - entry = kmalloc(sizeof(*entry), GFP_ATOMIC); + entry = kmalloc(sizeof *entry, GFP_ATOMIC); if (NULL == entry) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); Index: linux-kernel-sizeof/dat/dr.c =================================================================== --- linux-kernel-sizeof/dat/dr.c (revision 2427) +++ linux-kernel-sizeof/dat/dr.c (working copy) @@ -84,7 +84,7 @@ u32 dat_dr_insert(const struct dat_provi unsigned long flags; u32 status; - data = kmalloc(sizeof(*data), GFP_ATOMIC); + data = kmalloc(sizeof *data, GFP_ATOMIC); if (NULL == data) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); @@ -241,7 +241,7 @@ u32 dat_dr_list(int max_to_return, int * goto bail; } - array = kmalloc(array_size * sizeof(*array), GFP_ATOMIC); + array = kmalloc(array_size * sizeof *array, GFP_ATOMIC); if (NULL == array) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, Index: linux-kernel-sizeof/dat/core.c =================================================================== --- linux-kernel-sizeof/dat/core.c (revision 2427) +++ linux-kernel-sizeof/dat/core.c (working copy) @@ -66,7 +66,7 @@ void dat_dbg_print(DAT_DBG_TYPE type, co va_list args; va_start(args, fmt); - vsnprintf(buf, sizeof(buf), fmt, args); + vsnprintf(buf, sizeof buf, fmt, args); printk(KERN_ALERT "DAT: %s", buf); va_end(args); } From tduffy at sun.com Fri May 20 15:28:32 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 20 May 2005 15:28:32 -0700 Subject: [openib-general] [PATCHv2] kDAPL: kmalloc needs no cast In-Reply-To: <1116627591.3627.9.camel@duffman> References: <1116627591.3627.9.camel@duffman> Message-ID: <1116628112.3627.12.camel@duffman> This is a repost rebased off of r2427. Please apply after my v2 of the sizeof patch. Signed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat/dictionary.c linux-kernel-kmalloc/dat/dictionary.c --- linux-kernel-sizeof/dat/dictionary.c 2005-05-20 09:55:56.230003000 -0700 +++ linux-kernel-kmalloc/dat/dictionary.c 2005-05-20 15:19:47.778968000 -0700 @@ -97,7 +97,7 @@ u32 dat_dictionary_entry_create(struct d BUG_ON(NULL == p_entry); entry = kmalloc(sizeof *entry, GFP_ATOMIC); - if (NULL == entry) { + if (!entry) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto bail; diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat/dr.c linux-kernel-kmalloc/dat/dr.c --- linux-kernel-sizeof/dat/dr.c 2005-05-20 09:56:11.726007000 -0700 +++ linux-kernel-kmalloc/dat/dr.c 2005-05-20 15:19:47.787972000 -0700 @@ -85,7 +85,7 @@ u32 dat_dr_insert(const struct dat_provi u32 status; data = kmalloc(sizeof *data, GFP_ATOMIC); - if (NULL == data) { + if (!data) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto bail; @@ -242,7 +242,7 @@ u32 dat_dr_list(int max_to_return, int * } array = kmalloc(array_size * sizeof *array, GFP_ATOMIC); - if (NULL == array) { + if (!array) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_cookie.c linux-kernel-kmalloc/dat-provider/dapl_cookie.c --- linux-kernel-sizeof/dat-provider/dapl_cookie.c 2005-05-20 10:47:10.741014000 -0700 +++ linux-kernel-kmalloc/dat-provider/dapl_cookie.c 2005-05-20 15:19:47.793960000 -0700 @@ -124,7 +124,7 @@ u32 dapl_cb_create(DAPL_COOKIE_BUFFER *b size++; buffer->pool = kmalloc(size * sizeof *buffer->pool, GFP_ATOMIC); - if (NULL != buffer->pool) { + if (buffer->pool) { buffer->pool_size = size; atomic_set(&buffer->head, 0); atomic_set(&buffer->tail, 0); @@ -135,9 +135,8 @@ u32 dapl_cb_create(DAPL_COOKIE_BUFFER *b } return DAT_SUCCESS; - } else { + } else return DAT_INSUFFICIENT_RESOURCES; - } } /* diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_cr_util.c linux-kernel-kmalloc/dat-provider/dapl_cr_util.c --- linux-kernel-sizeof/dat-provider/dapl_cr_util.c 2005-05-20 09:39:00.781014000 -0700 +++ linux-kernel-kmalloc/dat-provider/dapl_cr_util.c 2005-05-20 15:19:47.797971000 -0700 @@ -42,10 +42,9 @@ DAPL_CR *dapl_cr_alloc(DAPL_IA * ia_ptr) DAPL_CR *cr_ptr; /* Allocate EP */ - cr_ptr = (DAPL_CR *) kmalloc(sizeof *cr_ptr, GFP_ATOMIC); - if (cr_ptr == NULL) { + cr_ptr = kmalloc(sizeof *cr_ptr, GFP_ATOMIC); + if (!cr_ptr) return NULL; - } /* zero the structure */ memset(cr_ptr, 0, sizeof *cr_ptr); diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_ep_connect.c linux-kernel-kmalloc/dat-provider/dapl_ep_connect.c --- linux-kernel-sizeof/dat-provider/dapl_ep_connect.c 2005-05-20 15:12:36.692972000 -0700 +++ linux-kernel-kmalloc/dat-provider/dapl_ep_connect.c 2005-05-20 15:22:16.513960000 -0700 @@ -280,8 +280,13 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han if (ep_ptr->param.ep_state == DAT_EP_STATE_ACTIVE_CONNECTION_PENDING && timeout != DAT_TIMEOUT_INFINITE) { - ep_ptr->cxn_timer = (struct dapl_timer_entry *) - kmalloc(sizeof *ep_ptr->cxn_timer, GFP_ATOMIC); + ep_ptr->cxn_timer = kmalloc(sizeof *ep_ptr->cxn_timer, + GFP_ATOMIC); + if (!ep_ptr->cxn_timer) { + dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, + DAT_RESOURCE_MEMORY); + goto bail; + } dapl_timer_set(ep_ptr->cxn_timer, dapl_ep_timeout, ep_ptr, timeout); diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_ep_modify.c linux-kernel-kmalloc/dat-provider/dapl_ep_modify.c --- linux-kernel-sizeof/dat-provider/dapl_ep_modify.c 2005-05-20 15:13:44.645961000 -0700 +++ linux-kernel-kmalloc/dat-provider/dapl_ep_modify.c 2005-05-20 15:19:47.835961000 -0700 @@ -386,7 +386,7 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand alloc_ep.send_iov_num = ep_attr1.max_request_iov; alloc_ep.send_iov = kmalloc(ep_attr1.max_request_iov * sizeof *alloc_ep.send_iov, GFP_ATOMIC); - if (alloc_ep.recv_iov == NULL) { + if (!alloc_ep.recv_iov) { dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto bail; @@ -396,7 +396,7 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand alloc_ep.recv_iov_num = ep_attr1.max_recv_iov; alloc_ep.recv_iov = kmalloc(ep_attr1.max_recv_iov * sizeof *alloc_ep.recv_iov, GFP_ATOMIC); - if (alloc_ep.recv_iov == NULL) { + if (!alloc_ep.recv_iov) { dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto bail; diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_ep_util.c linux-kernel-kmalloc/dat-provider/dapl_ep_util.c --- linux-kernel-sizeof/dat-provider/dapl_ep_util.c 2005-05-20 15:14:06.429964000 -0700 +++ linux-kernel-kmalloc/dat-provider/dapl_ep_util.c 2005-05-20 15:19:47.840972000 -0700 @@ -60,9 +60,8 @@ DAPL_EP *dapl_ep_alloc(DAPL_IA * ia_ptr, DAPL_EP *ep_ptr; /* Allocate EP */ - ep_ptr = (DAPL_EP *) kmalloc(sizeof *ep_ptr + sizeof (struct sockaddr), - GFP_ATOMIC); - if (ep_ptr == NULL) + ep_ptr = kmalloc(sizeof *ep_ptr + sizeof (struct sockaddr), GFP_ATOMIC); + if (!ep_ptr) goto bail; /* zero the structure */ @@ -126,8 +125,7 @@ DAPL_EP *dapl_ep_alloc(DAPL_IA * ia_ptr, ep_ptr->recv_iov = kmalloc(ep_ptr->recv_iov_num * sizeof *ep_ptr->recv_iov, GFP_ATOMIC); - - if (NULL == ep_ptr->recv_iov) { + if (!ep_ptr->recv_iov) { dapl_ep_dealloc(ep_ptr); ep_ptr = NULL; goto bail; @@ -135,8 +133,7 @@ DAPL_EP *dapl_ep_alloc(DAPL_IA * ia_ptr, ep_ptr->send_iov = kmalloc(ep_ptr->send_iov_num * sizeof *ep_ptr->send_iov, GFP_ATOMIC); - - if (NULL == ep_ptr->send_iov) { + if (!ep_ptr->send_iov) { dapl_ep_dealloc(ep_ptr); ep_ptr = NULL; goto bail; diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_evd_util.c linux-kernel-kmalloc/dat-provider/dapl_evd_util.c --- linux-kernel-sizeof/dat-provider/dapl_evd_util.c 2005-05-20 15:15:32.618966000 -0700 +++ linux-kernel-kmalloc/dat-provider/dapl_evd_util.c 2005-05-20 15:19:47.867963000 -0700 @@ -123,10 +123,9 @@ DAPL_EVD *dapl_evd_alloc(DAPL_IA * ia_pt DAPL_EVD *evd_ptr; /* Allocate EVD */ - evd_ptr = (DAPL_EVD *) kmalloc(sizeof *evd_ptr, GFP_ATOMIC); - if (!evd_ptr) { + evd_ptr = kmalloc(sizeof *evd_ptr, GFP_ATOMIC); + if (!evd_ptr) goto bail; - } /* zero the structure */ memset(evd_ptr, 0, sizeof *evd_ptr); @@ -169,10 +168,8 @@ u32 dapl_evd_event_alloc(DAPL_EVD *evd_p u32 dat_status = DAT_SUCCESS; /* Allocate EVENTs */ - event_ptr = - (struct dat_event *)kmalloc(evd_ptr->qlen * sizeof *event_ptr, - GFP_ATOMIC); - if (event_ptr == NULL) { + event_ptr = kmalloc(evd_ptr->qlen * sizeof *event_ptr, GFP_ATOMIC); + if (!event_ptr) { dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto bail; diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_hash.c linux-kernel-kmalloc/dat-provider/dapl_hash.c --- linux-kernel-sizeof/dat-provider/dapl_hash.c 2005-05-20 10:10:06.017003000 -0700 +++ linux-kernel-kmalloc/dat-provider/dapl_hash.c 2005-05-20 15:19:47.873960000 -0700 @@ -213,8 +213,8 @@ dapl_hash_add(DAPL_HASH_TABLEP p_table, p_table->num_entries++; status = TRUE; } else { - DAPL_HASH_ELEM *newelement = (DAPL_HASH_ELEM *) - kmalloc(sizeof *newelement, GFP_ATOMIC); + DAPL_HASH_ELEM *newelement = kmalloc(sizeof *newelement, + GFP_ATOMIC); /* * Add an element to the end of the chain */ @@ -232,10 +232,8 @@ dapl_hash_add(DAPL_HASH_TABLEP p_table, lastelement->next_element = newelement; p_table->num_entries++; status = TRUE; - } else { - /* allocation failed - should not happen */ + } else status = FALSE; - } } /* @@ -332,7 +330,7 @@ u32 dapl_hash_create(int table_size, DAP /* Allocate hash table */ p_table = kmalloc(sizeof *p_table, GFP_ATOMIC); - if (NULL == p_table) { + if (!p_table) { dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto bail; @@ -341,8 +339,8 @@ u32 dapl_hash_create(int table_size, DAP /* Init hash table, allocate and init and buckets */ memset(p_table, 0, sizeof *p_table); p_table->tbl_size = table_size; - p_table->table = (DAPL_HASH_ELEM *) kmalloc(table_length, GFP_ATOMIC); - if (NULL == p_table->table) { + p_table->table = kmalloc(table_length, GFP_ATOMIC); + if (!p_table->table) { kfree(p_table); dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_hca_util.c linux-kernel-kmalloc/dat-provider/dapl_hca_util.c --- linux-kernel-sizeof/dat-provider/dapl_hca_util.c 2005-05-20 15:11:31.050974000 -0700 +++ linux-kernel-kmalloc/dat-provider/dapl_hca_util.c 2005-05-20 15:19:47.878970000 -0700 @@ -61,7 +61,7 @@ DAPL_HCA *dapl_hca_alloc(char *name, str DAPL_HCA *hca_ptr; hca_ptr = kmalloc(sizeof *hca_ptr, GFP_ATOMIC); - if (NULL != hca_ptr) { + if (hca_ptr) { memset(hca_ptr, 0, sizeof *hca_ptr); if (DAT_SUCCESS == diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_ia_util.c linux-kernel-kmalloc/dat-provider/dapl_ia_util.c --- linux-kernel-sizeof/dat-provider/dapl_ia_util.c 2005-05-20 15:11:31.438968000 -0700 +++ linux-kernel-kmalloc/dat-provider/dapl_ia_util.c 2005-05-20 15:19:47.884961000 -0700 @@ -63,10 +63,9 @@ DAPL_IA *dapl_ia_alloc(struct dat_provid DAPL_IA *ia_ptr; /* Allocate IA */ - ia_ptr = (DAPL_IA *) kmalloc(sizeof *ia_ptr, GFP_ATOMIC); - if (ia_ptr == NULL) { + ia_ptr = kmalloc(sizeof *ia_ptr, GFP_ATOMIC); + if (!ia_ptr) return NULL; - } /* zero the structure */ memset(ia_ptr, 0, sizeof *ia_ptr); diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_lmr_util.c linux-kernel-kmalloc/dat-provider/dapl_lmr_util.c --- linux-kernel-sizeof/dat-provider/dapl_lmr_util.c 2005-05-20 09:45:43.253013000 -0700 +++ linux-kernel-kmalloc/dat-provider/dapl_lmr_util.c 2005-05-20 15:19:47.890970000 -0700 @@ -48,10 +48,9 @@ DAPL_LMR *dapl_lmr_alloc(DAPL_IA * ia, DAPL_LMR *lmr; /* Allocate LMR */ - lmr = (DAPL_LMR *) kmalloc(sizeof *lmr, GFP_ATOMIC); - if (NULL == lmr) { + lmr = kmalloc(sizeof *lmr, GFP_ATOMIC); + if (!lmr) return NULL; - } /* zero the structure */ memset(lmr, 0, sizeof *lmr); diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_openib_util.c linux-kernel-kmalloc/dat-provider/dapl_openib_util.c --- linux-kernel-sizeof/dat-provider/dapl_openib_util.c 2005-05-20 15:11:31.746962000 -0700 +++ linux-kernel-kmalloc/dat-provider/dapl_openib_util.c 2005-05-20 15:19:47.920964000 -0700 @@ -408,9 +408,8 @@ u32 dapl_ib_mr_register_physical(DAPL_IA u64 *array; array = (u64 *) phys_addr; - buf_list = (struct ib_phys_buf *) - kmalloc(length * sizeof *buf_list, GFP_ATOMIC); - if (NULL == buf_list) + buf_list = kmalloc(length * sizeof *buf_list, GFP_ATOMIC); + if (!buf_list) return DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_provider.c linux-kernel-kmalloc/dat-provider/dapl_provider.c --- linux-kernel-sizeof/dat-provider/dapl_provider.c 2005-05-20 09:53:27.437003000 -0700 +++ linux-kernel-kmalloc/dat-provider/dapl_provider.c 2005-05-20 15:19:47.926965000 -0700 @@ -151,7 +151,7 @@ u32 dapl_provider_list_create(void) /* create the head node */ g_dapl_provider_list.head = kmalloc(sizeof *g_dapl_provider_list.head, GFP_ATOMIC); - if (NULL == g_dapl_provider_list.head) { + if (!g_dapl_provider_list.head) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto bail; @@ -162,7 +162,7 @@ u32 dapl_provider_list_create(void) /* create the tail node */ g_dapl_provider_list.tail = kmalloc(sizeof *g_dapl_provider_list.tail, GFP_ATOMIC); - if (NULL == g_dapl_provider_list.tail) { + if (!g_dapl_provider_list.tail) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto bail; @@ -214,8 +214,7 @@ u32 dapl_provider_list_insert(const char unsigned int len; cur_node = kmalloc(sizeof *cur_node, GFP_ATOMIC); - - if (NULL == cur_node) { + if (!cur_node) { status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); goto bail; diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_pz_util.c linux-kernel-kmalloc/dat-provider/dapl_pz_util.c --- linux-kernel-sizeof/dat-provider/dapl_pz_util.c 2005-05-20 15:11:31.492027000 -0700 +++ linux-kernel-kmalloc/dat-provider/dapl_pz_util.c 2005-05-20 15:23:13.954961000 -0700 @@ -57,8 +57,8 @@ DAPL_PZ *dapl_pz_alloc(DAPL_IA * ia) DAPL_PZ *pz; /* Allocate PZ */ - pz = (DAPL_PZ *) kmalloc(sizeof *pz, GFP_ATOMIC); - if (NULL == pz) + pz = kmalloc(sizeof *pz, GFP_ATOMIC); + if (!pz) return NULL; /* zero the structure */ diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_ring_buffer_util.c linux-kernel-kmalloc/dat-provider/dapl_ring_buffer_util.c --- linux-kernel-sizeof/dat-provider/dapl_ring_buffer_util.c 2005-05-20 15:11:30.864961000 -0700 +++ linux-kernel-kmalloc/dat-provider/dapl_ring_buffer_util.c 2005-05-20 15:19:47.959972000 -0700 @@ -72,14 +72,13 @@ u32 dapl_rbuf_alloc(DAPL_RING_BUFFER *rb rsize <<= 1; } - rbuf->base = (void *) kmalloc(rsize * sizeof *rbuf->base, GFP_ATOMIC); - if (rbuf->base != NULL) { + rbuf->base = kmalloc(rsize * sizeof *rbuf->base, GFP_ATOMIC); + if (rbuf->base) { rbuf->lim = rsize - 1; atomic_set(&rbuf->head, 0); atomic_set(&rbuf->tail, 0); - } else { + } else return DAT_INSUFFICIENT_RESOURCES | DAT_RESOURCE_MEMORY; - } return DAT_SUCCESS; } diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_rmr_util.c linux-kernel-kmalloc/dat-provider/dapl_rmr_util.c --- linux-kernel-sizeof/dat-provider/dapl_rmr_util.c 2005-05-20 09:54:37.797008000 -0700 +++ linux-kernel-kmalloc/dat-provider/dapl_rmr_util.c 2005-05-20 15:19:47.964962000 -0700 @@ -33,10 +33,9 @@ DAPL_RMR *dapl_rmr_alloc(DAPL_PZ * pz) DAPL_RMR *rmr; /* Allocate LMR */ - rmr = (DAPL_RMR *) kmalloc(sizeof *rmr, GFP_ATOMIC); - if (NULL == rmr) { + rmr = kmalloc(sizeof *rmr, GFP_ATOMIC); + if (!rmr) return NULL; - } /* zero the structure */ memset(rmr, 0, sizeof *rmr); diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_sp_util.c linux-kernel-kmalloc/dat-provider/dapl_sp_util.c --- linux-kernel-sizeof/dat-provider/dapl_sp_util.c 2005-05-20 15:11:32.615964000 -0700 +++ linux-kernel-kmalloc/dat-provider/dapl_sp_util.c 2005-05-20 15:19:47.969972000 -0700 @@ -64,8 +64,8 @@ DAPL_SP *dapl_sp_alloc(DAPL_IA * ia_ptr, DAPL_SP *sp_ptr; /* Allocate EP */ - sp_ptr = (DAPL_SP *) kmalloc(sizeof *sp_ptr, GFP_ATOMIC); - if (sp_ptr == NULL) + sp_ptr = kmalloc(sizeof *sp_ptr, GFP_ATOMIC); + if (!sp_ptr) return NULL; /* zero the structure */ diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_srq_util.c linux-kernel-kmalloc/dat-provider/dapl_srq_util.c --- linux-kernel-sizeof/dat-provider/dapl_srq_util.c 2005-05-20 09:55:16.196002000 -0700 +++ linux-kernel-kmalloc/dat-provider/dapl_srq_util.c 2005-05-20 15:19:47.973961000 -0700 @@ -59,11 +59,9 @@ DAPL_SRQ *dapl_srq_alloc(DAPL_IA * ia_pt DAPL_SRQ *srq_ptr; /* Allocate SRQ */ - srq_ptr = (DAPL_SRQ *) kmalloc(sizeof *srq_ptr, GFP_ATOMIC); - - if (srq_ptr == NULL) { + srq_ptr = kmalloc(sizeof *srq_ptr, GFP_ATOMIC); + if (!srq_ptr) goto bail; - } /* zero the structure */ memset(srq_ptr, 0, sizeof *srq_ptr); diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_util.h linux-kernel-kmalloc/dat-provider/dapl_util.h --- linux-kernel-sizeof/dat-provider/dapl_util.h 2005-05-20 15:11:32.661977000 -0700 +++ linux-kernel-kmalloc/dat-provider/dapl_util.h 2005-05-20 15:19:47.999965000 -0700 @@ -144,7 +144,7 @@ static inline void *dapl_os_realloc(void void *newptr; newptr = kmalloc(size, GFP_ATOMIC); - if (newptr != NULL) { + if (newptr) { /* copy the mem array to the new */ memcpy(newptr, ptr, size); kfree(ptr); @@ -159,9 +159,8 @@ static inline void *dapl_os_realloc(void static inline char *dapl_os_strdup(const char *str) { char *ns = kmalloc(strlen(str) + 1, GFP_KERNEL); - if (ns) { + if (ns) strcpy(ns, str); - } return ns; } From mshefty at ichips.intel.com Fri May 20 15:35:31 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Fri, 20 May 2005 15:35:31 -0700 Subject: [openib-general] in need of a simple ulp In-Reply-To: <20050520213317.GC15688@aon.at> References: <426FCF0A.3070806@linuxmachines.com> <426FD098.9090300@ichips.intel.com> <427AC218.9070307@linuxmachines.com> <427AC385.5070308@ichips.intel.com> <427BA0AD.6010105@linuxmachines.com> <427FDA9D.7030905@ichips.intel.com> <428E4268.1040101@linuxmachines.com> <428E4B4A.2090705@ichips.intel.com> <20050520213317.GC15688@aon.at> Message-ID: <428E6633.4080204@ichips.intel.com> Bernhard Fischer wrote: >>In this case you won't know what's on a cqe until you poll the completion. >>An alternative way of structuring the code is to create two CQs, and tie >>one to the send queue, the other to the receive queue. > > hmz, yes, see "Quick unrelated question:" in > http://openib.org/pipermail/openib-general/2005-May/006796.html > > I do have ...s_cq and r_cq. For two nodes, using cqe=64 does obviously > work, but when i've got, say, a setup where 786 nodes each have r and s? > > This currently w{on't,ill in no way way} work (for me)!? Help!! > > Also, Sean, any thought on the qp limit question referenced there? The CQ and QP size limits are controlled by the HCA driver. I don't see that mthca has any specific limitations, and would appear to be limited by system memory. As a comparison, I was able to run tests with 1000 simultaneous connections, each transferring 1000 messages (send and receive QP sizes of 1000, and CQ size of 2000). The message size was 1000, but I cheat and reuse the same data buffer, since I don't care about its actual content. So, a cqe of 64 for 786 fully connected nodes seems doable, but will depend on how much space is allocated for the message buffers. As a side note, when I tried increasing the number of connections to 5000 under this same load, my system barfed with 'Out of Memory' errors... - Sean From jlentini at netapp.com Fri May 20 15:37:46 2005 From: jlentini at netapp.com (James Lentini) Date: Fri, 20 May 2005 18:37:46 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL & at: sizeof is not a function In-Reply-To: <1116611479.10540.23.camel@duffman> References: <1116611479.10540.23.camel@duffman> Message-ID: Committed in revision 2428. On Fri, 20 May 2005, Tom Duffy wrote: tduffy> sizeof is not a function. Also it is generally better to take the tduffy> sizeof the actual object your are memcpy()ing or kmalloc()ing in case tduffy> the type changes. tduffy> tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> Index: linux-kernel-sizeof/dat-provider/dapl_cookie.c tduffy> =================================================================== tduffy> --- linux-kernel-sizeof/dat-provider/dapl_cookie.c (revision 2423) tduffy> +++ linux-kernel-sizeof/dat-provider/dapl_cookie.c (working copy) tduffy> @@ -123,7 +123,7 @@ u32 dapl_cb_create(DAPL_COOKIE_BUFFER *b tduffy> */ tduffy> size++; tduffy> tduffy> - buffer->pool = kmalloc(size * sizeof(DAPL_COOKIE), GFP_ATOMIC); tduffy> + buffer->pool = kmalloc(size * sizeof *buffer->pool, GFP_ATOMIC); tduffy> if (NULL != buffer->pool) { tduffy> buffer->pool_size = size; tduffy> atomic_set(&buffer->head, 0); tduffy> Index: linux-kernel-sizeof/dat-provider/dapl_ep_connect.c tduffy> =================================================================== tduffy> --- linux-kernel-sizeof/dat-provider/dapl_ep_connect.c (revision 2423) tduffy> +++ linux-kernel-sizeof/dat-provider/dapl_ep_connect.c (working copy) tduffy> @@ -281,7 +281,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han tduffy> DAT_EP_STATE_ACTIVE_CONNECTION_PENDING tduffy> && timeout != DAT_TIMEOUT_INFINITE) { tduffy> ep_ptr->cxn_timer = (DAPL_OS_TIMER *) tduffy> - kmalloc(sizeof(DAPL_OS_TIMER), GFP_ATOMIC); tduffy> + kmalloc(sizeof *ep_ptr->cxn_timer, GFP_ATOMIC); tduffy> tduffy> dapl_timer_set(ep_ptr->cxn_timer, tduffy> dapl_ep_timeout, ep_ptr, timeout); tduffy> Index: linux-kernel-sizeof/dat-provider/dapl_ring_buffer_util.c tduffy> =================================================================== tduffy> --- linux-kernel-sizeof/dat-provider/dapl_ring_buffer_util.c (revision 2423) tduffy> +++ linux-kernel-sizeof/dat-provider/dapl_ring_buffer_util.c (working copy) tduffy> @@ -72,7 +72,7 @@ u32 dapl_rbuf_alloc(DAPL_RING_BUFFER *rb tduffy> rsize <<= 1; tduffy> } tduffy> tduffy> - rbuf->base = (void *) kmalloc(rsize * sizeof(void *), GFP_ATOMIC); tduffy> + rbuf->base = (void *) kmalloc(rsize * sizeof *rbuf->base, GFP_ATOMIC); tduffy> if (rbuf->base != NULL) { tduffy> rbuf->lim = rsize - 1; tduffy> atomic_set(&rbuf->head, 0); tduffy> Index: linux-kernel-sizeof/dat-provider/dapl_lmr_util.c tduffy> =================================================================== tduffy> --- linux-kernel-sizeof/dat-provider/dapl_lmr_util.c (revision 2423) tduffy> +++ linux-kernel-sizeof/dat-provider/dapl_lmr_util.c (working copy) tduffy> @@ -48,13 +48,13 @@ DAPL_LMR *dapl_lmr_alloc(DAPL_IA * ia, tduffy> DAPL_LMR *lmr; tduffy> tduffy> /* Allocate LMR */ tduffy> - lmr = (DAPL_LMR *) kmalloc(sizeof(DAPL_LMR), GFP_ATOMIC); tduffy> + lmr = (DAPL_LMR *) kmalloc(sizeof *lmr, GFP_ATOMIC); tduffy> if (NULL == lmr) { tduffy> return NULL; tduffy> } tduffy> tduffy> /* zero the structure */ tduffy> - memset(lmr, 0, sizeof(DAPL_LMR)); tduffy> + memset(lmr, 0, sizeof *lmr); tduffy> tduffy> /* tduffy> * initialize the header tduffy> Index: linux-kernel-sizeof/dat-provider/dapl_hca_util.c tduffy> =================================================================== tduffy> --- linux-kernel-sizeof/dat-provider/dapl_hca_util.c (revision 2423) tduffy> +++ linux-kernel-sizeof/dat-provider/dapl_hca_util.c (working copy) tduffy> @@ -60,9 +60,9 @@ DAPL_HCA *dapl_hca_alloc(char *name, str tduffy> { tduffy> DAPL_HCA *hca_ptr; tduffy> tduffy> - hca_ptr = kmalloc(sizeof(DAPL_HCA), GFP_ATOMIC); tduffy> + hca_ptr = kmalloc(sizeof *hca_ptr, GFP_ATOMIC); tduffy> if (NULL != hca_ptr) { tduffy> - memset(hca_ptr, 0, sizeof(DAPL_HCA)); tduffy> + memset(hca_ptr, 0, sizeof *hca_ptr); tduffy> tduffy> if (DAT_SUCCESS == tduffy> dapl_hash_create(DAPL_HASH_TABLE_DEFAULT_CAPACITY, tduffy> Index: linux-kernel-sizeof/dat-provider/dapl_cr_util.c tduffy> =================================================================== tduffy> --- linux-kernel-sizeof/dat-provider/dapl_cr_util.c (revision 2423) tduffy> +++ linux-kernel-sizeof/dat-provider/dapl_cr_util.c (working copy) tduffy> @@ -42,13 +42,13 @@ DAPL_CR *dapl_cr_alloc(DAPL_IA * ia_ptr) tduffy> DAPL_CR *cr_ptr; tduffy> tduffy> /* Allocate EP */ tduffy> - cr_ptr = (DAPL_CR *) kmalloc(sizeof(DAPL_CR), GFP_ATOMIC); tduffy> + cr_ptr = (DAPL_CR *) kmalloc(sizeof *cr_ptr, GFP_ATOMIC); tduffy> if (cr_ptr == NULL) { tduffy> return NULL; tduffy> } tduffy> tduffy> /* zero the structure */ tduffy> - memset(cr_ptr, 0, sizeof(DAPL_CR)); tduffy> + memset(cr_ptr, 0, sizeof *cr_ptr); tduffy> tduffy> /* tduffy> * initialize the header tduffy> Index: linux-kernel-sizeof/dat-provider/dapl_srq_util.c tduffy> =================================================================== tduffy> --- linux-kernel-sizeof/dat-provider/dapl_srq_util.c (revision 2423) tduffy> +++ linux-kernel-sizeof/dat-provider/dapl_srq_util.c (working copy) tduffy> @@ -59,14 +59,14 @@ DAPL_SRQ *dapl_srq_alloc(DAPL_IA * ia_pt tduffy> DAPL_SRQ *srq_ptr; tduffy> tduffy> /* Allocate SRQ */ tduffy> - srq_ptr = (DAPL_SRQ *) kmalloc(sizeof(DAPL_SRQ), GFP_ATOMIC); tduffy> + srq_ptr = (DAPL_SRQ *) kmalloc(sizeof *srq_ptr, GFP_ATOMIC); tduffy> tduffy> if (srq_ptr == NULL) { tduffy> goto bail; tduffy> } tduffy> tduffy> /* zero the structure */ tduffy> - memset(srq_ptr, 0, sizeof(DAPL_SRQ)); tduffy> + memset(srq_ptr, 0, sizeof *srq_ptr); tduffy> tduffy> /* tduffy> * initialize the header tduffy> Index: linux-kernel-sizeof/dat-provider/dapl_pz_util.c tduffy> =================================================================== tduffy> --- linux-kernel-sizeof/dat-provider/dapl_pz_util.c (revision 2423) tduffy> +++ linux-kernel-sizeof/dat-provider/dapl_pz_util.c (working copy) tduffy> @@ -57,13 +57,13 @@ DAPL_PZ *dapl_pz_alloc(DAPL_IA * ia) tduffy> DAPL_PZ *pz; tduffy> tduffy> /* Allocate PZ */ tduffy> - pz = (DAPL_PZ *) kmalloc(sizeof(DAPL_PZ), GFP_ATOMIC); tduffy> + pz = (DAPL_PZ *) kmalloc(sizeof *pz, GFP_ATOMIC); tduffy> if (NULL == pz) { tduffy> return NULL; tduffy> } tduffy> tduffy> /* zero the structure */ tduffy> - memset(pz, 0, sizeof(DAPL_PZ)); tduffy> + memset(pz, 0, sizeof *pz); tduffy> tduffy> /* tduffy> * initialize the header tduffy> Index: linux-kernel-sizeof/dat-provider/dapl_ia_util.c tduffy> =================================================================== tduffy> --- linux-kernel-sizeof/dat-provider/dapl_ia_util.c (revision 2423) tduffy> +++ linux-kernel-sizeof/dat-provider/dapl_ia_util.c (working copy) tduffy> @@ -63,13 +63,13 @@ DAPL_IA *dapl_ia_alloc(struct dat_provid tduffy> DAPL_IA *ia_ptr; tduffy> tduffy> /* Allocate IA */ tduffy> - ia_ptr = (DAPL_IA *) kmalloc(sizeof(DAPL_IA), GFP_ATOMIC); tduffy> + ia_ptr = (DAPL_IA *) kmalloc(sizeof *ia_ptr, GFP_ATOMIC); tduffy> if (ia_ptr == NULL) { tduffy> return NULL; tduffy> } tduffy> tduffy> /* zero the structure */ tduffy> - memset(ia_ptr, 0, sizeof(DAPL_IA)); tduffy> + memset(ia_ptr, 0, sizeof *ia_ptr); tduffy> tduffy> /* tduffy> * initialize the header tduffy> Index: linux-kernel-sizeof/dat-provider/dapl_provider.c tduffy> =================================================================== tduffy> --- linux-kernel-sizeof/dat-provider/dapl_provider.c (revision 2423) tduffy> +++ linux-kernel-sizeof/dat-provider/dapl_provider.c (working copy) tduffy> @@ -150,27 +150,25 @@ u32 dapl_provider_list_create(void) tduffy> tduffy> /* create the head node */ tduffy> g_dapl_provider_list.head = tduffy> - kmalloc(sizeof(DAPL_PROVIDER_LIST_NODE), GFP_ATOMIC); tduffy> + kmalloc(sizeof *g_dapl_provider_list.head, GFP_ATOMIC); tduffy> if (NULL == g_dapl_provider_list.head) { tduffy> status = tduffy> DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> goto bail; tduffy> } tduffy> tduffy> - memset(g_dapl_provider_list.head, 0, tduffy> - sizeof(DAPL_PROVIDER_LIST_NODE)); tduffy> + memset(g_dapl_provider_list.head, 0, sizeof *g_dapl_provider_list.head); tduffy> tduffy> /* create the tail node */ tduffy> g_dapl_provider_list.tail = tduffy> - kmalloc(sizeof(DAPL_PROVIDER_LIST_NODE), GFP_ATOMIC); tduffy> + kmalloc(sizeof *g_dapl_provider_list.tail, GFP_ATOMIC); tduffy> if (NULL == g_dapl_provider_list.tail) { tduffy> status = tduffy> DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> goto bail; tduffy> } tduffy> tduffy> - memset(g_dapl_provider_list.tail, 0, tduffy> - sizeof(DAPL_PROVIDER_LIST_NODE)); tduffy> + memset(g_dapl_provider_list.tail, 0, sizeof *g_dapl_provider_list.tail); tduffy> tduffy> g_dapl_provider_list.head->next = g_dapl_provider_list.tail; tduffy> g_dapl_provider_list.tail->prev = g_dapl_provider_list.head; tduffy> @@ -215,7 +213,7 @@ u32 dapl_provider_list_insert(const char tduffy> u32 status = DAT_SUCCESS; tduffy> unsigned int len; tduffy> tduffy> - cur_node = kmalloc(sizeof(DAPL_PROVIDER_LIST_NODE), GFP_ATOMIC); tduffy> + cur_node = kmalloc(sizeof *cur_node, GFP_ATOMIC); tduffy> tduffy> if (NULL == cur_node) { tduffy> status = tduffy> Index: linux-kernel-sizeof/dat-provider/dapl_openib_cm.c tduffy> =================================================================== tduffy> --- linux-kernel-sizeof/dat-provider/dapl_openib_cm.c (revision 2423) tduffy> +++ linux-kernel-sizeof/dat-provider/dapl_openib_cm.c (working copy) tduffy> @@ -427,7 +427,7 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han tduffy> ia_ptr = conn->ep->header.owner_ia; tduffy> tduffy> /* Setup QP/CM parameters */ tduffy> - memset(&conn->param, 0, sizeof(conn->param)); tduffy> + memset(&conn->param, 0, sizeof conn->param); tduffy> conn->param.qp_type = IB_QPT_RC; tduffy> conn->param.qp_num = conn->ep->qp_handle->qp_num; tduffy> conn->param.private_data = private_data; tduffy> @@ -441,7 +441,7 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han tduffy> conn->param.local_cm_response_timeout = DAPL_OPENIB_CM_RESPONSE_TIMEOUT; tduffy> conn->param.max_cm_retries = DAPL_OPENIB_MAX_CM_RETRIES; tduffy> tduffy> - memcpy(&conn->service_id, &remote_conn_qual, sizeof(u64)); tduffy> + memcpy(&conn->service_id, &remote_conn_qual, sizeof conn->service_id); tduffy> tduffy> conn->remote_ia_address = remote_ia_address; tduffy> conn->dapl_comp.fn = &dapl_rt_comp_handler; tduffy> @@ -771,7 +771,7 @@ u32 dapl_ib_accept_connection(DAT_CR_HAN tduffy> ep_ptr->cm_handle = conn; tduffy> conn->ep = ep_ptr; tduffy> tduffy> - memset(&passive_params, 0, sizeof(passive_params)); tduffy> + memset(&passive_params, 0, sizeof passive_params); tduffy> passive_params.private_data = prd_ptr; tduffy> passive_params.private_data_len = private_data_size; tduffy> passive_params.qp_num = ep_ptr->qp_handle->qp_num; tduffy> @@ -848,7 +848,7 @@ u32 dapl_ib_cm_remote_addr(DAT_HANDLE da tduffy> tduffy> if (!memcmp(&ia_ptr->hca_ptr->ib_trans.gid, &remote_gid, 16)) { tduffy> memcpy(remote_ia_address, &ia_ptr->hca_ptr->hca_address, tduffy> - sizeof(struct sockaddr)); tduffy> + sizeof *remote_ia_address); tduffy> return DAT_SUCCESS; tduffy> } tduffy> tduffy> Index: linux-kernel-sizeof/dat-provider/dapl_openib_util.c tduffy> =================================================================== tduffy> --- linux-kernel-sizeof/dat-provider/dapl_openib_util.c (revision 2423) tduffy> +++ linux-kernel-sizeof/dat-provider/dapl_openib_util.c (working copy) tduffy> @@ -164,7 +164,7 @@ u32 dapl_ib_open_hca(char *name, DAPL_HC tduffy> { tduffy> u32 dat_ret; tduffy> tduffy> - memset(&hca_ptr->ib_trans.gid.global, 0, sizeof(union ib_gid)); tduffy> + memset(&hca_ptr->ib_trans.gid.global, 0, sizeof hca_ptr->ib_trans.gid); tduffy> hca_ptr->ib_trans.lid = 0; tduffy> tduffy> dat_ret = dapl_ib_get_hca_ids(hca_ptr->ib_hca_handle, tduffy> @@ -410,7 +410,7 @@ u32 dapl_ib_mr_register_physical(DAPL_IA tduffy> tduffy> array = (u64 *) phys_addr; tduffy> buf_list = (struct ib_phys_buf *) tduffy> - kmalloc(length * sizeof(struct ib_phys_buf), GFP_ATOMIC); tduffy> + kmalloc(length * sizeof *buf_list, GFP_ATOMIC); tduffy> if (NULL == buf_list) tduffy> return DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, tduffy> DAT_RESOURCE_MEMORY); tduffy> Index: linux-kernel-sizeof/dat-provider/dapl_ep_modify.c tduffy> =================================================================== tduffy> --- linux-kernel-sizeof/dat-provider/dapl_ep_modify.c (revision 2423) tduffy> +++ linux-kernel-sizeof/dat-provider/dapl_ep_modify.c (working copy) tduffy> @@ -385,8 +385,7 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand tduffy> tduffy> alloc_ep.send_iov_num = ep_attr1.max_request_iov; tduffy> alloc_ep.send_iov = kmalloc(ep_attr1.max_request_iov tduffy> - * sizeof(ib_data_segment_t), tduffy> - GFP_ATOMIC); tduffy> + * sizeof *alloc_ep.send_iov, GFP_ATOMIC); tduffy> if (alloc_ep.recv_iov == NULL) { tduffy> dat_status = tduffy> DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> @@ -396,8 +395,7 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand tduffy> tduffy> alloc_ep.recv_iov_num = ep_attr1.max_recv_iov; tduffy> alloc_ep.recv_iov = kmalloc(ep_attr1.max_recv_iov tduffy> - * sizeof(ib_data_segment_t), tduffy> - GFP_ATOMIC); tduffy> + * sizeof *alloc_ep.recv_iov, GFP_ATOMIC); tduffy> if (alloc_ep.recv_iov == NULL) { tduffy> dat_status = tduffy> DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> Index: linux-kernel-sizeof/dat-provider/dapl_openib_qp.c tduffy> =================================================================== tduffy> --- linux-kernel-sizeof/dat-provider/dapl_openib_qp.c (revision 2423) tduffy> +++ linux-kernel-sizeof/dat-provider/dapl_openib_qp.c (working copy) tduffy> @@ -305,7 +305,7 @@ int dapl_modify_qp_state_to_init(ib_qp_h tduffy> int qp_attr_mask; tduffy> int ib_status; tduffy> tduffy> - memset(&qp_attr, 0, sizeof(qp_attr)); tduffy> + memset(&qp_attr, 0, sizeof qp_attr); tduffy> qp_attr.qp_state = IB_QPS_INIT; tduffy> qp_attr_mask = IB_QP_STATE; tduffy> qp_attr.qp_access_flags = tduffy> @@ -328,7 +328,7 @@ int dapl_modify_qp_state_to_error(ib_qp_ tduffy> int qp_attr_mask; tduffy> int ib_status; tduffy> tduffy> - memset(&qp_attr, 0, sizeof(qp_attr)); tduffy> + memset(&qp_attr, 0, sizeof qp_attr); tduffy> qp_attr.qp_state = IB_QPS_ERR; tduffy> qp_attr_mask = IB_QP_STATE; tduffy> ib_status = ib_modify_qp(qp, &qp_attr, qp_attr_mask); tduffy> @@ -342,7 +342,7 @@ int dapl_modify_qp_state_to_reset(ib_qp_ tduffy> int qp_attr_mask; tduffy> int ib_status; tduffy> tduffy> - memset(&qp_attr, 0, sizeof(qp_attr)); tduffy> + memset(&qp_attr, 0, sizeof qp_attr); tduffy> qp_attr.qp_state = IB_QPS_RESET; tduffy> qp_attr_mask = IB_QP_STATE; tduffy> ib_status = ib_modify_qp(qp, &qp_attr, qp_attr_mask); tduffy> @@ -356,7 +356,7 @@ int dapl_modify_qp_state_to_rtr(struct i tduffy> int qp_attr_mask; tduffy> int ib_status; tduffy> tduffy> - memset(&qp_attr, 0, sizeof(qp_attr)); tduffy> + memset(&qp_attr, 0, sizeof qp_attr); tduffy> qp_attr.qp_state = IB_QPS_RTR; tduffy> tduffy> ib_status = ib_cm_init_qp_attr(cm_id, &qp_attr, &qp_attr_mask); tduffy> @@ -376,7 +376,7 @@ int dapl_modify_qp_state_to_rts(struct i tduffy> int qp_attr_mask; tduffy> int ib_status; tduffy> tduffy> - memset(&qp_attr, 0, sizeof(qp_attr)); tduffy> + memset(&qp_attr, 0, sizeof qp_attr); tduffy> qp_attr.qp_state = IB_QPS_RTS; tduffy> tduffy> ib_status = ib_cm_init_qp_attr(cm_id, &qp_attr, &qp_attr_mask); tduffy> Index: linux-kernel-sizeof/dat-provider/dapl_cno_util.c tduffy> =================================================================== tduffy> --- linux-kernel-sizeof/dat-provider/dapl_cno_util.c (revision 2423) tduffy> +++ linux-kernel-sizeof/dat-provider/dapl_cno_util.c (working copy) tduffy> @@ -42,12 +42,12 @@ DAPL_CNO *dapl_cno_alloc(DAPL_IA *ia_ptr tduffy> { tduffy> DAPL_CNO *cno_ptr; tduffy> tduffy> - cno_ptr = kmalloc(sizeof(*cno_ptr), GFP_ATOMIC); tduffy> + cno_ptr = kmalloc(sizeof *cno_ptr, GFP_ATOMIC); tduffy> if (!cno_ptr) tduffy> return NULL; tduffy> tduffy> /* zero the structure */ tduffy> - memset(cno_ptr, 0, sizeof(*cno_ptr)); tduffy> + memset(cno_ptr, 0, sizeof *cno_ptr); tduffy> tduffy> /* tduffy> * Initialize the header. tduffy> Index: linux-kernel-sizeof/dat-provider/dapl_rmr_util.c tduffy> =================================================================== tduffy> --- linux-kernel-sizeof/dat-provider/dapl_rmr_util.c (revision 2423) tduffy> +++ linux-kernel-sizeof/dat-provider/dapl_rmr_util.c (working copy) tduffy> @@ -33,13 +33,13 @@ DAPL_RMR *dapl_rmr_alloc(DAPL_PZ * pz) tduffy> DAPL_RMR *rmr; tduffy> tduffy> /* Allocate LMR */ tduffy> - rmr = (DAPL_RMR *) kmalloc(sizeof(DAPL_RMR), GFP_ATOMIC); tduffy> + rmr = (DAPL_RMR *) kmalloc(sizeof *rmr, GFP_ATOMIC); tduffy> if (NULL == rmr) { tduffy> return NULL; tduffy> } tduffy> tduffy> /* zero the structure */ tduffy> - memset(rmr, 0, sizeof(DAPL_RMR)); tduffy> + memset(rmr, 0, sizeof *rmr); tduffy> tduffy> /* tduffy> * initialize the header tduffy> Index: linux-kernel-sizeof/dat-provider/dapl_ep_util.c tduffy> =================================================================== tduffy> --- linux-kernel-sizeof/dat-provider/dapl_ep_util.c (revision 2423) tduffy> +++ linux-kernel-sizeof/dat-provider/dapl_ep_util.c (working copy) tduffy> @@ -60,13 +60,13 @@ DAPL_EP *dapl_ep_alloc(DAPL_IA * ia_ptr, tduffy> DAPL_EP *ep_ptr; tduffy> tduffy> /* Allocate EP */ tduffy> - ep_ptr = (DAPL_EP *) kmalloc(sizeof(DAPL_EP) + sizeof(struct sockaddr), tduffy> + ep_ptr = (DAPL_EP *) kmalloc(sizeof *ep_ptr + sizeof (struct sockaddr), tduffy> GFP_ATOMIC); tduffy> if (ep_ptr == NULL) tduffy> goto bail; tduffy> tduffy> /* zero the structure */ tduffy> - memset(ep_ptr, 0, sizeof(DAPL_EP) + sizeof(struct sockaddr)); tduffy> + memset(ep_ptr, 0, sizeof *ep_ptr + sizeof (struct sockaddr)); tduffy> tduffy> /* tduffy> * initialize the header tduffy> @@ -125,7 +125,7 @@ DAPL_EP *dapl_ep_alloc(DAPL_IA * ia_ptr, tduffy> ep_ptr->send_iov_num = ep_ptr->param.ep_attr.max_request_iov; tduffy> tduffy> ep_ptr->recv_iov = kmalloc(ep_ptr->recv_iov_num * tduffy> - sizeof(ib_data_segment_t), GFP_ATOMIC); tduffy> + sizeof *ep_ptr->recv_iov, GFP_ATOMIC); tduffy> tduffy> if (NULL == ep_ptr->recv_iov) { tduffy> dapl_ep_dealloc(ep_ptr); tduffy> @@ -134,7 +134,7 @@ DAPL_EP *dapl_ep_alloc(DAPL_IA * ia_ptr, tduffy> } tduffy> tduffy> ep_ptr->send_iov = kmalloc(ep_ptr->send_iov_num * tduffy> - sizeof(ib_data_segment_t), GFP_ATOMIC); tduffy> + sizeof *ep_ptr->send_iov, GFP_ATOMIC); tduffy> tduffy> if (NULL == ep_ptr->send_iov) { tduffy> dapl_ep_dealloc(ep_ptr); tduffy> Index: linux-kernel-sizeof/dat-provider/dapl_evd_util.c tduffy> =================================================================== tduffy> --- linux-kernel-sizeof/dat-provider/dapl_evd_util.c (revision 2423) tduffy> +++ linux-kernel-sizeof/dat-provider/dapl_evd_util.c (working copy) tduffy> @@ -132,13 +132,13 @@ DAPL_EVD *dapl_evd_alloc(DAPL_IA * ia_pt tduffy> DAPL_EVD *evd_ptr; tduffy> tduffy> /* Allocate EVD */ tduffy> - evd_ptr = (DAPL_EVD *) kmalloc(sizeof(DAPL_EVD), GFP_ATOMIC); tduffy> + evd_ptr = (DAPL_EVD *) kmalloc(sizeof *evd_ptr, GFP_ATOMIC); tduffy> if (!evd_ptr) { tduffy> goto bail; tduffy> } tduffy> tduffy> /* zero the structure */ tduffy> - memset(evd_ptr, 0, sizeof(DAPL_EVD)); tduffy> + memset(evd_ptr, 0, sizeof *evd_ptr); tduffy> tduffy> /* tduffy> * initialize the header tduffy> Index: linux-kernel-sizeof/dat-provider/dapl_util.c tduffy> =================================================================== tduffy> --- linux-kernel-sizeof/dat-provider/dapl_util.c (revision 2423) tduffy> +++ linux-kernel-sizeof/dat-provider/dapl_util.c (working copy) tduffy> @@ -167,7 +167,7 @@ void dapl_dbg_log(DAPL_DBG_TYPE type, co tduffy> tduffy> if (type & g_dapl_dbg_mask) { tduffy> va_start(args, fmt); tduffy> - vsnprintf(buf, sizeof(buf), fmt, args); tduffy> + vsnprintf(buf, sizeof buf, fmt, args); tduffy> printk(KERN_ALERT "kDAPL: %s", buf); tduffy> va_end(args); tduffy> } tduffy> Index: linux-kernel-sizeof/dat-provider/dapl_sp_util.c tduffy> =================================================================== tduffy> --- linux-kernel-sizeof/dat-provider/dapl_sp_util.c (revision 2423) tduffy> +++ linux-kernel-sizeof/dat-provider/dapl_sp_util.c (working copy) tduffy> @@ -64,12 +64,12 @@ DAPL_SP *dapl_sp_alloc(DAPL_IA * ia_ptr, tduffy> DAPL_SP *sp_ptr; tduffy> tduffy> /* Allocate EP */ tduffy> - sp_ptr = (DAPL_SP *) kmalloc(sizeof(DAPL_SP), GFP_ATOMIC); tduffy> + sp_ptr = (DAPL_SP *) kmalloc(sizeof *sp_ptr, GFP_ATOMIC); tduffy> if (sp_ptr == NULL) tduffy> return NULL; tduffy> tduffy> /* zero the structure */ tduffy> - memset(sp_ptr, 0, sizeof(DAPL_SP)); tduffy> + memset(sp_ptr, 0, sizeof *sp_ptr); tduffy> tduffy> /* tduffy> * initialize the header tduffy> Index: linux-kernel-sizeof/dat-provider/dapl_hash.c tduffy> =================================================================== tduffy> --- linux-kernel-sizeof/dat-provider/dapl_hash.c (revision 2423) tduffy> +++ linux-kernel-sizeof/dat-provider/dapl_hash.c (working copy) tduffy> @@ -214,7 +214,7 @@ dapl_hash_add(DAPL_HASH_TABLEP p_table, tduffy> status = TRUE; tduffy> } else { tduffy> DAPL_HASH_ELEM *newelement = (DAPL_HASH_ELEM *) tduffy> - kmalloc(sizeof(DAPL_HASH_ELEM), GFP_ATOMIC); tduffy> + kmalloc(sizeof *newelement, GFP_ATOMIC); tduffy> /* tduffy> * Add an element to the end of the chain tduffy> */ tduffy> @@ -324,14 +324,14 @@ dapl_hash_delete_element(DAPL_HASH_ELEM tduffy> u32 dapl_hash_create(int table_size, DAPL_HASH_TABLE **pp_table) tduffy> { tduffy> DAPL_HASH_TABLE *p_table; tduffy> - int table_length = table_size * sizeof(DAPL_HASH_ELEM); tduffy> + int table_length = table_size * sizeof (struct DAPL_HASH_ELEM); tduffy> u32 dat_status = DAT_SUCCESS; tduffy> int i; tduffy> tduffy> dapl_os_assert(pp_table); tduffy> tduffy> /* Allocate hash table */ tduffy> - p_table = kmalloc(sizeof(DAPL_HASH_TABLE), GFP_ATOMIC); tduffy> + p_table = kmalloc(sizeof *p_table, GFP_ATOMIC); tduffy> if (NULL == p_table) { tduffy> dat_status = tduffy> DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> @@ -339,7 +339,7 @@ u32 dapl_hash_create(int table_size, DAP tduffy> } tduffy> tduffy> /* Init hash table, allocate and init and buckets */ tduffy> - memset(p_table, 0, sizeof(DAPL_HASH_TABLE)); tduffy> + memset(p_table, 0, sizeof *p_table); tduffy> p_table->tbl_size = table_size; tduffy> p_table->table = (DAPL_HASH_ELEM *) kmalloc(table_length, GFP_ATOMIC); tduffy> if (NULL == p_table->table) { tduffy> Index: linux-kernel-sizeof/patches/at.c tduffy> =================================================================== tduffy> --- linux-kernel-sizeof/patches/at.c (revision 2423) tduffy> +++ linux-kernel-sizeof/patches/at.c (working copy) tduffy> @@ -253,7 +253,7 @@ static int resolve_ip(struct ib_at_src * tduffy> src->dev = priv->ca; tduffy> src->port = priv->port; tduffy> src->pkey = cpu_to_be16(priv->pkey); tduffy> - memcpy(&src->gid, ipoib_dev->dev_addr + 4, sizeof(src->gid)); tduffy> + memcpy(&src->gid, ipoib_dev->dev_addr + 4, sizeof src->gid); tduffy> tduffy> if (!dgid) tduffy> return 0; tduffy> @@ -263,17 +263,13 @@ static int resolve_ip(struct ib_at_src * tduffy> * the IB device which was found. tduffy> */ tduffy> if (rt->u.dst.neighbour->dev->flags & IFF_LOOPBACK) { tduffy> - memcpy(dgid, ipoib_dev->dev_addr + 4, tduffy> - sizeof(union ib_gid)); tduffy> - tduffy> + memcpy(dgid, ipoib_dev->dev_addr + 4, sizeof *dgid); tduffy> return 1; tduffy> } tduffy> tduffy> if ((NUD_CONNECTED|NUD_DELAY|NUD_PROBE) & tduffy> rt->u.dst.neighbour->nud_state) { tduffy> - memcpy(dgid, rt->u.dst.neighbour->ha + 4, tduffy> - sizeof(union ib_gid)); tduffy> - tduffy> + memcpy(dgid, rt->u.dst.neighbour->ha + 4, sizeof *dgid); tduffy> return 1; tduffy> } tduffy> tduffy> @@ -300,7 +296,7 @@ static u64 alloc_req_id(void) tduffy> static void req_init(struct async *pend, void *data, int nelem, int type, tduffy> struct ib_at_completion *async_comp) tduffy> { tduffy> - memset(pend, 0, sizeof(*pend)); tduffy> + memset(pend, 0, sizeof *pend); tduffy> tduffy> pend->id = async_comp->req_id = alloc_req_id(); tduffy> pend->status = IB_AT_STATUS_INVALID; tduffy> @@ -454,8 +450,8 @@ static int same_path_req(struct async *a tduffy> struct path_req *pa = container_of(a, struct path_req, pend); tduffy> struct path_req *pb = container_of(b, struct path_req, pend); tduffy> tduffy> - return !memcmp(&pa->rt.sgid, &pb->rt.sgid, sizeof(pa->rt.sgid)) && tduffy> - !memcmp(&pa->rt.dgid, &pb->rt.dgid, sizeof(pa->rt.dgid)) && tduffy> + return !memcmp(&pa->rt.sgid, &pb->rt.sgid, sizeof pa->rt.sgid) && tduffy> + !memcmp(&pa->rt.dgid, &pb->rt.dgid, sizeof pa->rt.dgid) && tduffy> pa->rt.out_dev == pb->rt.out_dev && tduffy> pa->rt.out_port == pb->rt.out_port && tduffy> pa->rt.attr.pkey == pb->rt.attr.pkey && tduffy> @@ -784,7 +780,7 @@ static int ib_at_arp_recv(struct sk_buff tduffy> * queue IB arp packet onto work queue. tduffy> */ tduffy> DEBUG("recv IB ARP - queue work"); tduffy> - work = kmalloc(sizeof(*work), GFP_ATOMIC); tduffy> + work = kmalloc(sizeof *work, GFP_ATOMIC); tduffy> if (!work) tduffy> goto done; tduffy> tduffy> @@ -1000,7 +996,7 @@ static int ib_at_init(void) tduffy> * Init memory pools tduffy> */ tduffy> route_req_cache = kmem_cache_create("ib_at_routes", tduffy> - sizeof(struct route_req), tduffy> + sizeof (struct route_req), tduffy> 0, SLAB_HWCACHE_ALIGN, tduffy> NULL, NULL); tduffy> if (!route_req_cache) { tduffy> @@ -1010,7 +1006,7 @@ static int ib_at_init(void) tduffy> } tduffy> tduffy> path_req_cache = kmem_cache_create("ib_at_paths", tduffy> - sizeof(struct path_req), tduffy> + sizeof (struct path_req), tduffy> 0, SLAB_HWCACHE_ALIGN, tduffy> NULL, NULL); tduffy> if (!path_req_cache) { tduffy> Index: linux-kernel-sizeof/dat/dictionary.c tduffy> =================================================================== tduffy> --- linux-kernel-sizeof/dat/dictionary.c (revision 2423) tduffy> +++ linux-kernel-sizeof/dat/dictionary.c (working copy) tduffy> @@ -96,7 +96,7 @@ u32 dat_dictionary_entry_create(struct d tduffy> tduffy> BUG_ON(NULL == p_entry); tduffy> tduffy> - entry = kmalloc(sizeof(*entry), GFP_ATOMIC); tduffy> + entry = kmalloc(sizeof *entry, GFP_ATOMIC); tduffy> if (NULL == entry) { tduffy> status = tduffy> DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> Index: linux-kernel-sizeof/dat/dr.c tduffy> =================================================================== tduffy> --- linux-kernel-sizeof/dat/dr.c (revision 2423) tduffy> +++ linux-kernel-sizeof/dat/dr.c (working copy) tduffy> @@ -84,7 +84,7 @@ u32 dat_dr_insert(const struct dat_provi tduffy> unsigned long flags; tduffy> u32 status; tduffy> tduffy> - data = kmalloc(sizeof(*data), GFP_ATOMIC); tduffy> + data = kmalloc(sizeof *data, GFP_ATOMIC); tduffy> if (NULL == data) { tduffy> status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, tduffy> DAT_RESOURCE_MEMORY); tduffy> @@ -241,7 +241,7 @@ u32 dat_dr_list(int max_to_return, int * tduffy> goto bail; tduffy> } tduffy> tduffy> - array = kmalloc(array_size * sizeof(*array), GFP_ATOMIC); tduffy> + array = kmalloc(array_size * sizeof *array, GFP_ATOMIC); tduffy> if (NULL == array) { tduffy> status = tduffy> DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, tduffy> Index: linux-kernel-sizeof/dat/core.c tduffy> =================================================================== tduffy> --- linux-kernel-sizeof/dat/core.c (revision 2423) tduffy> +++ linux-kernel-sizeof/dat/core.c (working copy) tduffy> @@ -66,7 +66,7 @@ void dat_dbg_print(DAT_DBG_TYPE type, co tduffy> va_list args; tduffy> tduffy> va_start(args, fmt); tduffy> - vsnprintf(buf, sizeof(buf), fmt, args); tduffy> + vsnprintf(buf, sizeof buf, fmt, args); tduffy> printk(KERN_ALERT "DAT: %s", buf); tduffy> va_end(args); tduffy> } tduffy> From jlentini at netapp.com Fri May 20 15:47:07 2005 From: jlentini at netapp.com (James Lentini) Date: Fri, 20 May 2005 18:47:07 -0400 (EDT) Subject: [openib-general] [PATCHv2] kDAPL: kmalloc needs no cast In-Reply-To: <1116628112.3627.12.camel@duffman> References: <1116627591.3627.9.camel@duffman> <1116628112.3627.12.camel@duffman> Message-ID: Committed in revision 2429. On Fri, 20 May 2005, Tom Duffy wrote: tduffy> This is a repost rebased off of r2427. tduffy> tduffy> Please apply after my v2 of the sizeof patch. tduffy> tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat/dictionary.c linux-kernel-kmalloc/dat/dictionary.c tduffy> --- linux-kernel-sizeof/dat/dictionary.c 2005-05-20 09:55:56.230003000 -0700 tduffy> +++ linux-kernel-kmalloc/dat/dictionary.c 2005-05-20 15:19:47.778968000 -0700 tduffy> @@ -97,7 +97,7 @@ u32 dat_dictionary_entry_create(struct d tduffy> BUG_ON(NULL == p_entry); tduffy> tduffy> entry = kmalloc(sizeof *entry, GFP_ATOMIC); tduffy> - if (NULL == entry) { tduffy> + if (!entry) { tduffy> status = tduffy> DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> goto bail; tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat/dr.c linux-kernel-kmalloc/dat/dr.c tduffy> --- linux-kernel-sizeof/dat/dr.c 2005-05-20 09:56:11.726007000 -0700 tduffy> +++ linux-kernel-kmalloc/dat/dr.c 2005-05-20 15:19:47.787972000 -0700 tduffy> @@ -85,7 +85,7 @@ u32 dat_dr_insert(const struct dat_provi tduffy> u32 status; tduffy> tduffy> data = kmalloc(sizeof *data, GFP_ATOMIC); tduffy> - if (NULL == data) { tduffy> + if (!data) { tduffy> status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, tduffy> DAT_RESOURCE_MEMORY); tduffy> goto bail; tduffy> @@ -242,7 +242,7 @@ u32 dat_dr_list(int max_to_return, int * tduffy> } tduffy> tduffy> array = kmalloc(array_size * sizeof *array, GFP_ATOMIC); tduffy> - if (NULL == array) { tduffy> + if (!array) { tduffy> status = tduffy> DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, tduffy> DAT_RESOURCE_MEMORY); tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_cookie.c linux-kernel-kmalloc/dat-provider/dapl_cookie.c tduffy> --- linux-kernel-sizeof/dat-provider/dapl_cookie.c 2005-05-20 10:47:10.741014000 -0700 tduffy> +++ linux-kernel-kmalloc/dat-provider/dapl_cookie.c 2005-05-20 15:19:47.793960000 -0700 tduffy> @@ -124,7 +124,7 @@ u32 dapl_cb_create(DAPL_COOKIE_BUFFER *b tduffy> size++; tduffy> tduffy> buffer->pool = kmalloc(size * sizeof *buffer->pool, GFP_ATOMIC); tduffy> - if (NULL != buffer->pool) { tduffy> + if (buffer->pool) { tduffy> buffer->pool_size = size; tduffy> atomic_set(&buffer->head, 0); tduffy> atomic_set(&buffer->tail, 0); tduffy> @@ -135,9 +135,8 @@ u32 dapl_cb_create(DAPL_COOKIE_BUFFER *b tduffy> } tduffy> tduffy> return DAT_SUCCESS; tduffy> - } else { tduffy> + } else tduffy> return DAT_INSUFFICIENT_RESOURCES; tduffy> - } tduffy> } tduffy> tduffy> /* tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_cr_util.c linux-kernel-kmalloc/dat-provider/dapl_cr_util.c tduffy> --- linux-kernel-sizeof/dat-provider/dapl_cr_util.c 2005-05-20 09:39:00.781014000 -0700 tduffy> +++ linux-kernel-kmalloc/dat-provider/dapl_cr_util.c 2005-05-20 15:19:47.797971000 -0700 tduffy> @@ -42,10 +42,9 @@ DAPL_CR *dapl_cr_alloc(DAPL_IA * ia_ptr) tduffy> DAPL_CR *cr_ptr; tduffy> tduffy> /* Allocate EP */ tduffy> - cr_ptr = (DAPL_CR *) kmalloc(sizeof *cr_ptr, GFP_ATOMIC); tduffy> - if (cr_ptr == NULL) { tduffy> + cr_ptr = kmalloc(sizeof *cr_ptr, GFP_ATOMIC); tduffy> + if (!cr_ptr) tduffy> return NULL; tduffy> - } tduffy> tduffy> /* zero the structure */ tduffy> memset(cr_ptr, 0, sizeof *cr_ptr); tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_ep_connect.c linux-kernel-kmalloc/dat-provider/dapl_ep_connect.c tduffy> --- linux-kernel-sizeof/dat-provider/dapl_ep_connect.c 2005-05-20 15:12:36.692972000 -0700 tduffy> +++ linux-kernel-kmalloc/dat-provider/dapl_ep_connect.c 2005-05-20 15:22:16.513960000 -0700 tduffy> @@ -280,8 +280,13 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han tduffy> if (ep_ptr->param.ep_state == tduffy> DAT_EP_STATE_ACTIVE_CONNECTION_PENDING tduffy> && timeout != DAT_TIMEOUT_INFINITE) { tduffy> - ep_ptr->cxn_timer = (struct dapl_timer_entry *) tduffy> - kmalloc(sizeof *ep_ptr->cxn_timer, GFP_ATOMIC); tduffy> + ep_ptr->cxn_timer = kmalloc(sizeof *ep_ptr->cxn_timer, tduffy> + GFP_ATOMIC); tduffy> + if (!ep_ptr->cxn_timer) { tduffy> + dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, tduffy> + DAT_RESOURCE_MEMORY); tduffy> + goto bail; tduffy> + } tduffy> tduffy> dapl_timer_set(ep_ptr->cxn_timer, tduffy> dapl_ep_timeout, ep_ptr, timeout); tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_ep_modify.c linux-kernel-kmalloc/dat-provider/dapl_ep_modify.c tduffy> --- linux-kernel-sizeof/dat-provider/dapl_ep_modify.c 2005-05-20 15:13:44.645961000 -0700 tduffy> +++ linux-kernel-kmalloc/dat-provider/dapl_ep_modify.c 2005-05-20 15:19:47.835961000 -0700 tduffy> @@ -386,7 +386,7 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand tduffy> alloc_ep.send_iov_num = ep_attr1.max_request_iov; tduffy> alloc_ep.send_iov = kmalloc(ep_attr1.max_request_iov tduffy> * sizeof *alloc_ep.send_iov, GFP_ATOMIC); tduffy> - if (alloc_ep.recv_iov == NULL) { tduffy> + if (!alloc_ep.recv_iov) { tduffy> dat_status = tduffy> DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> goto bail; tduffy> @@ -396,7 +396,7 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand tduffy> alloc_ep.recv_iov_num = ep_attr1.max_recv_iov; tduffy> alloc_ep.recv_iov = kmalloc(ep_attr1.max_recv_iov tduffy> * sizeof *alloc_ep.recv_iov, GFP_ATOMIC); tduffy> - if (alloc_ep.recv_iov == NULL) { tduffy> + if (!alloc_ep.recv_iov) { tduffy> dat_status = tduffy> DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> goto bail; tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_ep_util.c linux-kernel-kmalloc/dat-provider/dapl_ep_util.c tduffy> --- linux-kernel-sizeof/dat-provider/dapl_ep_util.c 2005-05-20 15:14:06.429964000 -0700 tduffy> +++ linux-kernel-kmalloc/dat-provider/dapl_ep_util.c 2005-05-20 15:19:47.840972000 -0700 tduffy> @@ -60,9 +60,8 @@ DAPL_EP *dapl_ep_alloc(DAPL_IA * ia_ptr, tduffy> DAPL_EP *ep_ptr; tduffy> tduffy> /* Allocate EP */ tduffy> - ep_ptr = (DAPL_EP *) kmalloc(sizeof *ep_ptr + sizeof (struct sockaddr), tduffy> - GFP_ATOMIC); tduffy> - if (ep_ptr == NULL) tduffy> + ep_ptr = kmalloc(sizeof *ep_ptr + sizeof (struct sockaddr), GFP_ATOMIC); tduffy> + if (!ep_ptr) tduffy> goto bail; tduffy> tduffy> /* zero the structure */ tduffy> @@ -126,8 +125,7 @@ DAPL_EP *dapl_ep_alloc(DAPL_IA * ia_ptr, tduffy> tduffy> ep_ptr->recv_iov = kmalloc(ep_ptr->recv_iov_num * tduffy> sizeof *ep_ptr->recv_iov, GFP_ATOMIC); tduffy> - tduffy> - if (NULL == ep_ptr->recv_iov) { tduffy> + if (!ep_ptr->recv_iov) { tduffy> dapl_ep_dealloc(ep_ptr); tduffy> ep_ptr = NULL; tduffy> goto bail; tduffy> @@ -135,8 +133,7 @@ DAPL_EP *dapl_ep_alloc(DAPL_IA * ia_ptr, tduffy> tduffy> ep_ptr->send_iov = kmalloc(ep_ptr->send_iov_num * tduffy> sizeof *ep_ptr->send_iov, GFP_ATOMIC); tduffy> - tduffy> - if (NULL == ep_ptr->send_iov) { tduffy> + if (!ep_ptr->send_iov) { tduffy> dapl_ep_dealloc(ep_ptr); tduffy> ep_ptr = NULL; tduffy> goto bail; tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_evd_util.c linux-kernel-kmalloc/dat-provider/dapl_evd_util.c tduffy> --- linux-kernel-sizeof/dat-provider/dapl_evd_util.c 2005-05-20 15:15:32.618966000 -0700 tduffy> +++ linux-kernel-kmalloc/dat-provider/dapl_evd_util.c 2005-05-20 15:19:47.867963000 -0700 tduffy> @@ -123,10 +123,9 @@ DAPL_EVD *dapl_evd_alloc(DAPL_IA * ia_pt tduffy> DAPL_EVD *evd_ptr; tduffy> tduffy> /* Allocate EVD */ tduffy> - evd_ptr = (DAPL_EVD *) kmalloc(sizeof *evd_ptr, GFP_ATOMIC); tduffy> - if (!evd_ptr) { tduffy> + evd_ptr = kmalloc(sizeof *evd_ptr, GFP_ATOMIC); tduffy> + if (!evd_ptr) tduffy> goto bail; tduffy> - } tduffy> tduffy> /* zero the structure */ tduffy> memset(evd_ptr, 0, sizeof *evd_ptr); tduffy> @@ -169,10 +168,8 @@ u32 dapl_evd_event_alloc(DAPL_EVD *evd_p tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> /* Allocate EVENTs */ tduffy> - event_ptr = tduffy> - (struct dat_event *)kmalloc(evd_ptr->qlen * sizeof *event_ptr, tduffy> - GFP_ATOMIC); tduffy> - if (event_ptr == NULL) { tduffy> + event_ptr = kmalloc(evd_ptr->qlen * sizeof *event_ptr, GFP_ATOMIC); tduffy> + if (!event_ptr) { tduffy> dat_status = tduffy> DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> goto bail; tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_hash.c linux-kernel-kmalloc/dat-provider/dapl_hash.c tduffy> --- linux-kernel-sizeof/dat-provider/dapl_hash.c 2005-05-20 10:10:06.017003000 -0700 tduffy> +++ linux-kernel-kmalloc/dat-provider/dapl_hash.c 2005-05-20 15:19:47.873960000 -0700 tduffy> @@ -213,8 +213,8 @@ dapl_hash_add(DAPL_HASH_TABLEP p_table, tduffy> p_table->num_entries++; tduffy> status = TRUE; tduffy> } else { tduffy> - DAPL_HASH_ELEM *newelement = (DAPL_HASH_ELEM *) tduffy> - kmalloc(sizeof *newelement, GFP_ATOMIC); tduffy> + DAPL_HASH_ELEM *newelement = kmalloc(sizeof *newelement, tduffy> + GFP_ATOMIC); tduffy> /* tduffy> * Add an element to the end of the chain tduffy> */ tduffy> @@ -232,10 +232,8 @@ dapl_hash_add(DAPL_HASH_TABLEP p_table, tduffy> lastelement->next_element = newelement; tduffy> p_table->num_entries++; tduffy> status = TRUE; tduffy> - } else { tduffy> - /* allocation failed - should not happen */ tduffy> + } else tduffy> status = FALSE; tduffy> - } tduffy> } tduffy> tduffy> /* tduffy> @@ -332,7 +330,7 @@ u32 dapl_hash_create(int table_size, DAP tduffy> tduffy> /* Allocate hash table */ tduffy> p_table = kmalloc(sizeof *p_table, GFP_ATOMIC); tduffy> - if (NULL == p_table) { tduffy> + if (!p_table) { tduffy> dat_status = tduffy> DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> goto bail; tduffy> @@ -341,8 +339,8 @@ u32 dapl_hash_create(int table_size, DAP tduffy> /* Init hash table, allocate and init and buckets */ tduffy> memset(p_table, 0, sizeof *p_table); tduffy> p_table->tbl_size = table_size; tduffy> - p_table->table = (DAPL_HASH_ELEM *) kmalloc(table_length, GFP_ATOMIC); tduffy> - if (NULL == p_table->table) { tduffy> + p_table->table = kmalloc(table_length, GFP_ATOMIC); tduffy> + if (!p_table->table) { tduffy> kfree(p_table); tduffy> dat_status = tduffy> DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_hca_util.c linux-kernel-kmalloc/dat-provider/dapl_hca_util.c tduffy> --- linux-kernel-sizeof/dat-provider/dapl_hca_util.c 2005-05-20 15:11:31.050974000 -0700 tduffy> +++ linux-kernel-kmalloc/dat-provider/dapl_hca_util.c 2005-05-20 15:19:47.878970000 -0700 tduffy> @@ -61,7 +61,7 @@ DAPL_HCA *dapl_hca_alloc(char *name, str tduffy> DAPL_HCA *hca_ptr; tduffy> tduffy> hca_ptr = kmalloc(sizeof *hca_ptr, GFP_ATOMIC); tduffy> - if (NULL != hca_ptr) { tduffy> + if (hca_ptr) { tduffy> memset(hca_ptr, 0, sizeof *hca_ptr); tduffy> tduffy> if (DAT_SUCCESS == tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_ia_util.c linux-kernel-kmalloc/dat-provider/dapl_ia_util.c tduffy> --- linux-kernel-sizeof/dat-provider/dapl_ia_util.c 2005-05-20 15:11:31.438968000 -0700 tduffy> +++ linux-kernel-kmalloc/dat-provider/dapl_ia_util.c 2005-05-20 15:19:47.884961000 -0700 tduffy> @@ -63,10 +63,9 @@ DAPL_IA *dapl_ia_alloc(struct dat_provid tduffy> DAPL_IA *ia_ptr; tduffy> tduffy> /* Allocate IA */ tduffy> - ia_ptr = (DAPL_IA *) kmalloc(sizeof *ia_ptr, GFP_ATOMIC); tduffy> - if (ia_ptr == NULL) { tduffy> + ia_ptr = kmalloc(sizeof *ia_ptr, GFP_ATOMIC); tduffy> + if (!ia_ptr) tduffy> return NULL; tduffy> - } tduffy> tduffy> /* zero the structure */ tduffy> memset(ia_ptr, 0, sizeof *ia_ptr); tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_lmr_util.c linux-kernel-kmalloc/dat-provider/dapl_lmr_util.c tduffy> --- linux-kernel-sizeof/dat-provider/dapl_lmr_util.c 2005-05-20 09:45:43.253013000 -0700 tduffy> +++ linux-kernel-kmalloc/dat-provider/dapl_lmr_util.c 2005-05-20 15:19:47.890970000 -0700 tduffy> @@ -48,10 +48,9 @@ DAPL_LMR *dapl_lmr_alloc(DAPL_IA * ia, tduffy> DAPL_LMR *lmr; tduffy> tduffy> /* Allocate LMR */ tduffy> - lmr = (DAPL_LMR *) kmalloc(sizeof *lmr, GFP_ATOMIC); tduffy> - if (NULL == lmr) { tduffy> + lmr = kmalloc(sizeof *lmr, GFP_ATOMIC); tduffy> + if (!lmr) tduffy> return NULL; tduffy> - } tduffy> tduffy> /* zero the structure */ tduffy> memset(lmr, 0, sizeof *lmr); tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_openib_util.c linux-kernel-kmalloc/dat-provider/dapl_openib_util.c tduffy> --- linux-kernel-sizeof/dat-provider/dapl_openib_util.c 2005-05-20 15:11:31.746962000 -0700 tduffy> +++ linux-kernel-kmalloc/dat-provider/dapl_openib_util.c 2005-05-20 15:19:47.920964000 -0700 tduffy> @@ -408,9 +408,8 @@ u32 dapl_ib_mr_register_physical(DAPL_IA tduffy> u64 *array; tduffy> tduffy> array = (u64 *) phys_addr; tduffy> - buf_list = (struct ib_phys_buf *) tduffy> - kmalloc(length * sizeof *buf_list, GFP_ATOMIC); tduffy> - if (NULL == buf_list) tduffy> + buf_list = kmalloc(length * sizeof *buf_list, GFP_ATOMIC); tduffy> + if (!buf_list) tduffy> return DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, tduffy> DAT_RESOURCE_MEMORY); tduffy> tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_provider.c linux-kernel-kmalloc/dat-provider/dapl_provider.c tduffy> --- linux-kernel-sizeof/dat-provider/dapl_provider.c 2005-05-20 09:53:27.437003000 -0700 tduffy> +++ linux-kernel-kmalloc/dat-provider/dapl_provider.c 2005-05-20 15:19:47.926965000 -0700 tduffy> @@ -151,7 +151,7 @@ u32 dapl_provider_list_create(void) tduffy> /* create the head node */ tduffy> g_dapl_provider_list.head = tduffy> kmalloc(sizeof *g_dapl_provider_list.head, GFP_ATOMIC); tduffy> - if (NULL == g_dapl_provider_list.head) { tduffy> + if (!g_dapl_provider_list.head) { tduffy> status = tduffy> DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> goto bail; tduffy> @@ -162,7 +162,7 @@ u32 dapl_provider_list_create(void) tduffy> /* create the tail node */ tduffy> g_dapl_provider_list.tail = tduffy> kmalloc(sizeof *g_dapl_provider_list.tail, GFP_ATOMIC); tduffy> - if (NULL == g_dapl_provider_list.tail) { tduffy> + if (!g_dapl_provider_list.tail) { tduffy> status = tduffy> DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> goto bail; tduffy> @@ -214,8 +214,7 @@ u32 dapl_provider_list_insert(const char tduffy> unsigned int len; tduffy> tduffy> cur_node = kmalloc(sizeof *cur_node, GFP_ATOMIC); tduffy> - tduffy> - if (NULL == cur_node) { tduffy> + if (!cur_node) { tduffy> status = tduffy> DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> goto bail; tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_pz_util.c linux-kernel-kmalloc/dat-provider/dapl_pz_util.c tduffy> --- linux-kernel-sizeof/dat-provider/dapl_pz_util.c 2005-05-20 15:11:31.492027000 -0700 tduffy> +++ linux-kernel-kmalloc/dat-provider/dapl_pz_util.c 2005-05-20 15:23:13.954961000 -0700 tduffy> @@ -57,8 +57,8 @@ DAPL_PZ *dapl_pz_alloc(DAPL_IA * ia) tduffy> DAPL_PZ *pz; tduffy> tduffy> /* Allocate PZ */ tduffy> - pz = (DAPL_PZ *) kmalloc(sizeof *pz, GFP_ATOMIC); tduffy> - if (NULL == pz) tduffy> + pz = kmalloc(sizeof *pz, GFP_ATOMIC); tduffy> + if (!pz) tduffy> return NULL; tduffy> tduffy> /* zero the structure */ tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_ring_buffer_util.c linux-kernel-kmalloc/dat-provider/dapl_ring_buffer_util.c tduffy> --- linux-kernel-sizeof/dat-provider/dapl_ring_buffer_util.c 2005-05-20 15:11:30.864961000 -0700 tduffy> +++ linux-kernel-kmalloc/dat-provider/dapl_ring_buffer_util.c 2005-05-20 15:19:47.959972000 -0700 tduffy> @@ -72,14 +72,13 @@ u32 dapl_rbuf_alloc(DAPL_RING_BUFFER *rb tduffy> rsize <<= 1; tduffy> } tduffy> tduffy> - rbuf->base = (void *) kmalloc(rsize * sizeof *rbuf->base, GFP_ATOMIC); tduffy> - if (rbuf->base != NULL) { tduffy> + rbuf->base = kmalloc(rsize * sizeof *rbuf->base, GFP_ATOMIC); tduffy> + if (rbuf->base) { tduffy> rbuf->lim = rsize - 1; tduffy> atomic_set(&rbuf->head, 0); tduffy> atomic_set(&rbuf->tail, 0); tduffy> - } else { tduffy> + } else tduffy> return DAT_INSUFFICIENT_RESOURCES | DAT_RESOURCE_MEMORY; tduffy> - } tduffy> tduffy> return DAT_SUCCESS; tduffy> } tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_rmr_util.c linux-kernel-kmalloc/dat-provider/dapl_rmr_util.c tduffy> --- linux-kernel-sizeof/dat-provider/dapl_rmr_util.c 2005-05-20 09:54:37.797008000 -0700 tduffy> +++ linux-kernel-kmalloc/dat-provider/dapl_rmr_util.c 2005-05-20 15:19:47.964962000 -0700 tduffy> @@ -33,10 +33,9 @@ DAPL_RMR *dapl_rmr_alloc(DAPL_PZ * pz) tduffy> DAPL_RMR *rmr; tduffy> tduffy> /* Allocate LMR */ tduffy> - rmr = (DAPL_RMR *) kmalloc(sizeof *rmr, GFP_ATOMIC); tduffy> - if (NULL == rmr) { tduffy> + rmr = kmalloc(sizeof *rmr, GFP_ATOMIC); tduffy> + if (!rmr) tduffy> return NULL; tduffy> - } tduffy> tduffy> /* zero the structure */ tduffy> memset(rmr, 0, sizeof *rmr); tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_sp_util.c linux-kernel-kmalloc/dat-provider/dapl_sp_util.c tduffy> --- linux-kernel-sizeof/dat-provider/dapl_sp_util.c 2005-05-20 15:11:32.615964000 -0700 tduffy> +++ linux-kernel-kmalloc/dat-provider/dapl_sp_util.c 2005-05-20 15:19:47.969972000 -0700 tduffy> @@ -64,8 +64,8 @@ DAPL_SP *dapl_sp_alloc(DAPL_IA * ia_ptr, tduffy> DAPL_SP *sp_ptr; tduffy> tduffy> /* Allocate EP */ tduffy> - sp_ptr = (DAPL_SP *) kmalloc(sizeof *sp_ptr, GFP_ATOMIC); tduffy> - if (sp_ptr == NULL) tduffy> + sp_ptr = kmalloc(sizeof *sp_ptr, GFP_ATOMIC); tduffy> + if (!sp_ptr) tduffy> return NULL; tduffy> tduffy> /* zero the structure */ tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_srq_util.c linux-kernel-kmalloc/dat-provider/dapl_srq_util.c tduffy> --- linux-kernel-sizeof/dat-provider/dapl_srq_util.c 2005-05-20 09:55:16.196002000 -0700 tduffy> +++ linux-kernel-kmalloc/dat-provider/dapl_srq_util.c 2005-05-20 15:19:47.973961000 -0700 tduffy> @@ -59,11 +59,9 @@ DAPL_SRQ *dapl_srq_alloc(DAPL_IA * ia_pt tduffy> DAPL_SRQ *srq_ptr; tduffy> tduffy> /* Allocate SRQ */ tduffy> - srq_ptr = (DAPL_SRQ *) kmalloc(sizeof *srq_ptr, GFP_ATOMIC); tduffy> - tduffy> - if (srq_ptr == NULL) { tduffy> + srq_ptr = kmalloc(sizeof *srq_ptr, GFP_ATOMIC); tduffy> + if (!srq_ptr) tduffy> goto bail; tduffy> - } tduffy> tduffy> /* zero the structure */ tduffy> memset(srq_ptr, 0, sizeof *srq_ptr); tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel-sizeof/dat-provider/dapl_util.h linux-kernel-kmalloc/dat-provider/dapl_util.h tduffy> --- linux-kernel-sizeof/dat-provider/dapl_util.h 2005-05-20 15:11:32.661977000 -0700 tduffy> +++ linux-kernel-kmalloc/dat-provider/dapl_util.h 2005-05-20 15:19:47.999965000 -0700 tduffy> @@ -144,7 +144,7 @@ static inline void *dapl_os_realloc(void tduffy> void *newptr; tduffy> tduffy> newptr = kmalloc(size, GFP_ATOMIC); tduffy> - if (newptr != NULL) { tduffy> + if (newptr) { tduffy> /* copy the mem array to the new */ tduffy> memcpy(newptr, ptr, size); tduffy> kfree(ptr); tduffy> @@ -159,9 +159,8 @@ static inline void *dapl_os_realloc(void tduffy> static inline char *dapl_os_strdup(const char *str) tduffy> { tduffy> char *ns = kmalloc(strlen(str) + 1, GFP_KERNEL); tduffy> - if (ns) { tduffy> + if (ns) tduffy> strcpy(ns, str); tduffy> - } tduffy> return ns; tduffy> } tduffy> tduffy> From berny.f at aon.at Fri May 20 16:56:42 2005 From: berny.f at aon.at (Bernhard Fischer) Date: Sat, 21 May 2005 01:56:42 +0200 Subject: [openib-general] Re: [openib-commits] r2426 - gen2/users/jlentini/linux-kernel/dat-provider In-Reply-To: <20050520220508.CB9A12283D9@openib.ca.sandia.gov> References: <20050520220508.CB9A12283D9@openib.ca.sandia.gov> Message-ID: <20050520235642.GE15688@aon.at> On Fri, May 20, 2005 at 03:05:08PM -0700, jlentini at openib.org wrote: >Author: jlentini >Date: 2005-05-20 15:05:07 -0700 (Fri, 20 May 2005) >New Revision: 2426 Just tersely parsing, no mind to delve into any API implications whatsoever. [Late, no motivation to think, nor even read, not even speaking of writing anything publically. I'm over quota with my telephone-IP provider since 11th, which nag's our scanty pocket-money crucially, etc] >Modified: gen2/users/jlentini/linux-kernel/dat-provider/dapl.h >=================================================================== >--- gen2/users/jlentini/linux-kernel/dat-provider/dapl.h 2005-05-20 17:22:17 UTC (rev 2425) >+++ gen2/users/jlentini/linux-kernel/dat-provider/dapl.h 2005-05-20 22:05:07 UTC (rev 2426) >@@ -144,6 +140,24 @@ > * * > *********************************************************************/ > >+typedef void (*ib_async_handler_t) (struct ib_event *, void *); >+ >+/* >+ * ib_hca_transport_t structure. Define fields specific to this >+ * provider implementation necessary to maintain the HCA >+ * >+ * NOTE: null_ib_hca_handle is strictly internal to IB. DAPL allows a >+ * NULL DTO EVD handle, but IB does not. Create a CQ under the >+ * hood and make sure an error is generated if the user every "every tries to post"? -enoparse; please be more specific -- simpler. 'ever' would still be obfuscated. IOW please rephrase and put abbreviations into the wiki. >+ * tries to post, by setting the WQ length to 0 in ep_create >+ * and/or ep_modify. >+ */ >@@ -394,7 +403,7 @@ > > /* maintenence fields */ maintenance. also in: users/jlentini/userspace/dapl/include/dapl.h > boolean_t listening; /* PSP is registered & active */ >- ib_cm_srvc_handle_t cm_srvc_handle; /* Used by CM */ >+ struct ib_cm_id *cm_srvc_handle; /* Used by CM */ > DAPL_LLIST_HEAD cr_list_head; /* CR pending queue */ > int cr_list_count; /* count of CRs on queue */ > }; >--- gen2/users/jlentini/linux-kernel/dat-provider/dapl_evd_util.c 2005-05-20 17:22:17 UTC (rev 2425) >+++ gen2/users/jlentini/linux-kernel/dat-provider/dapl_evd_util.c 2005-05-20 22:05:07 UTC (rev 2426) >@@ -331,22 +322,22 @@ > dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK, > "\t dapl_evd_dto_callback : CQE \n"); > dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK, >- "\t\t work_req_id %lli\n", DAPL_GET_CQE_WRID(cqe_ptr)); >- if (DAPL_GET_CQE_STATUS(cqe_ptr) == 0) { >- if (DAPL_GET_CQE_OPTYPE(cqe_ptr) == OP_RECEIVE) { >+ "\t\t work_req_id %lli\n", cqe_ptr->wr_id); [[:space:]] --------------^ ? >+ if (cqe_ptr->status == 0) { >+ if (cqe_ptr->opcode == IB_WC_RECV) { > dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK, >- "\t\t op_type: OP_RECEIVE\n"); >+ "\t\t op_type: IB_WC_RECV\n"); ditto > dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK, > "\t\t bytes_num %d\n", ditto >- DAPL_GET_CQE_BYTESNUM(cqe_ptr)); >+ cqe_ptr->byte_len); > } else { > dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK, > "\t\t op_type: %s\n", ditto >- optable[DAPL_GET_CQE_OPTYPE(cqe_ptr)]); >+ optable[cqe_ptr->opcode]); > } > } > dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK, >- "\t\t status %d\n", DAPL_GET_CQE_STATUS(cqe_ptr)); >+ "\t\t status %d\n", cqe_ptr->status); ditto > dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK, > "\t >>>>>>>>>>>>>>>>>>>>>>><<<<<<<<<<<<<<<<<<<\n"); fooooooooooooooooooooooooooooooooooooooooooobar! merge errors? also in: gen2/branches/shaharf-ibat/src/userspace/management/libibmad/src/rpc.c gen2/ulps/iser/make.conf gen2/users/jlentini/linux-kernel/dat-provider/dapl_evd_util.c (multiple times) gen2/users/jlentini/linux-kernel/dat-provider/dapl_rsp_free.c (multiple gen2/users/jlentini/linux-kernel/dat-provider/dapl_rsp_create.c times) gen2/users/jlentini/linux-kernel/dat-provider/dapl_openib_cm.c (multiple times) gen2/users/jlentini/userspace/dapl/common/dapl_evd_util.c (ditto) gen2/users/jlentini/userspace/dapl/common/dapl_rsp_free.c (ditto) gen2/users/jlentini/userspace/dapl/common/dapl_rsp_create.c (ditto) gen2/trunk/src/userspace/management/libibmad/src/rpc.c > #endif >@@ -667,21 +657,21 @@ > > #ifdef DAPL_DBG > /* Current gen2 mthca is not setting the opcode in seccesful cqe */ >- /* The opcode will be OP_SEND or OP_RECEIVE acording the is_send bit */ >+ /* The opcode will be IB_WC_SEND or IB_WC_RECV acording the is_send bit */ "acording" -enoparse: s/acord/accord/g also in: gen2/branches/shaharf-ibat/src/userspace/management/osm/opensm/osm_state_mgr.c gen2/branches/roland-uverbs/src/userspace/management/osm/opensm/osm_sa_mcmember_record.c gen2/utils/src/linux-user/ibdm/datamodel/ibdm.i gen2/utils/src/linux-user/IBMgtSim/src/ibdm.i gen2/utils/src/linux-user/IBMgtSim/utils/RunSimTest there: callabcks: -enoparse; continue with acord: gen2/users/jlentini/linux-kernel/dat-provider/dapl_evd_util.c gen2/trunk/src/userspace/management/osm/opensm/osm_state_mgr.c "calla" -enoparse: s/calla/call gen2/utils/src/linux-user/IBMgtSim/utils/RunSimTest gen2/ulps/iser/include/iser_api.h gen2/trunk/src/userspace/management/osm/libvendor/osm_vendor_mlx_ts.c gen2/trunk/src/userspace/management/osm/libvendor/osm_vendor_ts.c gen2/trunk/src/userspace/management/osm/libvendor/osm_vendor_mlx_ts_anafa.c [If needed (i.e. if i'm aware of any request to do so) i'll expand those..] > /* We can not check the following assert for now */ [skip] >@@ -184,19 +184,8 @@ > } > } > /* force the EP into error state to force flush all posted DTOs. */ >- { >- struct dat_ep_attr ep_attr; >- struct dat_named_attr ep_state; >+ (void) dapl_modify_qp_state_to_error(ep_ptr->qp_handle); sun will hopefuly fix all of egrep -ri "(^[[:space:]]*\(void\))" openib.gen2/upstream/gen2/trunk/|egrep -v "svn-(text|base)" so i won't comment on that single occurance above ;) thank you, From tduffy at sun.com Fri May 20 17:09:48 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 20 May 2005 17:09:48 -0700 Subject: [openib-general] Re: [openib-commits] r2426 - gen2/users/jlentini/linux-kernel/dat-provider In-Reply-To: <20050520235642.GE15688@aon.at> References: <20050520220508.CB9A12283D9@openib.ca.sandia.gov> <20050520235642.GE15688@aon.at> Message-ID: <1116634188.16690.6.camel@duffman> On Sat, 2005-05-21 at 01:56 +0200, Bernhard Fischer wrote: > sun will hopefuly fix all of -enoparse hopefuly s/hopefuly/hopefully/ ;-) > egrep -ri "(^[[:space:]]*\(void\))" As we discussed earlier, it is not wrong to cast the return to (void) if indeed you are ignoring a non-void return. If, in fact, the function returns void, then casting is dumb. Also, if the function doesn't return anything useful and you always disregard the return, the function should be changes to return void. > openib.gen2/upstream/gen2/trunk/|egrep -v "svn-(text|base)" > so i won't comment on that single occurance above ;) -enoparse occurance s/occurance/occurrence/ Later, sun -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From tduffy at sun.com Fri May 20 17:34:30 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 20 May 2005 17:34:30 -0700 Subject: [openib-general] [PATCH] kDAPL: remove use of typedefs for structs in dapl.h Message-ID: <1116635670.16690.11.camel@duffman> James, I am sorry this is such a large patch. I had broken it out into 19 different patches, each changing one typedef, but after the large changes you made today, it was going to be a big PITA to regenerate them all. So, I just rebased my final tree with r2431 and generated one large patch. This patch removes the use of many of the struct typedefs in dapl.h. Signed-off-by: Tom Duffy Index: linux-kernel-resolv/dat-provider/dapl_evd_cq_async_error_callb.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_evd_cq_async_error_callb.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_evd_cq_async_error_callb.c (working copy) @@ -53,22 +53,19 @@ * */ -void -dapl_evd_cq_async_error_callback(struct ib_event *cause_ptr, void *context) +void dapl_evd_cq_async_error_callback(struct ib_event *cause_ptr, void *context) { - DAPL_EVD *async_evd; - DAPL_EVD *evd; + struct dapl_evd *async_evd; + struct dapl_evd *evd = context; u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK | DAPL_DBG_TYPE_EXCEPTION, "dapl_evd_cq_async_error_callback (%p, %p)\n", cause_ptr, context); - if (NULL == context) { + if (!evd) dapl_os_panic("NULL == context\n"); - } - evd = (DAPL_EVD *) context; async_evd = evd->header.owner_ia->async_error_evd; dat_status = dapl_evd_post_async_error_event(async_evd, Index: linux-kernel-resolv/dat-provider/dapl_cookie.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_cookie.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_cookie.c (working copy) @@ -84,9 +84,11 @@ * * *********************************************************************/ -u32 dapl_cb_get(DAPL_COOKIE_BUFFER *buffer, DAPL_COOKIE **cookie_ptr); +u32 dapl_cb_get(struct dapl_cookie_buffer *buffer, + struct dapl_cookie **cookie_ptr); -u32 dapl_cb_put(DAPL_COOKIE_BUFFER *buffer, DAPL_COOKIE *cookie); +u32 dapl_cb_put(struct dapl_cookie_buffer *buffer, + struct dapl_cookie *cookie); /********************************************************************* * * @@ -97,11 +99,11 @@ u32 dapl_cb_put(DAPL_COOKIE_BUFFER *buff /* * dapl_cb_create * - * Given a DAPL_COOKIE_BUFFER, allocate and initialize memory for + * Given a struct dapl_cookie_buffer, allocate and initialize memory for * the data structure. * * Input: - * buffer pointer to DAPL_COOKIE_BUFFER + * buffer pointer to struct dapl_cookie_buffer * ep endpoint to associate with cookies * size number of elements to allocate & manage * @@ -113,7 +115,8 @@ u32 dapl_cb_put(DAPL_COOKIE_BUFFER *buff * DAT_INSUFFICIENT_RESOURCES * */ -u32 dapl_cb_create(DAPL_COOKIE_BUFFER *buffer, DAPL_EP *ep, int size) +u32 dapl_cb_create(struct dapl_cookie_buffer *buffer, struct dapl_ep *ep, + int size) { int i; @@ -145,7 +148,7 @@ u32 dapl_cb_create(DAPL_COOKIE_BUFFER *b * Free the data structure * * Input: - * buffer pointer to DAPL_COOKIE_BUFFER + * buffer pointer to struct dapl_cookie_buffer * * Output: * none @@ -154,7 +157,7 @@ u32 dapl_cb_create(DAPL_COOKIE_BUFFER *b * none * */ -void dapl_cb_free(DAPL_COOKIE_BUFFER * buffer) +void dapl_cb_free(struct dapl_cookie_buffer *buffer) { if (NULL != buffer->pool) kfree(buffer->pool); @@ -166,7 +169,7 @@ void dapl_cb_free(DAPL_COOKIE_BUFFER * b * Remove an entry from the buffer * * Input: - * buffer pointer to DAPL_COOKIE_BUFFER + * buffer pointer to struct dapl_cookie_buffer * * Output: * cookie_ptr pointer to pointer to cookie @@ -177,7 +180,8 @@ void dapl_cb_free(DAPL_COOKIE_BUFFER * b * DAT_INSUFFICIENT_RESOURCES * */ -u32 dapl_cb_get(DAPL_COOKIE_BUFFER *buffer, DAPL_COOKIE **cookie_ptr) +u32 dapl_cb_get(struct dapl_cookie_buffer *buffer, + struct dapl_cookie **cookie_ptr) { u32 dat_status; int new_head; @@ -195,7 +199,7 @@ u32 dapl_cb_get(DAPL_COOKIE_BUFFER *buff *cookie_ptr = &buffer->pool[atomic_read(&buffer->head)]; dat_status = DAT_SUCCESS; } - bail: +bail: return dat_status; } @@ -205,7 +209,7 @@ u32 dapl_cb_get(DAPL_COOKIE_BUFFER *buff * Add entry(s) to the buffer * * Input: - * buffer pointer to DAPL_COOKIE_BUFFER + * buffer pointer to struct dapl_cookie_buffer * cookie pointer to cookie * * Output: @@ -216,7 +220,7 @@ u32 dapl_cb_get(DAPL_COOKIE_BUFFER *buff * DAT_INSUFFICIENT_EMPTY * */ -u32 dapl_cb_put(DAPL_COOKIE_BUFFER *buffer, DAPL_COOKIE *cookie) +u32 dapl_cb_put(struct dapl_cookie_buffer *buffer, struct dapl_cookie *cookie) { atomic_set(&buffer->tail, cookie->index); @@ -229,7 +233,7 @@ u32 dapl_cb_put(DAPL_COOKIE_BUFFER *buff * Allocate an RMR Bind cookie * * Input: - * buffer pointer to DAPL_COOKIE_BUFFER + * buffer pointer to struct dapl_cookie_buffer * rmr rmr to associate with the cookie * user_cookie user's cookie data * @@ -241,10 +245,11 @@ u32 dapl_cb_put(DAPL_COOKIE_BUFFER *buff * DAT_INSUFFICIENT_EMPTY * */ -u32 dapl_rmr_cookie_alloc(DAPL_COOKIE_BUFFER *buffer, DAPL_RMR *rmr, - DAT_RMR_COOKIE user_cookie, DAPL_COOKIE **cookie_ptr) +u32 dapl_rmr_cookie_alloc(struct dapl_cookie_buffer *buffer, + struct dapl_rmr *rmr, DAT_RMR_COOKIE user_cookie, + struct dapl_cookie **cookie_ptr) { - DAPL_COOKIE *cookie; + struct dapl_cookie *cookie; u32 dat_status; if (DAT_SUCCESS != dapl_cb_get(buffer, &cookie)) { @@ -261,7 +266,7 @@ u32 dapl_rmr_cookie_alloc(DAPL_COOKIE_BU *cookie_ptr = cookie; - bail: +bail: return dat_status; } @@ -271,7 +276,7 @@ u32 dapl_rmr_cookie_alloc(DAPL_COOKIE_BU * Allocate a DTO cookie * * Input: - * buffer pointer to DAPL_COOKIE_BUFFER + * buffer pointer to struct dapl_cookie_buffer * type DTO type * user_cookie user's cookie data * @@ -283,10 +288,10 @@ u32 dapl_rmr_cookie_alloc(DAPL_COOKIE_BU * DAT_INSUFFICIENT_EMPTY * */ -u32 dapl_dto_cookie_alloc(DAPL_COOKIE_BUFFER *buffer, DAPL_DTO_TYPE type, - DAT_DTO_COOKIE user_cookie, DAPL_COOKIE **cookie_ptr) +u32 dapl_dto_cookie_alloc(struct dapl_cookie_buffer *buffer, DAPL_DTO_TYPE type, + DAT_DTO_COOKIE user_cookie, struct dapl_cookie **cookie_ptr) { - DAPL_COOKIE *cookie; + struct dapl_cookie *cookie; if (DAT_SUCCESS != dapl_cb_get(buffer, &cookie)) { *cookie_ptr = NULL; @@ -303,7 +308,8 @@ u32 dapl_dto_cookie_alloc(DAPL_COOKIE_BU return DAT_SUCCESS; } -void dapl_cookie_dealloc(DAPL_COOKIE_BUFFER * buffer, DAPL_COOKIE * cookie) +void dapl_cookie_dealloc(struct dapl_cookie_buffer *buffer, + struct dapl_cookie * cookie) { dapl_cb_put(buffer, cookie); } Index: linux-kernel-resolv/dat-provider/dapl_ia_memtype_hint.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_ia_memtype_hint.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_ia_memtype_hint.c (working copy) @@ -69,13 +69,13 @@ u32 dapl_ia_memtype_hint(DAT_IA_HANDLE i u64 *suggested_length, u64 *suggested_alignment) { - DAPL_IA *ia_ptr; + struct dapl_ia *ia_ptr; u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ia_memtype_hint (%p)\n", ia_handle); - ia_ptr = (DAPL_IA *) ia_handle; + ia_ptr = (struct dapl_ia *)ia_handle; if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { dat_status = Index: linux-kernel-resolv/dat-provider/dapl_ep_connect.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_ep_connect.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_ep_connect.c (working copy) @@ -48,8 +48,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han int private_data_size, const void *private_data, enum dat_qos qos, enum dat_connect_flags connect_flags) { - DAPL_EP *ep_ptr; - DAPL_EP alloc_ep; + struct dapl_ep *ep_ptr, alloc_ep; u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API | DAPL_DBG_TYPE_CM, @@ -63,7 +62,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han timeout, private_data_size, private_data, qos, connect_flags); - ep_ptr = (DAPL_EP *) ep_handle; + ep_ptr = (struct dapl_ep *)ep_handle; /* * Verify parameter & state. The connection handle must be good @@ -174,7 +173,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han /* test connect handle */ || ep_ptr->param.connect_evd_handle == NULL || DAPL_BAD_HANDLE(ep_ptr->param.connect_evd_handle, DAPL_MAGIC_EVD) - || !(((DAPL_EVD *) ep_ptr->param.connect_evd_handle)-> + || !(((struct dapl_evd *)ep_ptr->param.connect_evd_handle)-> evd_flags & DAT_EVD_CONNECTION_FLAG) /* test optional completion handles */ || (ep_ptr->param.recv_evd_handle != DAT_HANDLE_NULL && @@ -261,8 +260,8 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han * this happens, return SUCCESS and generate the event */ if (dat_status == DAT_ERROR(DAT_INTERNAL_ERROR, 1)) { - dapl_evd_post_connection_event((DAPL_EVD *) ep_ptr-> - param. + dapl_evd_post_connection_event((struct dapl_evd *) + ep_ptr->param. connect_evd_handle, DAT_CONNECTION_EVENT_UNREACHABLE, (DAT_HANDLE) ep_ptr, 0, Index: linux-kernel-resolv/dat-provider/dapl_cookie.h =================================================================== --- linux-kernel-resolv/dat-provider/dapl_cookie.h (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_cookie.h (working copy) @@ -36,20 +36,21 @@ #include "dapl.h" -extern u32 dapl_cb_create(DAPL_COOKIE_BUFFER *buffer, DAPL_EP *ep, +extern u32 dapl_cb_create(struct dapl_cookie_buffer *buffer, struct dapl_ep *ep, int size); -extern void dapl_cb_free(DAPL_COOKIE_BUFFER *buffer); +extern void dapl_cb_free(struct dapl_cookie_buffer *buffer); -extern u32 dapl_rmr_cookie_alloc(DAPL_COOKIE_BUFFER *buffer, DAPL_RMR *rmr, +extern u32 dapl_rmr_cookie_alloc(struct dapl_cookie_buffer *buffer, + struct dapl_rmr *rmr, DAT_RMR_COOKIE user_cookie, - DAPL_COOKIE **cookie_ptr); + struct dapl_cookie **cookie_ptr); -extern u32 dapl_dto_cookie_alloc(DAPL_COOKIE_BUFFER *buffer, +extern u32 dapl_dto_cookie_alloc(struct dapl_cookie_buffer *buffer, DAPL_DTO_TYPE type, DAT_DTO_COOKIE user_cookie, - DAPL_COOKIE **cookie_ptr); + struct dapl_cookie **cookie_ptr); -extern void -dapl_cookie_dealloc(DAPL_COOKIE_BUFFER * buffer, DAPL_COOKIE * cookie); +extern void dapl_cookie_dealloc(struct dapl_cookie_buffer *buffer, + struct dapl_cookie *cookie); #endif /* DAPL_COOKIE_H */ Index: linux-kernel-resolv/dat-provider/dapl_cr_callback.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_cr_callback.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_cr_callback.c (working copy) @@ -45,11 +45,13 @@ /* * Prototypes */ -u32 dapl_connection_request(struct dapl_cm_id *ib_cm_handle, DAPL_SP *sp_ptr, - const void *prd_ptr, DAPL_EVD *evd_ptr); - -DAPL_EP *dapl_get_sp_ep(struct dapl_cm_id *ib_cm_handle, DAPL_SP * sp_ptr, - enum dat_event_number event); +u32 dapl_connection_request(struct dapl_cm_id *ib_cm_handle, + struct dapl_sp *sp_ptr, const void *prd_ptr, + struct dapl_evd *evd_ptr); + +struct dapl_ep *dapl_get_sp_ep(struct dapl_cm_id *ib_cm_handle, + struct dapl_sp *sp_ptr, + enum dat_event_number dat_event_num); /* * The callback function registered with verbs for passive side of @@ -60,9 +62,9 @@ void dapl_cr_callback(struct dapl_cm_id const void *private_data_ptr, /* event data */ const void *context) { - DAPL_EP *ep_ptr; - DAPL_EVD *evd_ptr; - DAPL_SP *sp_ptr; + struct dapl_ep *ep_ptr; + struct dapl_evd *evd_ptr; + struct dapl_sp *sp_ptr; u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_CM | DAPL_DBG_TYPE_CALLBACK, @@ -73,7 +75,7 @@ void dapl_cr_callback(struct dapl_cm_id * Passive side of the connection, context is a SP and * we need to look up the EP. */ - sp_ptr = (DAPL_SP *) context; + sp_ptr = (struct dapl_sp *)context; /* * The context pointer could have been cleaned up in a racing * CM callback, check to see if we should just exit here @@ -106,7 +108,7 @@ void dapl_cr_callback(struct dapl_cm_id if (ep_ptr == NULL) { return; } - evd_ptr = (DAPL_EVD *) ep_ptr->param.connect_evd_handle; + evd_ptr = (struct dapl_evd *)ep_ptr->param.connect_evd_handle; /* if something has happened to our EVD, bail. */ if (evd_ptr == NULL) { return; @@ -259,13 +261,14 @@ void dapl_cr_callback(struct dapl_cm_id * Create a CR record and link it on to the SP so we can update it * and free it later. Create an EP if specified by the PSP flags. */ -u32 dapl_connection_request(struct dapl_cm_id *ib_cm_handle, DAPL_SP *sp_ptr, - const void *prd_ptr, DAPL_EVD *evd_ptr) +u32 dapl_connection_request(struct dapl_cm_id *ib_cm_handle, + struct dapl_sp *sp_ptr, const void *prd_ptr, + struct dapl_evd *evd_ptr) { u32 dat_status; - DAPL_CR *cr_ptr; - DAPL_EP *ep_ptr; - DAPL_IA *ia_ptr; + struct dapl_cr *cr_ptr; + struct dapl_ep *ep_ptr; + struct dapl_ia *ia_ptr; DAT_SP_HANDLE sp_handle; cr_ptr = dapl_cr_alloc(sp_ptr->header.owner_ia); @@ -295,7 +298,7 @@ u32 dapl_connection_request(struct dapl_ } /* EP will be NULL unless RSP service point */ - ep_ptr = (DAPL_EP *) sp_ptr->ep_handle; + ep_ptr = (struct dapl_ep *)sp_ptr->ep_handle; if (sp_ptr->psp_flags == DAT_PSP_PROVIDER_FLAG) { /* @@ -362,11 +365,12 @@ u32 dapl_connection_request(struct dapl_ * up resources and obtain the EP pointer associated with a CR in * the SP. */ -DAPL_EP *dapl_get_sp_ep(struct dapl_cm_id *ib_cm_handle, - DAPL_SP * sp_ptr, enum dat_event_number event) +struct dapl_ep *dapl_get_sp_ep(struct dapl_cm_id *ib_cm_handle, + struct dapl_sp *sp_ptr, + enum dat_event_number event) { - DAPL_CR *cr_ptr; - DAPL_EP *ep_ptr; + struct dapl_cr *cr_ptr; + struct dapl_ep *ep_ptr; /* * acquire the lock, we may be racing with other threads here @@ -391,7 +395,7 @@ DAPL_EP *dapl_get_sp_ep(struct dapl_cm_i return NULL; } - ep_ptr = (DAPL_EP *) cr_ptr->param.local_ep_handle; + ep_ptr = (struct dapl_ep *)cr_ptr->param.local_ep_handle; /* Quick check to ensure our EP is still valid */ if ((DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP))) { @@ -427,8 +431,7 @@ DAPL_EP *dapl_get_sp_ep(struct dapl_cm_i sp_ptr); /* Decrement the ref count on the EVD */ if (sp_ptr->evd_handle) { - atomic_dec(& - ((DAPL_EVD *) sp_ptr-> + atomic_dec(&((struct dapl_evd *)sp_ptr-> evd_handle)->evd_ref_count); sp_ptr->evd_handle = NULL; } @@ -437,7 +440,8 @@ DAPL_EP *dapl_get_sp_ep(struct dapl_cm_i sp_ptr->header.flags); (void)dapl_ib_remove_conn_listener(sp_ptr->header. owner_ia, sp_ptr); - dapl_ia_unlink_sp((DAPL_IA *) sp_ptr->header.owner_ia, + dapl_ia_unlink_sp((struct dapl_ia *) + sp_ptr->header.owner_ia, sp_ptr); dapl_sp_dealloc(sp_ptr); dapl_cr_free(cr_ptr); Index: linux-kernel-resolv/dat-provider/dapl_evd_dto_callb.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_evd_dto_callb.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_evd_dto_callb.c (working copy) @@ -70,7 +70,7 @@ void dapl_evd_dto_callback(struct ib_cq *cq_handle, void *user_context) { - DAPL_EVD *evd_ptr; + struct dapl_evd *evd_ptr; u32 dat_status; DAPL_EVD_STATE state; @@ -78,7 +78,7 @@ void dapl_evd_dto_callback(struct ib_cq "dapl_evd_dto_callback(%p, %p)\n", cq_handle, user_context); - evd_ptr = (DAPL_EVD *) user_context; + evd_ptr = (struct dapl_evd *)user_context; dapl_os_assert(evd_ptr->ib_cq_handle == cq_handle); dapl_os_assert(evd_ptr->header.magic == DAPL_MAGIC_EVD); Index: linux-kernel-resolv/dat-provider/dapl_rmr_create.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_rmr_create.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_rmr_create.c (working copy) @@ -55,8 +55,8 @@ */ u32 dapl_rmr_create(DAT_PZ_HANDLE pz_handle, DAT_RMR_HANDLE *rmr_handle) { - DAPL_PZ *pz; - DAPL_RMR *rmr; + struct dapl_pz *pz; + struct dapl_rmr *rmr; u32 dat_status = DAT_SUCCESS; if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { @@ -65,7 +65,7 @@ u32 dapl_rmr_create(DAT_PZ_HANDLE pz_han goto bail; } - pz = (DAPL_PZ *) pz_handle; + pz = (struct dapl_pz *)pz_handle; rmr = dapl_rmr_alloc(pz); @@ -89,6 +89,6 @@ u32 dapl_rmr_create(DAT_PZ_HANDLE pz_han *rmr_handle = rmr; - bail: +bail: return dat_status; } Index: linux-kernel-resolv/dat-provider/dapl_lmr_sync_rdma_read.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_lmr_sync_rdma_read.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_lmr_sync_rdma_read.c (working copy) @@ -60,14 +60,14 @@ u32 dapl_lmr_sync_rdma_read(DAT_IA_HANDL const struct dat_lmr_triplet *local_segments, u64 num_segments) { - DAPL_IA *ia_ptr; + struct dapl_ia *ia_ptr; u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dat_lmr_sync_rdma_read (%p, %p, %ld)\n", ia_handle, local_segments, num_segments); - ia_ptr = (DAPL_IA *) ia_handle; + ia_ptr = (struct dapl_ia *)ia_handle; if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { dat_status = Index: linux-kernel-resolv/dat-provider/dapl_module.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_module.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_module.c (working copy) @@ -120,7 +120,7 @@ void DAT_PROVIDER_INIT_FUNC_NAME(const s struct ib_device *device, u8 port) { struct dat_provider *provider; - DAPL_HCA *hca_ptr; + struct dapl_hca *hca_ptr; u32 dat_status; provider = NULL; Index: linux-kernel-resolv/dat-provider/dapl_ring_buffer_util.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_ring_buffer_util.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_ring_buffer_util.c (working copy) @@ -40,12 +40,12 @@ /* * dapl_rbuf_alloc * - * Given a DAPL_RING_BUFFER, initialize it and provide memory for + * Given a struct dapl_ring_buffer, initialize it and provide memory for * the ringbuf itself. A passed in size will be adjusted to the next * largest power of two number to simplify management. * * Input: - * rbuf pointer to DAPL_RING_BUFFER + * rbuf pointer to struct dapl_ring_buffer * size number of elements to allocate & manage * * Output: @@ -56,7 +56,7 @@ * DAT_INSUFFICIENT_RESOURCES * */ -u32 dapl_rbuf_alloc(DAPL_RING_BUFFER *rbuf, int size) +u32 dapl_rbuf_alloc(struct dapl_ring_buffer *rbuf, int size) { unsigned int rsize; /* real size */ @@ -86,12 +86,12 @@ u32 dapl_rbuf_alloc(DAPL_RING_BUFFER *rb /* * dapl_rbuf_realloc * - * Resizes an empty DAPL_RING_BUFFER. This function is not thread safe; + * Resizes an empty struct dapl_ring_buffer. This function is not thread safe; * adding or removing elements from a ring buffer while resizing * will have indeterminate results. * * Input: - * rbuf pointer to DAPL_RING_BUFFER + * rbuf pointer to struct dapl_ring_buffer * size number of elements to allocate & manage * * Output: @@ -103,10 +103,10 @@ u32 dapl_rbuf_alloc(DAPL_RING_BUFFER *rb * DAT_INSUFFICIENT_RESOURCES * */ -u32 dapl_rbuf_realloc(DAPL_RING_BUFFER *rbuf, int size) +u32 dapl_rbuf_realloc(struct dapl_ring_buffer *rbuf, int size) { - DAPL_RING_BUFFER new_rbuf; - void *entry; + struct dapl_ring_buffer new_rbuf; + void *entry; u32 dat_status = DAT_SUCCESS; /* decreasing the size or retaining the old size is not allowed */ @@ -146,10 +146,10 @@ u32 dapl_rbuf_realloc(DAPL_RING_BUFFER * /* * dapl_rbuf_destroy * - * Release the buffer and reset pointers to a DAPL_RING_BUFFER + * Release the buffer and reset pointers to a struct dapl_ring_buffer * * Input: - * rbuf pointer to DAPL_RING_BUFFER + * rbuf pointer to struct dapl_ring_buffer * * Output: * none @@ -158,7 +158,7 @@ u32 dapl_rbuf_realloc(DAPL_RING_BUFFER * * none * */ -void dapl_rbuf_destroy(DAPL_RING_BUFFER * rbuf) +void dapl_rbuf_destroy(struct dapl_ring_buffer *rbuf) { if ((NULL == rbuf) || (NULL == rbuf->base)) { return; @@ -177,7 +177,7 @@ void dapl_rbuf_destroy(DAPL_RING_BUFFER * Add an entry to the ring buffer * * Input: - * rbuf pointer to DAPL_RING_BUFFER + * rbuf pointer to struct dapl_ring_buffer * entry entry to add * * Output: @@ -188,7 +188,7 @@ void dapl_rbuf_destroy(DAPL_RING_BUFFER * DAT_INSUFFICIENT_RESOURCES (queue full) * */ -u32 dapl_rbuf_add(DAPL_RING_BUFFER *rbuf, void *entry) +u32 dapl_rbuf_add(struct dapl_ring_buffer *rbuf, void *entry) { int pos; int val; @@ -214,7 +214,7 @@ u32 dapl_rbuf_add(DAPL_RING_BUFFER *rbuf * Remove an entry from the ring buffer * * Input: - * rbuf pointer to DAPL_RING_BUFFER + * rbuf pointer to struct dapl_ring_buffer * * Output: * entry entry removed from the ring buffer @@ -224,7 +224,7 @@ u32 dapl_rbuf_add(DAPL_RING_BUFFER *rbuf * DAT_QUEUE_EMPTY */ -void *dapl_rbuf_remove(DAPL_RING_BUFFER * rbuf) +void *dapl_rbuf_remove(struct dapl_ring_buffer *rbuf) { int pos; int val; @@ -250,7 +250,7 @@ void *dapl_rbuf_remove(DAPL_RING_BUFFER * Return the number of entries in use in the ring buffer * * Input: - * rbuf pointer to DAPL_RING_BUFFER + * rbuf pointer to struct dapl_ring_buffer * * * Output: @@ -260,7 +260,7 @@ void *dapl_rbuf_remove(DAPL_RING_BUFFER * count of entries * */ -int dapl_rbuf_count(DAPL_RING_BUFFER * rbuf) +int dapl_rbuf_count(struct dapl_ring_buffer *rbuf) { int count, head, tail; @@ -286,7 +286,7 @@ int dapl_rbuf_count(DAPL_RING_BUFFER * r * functions). * * Input: - * rbuf pointer to DAPL_RING_BUFFER + * rbuf pointer to struct dapl_ring_buffer * offset offset to adjust elemnt addresss by, * used for addresses of type void * * Output: @@ -295,7 +295,7 @@ int dapl_rbuf_count(DAPL_RING_BUFFER * r * Returns: * none */ -void dapl_rbuf_adjust(DAPL_RING_BUFFER * rbuf, unsigned long offset) +void dapl_rbuf_adjust(struct dapl_ring_buffer *rbuf, unsigned long offset) { int pos; Index: linux-kernel-resolv/dat-provider/dapl_psp_create.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_psp_create.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_psp_create.c (working copy) @@ -71,13 +71,13 @@ u32 dapl_psp_create(DAT_IA_HANDLE ia_han DAT_EVD_HANDLE evd_handle, enum dat_psp_flags psp_flags, DAT_PSP_HANDLE *psp_handle) { - DAPL_IA *ia_ptr; - DAPL_SP *sp_ptr; - DAPL_EVD *evd_ptr; + struct dapl_ia *ia_ptr; + struct dapl_sp *sp_ptr; + struct dapl_evd *evd_ptr; boolean_t sp_found; u32 dat_status = DAT_SUCCESS; - ia_ptr = (DAPL_IA *) ia_handle; + ia_ptr = (struct dapl_ia *)ia_handle; if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { dat_status = @@ -95,7 +95,7 @@ u32 dapl_psp_create(DAT_IA_HANDLE ia_han goto bail; } - evd_ptr = (DAPL_EVD *) evd_handle; + evd_ptr = (struct dapl_evd *)evd_handle; if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); @@ -141,7 +141,7 @@ u32 dapl_psp_create(DAT_IA_HANDLE ia_han /* * Take a reference on the EVD handle */ - atomic_inc(&((DAPL_EVD *) evd_handle)->evd_ref_count); + atomic_inc(&((struct dapl_evd *)evd_handle)->evd_ref_count); /* * Set up a listener for a connection. Connections can arrive @@ -168,7 +168,7 @@ u32 dapl_psp_create(DAT_IA_HANDLE ia_han * Have a problem setting up the connection, something * wrong! Decrements the EVD refcount & release it. */ - atomic_dec(&((DAPL_EVD *) evd_handle)-> + atomic_dec(&((struct dapl_evd *)evd_handle)-> evd_ref_count); sp_ptr->evd_handle = NULL; dapl_ia_unlink_sp(ia_ptr, sp_ptr); Index: linux-kernel-resolv/dat-provider/dapl_srq_resize.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_srq_resize.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_srq_resize.c (working copy) @@ -65,8 +65,8 @@ u32 dapl_srq_resize(DAT_SRQ_HANDLE srq_handle, int srq_max_recv_dto) { - DAPL_IA *ia_ptr; - DAPL_SRQ *srq_ptr; + struct dapl_ia *ia_ptr; + struct dapl_srq *srq_ptr; u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_resize (%p, %d)\n", @@ -78,7 +78,7 @@ u32 dapl_srq_resize(DAT_SRQ_HANDLE srq_h goto bail; } - srq_ptr = (DAPL_SRQ *) srq_handle; + srq_ptr = (struct dapl_srq *)srq_handle; ia_ptr = srq_ptr->header.owner_ia; /* Index: linux-kernel-resolv/dat-provider/dapl_ep_create.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_ep_create.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_ep_create.c (working copy) @@ -69,8 +69,8 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand DAT_EVD_HANDLE connect_evd_handle, const struct dat_ep_attr *ep_attr, DAT_EP_HANDLE *ep_handle) { - DAPL_IA *ia_ptr; - DAPL_EP *ep_ptr; + struct dapl_ia *ia_ptr; + struct dapl_ep *ep_ptr; struct dat_ep_attr *ep_attr_limit; u32 dat_status = DAT_SUCCESS; @@ -82,7 +82,7 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand request_evd_handle, connect_evd_handle, ep_attr, ep_handle); - ia_ptr = (DAPL_IA *) ia_handle; + ia_ptr = (struct dapl_ia *)ia_handle; /* * Verify parameters @@ -109,7 +109,7 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand /* If connect handle is !NULL verify handle is good */ if (connect_evd_handle != DAT_HANDLE_NULL && (DAPL_BAD_HANDLE(connect_evd_handle, DAPL_MAGIC_EVD) || - !(((DAPL_EVD *) connect_evd_handle)-> + !(((struct dapl_evd *)connect_evd_handle)-> evd_flags & DAT_EVD_CONNECTION_FLAG))) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CONN); @@ -119,7 +119,7 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand /* If recv_evd is !NULL, verify handle is good and flags are valid */ if (recv_evd_handle != DAT_HANDLE_NULL && (DAPL_BAD_HANDLE(recv_evd_handle, DAPL_MAGIC_EVD) || - !(((DAPL_EVD *) recv_evd_handle)->evd_flags & DAT_EVD_DTO_FLAG))) { + !(((struct dapl_evd *)recv_evd_handle)->evd_flags & DAT_EVD_DTO_FLAG))) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_RECV); goto bail; @@ -128,7 +128,7 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand /* If req_evd is !NULL, verify handle is good and flags are valid */ if (request_evd_handle != DAT_HANDLE_NULL && (DAPL_BAD_HANDLE(request_evd_handle, DAPL_MAGIC_EVD) || - !(((DAPL_EVD *) request_evd_handle)-> + !(((struct dapl_evd *)request_evd_handle)-> evd_flags & DAT_EVD_DTO_FLAG))) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, @@ -210,13 +210,11 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand /* * Make sure we handle the NULL DTO EVDs */ - if (recv_evd_handle == DAT_HANDLE_NULL && ep_attr == NULL) { + if (recv_evd_handle == DAT_HANDLE_NULL && ep_attr == NULL) ep_ptr->param.ep_attr.max_recv_dtos = 0; - } - if (request_evd_handle == DAT_HANDLE_NULL && ep_attr == NULL) { + if (request_evd_handle == DAT_HANDLE_NULL && ep_attr == NULL) ep_ptr->param.ep_attr.max_request_dtos = 0; - } /* * If the user has specified a PZ handle we allocate a QP for @@ -226,7 +224,7 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand */ if (pz_handle != DAT_HANDLE_NULL) { /* Take a reference on the PZ handle */ - atomic_inc(&((DAPL_PZ *) pz_handle)->pz_ref_count); + atomic_inc(&((struct dapl_pz *)pz_handle)->pz_ref_count); /* * Get a QP from the IB provider @@ -234,14 +232,12 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand dat_status = dapl_ib_qp_alloc(ia_ptr, ep_ptr, ep_ptr); if (dat_status != DAT_SUCCESS) { - atomic_dec(&((DAPL_PZ *) pz_handle)-> - pz_ref_count); + atomic_dec(&((struct dapl_pz *)pz_handle)->pz_ref_count); dapl_ep_dealloc(ep_ptr); goto bail; } - } else { + } else ep_ptr->qp_state = DAPL_QP_STATE_UNATTACHED; - } /* * Update ref counts. See the spec where the endpoint marks @@ -251,22 +247,20 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand * * N.B. This should really be done by a util routine. */ - atomic_inc(&((DAPL_EVD *) connect_evd_handle)->evd_ref_count); + atomic_inc(&((struct dapl_evd *)connect_evd_handle)->evd_ref_count); /* Optional handles */ - if (recv_evd_handle != DAT_HANDLE_NULL) { - atomic_inc(&((DAPL_EVD *) recv_evd_handle)-> + if (recv_evd_handle != DAT_HANDLE_NULL) + atomic_inc(&((struct dapl_evd *)recv_evd_handle)-> evd_ref_count); - } - if (request_evd_handle != DAT_HANDLE_NULL) { - atomic_inc(&((DAPL_EVD *) request_evd_handle)-> + if (request_evd_handle != DAT_HANDLE_NULL) + atomic_inc(&((struct dapl_evd *)request_evd_handle)-> evd_ref_count); - } /* Link it onto the IA */ dapl_ia_link_ep(ia_ptr, ep_ptr); *ep_handle = ep_ptr; - bail: +bail: return dat_status; } Index: linux-kernel-resolv/dat-provider/dapl_lmr_util.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_lmr_util.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_lmr_util.c (working copy) @@ -38,14 +38,12 @@ #include "dapl_lmr_util.h" #include "dapl_ia_util.h" -DAPL_LMR *dapl_lmr_alloc(DAPL_IA * ia, - enum dat_mem_type mem_type, - DAT_REGION_DESCRIPTION region_desc, - u64 length, - DAT_PZ_HANDLE pz_handle, - enum dat_mem_priv_flags mem_priv) +struct dapl_lmr *dapl_lmr_alloc(struct dapl_ia *ia, enum dat_mem_type mem_type, + DAT_REGION_DESCRIPTION region_desc, u64 length, + DAT_PZ_HANDLE pz_handle, + enum dat_mem_priv_flags mem_priv) { - DAPL_LMR *lmr; + struct dapl_lmr *lmr; /* Allocate LMR */ lmr = kmalloc(sizeof *lmr, GFP_ATOMIC); @@ -82,9 +80,10 @@ DAPL_LMR *dapl_lmr_alloc(DAPL_IA * ia, return lmr; } -void dapl_lmr_dealloc(DAPL_LMR * lmr) +void dapl_lmr_dealloc(struct dapl_lmr *lmr) { - lmr->header.magic = DAPL_MAGIC_INVALID; /* reset magic to prevent reuse */ + /* reset magic to prevent reuse */ + lmr->header.magic = DAPL_MAGIC_INVALID; dapl_ia_unlink_lmr(lmr->header.owner_ia, lmr); /* no need to destroy lmr->header.lock */ Index: linux-kernel-resolv/dat-provider/dapl_openib_dto.h =================================================================== --- linux-kernel-resolv/dat-provider/dapl_openib_dto.h (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_openib_dto.h (working copy) @@ -49,7 +49,8 @@ * * Provider specific Post RECV function */ -static inline u32 dapl_ib_post_recv(DAPL_EP *ep_ptr, DAPL_COOKIE *cookie, +static inline u32 dapl_ib_post_recv(struct dapl_ep *ep_ptr, + struct dapl_cookie *cookie, int num_segments, struct dat_lmr_triplet *local_iov) { @@ -98,8 +99,9 @@ static inline u32 dapl_ib_post_recv(DAPL * * Provider specific Post SEND function */ -static inline u32 dapl_ib_post_send(DAPL_EP *ep_ptr, enum ib_wr_opcode op_type, - DAPL_COOKIE *cookie, +static inline u32 dapl_ib_post_send(struct dapl_ep *ep_ptr, + enum ib_wr_opcode op_type, + struct dapl_cookie *cookie, int num_segments, struct dat_lmr_triplet *local_iov, const struct dat_rmr_triplet *remote_iov, Index: linux-kernel-resolv/dat-provider/dapl_hca_util.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_hca_util.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_hca_util.c (working copy) @@ -56,9 +56,9 @@ * none * */ -DAPL_HCA *dapl_hca_alloc(char *name, struct ib_device *device, u8 port) +struct dapl_hca *dapl_hca_alloc(char *name, struct ib_device *device, u8 port) { - DAPL_HCA *hca_ptr; + struct dapl_hca *hca_ptr; hca_ptr = kmalloc(sizeof *hca_ptr, GFP_ATOMIC); if (hca_ptr) { @@ -101,7 +101,7 @@ DAPL_HCA *dapl_hca_alloc(char *name, str * none * */ -void dapl_hca_free(DAPL_HCA * hca_ptr) +void dapl_hca_free(struct dapl_hca *hca_ptr) { (void)dapl_hash_free(hca_ptr->lmr_hash_table); kfree(hca_ptr->name); @@ -124,7 +124,7 @@ void dapl_hca_free(DAPL_HCA * hca_ptr) * none * */ -void dapl_hca_link_ia(DAPL_HCA * hca_ptr, DAPL_IA * ia_ptr) +void dapl_hca_link_ia(struct dapl_hca *hca_ptr, struct dapl_ia *ia_ptr) { spin_lock_irqsave(&hca_ptr->lock, hca_ptr->flags); dapl_llist_add_head(&hca_ptr->ia_list_head, @@ -148,7 +148,7 @@ void dapl_hca_link_ia(DAPL_HCA * hca_ptr * none * */ -void dapl_hca_unlink_ia(DAPL_HCA * hca_ptr, DAPL_IA * ia_ptr) +void dapl_hca_unlink_ia(struct dapl_hca *hca_ptr, struct dapl_ia *ia_ptr) { spin_lock_irqsave(&hca_ptr->lock, hca_ptr->flags); /* Index: linux-kernel-resolv/dat-provider/dapl_srq_create.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_srq_create.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_srq_create.c (working copy) @@ -68,15 +68,15 @@ u32 dapl_srq_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, struct dat_srq_attr *srq_attr, DAT_SRQ_HANDLE *srq_handle) { - DAPL_IA *ia_ptr; - DAPL_SRQ *srq_ptr; + struct dapl_ia *ia_ptr; + struct dapl_srq *srq_ptr; u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_create (%p, %p, %p, %p)\n", ia_handle, pz_handle, srq_attr, srq_handle); - ia_ptr = (DAPL_IA *) ia_handle; + ia_ptr = (struct dapl_ia *)ia_handle; /* * Verify parameters Index: linux-kernel-resolv/dat-provider/dapl_evd_connection_callb.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_evd_connection_callb.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_evd_connection_callb.c (working copy) @@ -65,8 +65,8 @@ void dapl_evd_connection_callback(struct const void *private_data_ptr, const void *context) { - DAPL_EP *ep_ptr; - DAPL_EVD *evd_ptr; + struct dapl_ep *ep_ptr; + struct dapl_evd *evd_ptr; int private_data_size; u32 dat_status; @@ -78,17 +78,15 @@ void dapl_evd_connection_callback(struct * Determine the type of handle passed back to us in the context * and sort out key parameters. */ - if (context == NULL - || ((DAPL_HEADER *) context)->magic != DAPL_MAGIC_EP) { + if (context == NULL || ((struct dapl_header *) context)->magic != DAPL_MAGIC_EP) return; - } /* * Active side of the connection, context is an EP and * PSP is irrelevant. */ - ep_ptr = (DAPL_EP *) context; - evd_ptr = (DAPL_EVD *) ep_ptr->param.connect_evd_handle; + ep_ptr = (struct dapl_ep *)context; + evd_ptr = (struct dapl_evd *)ep_ptr->param.connect_evd_handle; private_data_size = 0; /* Index: linux-kernel-resolv/dat-provider/dapl_srq_query.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_srq_query.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_srq_query.c (working copy) @@ -41,7 +41,7 @@ u32 dapl_srq_query(DAT_SRQ_HANDLE srq_handle, struct dat_srq_param *srq_param) { - DAPL_SRQ *srq_ptr; + struct dapl_srq *srq_ptr; u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, @@ -58,7 +58,7 @@ u32 dapl_srq_query(DAT_SRQ_HANDLE srq_ha goto bail; } - srq_ptr = (DAPL_SRQ *) srq_handle; + srq_ptr = (struct dapl_srq *)srq_handle; /* * XXX Need to calculate available_dto_count and outstanding_dto_count Index: linux-kernel-resolv/dat-provider/dapl_ring_buffer_util.h =================================================================== --- linux-kernel-resolv/dat-provider/dapl_ring_buffer_util.h (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_ring_buffer_util.h (working copy) @@ -42,19 +42,19 @@ /* * Prototypes */ -u32 dapl_rbuf_alloc(DAPL_RING_BUFFER *rbuf, int size); +u32 dapl_rbuf_alloc(struct dapl_ring_buffer *rbuf, int size); -u32 dapl_rbuf_realloc(DAPL_RING_BUFFER *rbuf, int size); +u32 dapl_rbuf_realloc(struct dapl_ring_buffer *rbuf, int size); -void dapl_rbuf_destroy(DAPL_RING_BUFFER * rbuf); +void dapl_rbuf_destroy(struct dapl_ring_buffer *rbuf); -u32 dapl_rbuf_add(DAPL_RING_BUFFER *rbuf, void *entry); +u32 dapl_rbuf_add(struct dapl_ring_buffer *rbuf, void *entry); -void *dapl_rbuf_remove(DAPL_RING_BUFFER * rbuf); +void *dapl_rbuf_remove(struct dapl_ring_buffer *rbuf); -int dapl_rbuf_count(DAPL_RING_BUFFER *rbuf); +int dapl_rbuf_count(struct dapl_ring_buffer *rbuf); -void dapl_rbuf_adjust(DAPL_RING_BUFFER * rbuf, unsigned long offset); +void dapl_rbuf_adjust(struct dapl_ring_buffer *rbuf, unsigned long offset); /* * Simple functions Index: linux-kernel-resolv/dat-provider/dapl_psp_create_any.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_psp_create_any.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_psp_create_any.c (working copy) @@ -74,15 +74,15 @@ u32 dapl_psp_create_any(DAT_IA_HANDLE ia DAT_EVD_HANDLE evd_handle, enum dat_psp_flags psp_flags, DAT_PSP_HANDLE *psp_handle) { - DAPL_IA *ia_ptr; - DAPL_SP *sp_ptr; - DAPL_EVD *evd_ptr; + struct dapl_ia *ia_ptr; + struct dapl_sp *sp_ptr; + struct dapl_evd *evd_ptr; u32 dat_status = DAT_SUCCESS; static DAT_CONN_QUAL hint_conn_qual = 1000; /* seed value */ DAT_CONN_QUAL lcl_conn_qual; DAT_CONN_QUAL limit_conn_qual; - ia_ptr = (DAPL_IA *) ia_handle; + ia_ptr = (struct dapl_ia *)ia_handle; if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { dat_status = @@ -104,7 +104,7 @@ u32 dapl_psp_create_any(DAT_IA_HANDLE ia goto bail; } - evd_ptr = (DAPL_EVD *) evd_handle; + evd_ptr = (struct dapl_evd *)evd_handle; if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); @@ -136,7 +136,7 @@ u32 dapl_psp_create_any(DAT_IA_HANDLE ia /* * Take a reference on the EVD handle */ - atomic_inc(&((DAPL_EVD *) evd_handle)->evd_ref_count); + atomic_inc(&((struct dapl_evd *)evd_handle)->evd_ref_count); /* Link it onto the IA */ dapl_ia_link_psp(ia_ptr, sp_ptr); @@ -176,7 +176,7 @@ u32 dapl_psp_create_any(DAT_IA_HANDLE ia /* * Have a problem setting up the connection, something wrong! */ - atomic_dec(&((DAPL_EVD *) evd_handle)->evd_ref_count); + atomic_dec(&((struct dapl_evd *)evd_handle)->evd_ref_count); sp_ptr->evd_handle = NULL; dapl_ia_unlink_sp(ia_ptr, sp_ptr); dapl_sp_dealloc(sp_ptr); Index: linux-kernel-resolv/dat-provider/dapl_lmr_util.h =================================================================== --- linux-kernel-resolv/dat-provider/dapl_lmr_util.h (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_lmr_util.h (working copy) @@ -39,19 +39,12 @@ #include "dapl_mr_util.h" -/********************************************************************* - * * - * Function Prototypes * - * * - *********************************************************************/ +extern struct dapl_lmr *dapl_lmr_alloc(struct dapl_ia *ia, + enum dat_mem_type mem_type, + DAT_REGION_DESCRIPTION region_desc, + u64 length, DAT_PZ_HANDLE pz_handle, + enum dat_mem_priv_flags mem_priv); -extern DAPL_LMR *dapl_lmr_alloc(DAPL_IA * ia, - enum dat_mem_type mem_type, - DAT_REGION_DESCRIPTION region_desc, - u64 length, - DAT_PZ_HANDLE pz_handle, - enum dat_mem_priv_flags mem_priv); +extern void dapl_lmr_dealloc(struct dapl_lmr *lmr); -extern void dapl_lmr_dealloc(DAPL_LMR * lmr); - -#endif /* DAPL_LMR_UTIL_H */ +#endif /* DAPL_LMR_UTIL_H */ Index: linux-kernel-resolv/dat-provider/dapl_hca_util.h =================================================================== --- linux-kernel-resolv/dat-provider/dapl_hca_util.h (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_hca_util.h (working copy) @@ -39,12 +39,12 @@ #include "dapl.h" -DAPL_HCA *dapl_hca_alloc(char *, struct ib_device *, u8 port); +struct dapl_hca *dapl_hca_alloc(char *name, struct ib_device *device, u8 port); -void dapl_hca_free(DAPL_HCA * hca_ptr); +void dapl_hca_free(struct dapl_hca *hca_ptr); -void dapl_hca_link_ia(DAPL_HCA * hca_ptr, DAPL_IA * ia_info); +void dapl_hca_link_ia(struct dapl_hca *hca_ptr, struct dapl_ia *ia_info); -void dapl_hca_unlink_ia(DAPL_HCA * hca_ptr, DAPL_IA * ia_info); +void dapl_hca_unlink_ia(struct dapl_hca *hca_ptr, struct dapl_ia *ia_info); #endif Index: linux-kernel-resolv/dat-provider/dapl_get_consumer_context.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_get_consumer_context.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_get_consumer_context.c (working copy) @@ -58,9 +58,9 @@ u32 dapl_get_consumer_context(DAT_HANDLE dat_handle, DAT_CONTEXT *context) { u32 dat_status = DAT_SUCCESS; - DAPL_HEADER *header; + struct dapl_header *header; - header = (DAPL_HEADER *) dat_handle; + header = (struct dapl_header *)dat_handle; if (((header) == NULL) || ((unsigned long)(header) & 3) || (header->magic != DAPL_MAGIC_IA && Index: linux-kernel-resolv/dat-provider/dapl_evd_resize.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_evd_resize.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_evd_resize.c (working copy) @@ -65,8 +65,8 @@ u32 dapl_evd_resize(DAT_EVD_HANDLE evd_handle, int evd_qlen) { - DAPL_IA *ia_ptr; - DAPL_EVD *evd_ptr; + struct dapl_ia *ia_ptr; + struct dapl_evd *evd_ptr; int pend_cnt; u32 status; @@ -78,7 +78,7 @@ u32 dapl_evd_resize(DAT_EVD_HANDLE evd_h goto bail; } - evd_ptr = (DAPL_EVD *) evd_handle; + evd_ptr = (struct dapl_evd *)evd_handle; ia_ptr = evd_ptr->header.owner_ia; if (evd_qlen == evd_ptr->qlen) { Index: linux-kernel-resolv/dat-provider/dapl_cr_util.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_cr_util.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_cr_util.c (working copy) @@ -37,9 +37,9 @@ /* * Create a CR. Part of the passive side of a connection */ -DAPL_CR *dapl_cr_alloc(DAPL_IA * ia_ptr) +struct dapl_cr *dapl_cr_alloc(struct dapl_ia *ia_ptr) { - DAPL_CR *cr_ptr; + struct dapl_cr *cr_ptr; /* Allocate EP */ cr_ptr = kmalloc(sizeof *cr_ptr, GFP_ATOMIC); @@ -67,11 +67,11 @@ DAPL_CR *dapl_cr_alloc(DAPL_IA * ia_ptr) /* * Free the passed in EP structure. */ -void dapl_cr_free(DAPL_CR * cr_ptr) +void dapl_cr_free(struct dapl_cr *cr_ptr) { dapl_os_assert(cr_ptr->header.magic == DAPL_MAGIC_CR || cr_ptr->header.magic == DAPL_MAGIC_CR_DESTROYED); - - cr_ptr->header.magic = DAPL_MAGIC_INVALID; /* reset magic to prevent reuse */ + /* reset magic to prevent reuse */ + cr_ptr->header.magic = DAPL_MAGIC_INVALID; kfree(cr_ptr); } Index: linux-kernel-resolv/dat-provider/dapl_rmr_free.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_rmr_free.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_rmr_free.c (working copy) @@ -55,7 +55,7 @@ */ u32 dapl_rmr_free(DAT_RMR_HANDLE rmr_handle) { - DAPL_RMR *rmr; + struct dapl_rmr *rmr; u32 dat_status = DAT_SUCCESS; if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) { @@ -64,7 +64,7 @@ u32 dapl_rmr_free(DAT_RMR_HANDLE rmr_han goto bail; } - rmr = (DAPL_RMR *) rmr_handle; + rmr = (struct dapl_rmr *)rmr_handle; /* * If the user did not perform an unbind op, release @@ -77,14 +77,13 @@ u32 dapl_rmr_free(DAT_RMR_HANDLE rmr_han dat_status = dapl_ib_mw_free(rmr); - if (dat_status != DAT_SUCCESS) { + if (dat_status != DAT_SUCCESS) goto bail; - } atomic_dec(&rmr->pz->pz_ref_count); dapl_rmr_dealloc(rmr); - bail: +bail: return dat_status; } Index: linux-kernel-resolv/dat-provider/dapl_srq_util.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_srq_util.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_srq_util.c (working copy) @@ -54,9 +54,10 @@ * pointer to srq * */ -DAPL_SRQ *dapl_srq_alloc(DAPL_IA * ia_ptr, const struct dat_srq_attr *srq_attr) +struct dapl_srq *dapl_srq_alloc(struct dapl_ia *ia_ptr, + const struct dat_srq_attr *srq_attr) { - DAPL_SRQ *srq_ptr; + struct dapl_srq *srq_ptr; /* Allocate SRQ */ srq_ptr = kmalloc(sizeof *srq_ptr, GFP_ATOMIC); @@ -89,14 +90,15 @@ DAPL_SRQ *dapl_srq_alloc(DAPL_IA * ia_pt srq_ptr->param.low_watermark = srq_attr->low_watermark; /* Get a cookie buffer to track outstanding recvs */ - if (DAT_SUCCESS != dapl_cb_create(&srq_ptr->recv_buffer, (DAPL_EP *) srq_ptr, /* just saves the value */ + if (DAT_SUCCESS != dapl_cb_create(&srq_ptr->recv_buffer, + (struct dapl_ep *)srq_ptr, srq_ptr->param.max_recv_dtos)) { dapl_srq_dealloc(srq_ptr); srq_ptr = NULL; goto bail; } - bail: +bail: return srq_ptr; } @@ -115,11 +117,12 @@ DAPL_SRQ *dapl_srq_alloc(DAPL_IA * ia_pt * none * */ -void dapl_srq_dealloc(DAPL_SRQ * srq_ptr) +void dapl_srq_dealloc(struct dapl_srq *srq_ptr) { dapl_os_assert(srq_ptr->header.magic == DAPL_MAGIC_SRQ); - srq_ptr->header.magic = DAPL_MAGIC_INVALID; /* reset magic to prevent reuse */ + /* reset magic to prevent reuse */ + srq_ptr->header.magic = DAPL_MAGIC_INVALID; dapl_ia_unlink_srq(srq_ptr->header.owner_ia, srq_ptr); dapl_cb_free(&srq_ptr->recv_buffer); /* no need to destroy srq_ptr->header.lock */ Index: linux-kernel-resolv/dat-provider/dapl_psp_free.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_psp_free.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_psp_free.c (working copy) @@ -60,12 +60,12 @@ */ u32 dapl_psp_free(DAT_PSP_HANDLE psp_handle) { - DAPL_IA *ia_ptr; - DAPL_SP *sp_ptr; + struct dapl_ia *ia_ptr; + struct dapl_sp *sp_ptr; DAPL_SP_STATE save_state; u32 dat_status = DAT_SUCCESS; - sp_ptr = (DAPL_SP *) psp_handle; + sp_ptr = (struct dapl_sp *)psp_handle; /* * Verify handle */ @@ -77,7 +77,7 @@ u32 dapl_psp_free(DAT_PSP_HANDLE psp_han goto bail; } - /* ia_ptr = (DAPL_IA *)sp_ptr->header.owner_ia; */ + /* ia_ptr = (struct dapl_ia *)sp_ptr->header.owner_ia; */ ia_ptr = sp_ptr->header.owner_ia; /* * Remove the connection listener if it has been established @@ -94,7 +94,7 @@ u32 dapl_psp_free(DAT_PSP_HANDLE psp_han * free the evd_handle will be NULL */ if (sp_ptr->evd_handle) { - atomic_dec(&((DAPL_EVD *) sp_ptr->evd_handle)-> + atomic_dec(&((struct dapl_evd *)sp_ptr->evd_handle)-> evd_ref_count); sp_ptr->evd_handle = NULL; } Index: linux-kernel-resolv/dat-provider/dapl_ep_free.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_ep_free.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_ep_free.c (working copy) @@ -63,15 +63,15 @@ */ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle) { - DAPL_EP *ep_ptr; - DAPL_IA *ia_ptr; + struct dapl_ep *ep_ptr; + struct dapl_ia *ia_ptr; struct dat_ep_param *param; enum ib_qp_state save_qp_state; u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_free (%p)\n", ep_handle); - ep_ptr = (DAPL_EP *) ep_handle; + ep_ptr = (struct dapl_ep *)ep_handle; param = &ep_ptr->param; /* @@ -129,22 +129,21 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle * is idempotent. */ if (param->pz_handle != NULL) { - atomic_dec(&((DAPL_PZ *) param->pz_handle)-> - pz_ref_count); + atomic_dec(&((struct dapl_pz *)param->pz_handle)->pz_ref_count); param->pz_handle = NULL; } if (param->recv_evd_handle != NULL) { - atomic_dec(&((DAPL_EVD *) param->recv_evd_handle)-> + atomic_dec(&((struct dapl_evd *)param->recv_evd_handle)-> evd_ref_count); param->recv_evd_handle = NULL; } if (param->request_evd_handle != NULL) { - atomic_dec(&((DAPL_EVD *) param->request_evd_handle)-> + atomic_dec(&((struct dapl_evd *)param->request_evd_handle)-> evd_ref_count); param->request_evd_handle = NULL; } if (param->connect_evd_handle != NULL) { - atomic_dec(&((DAPL_EVD *) param->connect_evd_handle)-> + atomic_dec(&((struct dapl_evd *)param->connect_evd_handle)-> evd_ref_count); param->connect_evd_handle = NULL; } @@ -181,7 +180,7 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle /* Free the resource */ dapl_ep_dealloc(ep_ptr); - bail: +bail: return dat_status; } Index: linux-kernel-resolv/dat-provider/dapl_ep_get_status.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_ep_get_status.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_ep_get_status.c (working copy) @@ -62,14 +62,14 @@ u32 dapl_ep_get_status(DAT_EP_HANDLE ep_handle, enum dat_ep_state *ep_state, boolean_t *in_dto_idle, boolean_t *out_dto_idle) { - DAPL_EP *ep_ptr; + struct dapl_ep *ep_ptr; u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_get_status (%p, %p, %p, %p)\n", ep_handle, ep_state, in_dto_idle, out_dto_idle); - ep_ptr = (DAPL_EP *) ep_handle; + ep_ptr = (struct dapl_ep *)ep_handle; /* * Verify parameter & state Index: linux-kernel-resolv/dat-provider/dapl_cr_util.h =================================================================== --- linux-kernel-resolv/dat-provider/dapl_cr_util.h (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_cr_util.h (working copy) @@ -36,12 +36,12 @@ #include "dapl.h" -DAPL_CR *dapl_cr_alloc(DAPL_IA * ia_ptr); +struct dapl_cr *dapl_cr_alloc(struct dapl_ia *ia_ptr); -void dapl_cr_free(DAPL_CR * cr_ptr); +void dapl_cr_free(struct dapl_cr *cr_ptr); void dapl_cr_callback(struct dapl_cm_id *ib_cm_handle, const enum dat_event_number event, const void *instant_data_p, const void *context); -#endif /* DAPL_CR_UTIL_H */ +#endif /* DAPL_CR_UTIL_H */ Index: linux-kernel-resolv/dat-provider/dapl_srq_util.h =================================================================== --- linux-kernel-resolv/dat-provider/dapl_srq_util.h (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_srq_util.h (working copy) @@ -39,11 +39,9 @@ #include "dapl.h" -/* function prototypes */ +extern struct dapl_srq *dapl_srq_alloc(struct dapl_ia *ia, + const struct dat_srq_attr *srq_attr); -extern DAPL_SRQ *dapl_srq_alloc(DAPL_IA * ia, - const struct dat_srq_attr *srq_attr); +extern void dapl_srq_dealloc(struct dapl_srq *srq_ptr); -extern void dapl_srq_dealloc(DAPL_SRQ * srq_ptr); - -#endif /* DAPL_SRQ_UTIL_H */ +#endif /* DAPL_SRQ_UTIL_H */ Index: linux-kernel-resolv/dat-provider/dapl_evd_post_se.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_evd_post_se.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_evd_post_se.c (working copy) @@ -61,10 +61,10 @@ u32 dapl_evd_post_se(DAT_EVD_HANDLE evd_handle, const struct dat_event *event) { - DAPL_EVD *evd_ptr; + struct dapl_evd *evd_ptr; u32 dat_status = DAT_SUCCESS; - evd_ptr = (DAPL_EVD *) evd_handle; + evd_ptr = (struct dapl_evd *)evd_handle; if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, 0); Index: linux-kernel-resolv/dat-provider/dapl_set_consumer_context.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_set_consumer_context.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_set_consumer_context.c (working copy) @@ -59,9 +59,9 @@ u32 dapl_set_consumer_context(DAT_HANDLE dat_handle, DAT_CONTEXT context) { u32 dat_status = DAT_SUCCESS; - DAPL_HEADER *header; + struct dapl_header *header; - header = (DAPL_HEADER *) dat_handle; + header = (struct dapl_header *)dat_handle; if (((header) == NULL) || ((unsigned long)(header) & 3) || (header->magic != DAPL_MAGIC_IA && Index: linux-kernel-resolv/dat-provider/dapl_mr_util.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_mr_util.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_mr_util.c (working copy) @@ -58,13 +58,13 @@ u64 dapl_mr_get_address(DAT_REGION_DESCRIPTION desc, enum dat_mem_type type) { - DAPL_LMR *lmr; + struct dapl_lmr *lmr; switch (type) { case DAT_MEM_TYPE_VIRTUAL: return (u64) (unsigned long) desc.for_va; case DAT_MEM_TYPE_LMR: - lmr = (DAPL_LMR *) desc.for_lmr_handle; + lmr = (struct dapl_lmr *)desc.for_lmr_handle; /* Since this function is recoursive we cannot inline it */ return dapl_mr_get_address(lmr->param.region_desc, Index: linux-kernel-resolv/dat-provider/dapl_pz_create.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_pz_create.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_pz_create.c (working copy) @@ -57,8 +57,8 @@ */ u32 dapl_pz_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE *pz_handle) { - DAPL_IA *ia; - DAPL_PZ *pz; + struct dapl_ia *ia; + struct dapl_pz *pz; u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, @@ -75,7 +75,7 @@ u32 dapl_pz_create(DAT_IA_HANDLE ia_hand goto bail; } - ia = (DAPL_IA *) ia_handle; + ia = (struct dapl_ia *)ia_handle; pz = dapl_pz_alloc(ia); if (NULL == pz) { Index: linux-kernel-resolv/dat-provider/dapl_ia_query.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_ia_query.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_ia_query.c (working copy) @@ -62,7 +62,7 @@ u32 dapl_ia_query(DAT_IA_HANDLE ia_handl struct dat_ia_attr *ia_attr, struct dat_provider_attr *provider_attr) { - DAPL_IA *ia_ptr; + struct dapl_ia *ia_ptr; u32 dat_status = DAT_SUCCESS; struct evd_merge_type { boolean_t array[6][6]; @@ -78,7 +78,7 @@ u32 dapl_ia_query(DAT_IA_HANDLE ia_handl ia_attr, provider_attr); - ia_ptr = (DAPL_IA *) ia_handle; + ia_ptr = (struct dapl_ia *)ia_handle; if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { dat_status = Index: linux-kernel-resolv/dat-provider/dapl_pz_query.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_pz_query.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_pz_query.c (working copy) @@ -38,7 +38,7 @@ u32 dapl_pz_query(DAT_PZ_HANDLE pz_handle, struct dat_pz_param *pz_param) { - DAPL_PZ *pz; + struct dapl_pz *pz; u32 status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_query (%p, %x, %p)\n", @@ -53,7 +53,7 @@ u32 dapl_pz_query(DAT_PZ_HANDLE pz_handl goto bail; } - pz = (DAPL_PZ *) pz_handle; + pz = (struct dapl_pz *)pz_handle; pz_param->ia_handle = (DAT_IA_HANDLE) pz->header.owner_ia; status = DAT_SUCCESS; Index: linux-kernel-resolv/dat-provider/dapl_evd_free.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_evd_free.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_evd_free.c (working copy) @@ -41,13 +41,13 @@ */ u32 dapl_evd_free(DAT_EVD_HANDLE evd_handle) { - DAPL_EVD *evd_ptr; - DAPL_CNO *cno_ptr; + struct dapl_evd *evd_ptr; + struct dapl_cno *cno_ptr; u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_evd_free (%p)\n", evd_handle); - evd_ptr = (DAPL_EVD *) evd_handle; + evd_ptr = (struct dapl_evd *)evd_handle; if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, 0); @@ -72,10 +72,10 @@ u32 dapl_evd_free(DAT_EVD_HANDLE evd_han dapl_ia_link_evd(evd_ptr->header.owner_ia, evd_ptr); } - if (cno_ptr != NULL) - dapl_cno_dealloc (cno_ptr); + if (cno_ptr) + dapl_cno_dealloc(cno_ptr); - bail: +bail: dapl_dbg_log(DAPL_DBG_TYPE_RTN, "dapl_evd_free () returns 0x%x\n", dat_status); Index: linux-kernel-resolv/dat-provider/dapl_lmr_query.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_lmr_query.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_lmr_query.c (working copy) @@ -38,7 +38,7 @@ u32 dapl_lmr_query(DAT_LMR_HANDLE lmr_handle, struct dat_lmr_param *lmr_param) { - DAPL_LMR *lmr; + struct dapl_lmr *lmr; u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, @@ -56,10 +56,10 @@ u32 dapl_lmr_query(DAT_LMR_HANDLE lmr_ha } dat_status = DAT_SUCCESS; - lmr = (DAPL_LMR *) lmr_handle; + lmr = (struct dapl_lmr *)lmr_handle; memcpy(lmr_param, &lmr->param, sizeof *lmr_param); - bail: +bail: return dat_status; } Index: linux-kernel-resolv/dat-provider/dapl_rmr_query.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_rmr_query.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_rmr_query.c (working copy) @@ -40,7 +40,7 @@ u32 dapl_rmr_query(DAT_RMR_HANDLE rmr_handle, struct dat_rmr_param *rmr_param) { - DAPL_RMR *rmr; + struct dapl_rmr *rmr; u32 dat_status = DAT_SUCCESS; if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) { @@ -53,10 +53,10 @@ u32 dapl_rmr_query(DAT_RMR_HANDLE rmr_ha goto bail; } - rmr = (DAPL_RMR *) rmr_handle; + rmr = (struct dapl_rmr *)rmr_handle; memcpy(rmr_param, &rmr->param, sizeof *rmr_param); - bail: +bail: return dat_status; } Index: linux-kernel-resolv/dat-provider/dapl_evd_modify_upcall.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_evd_modify_upcall.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_evd_modify_upcall.c (working copy) @@ -47,14 +47,14 @@ u32 dapl_evd_modify_upcall(DAT_EVD_HANDL enum dat_upcall_policy upcall_policy, /* ignore */ const struct dat_upcall_object *upcall) { - DAPL_EVD *evd_ptr; - DAPL_CNO *cno_ptr; + struct dapl_evd *evd_ptr; + struct dapl_cno *cno_ptr; u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_modify_upcall (%p)\n", evd_handle); - evd_ptr = (DAPL_EVD *) evd_handle; + evd_ptr = (struct dapl_evd *)evd_handle; cno_ptr = evd_ptr->cno_ptr; if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { @@ -62,12 +62,11 @@ u32 dapl_evd_modify_upcall(DAT_EVD_HANDL goto bail; } - if (cno_ptr != NULL) - { + if (cno_ptr != NULL) { cno_ptr->cno_upcall_policy = upcall_policy; cno_ptr->cno_upcall = *upcall; } - bail: +bail: return dat_status; } Index: linux-kernel-resolv/dat-provider/dapl_ia_util.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_ia_util.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_ia_util.c (working copy) @@ -58,9 +58,10 @@ * none * */ -DAPL_IA *dapl_ia_alloc(struct dat_provider *provider, DAPL_HCA * hca_ptr) +struct dapl_ia *dapl_ia_alloc(struct dat_provider *provider, + struct dapl_hca *hca_ptr) { - DAPL_IA *ia_ptr; + struct dapl_ia *ia_ptr; /* Allocate IA */ ia_ptr = kmalloc(sizeof *ia_ptr, GFP_ATOMIC); @@ -106,17 +107,17 @@ DAPL_IA *dapl_ia_alloc(struct dat_provid * Performs an abrupt close of the IA */ -u32 dapl_ia_abrupt_close(DAPL_IA *ia_ptr) +u32 dapl_ia_abrupt_close(struct dapl_ia *ia_ptr) { u32 dat_status = DAT_SUCCESS; - DAPL_EP *ep_ptr, *next_ep_ptr; - DAPL_LMR *lmr_ptr, *next_lmr_ptr; - DAPL_RMR *rmr_ptr, *next_rmr_ptr; - DAPL_PZ *pz_ptr, *next_pz_ptr; - DAPL_EVD *evd_ptr, *next_evd_ptr; - DAPL_SP *sp_ptr, *next_sp_ptr; /* for PSP and RSP queues */ - DAPL_CR *cr_ptr, *next_cr_ptr; - DAPL_HCA *hca_ptr; + struct dapl_ep *ep_ptr, *next_ep_ptr; + struct dapl_lmr *lmr_ptr, *next_lmr_ptr; + struct dapl_rmr *rmr_ptr, *next_rmr_ptr; + struct dapl_pz *pz_ptr, *next_pz_ptr; + struct dapl_evd *evd_ptr, *next_evd_ptr; + struct dapl_sp *sp_ptr, *next_sp_ptr; /* for PSP and RSP queues */ + struct dapl_cr *cr_ptr, *next_cr_ptr; + struct dapl_hca *hca_ptr; /* * clear all the data structures associated with the IA. @@ -135,11 +136,10 @@ u32 dapl_ia_abrupt_close(DAPL_IA *ia_ptr &rmr_ptr->header. ia_list_entry); dat_status = dapl_rmr_free(rmr_ptr); - if (dat_status != DAT_SUCCESS) { + if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): rmr_free(%p) returns %x\n", rmr_ptr, dat_status); - } rmr_ptr = next_rmr_ptr; } @@ -150,11 +150,10 @@ u32 dapl_ia_abrupt_close(DAPL_IA *ia_ptr &sp_ptr->header. ia_list_entry); dat_status = dapl_rsp_free(sp_ptr); - if (dat_status != DAT_SUCCESS) { + if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): rsp_free(%p) returns %x\n", sp_ptr, dat_status); - } sp_ptr = next_sp_ptr; } @@ -176,21 +175,19 @@ u32 dapl_ia_abrupt_close(DAPL_IA *ia_ptr DAT_EP_STATE_DISCONNECT_PENDING) { dat_status = dapl_ep_disconnect(ep_ptr, DAT_CLOSE_ABRUPT_FLAG); - if (dat_status != DAT_SUCCESS) { + if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): ep_disconnect(%p) returns %x\n", ep_ptr, dat_status); - } } /* force the EP into error state to force flush all posted DTOs. */ (void) dapl_modify_qp_state_to_error(ep_ptr->qp_handle); dat_status = dapl_ep_free(ep_ptr); - if (dat_status != DAT_SUCCESS) { + if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): ep_free(%p) returns %x\n", ep_ptr, dat_status); - } ep_ptr = next_ep_ptr; } @@ -201,11 +198,10 @@ u32 dapl_ia_abrupt_close(DAPL_IA *ia_ptr &lmr_ptr->header. ia_list_entry); dat_status = dapl_lmr_free(lmr_ptr); - if (dat_status != DAT_SUCCESS) { + if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): lmr_free(%p) returns %x\n", lmr_ptr, dat_status); - } lmr_ptr = next_lmr_ptr; } @@ -217,11 +213,10 @@ u32 dapl_ia_abrupt_close(DAPL_IA *ia_ptr * should be no competing threads after this. */ dat_status = dapl_ib_remove_conn_listener(ia_ptr, sp_ptr); - if (dat_status != DAT_SUCCESS) { + if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): psp cannot remove listener, returns %x\n", dat_status); - } next_sp_ptr = dapl_llist_next_entry(&ia_ptr->psp_list_head, &sp_ptr->header. @@ -245,11 +240,10 @@ u32 dapl_ia_abrupt_close(DAPL_IA *ia_ptr } dat_status = dapl_psp_free(sp_ptr); - if (dat_status != DAT_SUCCESS) { + if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): psp_free(%p) returns %x\n", sp_ptr, dat_status); - } sp_ptr = next_sp_ptr; } @@ -261,11 +255,10 @@ u32 dapl_ia_abrupt_close(DAPL_IA *ia_ptr &pz_ptr->header. ia_list_entry); dat_status = dapl_pz_free(pz_ptr); - if (dat_status != DAT_SUCCESS) { + if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): pz_free(%p) returns %x\n", pz_ptr, dat_status); - } pz_ptr = next_pz_ptr; } @@ -284,11 +277,10 @@ u32 dapl_ia_abrupt_close(DAPL_IA *ia_ptr if (evd_ptr != ia_ptr->async_error_evd) { /* it isn't the async EVD; delete it. */ dat_status = dapl_evd_free(evd_ptr); - if (dat_status != DAT_SUCCESS) { + if (dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): evd_free(%p) returns %x\n", evd_ptr, dat_status); - } } evd_ptr = next_evd_ptr; } @@ -305,11 +297,10 @@ u32 dapl_ia_abrupt_close(DAPL_IA *ia_ptr atomic_dec(&ia_ptr->async_error_evd->evd_ref_count); dat_status = dapl_evd_free(ia_ptr->async_error_evd); - if (DAT_SUCCESS != dat_status) { + if (DAT_SUCCESS != dat_status) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ia_close(ABRUPT): evd_free(%p) returns %x\n", ia_ptr->async_error_evd, dat_status); - } ia_ptr->async_error_evd = NULL; } @@ -337,13 +328,13 @@ u32 dapl_ia_abrupt_close(DAPL_IA *ia_ptr * */ -u32 dapl_ia_graceful_close(DAPL_IA *ia_ptr) +u32 dapl_ia_graceful_close(struct dapl_ia *ia_ptr) { u32 dat_status = DAT_SUCCESS; u32 cur_dat_status; - DAPL_EVD *evd_ptr; - DAPL_LLIST_ENTRY *entry; - DAPL_HCA *hca_ptr; + struct dapl_evd *evd_ptr; + struct dapl_llist_entry *entry; + struct dapl_hca *hca_ptr; if (!dapl_llist_is_empty(&ia_ptr->rmr_list_head) || !dapl_llist_is_empty(&ia_ptr->rsp_list_head) || @@ -416,14 +407,12 @@ u32 dapl_ia_graceful_close(DAPL_IA *ia_p if (ia_ptr->async_error_evd && (TRUE == ia_ptr->cleanup_async_error_evd)) { cur_dat_status = dapl_ia_teardown_callbacks(ia_ptr); - if (DAT_SUCCESS != cur_dat_status) { + if (DAT_SUCCESS != cur_dat_status) dat_status = cur_dat_status; - } atomic_dec(&ia_ptr->async_error_evd->evd_ref_count); cur_dat_status = dapl_evd_free(ia_ptr->async_error_evd); - if (DAT_SUCCESS != cur_dat_status) { + if (DAT_SUCCESS != cur_dat_status) dat_status = cur_dat_status; - } ia_ptr->async_error_evd = NULL; } @@ -432,14 +421,14 @@ u32 dapl_ia_graceful_close(DAPL_IA *ia_p dapl_ia_free(ia_ptr); - bail: +bail: return dat_status; } /* * free an IA INFO struct */ -void dapl_ia_free(DAPL_IA * ia_ptr) +void dapl_ia_free(struct dapl_ia *ia_ptr) { dapl_os_assert(ia_ptr->header.magic == DAPL_MAGIC_IA); @@ -464,7 +453,7 @@ void dapl_ia_free(DAPL_IA * ia_ptr) /* * Add an ep to the IA structure */ -void dapl_ia_link_ep(DAPL_IA * ia_ptr, DAPL_EP * ep_ptr) +void dapl_ia_link_ep(struct dapl_ia *ia_ptr, struct dapl_ep *ep_ptr) { spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); dapl_llist_add_head(&ia_ptr->ep_list_head, @@ -475,7 +464,7 @@ void dapl_ia_link_ep(DAPL_IA * ia_ptr, D /* * Remove an ep from the ia info structure */ -void dapl_ia_unlink_ep(DAPL_IA * ia_ptr, DAPL_EP * ep_ptr) +void dapl_ia_unlink_ep(struct dapl_ia *ia_ptr, struct dapl_ep *ep_ptr) { spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); dapl_llist_remove_entry(&ia_ptr->ep_list_head, @@ -486,7 +475,7 @@ void dapl_ia_unlink_ep(DAPL_IA * ia_ptr, /* * Add an srq to the IA structure */ -void dapl_ia_link_srq(DAPL_IA * ia_ptr, DAPL_SRQ * srq_ptr) +void dapl_ia_link_srq(struct dapl_ia *ia_ptr, struct dapl_srq *srq_ptr) { spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); dapl_llist_add_head(&ia_ptr->srq_list_head, @@ -497,7 +486,7 @@ void dapl_ia_link_srq(DAPL_IA * ia_ptr, /* * Remove an srq from the ia info structure */ -void dapl_ia_unlink_srq(DAPL_IA * ia_ptr, DAPL_SRQ * srq_ptr) +void dapl_ia_unlink_srq(struct dapl_ia *ia_ptr, struct dapl_srq *srq_ptr) { spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); dapl_llist_remove_entry(&ia_ptr->srq_list_head, @@ -508,7 +497,7 @@ void dapl_ia_unlink_srq(DAPL_IA * ia_ptr /* * Add an lmr to the IA structure */ -void dapl_ia_link_lmr(DAPL_IA * ia_ptr, DAPL_LMR * lmr_ptr) +void dapl_ia_link_lmr(struct dapl_ia *ia_ptr, struct dapl_lmr *lmr_ptr) { spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); dapl_llist_add_head(&ia_ptr->lmr_list_head, @@ -519,7 +508,7 @@ void dapl_ia_link_lmr(DAPL_IA * ia_ptr, /* * Remove an lmr from the ia info structure */ -void dapl_ia_unlink_lmr(DAPL_IA * ia_ptr, DAPL_LMR * lmr_ptr) +void dapl_ia_unlink_lmr(struct dapl_ia *ia_ptr, struct dapl_lmr *lmr_ptr) { spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); dapl_llist_remove_entry(&ia_ptr->lmr_list_head, @@ -530,7 +519,7 @@ void dapl_ia_unlink_lmr(DAPL_IA * ia_ptr /* * Add an rmr to the IA structure */ -void dapl_ia_link_rmr(DAPL_IA * ia_ptr, DAPL_RMR * rmr_ptr) +void dapl_ia_link_rmr(struct dapl_ia *ia_ptr, struct dapl_rmr *rmr_ptr) { spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); dapl_llist_add_head(&ia_ptr->rmr_list_head, @@ -541,7 +530,7 @@ void dapl_ia_link_rmr(DAPL_IA * ia_ptr, /* * Remove an rmr from the ia info structure */ -void dapl_ia_unlink_rmr(DAPL_IA * ia_ptr, DAPL_RMR * rmr_ptr) +void dapl_ia_unlink_rmr(struct dapl_ia *ia_ptr, struct dapl_rmr *rmr_ptr) { spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); dapl_llist_remove_entry(&ia_ptr->rmr_list_head, @@ -552,7 +541,7 @@ void dapl_ia_unlink_rmr(DAPL_IA * ia_ptr /* * Add an pz to the IA structure */ -void dapl_ia_link_pz(DAPL_IA * ia_ptr, DAPL_PZ * pz_ptr) +void dapl_ia_link_pz(struct dapl_ia *ia_ptr, struct dapl_pz *pz_ptr) { spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); dapl_llist_add_head(&ia_ptr->pz_list_head, @@ -563,7 +552,7 @@ void dapl_ia_link_pz(DAPL_IA * ia_ptr, D /* * Remove an pz from the ia info structure */ -void dapl_ia_unlink_pz(DAPL_IA * ia_ptr, DAPL_PZ * pz_ptr) +void dapl_ia_unlink_pz(struct dapl_ia *ia_ptr, struct dapl_pz *pz_ptr) { spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); dapl_llist_remove_entry(&ia_ptr->pz_list_head, @@ -574,7 +563,7 @@ void dapl_ia_unlink_pz(DAPL_IA * ia_ptr, /* * Add an evd to the IA structure */ -void dapl_ia_link_evd(DAPL_IA * ia_ptr, DAPL_EVD * evd_ptr) +void dapl_ia_link_evd(struct dapl_ia *ia_ptr, struct dapl_evd *evd_ptr) { spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); dapl_llist_add_head(&ia_ptr->evd_list_head, @@ -585,7 +574,7 @@ void dapl_ia_link_evd(DAPL_IA * ia_ptr, /* * Remove an evd from the ia info structure */ -void dapl_ia_unlink_evd(DAPL_IA * ia_ptr, DAPL_EVD * evd_ptr) +void dapl_ia_unlink_evd(struct dapl_ia *ia_ptr, struct dapl_evd *evd_ptr) { spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); dapl_llist_remove_entry(&ia_ptr->evd_list_head, @@ -596,7 +585,7 @@ void dapl_ia_unlink_evd(DAPL_IA * ia_ptr /* * Add an psp to the IA structure */ -void dapl_ia_link_psp(DAPL_IA * ia_ptr, DAPL_SP * sp_ptr) +void dapl_ia_link_psp(struct dapl_ia *ia_ptr, struct dapl_sp *sp_ptr) { spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); dapl_llist_add_head(&ia_ptr->psp_list_head, @@ -607,9 +596,9 @@ void dapl_ia_link_psp(DAPL_IA * ia_ptr, /* * Remove an sp from the appropriate ia rsp or psp queue */ -void dapl_ia_unlink_sp(DAPL_IA * ia_ptr, DAPL_SP * sp_ptr) +void dapl_ia_unlink_sp(struct dapl_ia *ia_ptr, struct dapl_sp *sp_ptr) { - DAPL_LLIST_HEAD *list_head; + struct dapl_llist_entry **list_head; if (sp_ptr->header.handle_type == DAT_HANDLE_TYPE_PSP) { list_head = &ia_ptr->psp_list_head; @@ -640,17 +629,16 @@ void dapl_ia_unlink_sp(DAPL_IA * ia_ptr, * none * */ -DAPL_SP *dapl_ia_sp_search(DAPL_IA * ia_ptr, - DAT_CONN_QUAL conn_qual, boolean_t is_psp) +struct dapl_sp *dapl_ia_sp_search(struct dapl_ia *ia_ptr, + DAT_CONN_QUAL conn_qual, boolean_t is_psp) { - DAPL_SP *sp_ptr; - DAPL_LLIST_HEAD *list_head; + struct dapl_sp *sp_ptr; + struct dapl_llist_entry **list_head; - if (is_psp) { + if (is_psp) list_head = &ia_ptr->psp_list_head; - } else { + else list_head = &ia_ptr->rsp_list_head; - } spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); @@ -686,7 +674,7 @@ DAPL_SP *dapl_ia_sp_search(DAPL_IA * ia_ * none * */ -void dapl_ia_link_rsp(DAPL_IA * ia_ptr, DAPL_SP * sp_ptr) +void dapl_ia_link_rsp(struct dapl_ia *ia_ptr, struct dapl_sp *sp_ptr) { spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); dapl_llist_add_head(&ia_ptr->rsp_list_head, @@ -694,7 +682,8 @@ void dapl_ia_link_rsp(DAPL_IA * ia_ptr, spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); } -u32 dapl_ia_setup_callbacks(DAPL_IA * ia_ptr, DAPL_EVD * async_evd_ptr) +u32 dapl_ia_setup_callbacks(struct dapl_ia *ia_ptr, + struct dapl_evd *async_evd_ptr) { u32 dat_status = DAT_SUCCESS; @@ -717,7 +706,7 @@ u32 dapl_ia_setup_callbacks(DAPL_IA * ia return dat_status; } -u32 dapl_ia_teardown_callbacks(DAPL_IA * ia_ptr) +u32 dapl_ia_teardown_callbacks(struct dapl_ia *ia_ptr) { u32 dat_status = DAT_SUCCESS; Index: linux-kernel-resolv/dat-provider/dapl_pz_util.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_pz_util.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_pz_util.c (working copy) @@ -52,13 +52,13 @@ * none * */ -DAPL_PZ *dapl_pz_alloc(DAPL_IA * ia) +struct dapl_pz *dapl_pz_alloc(struct dapl_ia *ia) { - DAPL_PZ *pz; + struct dapl_pz *pz; /* Allocate PZ */ pz = kmalloc(sizeof *pz, GFP_ATOMIC); - if (!pz) + if (!pz) return NULL; /* zero the structure */ @@ -100,9 +100,10 @@ DAPL_PZ *dapl_pz_alloc(DAPL_IA * ia) * none * */ -void dapl_pz_dealloc(DAPL_PZ * pz) +void dapl_pz_dealloc(struct dapl_pz *pz) { - pz->header.magic = DAPL_MAGIC_INVALID; /* reset magic to prevent reuse */ + /* reset magic to prevent reuse */ + pz->header.magic = DAPL_MAGIC_INVALID; dapl_ia_unlink_pz(pz->header.owner_ia, pz); /* no need to destroy pz->header.lock */ Index: linux-kernel-resolv/dat-provider/dapl_ep_query.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_ep_query.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_ep_query.c (working copy) @@ -59,14 +59,14 @@ */ u32 dapl_ep_query(DAT_EP_HANDLE ep_handle, struct dat_ep_param *ep_param) { - DAPL_EP *ep_ptr; + struct dapl_ep *ep_ptr; u32 status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_query (%p, %p)\n", ep_handle, ep_param); - ep_ptr = (DAPL_EP *) ep_handle; + ep_ptr = (struct dapl_ep *)ep_handle; /* * Verify parameter & state Index: linux-kernel-resolv/dat-provider/dapl_pz_util.h =================================================================== --- linux-kernel-resolv/dat-provider/dapl_pz_util.h (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_pz_util.h (working copy) @@ -39,8 +39,8 @@ #include "dapl.h" -extern DAPL_PZ *dapl_pz_alloc(DAPL_IA * ia); +extern struct dapl_pz *dapl_pz_alloc(struct dapl_ia *ia); -extern void dapl_pz_dealloc(DAPL_PZ * pz); +extern void dapl_pz_dealloc(struct dapl_pz *pz); -#endif /* DAPL_PZ_UTIL_H */ +#endif /* DAPL_PZ_UTIL_H */ Index: linux-kernel-resolv/dat-provider/dapl_evd_dequeue.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_evd_dequeue.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_evd_dequeue.c (working copy) @@ -64,14 +64,14 @@ u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_handle, struct dat_event *event) { - DAPL_EVD *evd_ptr; + struct dapl_evd *evd_ptr; struct dat_event *local_event; u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_evd_dequeue (%p, %p)\n", evd_handle, event); - evd_ptr = (DAPL_EVD *) evd_handle; + evd_ptr = (struct dapl_evd *)evd_handle; if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, 0); Index: linux-kernel-resolv/dat-provider/dapl_openib_cm.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_openib_cm.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_openib_cm.c (working copy) @@ -399,12 +399,12 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han DAT_CONN_QUAL remote_conn_qual, int private_data_size, void *private_data) { - DAPL_IA *ia_ptr; - DAPL_EP *ep_ptr; + struct dapl_ia *ia_ptr; + struct dapl_ep *ep_ptr; struct dapl_cm_id *conn; int status; - ep_ptr = (DAPL_EP *)ep_handle; + ep_ptr = (struct dapl_ep *)ep_handle; if (ep_ptr->qp_handle == NULL) { printk(KERN_ERR "dapl_ib_connect: qp handle invalid\n"); return DAT_INTERNAL_ERROR; @@ -486,7 +486,7 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han * DAT_INVALID_PARAMETER * */ -u32 dapl_ib_disconnect(DAPL_EP *ep_ptr, enum dat_close_flags close_flags) +u32 dapl_ib_disconnect(struct dapl_ep *ep_ptr, enum dat_close_flags close_flags) { struct dapl_cm_id *conn = ep_ptr->cm_handle; int status; @@ -510,24 +510,24 @@ u32 dapl_ib_disconnect(DAPL_EP *ep_ptr, return DAT_SUCCESS; } -/* - * dapl_ib_disconnect_clean - * - * Clean up outstanding connection data. This routine is invoked +/* + * dapl_ib_disconnect_clean + * + * Clean up outstanding connection data. This routine is invoked * after the final disconnect callback has occurred. - * - * Input: - * ep_ptr DAPL_EP - * active Indicates active side of connection - * - * Output: - * none - * - * Returns: - * void - * + * + * Input: + * ep_ptr struct dapl_ep + * active Indicates active side of connection + * + * Output: + * none + * + * Returns: + * void + * */ -void dapl_ib_disconnect_clean(DAPL_EP * ep_ptr, boolean_t active) +void dapl_ib_disconnect_clean(struct dapl_ep *ep_ptr, boolean_t active) { int status; @@ -566,9 +566,9 @@ void dapl_ib_disconnect_clean(DAPL_EP * * void * */ -void dapl_ib_reinit_ep(DAPL_EP * ep_ptr) +void dapl_ib_reinit_ep(struct dapl_ep *ep_ptr) { - DAPL_IA *ia_ptr; + struct dapl_ia *ia_ptr; int ib_status; ia_ptr = ep_ptr->header.owner_ia; @@ -615,7 +615,8 @@ void dapl_ib_reinit_ep(DAPL_EP * ep_ptr) * DAT_INVALID_PARAMETER * */ -u32 dapl_ib_setup_conn_listener(DAPL_IA *ia_ptr, u64 ServiceID, DAPL_SP *sp_ptr) +u32 dapl_ib_setup_conn_listener(struct dapl_ia *ia_ptr, u64 ServiceID, + struct dapl_sp *sp_ptr) { int status; @@ -658,7 +659,7 @@ u32 dapl_ib_setup_conn_listener(DAPL_IA * DAT_INVALID_PARAMETER * */ -u32 dapl_ib_remove_conn_listener(DAPL_IA *ia_ptr, DAPL_SP *sp_ptr) +u32 dapl_ib_remove_conn_listener(struct dapl_ia *ia_ptr, struct dapl_sp *sp_ptr) { //*** This will hang if called from CM thread context... //*** Move back to using WQ... @@ -731,16 +732,16 @@ u32 dapl_ib_accept_connection(DAT_CR_HAN int private_data_size, const void *prd_ptr) { - DAPL_CR *cr_ptr; - DAPL_EP *ep_ptr; - DAPL_IA *ia_ptr; + struct dapl_cr *cr_ptr; + struct dapl_ep *ep_ptr; + struct dapl_ia *ia_ptr; u32 dat_status; int status; struct ib_cm_rep_param passive_params; struct dapl_cm_id *conn; - cr_ptr = (DAPL_CR *) cr_handle; - ep_ptr = (DAPL_EP *) ep_handle; + cr_ptr = (struct dapl_cr *)cr_handle; + ep_ptr = (struct dapl_ep *)ep_handle; ia_ptr = ep_ptr->header.owner_ia; conn = cr_ptr->ib_cm_handle; @@ -833,13 +834,13 @@ static int ib_cm_get_remote_gid(struct d u32 dapl_ib_cm_remote_addr(DAT_HANDLE dat_handle, struct sockaddr_in6 *remote_ia_address) { - DAPL_IA *ia_ptr; + struct dapl_ia *ia_ptr; int rc; union ib_gid remote_gid; - DAPL_CR *cr_ptr = (DAPL_CR *) dat_handle; + struct dapl_cr *cr_ptr = (struct dapl_cr *)dat_handle; - ia_ptr = (DAPL_IA *) cr_ptr->sp_ptr->ia_handle; + ia_ptr = (struct dapl_ia *)cr_ptr->sp_ptr->ia_handle; rc = ib_cm_get_remote_gid(cr_ptr->ib_cm_handle, (u8 *) & remote_gid); if (rc < 0) Index: linux-kernel-resolv/dat-provider/dapl_ia_util.h =================================================================== --- linux-kernel-resolv/dat-provider/dapl_ia_util.h (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_ia_util.h (working copy) @@ -39,49 +39,51 @@ #include "dapl.h" -DAPL_IA *dapl_ia_alloc(struct dat_provider *provider, DAPL_HCA * hca_ptr); +struct dapl_ia *dapl_ia_alloc(struct dat_provider *provider, + struct dapl_hca *hca_ptr); -u32 dapl_ia_abrupt_close(DAPL_IA *ia_ptr); +u32 dapl_ia_abrupt_close(struct dapl_ia *ia_ptr); -u32 dapl_ia_graceful_close(DAPL_IA *ia_ptr); +u32 dapl_ia_graceful_close(struct dapl_ia *ia_ptr); -void dapl_ia_free(DAPL_IA * ia_ptr); +void dapl_ia_free(struct dapl_ia *ia_ptr); -void dapl_ia_link_ep(DAPL_IA * ia_ptr, DAPL_EP * ep_info); +void dapl_ia_link_ep(struct dapl_ia *ia_ptr, struct dapl_ep *ep_info); -void dapl_ia_unlink_ep(DAPL_IA * ia_ptr, DAPL_EP * ep_info); +void dapl_ia_unlink_ep(struct dapl_ia *ia_ptr, struct dapl_ep *ep_info); -void dapl_ia_link_srq(DAPL_IA * ia_ptr, DAPL_SRQ * srq_ptr); +void dapl_ia_link_srq(struct dapl_ia *ia_ptr, struct dapl_srq *srq_ptr); -void dapl_ia_unlink_srq(DAPL_IA * ia_ptr, DAPL_SRQ * srq_ptr); +void dapl_ia_unlink_srq(struct dapl_ia *ia_ptr, struct dapl_srq *srq_ptr); -void dapl_ia_link_lmr(DAPL_IA * ia_ptr, DAPL_LMR * lmr_info); +void dapl_ia_link_lmr(struct dapl_ia *ia_ptr, struct dapl_lmr *lmr_info); -void dapl_ia_unlink_lmr(DAPL_IA * ia_ptr, DAPL_LMR * lmr_info); +void dapl_ia_unlink_lmr(struct dapl_ia *ia_ptr, struct dapl_lmr *lmr_info); -void dapl_ia_link_rmr(DAPL_IA * ia_ptr, DAPL_RMR * rmr_info); +void dapl_ia_link_rmr(struct dapl_ia *ia_ptr, struct dapl_rmr *rmr_info); -void dapl_ia_unlink_rmr(DAPL_IA * ia_ptr, DAPL_RMR * rmr_info); +void dapl_ia_unlink_rmr(struct dapl_ia *ia_ptr, struct dapl_rmr *rmr_info); -void dapl_ia_link_pz(DAPL_IA * ia_ptr, DAPL_PZ * pz_info); +void dapl_ia_link_pz(struct dapl_ia *ia_ptr, struct dapl_pz *pz_info); -void dapl_ia_unlink_pz(DAPL_IA * ia_ptr, DAPL_PZ * pz_info); +void dapl_ia_unlink_pz(struct dapl_ia *ia_ptr, struct dapl_pz *pz_info); -void dapl_ia_link_evd(DAPL_IA * ia_ptr, DAPL_EVD * evd_info); +void dapl_ia_link_evd(struct dapl_ia *ia_ptr, struct dapl_evd *evd_info); -void dapl_ia_unlink_evd(DAPL_IA * ia_ptr, DAPL_EVD * evd_info); +void dapl_ia_unlink_evd(struct dapl_ia *ia_ptr, struct dapl_evd *evd_info); -void dapl_ia_link_psp(DAPL_IA * ia_ptr, DAPL_SP * sp_info); +void dapl_ia_link_psp(struct dapl_ia *ia_ptr, struct dapl_sp *sp_info); -void dapl_ia_unlink_sp(DAPL_IA * ia_ptr, DAPL_SP * sp_info); +void dapl_ia_unlink_sp(struct dapl_ia *ia_ptr, struct dapl_sp *sp_info); -void dapl_ia_link_rsp(DAPL_IA * ia_ptr, DAPL_SP * sp_info); +void dapl_ia_link_rsp(struct dapl_ia *ia_ptr, struct dapl_sp *sp_info); -DAPL_SP *dapl_ia_sp_search(DAPL_IA * ia_ptr, +struct dapl_sp *dapl_ia_sp_search(struct dapl_ia *ia_ptr, DAT_CONN_QUAL conn_qual, boolean_t is_psp); -u32 dapl_ia_setup_callbacks(DAPL_IA *ia_ptr, DAPL_EVD *async_evd_ptr); +u32 dapl_ia_setup_callbacks(struct dapl_ia *ia_ptr, + struct dapl_evd *async_evd_ptr); -u32 dapl_ia_teardown_callbacks(DAPL_IA *ia_ptr); +u32 dapl_ia_teardown_callbacks(struct dapl_ia *ia_ptr); #endif Index: linux-kernel-resolv/dat-provider/dapl_rmr_bind.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_rmr_bind.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_rmr_bind.c (working copy) @@ -46,16 +46,16 @@ * * *********************************************************************/ -static inline u32 dapl_rmr_bind_fuse(DAPL_RMR *rmr, +static inline u32 dapl_rmr_bind_fuse(struct dapl_rmr *rmr, const struct dat_lmr_triplet *lmr_triplet, enum dat_mem_priv_flags mem_priv, - DAPL_EP *ep_ptr, + struct dapl_ep *ep_ptr, DAT_RMR_COOKIE user_cookie, enum dat_completion_flags completion_flags, DAT_RMR_CONTEXT *rmr_context) { - DAPL_LMR *lmr; - DAPL_COOKIE *cookie; + struct dapl_lmr *lmr; + struct dapl_cookie *cookie; u32 status; boolean_t is_signaled; @@ -152,15 +152,16 @@ static inline u32 dapl_rmr_bind_fuse(DAP if (NULL != rmr_context) *rmr_context = rmr->param.rmr_context; - bail: +bail: return status; } -static inline u32 dapl_rmr_bind_unfuse(DAPL_RMR *rmr, DAPL_EP *ep_ptr, +static inline u32 dapl_rmr_bind_unfuse(struct dapl_rmr *rmr, + struct dapl_ep *ep_ptr, DAT_RMR_COOKIE user_cookie, enum dat_completion_flags cflags) { - DAPL_COOKIE *cookie; + struct dapl_cookie *cookie; u32 status = DAT_SUCCESS; boolean_t is_signaled; @@ -232,7 +233,7 @@ static inline u32 dapl_rmr_bind_unfuse(D rmr->ep = ep_ptr; rmr->lmr = NULL; - bail1: +bail1: return status; } @@ -254,8 +255,8 @@ u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_han enum dat_completion_flags completion_flags, DAT_RMR_CONTEXT *rmr_context) { - DAPL_RMR *rmr; - DAPL_EP *ep_ptr; + struct dapl_rmr *rmr; + struct dapl_ep *ep_ptr; if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); @@ -263,16 +264,15 @@ u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_han if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); - rmr = (DAPL_RMR *) rmr_handle; - ep_ptr = (DAPL_EP *) ep_handle; + rmr = (struct dapl_rmr *)rmr_handle; + ep_ptr = (struct dapl_ep *)ep_handle; /* if the rmr should be bound */ - if (0 != lmr_triplet->segment_length) { + if (0 != lmr_triplet->segment_length) return dapl_rmr_bind_fuse(rmr, lmr_triplet, mem_priv, ep_ptr, user_cookie, completion_flags, rmr_context); - } else { /* the rmr should be unbound */ + else /* the rmr should be unbound */ return dapl_rmr_bind_unfuse(rmr, ep_ptr, user_cookie, completion_flags); - } } Index: linux-kernel-resolv/dat-provider/dapl_cr_reject.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_cr_reject.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_cr_reject.c (working copy) @@ -42,11 +42,11 @@ */ u32 dapl_cr_reject(DAT_CR_HANDLE cr_handle) { - DAPL_CR *cr_ptr; - DAPL_EP *ep_ptr; + struct dapl_cr *cr_ptr; + struct dapl_ep *ep_ptr; enum dat_ep_state entry_ep_state; DAT_EP_HANDLE entry_ep_handle; - DAPL_SP *sp_ptr; + struct dapl_sp *sp_ptr; u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_cr_reject (%p)\n", cr_handle); @@ -57,16 +57,16 @@ u32 dapl_cr_reject(DAT_CR_HANDLE cr_hand goto bail; } - cr_ptr = (DAPL_CR *) cr_handle; + cr_ptr = (struct dapl_cr *)cr_handle; /* * Clean up provider created EP if there is one: only if * DAT_PSP_PROVIDER_FLAG was set on the PSP */ - ep_ptr = (DAPL_EP *) cr_ptr->param.local_ep_handle; + ep_ptr = (struct dapl_ep *)cr_ptr->param.local_ep_handle; entry_ep_handle = cr_ptr->param.local_ep_handle; entry_ep_state = 0; - if (ep_ptr != NULL) { + if (ep_ptr) { entry_ep_state = ep_ptr->param.ep_state; ep_ptr->param.ep_state = DAT_EP_STATE_UNCONNECTED; cr_ptr->param.local_ep_handle = NULL; @@ -87,10 +87,8 @@ u32 dapl_cr_reject(DAT_CR_HANDLE cr_hand * RSP and user-provided EPs are in the control of the user. */ sp_ptr = cr_ptr->sp_ptr; - if (ep_ptr != NULL && - sp_ptr->psp_flags == DAT_PSP_PROVIDER_FLAG) { + if (ep_ptr && sp_ptr->psp_flags == DAT_PSP_PROVIDER_FLAG) (void)dapl_ep_free(ep_ptr); - } /* Remove the CR from the queue, then free it */ spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); @@ -101,6 +99,6 @@ u32 dapl_cr_reject(DAT_CR_HANDLE cr_hand dapl_cr_free(cr_ptr); } - bail: +bail: return dat_status; } Index: linux-kernel-resolv/dat-provider/dapl_ep_reset.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_ep_reset.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_ep_reset.c (working copy) @@ -63,10 +63,10 @@ */ u32 dapl_ep_reset(DAT_EP_HANDLE ep_handle) { - DAPL_EP *ep_ptr; + struct dapl_ep *ep_ptr; u32 dat_status = DAT_SUCCESS; - ep_ptr = (DAPL_EP *) ep_handle; + ep_ptr = (struct dapl_ep *)ep_handle; /* * Verify parameter & state Index: linux-kernel-resolv/dat-provider/dapl_ep_recv_query.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_ep_recv_query.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_ep_recv_query.c (working copy) @@ -57,13 +57,13 @@ u32 dapl_ep_recv_query(DAT_EP_HANDLE ep_handle, int *nbufs_allocate, int *bufs_alloc_span) { - DAPL_EP *ep_ptr; + struct dapl_ep *ep_ptr; u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_recv_query (%p, %p, %p)\n", ep_handle, nbufs_allocate, bufs_alloc_span); - ep_ptr = (DAPL_EP *) ep_handle; + ep_ptr = (struct dapl_ep *)ep_handle; /* * Verify parameter & state Index: linux-kernel-resolv/dat-provider/dapl_openib_util.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_openib_util.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_openib_util.c (working copy) @@ -159,7 +159,7 @@ static inline int dapl_ib_convert_mem_pr * DAT_INSUFFICIENT_RESOURCES * */ -u32 dapl_ib_open_hca(char *name, DAPL_HCA * hca_ptr) +u32 dapl_ib_open_hca(char *name, struct dapl_hca *hca_ptr) { u32 dat_ret; @@ -196,7 +196,7 @@ u32 dapl_ib_open_hca(char *name, DAPL_HC * DAT_INSUFFICIENT_RESOURCES * */ -u32 dapl_ib_close_hca(DAPL_HCA *hca_ptr) +u32 dapl_ib_close_hca(struct dapl_hca *hca_ptr) { return DAT_SUCCESS; } @@ -219,8 +219,8 @@ u32 dapl_ib_close_hca(DAPL_HCA *hca_ptr) * DAT_INSUFFICIENT_RESOURCES * */ -u32 dapl_ib_cq_alloc(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr, int *cqlen, - ib_comp_handler callback) +u32 dapl_ib_cq_alloc(struct dapl_ia *ia_ptr, struct dapl_evd *evd_ptr, + int *cqlen, ib_comp_handler callback) { int ib_status; @@ -253,7 +253,7 @@ u32 dapl_ib_cq_alloc(DAPL_IA *ia_ptr, DA * DAT_INSUFFICIENT_RESOURCES * */ -u32 dapl_ib_cq_free(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr) +u32 dapl_ib_cq_free(struct dapl_ia *ia_ptr, struct dapl_evd *evd_ptr) { int ib_status; @@ -261,7 +261,8 @@ u32 dapl_ib_cq_free(DAPL_IA *ia_ptr, DAP return dapl_ib_status_convert(ib_status); } -u32 dapl_ib_cq_resize(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr, int *cqlen) +u32 dapl_ib_cq_resize(struct dapl_ia *ia_ptr, struct dapl_evd *evd_ptr, + int *cqlen) { int ib_status; @@ -293,7 +294,7 @@ u32 dapl_ib_cq_resize(DAPL_IA *ia_ptr, D * DAT_INSUFFICIENT_RESOURCES * */ -u32 dapl_set_cq_notify(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr) +u32 dapl_set_cq_notify(struct dapl_ia *ia_ptr, struct dapl_evd *evd_ptr) { int ib_status; @@ -318,7 +319,7 @@ u32 dapl_set_cq_notify(DAPL_IA *ia_ptr, * DAT_INSUFFICIENT_RESOURCES * */ -u32 dapl_ib_pd_alloc(DAPL_IA *ia, DAPL_PZ *pz) +u32 dapl_ib_pd_alloc(struct dapl_ia *ia, struct dapl_pz *pz) { struct ib_pd *pd; int ib_status; @@ -353,7 +354,7 @@ u32 dapl_ib_pd_alloc(DAPL_IA *ia, DAPL_P * DAT_INSUFFICIENT_RESOURCES * */ -u32 dapl_ib_pd_free(DAPL_PZ *pz) +u32 dapl_ib_pd_free(struct dapl_pz *pz) { int ib_status; @@ -388,14 +389,15 @@ u32 dapl_ib_pd_free(DAPL_PZ *pz) * DAT_INSUFFICIENT_RESOURCES * */ -u32 dapl_ib_mr_register(DAPL_IA *ia, DAPL_LMR *lmr, void *virt_addr, - u64 length, enum dat_mem_priv_flags privileges) +u32 dapl_ib_mr_register(struct dapl_ia *ia, struct dapl_lmr *lmr, + void *virt_addr, u64 length, + enum dat_mem_priv_flags privileges) { /* ITBD virtual memory registration ??? */ return DAT_SUCCESS; } -u32 dapl_ib_mr_register_physical(DAPL_IA *ia_ptr, DAPL_LMR *lmr, +u32 dapl_ib_mr_register_physical(struct dapl_ia *ia_ptr, struct dapl_lmr *lmr, void *phys_addr, u64 length, enum dat_mem_priv_flags privileges) { @@ -421,7 +423,7 @@ u32 dapl_ib_mr_register_physical(DAPL_IA iova = buf_list[0].addr; acl = dapl_ib_convert_mem_privileges(privileges); acl |= IB_ACCESS_MW_BIND; - mr = ib_reg_phys_mr((struct ib_pd *)((DAPL_PZ *) lmr->param.pz_handle)-> + mr = ib_reg_phys_mr((struct ib_pd *)((struct dapl_pz *)lmr->param.pz_handle)-> pd_handle, buf_list, length, acl, &iova); kfree(buf_list); if (IS_ERR(mr)) { @@ -469,7 +471,7 @@ u32 dapl_ib_mr_register_physical(DAPL_IA * DAT_INSUFFICIENT_RESOURCES * */ -u32 dapl_ib_mr_deregister(DAPL_LMR *lmr) +u32 dapl_ib_mr_deregister(struct dapl_lmr *lmr) { int ib_status; @@ -503,12 +505,12 @@ u32 dapl_ib_mr_deregister(DAPL_LMR *lmr) * DAT_INSUFFICIENT_RESOURCES * */ -u32 dapl_ib_mw_alloc(DAPL_RMR *rmr) +u32 dapl_ib_mw_alloc(struct dapl_rmr *rmr) { int ib_status; struct ib_mw *mw; - mw = ib_alloc_mw((struct ib_pd *)((DAPL_PZ *) rmr->param.pz_handle)-> + mw = ib_alloc_mw((struct ib_pd *)((struct dapl_pz *)rmr->param.pz_handle)-> pd_handle); if (IS_ERR(mw)) { ib_status = PTR_ERR(mw); @@ -540,7 +542,7 @@ u32 dapl_ib_mw_alloc(DAPL_RMR *rmr) * DAT_INSUFFICIENT_RESOURCES * */ -u32 dapl_ib_mw_free(DAPL_RMR *rmr) +u32 dapl_ib_mw_free(struct dapl_rmr *rmr) { int ib_status; @@ -573,10 +575,10 @@ u32 dapl_ib_mw_free(DAPL_RMR *rmr) * DAT_INSUFFICIENT_RESOURCES * */ -u32 dapl_ib_mw_bind(DAPL_RMR *rmr, DAPL_LMR *lmr, DAPL_EP *ep, - DAPL_COOKIE *cookie, u64 virtual_address, - u64 length, enum dat_mem_priv_flags mem_priv, - boolean_t is_signaled) +u32 dapl_ib_mw_bind(struct dapl_rmr *rmr, struct dapl_lmr *lmr, + struct dapl_ep *ep, struct dapl_cookie *cookie, + u64 virtual_address, u64 length, + enum dat_mem_priv_flags mem_priv, boolean_t is_signaled) { int ib_status; struct ib_mw_bind mw_bind_prop; @@ -624,8 +626,8 @@ u32 dapl_ib_mw_bind(DAPL_RMR *rmr, DAPL_ * DAT_INSUFFICIENT_RESOURCES * */ -u32 dapl_ib_mw_unbind(DAPL_RMR *rmr, DAPL_EP *ep, DAPL_COOKIE *cookie, - boolean_t is_signaled) +u32 dapl_ib_mw_unbind(struct dapl_rmr *rmr, struct dapl_ep *ep, + struct dapl_cookie *cookie, boolean_t is_signaled) { int ib_status; struct ib_mw_bind mw_bind_prop; @@ -671,7 +673,7 @@ u32 dapl_ib_mw_unbind(DAPL_RMR *rmr, DAP * DAT_INVALID_PARAMETER * */ -u32 dapl_ib_setup_async_callback(DAPL_IA *ia_ptr, +u32 dapl_ib_setup_async_callback(struct dapl_ia *ia_ptr, DAPL_ASYNC_HANDLER_TYPE handler_type, ib_async_handler_t callback, void *context) { @@ -696,7 +698,7 @@ u32 dapl_ib_setup_async_callback(DAPL_IA * DAT_SUCCESS * DAT_INVALID_PARAMETER */ -u32 dapl_ib_query_hca(DAPL_HCA *hca_ptr, struct dat_ia_attr *ia_attr, +u32 dapl_ib_query_hca(struct dapl_hca *hca_ptr, struct dat_ia_attr *ia_attr, struct dat_ep_attr *ep_attr, struct sockaddr_in6 *ip_addr) { int ib_status; @@ -797,7 +799,7 @@ u32 dapl_ib_query_hca(DAPL_HCA *hca_ptr, return DAT_SUCCESS; } -u32 dapl_ib_completion_poll(DAPL_HCA *hca_ptr, DAPL_EVD *evd_ptr, +u32 dapl_ib_completion_poll(struct dapl_hca *hca_ptr, struct dapl_evd *evd_ptr, struct ib_wc *cqe_ptr) { int ib_status; @@ -810,7 +812,7 @@ u32 dapl_ib_completion_poll(DAPL_HCA *hc return dapl_ib_status_convert(ib_status); } -u32 dapl_ib_completion_notify(DAPL_EVD *evd_ptr) +u32 dapl_ib_completion_notify(struct dapl_evd *evd_ptr) { int ib_status; Index: linux-kernel-resolv/dat-provider/dapl_ep_modify.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_ep_modify.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_ep_modify.c (working copy) @@ -56,12 +56,12 @@ static inline u32 dapl_ep_modify_validate_parameters(DAT_EP_HANDLE ep_handle, enum dat_ep_param_mask mask, const struct dat_ep_param *ep_param, - DAPL_IA ** ia_ptr, - DAPL_EP ** ep_ptr, + struct dapl_ia **ia_ptr, + struct dapl_ep **ep_ptr, struct dat_ep_attr *ep_attr_ptr) { - DAPL_IA *ia; - DAPL_EP *ep; + struct dapl_ia *ia; + struct dapl_ep *ep; struct dat_ep_attr ep_attr; struct dat_ep_attr ep_attr_limit; struct dat_ep_attr ep_attr_request; @@ -76,7 +76,7 @@ dapl_ep_modify_validate_parameters(DAT_E goto bail; } - ep = (DAPL_EP *) ep_handle; + ep = (struct dapl_ep *)ep_handle; ia = ep->header.owner_ia; /* @@ -139,7 +139,7 @@ dapl_ep_modify_validate_parameters(DAT_E if (mask & DAT_EP_FIELD_RECV_EVD_HANDLE) { if (ep_param->recv_evd_handle != NULL && (DAPL_BAD_HANDLE(ep_param->recv_evd_handle, DAPL_MAGIC_EVD) - || !((DAPL_EVD *) ep_param->recv_evd_handle)-> + || !((struct dapl_evd *)ep_param->recv_evd_handle)-> evd_flags & DAT_EVD_DTO_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); @@ -161,7 +161,7 @@ dapl_ep_modify_validate_parameters(DAT_E if (ep_param->connect_evd_handle != NULL && DAPL_BAD_HANDLE(ep_param->connect_evd_handle, DAPL_MAGIC_EVD) - && !(((DAPL_EVD *) ep_param->connect_evd_handle)-> + && !(((struct dapl_evd *)ep_param->connect_evd_handle)-> evd_flags & DAT_EVD_CONNECTION_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); @@ -298,13 +298,13 @@ dapl_ep_modify_validate_parameters(DAT_E u32 dapl_ep_modify(DAT_EP_HANDLE ep_handle, enum dat_ep_param_mask mask, const struct dat_ep_param *ep_param) { - DAPL_IA *ia; - DAPL_EP *ep1, *ep2; + struct dapl_ia *ia; + struct dapl_ep *ep1, *ep2; struct dat_ep_attr ep_attr1, ep_attr2; - DAPL_EP new_ep, copy_of_old_ep; - DAPL_EP alloc_ep; /* Holder for resources. */ - DAPL_PZ *tmp_pz; - DAPL_EVD *tmp_evd; + struct dapl_ep new_ep, copy_of_old_ep; + struct dapl_ep alloc_ep; /* Holder for resources. */ + struct dapl_pz *tmp_pz; + struct dapl_evd *tmp_evd; u32 dat_status; /* Flag indicating we've allocated a new one of these. */ @@ -325,9 +325,8 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand mask, ep_param, &ia, &ep1, &ep_attr1); - if (DAT_SUCCESS != dat_status) { + if (DAT_SUCCESS != dat_status) goto bail; - } /* * Setup the alloc_ep with the appropriate parameters (primarily @@ -335,23 +334,17 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand */ alloc_ep = *ep1; alloc_ep.param.ep_attr = ep_attr1; - if (mask & DAT_EP_FIELD_PZ_HANDLE) { + if (mask & DAT_EP_FIELD_PZ_HANDLE) alloc_ep.param.pz_handle = ep_param->pz_handle; - } - if (mask & DAT_EP_FIELD_RECV_EVD_HANDLE) { + if (mask & DAT_EP_FIELD_RECV_EVD_HANDLE) alloc_ep.param.recv_evd_handle = ep_param->recv_evd_handle; - } - if (mask & DAT_EP_FIELD_REQUEST_EVD_HANDLE) { - alloc_ep.param.request_evd_handle = - ep_param->request_evd_handle; - } + if (mask & DAT_EP_FIELD_REQUEST_EVD_HANDLE) + alloc_ep.param.request_evd_handle = ep_param->request_evd_handle; - if (mask & DAT_EP_FIELD_CONNECT_EVD_HANDLE) { - alloc_ep.param.connect_evd_handle = - ep_param->connect_evd_handle; - } + if (mask & DAT_EP_FIELD_CONNECT_EVD_HANDLE) + alloc_ep.param.connect_evd_handle = ep_param->connect_evd_handle; /* * Allocate everything that might be needed. @@ -364,16 +357,14 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand */ dat_status = dapl_cb_create(&alloc_ep.req_buffer, ep1, /* For pointer in buffer bool. */ ep_attr1.max_request_dtos); - if (DAT_SUCCESS != dat_status) { + if (DAT_SUCCESS != dat_status) goto bail; - } rqst_cb_allocated = TRUE; dat_status = dapl_cb_create(&alloc_ep.recv_buffer, ep1, /* For pointer in buffer bool. */ ep_attr1.max_recv_dtos); - if (DAT_SUCCESS != dat_status) { + if (DAT_SUCCESS != dat_status) goto bail; - } recv_cb_allocated = TRUE; alloc_ep.send_iov_num = ep_attr1.max_request_iov; @@ -397,9 +388,8 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand recv_iov_allocated = TRUE; dat_status = dapl_ib_qp_alloc(ia, &alloc_ep, ep1); - if (dat_status != DAT_SUCCESS) { + if (dat_status != DAT_SUCCESS) goto bail; - } qp_allocated = TRUE; /* @@ -527,92 +517,79 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand * and then decrementing old ones (so if they're the same * the refcount never drops to zero). */ - tmp_pz = (DAPL_PZ *) new_ep.param.pz_handle; - if (NULL != tmp_pz) { + tmp_pz = (struct dapl_pz *)new_ep.param.pz_handle; + if (tmp_pz) atomic_inc(&tmp_pz->pz_ref_count); - } - tmp_evd = (DAPL_EVD *) new_ep.param.recv_evd_handle; - if (NULL != tmp_evd) { + tmp_evd = (struct dapl_evd *)new_ep.param.recv_evd_handle; + if (tmp_evd) atomic_inc(&tmp_evd->evd_ref_count); - } - tmp_evd = (DAPL_EVD *) new_ep.param.request_evd_handle; - if (NULL != tmp_evd) { + tmp_evd = (struct dapl_evd *)new_ep.param.request_evd_handle; + if (tmp_evd) atomic_inc(&tmp_evd->evd_ref_count); - } - tmp_evd = (DAPL_EVD *) new_ep.param.connect_evd_handle; - if (NULL != tmp_evd) { + tmp_evd = (struct dapl_evd *)new_ep.param.connect_evd_handle; + if (tmp_evd) atomic_inc(&tmp_evd->evd_ref_count); - } /* decreament the old reference counts */ - tmp_pz = (DAPL_PZ *) copy_of_old_ep.param.pz_handle; - if (NULL != tmp_pz) { + tmp_pz = (struct dapl_pz *)copy_of_old_ep.param.pz_handle; + if (tmp_pz) atomic_dec(&tmp_pz->pz_ref_count); - } - tmp_evd = (DAPL_EVD *) copy_of_old_ep.param.recv_evd_handle; - if (NULL != tmp_evd) { + tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.recv_evd_handle; + if (tmp_evd) atomic_dec(&tmp_evd->evd_ref_count); - } - tmp_evd = (DAPL_EVD *) copy_of_old_ep.param.request_evd_handle; - if (NULL != tmp_evd) { + tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.request_evd_handle; + if (tmp_evd) atomic_dec(&tmp_evd->evd_ref_count); - } - tmp_evd = (DAPL_EVD *) copy_of_old_ep.param.connect_evd_handle; - if (NULL != tmp_evd) { + tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.connect_evd_handle; + if (tmp_evd) atomic_dec(&tmp_evd->evd_ref_count); - } - bail: +bail: if (qp_allocated) { u32 local_dat_status; - if (dat_status != DAT_SUCCESS || !qp_used) { + if (dat_status != DAT_SUCCESS || !qp_used) local_dat_status = dapl_ib_qp_free(ia, &alloc_ep); - } else { + else local_dat_status = dapl_ib_qp_free(ia, ©_of_old_ep); - } - if (local_dat_status != DAT_SUCCESS) { + if (local_dat_status != DAT_SUCCESS) dapl_dbg_log(DAPL_DBG_TYPE_WARN, "ep_modify: Failed to free QP; status %x\n", local_dat_status); - } } if (rqst_cb_allocated) { - if (dat_status != DAT_SUCCESS || !rqst_cb_used) { + if (dat_status != DAT_SUCCESS || !rqst_cb_used) dapl_cb_free(&alloc_ep.req_buffer); - } else { + else dapl_cb_free(©_of_old_ep.req_buffer); - } } if (recv_cb_allocated) { - if (dat_status != DAT_SUCCESS || !recv_cb_used) { + if (dat_status != DAT_SUCCESS || !recv_cb_used) dapl_cb_free(&alloc_ep.recv_buffer); - } else { + else dapl_cb_free(©_of_old_ep.recv_buffer); - } } if (rqst_iov_allocated) { - if (dat_status != DAT_SUCCESS || !rqst_iov_used) { + if (dat_status != DAT_SUCCESS || !rqst_iov_used) kfree(alloc_ep.send_iov); - } else { + else kfree(copy_of_old_ep.send_iov); - } } if (recv_iov_allocated) { - if (dat_status != DAT_SUCCESS || !recv_iov_used) { + if (dat_status != DAT_SUCCESS || !recv_iov_used) kfree(alloc_ep.recv_iov); - } else { + else kfree(copy_of_old_ep.recv_iov); - } } + return dat_status; } Index: linux-kernel-resolv/dat-provider/dapl_openib_qp.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_openib_qp.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_openib_qp.c (working copy) @@ -44,7 +44,8 @@ #include "dapl_evd_util.h" #include "dapl_openib_util.h" -struct ib_cq * dapl_get_dto_cq(DAPL_IA * ia_ptr, DAPL_EVD * recv_evd_ptr); +struct ib_cq *dapl_get_dto_cq(struct dapl_ia *ia_ptr, + struct dapl_evd *recv_evd_ptr); /* * dapl_ib_qp_alloc @@ -66,7 +67,8 @@ struct ib_cq * dapl_get_dto_cq(DAPL_IA * * DAT_INSUFFICIENT_RESOURCES * */ -u32 dapl_ib_qp_alloc(DAPL_IA *ia_ptr, DAPL_EP *ep_ptr, DAPL_EP *ep_ctx_ptr) +u32 dapl_ib_qp_alloc(struct dapl_ia *ia_ptr, struct dapl_ep *ep_ptr, + struct dapl_ep *ep_ctx_ptr) { struct dat_ep_attr *attr; int ib_status; @@ -80,15 +82,15 @@ u32 dapl_ib_qp_alloc(DAPL_IA *ia_ptr, DA dapl_os_assert(ep_ptr->param.pz_handle != NULL); ib_pd_handle = - (struct ib_pd *)((DAPL_PZ *) ep_ptr->param.pz_handle)->pd_handle; + (struct ib_pd *)((struct dapl_pz *)ep_ptr->param.pz_handle)->pd_handle; ib_hca_handle = ia_ptr->hca_ptr->ib_hca_handle; cq_recv = NULL; cq_send = NULL; cq_recv = dapl_get_dto_cq(ia_ptr, - (DAPL_EVD *) ep_ptr->param.recv_evd_handle); + (struct dapl_evd *)ep_ptr->param.recv_evd_handle); cq_send = dapl_get_dto_cq(ia_ptr, - (DAPL_EVD *) ep_ptr->param. + (struct dapl_evd *)ep_ptr->param. request_evd_handle); qp_attr.send_cq = cq_send; @@ -151,7 +153,7 @@ u32 dapl_ib_qp_alloc(DAPL_IA *ia_ptr, DA * Status of destroy operation. * */ -u32 dapl_ib_qp_free(DAPL_IA *ia_ptr, DAPL_EP *ep_ptr) +u32 dapl_ib_qp_free(struct dapl_ia *ia_ptr, struct dapl_ep *ep_ptr) { int ib_status; @@ -178,7 +180,7 @@ u32 dapl_ib_qp_free(DAPL_IA *ia_ptr, DAP * Returns * A valid CQ handle */ -struct ib_cq * dapl_get_dto_cq(DAPL_IA * ia_ptr, DAPL_EVD * evd_ptr) +struct ib_cq *dapl_get_dto_cq(struct dapl_ia *ia_ptr, struct dapl_evd *evd_ptr) { struct ib_cq *cq_handle; int cq_entries; @@ -232,7 +234,7 @@ struct ib_cq * dapl_get_dto_cq(DAPL_IA * * DAT_INVALID_PARAMETER * */ -u32 dapl_ib_qp_modify(DAPL_IA *ia_ptr, DAPL_EP *ep_ptr, +u32 dapl_ib_qp_modify(struct dapl_ia *ia_ptr, struct dapl_ep *ep_ptr, struct dat_ep_attr *ep_attr) { struct ib_qp_attr qp_attr; Index: linux-kernel-resolv/dat-provider/dapl_evd_kcreate.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_evd_kcreate.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_evd_kcreate.c (working copy) @@ -53,9 +53,9 @@ u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_ha const struct dat_upcall_object *upcall, enum dat_evd_flags evd_flags, DAT_EVD_HANDLE *evd_handle) { - DAPL_IA *ia_ptr; - DAPL_EVD *evd_ptr; - DAPL_CNO *cno_ptr; + struct dapl_ia *ia_ptr; + struct dapl_evd *evd_ptr; + struct dapl_cno *cno_ptr; u32 dat_status = DAT_SUCCESS; DAT_EVD_HANDLE evdhandle = (DAT_EVD_HANDLE) NULL; @@ -64,7 +64,7 @@ u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_ha ia_handle, evd_min_qlen, upcall_policy, evd_flags, evd_handle); - ia_ptr = (DAPL_IA *) ia_handle; + ia_ptr = (struct dapl_ia *)ia_handle; cno_ptr = NULL; evd_ptr = NULL; *evd_handle = NULL; @@ -94,38 +94,31 @@ u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_ha goto bail; } - cno_ptr = dapl_cno_alloc (ia_ptr, upcall); + cno_ptr = dapl_cno_alloc(ia_ptr, upcall); - if (!cno_ptr) - { + if (!cno_ptr) { dat_status = DAT_INSUFFICIENT_RESOURCES; goto bail; } - cno_ptr->cno_upcall_policy = upcall_policy; - + cno_ptr->cno_upcall_policy = upcall_policy; dat_status = dapl_evd_internal_create(ia_ptr, cno_ptr, evd_min_qlen, evd_flags, &evd_ptr); - if (dat_status != DAT_SUCCESS) { + if (dat_status != DAT_SUCCESS) goto bail; - } evdhandle = *evd_handle = (DAT_EVD_HANDLE) evd_ptr; - bail: +bail: if (dat_status != DAT_SUCCESS && cno_ptr != NULL) - { dapl_cno_dealloc (cno_ptr); - } - if (dat_status != DAT_SUCCESS) { - if (evd_ptr) { + if (dat_status != DAT_SUCCESS) + if (evd_ptr) dapl_evd_free(evd_ptr); - } - } return dat_status; } Index: linux-kernel-resolv/dat-provider/dapl_cno_util.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_cno_util.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_cno_util.c (working copy) @@ -38,9 +38,10 @@ /* * alloc and initialize an EVD struct */ -DAPL_CNO *dapl_cno_alloc(DAPL_IA *ia_ptr, const struct dat_upcall_object *upcall) +struct dapl_cno *dapl_cno_alloc(struct dapl_ia *ia_ptr, + const struct dat_upcall_object *upcall) { - DAPL_CNO *cno_ptr; + struct dapl_cno *cno_ptr; cno_ptr = kmalloc(sizeof *cno_ptr, GFP_ATOMIC); if (!cno_ptr) @@ -71,7 +72,7 @@ DAPL_CNO *dapl_cno_alloc(DAPL_IA *ia_ptr /* * Free the passed in CNO structure. */ -void dapl_cno_dealloc(DAPL_CNO *cno_ptr) +void dapl_cno_dealloc(struct dapl_cno *cno_ptr) { dapl_os_assert(cno_ptr->header.magic == DAPL_MAGIC_CNO); @@ -98,7 +99,7 @@ void dapl_cno_dealloc(DAPL_CNO *cno_ptr) * Returns: * None */ -void dapl_cno_trigger(DAPL_CNO *cno_ptr, DAPL_EVD *evd_ptr) +void dapl_cno_trigger(struct dapl_cno *cno_ptr, struct dapl_evd *evd_ptr) { u32 dat_status = DAT_SUCCESS; struct dat_event event; Index: linux-kernel-resolv/dat-provider/dapl_timer_util.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_timer_util.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_timer_util.c (working copy) @@ -53,7 +53,7 @@ #include "dapl_timer_util.h" struct dapl_timer_head { - DAPL_LLIST_HEAD timer_list_head; + struct dapl_llist_entry *timer_list_head; spinlock_t lock; unsigned long flags; DAPL_OS_WAIT_OBJECT wait_object; @@ -129,7 +129,8 @@ u32 dapl_timer_set(struct dapl_timer_ent if (dapl_llist_is_empty(&g_dapl_timer_head.timer_list_head)) { /* Case 1: add entry to head of list */ dapl_llist_add_head(&g_dapl_timer_head.timer_list_head, - (DAPL_LLIST_ENTRY *) & timer->list_entry, + (struct dapl_llist_entry *) + &timer->list_entry, timer); wakeup_tmo_thread = TRUE; } else { @@ -139,42 +140,41 @@ u32 dapl_timer_set(struct dapl_timer_ent if (timer->expires < list_ptr->expires) { /* Case 2: add entry to head of list */ dapl_llist_add_head(&g_dapl_timer_head.timer_list_head, - (DAPL_LLIST_ENTRY *) & timer-> - list_entry, timer); + (struct dapl_llist_entry *) + &timer->list_entry, timer); wakeup_tmo_thread = TRUE; } else { /* Case 3: figure out where entry goes in sorted list */ list_ptr = dapl_llist_next_entry(&g_dapl_timer_head. timer_list_head, - (DAPL_LLIST_ENTRY *) & - list_ptr->list_entry); + (struct dapl_llist_entry *) + &list_ptr->list_entry); while (list_ptr != NULL) { if (timer->expires < list_ptr->expires) { dapl_llist_add_entry(&g_dapl_timer_head. timer_list_head, - (DAPL_LLIST_ENTRY - *) & list_ptr-> - list_entry, - (DAPL_LLIST_ENTRY - *) & timer-> - list_entry, timer); + (struct dapl_llist_entry *) + &list_ptr->list_entry, + (struct dapl_llist_entry *) + &timer->list_entry, + timer); break; } list_ptr = dapl_llist_next_entry(&g_dapl_timer_head. timer_list_head, - (DAPL_LLIST_ENTRY *) & - list_ptr->list_entry); + (struct dapl_llist_entry *) + &list_ptr->list_entry); } if (list_ptr == NULL) { /* entry goes to the end of the list */ dapl_llist_add_tail(&g_dapl_timer_head. timer_list_head, - (DAPL_LLIST_ENTRY *) & - timer->list_entry, timer); + (struct dapl_llist_entry *) + &timer->list_entry, timer); } } @@ -209,8 +209,8 @@ void dapl_timer_cancel(struct dapl_timer if (!dapl_llist_is_empty(&g_dapl_timer_head.timer_list_head) && timer->list_entry.list_head == &g_dapl_timer_head.timer_list_head) { dapl_llist_remove_entry(&g_dapl_timer_head.timer_list_head, - (DAPL_LLIST_ENTRY *) & timer-> - list_entry); + (struct dapl_llist_entry *) + &timer->list_entry); } /* * If this was the first entry on the queue we could awaken the Index: linux-kernel-resolv/dat-provider/dapl_openib_util.h =================================================================== --- linux-kernel-resolv/dat-provider/dapl_openib_util.h (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_openib_util.h (working copy) @@ -62,15 +62,16 @@ u32 dapl_ib_enum_hcas(const char *vendor u32 dapl_ib_get_instance_data(DAPL_HCA_NAME hca_name, char *instance); -u32 dapl_ib_open_hca(char *namestr, DAPL_HCA *hca_ptr); +u32 dapl_ib_open_hca(char *namestr, struct dapl_hca *hca_ptr); -u32 dapl_ib_close_hca(DAPL_HCA *hca_ptr); +u32 dapl_ib_close_hca(struct dapl_hca *hca_ptr); -u32 dapl_ib_qp_alloc(DAPL_IA *ia_ptr, DAPL_EP *ep_ptr, DAPL_EP *ep_ctx_ptr); +u32 dapl_ib_qp_alloc(struct dapl_ia *ia_ptr, struct dapl_ep *ep_ptr, + struct dapl_ep *ep_ctx_ptr); -u32 dapl_ib_qp_free(DAPL_IA *ia_ptr, DAPL_EP *ep_ptr); +u32 dapl_ib_qp_free(struct dapl_ia *ia_ptr, struct dapl_ep *ep_ptr); -u32 dapl_ib_qp_modify(DAPL_IA *ia_ptr, DAPL_EP *ep_ptr, +u32 dapl_ib_qp_modify(struct dapl_ia *ia_ptr, struct dapl_ep *ep_ptr, struct dat_ep_attr *ep_attr); u32 dapl_ib_connect(DAT_EP_HANDLE ep_handle, @@ -78,12 +79,12 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han DAT_CONN_QUAL remote_conn_qual, int private_data_size, void *private_data); -u32 dapl_ib_disconnect(DAPL_EP *ep_ptr, enum dat_close_flags close_flags); +u32 dapl_ib_disconnect(struct dapl_ep *ep_ptr, enum dat_close_flags close_flags); -u32 dapl_ib_setup_conn_listener(DAPL_IA *ia_ptr, u64 ServiceID, - DAPL_SP *sp_ptr); +u32 dapl_ib_setup_conn_listener(struct dapl_ia *ia_ptr, u64 ServiceID, + struct dapl_sp *sp_ptr); -u32 dapl_ib_remove_conn_listener(DAPL_IA *ia_ptr, DAPL_SP *sp_ptr); +u32 dapl_ib_remove_conn_listener(struct dapl_ia *ia_ptr, struct dapl_sp *sp_ptr); u32 dapl_ib_accept_connection(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, @@ -92,61 +93,63 @@ u32 dapl_ib_accept_connection(DAT_CR_HAN u32 dapl_ib_reject_connection(struct dapl_cm_id *cm_handle); -u32 dapl_ib_setup_async_callback(DAPL_IA *ia_ptr, +u32 dapl_ib_setup_async_callback(struct dapl_ia *ia_ptr, DAPL_ASYNC_HANDLER_TYPE handler_type, ib_async_handler_t callback, void *context); -u32 dapl_ib_cq_alloc(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr, int *cqlen, - ib_comp_handler callback); +u32 dapl_ib_cq_alloc(struct dapl_ia *ia_ptr, struct dapl_evd *evd_ptr, + int *cqlen, ib_comp_handler callback); -u32 dapl_ib_cq_free(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr); +u32 dapl_ib_cq_free(struct dapl_ia *ia_ptr, struct dapl_evd *evd_ptr); -u32 dapl_set_cq_notify(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr); +u32 dapl_set_cq_notify(struct dapl_ia *ia_ptr, struct dapl_evd *evd_ptr); -u32 dapl_ib_cq_resize(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr, int *cqlen); +u32 dapl_ib_cq_resize(struct dapl_ia *ia_ptr, struct dapl_evd *evd_ptr, + int *cqlen); -u32 dapl_ib_pd_alloc(DAPL_IA *ia_ptr, DAPL_PZ *pz); +u32 dapl_ib_pd_alloc(struct dapl_ia *ia_ptr, struct dapl_pz *pz); -u32 dapl_ib_pd_free(DAPL_PZ *pz); +u32 dapl_ib_pd_free(struct dapl_pz *pz); -u32 dapl_ib_mr_register(DAPL_IA *ia_ptr, DAPL_LMR *lmr, void *virt_addr, - u64 length, enum dat_mem_priv_flags privileges); +u32 dapl_ib_mr_register(struct dapl_ia *ia_ptr, struct dapl_lmr *lmr, + void *virt_addr, u64 length, + enum dat_mem_priv_flags privileges); -u32 dapl_ib_mr_register_physical(DAPL_IA *ia_ptr, DAPL_LMR *lmr, +u32 dapl_ib_mr_register_physical(struct dapl_ia *ia_ptr, struct dapl_lmr *lmr, void *phys_addr, u64 length, enum dat_mem_priv_flags privileges); -u32 dapl_ib_mr_deregister(DAPL_LMR *lmr); +u32 dapl_ib_mr_deregister(struct dapl_lmr *lmr); -u32 dapl_ib_mr_register_shared(DAPL_IA *ia_ptr, DAPL_LMR *lmr, +u32 dapl_ib_mr_register_shared(struct dapl_ia *ia_ptr, struct dapl_lmr *lmr, enum dat_mem_priv_flags privileges); -u32 dapl_ib_mw_alloc(DAPL_RMR *rmr); +u32 dapl_ib_mw_alloc(struct dapl_rmr *rmr); -u32 dapl_ib_mw_free(DAPL_RMR *rmr); +u32 dapl_ib_mw_free(struct dapl_rmr *rmr); -u32 dapl_ib_mw_bind(DAPL_RMR *rmr, DAPL_LMR *lmr, DAPL_EP *ep, - DAPL_COOKIE *cookie, u64 virtual_address, - u64 length, enum dat_mem_priv_flags mem_priv, - boolean_t is_signaled); +u32 dapl_ib_mw_bind(struct dapl_rmr *rmr, struct dapl_lmr *lmr, + struct dapl_ep *ep, struct dapl_cookie *cookie, + u64 virtual_address, u64 length, + enum dat_mem_priv_flags mem_priv, boolean_t is_signaled); -u32 dapl_ib_mw_unbind(DAPL_RMR *rmr, DAPL_EP *ep, DAPL_COOKIE *cookie, - boolean_t is_signaled); +u32 dapl_ib_mw_unbind(struct dapl_rmr *rmr, struct dapl_ep *ep, + struct dapl_cookie *cookie, boolean_t is_signaled); -u32 dapl_ib_query_hca(DAPL_HCA *hca_ptr, struct dat_ia_attr *ia_attr, +u32 dapl_ib_query_hca(struct dapl_hca *hca_ptr, struct dat_ia_attr *ia_attr, struct dat_ep_attr *ep_attr, struct sockaddr_in6 *ip_addr); -u32 dapl_ib_completion_poll(DAPL_HCA *hca_ptr, DAPL_EVD *evd_ptr, +u32 dapl_ib_completion_poll(struct dapl_hca *hca_ptr, struct dapl_evd *evd_ptr, struct ib_wc *cqe_ptr); -u32 dapl_ib_completion_notify(DAPL_EVD *evd_ptr); +u32 dapl_ib_completion_notify(struct dapl_evd *evd_ptr); enum dat_dto_completion_status dapl_ib_get_dto_status(struct ib_wc *cqe_ptr); -void dapl_ib_reinit_ep(DAPL_EP * ep_ptr); +void dapl_ib_reinit_ep(struct dapl_ep * ep_ptr); -void dapl_ib_disconnect_clean(DAPL_EP * ep_ptr, boolean_t passive); +void dapl_ib_disconnect_clean(struct dapl_ep * ep_ptr, boolean_t passive); u32 dapl_ib_get_async_event(struct ib_event *cause_ptr, enum dat_event_number *async_event); Index: linux-kernel-resolv/dat-provider/dapl_evd_kquery.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_evd_kquery.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_evd_kquery.c (working copy) @@ -40,7 +40,7 @@ u32 dapl_evd_kquery(DAT_EVD_HANDLE evd_handle, struct dat_evd_param *evd_param) { - DAPL_EVD *evd_ptr; + struct dapl_evd *evd_ptr; u32 dat_status = DAT_SUCCESS; if (NULL == evd_param) { @@ -58,7 +58,7 @@ u32 dapl_evd_kquery(DAT_EVD_HANDLE evd_h goto bail; } - evd_ptr = (DAPL_EVD *) evd_handle; + evd_ptr = (struct dapl_evd *)evd_handle; /* * We may be racing against the thread safe modify * calls here (dat_evd_{enable,disable,{set,clear}_unwaitable}). Index: linux-kernel-resolv/dat-provider/dapl_ia_close.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_ia_close.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_ia_close.c (working copy) @@ -59,13 +59,13 @@ */ u32 dapl_ia_close(DAT_IA_HANDLE ia_handle, enum dat_close_flags ia_flags) { - DAPL_IA *ia_ptr; + struct dapl_ia *ia_ptr; u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ia_close (%p, %d)\n", ia_handle, ia_flags); - ia_ptr = (DAPL_IA *) ia_handle; + ia_ptr = (struct dapl_ia *)ia_handle; if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { dat_status = Index: linux-kernel-resolv/dat-provider/dapl_evd_qp_async_error_callb.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_evd_qp_async_error_callb.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_evd_qp_async_error_callb.c (working copy) @@ -54,13 +54,13 @@ void dapl_evd_qp_async_error_callback(struct ib_event *cause, void *context) { - DAPL_EP *ep_ptr; - DAPL_EVD *async_evd; + struct dapl_ep *ep_ptr; + struct dapl_evd *async_evd; enum dat_event_number async_event; u32 dat_status; - ep_ptr = (DAPL_EP *) context; - async_evd = (DAPL_EVD *) ep_ptr->header.owner_ia->async_error_evd; + ep_ptr = (struct dapl_ep *)context; + async_evd = (struct dapl_evd *)ep_ptr->header.owner_ia->async_error_evd; dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK | DAPL_DBG_TYPE_EXCEPTION, "--> dapl_evd_qp_async_error_callback: ep %p qp %p (%x) state %d\n", Index: linux-kernel-resolv/dat-provider/dapl_cno_util.h =================================================================== --- linux-kernel-resolv/dat-provider/dapl_cno_util.h (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_cno_util.h (working copy) @@ -36,16 +36,11 @@ #include "dapl.h" -DAPL_CNO * -dapl_cno_alloc(DAPL_IA *ia_ptr, const struct dat_upcall_object *upcall); +struct dapl_cno *dapl_cno_alloc(struct dapl_ia *ia_ptr, + const struct dat_upcall_object *upcall); -void -dapl_cno_dealloc ( - DAPL_CNO *cno_ptr) ; +void dapl_cno_dealloc(struct dapl_cno *cno_ptr); -void -dapl_cno_trigger( - DAPL_CNO *cno_ptr, - DAPL_EVD *evd_ptr); +void dapl_cno_trigger(struct dapl_cno *cno_ptr, struct dapl_evd *evd_ptr); #endif Index: linux-kernel-resolv/dat-provider/dapl_srq_set_lw.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_srq_set_lw.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_srq_set_lw.c (working copy) @@ -65,7 +65,7 @@ u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, int low_watermark) { - DAPL_SRQ *srq_ptr; + struct dapl_srq *srq_ptr; u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_set_lw (%p, %d)\n", @@ -77,7 +77,7 @@ u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_h goto bail; } - srq_ptr = (DAPL_SRQ *) srq_handle; + srq_ptr = (struct dapl_srq *)srq_handle; /* XXX Put implementation here XXX */ Index: linux-kernel-resolv/dat-provider/dapl_evd_un_async_error_callb.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_evd_un_async_error_callb.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_evd_un_async_error_callb.c (working copy) @@ -46,7 +46,7 @@ void dapl_evd_un_async_error_callback(struct ib_event *cause, void *context) { - DAPL_EVD *async_evd; + struct dapl_evd *async_evd; enum dat_event_number async_event; u32 dat_status; @@ -59,7 +59,7 @@ void dapl_evd_un_async_error_callback(st return; } - async_evd = (DAPL_EVD *) context; + async_evd = (struct dapl_evd *)context; dat_status = dapl_ib_get_async_event(cause, &async_event); if (dat_status == DAT_SUCCESS) { Index: linux-kernel-resolv/dat-provider/dapl_cr_accept.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_cr_accept.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_cr_accept.c (working copy) @@ -42,9 +42,9 @@ u32 dapl_cr_accept(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, int private_data_size, const void *private_data) { - DAPL_EP *ep_ptr; + struct dapl_ep *ep_ptr; u32 dat_status; - DAPL_CR *cr_ptr; + struct dapl_cr *cr_ptr; enum dat_ep_state entry_ep_state; DAT_EP_HANDLE entry_ep_handle; @@ -58,7 +58,7 @@ u32 dapl_cr_accept(DAT_CR_HANDLE cr_hand goto bail; } - cr_ptr = (DAPL_CR *) cr_handle; + cr_ptr = (struct dapl_cr *)cr_handle; /* * Return an error if we have an ep_handle and the CR already has an @@ -91,21 +91,21 @@ u32 dapl_cr_accept(DAT_CR_HANDLE cr_hand */ if (ep_handle == NULL) { ep_handle = cr_ptr->param.local_ep_handle; - if ((((DAPL_EP *) ep_handle)->param.ep_state != + if ((((struct dapl_ep *)ep_handle)->param.ep_state != DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING) - && (((DAPL_EP *) ep_handle)->param.ep_state != + && (((struct dapl_ep *)ep_handle)->param.ep_state != DAT_EP_STATE_PASSIVE_CONNECTION_PENDING)) { return DAT_INVALID_STATE; } } else { /* ensure this EP isn't connected or in use */ - if (((DAPL_EP *) ep_handle)->param.ep_state != + if (((struct dapl_ep *)ep_handle)->param.ep_state != DAT_EP_STATE_UNCONNECTED) { return DAT_INVALID_STATE; } } - ep_ptr = (DAPL_EP *) ep_handle; + ep_ptr = (struct dapl_ep *)ep_handle; /* * Verify the attributes of the EP handle before we connect it. Test @@ -124,7 +124,7 @@ u32 dapl_cr_accept(DAT_CR_HANDLE cr_hand /* test connect handle */ || ep_ptr->param.connect_evd_handle == NULL || DAPL_BAD_HANDLE(ep_ptr->param.connect_evd_handle, DAPL_MAGIC_EVD) - || !(((DAPL_EVD *) ep_ptr->param.connect_evd_handle)-> + || !(((struct dapl_evd *)ep_ptr->param.connect_evd_handle)-> evd_flags & DAT_EVD_CONNECTION_FLAG) /* test optional completion handles */ || (ep_ptr->param.recv_evd_handle != DAT_HANDLE_NULL && Index: linux-kernel-resolv/dat-provider/dapl_rsp_create.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_rsp_create.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_rsp_create.c (working copy) @@ -72,14 +72,14 @@ u32 dapl_rsp_create(DAT_IA_HANDLE ia_han DAT_EP_HANDLE ep_handle, DAT_EVD_HANDLE evd_handle, DAT_RSP_HANDLE *rsp_handle) { - DAPL_IA *ia_ptr; - DAPL_SP *sp_ptr; - DAPL_EVD *evd_ptr; - DAPL_EP *ep_ptr; + struct dapl_ia *ia_ptr; + struct dapl_sp *sp_ptr; + struct dapl_evd *evd_ptr; + struct dapl_ep *ep_ptr; boolean_t sp_found; u32 dat_status = DAT_SUCCESS; - ia_ptr = (DAPL_IA *) ia_handle; + ia_ptr = (struct dapl_ia *)ia_handle; dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_rsp_free conn_qual: %x EP: %p\n", @@ -106,14 +106,14 @@ u32 dapl_rsp_create(DAT_IA_HANDLE ia_han goto bail; } - ep_ptr = (DAPL_EP *) ep_handle; + ep_ptr = (struct dapl_ep *)ep_handle; if (ep_ptr->param.ep_state != DAT_EP_STATE_UNCONNECTED) { dat_status = DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep_ptr)); goto bail; } - evd_ptr = (DAPL_EVD *) evd_handle; + evd_ptr = (struct dapl_evd *)evd_handle; if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); @@ -147,7 +147,7 @@ u32 dapl_rsp_create(DAT_IA_HANDLE ia_han /* * Take a reference on the EVD handle */ - atomic_inc(&((DAPL_EVD *) evd_handle)->evd_ref_count); + atomic_inc(&((struct dapl_evd *)evd_handle)->evd_ref_count); /* * Update the EP state indicating the provider now owns it @@ -174,7 +174,7 @@ u32 dapl_rsp_create(DAT_IA_HANDLE ia_han * wrong! Decrements the EVD refcount & release it. Set * the state to FREE, so we know the call failed. */ - atomic_dec(&((DAPL_EVD *) evd_handle)-> + atomic_dec(&((struct dapl_evd *)evd_handle)-> evd_ref_count); sp_ptr->evd_handle = NULL; sp_ptr->state = DAPL_SP_STATE_FREE; Index: linux-kernel-resolv/dat-provider/dapl_srq_post_recv.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_srq_post_recv.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_srq_post_recv.c (working copy) @@ -71,8 +71,8 @@ u32 dapl_srq_post_recv(DAT_SRQ_HANDLE sr struct dat_lmr_triplet *local_iov, DAT_DTO_COOKIE user_cookie) { - DAPL_SRQ *srq_ptr; - DAPL_COOKIE *cookie; + struct dapl_srq *srq_ptr; + struct dapl_cookie *cookie; u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, @@ -85,7 +85,7 @@ u32 dapl_srq_post_recv(DAT_SRQ_HANDLE sr goto bail; } - srq_ptr = (DAPL_SRQ *) srq_handle; + srq_ptr = (struct dapl_srq *)srq_handle; /* * Synchronization ok since this buffer is only used for receive Index: linux-kernel-resolv/dat-provider/dapl_lmr_free.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_lmr_free.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_lmr_free.c (working copy) @@ -59,7 +59,7 @@ u32 dapl_lmr_free(DAT_LMR_HANDLE lmr_handle) { - DAPL_LMR *lmr; + struct dapl_lmr *lmr; u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_free (%p)\n", lmr_handle); @@ -70,36 +70,34 @@ u32 dapl_lmr_free(DAT_LMR_HANDLE lmr_han goto bail; } - lmr = (DAPL_LMR *) lmr_handle; + lmr = (struct dapl_lmr *)lmr_handle; switch (lmr->param.mem_type) { case DAT_MEM_TYPE_PHYSICAL: case DAT_MEM_TYPE_VIRTUAL: case DAT_MEM_TYPE_LMR: { - DAPL_PZ *pz; + struct dapl_pz *pz; - if (0 != atomic_read(&lmr->lmr_ref_count)) { + if (0 != atomic_read(&lmr->lmr_ref_count)) return DAT_INVALID_STATE; - } dat_status = dapl_hash_remove(lmr->header.owner_ia->hca_ptr-> lmr_hash_table, lmr->param.lmr_context, NULL); - if (dat_status != DAT_SUCCESS) { + if (dat_status != DAT_SUCCESS) goto bail; - } dat_status = dapl_ib_mr_deregister(lmr); if (dat_status == DAT_SUCCESS) { - pz = (DAPL_PZ *) lmr->param.pz_handle; + pz = (struct dapl_pz *)lmr->param.pz_handle; atomic_dec(&pz->pz_ref_count); dapl_lmr_dealloc(lmr); - } else { + } else /* * Deregister failed; put it back in the * hash table. @@ -107,7 +105,6 @@ u32 dapl_lmr_free(DAT_LMR_HANDLE lmr_han dapl_hash_insert(lmr->header.owner_ia-> hca_ptr->lmr_hash_table, lmr->param.lmr_context, lmr); - } break; } @@ -124,6 +121,6 @@ u32 dapl_lmr_free(DAT_LMR_HANDLE lmr_han break; } } - bail: +bail: return dat_status; } Index: linux-kernel-resolv/dat-provider/dapl_get_handle_type.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_get_handle_type.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_get_handle_type.c (working copy) @@ -59,9 +59,9 @@ u32 dapl_get_handle_type(DAT_HANDLE dat_handle, enum dat_handle_type *type) { u32 dat_status = DAT_SUCCESS; - DAPL_HEADER *header; + struct dapl_header *header; - header = (DAPL_HEADER *) dat_handle; + header = (struct dapl_header *)dat_handle; if (((header) == NULL) || ((unsigned long)(header) & 3) || (header->magic != DAPL_MAGIC_IA && Index: linux-kernel-resolv/dat-provider/dapl_rmr_util.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_rmr_util.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_rmr_util.c (working copy) @@ -28,9 +28,9 @@ #include "dapl_rmr_util.h" #include "dapl_ia_util.h" -DAPL_RMR *dapl_rmr_alloc(DAPL_PZ * pz) +struct dapl_rmr *dapl_rmr_alloc(struct dapl_pz *pz) { - DAPL_RMR *rmr; + struct dapl_rmr *rmr; /* Allocate LMR */ rmr = kmalloc(sizeof *rmr, GFP_ATOMIC); @@ -70,9 +70,10 @@ DAPL_RMR *dapl_rmr_alloc(DAPL_PZ * pz) return rmr; } -void dapl_rmr_dealloc(DAPL_RMR * rmr) +void dapl_rmr_dealloc(struct dapl_rmr *rmr) { - rmr->header.magic = DAPL_MAGIC_INVALID; /* reset magic to prevent reuse */ + /* reset magic to prevent reuse */ + rmr->header.magic = DAPL_MAGIC_INVALID; dapl_ia_unlink_rmr(rmr->header.owner_ia, rmr); /* no need to destroy rmr->header.lock */ Index: linux-kernel-resolv/dat-provider/dapl_ep_util.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_ep_util.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_ep_util.c (working copy) @@ -55,9 +55,10 @@ * none * */ -DAPL_EP *dapl_ep_alloc(DAPL_IA * ia_ptr, const struct dat_ep_attr *ep_attr) +struct dapl_ep *dapl_ep_alloc(struct dapl_ia *ia_ptr, + const struct dat_ep_attr *ep_attr) { - DAPL_EP *ep_ptr; + struct dapl_ep *ep_ptr; /* Allocate EP */ ep_ptr = kmalloc(sizeof *ep_ptr + sizeof (struct sockaddr), GFP_ATOMIC); @@ -94,8 +95,7 @@ DAPL_EP *dapl_ep_alloc(DAPL_IA * ia_ptr, (ep_ptr->param.ep_attr.max_request_dtos / 256) + 1; ep_ptr->param.ep_attr.max_recv_dtos = (ep_ptr->param.ep_attr.max_recv_dtos / 256) + 1; - } - else + } else ep_ptr->param.ep_attr = *ep_attr; ep_ptr->qp_handle = NULL; @@ -139,7 +139,7 @@ DAPL_EP *dapl_ep_alloc(DAPL_IA * ia_ptr, goto bail; } - bail: +bail: return ep_ptr; } @@ -158,26 +158,24 @@ DAPL_EP *dapl_ep_alloc(DAPL_IA * ia_ptr, * none * */ -void dapl_ep_dealloc(DAPL_EP * ep_ptr) +void dapl_ep_dealloc(struct dapl_ep *ep_ptr) { dapl_os_assert(ep_ptr->header.magic == DAPL_MAGIC_EP); - ep_ptr->header.magic = DAPL_MAGIC_INVALID; /* reset magic to prevent reuse */ + /* reset magic to prevent reuse */ + ep_ptr->header.magic = DAPL_MAGIC_INVALID; dapl_cb_free(&ep_ptr->req_buffer); dapl_cb_free(&ep_ptr->recv_buffer); - if (NULL != ep_ptr->recv_iov) { + if (ep_ptr->recv_iov) kfree(ep_ptr->recv_iov); - } - if (NULL != ep_ptr->send_iov) { + if (ep_ptr->send_iov) kfree(ep_ptr->send_iov); - } - if (NULL != ep_ptr->cxn_timer) { + if (ep_ptr->cxn_timer) kfree(ep_ptr->cxn_timer); - } kfree(ep_ptr); } @@ -194,9 +192,8 @@ u32 dapl_ep_check_recv_completion_flags( * an EVD completes, even though the dapl handler is invoked. */ - if (flags & DAT_COMPLETION_SUPPRESS_FLAG) { + if (flags & DAT_COMPLETION_SUPPRESS_FLAG) return DAT_INVALID_PARAMETER; - } return DAT_SUCCESS; } @@ -208,8 +205,8 @@ u32 dapl_ep_post_send_req(DAT_EP_HANDLE enum dat_completion_flags completion_flags, DAPL_DTO_TYPE dto_type, enum ib_wr_opcode op_type) { - DAPL_EP *ep_ptr; - DAPL_COOKIE *cookie; + struct dapl_ep *ep_ptr; + struct dapl_cookie *cookie; u32 dat_status; if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) { @@ -218,7 +215,7 @@ u32 dapl_ep_post_send_req(DAT_EP_HANDLE goto bail; } - ep_ptr = (DAPL_EP *) ep_handle; + ep_ptr = (struct dapl_ep *)ep_handle; /* * Synchronization ok since this buffer is only used for send @@ -226,9 +223,8 @@ u32 dapl_ep_post_send_req(DAT_EP_HANDLE */ dat_status = dapl_dto_cookie_alloc(&ep_ptr->req_buffer, dto_type, user_cookie, &cookie); - if (dat_status != DAT_SUCCESS) { + if (dat_status != DAT_SUCCESS) goto bail; - } /* * Take reference before posting to avoid race conditions with @@ -247,7 +243,7 @@ u32 dapl_ep_post_send_req(DAT_EP_HANDLE dapl_cookie_dealloc(&ep_ptr->req_buffer, cookie); } - bail: +bail: return dat_status; } @@ -259,11 +255,11 @@ u32 dapl_ep_post_send_req(DAT_EP_HANDLE */ void dapl_ep_timeout(void *arg) { - DAPL_EP *ep_ptr; + struct dapl_ep *ep_ptr; dapl_dbg_log(DAPL_DBG_TYPE_CM, "--> dapl_ep_timeout! ep %lx\n", arg); - ep_ptr = (DAPL_EP *) arg; + ep_ptr = (struct dapl_ep *)arg; /* reset the EP state */ ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; @@ -271,7 +267,7 @@ void dapl_ep_timeout(void *arg) /* Clean up the EP and put the underlying QP into the ERROR state. */ dapl_ib_disconnect_clean(ep_ptr, TRUE); - (void)dapl_evd_post_connection_event((DAPL_EVD *) ep_ptr->param. + (void)dapl_evd_post_connection_event((struct dapl_evd *)ep_ptr->param. connect_evd_handle, DAT_CONNECTION_EVENT_TIMED_OUT, (DAT_HANDLE) ep_ptr, 0, NULL); @@ -283,7 +279,7 @@ void dapl_ep_timeout(void *arg) * Return the INVALID_STATE connection subtype associated with an * INVALID_STATE on an EP. Strictly for error reporting. */ -enum dat_return_subtype dapl_ep_state_subtype(DAPL_EP * ep_ptr) +enum dat_return_subtype dapl_ep_state_subtype(struct dapl_ep *ep_ptr) { enum dat_return_subtype dat_status; Index: linux-kernel-resolv/dat-provider/dapl_rmr_util.h =================================================================== --- linux-kernel-resolv/dat-provider/dapl_rmr_util.h (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_rmr_util.h (working copy) @@ -39,15 +39,9 @@ #include "dapl_mr_util.h" -/********************************************************************* - * * - * Function Prototypes * - * * - *********************************************************************/ +extern struct dapl_rmr *dapl_rmr_alloc(struct dapl_pz *pz); -extern DAPL_RMR *dapl_rmr_alloc(DAPL_PZ * pz); - -extern void dapl_rmr_dealloc(DAPL_RMR * rmr); +extern void dapl_rmr_dealloc(struct dapl_rmr *rmr); static inline boolean_t dapl_rmr_validate_completion_flag(enum dat_completion_flags mask, Index: linux-kernel-resolv/dat-provider/dapl_lmr_sync_rdma_write.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_lmr_sync_rdma_write.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_lmr_sync_rdma_write.c (working copy) @@ -45,14 +45,14 @@ u32 dapl_lmr_sync_rdma_write(DAT_IA_HAND const struct dat_lmr_triplet *local_segments, u64 num_segments) { - DAPL_IA *ia_ptr; + struct dapl_ia *ia_ptr; u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dat_lmr_sync_rdma_write (%p, %p, %ld)\n", ia_handle, local_segments, num_segments); - ia_ptr = (DAPL_IA *) ia_handle; + ia_ptr = (struct dapl_ia *)ia_handle; if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { dat_status = Index: linux-kernel-resolv/dat-provider/dapl_rsp_free.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_rsp_free.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_rsp_free.c (working copy) @@ -60,12 +60,12 @@ */ u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_handle) { - DAPL_IA *ia_ptr; - DAPL_SP *sp_ptr; - DAPL_EP *ep_ptr; + struct dapl_ia *ia_ptr; + struct dapl_sp *sp_ptr; + struct dapl_ep *ep_ptr; u32 dat_status = DAT_SUCCESS; - sp_ptr = (DAPL_SP *) rsp_handle; + sp_ptr = (struct dapl_sp *)rsp_handle; /* * Verify handle */ @@ -76,7 +76,7 @@ u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_han goto bail; } - /* ia_ptr = (DAPL_IA *)sp_ptr->header.owner_ia; */ + /* ia_ptr = (struct dapl_ia *)sp_ptr->header.owner_ia; */ ia_ptr = sp_ptr->header.owner_ia; /* @@ -90,17 +90,16 @@ u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_han * Make sure we don't leave a dangling EP. If the state is still * RESERVED then the RSP still owns it. */ - ep_ptr = (DAPL_EP *) sp_ptr->ep_handle; - if (ep_ptr != NULL && ep_ptr->param.ep_state == DAT_EP_STATE_RESERVED) { + ep_ptr = (struct dapl_ep *)sp_ptr->ep_handle; + if (ep_ptr != NULL && ep_ptr->param.ep_state == DAT_EP_STATE_RESERVED) ep_ptr->param.ep_state = DAT_EP_STATE_UNCONNECTED; - } sp_ptr->ep_handle = NULL; /* Release reference on EVD. If an error was encountered in a previous * free the evd_handle will be NULL */ if (sp_ptr->evd_handle) { - atomic_dec(&((DAPL_EVD *) sp_ptr->evd_handle)-> + atomic_dec(&((struct dapl_evd *)sp_ptr->evd_handle)-> evd_ref_count); sp_ptr->evd_handle = NULL; } @@ -138,6 +137,6 @@ u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_han sp_ptr->header.flags); } - bail: +bail: return dat_status; } Index: linux-kernel-resolv/dat-provider/dapl_srq_free.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_srq_free.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_srq_free.c (working copy) @@ -61,14 +61,14 @@ */ u32 dapl_srq_free(DAT_SRQ_HANDLE srq_handle) { - DAPL_SRQ *srq_ptr; - DAPL_IA *ia_ptr; + struct dapl_srq *srq_ptr; + struct dapl_ia *ia_ptr; struct dat_srq_param *param; u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_free (%p)\n", srq_handle); - srq_ptr = (DAPL_SRQ *) srq_handle; + srq_ptr = (struct dapl_srq *)srq_handle; param = &srq_ptr->param; /* Index: linux-kernel-resolv/dat-provider/dapl_ep_util.h =================================================================== --- linux-kernel-resolv/dat-provider/dapl_ep_util.h (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_ep_util.h (working copy) @@ -42,9 +42,10 @@ /* function prototypes */ -extern DAPL_EP *dapl_ep_alloc(DAPL_IA * ia, const struct dat_ep_attr *ep_attr); +extern struct dapl_ep *dapl_ep_alloc(struct dapl_ia *ia, + const struct dat_ep_attr *ep_attr); -extern void dapl_ep_dealloc(DAPL_EP * ep_ptr); +extern void dapl_ep_dealloc(struct dapl_ep *ep_ptr); extern u32 dapl_ep_check_recv_completion_flags(enum dat_completion_flags flags); @@ -58,6 +59,6 @@ extern u32 dapl_ep_post_send_req(DAT_EP_ void dapl_ep_timeout(void *arg); -enum dat_return_subtype dapl_ep_state_subtype(DAPL_EP * ep_ptr); +enum dat_return_subtype dapl_ep_state_subtype(struct dapl_ep *ep_ptr); #endif /* DAPL_EP_UTIL_H */ Index: linux-kernel-resolv/dat-provider/dapl_ep_create_with_srq.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_ep_create_with_srq.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_ep_create_with_srq.c (working copy) @@ -77,8 +77,8 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL const struct dat_ep_attr *ep_attr, DAT_EP_HANDLE *ep_handle) { - DAPL_IA *ia_ptr; - DAPL_EP *ep_ptr; + struct dapl_ia *ia_ptr; + struct dapl_ep *ep_ptr; struct dat_ep_attr *ep_attr_limit; u32 dat_status = DAT_SUCCESS; @@ -90,7 +90,7 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL request_evd_handle, connect_evd_handle, srq_handle, ep_attr, ep_handle); - ia_ptr = (DAPL_IA *) ia_handle; + ia_ptr = (struct dapl_ia *)ia_handle; /* * Verify parameters @@ -116,7 +116,7 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL /* If connect handle is !NULL verify handle is good */ if (connect_evd_handle != DAT_HANDLE_NULL && (DAPL_BAD_HANDLE(connect_evd_handle, DAPL_MAGIC_EVD) || - !(((DAPL_EVD *) connect_evd_handle)-> + !(((struct dapl_evd *)connect_evd_handle)-> evd_flags & DAT_EVD_CONNECTION_FLAG))) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CONN); @@ -125,7 +125,7 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL /* If recv_evd is !NULL, verify handle is good and flags are valid */ if (recv_evd_handle != DAT_HANDLE_NULL && (DAPL_BAD_HANDLE(recv_evd_handle, DAPL_MAGIC_EVD) || - !(((DAPL_EVD *) recv_evd_handle)->evd_flags & DAT_EVD_DTO_FLAG))) { + !(((struct dapl_evd *)recv_evd_handle)->evd_flags & DAT_EVD_DTO_FLAG))) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_RECV); goto bail; @@ -134,7 +134,7 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL /* If req_evd is !NULL, verify handle is good and flags are valid */ if (request_evd_handle != DAT_HANDLE_NULL && (DAPL_BAD_HANDLE(request_evd_handle, DAPL_MAGIC_EVD) || - !(((DAPL_EVD *) request_evd_handle)-> + !(((struct dapl_evd *)request_evd_handle)-> evd_flags & DAT_EVD_DTO_FLAG))) { dat_status = DAT_ERROR(DAT_INVALID_HANDLE, @@ -253,7 +253,7 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL */ if (pz_handle != DAT_HANDLE_NULL) { /* Take a reference on the PZ handle */ - atomic_inc(&((DAPL_PZ *) pz_handle)->pz_ref_count); + atomic_inc(&((struct dapl_pz *)pz_handle)->pz_ref_count); /* * Get a QP from the IB provider @@ -261,14 +261,12 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL dat_status = dapl_ib_qp_alloc(ia_ptr, ep_ptr, ep_ptr); if (dat_status != DAT_SUCCESS) { - atomic_dec(&((DAPL_PZ *) pz_handle)-> - pz_ref_count); + atomic_dec(&((struct dapl_pz *)pz_handle)->pz_ref_count); dapl_ep_dealloc(ep_ptr); goto bail; } - } else { + } else ep_ptr->qp_state = DAPL_QP_STATE_UNATTACHED; - } /* * Update ref counts. See the spec where the endpoint marks @@ -278,16 +276,14 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL * * N.B. This should really be done by a util routine. */ - atomic_inc(&((DAPL_EVD *) connect_evd_handle)->evd_ref_count); + atomic_inc(&((struct dapl_evd *)connect_evd_handle)->evd_ref_count); /* Optional handles */ - if (recv_evd_handle != DAT_HANDLE_NULL) { - atomic_inc(&((DAPL_EVD *) recv_evd_handle)-> + if (recv_evd_handle != DAT_HANDLE_NULL) + atomic_inc(&((struct dapl_evd *)recv_evd_handle)-> evd_ref_count); - } - if (request_evd_handle != DAT_HANDLE_NULL) { - atomic_inc(&((DAPL_EVD *) request_evd_handle)-> + if (request_evd_handle != DAT_HANDLE_NULL) + atomic_inc(&((struct dapl_evd *)request_evd_handle)-> evd_ref_count); - } /* Link it onto the IA */ dapl_ia_link_ep(ia_ptr, ep_ptr); @@ -296,6 +292,6 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL #endif /* notdef */ - bail: +bail: return dat_status; } Index: linux-kernel-resolv/dat-provider/dapl_ia_open.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_ia_open.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_ia_open.c (working copy) @@ -57,12 +57,9 @@ u32 dapl_ia_open(const char *name, int a { u32 dat_status = DAT_SUCCESS; struct dat_provider *provider; - DAPL_HCA *hca_ptr; - DAPL_IA *ia_ptr; - DAPL_EVD *evd_ptr; - - hca_ptr = NULL; - ia_ptr = NULL; + struct dapl_hca *hca_ptr = NULL; + struct dapl_ia *ia_ptr = NULL; + struct dapl_evd *evd_ptr; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ia_open (%s, %d, %p, %p)\n", @@ -87,7 +84,7 @@ u32 dapl_ia_open(const char *name, int a /* initialize the caller's param */ *ia_handle_ptr = DAT_HANDLE_NULL; - hca_ptr = (DAPL_HCA *) provider->extension; + hca_ptr = (struct dapl_hca *)provider->extension; atomic_inc(&hca_ptr->handle_ref_count); /* Allocate and initialize ia structure */ @@ -103,7 +100,7 @@ u32 dapl_ia_open(const char *name, int a * use the one passed in (if non-NULL) or create one */ - evd_ptr = (DAPL_EVD *) * async_evd_handle_ptr; + evd_ptr = (struct dapl_evd *)*async_evd_handle_ptr; if (evd_ptr) { if (DAPL_BAD_HANDLE(evd_ptr, DAPL_MAGIC_EVD) || !(evd_ptr->evd_flags & DAT_EVD_ASYNC_FLAG)) { Index: linux-kernel-resolv/dat-provider/dapl_evd_util.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_evd_util.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_evd_util.c (working copy) @@ -41,7 +41,7 @@ static inline void dapl_evd_eh_print_cqe(struct ib_wc *cqe); -u32 dapl_evd_event_alloc(DAPL_EVD *evd_ptr, int qlen); +u32 dapl_evd_event_alloc(struct dapl_evd *evd_ptr, int qlen); /* * actually create the evd. this is called after all parameter checking @@ -49,11 +49,11 @@ u32 dapl_evd_event_alloc(DAPL_EVD *evd_p * to create the default async evd. */ -u32 dapl_evd_internal_create(DAPL_IA *ia_ptr, DAPL_CNO *cno_ptr, +u32 dapl_evd_internal_create(struct dapl_ia *ia_ptr, struct dapl_cno *cno_ptr, int min_qlen, enum dat_evd_flags evd_flags, - DAPL_EVD **evd_ptr_ptr) + struct dapl_evd **evd_ptr_ptr) { - DAPL_EVD *evd_ptr; + struct dapl_evd *evd_ptr; int cq_len; u32 dat_status = DAT_SUCCESS; @@ -117,10 +117,11 @@ bail: /* * alloc and initialize an EVD struct */ -DAPL_EVD *dapl_evd_alloc(DAPL_IA * ia_ptr, DAPL_CNO * cno_ptr, - enum dat_evd_flags evd_flags, int qlen) +struct dapl_evd *dapl_evd_alloc(struct dapl_ia *ia_ptr, + struct dapl_cno *cno_ptr, + enum dat_evd_flags evd_flags, int qlen) { - DAPL_EVD *evd_ptr; + struct dapl_evd *evd_ptr; /* Allocate EVD */ evd_ptr = kmalloc(sizeof *evd_ptr, GFP_ATOMIC); @@ -161,7 +162,7 @@ bail: /* * alloc events into an EVD. */ -u32 dapl_evd_event_alloc(DAPL_EVD *evd_ptr, int qlen) +u32 dapl_evd_event_alloc(struct dapl_evd *evd_ptr, int qlen) { struct dat_event *event_ptr; int i; @@ -201,7 +202,7 @@ bail: /* * realloc events into an EVD. */ -u32 dapl_evd_event_realloc(DAPL_EVD *evd_ptr, int qlen) +u32 dapl_evd_event_realloc(struct dapl_evd *evd_ptr, int qlen) { struct dat_event *events; int i, old_qlen, diff; @@ -251,10 +252,10 @@ bail: * will clean up all of the internal data structures and report the * error. */ -u32 dapl_evd_dealloc(DAPL_EVD *evd_ptr) +u32 dapl_evd_dealloc(struct dapl_evd *evd_ptr) { u32 dat_status = DAT_SUCCESS; - DAPL_IA *ia_ptr; + struct dapl_ia *ia_ptr; dapl_os_assert(evd_ptr->header.magic == DAPL_MAGIC_EVD); dapl_os_assert(atomic_read(&evd_ptr->evd_ref_count) == 0); @@ -363,7 +364,7 @@ void dapl_evd_eh_print_cqe(struct ib_wc * that the lock is held. */ -static struct dat_event *dapl_evd_get_event(DAPL_EVD * evd_ptr) +static struct dat_event *dapl_evd_get_event(struct dapl_evd *evd_ptr) { struct dat_event *event; @@ -389,11 +390,11 @@ static struct dat_event *dapl_evd_get_ev * entry to this function. */ -static void dapl_evd_post_event(DAPL_EVD *evd_ptr, +static void dapl_evd_post_event(struct dapl_evd *evd_ptr, const struct dat_event *event_ptr) { u32 dat_status; - DAPL_CNO * cno_to_trigger = NULL; + struct dapl_cno *cno_to_trigger = NULL; dapl_dbg_log(DAPL_DBG_TYPE_EVD, "dapl_evd_post_event: Called with event # %x\n", @@ -420,10 +421,10 @@ static void dapl_evd_post_event(DAPL_EVD /* * format an overflow event for posting */ -static void -dapl_evd_format_overflow_event(DAPL_EVD * evd_ptr, struct dat_event * event_ptr) +static void dapl_evd_format_overflow_event(struct dapl_evd *evd_ptr, + struct dat_event *event_ptr) { - DAPL_IA *ia_ptr; + struct dapl_ia *ia_ptr; ia_ptr = evd_ptr->header.owner_ia; @@ -436,9 +437,8 @@ dapl_evd_format_overflow_event(DAPL_EVD /* * post an overflow event */ -static void -dapl_evd_post_overflow_event(DAPL_EVD * async_evd_ptr, - DAPL_EVD * overflow_evd_ptr) +static void dapl_evd_post_overflow_event(struct dapl_evd *async_evd_ptr, + struct dapl_evd *overflow_evd_ptr) { struct dat_event *overflow_event; @@ -464,7 +464,7 @@ dapl_evd_post_overflow_event(DAPL_EVD * return; } -static struct dat_event *dapl_evd_get_and_init_event(DAPL_EVD *evd_ptr, +static struct dat_event *dapl_evd_get_and_init_event(struct dapl_evd *evd_ptr, enum dat_event_number evno) { struct dat_event *event_ptr; @@ -481,7 +481,7 @@ static struct dat_event *dapl_evd_get_an return event_ptr; } -u32 dapl_evd_post_cr_arrival_event(DAPL_EVD *evd_ptr, +u32 dapl_evd_post_cr_arrival_event(struct dapl_evd *evd_ptr, DAT_SP_HANDLE sp_handle, struct sockaddr *ia_address_ptr, DAT_CONN_QUAL conn_qual, @@ -511,7 +511,7 @@ u32 dapl_evd_post_cr_arrival_event(DAPL_ return DAT_SUCCESS; } -u32 dapl_evd_post_connection_event(DAPL_EVD *evd_ptr, +u32 dapl_evd_post_connection_event(struct dapl_evd *evd_ptr, enum dat_event_number event_number, DAT_EP_HANDLE ep_handle, int private_data_size, @@ -540,9 +540,9 @@ u32 dapl_evd_post_connection_event(DAPL_ return DAT_SUCCESS; } -u32 dapl_evd_post_async_error_event(DAPL_EVD *evd_ptr, +u32 dapl_evd_post_async_error_event(struct dapl_evd *evd_ptr, enum dat_event_number event_number, - DAPL_IA *ia_ptr) + struct dapl_ia *ia_ptr) { struct dat_event *event_ptr; event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number); @@ -564,7 +564,7 @@ u32 dapl_evd_post_async_error_event(DAPL return DAT_SUCCESS; } -u32 dapl_evd_post_software_event(DAPL_EVD *evd_ptr, +u32 dapl_evd_post_software_event(struct dapl_evd *evd_ptr, enum dat_event_number event_number, void *pointer) { @@ -604,12 +604,12 @@ u32 dapl_evd_post_software_event(DAPL_EV * none * */ -static void -dapl_evd_cqe_to_event(DAPL_EVD * evd_ptr, struct ib_wc *cqe_ptr, - struct dat_event *event_ptr) +static void dapl_evd_cqe_to_event(struct dapl_evd *evd_ptr, + struct ib_wc *cqe_ptr, + struct dat_event *event_ptr) { - DAPL_EP *ep_ptr; - DAPL_COOKIE *cookie; + struct dapl_ep *ep_ptr; + struct dapl_cookie *cookie; enum dat_dto_completion_status dto_status; /* @@ -618,7 +618,7 @@ dapl_evd_cqe_to_event(DAPL_EVD * evd_ptr */ dto_status = dapl_ib_get_dto_status(cqe_ptr); - cookie = (DAPL_COOKIE *) (unsigned long) cqe_ptr->wr_id; + cookie = (struct dapl_cookie *) (unsigned long) cqe_ptr->wr_id; dapl_os_assert((NULL != cookie)); ep_ptr = cookie->ep; @@ -633,7 +633,7 @@ dapl_evd_cqe_to_event(DAPL_EVD * evd_ptr switch (cookie->type) { case DAPL_COOKIE_TYPE_DTO: { - DAPL_COOKIE_BUFFER *buffer; + struct dapl_cookie_buffer *buffer; if (DAPL_DTO_TYPE_RECV == cookie->val.dto.type) { atomic_dec(&ep_ptr->recv_count); @@ -740,7 +740,7 @@ dapl_evd_cqe_to_event(DAPL_EVD * evd_ptr * Status of operation * */ -u32 dapl_evd_cq_poll_to_event(DAPL_EVD *evd_ptr, struct dat_event *event) +u32 dapl_evd_cq_poll_to_event(struct dapl_evd *evd_ptr, struct dat_event *event) { u32 dat_status; struct ib_wc cur_cqe; Index: linux-kernel-resolv/dat-provider/dapl_ep_disconnect.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_ep_disconnect.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_ep_disconnect.c (working copy) @@ -66,16 +66,16 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_handle, enum dat_close_flags disconnect_flags) { - DAPL_EP *ep_ptr; - DAPL_EVD *evd_ptr; - DAPL_CR *cr_ptr; + struct dapl_ep *ep_ptr; + struct dapl_evd *evd_ptr; + struct dapl_cr *cr_ptr; u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API | DAPL_DBG_TYPE_CM, "dapl_ep_disconnect (%p, %x)\n", ep_handle, disconnect_flags); - ep_ptr = (DAPL_EP *) ep_handle; + ep_ptr = (struct dapl_ep *)ep_handle; /* * Verify parameter & state @@ -144,7 +144,7 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ /* clean up connection state */ dapl_sp_remove_ep(ep_ptr); - evd_ptr = (DAPL_EVD *) ep_ptr->param.connect_evd_handle; + evd_ptr = (struct dapl_evd *)ep_ptr->param.connect_evd_handle; dapl_evd_post_connection_event(evd_ptr, DAT_CONNECTION_EVENT_DISCONNECTED, (DAT_HANDLE) ep_ptr, 0, NULL); Index: linux-kernel-resolv/dat-provider/dapl_sp_util.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_sp_util.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_sp_util.c (working copy) @@ -59,13 +59,13 @@ * pointer to sp info struct * */ -DAPL_SP *dapl_sp_alloc(DAPL_IA * ia_ptr, boolean_t is_psp) +struct dapl_sp *dapl_sp_alloc(struct dapl_ia *ia_ptr, boolean_t is_psp) { - DAPL_SP *sp_ptr; + struct dapl_sp *sp_ptr; /* Allocate EP */ sp_ptr = kmalloc(sizeof *sp_ptr, GFP_ATOMIC); - if (!sp_ptr) + if (!sp_ptr) return NULL; /* zero the structure */ @@ -111,7 +111,7 @@ DAPL_SP *dapl_sp_alloc(DAPL_IA * ia_ptr, * none * */ -void dapl_sp_dealloc(DAPL_SP * sp_ptr) +void dapl_sp_dealloc(struct dapl_sp *sp_ptr) { dapl_os_assert(sp_ptr->header.magic == DAPL_MAGIC_PSP || sp_ptr->header.magic == DAPL_MAGIC_RSP); @@ -138,7 +138,7 @@ void dapl_sp_dealloc(DAPL_SP * sp_ptr) * none * */ -void dapl_sp_link_cr(DAPL_SP *sp_ptr, DAPL_CR *cr_ptr) +void dapl_sp_link_cr(struct dapl_sp *sp_ptr, struct dapl_cr *cr_ptr) { spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); dapl_llist_add_tail(&sp_ptr->cr_list_head, @@ -166,14 +166,15 @@ void dapl_sp_link_cr(DAPL_SP *sp_ptr, DA * cr_ptr Pointer to matching DAPL_CR * */ -DAPL_CR *dapl_sp_search_cr(DAPL_SP * sp_ptr, struct dapl_cm_id *ib_cm_handle) +struct dapl_cr *dapl_sp_search_cr(struct dapl_sp *sp_ptr, + struct dapl_cm_id *ib_cm_handle) { - DAPL_CR *cr_ptr; + struct dapl_cr *cr_ptr; if (dapl_llist_is_empty(&sp_ptr->cr_list_head)) return NULL; - cr_ptr = (DAPL_CR *) dapl_llist_peek_head(&sp_ptr->cr_list_head); + cr_ptr = (struct dapl_cr *)dapl_llist_peek_head(&sp_ptr->cr_list_head); do { if (cr_ptr->ib_cm_handle == ib_cm_handle) @@ -203,7 +204,7 @@ DAPL_CR *dapl_sp_search_cr(DAPL_SP * sp_ * void * */ -void dapl_sp_remove_cr(DAPL_SP * sp_ptr, DAPL_CR * cr_ptr) +void dapl_sp_remove_cr(struct dapl_sp *sp_ptr, struct dapl_cr *cr_ptr) { if (dapl_llist_is_empty(&sp_ptr->cr_list_head)) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, @@ -233,10 +234,10 @@ void dapl_sp_remove_cr(DAPL_SP * sp_ptr, * void * */ -void dapl_sp_remove_ep(DAPL_EP * ep_ptr) +void dapl_sp_remove_ep(struct dapl_ep *ep_ptr) { - DAPL_SP *sp_ptr; - DAPL_CR *cr_ptr; + struct dapl_sp *sp_ptr; + struct dapl_cr *cr_ptr; cr_ptr = ep_ptr->cr_ptr; @@ -253,7 +254,5 @@ void dapl_sp_remove_ep(DAPL_EP * ep_ptr) /* free memory outside of the lock */ dapl_cr_free(cr_ptr); - - return; } } Index: linux-kernel-resolv/dat-provider/dapl_evd_util.h =================================================================== --- linux-kernel-resolv/dat-provider/dapl_evd_util.h (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_evd_util.h (working copy) @@ -40,16 +40,17 @@ #include "dapl.h" -u32 dapl_evd_internal_create(DAPL_IA *ia_ptr, DAPL_CNO *cno_ptr, +u32 dapl_evd_internal_create(struct dapl_ia *ia_ptr, struct dapl_cno *cno_ptr, int min_qlen, enum dat_evd_flags evd_flags, - DAPL_EVD **evd_ptr_ptr); + struct dapl_evd **evd_ptr_ptr); -DAPL_EVD *dapl_evd_alloc(DAPL_IA * ia_ptr, DAPL_CNO * cno_ptr, - enum dat_evd_flags evd_flags, int qlen); +struct dapl_evd *dapl_evd_alloc(struct dapl_ia *ia_ptr, + struct dapl_cno *cno_ptr, + enum dat_evd_flags evd_flags, int qlen); -u32 dapl_evd_dealloc(DAPL_EVD *evd_ptr); +u32 dapl_evd_dealloc(struct dapl_evd *evd_ptr); -u32 dapl_evd_event_realloc(DAPL_EVD *evd_ptr, int qlen); +u32 dapl_evd_event_realloc(struct dapl_evd *evd_ptr, int qlen); /* * Each of these functions will retrieve a free event from @@ -62,23 +63,23 @@ u32 dapl_evd_event_realloc(DAPL_EVD *evd * DAT_SUCCESS otherwise. */ -u32 dapl_evd_post_cr_arrival_event(DAPL_EVD *evd_ptr, +u32 dapl_evd_post_cr_arrival_event(struct dapl_evd *evd_ptr, DAT_SP_HANDLE sp_handle, struct sockaddr *ia_address_ptr, DAT_CONN_QUAL conn_qual, DAT_CR_HANDLE cr_handle); -u32 dapl_evd_post_connection_event(DAPL_EVD *evd_ptr, +u32 dapl_evd_post_connection_event(struct dapl_evd *evd_ptr, enum dat_event_number event_number, DAT_EP_HANDLE ep_handle, int private_data_size, void *private_data); -u32 dapl_evd_post_async_error_event(DAPL_EVD *evd_ptr, +u32 dapl_evd_post_async_error_event(struct dapl_evd *evd_ptr, enum dat_event_number event_number, - DAPL_IA *ia_ptr); + struct dapl_ia *ia_ptr); -u32 dapl_evd_post_software_event(DAPL_EVD *evd_ptr, +u32 dapl_evd_post_software_event(struct dapl_evd *evd_ptr, enum dat_event_number event_number, void *pointer); @@ -105,7 +106,7 @@ extern void dapl_evd_cq_async_error_call extern void dapl_evd_qp_async_error_callback(struct ib_event *cause_ptr, void *context); -extern u32 dapl_evd_cq_poll_to_event(DAPL_EVD *evd_ptr, +extern u32 dapl_evd_cq_poll_to_event(struct dapl_evd *evd_ptr, struct dat_event *event); #endif Index: linux-kernel-resolv/dat-provider/dapl_hash.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_hash.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_hash.c (working copy) @@ -161,7 +161,7 @@ dapl_hash_rehash(DAPL_HASH_ELEM * elemen * TRUE on success */ static boolean_t -dapl_hash_add(DAPL_HASH_TABLEP p_table, DAPL_HASH_KEY key, void *datum, +dapl_hash_add(struct dapl_hash_table *table, DAPL_HASH_KEY key, void *datum, boolean_t allow_dup, boolean_t *report_dup) { void *olddatum; @@ -184,7 +184,7 @@ dapl_hash_add(DAPL_HASH_TABLEP p_table, return FALSE; } - DAPL_HASHLOOKUP(p_table, key, olddatum, found); + DAPL_HASHLOOKUP(table, key, olddatum, found); if (found) { /* * key exists already @@ -201,16 +201,16 @@ dapl_hash_add(DAPL_HASH_TABLEP p_table, } } - hashValue = DAPL_DOHASH(key, p_table->tbl_size); + hashValue = DAPL_DOHASH(key, table->tbl_size); key = save_key; - if (NO_DATUM(p_table->table[hashValue].datum)) { + if (NO_DATUM(table->table[hashValue].datum)) { /* * Empty head - just fill it in */ - p_table->table[hashValue].key = key; - p_table->table[hashValue].datum = datum; - p_table->table[hashValue].next_element = NULL; - p_table->num_entries++; + table->table[hashValue].key = key; + table->table[hashValue].datum = datum; + table->table[hashValue].next_element = NULL; + table->num_entries++; status = TRUE; } else { DAPL_HASH_ELEM *newelement = kmalloc(sizeof *newelement, @@ -223,14 +223,14 @@ dapl_hash_add(DAPL_HASH_TABLEP p_table, newelement->key = key; newelement->datum = datum; newelement->next_element = NULL; - for (lastelement = &p_table->table[hashValue]; + for (lastelement = &table->table[hashValue]; lastelement->next_element; lastelement = lastelement->next_element) { /* Walk to the end of the chain */ chain_len++; } lastelement->next_element = newelement; - p_table->num_entries++; + table->num_entries++; status = TRUE; } else status = FALSE; @@ -241,14 +241,14 @@ dapl_hash_add(DAPL_HASH_TABLEP p_table, * length. */ chain_len++; - p_table->hash_tbl_inserts++; - p_table->hash_tbl_total += p_table->num_entries; - p_table->hash_chn_total += chain_len; - if (p_table->num_entries > p_table->hash_tbl_max) { - p_table->hash_tbl_max = p_table->num_entries; + table->hash_tbl_inserts++; + table->hash_tbl_total += table->num_entries; + table->hash_chn_total += chain_len; + if (table->num_entries > table->hash_tbl_max) { + table->hash_tbl_max = table->num_entries; } - if (chain_len > p_table->hash_chn_max) { - p_table->hash_chn_max = chain_len; + if (chain_len > table->hash_chn_max) { + table->hash_chn_max = chain_len; } return status; @@ -319,9 +319,9 @@ dapl_hash_delete_element(DAPL_HASH_ELEM /* * Create a new hash table with at least 'table_size' hash buckets. */ -u32 dapl_hash_create(int table_size, DAPL_HASH_TABLE **pp_table) +u32 dapl_hash_create(int table_size, struct dapl_hash_table **pp_table) { - DAPL_HASH_TABLE *p_table; + struct dapl_hash_table *p_table; int table_length = table_size * sizeof (struct DAPL_HASH_ELEM); u32 dat_status = DAT_SUCCESS; int i; @@ -356,14 +356,14 @@ u32 dapl_hash_create(int table_size, DAP *pp_table = p_table; - bail: +bail: return DAT_SUCCESS; } /* * Destroy a hash table */ -u32 dapl_hash_free(DAPL_HASH_TABLE *p_table) +u32 dapl_hash_free(struct dapl_hash_table *p_table) { dapl_os_assert(p_table && p_table->table); @@ -378,7 +378,7 @@ u32 dapl_hash_free(DAPL_HASH_TABLE *p_ta * Returns the number of elements stored in the table */ -u32 dapl_hash_size(DAPL_HASH_TABLE *p_table, int *p_size) +u32 dapl_hash_size(struct dapl_hash_table *p_table, int *p_size) { dapl_os_assert(p_table && p_size); @@ -392,7 +392,7 @@ u32 dapl_hash_size(DAPL_HASH_TABLE *p_ta * Duplicates are not expected, and return in error, having done nothing. */ -u32 dapl_hash_insert(DAPL_HASH_TABLE *p_table, DAPL_HASH_KEY key, +u32 dapl_hash_insert(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, DAPL_HASH_DATA data) { u32 dat_status = DAT_SUCCESS; @@ -415,7 +415,7 @@ u32 dapl_hash_insert(DAPL_HASH_TABLE *p_ * data is returned in the DAPL_HASH_DATA * pointer if that pointer is not NULL. */ -u32 dapl_hash_search(DAPL_HASH_TABLE *p_table, DAPL_HASH_KEY key, +u32 dapl_hash_search(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, DAPL_HASH_DATA *p_data) { u32 dat_status; @@ -439,7 +439,7 @@ u32 dapl_hash_search(DAPL_HASH_TABLE *p_ return dat_status; } -u32 dapl_hash_remove(DAPL_HASH_TABLE *p_table, DAPL_HASH_KEY key, +u32 dapl_hash_remove(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, DAPL_HASH_DATA *p_data) { u32 dat_status; Index: linux-kernel-resolv/dat-provider/dapl_sp_util.h =================================================================== --- linux-kernel-resolv/dat-provider/dapl_sp_util.h (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_sp_util.h (working copy) @@ -37,16 +37,17 @@ #ifndef DAPL_PSP_UTIL_H #define DAPL_PSP_UTIL_H -DAPL_SP *dapl_sp_alloc(DAPL_IA * ia_ptr, boolean_t is_psp); +struct dapl_sp *dapl_sp_alloc(struct dapl_ia *ia_ptr, boolean_t is_psp); -void dapl_sp_dealloc(DAPL_SP * sp_ptr); +void dapl_sp_dealloc(struct dapl_sp *sp_ptr); -void dapl_sp_link_cr(DAPL_SP * sp_ptr, DAPL_CR * cr_ptr); +void dapl_sp_link_cr(struct dapl_sp *sp_ptr, struct dapl_cr *cr_ptr); -DAPL_CR *dapl_sp_search_cr(DAPL_SP * sp_ptr, struct dapl_cm_id *ib_cm_handle); +struct dapl_cr *dapl_sp_search_cr(struct dapl_sp *sp_ptr, + struct dapl_cm_id *ib_cm_handle); -void dapl_sp_remove_cr(DAPL_SP * sp_ptr, DAPL_CR * cr_ptr); +void dapl_sp_remove_cr(struct dapl_sp *sp_ptr, struct dapl_cr *cr_ptr); -void dapl_sp_remove_ep(DAPL_EP * ep_ptr); +void dapl_sp_remove_ep(struct dapl_ep *ep_ptr); -#endif /* DAPL_PSP_UTIL_H */ +#endif /* DAPL_PSP_UTIL_H */ Index: linux-kernel-resolv/dat-provider/dapl_ep_dup_connect.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_ep_dup_connect.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_ep_dup_connect.c (working copy) @@ -73,12 +73,12 @@ u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep DAT_TIMEOUT timeout, int private_data_size, const void *private_data, enum dat_qos qos) { - DAPL_EP *ep_dup_ptr; + struct dapl_ep *ep_dup_ptr; struct sockaddr *remote_ia_address_ptr; DAT_CONN_QUAL remote_conn_qual; u32 dat_status; - ep_dup_ptr = (DAPL_EP *) ep_dup_handle; + ep_dup_ptr = (struct dapl_ep *)ep_dup_handle; /* * Verify the dup handle, which must be connected. All other Index: linux-kernel-resolv/dat-provider/dapl.h =================================================================== --- linux-kernel-resolv/dat-provider/dapl.h (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl.h (working copy) @@ -123,7 +123,7 @@ typedef enum dapl_evd_completion { #define DAPL_BAD_HANDLE(h, magicNum) ( \ ((h) == NULL) || \ ((unsigned long)(h) & 3) || \ - (((DAPL_HEADER *)(h))->magic != (magicNum))) + (((struct dapl_header *)(h))->magic != (magicNum))) #define DAPL_MIN(a, b) ((a < b) ? (a) : (b)) #define DAPL_MAX(a, b) ((a > b) ? (a) : (b)) @@ -158,35 +158,9 @@ typedef struct ib_hca_transport { u16 lid; } ib_hca_transport_t; -typedef struct dapl_llist_entry DAPL_LLIST_ENTRY; -typedef DAPL_LLIST_ENTRY *DAPL_LLIST_HEAD; -typedef struct dapl_ring_buffer DAPL_RING_BUFFER; -typedef struct dapl_cookie_buffer DAPL_COOKIE_BUFFER; - -typedef struct dapl_hash_table DAPL_HASH_TABLE; -typedef struct dapl_hash_table *DAPL_HASH_TABLEP; typedef u64 DAPL_HASH_KEY; typedef void *DAPL_HASH_DATA; -typedef struct dapl_hca DAPL_HCA; - -typedef struct dapl_header DAPL_HEADER; - -typedef struct dapl_ia DAPL_IA; -typedef struct dapl_cno DAPL_CNO; -typedef struct dapl_evd DAPL_EVD; -typedef struct dapl_ep DAPL_EP; -typedef struct dapl_srq DAPL_SRQ; -typedef struct dapl_pz DAPL_PZ; -typedef struct dapl_lmr DAPL_LMR; -typedef struct dapl_rmr DAPL_RMR; -typedef struct dapl_sp DAPL_SP; -typedef struct dapl_cr DAPL_CR; - -typedef struct dapl_cookie DAPL_COOKIE; -typedef struct dapl_dto_cookie DAPL_DTO_COOKIE; -typedef struct dapl_rmr_cookie DAPL_RMR_COOKIE; - /********************************************************************* * * * Structures * @@ -197,7 +171,7 @@ struct dapl_llist_entry { struct dapl_llist_entry *flink; struct dapl_llist_entry *blink; void *data; - DAPL_LLIST_HEAD *list_head; /* for consistency checking */ + struct dapl_llist_entry **list_head; /* for consistency checking */ }; struct dapl_ring_buffer { @@ -208,7 +182,7 @@ struct dapl_ring_buffer { }; struct dapl_cookie_buffer { - DAPL_COOKIE *pool; + struct dapl_cookie *pool; int pool_size; atomic_t head; atomic_t tail; @@ -217,7 +191,7 @@ struct dapl_cookie_buffer { struct dapl_hca { spinlock_t lock; unsigned long flags; - DAPL_LLIST_HEAD ia_list_head; /* list of all open IAs */ + struct dapl_llist_entry *ia_list_head; /* list of all open IAs */ atomic_t handle_ref_count; /* count of ia_opens on handle */ struct sockaddr_in6 hca_address; /* local address of HCA */ char *name; /* provider name */ @@ -225,7 +199,7 @@ struct dapl_hca { u8 port_num; /* number of physical port */ ib_hca_transport_t ib_trans; /* Values specific transport API */ /* Memory Subsystem Support */ - DAPL_HASH_TABLE *lmr_hash_table; + struct dapl_hash_table *lmr_hash_table; /* Limits & useful HCA attributes */ struct dat_ia_attr ia_attr; struct dat_ep_attr ep_attr; @@ -236,8 +210,8 @@ struct dapl_header { struct dat_provider *provider; /* required by DAT - must be first */ DAPL_MAGIC magic; /* magic number for verification */ enum dat_handle_type handle_type; /* struct type */ - DAPL_IA *owner_ia; /* ia which owns this stuct */ - DAPL_LLIST_ENTRY ia_list_entry; /* link entry on ia struct */ + struct dapl_ia *owner_ia; /* ia which owns this struct */ + struct dapl_llist_entry ia_list_entry; /* link entry on ia struct */ DAT_CONTEXT user_context; /* user context - opaque to DAPL */ spinlock_t lock; /* lock - in header for easier macros */ unsigned long flags; /* saved lock flag values */ @@ -245,32 +219,32 @@ struct dapl_header { /* DAPL_IA maps to DAT_IA_HANDLE */ struct dapl_ia { - DAPL_HEADER header; - DAPL_HCA *hca_ptr; - DAPL_EVD *async_error_evd; + struct dapl_header header; /* WARNING: must be first */ + struct dapl_hca *hca_ptr; + struct dapl_evd *async_error_evd; boolean_t cleanup_async_error_evd; - DAPL_LLIST_ENTRY hca_ia_list_entry; /* HCAs list of IAs */ - DAPL_LLIST_HEAD ep_list_head; /* EP queue */ - DAPL_LLIST_HEAD lmr_list_head; /* LMR queue */ - DAPL_LLIST_HEAD rmr_list_head; /* RMR queue */ - DAPL_LLIST_HEAD pz_list_head; /* PZ queue */ - DAPL_LLIST_HEAD evd_list_head; /* EVD queue */ - DAPL_LLIST_HEAD psp_list_head; /* PSP queue */ - DAPL_LLIST_HEAD rsp_list_head; /* RSP queue */ - DAPL_LLIST_HEAD srq_list_head; /* SRQ queue */ + struct dapl_llist_entry hca_ia_list_entry; /* HCAs list of IAs */ + struct dapl_llist_entry *ep_list_head; /* EP queue */ + struct dapl_llist_entry *lmr_list_head; /* LMR queue */ + struct dapl_llist_entry *rmr_list_head; /* RMR queue */ + struct dapl_llist_entry *pz_list_head; /* PZ queue */ + struct dapl_llist_entry *evd_list_head; /* EVD queue */ + struct dapl_llist_entry *psp_list_head; /* PSP queue */ + struct dapl_llist_entry *rsp_list_head; /* RSP queue */ + struct dapl_llist_entry *srq_list_head; /* SRQ queue */ }; /* DAPL_CNO maps to DAT_CNO_HANDLE */ struct dapl_cno { - DAPL_HEADER header; + struct dapl_header header; /* WARNING: must be first */ struct dat_upcall_object cno_upcall; enum dat_upcall_policy cno_upcall_policy; }; /* DAPL_EVD maps to DAT_EVD_HANDLE */ struct dapl_evd { - DAPL_HEADER header; + struct dapl_header header; /* WARNING: must be first */ DAPL_EVD_STATE evd_state; enum dat_evd_flags evd_flags; @@ -292,22 +266,22 @@ struct dapl_evd { /* the actual events */ int qlen; struct dat_event *events; - DAPL_RING_BUFFER free_event_queue; - DAPL_RING_BUFFER pending_event_queue; - DAPL_CNO *cno_ptr; + struct dapl_ring_buffer free_event_queue; + struct dapl_ring_buffer pending_event_queue; + struct dapl_cno *cno_ptr; }; /* uDAPL timer entry, used to queue timeouts */ struct dapl_timer_entry { - DAPL_LLIST_ENTRY list_entry; /* link entry on ia struct */ + struct dapl_llist_entry list_entry; /* link entry on ia struct */ DAPL_OS_TIMEVAL expires; void (*function) (void *); void *data; }; -/* DAPL_EP maps to DAT_EP_HANDLE */ +/* struct dapl_ep maps to DAT_EP_HANDLE */ struct dapl_ep { - DAPL_HEADER header; + struct dapl_header header; /* WARNING: must be first */ /* What the DAT Consumer asked for */ struct dat_ep_param param; @@ -336,8 +310,8 @@ struct dapl_ep { atomic_t req_count; atomic_t recv_count; - DAPL_COOKIE_BUFFER req_buffer; - DAPL_COOKIE_BUFFER recv_buffer; + struct dapl_cookie_buffer req_buffer; + struct dapl_cookie_buffer recv_buffer; struct ib_sge *recv_iov; int recv_iov_num; @@ -346,37 +320,37 @@ struct dapl_ep { int send_iov_num; }; -/* DAPL_SRQ maps to DAT_SRQ_HANDLE */ +/* struct dapl_srq maps to DAT_SRQ_HANDLE */ struct dapl_srq { - DAPL_HEADER header; + struct dapl_header header; /* WARNING: must be first */ struct dat_srq_param param; atomic_t srq_ref_count; - DAPL_COOKIE_BUFFER recv_buffer; + struct dapl_cookie_buffer recv_buffer; atomic_t recv_count; }; -/* DAPL_PZ maps to DAT_PZ_HANDLE */ +/* struct dapl_pz maps to DAT_PZ_HANDLE */ struct dapl_pz { - DAPL_HEADER header; + struct dapl_header header; /* WARNING: must be first */ struct ib_pd *pd_handle; atomic_t pz_ref_count; }; -/* DAPL_LMR maps to DAT_LMR_HANDLE */ +/* struct dapl_lmr maps to DAT_LMR_HANDLE */ struct dapl_lmr { - DAPL_HEADER header; + struct dapl_header header; /* WARNING: must be first */ struct dat_lmr_param param; struct ib_mr *mr_handle; atomic_t lmr_ref_count; }; -/* DAPL_RMR maps to DAT_RMR_HANDLE */ +/* struct dapl_rmr maps to DAT_RMR_HANDLE */ struct dapl_rmr { - DAPL_HEADER header; + struct dapl_header header; /* WARNING: must be first */ struct dat_rmr_param param; - DAPL_EP *ep; - DAPL_PZ *pz; - DAPL_LMR *lmr; + struct dapl_ep *ep; + struct dapl_pz *pz; + struct dapl_lmr *lmr; struct ib_mw *mw_handle; }; @@ -389,9 +363,9 @@ typedef enum dapl_sp_state { DAPL_SP_STATE_RSP_PENDING } DAPL_SP_STATE; -/* DAPL_SP maps to DAT_PSP_HANDLE and DAT_RSP_HANDLE */ +/* struct dapl_sp maps to DAT_PSP_HANDLE and DAT_RSP_HANDLE */ struct dapl_sp { - DAPL_HEADER header; + struct dapl_header header; /* WARNING: must be first */ DAPL_SP_STATE state; /* type and queue of the SP */ /* PSP/RSP PARAM fields */ @@ -404,13 +378,13 @@ struct dapl_sp { /* maintenence fields */ boolean_t listening; /* PSP is registered & active */ struct ib_cm_id *cm_srvc_handle; /* Used by CM */ - DAPL_LLIST_HEAD cr_list_head; /* CR pending queue */ + struct dapl_llist_entry *cr_list_head; /* CR pending queue */ int cr_list_count; /* count of CRs on queue */ }; -/* DAPL_CR maps to DAT_CR_HANDLE */ +/* struct dapl_cr maps to DAT_CR_HANDLE */ struct dapl_cr { - DAPL_HEADER header; + struct dapl_header header; /* WARNING: must be first */ /* for convenience the data is kept as a struct dat_cr_param. * however, the "local_endpoint" field is always NULL @@ -430,7 +404,7 @@ struct dapl_cr { * Need to be able to associate the CR back to the PSP for * dapl_cr_reject. */ - DAPL_SP *sp_ptr; + struct dapl_sp *sp_ptr; }; typedef enum dapl_dto_type { @@ -446,27 +420,27 @@ typedef enum dapl_cookie_type { DAPL_COOKIE_TYPE_RMR, } DAPL_COOKIE_TYPE; -/* DAPL_DTO_COOKIE used as context for DTO WQEs */ +/* struct dapl_dto_cookie used as context for DTO WQEs */ struct dapl_dto_cookie { DAPL_DTO_TYPE type; DAT_DTO_COOKIE cookie; int size; /* used for SEND and RDMA write */ }; -/* DAPL_RMR_COOKIE used as context for bind WQEs */ +/* struct dapl_rmr_cookie used as context for bind WQEs */ struct dapl_rmr_cookie { - DAPL_RMR *rmr; + struct dapl_rmr *rmr; DAT_RMR_COOKIE cookie; }; -/* DAPL_COOKIE used as context for WQEs */ +/* struct dapl_cookie used as context for WQEs */ struct dapl_cookie { DAPL_COOKIE_TYPE type; /* Must be first, to define struct. */ - DAPL_EP *ep; + struct dapl_ep *ep; int index; union { - DAPL_DTO_COOKIE dto; - DAPL_RMR_COOKIE rmr; + struct dapl_dto_cookie dto; + struct dapl_rmr_cookie rmr; } val; }; @@ -737,34 +711,35 @@ extern u32 dapl_srq_set_lw(DAT_SRQ_HANDL * DAPL internal utility function prototpyes */ -extern void dapl_llist_init_head(DAPL_LLIST_HEAD * head); +extern void dapl_llist_init_head(struct dapl_llist_entry **head); -extern void dapl_llist_init_entry(DAPL_LLIST_ENTRY * entry); +extern void dapl_llist_init_entry(struct dapl_llist_entry *entry); -extern boolean_t dapl_llist_is_empty(DAPL_LLIST_HEAD * head); +extern boolean_t dapl_llist_is_empty(struct dapl_llist_entry **head); -extern void dapl_llist_add_head(DAPL_LLIST_HEAD * head, - DAPL_LLIST_ENTRY * entry, void *data); +extern void dapl_llist_add_head(struct dapl_llist_entry **head, + struct dapl_llist_entry *entry, void *data); -extern void dapl_llist_add_tail(DAPL_LLIST_HEAD * head, - DAPL_LLIST_ENTRY * entry, void *data); +extern void dapl_llist_add_tail(struct dapl_llist_entry **head, + struct dapl_llist_entry *entry, void *data); -extern void dapl_llist_add_entry(DAPL_LLIST_HEAD * head, - DAPL_LLIST_ENTRY * entry, - DAPL_LLIST_ENTRY * new_entry, void *data); +extern void dapl_llist_add_entry(struct dapl_llist_entry **head, + struct dapl_llist_entry *entry, + struct dapl_llist_entry *new_entry, + void *data); -extern void *dapl_llist_remove_head(DAPL_LLIST_HEAD * head); +extern void *dapl_llist_remove_head(struct dapl_llist_entry **head); -extern void *dapl_llist_remove_tail(DAPL_LLIST_HEAD * head); +extern void *dapl_llist_remove_tail(struct dapl_llist_entry **head); -extern void *dapl_llist_remove_entry(DAPL_LLIST_HEAD * head, - DAPL_LLIST_ENTRY * entry); +extern void *dapl_llist_remove_entry(struct dapl_llist_entry **head, + struct dapl_llist_entry *entry); -extern void *dapl_llist_peek_head(DAPL_LLIST_HEAD * head); +extern void *dapl_llist_peek_head(struct dapl_llist_entry **head); -extern void *dapl_llist_next_entry(DAPL_LLIST_HEAD * head, - DAPL_LLIST_ENTRY * cur_ent); +extern void *dapl_llist_next_entry(struct dapl_llist_entry **head, + struct dapl_llist_entry *cur_ent); -extern void dapl_llist_debug_print_list(DAPL_LLIST_HEAD * head); +extern void dapl_llist_debug_print_list(struct dapl_llist_entry **head); #endif Index: linux-kernel-resolv/dat-provider/dapl_psp_query.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_psp_query.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_psp_query.c (working copy) @@ -40,11 +40,11 @@ u32 dapl_psp_query(DAT_PSP_HANDLE psp_handle, struct dat_psp_param *psp_param) { - DAPL_SP *sp_ptr; + struct dapl_sp *sp_ptr; u32 status; if (DAPL_BAD_HANDLE(psp_handle, DAPL_MAGIC_PSP) || - ((DAPL_SP *) psp_handle)->listening != TRUE) { + ((struct dapl_sp *)psp_handle)->listening != TRUE) { status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PSP); goto bail; @@ -55,7 +55,7 @@ u32 dapl_psp_query(DAT_PSP_HANDLE psp_ha goto bail; } - sp_ptr = (DAPL_SP *) psp_handle; + sp_ptr = (struct dapl_sp *)psp_handle; psp_param->ia_handle = sp_ptr->ia_handle; psp_param->conn_qual = sp_ptr->conn_qual; @@ -64,6 +64,6 @@ u32 dapl_psp_query(DAT_PSP_HANDLE psp_ha status = DAT_SUCCESS; - bail: +bail: return status; } Index: linux-kernel-resolv/dat-provider/dapl_llist.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_llist.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_llist.c (working copy) @@ -57,7 +57,7 @@ * * Purpose: initialize a linked list head */ -void dapl_llist_init_head(DAPL_LLIST_HEAD * head) +void dapl_llist_init_head(struct dapl_llist_entry **head) { *head = NULL; } @@ -67,7 +67,7 @@ void dapl_llist_init_head(DAPL_LLIST_HEA * * Purpose: initialize a linked list entry */ -void dapl_llist_init_entry(DAPL_LLIST_ENTRY * entry) +void dapl_llist_init_entry(struct dapl_llist_entry *entry) { entry->blink = NULL; entry->flink = NULL; @@ -80,7 +80,7 @@ void dapl_llist_init_entry(DAPL_LLIST_EN * * Purpose: returns TRUE if the linked list is empty */ -boolean_t dapl_llist_is_empty(DAPL_LLIST_HEAD * head) +boolean_t dapl_llist_is_empty(struct dapl_llist_entry **head) { return (*head == NULL); } @@ -91,10 +91,10 @@ boolean_t dapl_llist_is_empty(DAPL_LLIST * Purpose: Add an entry to the head of a linked list */ void -dapl_llist_add_head(DAPL_LLIST_HEAD * head, - DAPL_LLIST_ENTRY * entry, void *data) +dapl_llist_add_head(struct dapl_llist_entry **head, + struct dapl_llist_entry *entry, void *data) { - DAPL_LLIST_ENTRY *first; + struct dapl_llist_entry *first; /* deal with empty list */ if (dapl_llist_is_empty(head)) { @@ -119,10 +119,10 @@ dapl_llist_add_head(DAPL_LLIST_HEAD * he * Purpose: Add an entry to the tail of a linked list */ void -dapl_llist_add_tail(DAPL_LLIST_HEAD * head, - DAPL_LLIST_ENTRY * entry, void *data) +dapl_llist_add_tail(struct dapl_llist_entry **head, + struct dapl_llist_entry *entry, void *data) { - DAPL_LLIST_ENTRY *last; + struct dapl_llist_entry *last; /* deal with empty list */ if (dapl_llist_is_empty(head)) { @@ -147,12 +147,12 @@ dapl_llist_add_tail(DAPL_LLIST_HEAD * he * not verify the list or the validity of the entries passed in. */ void -dapl_llist_add_entry(DAPL_LLIST_HEAD * head, - DAPL_LLIST_ENTRY * entry, - DAPL_LLIST_ENTRY * new_entry, void *data) +dapl_llist_add_entry(struct dapl_llist_entry **head, + struct dapl_llist_entry *entry, + struct dapl_llist_entry *new_entry, void *data) { - DAPL_LLIST_ENTRY *first; - DAPL_LLIST_ENTRY *prev; + struct dapl_llist_entry *first; + struct dapl_llist_entry *prev; /* get the head pointer */ first = *head; @@ -180,9 +180,9 @@ dapl_llist_add_entry(DAPL_LLIST_HEAD * h * * Purpose: Remove the first entry of a linked list */ -void *dapl_llist_remove_head(DAPL_LLIST_HEAD * head) +void *dapl_llist_remove_head(struct dapl_llist_entry **head) { - DAPL_LLIST_ENTRY *first; + struct dapl_llist_entry *first; dapl_os_assert(!dapl_llist_is_empty(head)); first = *head; @@ -206,9 +206,9 @@ void *dapl_llist_remove_head(DAPL_LLIST_ * * Purpose: Remove the last entry of a linked list */ -void *dapl_llist_remove_tail(DAPL_LLIST_HEAD * head) +void *dapl_llist_remove_tail(struct dapl_llist_entry **head) { - DAPL_LLIST_ENTRY *last; + struct dapl_llist_entry *last; dapl_os_assert(!dapl_llist_is_empty(head)); last = (*head)->blink; @@ -232,9 +232,10 @@ void *dapl_llist_remove_tail(DAPL_LLIST_ * * Purpose: Remove the specified entry from a linked list */ -void *dapl_llist_remove_entry(DAPL_LLIST_HEAD * head, DAPL_LLIST_ENTRY * entry) +void *dapl_llist_remove_entry(struct dapl_llist_entry **head, + struct dapl_llist_entry *entry) { - DAPL_LLIST_ENTRY *first; + struct dapl_llist_entry *first; dapl_os_assert(!dapl_llist_is_empty(head)); first = *head; @@ -263,9 +264,9 @@ void *dapl_llist_remove_entry(DAPL_LLIST * dapl_llist_peek_head */ -void *dapl_llist_peek_head(DAPL_LLIST_HEAD * head) +void *dapl_llist_peek_head(struct dapl_llist_entry **head) { - DAPL_LLIST_ENTRY *first; + struct dapl_llist_entry *first; dapl_os_assert(!dapl_llist_is_empty(head)); first = *head; @@ -279,9 +280,10 @@ void *dapl_llist_peek_head(DAPL_LLIST_HE * head */ -void *dapl_llist_next_entry(DAPL_LLIST_HEAD * head, DAPL_LLIST_ENTRY * cur_ent) +void *dapl_llist_next_entry(struct dapl_llist_entry **head, + struct dapl_llist_entry *cur_ent) { - DAPL_LLIST_ENTRY *next; + struct dapl_llist_entry *next; dapl_os_assert(!dapl_llist_is_empty(head)); if (cur_ent == NULL) { @@ -301,10 +303,10 @@ void *dapl_llist_next_entry(DAPL_LLIST_H * * Purpose: Prints the linked list for debugging */ -void dapl_llist_debug_print_list(DAPL_LLIST_HEAD * head) +void dapl_llist_debug_print_list(struct dapl_llist_entry **head) { - DAPL_LLIST_ENTRY *entry; - DAPL_LLIST_ENTRY *first; + struct dapl_llist_entry *entry; + struct dapl_llist_entry *first; first = *head; if (!first) { dapl_dbg_log(DAPL_DBG_TYPE_UTIL, "EMPTY_LIST\n"); Index: linux-kernel-resolv/dat-provider/dapl_hash.h =================================================================== --- linux-kernel-resolv/dat-provider/dapl_hash.h (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_hash.h (working copy) @@ -67,19 +67,19 @@ * * *********************************************************************/ -extern u32 dapl_hash_create(int capacity, DAPL_HASH_TABLE **pp_table); +extern u32 dapl_hash_create(int capacity, struct dapl_hash_table **pp_table); -extern u32 dapl_hash_free(DAPL_HASH_TABLE *p_table); +extern u32 dapl_hash_free(struct dapl_hash_table *p_table); -extern u32 dapl_hash_size(DAPL_HASH_TABLE *p_table, int *p_size); +extern u32 dapl_hash_size(struct dapl_hash_table *p_table, int *p_size); -extern u32 dapl_hash_insert(DAPL_HASH_TABLE *p_table, DAPL_HASH_KEY key, +extern u32 dapl_hash_insert(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, DAPL_HASH_DATA data); -extern u32 dapl_hash_search(DAPL_HASH_TABLE *p_table, DAPL_HASH_KEY key, +extern u32 dapl_hash_search(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, DAPL_HASH_DATA *p_data); -extern u32 dapl_hash_remove(DAPL_HASH_TABLE *p_table, DAPL_HASH_KEY key, +extern u32 dapl_hash_remove(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, DAPL_HASH_DATA *p_data); #endif /* DAPL_HASH_H */ Index: linux-kernel-resolv/dat-provider/dapl_lmr_kcreate.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_lmr_kcreate.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_lmr_kcreate.c (working copy) @@ -37,8 +37,8 @@ #include "dapl_lmr_util.h" #include "dapl_openib_util.h" -static inline u32 dapl_lmr_create_virtual(DAPL_IA *ia, void *virt_addr, - u64 length, DAPL_PZ *pz, +static inline u32 dapl_lmr_create_virtual(struct dapl_ia *ia, void *virt_addr, + u64 length, struct dapl_pz *pz, enum dat_mem_priv_flags privileges, DAT_LMR_HANDLE *lmr_handle, DAT_LMR_CONTEXT *lmr_context, @@ -46,7 +46,7 @@ static inline u32 dapl_lmr_create_virtua u64 *registered_length, u64 *registered_address) { - DAPL_LMR *lmr; + struct dapl_lmr *lmr; DAT_REGION_DESCRIPTION reg_desc; u32 dat_status = DAT_SUCCESS; @@ -114,9 +114,9 @@ bail: } -static inline u32 dapl_lmr_create_physical(DAPL_IA *ia, +static inline u32 dapl_lmr_create_physical(struct dapl_ia *ia, DAT_REGION_DESCRIPTION phys_addr, - u64 page_count, DAPL_PZ *pz, + u64 page_count, struct dapl_pz *pz, enum dat_mem_priv_flags privileges, DAT_LMR_HANDLE *lmr_handle, DAT_LMR_CONTEXT *lmr_context, @@ -124,7 +124,7 @@ static inline u32 dapl_lmr_create_physic u64 *registered_length, u64 *registered_address) { - DAPL_LMR *lmr; + struct dapl_lmr *lmr; u32 dat_status = DAT_SUCCESS; u64 *array = phys_addr.for_array; @@ -190,8 +190,9 @@ bail: return dat_status; } -static inline u32 dapl_lmr_create_lmr(DAPL_IA *ia, DAPL_LMR *original_lmr, - DAPL_PZ *pz, +static inline u32 dapl_lmr_create_lmr(struct dapl_ia *ia, + struct dapl_lmr *original_lmr, + struct dapl_pz *pz, enum dat_mem_priv_flags privileges, DAT_LMR_HANDLE *lmr_handle, DAT_LMR_CONTEXT *lmr_context, @@ -199,7 +200,7 @@ static inline u32 dapl_lmr_create_lmr(DA u64 *registered_length, u64 *registered_address) { - DAPL_LMR *lmr; + struct dapl_lmr *lmr; DAT_REGION_DESCRIPTION reg_desc; u32 dat_status; @@ -309,8 +310,8 @@ u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_ha enum dat_mem_optimize_flags optimization, DAT_LMR_HANDLE *lmr_handle, DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address) { - DAPL_IA *ia; - DAPL_PZ *pz; + struct dapl_ia *ia; + struct dapl_pz *pz; u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, @@ -328,8 +329,8 @@ u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_ha goto bail; } - ia = (DAPL_IA *) ia_handle; - pz = (DAPL_PZ *) pz_handle; + ia = (struct dapl_ia *)ia_handle; + pz = (struct dapl_pz *)pz_handle; switch (mem_type) { case DAT_MEM_TYPE_VIRTUAL: @@ -358,7 +359,7 @@ u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_ha } case DAT_MEM_TYPE_LMR: { - DAPL_LMR *lmr; + struct dapl_lmr *lmr; if (DAPL_BAD_HANDLE (region_description.for_lmr_handle, @@ -369,7 +370,7 @@ u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_ha goto bail; } - lmr = (DAPL_LMR *) region_description.for_lmr_handle; + lmr = (struct dapl_lmr *)region_description.for_lmr_handle; dat_status = dapl_lmr_create_lmr(ia, lmr, pz, privileges, @@ -394,6 +395,6 @@ u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_ha } } - bail: +bail: return dat_status; } Index: linux-kernel-resolv/dat-provider/dapl_ep_post_recv.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_ep_post_recv.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_ep_post_recv.c (working copy) @@ -71,8 +71,8 @@ u32 dapl_ep_post_recv(DAT_EP_HANDLE ep_h DAT_DTO_COOKIE user_cookie, enum dat_completion_flags completion_flags) { - DAPL_EP *ep_ptr; - DAPL_COOKIE *cookie; + struct dapl_ep *ep_ptr; + struct dapl_cookie *cookie; u32 dat_status; dapl_dbg_log(DAPL_DBG_TYPE_API, @@ -87,7 +87,7 @@ u32 dapl_ep_post_recv(DAT_EP_HANDLE ep_h goto bail; } - ep_ptr = (DAPL_EP *) ep_handle; + ep_ptr = (struct dapl_ep *)ep_handle; /* * Synchronization ok since this buffer is only used for receive Index: linux-kernel-resolv/dat-provider/dapl_cr_query.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_cr_query.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_cr_query.c (working copy) @@ -36,7 +36,7 @@ u32 dapl_cr_query(DAT_CR_HANDLE cr_handle, struct dat_cr_param *cr_param) { - DAPL_CR *cr_ptr; + struct dapl_cr *cr_ptr; u32 status; dapl_dbg_log(DAPL_DBG_TYPE_API, @@ -53,7 +53,7 @@ u32 dapl_cr_query(DAT_CR_HANDLE cr_handl goto bail; } - cr_ptr = (DAPL_CR *) cr_handle; + cr_ptr = (struct dapl_cr *)cr_handle; /* obtain the remote IP address */ (void) dapl_ib_cm_remote_addr((DAT_HANDLE) cr_handle, @@ -64,6 +64,6 @@ u32 dapl_cr_query(DAT_CR_HANDLE cr_handl status = DAT_SUCCESS; - bail: +bail: return status; } Index: linux-kernel-resolv/dat-provider/dapl_rsp_query.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_rsp_query.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_rsp_query.c (working copy) @@ -40,7 +40,7 @@ u32 dapl_rsp_query(DAT_RSP_HANDLE rsp_handle, struct dat_rsp_param *rsp_param) { - DAPL_SP *sp_ptr; + struct dapl_sp *sp_ptr; u32 status; if (DAPL_BAD_HANDLE(rsp_handle, DAPL_MAGIC_RSP)) { @@ -53,7 +53,7 @@ u32 dapl_rsp_query(DAT_RSP_HANDLE rsp_ha goto bail; } - sp_ptr = (DAPL_SP *) rsp_handle; + sp_ptr = (struct dapl_sp *)rsp_handle; /* * Fill in the RSP params @@ -65,6 +65,6 @@ u32 dapl_rsp_query(DAT_RSP_HANDLE rsp_ha status = DAT_SUCCESS; - bail: +bail: return status; } Index: linux-kernel-resolv/dat-provider/dapl_pz_free.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_pz_free.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_pz_free.c (working copy) @@ -56,7 +56,7 @@ */ u32 dapl_pz_free(DAT_PZ_HANDLE pz_handle) { - DAPL_PZ *pz; + struct dapl_pz *pz; u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_free (%p)\n", pz_handle); @@ -67,7 +67,7 @@ u32 dapl_pz_free(DAT_PZ_HANDLE pz_handle goto bail; } - pz = (DAPL_PZ *) pz_handle; + pz = (struct dapl_pz *)pz_handle; if (0 != atomic_read(&pz->pz_ref_count)) { dat_status = Index: linux-kernel-resolv/dat-provider/dapl_ep_set_watermark.c =================================================================== --- linux-kernel-resolv/dat-provider/dapl_ep_set_watermark.c (revision 2431) +++ linux-kernel-resolv/dat-provider/dapl_ep_set_watermark.c (working copy) @@ -63,13 +63,13 @@ u32 dapl_ep_set_watermark(DAT_EP_HANDLE int soft_high_watermark, int hard_high_watermark) { - DAPL_EP *ep_ptr; + struct dapl_ep *ep_ptr; u32 dat_status = DAT_SUCCESS; dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_set_watermark (%p, %d, %d)\n", ep_handle, soft_high_watermark, hard_high_watermark); - ep_ptr = (DAPL_EP *) ep_handle; + ep_ptr = (struct dapl_ep *)ep_handle; /* * Verify parameter & state From jcarr at linuxmachines.com Fri May 20 17:57:15 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Fri, 20 May 2005 17:57:15 -0700 Subject: [openib-general] in need of a simple ulp In-Reply-To: <428E4B4A.2090705@ichips.intel.com> References: <426FCF0A.3070806@linuxmachines.com> <426FD098.9090300@ichips.intel.com> <427AC218.9070307@linuxmachines.com> <427AC385.5070308@ichips.intel.com> <427BA0AD.6010105@linuxmachines.com> <427FDA9D.7030905@ichips.intel.com> <428E4268.1040101@linuxmachines.com> <428E4B4A.2090705@ichips.intel.com> Message-ID: <428E876B.8070704@linuxmachines.com> Sean Hefty wrote: > In this case you won't know what's on a cqe until you poll the > completion. The cmpost test only poll's on the completion on the client side. I'll change it to run ib_poll_cq() on the server also. Now I'm so confused I'm not sure why I wanted this in the first place though :) > An alternative way of structuring the code is to create two > CQs, and tie one to the send queue, the other to the receive queue. Sure that makes sense. Another thing I've noticed is that I can't seem to get comp_handler() or event_handler() to run. You pass them into ib_create_cq(), but perhaps this isn't an implemented feature or some other things must be set for this to work. Jeff From jlentini at netapp.com Fri May 20 21:04:49 2005 From: jlentini at netapp.com (James Lentini) Date: Sat, 21 May 2005 00:04:49 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: remove use of typedefs for structs in dapl.h In-Reply-To: <1116635670.16690.11.camel@duffman> References: <1116635670.16690.11.camel@duffman> Message-ID: No problem. Committed in revision 2433. On Fri, 20 May 2005, Tom Duffy wrote: tduffy> James, tduffy> tduffy> I am sorry this is such a large patch. I had broken it out into 19 tduffy> different patches, each changing one typedef, but after the large tduffy> changes you made today, it was going to be a big PITA to regenerate them tduffy> all. So, I just rebased my final tree with r2431 and generated one tduffy> large patch. tduffy> tduffy> This patch removes the use of many of the struct typedefs in dapl.h. tduffy> tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> Index: linux-kernel-resolv/dat-provider/dapl_evd_cq_async_error_callb.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_evd_cq_async_error_callb.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_evd_cq_async_error_callb.c (working copy) tduffy> @@ -53,22 +53,19 @@ tduffy> * tduffy> */ tduffy> tduffy> -void tduffy> -dapl_evd_cq_async_error_callback(struct ib_event *cause_ptr, void *context) tduffy> +void dapl_evd_cq_async_error_callback(struct ib_event *cause_ptr, void *context) tduffy> { tduffy> - DAPL_EVD *async_evd; tduffy> - DAPL_EVD *evd; tduffy> + struct dapl_evd *async_evd; tduffy> + struct dapl_evd *evd = context; tduffy> u32 dat_status; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK | DAPL_DBG_TYPE_EXCEPTION, tduffy> "dapl_evd_cq_async_error_callback (%p, %p)\n", tduffy> cause_ptr, context); tduffy> tduffy> - if (NULL == context) { tduffy> + if (!evd) tduffy> dapl_os_panic("NULL == context\n"); tduffy> - } tduffy> tduffy> - evd = (DAPL_EVD *) context; tduffy> async_evd = evd->header.owner_ia->async_error_evd; tduffy> tduffy> dat_status = dapl_evd_post_async_error_event(async_evd, tduffy> Index: linux-kernel-resolv/dat-provider/dapl_cookie.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_cookie.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_cookie.c (working copy) tduffy> @@ -84,9 +84,11 @@ tduffy> * * tduffy> *********************************************************************/ tduffy> tduffy> -u32 dapl_cb_get(DAPL_COOKIE_BUFFER *buffer, DAPL_COOKIE **cookie_ptr); tduffy> +u32 dapl_cb_get(struct dapl_cookie_buffer *buffer, tduffy> + struct dapl_cookie **cookie_ptr); tduffy> tduffy> -u32 dapl_cb_put(DAPL_COOKIE_BUFFER *buffer, DAPL_COOKIE *cookie); tduffy> +u32 dapl_cb_put(struct dapl_cookie_buffer *buffer, tduffy> + struct dapl_cookie *cookie); tduffy> tduffy> /********************************************************************* tduffy> * * tduffy> @@ -97,11 +99,11 @@ u32 dapl_cb_put(DAPL_COOKIE_BUFFER *buff tduffy> /* tduffy> * dapl_cb_create tduffy> * tduffy> - * Given a DAPL_COOKIE_BUFFER, allocate and initialize memory for tduffy> + * Given a struct dapl_cookie_buffer, allocate and initialize memory for tduffy> * the data structure. tduffy> * tduffy> * Input: tduffy> - * buffer pointer to DAPL_COOKIE_BUFFER tduffy> + * buffer pointer to struct dapl_cookie_buffer tduffy> * ep endpoint to associate with cookies tduffy> * size number of elements to allocate & manage tduffy> * tduffy> @@ -113,7 +115,8 @@ u32 dapl_cb_put(DAPL_COOKIE_BUFFER *buff tduffy> * DAT_INSUFFICIENT_RESOURCES tduffy> * tduffy> */ tduffy> -u32 dapl_cb_create(DAPL_COOKIE_BUFFER *buffer, DAPL_EP *ep, int size) tduffy> +u32 dapl_cb_create(struct dapl_cookie_buffer *buffer, struct dapl_ep *ep, tduffy> + int size) tduffy> { tduffy> int i; tduffy> tduffy> @@ -145,7 +148,7 @@ u32 dapl_cb_create(DAPL_COOKIE_BUFFER *b tduffy> * Free the data structure tduffy> * tduffy> * Input: tduffy> - * buffer pointer to DAPL_COOKIE_BUFFER tduffy> + * buffer pointer to struct dapl_cookie_buffer tduffy> * tduffy> * Output: tduffy> * none tduffy> @@ -154,7 +157,7 @@ u32 dapl_cb_create(DAPL_COOKIE_BUFFER *b tduffy> * none tduffy> * tduffy> */ tduffy> -void dapl_cb_free(DAPL_COOKIE_BUFFER * buffer) tduffy> +void dapl_cb_free(struct dapl_cookie_buffer *buffer) tduffy> { tduffy> if (NULL != buffer->pool) tduffy> kfree(buffer->pool); tduffy> @@ -166,7 +169,7 @@ void dapl_cb_free(DAPL_COOKIE_BUFFER * b tduffy> * Remove an entry from the buffer tduffy> * tduffy> * Input: tduffy> - * buffer pointer to DAPL_COOKIE_BUFFER tduffy> + * buffer pointer to struct dapl_cookie_buffer tduffy> * tduffy> * Output: tduffy> * cookie_ptr pointer to pointer to cookie tduffy> @@ -177,7 +180,8 @@ void dapl_cb_free(DAPL_COOKIE_BUFFER * b tduffy> * DAT_INSUFFICIENT_RESOURCES tduffy> * tduffy> */ tduffy> -u32 dapl_cb_get(DAPL_COOKIE_BUFFER *buffer, DAPL_COOKIE **cookie_ptr) tduffy> +u32 dapl_cb_get(struct dapl_cookie_buffer *buffer, tduffy> + struct dapl_cookie **cookie_ptr) tduffy> { tduffy> u32 dat_status; tduffy> int new_head; tduffy> @@ -195,7 +199,7 @@ u32 dapl_cb_get(DAPL_COOKIE_BUFFER *buff tduffy> *cookie_ptr = &buffer->pool[atomic_read(&buffer->head)]; tduffy> dat_status = DAT_SUCCESS; tduffy> } tduffy> - bail: tduffy> +bail: tduffy> return dat_status; tduffy> } tduffy> tduffy> @@ -205,7 +209,7 @@ u32 dapl_cb_get(DAPL_COOKIE_BUFFER *buff tduffy> * Add entry(s) to the buffer tduffy> * tduffy> * Input: tduffy> - * buffer pointer to DAPL_COOKIE_BUFFER tduffy> + * buffer pointer to struct dapl_cookie_buffer tduffy> * cookie pointer to cookie tduffy> * tduffy> * Output: tduffy> @@ -216,7 +220,7 @@ u32 dapl_cb_get(DAPL_COOKIE_BUFFER *buff tduffy> * DAT_INSUFFICIENT_EMPTY tduffy> * tduffy> */ tduffy> -u32 dapl_cb_put(DAPL_COOKIE_BUFFER *buffer, DAPL_COOKIE *cookie) tduffy> +u32 dapl_cb_put(struct dapl_cookie_buffer *buffer, struct dapl_cookie *cookie) tduffy> { tduffy> atomic_set(&buffer->tail, cookie->index); tduffy> tduffy> @@ -229,7 +233,7 @@ u32 dapl_cb_put(DAPL_COOKIE_BUFFER *buff tduffy> * Allocate an RMR Bind cookie tduffy> * tduffy> * Input: tduffy> - * buffer pointer to DAPL_COOKIE_BUFFER tduffy> + * buffer pointer to struct dapl_cookie_buffer tduffy> * rmr rmr to associate with the cookie tduffy> * user_cookie user's cookie data tduffy> * tduffy> @@ -241,10 +245,11 @@ u32 dapl_cb_put(DAPL_COOKIE_BUFFER *buff tduffy> * DAT_INSUFFICIENT_EMPTY tduffy> * tduffy> */ tduffy> -u32 dapl_rmr_cookie_alloc(DAPL_COOKIE_BUFFER *buffer, DAPL_RMR *rmr, tduffy> - DAT_RMR_COOKIE user_cookie, DAPL_COOKIE **cookie_ptr) tduffy> +u32 dapl_rmr_cookie_alloc(struct dapl_cookie_buffer *buffer, tduffy> + struct dapl_rmr *rmr, DAT_RMR_COOKIE user_cookie, tduffy> + struct dapl_cookie **cookie_ptr) tduffy> { tduffy> - DAPL_COOKIE *cookie; tduffy> + struct dapl_cookie *cookie; tduffy> u32 dat_status; tduffy> tduffy> if (DAT_SUCCESS != dapl_cb_get(buffer, &cookie)) { tduffy> @@ -261,7 +266,7 @@ u32 dapl_rmr_cookie_alloc(DAPL_COOKIE_BU tduffy> tduffy> *cookie_ptr = cookie; tduffy> tduffy> - bail: tduffy> +bail: tduffy> return dat_status; tduffy> } tduffy> tduffy> @@ -271,7 +276,7 @@ u32 dapl_rmr_cookie_alloc(DAPL_COOKIE_BU tduffy> * Allocate a DTO cookie tduffy> * tduffy> * Input: tduffy> - * buffer pointer to DAPL_COOKIE_BUFFER tduffy> + * buffer pointer to struct dapl_cookie_buffer tduffy> * type DTO type tduffy> * user_cookie user's cookie data tduffy> * tduffy> @@ -283,10 +288,10 @@ u32 dapl_rmr_cookie_alloc(DAPL_COOKIE_BU tduffy> * DAT_INSUFFICIENT_EMPTY tduffy> * tduffy> */ tduffy> -u32 dapl_dto_cookie_alloc(DAPL_COOKIE_BUFFER *buffer, DAPL_DTO_TYPE type, tduffy> - DAT_DTO_COOKIE user_cookie, DAPL_COOKIE **cookie_ptr) tduffy> +u32 dapl_dto_cookie_alloc(struct dapl_cookie_buffer *buffer, DAPL_DTO_TYPE type, tduffy> + DAT_DTO_COOKIE user_cookie, struct dapl_cookie **cookie_ptr) tduffy> { tduffy> - DAPL_COOKIE *cookie; tduffy> + struct dapl_cookie *cookie; tduffy> tduffy> if (DAT_SUCCESS != dapl_cb_get(buffer, &cookie)) { tduffy> *cookie_ptr = NULL; tduffy> @@ -303,7 +308,8 @@ u32 dapl_dto_cookie_alloc(DAPL_COOKIE_BU tduffy> return DAT_SUCCESS; tduffy> } tduffy> tduffy> -void dapl_cookie_dealloc(DAPL_COOKIE_BUFFER * buffer, DAPL_COOKIE * cookie) tduffy> +void dapl_cookie_dealloc(struct dapl_cookie_buffer *buffer, tduffy> + struct dapl_cookie * cookie) tduffy> { tduffy> dapl_cb_put(buffer, cookie); tduffy> } tduffy> Index: linux-kernel-resolv/dat-provider/dapl_ia_memtype_hint.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_ia_memtype_hint.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_ia_memtype_hint.c (working copy) tduffy> @@ -69,13 +69,13 @@ u32 dapl_ia_memtype_hint(DAT_IA_HANDLE i tduffy> u64 *suggested_length, tduffy> u64 *suggested_alignment) tduffy> { tduffy> - DAPL_IA *ia_ptr; tduffy> + struct dapl_ia *ia_ptr; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ia_memtype_hint (%p)\n", tduffy> ia_handle); tduffy> tduffy> - ia_ptr = (DAPL_IA *) ia_handle; tduffy> + ia_ptr = (struct dapl_ia *)ia_handle; tduffy> tduffy> if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { tduffy> dat_status = tduffy> Index: linux-kernel-resolv/dat-provider/dapl_ep_connect.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_ep_connect.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_ep_connect.c (working copy) tduffy> @@ -48,8 +48,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han tduffy> int private_data_size, const void *private_data, tduffy> enum dat_qos qos, enum dat_connect_flags connect_flags) tduffy> { tduffy> - DAPL_EP *ep_ptr; tduffy> - DAPL_EP alloc_ep; tduffy> + struct dapl_ep *ep_ptr, alloc_ep; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_API | DAPL_DBG_TYPE_CM, tduffy> @@ -63,7 +62,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han tduffy> timeout, tduffy> private_data_size, private_data, qos, connect_flags); tduffy> tduffy> - ep_ptr = (DAPL_EP *) ep_handle; tduffy> + ep_ptr = (struct dapl_ep *)ep_handle; tduffy> tduffy> /* tduffy> * Verify parameter & state. The connection handle must be good tduffy> @@ -174,7 +173,7 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han tduffy> /* test connect handle */ tduffy> || ep_ptr->param.connect_evd_handle == NULL tduffy> || DAPL_BAD_HANDLE(ep_ptr->param.connect_evd_handle, DAPL_MAGIC_EVD) tduffy> - || !(((DAPL_EVD *) ep_ptr->param.connect_evd_handle)-> tduffy> + || !(((struct dapl_evd *)ep_ptr->param.connect_evd_handle)-> tduffy> evd_flags & DAT_EVD_CONNECTION_FLAG) tduffy> /* test optional completion handles */ tduffy> || (ep_ptr->param.recv_evd_handle != DAT_HANDLE_NULL && tduffy> @@ -261,8 +260,8 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han tduffy> * this happens, return SUCCESS and generate the event tduffy> */ tduffy> if (dat_status == DAT_ERROR(DAT_INTERNAL_ERROR, 1)) { tduffy> - dapl_evd_post_connection_event((DAPL_EVD *) ep_ptr-> tduffy> - param. tduffy> + dapl_evd_post_connection_event((struct dapl_evd *) tduffy> + ep_ptr->param. tduffy> connect_evd_handle, tduffy> DAT_CONNECTION_EVENT_UNREACHABLE, tduffy> (DAT_HANDLE) ep_ptr, 0, tduffy> Index: linux-kernel-resolv/dat-provider/dapl_cookie.h tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_cookie.h (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_cookie.h (working copy) tduffy> @@ -36,20 +36,21 @@ tduffy> tduffy> #include "dapl.h" tduffy> tduffy> -extern u32 dapl_cb_create(DAPL_COOKIE_BUFFER *buffer, DAPL_EP *ep, tduffy> +extern u32 dapl_cb_create(struct dapl_cookie_buffer *buffer, struct dapl_ep *ep, tduffy> int size); tduffy> tduffy> -extern void dapl_cb_free(DAPL_COOKIE_BUFFER *buffer); tduffy> +extern void dapl_cb_free(struct dapl_cookie_buffer *buffer); tduffy> tduffy> -extern u32 dapl_rmr_cookie_alloc(DAPL_COOKIE_BUFFER *buffer, DAPL_RMR *rmr, tduffy> +extern u32 dapl_rmr_cookie_alloc(struct dapl_cookie_buffer *buffer, tduffy> + struct dapl_rmr *rmr, tduffy> DAT_RMR_COOKIE user_cookie, tduffy> - DAPL_COOKIE **cookie_ptr); tduffy> + struct dapl_cookie **cookie_ptr); tduffy> tduffy> -extern u32 dapl_dto_cookie_alloc(DAPL_COOKIE_BUFFER *buffer, tduffy> +extern u32 dapl_dto_cookie_alloc(struct dapl_cookie_buffer *buffer, tduffy> DAPL_DTO_TYPE type, DAT_DTO_COOKIE user_cookie, tduffy> - DAPL_COOKIE **cookie_ptr); tduffy> + struct dapl_cookie **cookie_ptr); tduffy> tduffy> -extern void tduffy> -dapl_cookie_dealloc(DAPL_COOKIE_BUFFER * buffer, DAPL_COOKIE * cookie); tduffy> +extern void dapl_cookie_dealloc(struct dapl_cookie_buffer *buffer, tduffy> + struct dapl_cookie *cookie); tduffy> tduffy> #endif /* DAPL_COOKIE_H */ tduffy> Index: linux-kernel-resolv/dat-provider/dapl_cr_callback.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_cr_callback.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_cr_callback.c (working copy) tduffy> @@ -45,11 +45,13 @@ tduffy> /* tduffy> * Prototypes tduffy> */ tduffy> -u32 dapl_connection_request(struct dapl_cm_id *ib_cm_handle, DAPL_SP *sp_ptr, tduffy> - const void *prd_ptr, DAPL_EVD *evd_ptr); tduffy> - tduffy> -DAPL_EP *dapl_get_sp_ep(struct dapl_cm_id *ib_cm_handle, DAPL_SP * sp_ptr, tduffy> - enum dat_event_number event); tduffy> +u32 dapl_connection_request(struct dapl_cm_id *ib_cm_handle, tduffy> + struct dapl_sp *sp_ptr, const void *prd_ptr, tduffy> + struct dapl_evd *evd_ptr); tduffy> + tduffy> +struct dapl_ep *dapl_get_sp_ep(struct dapl_cm_id *ib_cm_handle, tduffy> + struct dapl_sp *sp_ptr, tduffy> + enum dat_event_number dat_event_num); tduffy> tduffy> /* tduffy> * The callback function registered with verbs for passive side of tduffy> @@ -60,9 +62,9 @@ void dapl_cr_callback(struct dapl_cm_id tduffy> const void *private_data_ptr, /* event data */ tduffy> const void *context) tduffy> { tduffy> - DAPL_EP *ep_ptr; tduffy> - DAPL_EVD *evd_ptr; tduffy> - DAPL_SP *sp_ptr; tduffy> + struct dapl_ep *ep_ptr; tduffy> + struct dapl_evd *evd_ptr; tduffy> + struct dapl_sp *sp_ptr; tduffy> u32 dat_status; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_CM | DAPL_DBG_TYPE_CALLBACK, tduffy> @@ -73,7 +75,7 @@ void dapl_cr_callback(struct dapl_cm_id tduffy> * Passive side of the connection, context is a SP and tduffy> * we need to look up the EP. tduffy> */ tduffy> - sp_ptr = (DAPL_SP *) context; tduffy> + sp_ptr = (struct dapl_sp *)context; tduffy> /* tduffy> * The context pointer could have been cleaned up in a racing tduffy> * CM callback, check to see if we should just exit here tduffy> @@ -106,7 +108,7 @@ void dapl_cr_callback(struct dapl_cm_id tduffy> if (ep_ptr == NULL) { tduffy> return; tduffy> } tduffy> - evd_ptr = (DAPL_EVD *) ep_ptr->param.connect_evd_handle; tduffy> + evd_ptr = (struct dapl_evd *)ep_ptr->param.connect_evd_handle; tduffy> /* if something has happened to our EVD, bail. */ tduffy> if (evd_ptr == NULL) { tduffy> return; tduffy> @@ -259,13 +261,14 @@ void dapl_cr_callback(struct dapl_cm_id tduffy> * Create a CR record and link it on to the SP so we can update it tduffy> * and free it later. Create an EP if specified by the PSP flags. tduffy> */ tduffy> -u32 dapl_connection_request(struct dapl_cm_id *ib_cm_handle, DAPL_SP *sp_ptr, tduffy> - const void *prd_ptr, DAPL_EVD *evd_ptr) tduffy> +u32 dapl_connection_request(struct dapl_cm_id *ib_cm_handle, tduffy> + struct dapl_sp *sp_ptr, const void *prd_ptr, tduffy> + struct dapl_evd *evd_ptr) tduffy> { tduffy> u32 dat_status; tduffy> - DAPL_CR *cr_ptr; tduffy> - DAPL_EP *ep_ptr; tduffy> - DAPL_IA *ia_ptr; tduffy> + struct dapl_cr *cr_ptr; tduffy> + struct dapl_ep *ep_ptr; tduffy> + struct dapl_ia *ia_ptr; tduffy> DAT_SP_HANDLE sp_handle; tduffy> tduffy> cr_ptr = dapl_cr_alloc(sp_ptr->header.owner_ia); tduffy> @@ -295,7 +298,7 @@ u32 dapl_connection_request(struct dapl_ tduffy> } tduffy> tduffy> /* EP will be NULL unless RSP service point */ tduffy> - ep_ptr = (DAPL_EP *) sp_ptr->ep_handle; tduffy> + ep_ptr = (struct dapl_ep *)sp_ptr->ep_handle; tduffy> tduffy> if (sp_ptr->psp_flags == DAT_PSP_PROVIDER_FLAG) { tduffy> /* tduffy> @@ -362,11 +365,12 @@ u32 dapl_connection_request(struct dapl_ tduffy> * up resources and obtain the EP pointer associated with a CR in tduffy> * the SP. tduffy> */ tduffy> -DAPL_EP *dapl_get_sp_ep(struct dapl_cm_id *ib_cm_handle, tduffy> - DAPL_SP * sp_ptr, enum dat_event_number event) tduffy> +struct dapl_ep *dapl_get_sp_ep(struct dapl_cm_id *ib_cm_handle, tduffy> + struct dapl_sp *sp_ptr, tduffy> + enum dat_event_number event) tduffy> { tduffy> - DAPL_CR *cr_ptr; tduffy> - DAPL_EP *ep_ptr; tduffy> + struct dapl_cr *cr_ptr; tduffy> + struct dapl_ep *ep_ptr; tduffy> tduffy> /* tduffy> * acquire the lock, we may be racing with other threads here tduffy> @@ -391,7 +395,7 @@ DAPL_EP *dapl_get_sp_ep(struct dapl_cm_i tduffy> return NULL; tduffy> } tduffy> tduffy> - ep_ptr = (DAPL_EP *) cr_ptr->param.local_ep_handle; tduffy> + ep_ptr = (struct dapl_ep *)cr_ptr->param.local_ep_handle; tduffy> tduffy> /* Quick check to ensure our EP is still valid */ tduffy> if ((DAPL_BAD_HANDLE(ep_ptr, DAPL_MAGIC_EP))) { tduffy> @@ -427,8 +431,7 @@ DAPL_EP *dapl_get_sp_ep(struct dapl_cm_i tduffy> sp_ptr); tduffy> /* Decrement the ref count on the EVD */ tduffy> if (sp_ptr->evd_handle) { tduffy> - atomic_dec(& tduffy> - ((DAPL_EVD *) sp_ptr-> tduffy> + atomic_dec(&((struct dapl_evd *)sp_ptr-> tduffy> evd_handle)->evd_ref_count); tduffy> sp_ptr->evd_handle = NULL; tduffy> } tduffy> @@ -437,7 +440,8 @@ DAPL_EP *dapl_get_sp_ep(struct dapl_cm_i tduffy> sp_ptr->header.flags); tduffy> (void)dapl_ib_remove_conn_listener(sp_ptr->header. tduffy> owner_ia, sp_ptr); tduffy> - dapl_ia_unlink_sp((DAPL_IA *) sp_ptr->header.owner_ia, tduffy> + dapl_ia_unlink_sp((struct dapl_ia *) tduffy> + sp_ptr->header.owner_ia, tduffy> sp_ptr); tduffy> dapl_sp_dealloc(sp_ptr); tduffy> dapl_cr_free(cr_ptr); tduffy> Index: linux-kernel-resolv/dat-provider/dapl_evd_dto_callb.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_evd_dto_callb.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_evd_dto_callb.c (working copy) tduffy> @@ -70,7 +70,7 @@ tduffy> tduffy> void dapl_evd_dto_callback(struct ib_cq *cq_handle, void *user_context) tduffy> { tduffy> - DAPL_EVD *evd_ptr; tduffy> + struct dapl_evd *evd_ptr; tduffy> u32 dat_status; tduffy> DAPL_EVD_STATE state; tduffy> tduffy> @@ -78,7 +78,7 @@ void dapl_evd_dto_callback(struct ib_cq tduffy> "dapl_evd_dto_callback(%p, %p)\n", tduffy> cq_handle, user_context); tduffy> tduffy> - evd_ptr = (DAPL_EVD *) user_context; tduffy> + evd_ptr = (struct dapl_evd *)user_context; tduffy> tduffy> dapl_os_assert(evd_ptr->ib_cq_handle == cq_handle); tduffy> dapl_os_assert(evd_ptr->header.magic == DAPL_MAGIC_EVD); tduffy> Index: linux-kernel-resolv/dat-provider/dapl_rmr_create.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_rmr_create.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_rmr_create.c (working copy) tduffy> @@ -55,8 +55,8 @@ tduffy> */ tduffy> u32 dapl_rmr_create(DAT_PZ_HANDLE pz_handle, DAT_RMR_HANDLE *rmr_handle) tduffy> { tduffy> - DAPL_PZ *pz; tduffy> - DAPL_RMR *rmr; tduffy> + struct dapl_pz *pz; tduffy> + struct dapl_rmr *rmr; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { tduffy> @@ -65,7 +65,7 @@ u32 dapl_rmr_create(DAT_PZ_HANDLE pz_han tduffy> goto bail; tduffy> } tduffy> tduffy> - pz = (DAPL_PZ *) pz_handle; tduffy> + pz = (struct dapl_pz *)pz_handle; tduffy> tduffy> rmr = dapl_rmr_alloc(pz); tduffy> tduffy> @@ -89,6 +89,6 @@ u32 dapl_rmr_create(DAT_PZ_HANDLE pz_han tduffy> tduffy> *rmr_handle = rmr; tduffy> tduffy> - bail: tduffy> +bail: tduffy> return dat_status; tduffy> } tduffy> Index: linux-kernel-resolv/dat-provider/dapl_lmr_sync_rdma_read.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_lmr_sync_rdma_read.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_lmr_sync_rdma_read.c (working copy) tduffy> @@ -60,14 +60,14 @@ u32 dapl_lmr_sync_rdma_read(DAT_IA_HANDL tduffy> const struct dat_lmr_triplet *local_segments, tduffy> u64 num_segments) tduffy> { tduffy> - DAPL_IA *ia_ptr; tduffy> + struct dapl_ia *ia_ptr; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> "dat_lmr_sync_rdma_read (%p, %p, %ld)\n", tduffy> ia_handle, local_segments, num_segments); tduffy> tduffy> - ia_ptr = (DAPL_IA *) ia_handle; tduffy> + ia_ptr = (struct dapl_ia *)ia_handle; tduffy> tduffy> if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { tduffy> dat_status = tduffy> Index: linux-kernel-resolv/dat-provider/dapl_module.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_module.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_module.c (working copy) tduffy> @@ -120,7 +120,7 @@ void DAT_PROVIDER_INIT_FUNC_NAME(const s tduffy> struct ib_device *device, u8 port) tduffy> { tduffy> struct dat_provider *provider; tduffy> - DAPL_HCA *hca_ptr; tduffy> + struct dapl_hca *hca_ptr; tduffy> u32 dat_status; tduffy> tduffy> provider = NULL; tduffy> Index: linux-kernel-resolv/dat-provider/dapl_ring_buffer_util.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_ring_buffer_util.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_ring_buffer_util.c (working copy) tduffy> @@ -40,12 +40,12 @@ tduffy> /* tduffy> * dapl_rbuf_alloc tduffy> * tduffy> - * Given a DAPL_RING_BUFFER, initialize it and provide memory for tduffy> + * Given a struct dapl_ring_buffer, initialize it and provide memory for tduffy> * the ringbuf itself. A passed in size will be adjusted to the next tduffy> * largest power of two number to simplify management. tduffy> * tduffy> * Input: tduffy> - * rbuf pointer to DAPL_RING_BUFFER tduffy> + * rbuf pointer to struct dapl_ring_buffer tduffy> * size number of elements to allocate & manage tduffy> * tduffy> * Output: tduffy> @@ -56,7 +56,7 @@ tduffy> * DAT_INSUFFICIENT_RESOURCES tduffy> * tduffy> */ tduffy> -u32 dapl_rbuf_alloc(DAPL_RING_BUFFER *rbuf, int size) tduffy> +u32 dapl_rbuf_alloc(struct dapl_ring_buffer *rbuf, int size) tduffy> { tduffy> unsigned int rsize; /* real size */ tduffy> tduffy> @@ -86,12 +86,12 @@ u32 dapl_rbuf_alloc(DAPL_RING_BUFFER *rb tduffy> /* tduffy> * dapl_rbuf_realloc tduffy> * tduffy> - * Resizes an empty DAPL_RING_BUFFER. This function is not thread safe; tduffy> + * Resizes an empty struct dapl_ring_buffer. This function is not thread safe; tduffy> * adding or removing elements from a ring buffer while resizing tduffy> * will have indeterminate results. tduffy> * tduffy> * Input: tduffy> - * rbuf pointer to DAPL_RING_BUFFER tduffy> + * rbuf pointer to struct dapl_ring_buffer tduffy> * size number of elements to allocate & manage tduffy> * tduffy> * Output: tduffy> @@ -103,10 +103,10 @@ u32 dapl_rbuf_alloc(DAPL_RING_BUFFER *rb tduffy> * DAT_INSUFFICIENT_RESOURCES tduffy> * tduffy> */ tduffy> -u32 dapl_rbuf_realloc(DAPL_RING_BUFFER *rbuf, int size) tduffy> +u32 dapl_rbuf_realloc(struct dapl_ring_buffer *rbuf, int size) tduffy> { tduffy> - DAPL_RING_BUFFER new_rbuf; tduffy> - void *entry; tduffy> + struct dapl_ring_buffer new_rbuf; tduffy> + void *entry; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> /* decreasing the size or retaining the old size is not allowed */ tduffy> @@ -146,10 +146,10 @@ u32 dapl_rbuf_realloc(DAPL_RING_BUFFER * tduffy> /* tduffy> * dapl_rbuf_destroy tduffy> * tduffy> - * Release the buffer and reset pointers to a DAPL_RING_BUFFER tduffy> + * Release the buffer and reset pointers to a struct dapl_ring_buffer tduffy> * tduffy> * Input: tduffy> - * rbuf pointer to DAPL_RING_BUFFER tduffy> + * rbuf pointer to struct dapl_ring_buffer tduffy> * tduffy> * Output: tduffy> * none tduffy> @@ -158,7 +158,7 @@ u32 dapl_rbuf_realloc(DAPL_RING_BUFFER * tduffy> * none tduffy> * tduffy> */ tduffy> -void dapl_rbuf_destroy(DAPL_RING_BUFFER * rbuf) tduffy> +void dapl_rbuf_destroy(struct dapl_ring_buffer *rbuf) tduffy> { tduffy> if ((NULL == rbuf) || (NULL == rbuf->base)) { tduffy> return; tduffy> @@ -177,7 +177,7 @@ void dapl_rbuf_destroy(DAPL_RING_BUFFER tduffy> * Add an entry to the ring buffer tduffy> * tduffy> * Input: tduffy> - * rbuf pointer to DAPL_RING_BUFFER tduffy> + * rbuf pointer to struct dapl_ring_buffer tduffy> * entry entry to add tduffy> * tduffy> * Output: tduffy> @@ -188,7 +188,7 @@ void dapl_rbuf_destroy(DAPL_RING_BUFFER tduffy> * DAT_INSUFFICIENT_RESOURCES (queue full) tduffy> * tduffy> */ tduffy> -u32 dapl_rbuf_add(DAPL_RING_BUFFER *rbuf, void *entry) tduffy> +u32 dapl_rbuf_add(struct dapl_ring_buffer *rbuf, void *entry) tduffy> { tduffy> int pos; tduffy> int val; tduffy> @@ -214,7 +214,7 @@ u32 dapl_rbuf_add(DAPL_RING_BUFFER *rbuf tduffy> * Remove an entry from the ring buffer tduffy> * tduffy> * Input: tduffy> - * rbuf pointer to DAPL_RING_BUFFER tduffy> + * rbuf pointer to struct dapl_ring_buffer tduffy> * tduffy> * Output: tduffy> * entry entry removed from the ring buffer tduffy> @@ -224,7 +224,7 @@ u32 dapl_rbuf_add(DAPL_RING_BUFFER *rbuf tduffy> * DAT_QUEUE_EMPTY tduffy> tduffy> */ tduffy> -void *dapl_rbuf_remove(DAPL_RING_BUFFER * rbuf) tduffy> +void *dapl_rbuf_remove(struct dapl_ring_buffer *rbuf) tduffy> { tduffy> int pos; tduffy> int val; tduffy> @@ -250,7 +250,7 @@ void *dapl_rbuf_remove(DAPL_RING_BUFFER tduffy> * Return the number of entries in use in the ring buffer tduffy> * tduffy> * Input: tduffy> - * rbuf pointer to DAPL_RING_BUFFER tduffy> + * rbuf pointer to struct dapl_ring_buffer tduffy> * tduffy> * tduffy> * Output: tduffy> @@ -260,7 +260,7 @@ void *dapl_rbuf_remove(DAPL_RING_BUFFER tduffy> * count of entries tduffy> * tduffy> */ tduffy> -int dapl_rbuf_count(DAPL_RING_BUFFER * rbuf) tduffy> +int dapl_rbuf_count(struct dapl_ring_buffer *rbuf) tduffy> { tduffy> int count, head, tail; tduffy> tduffy> @@ -286,7 +286,7 @@ int dapl_rbuf_count(DAPL_RING_BUFFER * r tduffy> * functions). tduffy> * tduffy> * Input: tduffy> - * rbuf pointer to DAPL_RING_BUFFER tduffy> + * rbuf pointer to struct dapl_ring_buffer tduffy> * offset offset to adjust elemnt addresss by, tduffy> * used for addresses of type void * tduffy> * Output: tduffy> @@ -295,7 +295,7 @@ int dapl_rbuf_count(DAPL_RING_BUFFER * r tduffy> * Returns: tduffy> * none tduffy> */ tduffy> -void dapl_rbuf_adjust(DAPL_RING_BUFFER * rbuf, unsigned long offset) tduffy> +void dapl_rbuf_adjust(struct dapl_ring_buffer *rbuf, unsigned long offset) tduffy> { tduffy> int pos; tduffy> tduffy> Index: linux-kernel-resolv/dat-provider/dapl_psp_create.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_psp_create.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_psp_create.c (working copy) tduffy> @@ -71,13 +71,13 @@ u32 dapl_psp_create(DAT_IA_HANDLE ia_han tduffy> DAT_EVD_HANDLE evd_handle, enum dat_psp_flags psp_flags, tduffy> DAT_PSP_HANDLE *psp_handle) tduffy> { tduffy> - DAPL_IA *ia_ptr; tduffy> - DAPL_SP *sp_ptr; tduffy> - DAPL_EVD *evd_ptr; tduffy> + struct dapl_ia *ia_ptr; tduffy> + struct dapl_sp *sp_ptr; tduffy> + struct dapl_evd *evd_ptr; tduffy> boolean_t sp_found; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> - ia_ptr = (DAPL_IA *) ia_handle; tduffy> + ia_ptr = (struct dapl_ia *)ia_handle; tduffy> tduffy> if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { tduffy> dat_status = tduffy> @@ -95,7 +95,7 @@ u32 dapl_psp_create(DAT_IA_HANDLE ia_han tduffy> goto bail; tduffy> } tduffy> tduffy> - evd_ptr = (DAPL_EVD *) evd_handle; tduffy> + evd_ptr = (struct dapl_evd *)evd_handle; tduffy> if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { tduffy> dat_status = tduffy> DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); tduffy> @@ -141,7 +141,7 @@ u32 dapl_psp_create(DAT_IA_HANDLE ia_han tduffy> /* tduffy> * Take a reference on the EVD handle tduffy> */ tduffy> - atomic_inc(&((DAPL_EVD *) evd_handle)->evd_ref_count); tduffy> + atomic_inc(&((struct dapl_evd *)evd_handle)->evd_ref_count); tduffy> tduffy> /* tduffy> * Set up a listener for a connection. Connections can arrive tduffy> @@ -168,7 +168,7 @@ u32 dapl_psp_create(DAT_IA_HANDLE ia_han tduffy> * Have a problem setting up the connection, something tduffy> * wrong! Decrements the EVD refcount & release it. tduffy> */ tduffy> - atomic_dec(&((DAPL_EVD *) evd_handle)-> tduffy> + atomic_dec(&((struct dapl_evd *)evd_handle)-> tduffy> evd_ref_count); tduffy> sp_ptr->evd_handle = NULL; tduffy> dapl_ia_unlink_sp(ia_ptr, sp_ptr); tduffy> Index: linux-kernel-resolv/dat-provider/dapl_srq_resize.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_srq_resize.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_srq_resize.c (working copy) tduffy> @@ -65,8 +65,8 @@ tduffy> tduffy> u32 dapl_srq_resize(DAT_SRQ_HANDLE srq_handle, int srq_max_recv_dto) tduffy> { tduffy> - DAPL_IA *ia_ptr; tduffy> - DAPL_SRQ *srq_ptr; tduffy> + struct dapl_ia *ia_ptr; tduffy> + struct dapl_srq *srq_ptr; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_resize (%p, %d)\n", tduffy> @@ -78,7 +78,7 @@ u32 dapl_srq_resize(DAT_SRQ_HANDLE srq_h tduffy> goto bail; tduffy> } tduffy> tduffy> - srq_ptr = (DAPL_SRQ *) srq_handle; tduffy> + srq_ptr = (struct dapl_srq *)srq_handle; tduffy> ia_ptr = srq_ptr->header.owner_ia; tduffy> tduffy> /* tduffy> Index: linux-kernel-resolv/dat-provider/dapl_ep_create.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_ep_create.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_ep_create.c (working copy) tduffy> @@ -69,8 +69,8 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand tduffy> DAT_EVD_HANDLE connect_evd_handle, tduffy> const struct dat_ep_attr *ep_attr, DAT_EP_HANDLE *ep_handle) tduffy> { tduffy> - DAPL_IA *ia_ptr; tduffy> - DAPL_EP *ep_ptr; tduffy> + struct dapl_ia *ia_ptr; tduffy> + struct dapl_ep *ep_ptr; tduffy> struct dat_ep_attr *ep_attr_limit; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> @@ -82,7 +82,7 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand tduffy> request_evd_handle, tduffy> connect_evd_handle, ep_attr, ep_handle); tduffy> tduffy> - ia_ptr = (DAPL_IA *) ia_handle; tduffy> + ia_ptr = (struct dapl_ia *)ia_handle; tduffy> tduffy> /* tduffy> * Verify parameters tduffy> @@ -109,7 +109,7 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand tduffy> /* If connect handle is !NULL verify handle is good */ tduffy> if (connect_evd_handle != DAT_HANDLE_NULL && tduffy> (DAPL_BAD_HANDLE(connect_evd_handle, DAPL_MAGIC_EVD) || tduffy> - !(((DAPL_EVD *) connect_evd_handle)-> tduffy> + !(((struct dapl_evd *)connect_evd_handle)-> tduffy> evd_flags & DAT_EVD_CONNECTION_FLAG))) { tduffy> dat_status = tduffy> DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CONN); tduffy> @@ -119,7 +119,7 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand tduffy> /* If recv_evd is !NULL, verify handle is good and flags are valid */ tduffy> if (recv_evd_handle != DAT_HANDLE_NULL && tduffy> (DAPL_BAD_HANDLE(recv_evd_handle, DAPL_MAGIC_EVD) || tduffy> - !(((DAPL_EVD *) recv_evd_handle)->evd_flags & DAT_EVD_DTO_FLAG))) { tduffy> + !(((struct dapl_evd *)recv_evd_handle)->evd_flags & DAT_EVD_DTO_FLAG))) { tduffy> dat_status = tduffy> DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_RECV); tduffy> goto bail; tduffy> @@ -128,7 +128,7 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand tduffy> /* If req_evd is !NULL, verify handle is good and flags are valid */ tduffy> if (request_evd_handle != DAT_HANDLE_NULL && tduffy> (DAPL_BAD_HANDLE(request_evd_handle, DAPL_MAGIC_EVD) || tduffy> - !(((DAPL_EVD *) request_evd_handle)-> tduffy> + !(((struct dapl_evd *)request_evd_handle)-> tduffy> evd_flags & DAT_EVD_DTO_FLAG))) { tduffy> dat_status = tduffy> DAT_ERROR(DAT_INVALID_HANDLE, tduffy> @@ -210,13 +210,11 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand tduffy> /* tduffy> * Make sure we handle the NULL DTO EVDs tduffy> */ tduffy> - if (recv_evd_handle == DAT_HANDLE_NULL && ep_attr == NULL) { tduffy> + if (recv_evd_handle == DAT_HANDLE_NULL && ep_attr == NULL) tduffy> ep_ptr->param.ep_attr.max_recv_dtos = 0; tduffy> - } tduffy> tduffy> - if (request_evd_handle == DAT_HANDLE_NULL && ep_attr == NULL) { tduffy> + if (request_evd_handle == DAT_HANDLE_NULL && ep_attr == NULL) tduffy> ep_ptr->param.ep_attr.max_request_dtos = 0; tduffy> - } tduffy> tduffy> /* tduffy> * If the user has specified a PZ handle we allocate a QP for tduffy> @@ -226,7 +224,7 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand tduffy> */ tduffy> if (pz_handle != DAT_HANDLE_NULL) { tduffy> /* Take a reference on the PZ handle */ tduffy> - atomic_inc(&((DAPL_PZ *) pz_handle)->pz_ref_count); tduffy> + atomic_inc(&((struct dapl_pz *)pz_handle)->pz_ref_count); tduffy> tduffy> /* tduffy> * Get a QP from the IB provider tduffy> @@ -234,14 +232,12 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand tduffy> dat_status = dapl_ib_qp_alloc(ia_ptr, ep_ptr, ep_ptr); tduffy> tduffy> if (dat_status != DAT_SUCCESS) { tduffy> - atomic_dec(&((DAPL_PZ *) pz_handle)-> tduffy> - pz_ref_count); tduffy> + atomic_dec(&((struct dapl_pz *)pz_handle)->pz_ref_count); tduffy> dapl_ep_dealloc(ep_ptr); tduffy> goto bail; tduffy> } tduffy> - } else { tduffy> + } else tduffy> ep_ptr->qp_state = DAPL_QP_STATE_UNATTACHED; tduffy> - } tduffy> tduffy> /* tduffy> * Update ref counts. See the spec where the endpoint marks tduffy> @@ -251,22 +247,20 @@ u32 dapl_ep_create(DAT_IA_HANDLE ia_hand tduffy> * tduffy> * N.B. This should really be done by a util routine. tduffy> */ tduffy> - atomic_inc(&((DAPL_EVD *) connect_evd_handle)->evd_ref_count); tduffy> + atomic_inc(&((struct dapl_evd *)connect_evd_handle)->evd_ref_count); tduffy> /* Optional handles */ tduffy> - if (recv_evd_handle != DAT_HANDLE_NULL) { tduffy> - atomic_inc(&((DAPL_EVD *) recv_evd_handle)-> tduffy> + if (recv_evd_handle != DAT_HANDLE_NULL) tduffy> + atomic_inc(&((struct dapl_evd *)recv_evd_handle)-> tduffy> evd_ref_count); tduffy> - } tduffy> - if (request_evd_handle != DAT_HANDLE_NULL) { tduffy> - atomic_inc(&((DAPL_EVD *) request_evd_handle)-> tduffy> + if (request_evd_handle != DAT_HANDLE_NULL) tduffy> + atomic_inc(&((struct dapl_evd *)request_evd_handle)-> tduffy> evd_ref_count); tduffy> - } tduffy> tduffy> /* Link it onto the IA */ tduffy> dapl_ia_link_ep(ia_ptr, ep_ptr); tduffy> tduffy> *ep_handle = ep_ptr; tduffy> tduffy> - bail: tduffy> +bail: tduffy> return dat_status; tduffy> } tduffy> Index: linux-kernel-resolv/dat-provider/dapl_lmr_util.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_lmr_util.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_lmr_util.c (working copy) tduffy> @@ -38,14 +38,12 @@ tduffy> #include "dapl_lmr_util.h" tduffy> #include "dapl_ia_util.h" tduffy> tduffy> -DAPL_LMR *dapl_lmr_alloc(DAPL_IA * ia, tduffy> - enum dat_mem_type mem_type, tduffy> - DAT_REGION_DESCRIPTION region_desc, tduffy> - u64 length, tduffy> - DAT_PZ_HANDLE pz_handle, tduffy> - enum dat_mem_priv_flags mem_priv) tduffy> +struct dapl_lmr *dapl_lmr_alloc(struct dapl_ia *ia, enum dat_mem_type mem_type, tduffy> + DAT_REGION_DESCRIPTION region_desc, u64 length, tduffy> + DAT_PZ_HANDLE pz_handle, tduffy> + enum dat_mem_priv_flags mem_priv) tduffy> { tduffy> - DAPL_LMR *lmr; tduffy> + struct dapl_lmr *lmr; tduffy> tduffy> /* Allocate LMR */ tduffy> lmr = kmalloc(sizeof *lmr, GFP_ATOMIC); tduffy> @@ -82,9 +80,10 @@ DAPL_LMR *dapl_lmr_alloc(DAPL_IA * ia, tduffy> return lmr; tduffy> } tduffy> tduffy> -void dapl_lmr_dealloc(DAPL_LMR * lmr) tduffy> +void dapl_lmr_dealloc(struct dapl_lmr *lmr) tduffy> { tduffy> - lmr->header.magic = DAPL_MAGIC_INVALID; /* reset magic to prevent reuse */ tduffy> + /* reset magic to prevent reuse */ tduffy> + lmr->header.magic = DAPL_MAGIC_INVALID; tduffy> dapl_ia_unlink_lmr(lmr->header.owner_ia, lmr); tduffy> /* no need to destroy lmr->header.lock */ tduffy> tduffy> Index: linux-kernel-resolv/dat-provider/dapl_openib_dto.h tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_openib_dto.h (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_openib_dto.h (working copy) tduffy> @@ -49,7 +49,8 @@ tduffy> * tduffy> * Provider specific Post RECV function tduffy> */ tduffy> -static inline u32 dapl_ib_post_recv(DAPL_EP *ep_ptr, DAPL_COOKIE *cookie, tduffy> +static inline u32 dapl_ib_post_recv(struct dapl_ep *ep_ptr, tduffy> + struct dapl_cookie *cookie, tduffy> int num_segments, tduffy> struct dat_lmr_triplet *local_iov) tduffy> { tduffy> @@ -98,8 +99,9 @@ static inline u32 dapl_ib_post_recv(DAPL tduffy> * tduffy> * Provider specific Post SEND function tduffy> */ tduffy> -static inline u32 dapl_ib_post_send(DAPL_EP *ep_ptr, enum ib_wr_opcode op_type, tduffy> - DAPL_COOKIE *cookie, tduffy> +static inline u32 dapl_ib_post_send(struct dapl_ep *ep_ptr, tduffy> + enum ib_wr_opcode op_type, tduffy> + struct dapl_cookie *cookie, tduffy> int num_segments, tduffy> struct dat_lmr_triplet *local_iov, tduffy> const struct dat_rmr_triplet *remote_iov, tduffy> Index: linux-kernel-resolv/dat-provider/dapl_hca_util.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_hca_util.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_hca_util.c (working copy) tduffy> @@ -56,9 +56,9 @@ tduffy> * none tduffy> * tduffy> */ tduffy> -DAPL_HCA *dapl_hca_alloc(char *name, struct ib_device *device, u8 port) tduffy> +struct dapl_hca *dapl_hca_alloc(char *name, struct ib_device *device, u8 port) tduffy> { tduffy> - DAPL_HCA *hca_ptr; tduffy> + struct dapl_hca *hca_ptr; tduffy> tduffy> hca_ptr = kmalloc(sizeof *hca_ptr, GFP_ATOMIC); tduffy> if (hca_ptr) { tduffy> @@ -101,7 +101,7 @@ DAPL_HCA *dapl_hca_alloc(char *name, str tduffy> * none tduffy> * tduffy> */ tduffy> -void dapl_hca_free(DAPL_HCA * hca_ptr) tduffy> +void dapl_hca_free(struct dapl_hca *hca_ptr) tduffy> { tduffy> (void)dapl_hash_free(hca_ptr->lmr_hash_table); tduffy> kfree(hca_ptr->name); tduffy> @@ -124,7 +124,7 @@ void dapl_hca_free(DAPL_HCA * hca_ptr) tduffy> * none tduffy> * tduffy> */ tduffy> -void dapl_hca_link_ia(DAPL_HCA * hca_ptr, DAPL_IA * ia_ptr) tduffy> +void dapl_hca_link_ia(struct dapl_hca *hca_ptr, struct dapl_ia *ia_ptr) tduffy> { tduffy> spin_lock_irqsave(&hca_ptr->lock, hca_ptr->flags); tduffy> dapl_llist_add_head(&hca_ptr->ia_list_head, tduffy> @@ -148,7 +148,7 @@ void dapl_hca_link_ia(DAPL_HCA * hca_ptr tduffy> * none tduffy> * tduffy> */ tduffy> -void dapl_hca_unlink_ia(DAPL_HCA * hca_ptr, DAPL_IA * ia_ptr) tduffy> +void dapl_hca_unlink_ia(struct dapl_hca *hca_ptr, struct dapl_ia *ia_ptr) tduffy> { tduffy> spin_lock_irqsave(&hca_ptr->lock, hca_ptr->flags); tduffy> /* tduffy> Index: linux-kernel-resolv/dat-provider/dapl_srq_create.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_srq_create.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_srq_create.c (working copy) tduffy> @@ -68,15 +68,15 @@ tduffy> u32 dapl_srq_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, tduffy> struct dat_srq_attr *srq_attr, DAT_SRQ_HANDLE *srq_handle) tduffy> { tduffy> - DAPL_IA *ia_ptr; tduffy> - DAPL_SRQ *srq_ptr; tduffy> + struct dapl_ia *ia_ptr; tduffy> + struct dapl_srq *srq_ptr; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> "dapl_srq_create (%p, %p, %p, %p)\n", tduffy> ia_handle, pz_handle, srq_attr, srq_handle); tduffy> tduffy> - ia_ptr = (DAPL_IA *) ia_handle; tduffy> + ia_ptr = (struct dapl_ia *)ia_handle; tduffy> tduffy> /* tduffy> * Verify parameters tduffy> Index: linux-kernel-resolv/dat-provider/dapl_evd_connection_callb.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_evd_connection_callb.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_evd_connection_callb.c (working copy) tduffy> @@ -65,8 +65,8 @@ void dapl_evd_connection_callback(struct tduffy> const void *private_data_ptr, tduffy> const void *context) tduffy> { tduffy> - DAPL_EP *ep_ptr; tduffy> - DAPL_EVD *evd_ptr; tduffy> + struct dapl_ep *ep_ptr; tduffy> + struct dapl_evd *evd_ptr; tduffy> int private_data_size; tduffy> u32 dat_status; tduffy> tduffy> @@ -78,17 +78,15 @@ void dapl_evd_connection_callback(struct tduffy> * Determine the type of handle passed back to us in the context tduffy> * and sort out key parameters. tduffy> */ tduffy> - if (context == NULL tduffy> - || ((DAPL_HEADER *) context)->magic != DAPL_MAGIC_EP) { tduffy> + if (context == NULL || ((struct dapl_header *) context)->magic != DAPL_MAGIC_EP) tduffy> return; tduffy> - } tduffy> tduffy> /* tduffy> * Active side of the connection, context is an EP and tduffy> * PSP is irrelevant. tduffy> */ tduffy> - ep_ptr = (DAPL_EP *) context; tduffy> - evd_ptr = (DAPL_EVD *) ep_ptr->param.connect_evd_handle; tduffy> + ep_ptr = (struct dapl_ep *)context; tduffy> + evd_ptr = (struct dapl_evd *)ep_ptr->param.connect_evd_handle; tduffy> tduffy> private_data_size = 0; tduffy> /* tduffy> Index: linux-kernel-resolv/dat-provider/dapl_srq_query.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_srq_query.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_srq_query.c (working copy) tduffy> @@ -41,7 +41,7 @@ tduffy> tduffy> u32 dapl_srq_query(DAT_SRQ_HANDLE srq_handle, struct dat_srq_param *srq_param) tduffy> { tduffy> - DAPL_SRQ *srq_ptr; tduffy> + struct dapl_srq *srq_ptr; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> @@ -58,7 +58,7 @@ u32 dapl_srq_query(DAT_SRQ_HANDLE srq_ha tduffy> goto bail; tduffy> } tduffy> tduffy> - srq_ptr = (DAPL_SRQ *) srq_handle; tduffy> + srq_ptr = (struct dapl_srq *)srq_handle; tduffy> tduffy> /* tduffy> * XXX Need to calculate available_dto_count and outstanding_dto_count tduffy> Index: linux-kernel-resolv/dat-provider/dapl_ring_buffer_util.h tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_ring_buffer_util.h (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_ring_buffer_util.h (working copy) tduffy> @@ -42,19 +42,19 @@ tduffy> /* tduffy> * Prototypes tduffy> */ tduffy> -u32 dapl_rbuf_alloc(DAPL_RING_BUFFER *rbuf, int size); tduffy> +u32 dapl_rbuf_alloc(struct dapl_ring_buffer *rbuf, int size); tduffy> tduffy> -u32 dapl_rbuf_realloc(DAPL_RING_BUFFER *rbuf, int size); tduffy> +u32 dapl_rbuf_realloc(struct dapl_ring_buffer *rbuf, int size); tduffy> tduffy> -void dapl_rbuf_destroy(DAPL_RING_BUFFER * rbuf); tduffy> +void dapl_rbuf_destroy(struct dapl_ring_buffer *rbuf); tduffy> tduffy> -u32 dapl_rbuf_add(DAPL_RING_BUFFER *rbuf, void *entry); tduffy> +u32 dapl_rbuf_add(struct dapl_ring_buffer *rbuf, void *entry); tduffy> tduffy> -void *dapl_rbuf_remove(DAPL_RING_BUFFER * rbuf); tduffy> +void *dapl_rbuf_remove(struct dapl_ring_buffer *rbuf); tduffy> tduffy> -int dapl_rbuf_count(DAPL_RING_BUFFER *rbuf); tduffy> +int dapl_rbuf_count(struct dapl_ring_buffer *rbuf); tduffy> tduffy> -void dapl_rbuf_adjust(DAPL_RING_BUFFER * rbuf, unsigned long offset); tduffy> +void dapl_rbuf_adjust(struct dapl_ring_buffer *rbuf, unsigned long offset); tduffy> tduffy> /* tduffy> * Simple functions tduffy> Index: linux-kernel-resolv/dat-provider/dapl_psp_create_any.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_psp_create_any.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_psp_create_any.c (working copy) tduffy> @@ -74,15 +74,15 @@ u32 dapl_psp_create_any(DAT_IA_HANDLE ia tduffy> DAT_EVD_HANDLE evd_handle, enum dat_psp_flags psp_flags, tduffy> DAT_PSP_HANDLE *psp_handle) tduffy> { tduffy> - DAPL_IA *ia_ptr; tduffy> - DAPL_SP *sp_ptr; tduffy> - DAPL_EVD *evd_ptr; tduffy> + struct dapl_ia *ia_ptr; tduffy> + struct dapl_sp *sp_ptr; tduffy> + struct dapl_evd *evd_ptr; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> static DAT_CONN_QUAL hint_conn_qual = 1000; /* seed value */ tduffy> DAT_CONN_QUAL lcl_conn_qual; tduffy> DAT_CONN_QUAL limit_conn_qual; tduffy> tduffy> - ia_ptr = (DAPL_IA *) ia_handle; tduffy> + ia_ptr = (struct dapl_ia *)ia_handle; tduffy> tduffy> if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { tduffy> dat_status = tduffy> @@ -104,7 +104,7 @@ u32 dapl_psp_create_any(DAT_IA_HANDLE ia tduffy> goto bail; tduffy> } tduffy> tduffy> - evd_ptr = (DAPL_EVD *) evd_handle; tduffy> + evd_ptr = (struct dapl_evd *)evd_handle; tduffy> if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { tduffy> dat_status = tduffy> DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); tduffy> @@ -136,7 +136,7 @@ u32 dapl_psp_create_any(DAT_IA_HANDLE ia tduffy> /* tduffy> * Take a reference on the EVD handle tduffy> */ tduffy> - atomic_inc(&((DAPL_EVD *) evd_handle)->evd_ref_count); tduffy> + atomic_inc(&((struct dapl_evd *)evd_handle)->evd_ref_count); tduffy> tduffy> /* Link it onto the IA */ tduffy> dapl_ia_link_psp(ia_ptr, sp_ptr); tduffy> @@ -176,7 +176,7 @@ u32 dapl_psp_create_any(DAT_IA_HANDLE ia tduffy> /* tduffy> * Have a problem setting up the connection, something wrong! tduffy> */ tduffy> - atomic_dec(&((DAPL_EVD *) evd_handle)->evd_ref_count); tduffy> + atomic_dec(&((struct dapl_evd *)evd_handle)->evd_ref_count); tduffy> sp_ptr->evd_handle = NULL; tduffy> dapl_ia_unlink_sp(ia_ptr, sp_ptr); tduffy> dapl_sp_dealloc(sp_ptr); tduffy> Index: linux-kernel-resolv/dat-provider/dapl_lmr_util.h tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_lmr_util.h (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_lmr_util.h (working copy) tduffy> @@ -39,19 +39,12 @@ tduffy> tduffy> #include "dapl_mr_util.h" tduffy> tduffy> -/********************************************************************* tduffy> - * * tduffy> - * Function Prototypes * tduffy> - * * tduffy> - *********************************************************************/ tduffy> +extern struct dapl_lmr *dapl_lmr_alloc(struct dapl_ia *ia, tduffy> + enum dat_mem_type mem_type, tduffy> + DAT_REGION_DESCRIPTION region_desc, tduffy> + u64 length, DAT_PZ_HANDLE pz_handle, tduffy> + enum dat_mem_priv_flags mem_priv); tduffy> tduffy> -extern DAPL_LMR *dapl_lmr_alloc(DAPL_IA * ia, tduffy> - enum dat_mem_type mem_type, tduffy> - DAT_REGION_DESCRIPTION region_desc, tduffy> - u64 length, tduffy> - DAT_PZ_HANDLE pz_handle, tduffy> - enum dat_mem_priv_flags mem_priv); tduffy> +extern void dapl_lmr_dealloc(struct dapl_lmr *lmr); tduffy> tduffy> -extern void dapl_lmr_dealloc(DAPL_LMR * lmr); tduffy> - tduffy> -#endif /* DAPL_LMR_UTIL_H */ tduffy> +#endif /* DAPL_LMR_UTIL_H */ tduffy> Index: linux-kernel-resolv/dat-provider/dapl_hca_util.h tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_hca_util.h (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_hca_util.h (working copy) tduffy> @@ -39,12 +39,12 @@ tduffy> tduffy> #include "dapl.h" tduffy> tduffy> -DAPL_HCA *dapl_hca_alloc(char *, struct ib_device *, u8 port); tduffy> +struct dapl_hca *dapl_hca_alloc(char *name, struct ib_device *device, u8 port); tduffy> tduffy> -void dapl_hca_free(DAPL_HCA * hca_ptr); tduffy> +void dapl_hca_free(struct dapl_hca *hca_ptr); tduffy> tduffy> -void dapl_hca_link_ia(DAPL_HCA * hca_ptr, DAPL_IA * ia_info); tduffy> +void dapl_hca_link_ia(struct dapl_hca *hca_ptr, struct dapl_ia *ia_info); tduffy> tduffy> -void dapl_hca_unlink_ia(DAPL_HCA * hca_ptr, DAPL_IA * ia_info); tduffy> +void dapl_hca_unlink_ia(struct dapl_hca *hca_ptr, struct dapl_ia *ia_info); tduffy> tduffy> #endif tduffy> Index: linux-kernel-resolv/dat-provider/dapl_get_consumer_context.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_get_consumer_context.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_get_consumer_context.c (working copy) tduffy> @@ -58,9 +58,9 @@ tduffy> u32 dapl_get_consumer_context(DAT_HANDLE dat_handle, DAT_CONTEXT *context) tduffy> { tduffy> u32 dat_status = DAT_SUCCESS; tduffy> - DAPL_HEADER *header; tduffy> + struct dapl_header *header; tduffy> tduffy> - header = (DAPL_HEADER *) dat_handle; tduffy> + header = (struct dapl_header *)dat_handle; tduffy> if (((header) == NULL) || tduffy> ((unsigned long)(header) & 3) || tduffy> (header->magic != DAPL_MAGIC_IA && tduffy> Index: linux-kernel-resolv/dat-provider/dapl_evd_resize.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_evd_resize.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_evd_resize.c (working copy) tduffy> @@ -65,8 +65,8 @@ tduffy> tduffy> u32 dapl_evd_resize(DAT_EVD_HANDLE evd_handle, int evd_qlen) tduffy> { tduffy> - DAPL_IA *ia_ptr; tduffy> - DAPL_EVD *evd_ptr; tduffy> + struct dapl_ia *ia_ptr; tduffy> + struct dapl_evd *evd_ptr; tduffy> int pend_cnt; tduffy> u32 status; tduffy> tduffy> @@ -78,7 +78,7 @@ u32 dapl_evd_resize(DAT_EVD_HANDLE evd_h tduffy> goto bail; tduffy> } tduffy> tduffy> - evd_ptr = (DAPL_EVD *) evd_handle; tduffy> + evd_ptr = (struct dapl_evd *)evd_handle; tduffy> ia_ptr = evd_ptr->header.owner_ia; tduffy> tduffy> if (evd_qlen == evd_ptr->qlen) { tduffy> Index: linux-kernel-resolv/dat-provider/dapl_cr_util.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_cr_util.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_cr_util.c (working copy) tduffy> @@ -37,9 +37,9 @@ tduffy> /* tduffy> * Create a CR. Part of the passive side of a connection tduffy> */ tduffy> -DAPL_CR *dapl_cr_alloc(DAPL_IA * ia_ptr) tduffy> +struct dapl_cr *dapl_cr_alloc(struct dapl_ia *ia_ptr) tduffy> { tduffy> - DAPL_CR *cr_ptr; tduffy> + struct dapl_cr *cr_ptr; tduffy> tduffy> /* Allocate EP */ tduffy> cr_ptr = kmalloc(sizeof *cr_ptr, GFP_ATOMIC); tduffy> @@ -67,11 +67,11 @@ DAPL_CR *dapl_cr_alloc(DAPL_IA * ia_ptr) tduffy> /* tduffy> * Free the passed in EP structure. tduffy> */ tduffy> -void dapl_cr_free(DAPL_CR * cr_ptr) tduffy> +void dapl_cr_free(struct dapl_cr *cr_ptr) tduffy> { tduffy> dapl_os_assert(cr_ptr->header.magic == DAPL_MAGIC_CR || tduffy> cr_ptr->header.magic == DAPL_MAGIC_CR_DESTROYED); tduffy> - tduffy> - cr_ptr->header.magic = DAPL_MAGIC_INVALID; /* reset magic to prevent reuse */ tduffy> + /* reset magic to prevent reuse */ tduffy> + cr_ptr->header.magic = DAPL_MAGIC_INVALID; tduffy> kfree(cr_ptr); tduffy> } tduffy> Index: linux-kernel-resolv/dat-provider/dapl_rmr_free.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_rmr_free.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_rmr_free.c (working copy) tduffy> @@ -55,7 +55,7 @@ tduffy> */ tduffy> u32 dapl_rmr_free(DAT_RMR_HANDLE rmr_handle) tduffy> { tduffy> - DAPL_RMR *rmr; tduffy> + struct dapl_rmr *rmr; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) { tduffy> @@ -64,7 +64,7 @@ u32 dapl_rmr_free(DAT_RMR_HANDLE rmr_han tduffy> goto bail; tduffy> } tduffy> tduffy> - rmr = (DAPL_RMR *) rmr_handle; tduffy> + rmr = (struct dapl_rmr *)rmr_handle; tduffy> tduffy> /* tduffy> * If the user did not perform an unbind op, release tduffy> @@ -77,14 +77,13 @@ u32 dapl_rmr_free(DAT_RMR_HANDLE rmr_han tduffy> tduffy> dat_status = dapl_ib_mw_free(rmr); tduffy> tduffy> - if (dat_status != DAT_SUCCESS) { tduffy> + if (dat_status != DAT_SUCCESS) tduffy> goto bail; tduffy> - } tduffy> tduffy> atomic_dec(&rmr->pz->pz_ref_count); tduffy> tduffy> dapl_rmr_dealloc(rmr); tduffy> tduffy> - bail: tduffy> +bail: tduffy> return dat_status; tduffy> } tduffy> Index: linux-kernel-resolv/dat-provider/dapl_srq_util.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_srq_util.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_srq_util.c (working copy) tduffy> @@ -54,9 +54,10 @@ tduffy> * pointer to srq tduffy> * tduffy> */ tduffy> -DAPL_SRQ *dapl_srq_alloc(DAPL_IA * ia_ptr, const struct dat_srq_attr *srq_attr) tduffy> +struct dapl_srq *dapl_srq_alloc(struct dapl_ia *ia_ptr, tduffy> + const struct dat_srq_attr *srq_attr) tduffy> { tduffy> - DAPL_SRQ *srq_ptr; tduffy> + struct dapl_srq *srq_ptr; tduffy> tduffy> /* Allocate SRQ */ tduffy> srq_ptr = kmalloc(sizeof *srq_ptr, GFP_ATOMIC); tduffy> @@ -89,14 +90,15 @@ DAPL_SRQ *dapl_srq_alloc(DAPL_IA * ia_pt tduffy> srq_ptr->param.low_watermark = srq_attr->low_watermark; tduffy> tduffy> /* Get a cookie buffer to track outstanding recvs */ tduffy> - if (DAT_SUCCESS != dapl_cb_create(&srq_ptr->recv_buffer, (DAPL_EP *) srq_ptr, /* just saves the value */ tduffy> + if (DAT_SUCCESS != dapl_cb_create(&srq_ptr->recv_buffer, tduffy> + (struct dapl_ep *)srq_ptr, tduffy> srq_ptr->param.max_recv_dtos)) { tduffy> dapl_srq_dealloc(srq_ptr); tduffy> srq_ptr = NULL; tduffy> goto bail; tduffy> } tduffy> tduffy> - bail: tduffy> +bail: tduffy> return srq_ptr; tduffy> } tduffy> tduffy> @@ -115,11 +117,12 @@ DAPL_SRQ *dapl_srq_alloc(DAPL_IA * ia_pt tduffy> * none tduffy> * tduffy> */ tduffy> -void dapl_srq_dealloc(DAPL_SRQ * srq_ptr) tduffy> +void dapl_srq_dealloc(struct dapl_srq *srq_ptr) tduffy> { tduffy> dapl_os_assert(srq_ptr->header.magic == DAPL_MAGIC_SRQ); tduffy> tduffy> - srq_ptr->header.magic = DAPL_MAGIC_INVALID; /* reset magic to prevent reuse */ tduffy> + /* reset magic to prevent reuse */ tduffy> + srq_ptr->header.magic = DAPL_MAGIC_INVALID; tduffy> dapl_ia_unlink_srq(srq_ptr->header.owner_ia, srq_ptr); tduffy> dapl_cb_free(&srq_ptr->recv_buffer); tduffy> /* no need to destroy srq_ptr->header.lock */ tduffy> Index: linux-kernel-resolv/dat-provider/dapl_psp_free.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_psp_free.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_psp_free.c (working copy) tduffy> @@ -60,12 +60,12 @@ tduffy> */ tduffy> u32 dapl_psp_free(DAT_PSP_HANDLE psp_handle) tduffy> { tduffy> - DAPL_IA *ia_ptr; tduffy> - DAPL_SP *sp_ptr; tduffy> + struct dapl_ia *ia_ptr; tduffy> + struct dapl_sp *sp_ptr; tduffy> DAPL_SP_STATE save_state; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> - sp_ptr = (DAPL_SP *) psp_handle; tduffy> + sp_ptr = (struct dapl_sp *)psp_handle; tduffy> /* tduffy> * Verify handle tduffy> */ tduffy> @@ -77,7 +77,7 @@ u32 dapl_psp_free(DAT_PSP_HANDLE psp_han tduffy> goto bail; tduffy> } tduffy> tduffy> - /* ia_ptr = (DAPL_IA *)sp_ptr->header.owner_ia; */ tduffy> + /* ia_ptr = (struct dapl_ia *)sp_ptr->header.owner_ia; */ tduffy> ia_ptr = sp_ptr->header.owner_ia; tduffy> /* tduffy> * Remove the connection listener if it has been established tduffy> @@ -94,7 +94,7 @@ u32 dapl_psp_free(DAT_PSP_HANDLE psp_han tduffy> * free the evd_handle will be NULL tduffy> */ tduffy> if (sp_ptr->evd_handle) { tduffy> - atomic_dec(&((DAPL_EVD *) sp_ptr->evd_handle)-> tduffy> + atomic_dec(&((struct dapl_evd *)sp_ptr->evd_handle)-> tduffy> evd_ref_count); tduffy> sp_ptr->evd_handle = NULL; tduffy> } tduffy> Index: linux-kernel-resolv/dat-provider/dapl_ep_free.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_ep_free.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_ep_free.c (working copy) tduffy> @@ -63,15 +63,15 @@ tduffy> */ tduffy> u32 dapl_ep_free(DAT_EP_HANDLE ep_handle) tduffy> { tduffy> - DAPL_EP *ep_ptr; tduffy> - DAPL_IA *ia_ptr; tduffy> + struct dapl_ep *ep_ptr; tduffy> + struct dapl_ia *ia_ptr; tduffy> struct dat_ep_param *param; tduffy> enum ib_qp_state save_qp_state; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_free (%p)\n", ep_handle); tduffy> tduffy> - ep_ptr = (DAPL_EP *) ep_handle; tduffy> + ep_ptr = (struct dapl_ep *)ep_handle; tduffy> param = &ep_ptr->param; tduffy> tduffy> /* tduffy> @@ -129,22 +129,21 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle tduffy> * is idempotent. tduffy> */ tduffy> if (param->pz_handle != NULL) { tduffy> - atomic_dec(&((DAPL_PZ *) param->pz_handle)-> tduffy> - pz_ref_count); tduffy> + atomic_dec(&((struct dapl_pz *)param->pz_handle)->pz_ref_count); tduffy> param->pz_handle = NULL; tduffy> } tduffy> if (param->recv_evd_handle != NULL) { tduffy> - atomic_dec(&((DAPL_EVD *) param->recv_evd_handle)-> tduffy> + atomic_dec(&((struct dapl_evd *)param->recv_evd_handle)-> tduffy> evd_ref_count); tduffy> param->recv_evd_handle = NULL; tduffy> } tduffy> if (param->request_evd_handle != NULL) { tduffy> - atomic_dec(&((DAPL_EVD *) param->request_evd_handle)-> tduffy> + atomic_dec(&((struct dapl_evd *)param->request_evd_handle)-> tduffy> evd_ref_count); tduffy> param->request_evd_handle = NULL; tduffy> } tduffy> if (param->connect_evd_handle != NULL) { tduffy> - atomic_dec(&((DAPL_EVD *) param->connect_evd_handle)-> tduffy> + atomic_dec(&((struct dapl_evd *)param->connect_evd_handle)-> tduffy> evd_ref_count); tduffy> param->connect_evd_handle = NULL; tduffy> } tduffy> @@ -181,7 +180,7 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle tduffy> /* Free the resource */ tduffy> dapl_ep_dealloc(ep_ptr); tduffy> tduffy> - bail: tduffy> +bail: tduffy> return dat_status; tduffy> tduffy> } tduffy> Index: linux-kernel-resolv/dat-provider/dapl_ep_get_status.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_ep_get_status.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_ep_get_status.c (working copy) tduffy> @@ -62,14 +62,14 @@ tduffy> u32 dapl_ep_get_status(DAT_EP_HANDLE ep_handle, enum dat_ep_state *ep_state, tduffy> boolean_t *in_dto_idle, boolean_t *out_dto_idle) tduffy> { tduffy> - DAPL_EP *ep_ptr; tduffy> + struct dapl_ep *ep_ptr; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> "dapl_ep_get_status (%p, %p, %p, %p)\n", tduffy> ep_handle, ep_state, in_dto_idle, out_dto_idle); tduffy> tduffy> - ep_ptr = (DAPL_EP *) ep_handle; tduffy> + ep_ptr = (struct dapl_ep *)ep_handle; tduffy> tduffy> /* tduffy> * Verify parameter & state tduffy> Index: linux-kernel-resolv/dat-provider/dapl_cr_util.h tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_cr_util.h (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_cr_util.h (working copy) tduffy> @@ -36,12 +36,12 @@ tduffy> tduffy> #include "dapl.h" tduffy> tduffy> -DAPL_CR *dapl_cr_alloc(DAPL_IA * ia_ptr); tduffy> +struct dapl_cr *dapl_cr_alloc(struct dapl_ia *ia_ptr); tduffy> tduffy> -void dapl_cr_free(DAPL_CR * cr_ptr); tduffy> +void dapl_cr_free(struct dapl_cr *cr_ptr); tduffy> tduffy> void dapl_cr_callback(struct dapl_cm_id *ib_cm_handle, tduffy> const enum dat_event_number event, tduffy> const void *instant_data_p, const void *context); tduffy> tduffy> -#endif /* DAPL_CR_UTIL_H */ tduffy> +#endif /* DAPL_CR_UTIL_H */ tduffy> Index: linux-kernel-resolv/dat-provider/dapl_srq_util.h tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_srq_util.h (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_srq_util.h (working copy) tduffy> @@ -39,11 +39,9 @@ tduffy> tduffy> #include "dapl.h" tduffy> tduffy> -/* function prototypes */ tduffy> +extern struct dapl_srq *dapl_srq_alloc(struct dapl_ia *ia, tduffy> + const struct dat_srq_attr *srq_attr); tduffy> tduffy> -extern DAPL_SRQ *dapl_srq_alloc(DAPL_IA * ia, tduffy> - const struct dat_srq_attr *srq_attr); tduffy> +extern void dapl_srq_dealloc(struct dapl_srq *srq_ptr); tduffy> tduffy> -extern void dapl_srq_dealloc(DAPL_SRQ * srq_ptr); tduffy> - tduffy> -#endif /* DAPL_SRQ_UTIL_H */ tduffy> +#endif /* DAPL_SRQ_UTIL_H */ tduffy> Index: linux-kernel-resolv/dat-provider/dapl_evd_post_se.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_evd_post_se.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_evd_post_se.c (working copy) tduffy> @@ -61,10 +61,10 @@ tduffy> tduffy> u32 dapl_evd_post_se(DAT_EVD_HANDLE evd_handle, const struct dat_event *event) tduffy> { tduffy> - DAPL_EVD *evd_ptr; tduffy> + struct dapl_evd *evd_ptr; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> - evd_ptr = (DAPL_EVD *) evd_handle; tduffy> + evd_ptr = (struct dapl_evd *)evd_handle; tduffy> tduffy> if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { tduffy> dat_status = DAT_ERROR(DAT_INVALID_HANDLE, 0); tduffy> Index: linux-kernel-resolv/dat-provider/dapl_set_consumer_context.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_set_consumer_context.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_set_consumer_context.c (working copy) tduffy> @@ -59,9 +59,9 @@ tduffy> u32 dapl_set_consumer_context(DAT_HANDLE dat_handle, DAT_CONTEXT context) tduffy> { tduffy> u32 dat_status = DAT_SUCCESS; tduffy> - DAPL_HEADER *header; tduffy> + struct dapl_header *header; tduffy> tduffy> - header = (DAPL_HEADER *) dat_handle; tduffy> + header = (struct dapl_header *)dat_handle; tduffy> if (((header) == NULL) || tduffy> ((unsigned long)(header) & 3) || tduffy> (header->magic != DAPL_MAGIC_IA && tduffy> Index: linux-kernel-resolv/dat-provider/dapl_mr_util.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_mr_util.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_mr_util.c (working copy) tduffy> @@ -58,13 +58,13 @@ tduffy> tduffy> u64 dapl_mr_get_address(DAT_REGION_DESCRIPTION desc, enum dat_mem_type type) tduffy> { tduffy> - DAPL_LMR *lmr; tduffy> + struct dapl_lmr *lmr; tduffy> tduffy> switch (type) { tduffy> case DAT_MEM_TYPE_VIRTUAL: tduffy> return (u64) (unsigned long) desc.for_va; tduffy> case DAT_MEM_TYPE_LMR: tduffy> - lmr = (DAPL_LMR *) desc.for_lmr_handle; tduffy> + lmr = (struct dapl_lmr *)desc.for_lmr_handle; tduffy> tduffy> /* Since this function is recoursive we cannot inline it */ tduffy> return dapl_mr_get_address(lmr->param.region_desc, tduffy> Index: linux-kernel-resolv/dat-provider/dapl_pz_create.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_pz_create.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_pz_create.c (working copy) tduffy> @@ -57,8 +57,8 @@ tduffy> */ tduffy> u32 dapl_pz_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE *pz_handle) tduffy> { tduffy> - DAPL_IA *ia; tduffy> - DAPL_PZ *pz; tduffy> + struct dapl_ia *ia; tduffy> + struct dapl_pz *pz; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> @@ -75,7 +75,7 @@ u32 dapl_pz_create(DAT_IA_HANDLE ia_hand tduffy> goto bail; tduffy> } tduffy> tduffy> - ia = (DAPL_IA *) ia_handle; tduffy> + ia = (struct dapl_ia *)ia_handle; tduffy> tduffy> pz = dapl_pz_alloc(ia); tduffy> if (NULL == pz) { tduffy> Index: linux-kernel-resolv/dat-provider/dapl_ia_query.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_ia_query.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_ia_query.c (working copy) tduffy> @@ -62,7 +62,7 @@ u32 dapl_ia_query(DAT_IA_HANDLE ia_handl tduffy> struct dat_ia_attr *ia_attr, tduffy> struct dat_provider_attr *provider_attr) tduffy> { tduffy> - DAPL_IA *ia_ptr; tduffy> + struct dapl_ia *ia_ptr; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> struct evd_merge_type { tduffy> boolean_t array[6][6]; tduffy> @@ -78,7 +78,7 @@ u32 dapl_ia_query(DAT_IA_HANDLE ia_handl tduffy> ia_attr, tduffy> provider_attr); tduffy> tduffy> - ia_ptr = (DAPL_IA *) ia_handle; tduffy> + ia_ptr = (struct dapl_ia *)ia_handle; tduffy> tduffy> if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { tduffy> dat_status = tduffy> Index: linux-kernel-resolv/dat-provider/dapl_pz_query.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_pz_query.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_pz_query.c (working copy) tduffy> @@ -38,7 +38,7 @@ tduffy> tduffy> u32 dapl_pz_query(DAT_PZ_HANDLE pz_handle, struct dat_pz_param *pz_param) tduffy> { tduffy> - DAPL_PZ *pz; tduffy> + struct dapl_pz *pz; tduffy> u32 status; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_query (%p, %x, %p)\n", tduffy> @@ -53,7 +53,7 @@ u32 dapl_pz_query(DAT_PZ_HANDLE pz_handl tduffy> goto bail; tduffy> } tduffy> tduffy> - pz = (DAPL_PZ *) pz_handle; tduffy> + pz = (struct dapl_pz *)pz_handle; tduffy> pz_param->ia_handle = (DAT_IA_HANDLE) pz->header.owner_ia; tduffy> status = DAT_SUCCESS; tduffy> tduffy> Index: linux-kernel-resolv/dat-provider/dapl_evd_free.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_evd_free.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_evd_free.c (working copy) tduffy> @@ -41,13 +41,13 @@ tduffy> */ tduffy> u32 dapl_evd_free(DAT_EVD_HANDLE evd_handle) tduffy> { tduffy> - DAPL_EVD *evd_ptr; tduffy> - DAPL_CNO *cno_ptr; tduffy> + struct dapl_evd *evd_ptr; tduffy> + struct dapl_cno *cno_ptr; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_evd_free (%p)\n", evd_handle); tduffy> tduffy> - evd_ptr = (DAPL_EVD *) evd_handle; tduffy> + evd_ptr = (struct dapl_evd *)evd_handle; tduffy> tduffy> if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { tduffy> dat_status = DAT_ERROR(DAT_INVALID_HANDLE, 0); tduffy> @@ -72,10 +72,10 @@ u32 dapl_evd_free(DAT_EVD_HANDLE evd_han tduffy> dapl_ia_link_evd(evd_ptr->header.owner_ia, evd_ptr); tduffy> } tduffy> tduffy> - if (cno_ptr != NULL) tduffy> - dapl_cno_dealloc (cno_ptr); tduffy> + if (cno_ptr) tduffy> + dapl_cno_dealloc(cno_ptr); tduffy> tduffy> - bail: tduffy> +bail: tduffy> dapl_dbg_log(DAPL_DBG_TYPE_RTN, tduffy> "dapl_evd_free () returns 0x%x\n", dat_status); tduffy> tduffy> Index: linux-kernel-resolv/dat-provider/dapl_lmr_query.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_lmr_query.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_lmr_query.c (working copy) tduffy> @@ -38,7 +38,7 @@ tduffy> tduffy> u32 dapl_lmr_query(DAT_LMR_HANDLE lmr_handle, struct dat_lmr_param *lmr_param) tduffy> { tduffy> - DAPL_LMR *lmr; tduffy> + struct dapl_lmr *lmr; tduffy> u32 dat_status; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> @@ -56,10 +56,10 @@ u32 dapl_lmr_query(DAT_LMR_HANDLE lmr_ha tduffy> } tduffy> tduffy> dat_status = DAT_SUCCESS; tduffy> - lmr = (DAPL_LMR *) lmr_handle; tduffy> + lmr = (struct dapl_lmr *)lmr_handle; tduffy> tduffy> memcpy(lmr_param, &lmr->param, sizeof *lmr_param); tduffy> tduffy> - bail: tduffy> +bail: tduffy> return dat_status; tduffy> } tduffy> Index: linux-kernel-resolv/dat-provider/dapl_rmr_query.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_rmr_query.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_rmr_query.c (working copy) tduffy> @@ -40,7 +40,7 @@ tduffy> tduffy> u32 dapl_rmr_query(DAT_RMR_HANDLE rmr_handle, struct dat_rmr_param *rmr_param) tduffy> { tduffy> - DAPL_RMR *rmr; tduffy> + struct dapl_rmr *rmr; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) { tduffy> @@ -53,10 +53,10 @@ u32 dapl_rmr_query(DAT_RMR_HANDLE rmr_ha tduffy> goto bail; tduffy> } tduffy> tduffy> - rmr = (DAPL_RMR *) rmr_handle; tduffy> + rmr = (struct dapl_rmr *)rmr_handle; tduffy> tduffy> memcpy(rmr_param, &rmr->param, sizeof *rmr_param); tduffy> tduffy> - bail: tduffy> +bail: tduffy> return dat_status; tduffy> } tduffy> Index: linux-kernel-resolv/dat-provider/dapl_evd_modify_upcall.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_evd_modify_upcall.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_evd_modify_upcall.c (working copy) tduffy> @@ -47,14 +47,14 @@ u32 dapl_evd_modify_upcall(DAT_EVD_HANDL tduffy> enum dat_upcall_policy upcall_policy, /* ignore */ tduffy> const struct dat_upcall_object *upcall) tduffy> { tduffy> - DAPL_EVD *evd_ptr; tduffy> - DAPL_CNO *cno_ptr; tduffy> + struct dapl_evd *evd_ptr; tduffy> + struct dapl_cno *cno_ptr; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_modify_upcall (%p)\n", tduffy> evd_handle); tduffy> tduffy> - evd_ptr = (DAPL_EVD *) evd_handle; tduffy> + evd_ptr = (struct dapl_evd *)evd_handle; tduffy> cno_ptr = evd_ptr->cno_ptr; tduffy> tduffy> if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { tduffy> @@ -62,12 +62,11 @@ u32 dapl_evd_modify_upcall(DAT_EVD_HANDL tduffy> goto bail; tduffy> } tduffy> tduffy> - if (cno_ptr != NULL) tduffy> - { tduffy> + if (cno_ptr != NULL) { tduffy> cno_ptr->cno_upcall_policy = upcall_policy; tduffy> cno_ptr->cno_upcall = *upcall; tduffy> } tduffy> tduffy> - bail: tduffy> +bail: tduffy> return dat_status; tduffy> } tduffy> Index: linux-kernel-resolv/dat-provider/dapl_ia_util.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_ia_util.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_ia_util.c (working copy) tduffy> @@ -58,9 +58,10 @@ tduffy> * none tduffy> * tduffy> */ tduffy> -DAPL_IA *dapl_ia_alloc(struct dat_provider *provider, DAPL_HCA * hca_ptr) tduffy> +struct dapl_ia *dapl_ia_alloc(struct dat_provider *provider, tduffy> + struct dapl_hca *hca_ptr) tduffy> { tduffy> - DAPL_IA *ia_ptr; tduffy> + struct dapl_ia *ia_ptr; tduffy> tduffy> /* Allocate IA */ tduffy> ia_ptr = kmalloc(sizeof *ia_ptr, GFP_ATOMIC); tduffy> @@ -106,17 +107,17 @@ DAPL_IA *dapl_ia_alloc(struct dat_provid tduffy> * Performs an abrupt close of the IA tduffy> */ tduffy> tduffy> -u32 dapl_ia_abrupt_close(DAPL_IA *ia_ptr) tduffy> +u32 dapl_ia_abrupt_close(struct dapl_ia *ia_ptr) tduffy> { tduffy> u32 dat_status = DAT_SUCCESS; tduffy> - DAPL_EP *ep_ptr, *next_ep_ptr; tduffy> - DAPL_LMR *lmr_ptr, *next_lmr_ptr; tduffy> - DAPL_RMR *rmr_ptr, *next_rmr_ptr; tduffy> - DAPL_PZ *pz_ptr, *next_pz_ptr; tduffy> - DAPL_EVD *evd_ptr, *next_evd_ptr; tduffy> - DAPL_SP *sp_ptr, *next_sp_ptr; /* for PSP and RSP queues */ tduffy> - DAPL_CR *cr_ptr, *next_cr_ptr; tduffy> - DAPL_HCA *hca_ptr; tduffy> + struct dapl_ep *ep_ptr, *next_ep_ptr; tduffy> + struct dapl_lmr *lmr_ptr, *next_lmr_ptr; tduffy> + struct dapl_rmr *rmr_ptr, *next_rmr_ptr; tduffy> + struct dapl_pz *pz_ptr, *next_pz_ptr; tduffy> + struct dapl_evd *evd_ptr, *next_evd_ptr; tduffy> + struct dapl_sp *sp_ptr, *next_sp_ptr; /* for PSP and RSP queues */ tduffy> + struct dapl_cr *cr_ptr, *next_cr_ptr; tduffy> + struct dapl_hca *hca_ptr; tduffy> tduffy> /* tduffy> * clear all the data structures associated with the IA. tduffy> @@ -135,11 +136,10 @@ u32 dapl_ia_abrupt_close(DAPL_IA *ia_ptr tduffy> &rmr_ptr->header. tduffy> ia_list_entry); tduffy> dat_status = dapl_rmr_free(rmr_ptr); tduffy> - if (dat_status != DAT_SUCCESS) { tduffy> + if (dat_status != DAT_SUCCESS) tduffy> dapl_dbg_log(DAPL_DBG_TYPE_WARN, tduffy> "ia_close(ABRUPT): rmr_free(%p) returns %x\n", tduffy> rmr_ptr, dat_status); tduffy> - } tduffy> rmr_ptr = next_rmr_ptr; tduffy> } tduffy> tduffy> @@ -150,11 +150,10 @@ u32 dapl_ia_abrupt_close(DAPL_IA *ia_ptr tduffy> &sp_ptr->header. tduffy> ia_list_entry); tduffy> dat_status = dapl_rsp_free(sp_ptr); tduffy> - if (dat_status != DAT_SUCCESS) { tduffy> + if (dat_status != DAT_SUCCESS) tduffy> dapl_dbg_log(DAPL_DBG_TYPE_WARN, tduffy> "ia_close(ABRUPT): rsp_free(%p) returns %x\n", tduffy> sp_ptr, dat_status); tduffy> - } tduffy> sp_ptr = next_sp_ptr; tduffy> } tduffy> tduffy> @@ -176,21 +175,19 @@ u32 dapl_ia_abrupt_close(DAPL_IA *ia_ptr tduffy> DAT_EP_STATE_DISCONNECT_PENDING) { tduffy> dat_status = tduffy> dapl_ep_disconnect(ep_ptr, DAT_CLOSE_ABRUPT_FLAG); tduffy> - if (dat_status != DAT_SUCCESS) { tduffy> + if (dat_status != DAT_SUCCESS) tduffy> dapl_dbg_log(DAPL_DBG_TYPE_WARN, tduffy> "ia_close(ABRUPT): ep_disconnect(%p) returns %x\n", tduffy> ep_ptr, dat_status); tduffy> - } tduffy> } tduffy> /* force the EP into error state to force flush all posted DTOs. */ tduffy> (void) dapl_modify_qp_state_to_error(ep_ptr->qp_handle); tduffy> tduffy> dat_status = dapl_ep_free(ep_ptr); tduffy> - if (dat_status != DAT_SUCCESS) { tduffy> + if (dat_status != DAT_SUCCESS) tduffy> dapl_dbg_log(DAPL_DBG_TYPE_WARN, tduffy> "ia_close(ABRUPT): ep_free(%p) returns %x\n", tduffy> ep_ptr, dat_status); tduffy> - } tduffy> ep_ptr = next_ep_ptr; tduffy> } tduffy> tduffy> @@ -201,11 +198,10 @@ u32 dapl_ia_abrupt_close(DAPL_IA *ia_ptr tduffy> &lmr_ptr->header. tduffy> ia_list_entry); tduffy> dat_status = dapl_lmr_free(lmr_ptr); tduffy> - if (dat_status != DAT_SUCCESS) { tduffy> + if (dat_status != DAT_SUCCESS) tduffy> dapl_dbg_log(DAPL_DBG_TYPE_WARN, tduffy> "ia_close(ABRUPT): lmr_free(%p) returns %x\n", tduffy> lmr_ptr, dat_status); tduffy> - } tduffy> lmr_ptr = next_lmr_ptr; tduffy> } tduffy> tduffy> @@ -217,11 +213,10 @@ u32 dapl_ia_abrupt_close(DAPL_IA *ia_ptr tduffy> * should be no competing threads after this. tduffy> */ tduffy> dat_status = dapl_ib_remove_conn_listener(ia_ptr, sp_ptr); tduffy> - if (dat_status != DAT_SUCCESS) { tduffy> + if (dat_status != DAT_SUCCESS) tduffy> dapl_dbg_log(DAPL_DBG_TYPE_WARN, tduffy> "ia_close(ABRUPT): psp cannot remove listener, returns %x\n", tduffy> dat_status); tduffy> - } tduffy> tduffy> next_sp_ptr = dapl_llist_next_entry(&ia_ptr->psp_list_head, tduffy> &sp_ptr->header. tduffy> @@ -245,11 +240,10 @@ u32 dapl_ia_abrupt_close(DAPL_IA *ia_ptr tduffy> } tduffy> tduffy> dat_status = dapl_psp_free(sp_ptr); tduffy> - if (dat_status != DAT_SUCCESS) { tduffy> + if (dat_status != DAT_SUCCESS) tduffy> dapl_dbg_log(DAPL_DBG_TYPE_WARN, tduffy> "ia_close(ABRUPT): psp_free(%p) returns %x\n", tduffy> sp_ptr, dat_status); tduffy> - } tduffy> tduffy> sp_ptr = next_sp_ptr; tduffy> } tduffy> @@ -261,11 +255,10 @@ u32 dapl_ia_abrupt_close(DAPL_IA *ia_ptr tduffy> &pz_ptr->header. tduffy> ia_list_entry); tduffy> dat_status = dapl_pz_free(pz_ptr); tduffy> - if (dat_status != DAT_SUCCESS) { tduffy> + if (dat_status != DAT_SUCCESS) tduffy> dapl_dbg_log(DAPL_DBG_TYPE_WARN, tduffy> "ia_close(ABRUPT): pz_free(%p) returns %x\n", tduffy> pz_ptr, dat_status); tduffy> - } tduffy> pz_ptr = next_pz_ptr; tduffy> } tduffy> tduffy> @@ -284,11 +277,10 @@ u32 dapl_ia_abrupt_close(DAPL_IA *ia_ptr tduffy> if (evd_ptr != ia_ptr->async_error_evd) { tduffy> /* it isn't the async EVD; delete it. */ tduffy> dat_status = dapl_evd_free(evd_ptr); tduffy> - if (dat_status != DAT_SUCCESS) { tduffy> + if (dat_status != DAT_SUCCESS) tduffy> dapl_dbg_log(DAPL_DBG_TYPE_WARN, tduffy> "ia_close(ABRUPT): evd_free(%p) returns %x\n", tduffy> evd_ptr, dat_status); tduffy> - } tduffy> } tduffy> evd_ptr = next_evd_ptr; tduffy> } tduffy> @@ -305,11 +297,10 @@ u32 dapl_ia_abrupt_close(DAPL_IA *ia_ptr tduffy> atomic_dec(&ia_ptr->async_error_evd->evd_ref_count); tduffy> dat_status = dapl_evd_free(ia_ptr->async_error_evd); tduffy> tduffy> - if (DAT_SUCCESS != dat_status) { tduffy> + if (DAT_SUCCESS != dat_status) tduffy> dapl_dbg_log(DAPL_DBG_TYPE_WARN, tduffy> "ia_close(ABRUPT): evd_free(%p) returns %x\n", tduffy> ia_ptr->async_error_evd, dat_status); tduffy> - } tduffy> tduffy> ia_ptr->async_error_evd = NULL; tduffy> } tduffy> @@ -337,13 +328,13 @@ u32 dapl_ia_abrupt_close(DAPL_IA *ia_ptr tduffy> * tduffy> */ tduffy> tduffy> -u32 dapl_ia_graceful_close(DAPL_IA *ia_ptr) tduffy> +u32 dapl_ia_graceful_close(struct dapl_ia *ia_ptr) tduffy> { tduffy> u32 dat_status = DAT_SUCCESS; tduffy> u32 cur_dat_status; tduffy> - DAPL_EVD *evd_ptr; tduffy> - DAPL_LLIST_ENTRY *entry; tduffy> - DAPL_HCA *hca_ptr; tduffy> + struct dapl_evd *evd_ptr; tduffy> + struct dapl_llist_entry *entry; tduffy> + struct dapl_hca *hca_ptr; tduffy> tduffy> if (!dapl_llist_is_empty(&ia_ptr->rmr_list_head) || tduffy> !dapl_llist_is_empty(&ia_ptr->rsp_list_head) || tduffy> @@ -416,14 +407,12 @@ u32 dapl_ia_graceful_close(DAPL_IA *ia_p tduffy> if (ia_ptr->async_error_evd && tduffy> (TRUE == ia_ptr->cleanup_async_error_evd)) { tduffy> cur_dat_status = dapl_ia_teardown_callbacks(ia_ptr); tduffy> - if (DAT_SUCCESS != cur_dat_status) { tduffy> + if (DAT_SUCCESS != cur_dat_status) tduffy> dat_status = cur_dat_status; tduffy> - } tduffy> atomic_dec(&ia_ptr->async_error_evd->evd_ref_count); tduffy> cur_dat_status = dapl_evd_free(ia_ptr->async_error_evd); tduffy> - if (DAT_SUCCESS != cur_dat_status) { tduffy> + if (DAT_SUCCESS != cur_dat_status) tduffy> dat_status = cur_dat_status; tduffy> - } tduffy> tduffy> ia_ptr->async_error_evd = NULL; tduffy> } tduffy> @@ -432,14 +421,14 @@ u32 dapl_ia_graceful_close(DAPL_IA *ia_p tduffy> tduffy> dapl_ia_free(ia_ptr); tduffy> tduffy> - bail: tduffy> +bail: tduffy> return dat_status; tduffy> } tduffy> tduffy> /* tduffy> * free an IA INFO struct tduffy> */ tduffy> -void dapl_ia_free(DAPL_IA * ia_ptr) tduffy> +void dapl_ia_free(struct dapl_ia *ia_ptr) tduffy> { tduffy> dapl_os_assert(ia_ptr->header.magic == DAPL_MAGIC_IA); tduffy> tduffy> @@ -464,7 +453,7 @@ void dapl_ia_free(DAPL_IA * ia_ptr) tduffy> /* tduffy> * Add an ep to the IA structure tduffy> */ tduffy> -void dapl_ia_link_ep(DAPL_IA * ia_ptr, DAPL_EP * ep_ptr) tduffy> +void dapl_ia_link_ep(struct dapl_ia *ia_ptr, struct dapl_ep *ep_ptr) tduffy> { tduffy> spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); tduffy> dapl_llist_add_head(&ia_ptr->ep_list_head, tduffy> @@ -475,7 +464,7 @@ void dapl_ia_link_ep(DAPL_IA * ia_ptr, D tduffy> /* tduffy> * Remove an ep from the ia info structure tduffy> */ tduffy> -void dapl_ia_unlink_ep(DAPL_IA * ia_ptr, DAPL_EP * ep_ptr) tduffy> +void dapl_ia_unlink_ep(struct dapl_ia *ia_ptr, struct dapl_ep *ep_ptr) tduffy> { tduffy> spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); tduffy> dapl_llist_remove_entry(&ia_ptr->ep_list_head, tduffy> @@ -486,7 +475,7 @@ void dapl_ia_unlink_ep(DAPL_IA * ia_ptr, tduffy> /* tduffy> * Add an srq to the IA structure tduffy> */ tduffy> -void dapl_ia_link_srq(DAPL_IA * ia_ptr, DAPL_SRQ * srq_ptr) tduffy> +void dapl_ia_link_srq(struct dapl_ia *ia_ptr, struct dapl_srq *srq_ptr) tduffy> { tduffy> spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); tduffy> dapl_llist_add_head(&ia_ptr->srq_list_head, tduffy> @@ -497,7 +486,7 @@ void dapl_ia_link_srq(DAPL_IA * ia_ptr, tduffy> /* tduffy> * Remove an srq from the ia info structure tduffy> */ tduffy> -void dapl_ia_unlink_srq(DAPL_IA * ia_ptr, DAPL_SRQ * srq_ptr) tduffy> +void dapl_ia_unlink_srq(struct dapl_ia *ia_ptr, struct dapl_srq *srq_ptr) tduffy> { tduffy> spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); tduffy> dapl_llist_remove_entry(&ia_ptr->srq_list_head, tduffy> @@ -508,7 +497,7 @@ void dapl_ia_unlink_srq(DAPL_IA * ia_ptr tduffy> /* tduffy> * Add an lmr to the IA structure tduffy> */ tduffy> -void dapl_ia_link_lmr(DAPL_IA * ia_ptr, DAPL_LMR * lmr_ptr) tduffy> +void dapl_ia_link_lmr(struct dapl_ia *ia_ptr, struct dapl_lmr *lmr_ptr) tduffy> { tduffy> spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); tduffy> dapl_llist_add_head(&ia_ptr->lmr_list_head, tduffy> @@ -519,7 +508,7 @@ void dapl_ia_link_lmr(DAPL_IA * ia_ptr, tduffy> /* tduffy> * Remove an lmr from the ia info structure tduffy> */ tduffy> -void dapl_ia_unlink_lmr(DAPL_IA * ia_ptr, DAPL_LMR * lmr_ptr) tduffy> +void dapl_ia_unlink_lmr(struct dapl_ia *ia_ptr, struct dapl_lmr *lmr_ptr) tduffy> { tduffy> spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); tduffy> dapl_llist_remove_entry(&ia_ptr->lmr_list_head, tduffy> @@ -530,7 +519,7 @@ void dapl_ia_unlink_lmr(DAPL_IA * ia_ptr tduffy> /* tduffy> * Add an rmr to the IA structure tduffy> */ tduffy> -void dapl_ia_link_rmr(DAPL_IA * ia_ptr, DAPL_RMR * rmr_ptr) tduffy> +void dapl_ia_link_rmr(struct dapl_ia *ia_ptr, struct dapl_rmr *rmr_ptr) tduffy> { tduffy> spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); tduffy> dapl_llist_add_head(&ia_ptr->rmr_list_head, tduffy> @@ -541,7 +530,7 @@ void dapl_ia_link_rmr(DAPL_IA * ia_ptr, tduffy> /* tduffy> * Remove an rmr from the ia info structure tduffy> */ tduffy> -void dapl_ia_unlink_rmr(DAPL_IA * ia_ptr, DAPL_RMR * rmr_ptr) tduffy> +void dapl_ia_unlink_rmr(struct dapl_ia *ia_ptr, struct dapl_rmr *rmr_ptr) tduffy> { tduffy> spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); tduffy> dapl_llist_remove_entry(&ia_ptr->rmr_list_head, tduffy> @@ -552,7 +541,7 @@ void dapl_ia_unlink_rmr(DAPL_IA * ia_ptr tduffy> /* tduffy> * Add an pz to the IA structure tduffy> */ tduffy> -void dapl_ia_link_pz(DAPL_IA * ia_ptr, DAPL_PZ * pz_ptr) tduffy> +void dapl_ia_link_pz(struct dapl_ia *ia_ptr, struct dapl_pz *pz_ptr) tduffy> { tduffy> spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); tduffy> dapl_llist_add_head(&ia_ptr->pz_list_head, tduffy> @@ -563,7 +552,7 @@ void dapl_ia_link_pz(DAPL_IA * ia_ptr, D tduffy> /* tduffy> * Remove an pz from the ia info structure tduffy> */ tduffy> -void dapl_ia_unlink_pz(DAPL_IA * ia_ptr, DAPL_PZ * pz_ptr) tduffy> +void dapl_ia_unlink_pz(struct dapl_ia *ia_ptr, struct dapl_pz *pz_ptr) tduffy> { tduffy> spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); tduffy> dapl_llist_remove_entry(&ia_ptr->pz_list_head, tduffy> @@ -574,7 +563,7 @@ void dapl_ia_unlink_pz(DAPL_IA * ia_ptr, tduffy> /* tduffy> * Add an evd to the IA structure tduffy> */ tduffy> -void dapl_ia_link_evd(DAPL_IA * ia_ptr, DAPL_EVD * evd_ptr) tduffy> +void dapl_ia_link_evd(struct dapl_ia *ia_ptr, struct dapl_evd *evd_ptr) tduffy> { tduffy> spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); tduffy> dapl_llist_add_head(&ia_ptr->evd_list_head, tduffy> @@ -585,7 +574,7 @@ void dapl_ia_link_evd(DAPL_IA * ia_ptr, tduffy> /* tduffy> * Remove an evd from the ia info structure tduffy> */ tduffy> -void dapl_ia_unlink_evd(DAPL_IA * ia_ptr, DAPL_EVD * evd_ptr) tduffy> +void dapl_ia_unlink_evd(struct dapl_ia *ia_ptr, struct dapl_evd *evd_ptr) tduffy> { tduffy> spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); tduffy> dapl_llist_remove_entry(&ia_ptr->evd_list_head, tduffy> @@ -596,7 +585,7 @@ void dapl_ia_unlink_evd(DAPL_IA * ia_ptr tduffy> /* tduffy> * Add an psp to the IA structure tduffy> */ tduffy> -void dapl_ia_link_psp(DAPL_IA * ia_ptr, DAPL_SP * sp_ptr) tduffy> +void dapl_ia_link_psp(struct dapl_ia *ia_ptr, struct dapl_sp *sp_ptr) tduffy> { tduffy> spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); tduffy> dapl_llist_add_head(&ia_ptr->psp_list_head, tduffy> @@ -607,9 +596,9 @@ void dapl_ia_link_psp(DAPL_IA * ia_ptr, tduffy> /* tduffy> * Remove an sp from the appropriate ia rsp or psp queue tduffy> */ tduffy> -void dapl_ia_unlink_sp(DAPL_IA * ia_ptr, DAPL_SP * sp_ptr) tduffy> +void dapl_ia_unlink_sp(struct dapl_ia *ia_ptr, struct dapl_sp *sp_ptr) tduffy> { tduffy> - DAPL_LLIST_HEAD *list_head; tduffy> + struct dapl_llist_entry **list_head; tduffy> tduffy> if (sp_ptr->header.handle_type == DAT_HANDLE_TYPE_PSP) { tduffy> list_head = &ia_ptr->psp_list_head; tduffy> @@ -640,17 +629,16 @@ void dapl_ia_unlink_sp(DAPL_IA * ia_ptr, tduffy> * none tduffy> * tduffy> */ tduffy> -DAPL_SP *dapl_ia_sp_search(DAPL_IA * ia_ptr, tduffy> - DAT_CONN_QUAL conn_qual, boolean_t is_psp) tduffy> +struct dapl_sp *dapl_ia_sp_search(struct dapl_ia *ia_ptr, tduffy> + DAT_CONN_QUAL conn_qual, boolean_t is_psp) tduffy> { tduffy> - DAPL_SP *sp_ptr; tduffy> - DAPL_LLIST_HEAD *list_head; tduffy> + struct dapl_sp *sp_ptr; tduffy> + struct dapl_llist_entry **list_head; tduffy> tduffy> - if (is_psp) { tduffy> + if (is_psp) tduffy> list_head = &ia_ptr->psp_list_head; tduffy> - } else { tduffy> + else tduffy> list_head = &ia_ptr->rsp_list_head; tduffy> - } tduffy> tduffy> spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); tduffy> tduffy> @@ -686,7 +674,7 @@ DAPL_SP *dapl_ia_sp_search(DAPL_IA * ia_ tduffy> * none tduffy> * tduffy> */ tduffy> -void dapl_ia_link_rsp(DAPL_IA * ia_ptr, DAPL_SP * sp_ptr) tduffy> +void dapl_ia_link_rsp(struct dapl_ia *ia_ptr, struct dapl_sp *sp_ptr) tduffy> { tduffy> spin_lock_irqsave(&ia_ptr->header.lock, ia_ptr->header.flags); tduffy> dapl_llist_add_head(&ia_ptr->rsp_list_head, tduffy> @@ -694,7 +682,8 @@ void dapl_ia_link_rsp(DAPL_IA * ia_ptr, tduffy> spin_unlock_irqrestore(&ia_ptr->header.lock, ia_ptr->header.flags); tduffy> } tduffy> tduffy> -u32 dapl_ia_setup_callbacks(DAPL_IA * ia_ptr, DAPL_EVD * async_evd_ptr) tduffy> +u32 dapl_ia_setup_callbacks(struct dapl_ia *ia_ptr, tduffy> + struct dapl_evd *async_evd_ptr) tduffy> { tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> @@ -717,7 +706,7 @@ u32 dapl_ia_setup_callbacks(DAPL_IA * ia tduffy> return dat_status; tduffy> } tduffy> tduffy> -u32 dapl_ia_teardown_callbacks(DAPL_IA * ia_ptr) tduffy> +u32 dapl_ia_teardown_callbacks(struct dapl_ia *ia_ptr) tduffy> { tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> Index: linux-kernel-resolv/dat-provider/dapl_pz_util.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_pz_util.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_pz_util.c (working copy) tduffy> @@ -52,13 +52,13 @@ tduffy> * none tduffy> * tduffy> */ tduffy> -DAPL_PZ *dapl_pz_alloc(DAPL_IA * ia) tduffy> +struct dapl_pz *dapl_pz_alloc(struct dapl_ia *ia) tduffy> { tduffy> - DAPL_PZ *pz; tduffy> + struct dapl_pz *pz; tduffy> tduffy> /* Allocate PZ */ tduffy> pz = kmalloc(sizeof *pz, GFP_ATOMIC); tduffy> - if (!pz) tduffy> + if (!pz) tduffy> return NULL; tduffy> tduffy> /* zero the structure */ tduffy> @@ -100,9 +100,10 @@ DAPL_PZ *dapl_pz_alloc(DAPL_IA * ia) tduffy> * none tduffy> * tduffy> */ tduffy> -void dapl_pz_dealloc(DAPL_PZ * pz) tduffy> +void dapl_pz_dealloc(struct dapl_pz *pz) tduffy> { tduffy> - pz->header.magic = DAPL_MAGIC_INVALID; /* reset magic to prevent reuse */ tduffy> + /* reset magic to prevent reuse */ tduffy> + pz->header.magic = DAPL_MAGIC_INVALID; tduffy> dapl_ia_unlink_pz(pz->header.owner_ia, pz); tduffy> /* no need to destroy pz->header.lock */ tduffy> tduffy> Index: linux-kernel-resolv/dat-provider/dapl_ep_query.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_ep_query.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_ep_query.c (working copy) tduffy> @@ -59,14 +59,14 @@ tduffy> */ tduffy> u32 dapl_ep_query(DAT_EP_HANDLE ep_handle, struct dat_ep_param *ep_param) tduffy> { tduffy> - DAPL_EP *ep_ptr; tduffy> + struct dapl_ep *ep_ptr; tduffy> u32 status = DAT_SUCCESS; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> "dapl_ep_query (%p, %p)\n", tduffy> ep_handle, ep_param); tduffy> tduffy> - ep_ptr = (DAPL_EP *) ep_handle; tduffy> + ep_ptr = (struct dapl_ep *)ep_handle; tduffy> tduffy> /* tduffy> * Verify parameter & state tduffy> Index: linux-kernel-resolv/dat-provider/dapl_pz_util.h tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_pz_util.h (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_pz_util.h (working copy) tduffy> @@ -39,8 +39,8 @@ tduffy> tduffy> #include "dapl.h" tduffy> tduffy> -extern DAPL_PZ *dapl_pz_alloc(DAPL_IA * ia); tduffy> +extern struct dapl_pz *dapl_pz_alloc(struct dapl_ia *ia); tduffy> tduffy> -extern void dapl_pz_dealloc(DAPL_PZ * pz); tduffy> +extern void dapl_pz_dealloc(struct dapl_pz *pz); tduffy> tduffy> -#endif /* DAPL_PZ_UTIL_H */ tduffy> +#endif /* DAPL_PZ_UTIL_H */ tduffy> Index: linux-kernel-resolv/dat-provider/dapl_evd_dequeue.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_evd_dequeue.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_evd_dequeue.c (working copy) tduffy> @@ -64,14 +64,14 @@ tduffy> tduffy> u32 dapl_evd_dequeue(DAT_EVD_HANDLE evd_handle, struct dat_event *event) tduffy> { tduffy> - DAPL_EVD *evd_ptr; tduffy> + struct dapl_evd *evd_ptr; tduffy> struct dat_event *local_event; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> "dapl_evd_dequeue (%p, %p)\n", evd_handle, event); tduffy> tduffy> - evd_ptr = (DAPL_EVD *) evd_handle; tduffy> + evd_ptr = (struct dapl_evd *)evd_handle; tduffy> tduffy> if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { tduffy> dat_status = DAT_ERROR(DAT_INVALID_HANDLE, 0); tduffy> Index: linux-kernel-resolv/dat-provider/dapl_openib_cm.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_openib_cm.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_openib_cm.c (working copy) tduffy> @@ -399,12 +399,12 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han tduffy> DAT_CONN_QUAL remote_conn_qual, tduffy> int private_data_size, void *private_data) tduffy> { tduffy> - DAPL_IA *ia_ptr; tduffy> - DAPL_EP *ep_ptr; tduffy> + struct dapl_ia *ia_ptr; tduffy> + struct dapl_ep *ep_ptr; tduffy> struct dapl_cm_id *conn; tduffy> int status; tduffy> tduffy> - ep_ptr = (DAPL_EP *)ep_handle; tduffy> + ep_ptr = (struct dapl_ep *)ep_handle; tduffy> if (ep_ptr->qp_handle == NULL) { tduffy> printk(KERN_ERR "dapl_ib_connect: qp handle invalid\n"); tduffy> return DAT_INTERNAL_ERROR; tduffy> @@ -486,7 +486,7 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han tduffy> * DAT_INVALID_PARAMETER tduffy> * tduffy> */ tduffy> -u32 dapl_ib_disconnect(DAPL_EP *ep_ptr, enum dat_close_flags close_flags) tduffy> +u32 dapl_ib_disconnect(struct dapl_ep *ep_ptr, enum dat_close_flags close_flags) tduffy> { tduffy> struct dapl_cm_id *conn = ep_ptr->cm_handle; tduffy> int status; tduffy> @@ -510,24 +510,24 @@ u32 dapl_ib_disconnect(DAPL_EP *ep_ptr, tduffy> return DAT_SUCCESS; tduffy> } tduffy> tduffy> -/* tduffy> - * dapl_ib_disconnect_clean tduffy> - * tduffy> - * Clean up outstanding connection data. This routine is invoked tduffy> +/* tduffy> + * dapl_ib_disconnect_clean tduffy> + * tduffy> + * Clean up outstanding connection data. This routine is invoked tduffy> * after the final disconnect callback has occurred. tduffy> - * tduffy> - * Input: tduffy> - * ep_ptr DAPL_EP tduffy> - * active Indicates active side of connection tduffy> - * tduffy> - * Output: tduffy> - * none tduffy> - * tduffy> - * Returns: tduffy> - * void tduffy> - * tduffy> + * tduffy> + * Input: tduffy> + * ep_ptr struct dapl_ep tduffy> + * active Indicates active side of connection tduffy> + * tduffy> + * Output: tduffy> + * none tduffy> + * tduffy> + * Returns: tduffy> + * void tduffy> + * tduffy> */ tduffy> -void dapl_ib_disconnect_clean(DAPL_EP * ep_ptr, boolean_t active) tduffy> +void dapl_ib_disconnect_clean(struct dapl_ep *ep_ptr, boolean_t active) tduffy> { tduffy> int status; tduffy> tduffy> @@ -566,9 +566,9 @@ void dapl_ib_disconnect_clean(DAPL_EP * tduffy> * void tduffy> * tduffy> */ tduffy> -void dapl_ib_reinit_ep(DAPL_EP * ep_ptr) tduffy> +void dapl_ib_reinit_ep(struct dapl_ep *ep_ptr) tduffy> { tduffy> - DAPL_IA *ia_ptr; tduffy> + struct dapl_ia *ia_ptr; tduffy> int ib_status; tduffy> tduffy> ia_ptr = ep_ptr->header.owner_ia; tduffy> @@ -615,7 +615,8 @@ void dapl_ib_reinit_ep(DAPL_EP * ep_ptr) tduffy> * DAT_INVALID_PARAMETER tduffy> * tduffy> */ tduffy> -u32 dapl_ib_setup_conn_listener(DAPL_IA *ia_ptr, u64 ServiceID, DAPL_SP *sp_ptr) tduffy> +u32 dapl_ib_setup_conn_listener(struct dapl_ia *ia_ptr, u64 ServiceID, tduffy> + struct dapl_sp *sp_ptr) tduffy> { tduffy> int status; tduffy> tduffy> @@ -658,7 +659,7 @@ u32 dapl_ib_setup_conn_listener(DAPL_IA tduffy> * DAT_INVALID_PARAMETER tduffy> * tduffy> */ tduffy> -u32 dapl_ib_remove_conn_listener(DAPL_IA *ia_ptr, DAPL_SP *sp_ptr) tduffy> +u32 dapl_ib_remove_conn_listener(struct dapl_ia *ia_ptr, struct dapl_sp *sp_ptr) tduffy> { tduffy> //*** This will hang if called from CM thread context... tduffy> //*** Move back to using WQ... tduffy> @@ -731,16 +732,16 @@ u32 dapl_ib_accept_connection(DAT_CR_HAN tduffy> int private_data_size, tduffy> const void *prd_ptr) tduffy> { tduffy> - DAPL_CR *cr_ptr; tduffy> - DAPL_EP *ep_ptr; tduffy> - DAPL_IA *ia_ptr; tduffy> + struct dapl_cr *cr_ptr; tduffy> + struct dapl_ep *ep_ptr; tduffy> + struct dapl_ia *ia_ptr; tduffy> u32 dat_status; tduffy> int status; tduffy> struct ib_cm_rep_param passive_params; tduffy> struct dapl_cm_id *conn; tduffy> tduffy> - cr_ptr = (DAPL_CR *) cr_handle; tduffy> - ep_ptr = (DAPL_EP *) ep_handle; tduffy> + cr_ptr = (struct dapl_cr *)cr_handle; tduffy> + ep_ptr = (struct dapl_ep *)ep_handle; tduffy> ia_ptr = ep_ptr->header.owner_ia; tduffy> conn = cr_ptr->ib_cm_handle; tduffy> tduffy> @@ -833,13 +834,13 @@ static int ib_cm_get_remote_gid(struct d tduffy> u32 dapl_ib_cm_remote_addr(DAT_HANDLE dat_handle, tduffy> struct sockaddr_in6 *remote_ia_address) tduffy> { tduffy> - DAPL_IA *ia_ptr; tduffy> + struct dapl_ia *ia_ptr; tduffy> int rc; tduffy> union ib_gid remote_gid; tduffy> tduffy> - DAPL_CR *cr_ptr = (DAPL_CR *) dat_handle; tduffy> + struct dapl_cr *cr_ptr = (struct dapl_cr *)dat_handle; tduffy> tduffy> - ia_ptr = (DAPL_IA *) cr_ptr->sp_ptr->ia_handle; tduffy> + ia_ptr = (struct dapl_ia *)cr_ptr->sp_ptr->ia_handle; tduffy> tduffy> rc = ib_cm_get_remote_gid(cr_ptr->ib_cm_handle, (u8 *) & remote_gid); tduffy> if (rc < 0) tduffy> Index: linux-kernel-resolv/dat-provider/dapl_ia_util.h tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_ia_util.h (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_ia_util.h (working copy) tduffy> @@ -39,49 +39,51 @@ tduffy> tduffy> #include "dapl.h" tduffy> tduffy> -DAPL_IA *dapl_ia_alloc(struct dat_provider *provider, DAPL_HCA * hca_ptr); tduffy> +struct dapl_ia *dapl_ia_alloc(struct dat_provider *provider, tduffy> + struct dapl_hca *hca_ptr); tduffy> tduffy> -u32 dapl_ia_abrupt_close(DAPL_IA *ia_ptr); tduffy> +u32 dapl_ia_abrupt_close(struct dapl_ia *ia_ptr); tduffy> tduffy> -u32 dapl_ia_graceful_close(DAPL_IA *ia_ptr); tduffy> +u32 dapl_ia_graceful_close(struct dapl_ia *ia_ptr); tduffy> tduffy> -void dapl_ia_free(DAPL_IA * ia_ptr); tduffy> +void dapl_ia_free(struct dapl_ia *ia_ptr); tduffy> tduffy> -void dapl_ia_link_ep(DAPL_IA * ia_ptr, DAPL_EP * ep_info); tduffy> +void dapl_ia_link_ep(struct dapl_ia *ia_ptr, struct dapl_ep *ep_info); tduffy> tduffy> -void dapl_ia_unlink_ep(DAPL_IA * ia_ptr, DAPL_EP * ep_info); tduffy> +void dapl_ia_unlink_ep(struct dapl_ia *ia_ptr, struct dapl_ep *ep_info); tduffy> tduffy> -void dapl_ia_link_srq(DAPL_IA * ia_ptr, DAPL_SRQ * srq_ptr); tduffy> +void dapl_ia_link_srq(struct dapl_ia *ia_ptr, struct dapl_srq *srq_ptr); tduffy> tduffy> -void dapl_ia_unlink_srq(DAPL_IA * ia_ptr, DAPL_SRQ * srq_ptr); tduffy> +void dapl_ia_unlink_srq(struct dapl_ia *ia_ptr, struct dapl_srq *srq_ptr); tduffy> tduffy> -void dapl_ia_link_lmr(DAPL_IA * ia_ptr, DAPL_LMR * lmr_info); tduffy> +void dapl_ia_link_lmr(struct dapl_ia *ia_ptr, struct dapl_lmr *lmr_info); tduffy> tduffy> -void dapl_ia_unlink_lmr(DAPL_IA * ia_ptr, DAPL_LMR * lmr_info); tduffy> +void dapl_ia_unlink_lmr(struct dapl_ia *ia_ptr, struct dapl_lmr *lmr_info); tduffy> tduffy> -void dapl_ia_link_rmr(DAPL_IA * ia_ptr, DAPL_RMR * rmr_info); tduffy> +void dapl_ia_link_rmr(struct dapl_ia *ia_ptr, struct dapl_rmr *rmr_info); tduffy> tduffy> -void dapl_ia_unlink_rmr(DAPL_IA * ia_ptr, DAPL_RMR * rmr_info); tduffy> +void dapl_ia_unlink_rmr(struct dapl_ia *ia_ptr, struct dapl_rmr *rmr_info); tduffy> tduffy> -void dapl_ia_link_pz(DAPL_IA * ia_ptr, DAPL_PZ * pz_info); tduffy> +void dapl_ia_link_pz(struct dapl_ia *ia_ptr, struct dapl_pz *pz_info); tduffy> tduffy> -void dapl_ia_unlink_pz(DAPL_IA * ia_ptr, DAPL_PZ * pz_info); tduffy> +void dapl_ia_unlink_pz(struct dapl_ia *ia_ptr, struct dapl_pz *pz_info); tduffy> tduffy> -void dapl_ia_link_evd(DAPL_IA * ia_ptr, DAPL_EVD * evd_info); tduffy> +void dapl_ia_link_evd(struct dapl_ia *ia_ptr, struct dapl_evd *evd_info); tduffy> tduffy> -void dapl_ia_unlink_evd(DAPL_IA * ia_ptr, DAPL_EVD * evd_info); tduffy> +void dapl_ia_unlink_evd(struct dapl_ia *ia_ptr, struct dapl_evd *evd_info); tduffy> tduffy> -void dapl_ia_link_psp(DAPL_IA * ia_ptr, DAPL_SP * sp_info); tduffy> +void dapl_ia_link_psp(struct dapl_ia *ia_ptr, struct dapl_sp *sp_info); tduffy> tduffy> -void dapl_ia_unlink_sp(DAPL_IA * ia_ptr, DAPL_SP * sp_info); tduffy> +void dapl_ia_unlink_sp(struct dapl_ia *ia_ptr, struct dapl_sp *sp_info); tduffy> tduffy> -void dapl_ia_link_rsp(DAPL_IA * ia_ptr, DAPL_SP * sp_info); tduffy> +void dapl_ia_link_rsp(struct dapl_ia *ia_ptr, struct dapl_sp *sp_info); tduffy> tduffy> -DAPL_SP *dapl_ia_sp_search(DAPL_IA * ia_ptr, tduffy> +struct dapl_sp *dapl_ia_sp_search(struct dapl_ia *ia_ptr, tduffy> DAT_CONN_QUAL conn_qual, boolean_t is_psp); tduffy> tduffy> -u32 dapl_ia_setup_callbacks(DAPL_IA *ia_ptr, DAPL_EVD *async_evd_ptr); tduffy> +u32 dapl_ia_setup_callbacks(struct dapl_ia *ia_ptr, tduffy> + struct dapl_evd *async_evd_ptr); tduffy> tduffy> -u32 dapl_ia_teardown_callbacks(DAPL_IA *ia_ptr); tduffy> +u32 dapl_ia_teardown_callbacks(struct dapl_ia *ia_ptr); tduffy> tduffy> #endif tduffy> Index: linux-kernel-resolv/dat-provider/dapl_rmr_bind.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_rmr_bind.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_rmr_bind.c (working copy) tduffy> @@ -46,16 +46,16 @@ tduffy> * * tduffy> *********************************************************************/ tduffy> tduffy> -static inline u32 dapl_rmr_bind_fuse(DAPL_RMR *rmr, tduffy> +static inline u32 dapl_rmr_bind_fuse(struct dapl_rmr *rmr, tduffy> const struct dat_lmr_triplet *lmr_triplet, tduffy> enum dat_mem_priv_flags mem_priv, tduffy> - DAPL_EP *ep_ptr, tduffy> + struct dapl_ep *ep_ptr, tduffy> DAT_RMR_COOKIE user_cookie, tduffy> enum dat_completion_flags completion_flags, tduffy> DAT_RMR_CONTEXT *rmr_context) tduffy> { tduffy> - DAPL_LMR *lmr; tduffy> - DAPL_COOKIE *cookie; tduffy> + struct dapl_lmr *lmr; tduffy> + struct dapl_cookie *cookie; tduffy> u32 status; tduffy> boolean_t is_signaled; tduffy> tduffy> @@ -152,15 +152,16 @@ static inline u32 dapl_rmr_bind_fuse(DAP tduffy> if (NULL != rmr_context) tduffy> *rmr_context = rmr->param.rmr_context; tduffy> tduffy> - bail: tduffy> +bail: tduffy> return status; tduffy> } tduffy> tduffy> -static inline u32 dapl_rmr_bind_unfuse(DAPL_RMR *rmr, DAPL_EP *ep_ptr, tduffy> +static inline u32 dapl_rmr_bind_unfuse(struct dapl_rmr *rmr, tduffy> + struct dapl_ep *ep_ptr, tduffy> DAT_RMR_COOKIE user_cookie, tduffy> enum dat_completion_flags cflags) tduffy> { tduffy> - DAPL_COOKIE *cookie; tduffy> + struct dapl_cookie *cookie; tduffy> u32 status = DAT_SUCCESS; tduffy> boolean_t is_signaled; tduffy> tduffy> @@ -232,7 +233,7 @@ static inline u32 dapl_rmr_bind_unfuse(D tduffy> rmr->ep = ep_ptr; tduffy> rmr->lmr = NULL; tduffy> tduffy> - bail1: tduffy> +bail1: tduffy> return status; tduffy> } tduffy> tduffy> @@ -254,8 +255,8 @@ u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_han tduffy> enum dat_completion_flags completion_flags, tduffy> DAT_RMR_CONTEXT *rmr_context) tduffy> { tduffy> - DAPL_RMR *rmr; tduffy> - DAPL_EP *ep_ptr; tduffy> + struct dapl_rmr *rmr; tduffy> + struct dapl_ep *ep_ptr; tduffy> tduffy> if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) tduffy> return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); tduffy> @@ -263,16 +264,15 @@ u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_han tduffy> if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) tduffy> return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); tduffy> tduffy> - rmr = (DAPL_RMR *) rmr_handle; tduffy> - ep_ptr = (DAPL_EP *) ep_handle; tduffy> + rmr = (struct dapl_rmr *)rmr_handle; tduffy> + ep_ptr = (struct dapl_ep *)ep_handle; tduffy> tduffy> /* if the rmr should be bound */ tduffy> - if (0 != lmr_triplet->segment_length) { tduffy> + if (0 != lmr_triplet->segment_length) tduffy> return dapl_rmr_bind_fuse(rmr, lmr_triplet, mem_priv, ep_ptr, tduffy> user_cookie, completion_flags, tduffy> rmr_context); tduffy> - } else { /* the rmr should be unbound */ tduffy> + else /* the rmr should be unbound */ tduffy> return dapl_rmr_bind_unfuse(rmr, ep_ptr, user_cookie, tduffy> completion_flags); tduffy> - } tduffy> } tduffy> Index: linux-kernel-resolv/dat-provider/dapl_cr_reject.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_cr_reject.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_cr_reject.c (working copy) tduffy> @@ -42,11 +42,11 @@ tduffy> */ tduffy> u32 dapl_cr_reject(DAT_CR_HANDLE cr_handle) tduffy> { tduffy> - DAPL_CR *cr_ptr; tduffy> - DAPL_EP *ep_ptr; tduffy> + struct dapl_cr *cr_ptr; tduffy> + struct dapl_ep *ep_ptr; tduffy> enum dat_ep_state entry_ep_state; tduffy> DAT_EP_HANDLE entry_ep_handle; tduffy> - DAPL_SP *sp_ptr; tduffy> + struct dapl_sp *sp_ptr; tduffy> u32 dat_status; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_cr_reject (%p)\n", cr_handle); tduffy> @@ -57,16 +57,16 @@ u32 dapl_cr_reject(DAT_CR_HANDLE cr_hand tduffy> goto bail; tduffy> } tduffy> tduffy> - cr_ptr = (DAPL_CR *) cr_handle; tduffy> + cr_ptr = (struct dapl_cr *)cr_handle; tduffy> tduffy> /* tduffy> * Clean up provider created EP if there is one: only if tduffy> * DAT_PSP_PROVIDER_FLAG was set on the PSP tduffy> */ tduffy> - ep_ptr = (DAPL_EP *) cr_ptr->param.local_ep_handle; tduffy> + ep_ptr = (struct dapl_ep *)cr_ptr->param.local_ep_handle; tduffy> entry_ep_handle = cr_ptr->param.local_ep_handle; tduffy> entry_ep_state = 0; tduffy> - if (ep_ptr != NULL) { tduffy> + if (ep_ptr) { tduffy> entry_ep_state = ep_ptr->param.ep_state; tduffy> ep_ptr->param.ep_state = DAT_EP_STATE_UNCONNECTED; tduffy> cr_ptr->param.local_ep_handle = NULL; tduffy> @@ -87,10 +87,8 @@ u32 dapl_cr_reject(DAT_CR_HANDLE cr_hand tduffy> * RSP and user-provided EPs are in the control of the user. tduffy> */ tduffy> sp_ptr = cr_ptr->sp_ptr; tduffy> - if (ep_ptr != NULL && tduffy> - sp_ptr->psp_flags == DAT_PSP_PROVIDER_FLAG) { tduffy> + if (ep_ptr && sp_ptr->psp_flags == DAT_PSP_PROVIDER_FLAG) tduffy> (void)dapl_ep_free(ep_ptr); tduffy> - } tduffy> tduffy> /* Remove the CR from the queue, then free it */ tduffy> spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); tduffy> @@ -101,6 +99,6 @@ u32 dapl_cr_reject(DAT_CR_HANDLE cr_hand tduffy> dapl_cr_free(cr_ptr); tduffy> } tduffy> tduffy> - bail: tduffy> +bail: tduffy> return dat_status; tduffy> } tduffy> Index: linux-kernel-resolv/dat-provider/dapl_ep_reset.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_ep_reset.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_ep_reset.c (working copy) tduffy> @@ -63,10 +63,10 @@ tduffy> */ tduffy> u32 dapl_ep_reset(DAT_EP_HANDLE ep_handle) tduffy> { tduffy> - DAPL_EP *ep_ptr; tduffy> + struct dapl_ep *ep_ptr; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> - ep_ptr = (DAPL_EP *) ep_handle; tduffy> + ep_ptr = (struct dapl_ep *)ep_handle; tduffy> tduffy> /* tduffy> * Verify parameter & state tduffy> Index: linux-kernel-resolv/dat-provider/dapl_ep_recv_query.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_ep_recv_query.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_ep_recv_query.c (working copy) tduffy> @@ -57,13 +57,13 @@ tduffy> u32 dapl_ep_recv_query(DAT_EP_HANDLE ep_handle, tduffy> int *nbufs_allocate, int *bufs_alloc_span) tduffy> { tduffy> - DAPL_EP *ep_ptr; tduffy> + struct dapl_ep *ep_ptr; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_recv_query (%p, %p, %p)\n", tduffy> ep_handle, nbufs_allocate, bufs_alloc_span); tduffy> tduffy> - ep_ptr = (DAPL_EP *) ep_handle; tduffy> + ep_ptr = (struct dapl_ep *)ep_handle; tduffy> tduffy> /* tduffy> * Verify parameter & state tduffy> Index: linux-kernel-resolv/dat-provider/dapl_openib_util.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_openib_util.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_openib_util.c (working copy) tduffy> @@ -159,7 +159,7 @@ static inline int dapl_ib_convert_mem_pr tduffy> * DAT_INSUFFICIENT_RESOURCES tduffy> * tduffy> */ tduffy> -u32 dapl_ib_open_hca(char *name, DAPL_HCA * hca_ptr) tduffy> +u32 dapl_ib_open_hca(char *name, struct dapl_hca *hca_ptr) tduffy> { tduffy> u32 dat_ret; tduffy> tduffy> @@ -196,7 +196,7 @@ u32 dapl_ib_open_hca(char *name, DAPL_HC tduffy> * DAT_INSUFFICIENT_RESOURCES tduffy> * tduffy> */ tduffy> -u32 dapl_ib_close_hca(DAPL_HCA *hca_ptr) tduffy> +u32 dapl_ib_close_hca(struct dapl_hca *hca_ptr) tduffy> { tduffy> return DAT_SUCCESS; tduffy> } tduffy> @@ -219,8 +219,8 @@ u32 dapl_ib_close_hca(DAPL_HCA *hca_ptr) tduffy> * DAT_INSUFFICIENT_RESOURCES tduffy> * tduffy> */ tduffy> -u32 dapl_ib_cq_alloc(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr, int *cqlen, tduffy> - ib_comp_handler callback) tduffy> +u32 dapl_ib_cq_alloc(struct dapl_ia *ia_ptr, struct dapl_evd *evd_ptr, tduffy> + int *cqlen, ib_comp_handler callback) tduffy> { tduffy> int ib_status; tduffy> tduffy> @@ -253,7 +253,7 @@ u32 dapl_ib_cq_alloc(DAPL_IA *ia_ptr, DA tduffy> * DAT_INSUFFICIENT_RESOURCES tduffy> * tduffy> */ tduffy> -u32 dapl_ib_cq_free(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr) tduffy> +u32 dapl_ib_cq_free(struct dapl_ia *ia_ptr, struct dapl_evd *evd_ptr) tduffy> { tduffy> int ib_status; tduffy> tduffy> @@ -261,7 +261,8 @@ u32 dapl_ib_cq_free(DAPL_IA *ia_ptr, DAP tduffy> return dapl_ib_status_convert(ib_status); tduffy> } tduffy> tduffy> -u32 dapl_ib_cq_resize(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr, int *cqlen) tduffy> +u32 dapl_ib_cq_resize(struct dapl_ia *ia_ptr, struct dapl_evd *evd_ptr, tduffy> + int *cqlen) tduffy> { tduffy> int ib_status; tduffy> tduffy> @@ -293,7 +294,7 @@ u32 dapl_ib_cq_resize(DAPL_IA *ia_ptr, D tduffy> * DAT_INSUFFICIENT_RESOURCES tduffy> * tduffy> */ tduffy> -u32 dapl_set_cq_notify(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr) tduffy> +u32 dapl_set_cq_notify(struct dapl_ia *ia_ptr, struct dapl_evd *evd_ptr) tduffy> { tduffy> int ib_status; tduffy> tduffy> @@ -318,7 +319,7 @@ u32 dapl_set_cq_notify(DAPL_IA *ia_ptr, tduffy> * DAT_INSUFFICIENT_RESOURCES tduffy> * tduffy> */ tduffy> -u32 dapl_ib_pd_alloc(DAPL_IA *ia, DAPL_PZ *pz) tduffy> +u32 dapl_ib_pd_alloc(struct dapl_ia *ia, struct dapl_pz *pz) tduffy> { tduffy> struct ib_pd *pd; tduffy> int ib_status; tduffy> @@ -353,7 +354,7 @@ u32 dapl_ib_pd_alloc(DAPL_IA *ia, DAPL_P tduffy> * DAT_INSUFFICIENT_RESOURCES tduffy> * tduffy> */ tduffy> -u32 dapl_ib_pd_free(DAPL_PZ *pz) tduffy> +u32 dapl_ib_pd_free(struct dapl_pz *pz) tduffy> { tduffy> int ib_status; tduffy> tduffy> @@ -388,14 +389,15 @@ u32 dapl_ib_pd_free(DAPL_PZ *pz) tduffy> * DAT_INSUFFICIENT_RESOURCES tduffy> * tduffy> */ tduffy> -u32 dapl_ib_mr_register(DAPL_IA *ia, DAPL_LMR *lmr, void *virt_addr, tduffy> - u64 length, enum dat_mem_priv_flags privileges) tduffy> +u32 dapl_ib_mr_register(struct dapl_ia *ia, struct dapl_lmr *lmr, tduffy> + void *virt_addr, u64 length, tduffy> + enum dat_mem_priv_flags privileges) tduffy> { tduffy> /* ITBD virtual memory registration ??? */ tduffy> return DAT_SUCCESS; tduffy> } tduffy> tduffy> -u32 dapl_ib_mr_register_physical(DAPL_IA *ia_ptr, DAPL_LMR *lmr, tduffy> +u32 dapl_ib_mr_register_physical(struct dapl_ia *ia_ptr, struct dapl_lmr *lmr, tduffy> void *phys_addr, u64 length, tduffy> enum dat_mem_priv_flags privileges) tduffy> { tduffy> @@ -421,7 +423,7 @@ u32 dapl_ib_mr_register_physical(DAPL_IA tduffy> iova = buf_list[0].addr; tduffy> acl = dapl_ib_convert_mem_privileges(privileges); tduffy> acl |= IB_ACCESS_MW_BIND; tduffy> - mr = ib_reg_phys_mr((struct ib_pd *)((DAPL_PZ *) lmr->param.pz_handle)-> tduffy> + mr = ib_reg_phys_mr((struct ib_pd *)((struct dapl_pz *)lmr->param.pz_handle)-> tduffy> pd_handle, buf_list, length, acl, &iova); tduffy> kfree(buf_list); tduffy> if (IS_ERR(mr)) { tduffy> @@ -469,7 +471,7 @@ u32 dapl_ib_mr_register_physical(DAPL_IA tduffy> * DAT_INSUFFICIENT_RESOURCES tduffy> * tduffy> */ tduffy> -u32 dapl_ib_mr_deregister(DAPL_LMR *lmr) tduffy> +u32 dapl_ib_mr_deregister(struct dapl_lmr *lmr) tduffy> { tduffy> int ib_status; tduffy> tduffy> @@ -503,12 +505,12 @@ u32 dapl_ib_mr_deregister(DAPL_LMR *lmr) tduffy> * DAT_INSUFFICIENT_RESOURCES tduffy> * tduffy> */ tduffy> -u32 dapl_ib_mw_alloc(DAPL_RMR *rmr) tduffy> +u32 dapl_ib_mw_alloc(struct dapl_rmr *rmr) tduffy> { tduffy> int ib_status; tduffy> struct ib_mw *mw; tduffy> tduffy> - mw = ib_alloc_mw((struct ib_pd *)((DAPL_PZ *) rmr->param.pz_handle)-> tduffy> + mw = ib_alloc_mw((struct ib_pd *)((struct dapl_pz *)rmr->param.pz_handle)-> tduffy> pd_handle); tduffy> if (IS_ERR(mw)) { tduffy> ib_status = PTR_ERR(mw); tduffy> @@ -540,7 +542,7 @@ u32 dapl_ib_mw_alloc(DAPL_RMR *rmr) tduffy> * DAT_INSUFFICIENT_RESOURCES tduffy> * tduffy> */ tduffy> -u32 dapl_ib_mw_free(DAPL_RMR *rmr) tduffy> +u32 dapl_ib_mw_free(struct dapl_rmr *rmr) tduffy> { tduffy> int ib_status; tduffy> tduffy> @@ -573,10 +575,10 @@ u32 dapl_ib_mw_free(DAPL_RMR *rmr) tduffy> * DAT_INSUFFICIENT_RESOURCES tduffy> * tduffy> */ tduffy> -u32 dapl_ib_mw_bind(DAPL_RMR *rmr, DAPL_LMR *lmr, DAPL_EP *ep, tduffy> - DAPL_COOKIE *cookie, u64 virtual_address, tduffy> - u64 length, enum dat_mem_priv_flags mem_priv, tduffy> - boolean_t is_signaled) tduffy> +u32 dapl_ib_mw_bind(struct dapl_rmr *rmr, struct dapl_lmr *lmr, tduffy> + struct dapl_ep *ep, struct dapl_cookie *cookie, tduffy> + u64 virtual_address, u64 length, tduffy> + enum dat_mem_priv_flags mem_priv, boolean_t is_signaled) tduffy> { tduffy> int ib_status; tduffy> struct ib_mw_bind mw_bind_prop; tduffy> @@ -624,8 +626,8 @@ u32 dapl_ib_mw_bind(DAPL_RMR *rmr, DAPL_ tduffy> * DAT_INSUFFICIENT_RESOURCES tduffy> * tduffy> */ tduffy> -u32 dapl_ib_mw_unbind(DAPL_RMR *rmr, DAPL_EP *ep, DAPL_COOKIE *cookie, tduffy> - boolean_t is_signaled) tduffy> +u32 dapl_ib_mw_unbind(struct dapl_rmr *rmr, struct dapl_ep *ep, tduffy> + struct dapl_cookie *cookie, boolean_t is_signaled) tduffy> { tduffy> int ib_status; tduffy> struct ib_mw_bind mw_bind_prop; tduffy> @@ -671,7 +673,7 @@ u32 dapl_ib_mw_unbind(DAPL_RMR *rmr, DAP tduffy> * DAT_INVALID_PARAMETER tduffy> * tduffy> */ tduffy> -u32 dapl_ib_setup_async_callback(DAPL_IA *ia_ptr, tduffy> +u32 dapl_ib_setup_async_callback(struct dapl_ia *ia_ptr, tduffy> DAPL_ASYNC_HANDLER_TYPE handler_type, tduffy> ib_async_handler_t callback, void *context) tduffy> { tduffy> @@ -696,7 +698,7 @@ u32 dapl_ib_setup_async_callback(DAPL_IA tduffy> * DAT_SUCCESS tduffy> * DAT_INVALID_PARAMETER tduffy> */ tduffy> -u32 dapl_ib_query_hca(DAPL_HCA *hca_ptr, struct dat_ia_attr *ia_attr, tduffy> +u32 dapl_ib_query_hca(struct dapl_hca *hca_ptr, struct dat_ia_attr *ia_attr, tduffy> struct dat_ep_attr *ep_attr, struct sockaddr_in6 *ip_addr) tduffy> { tduffy> int ib_status; tduffy> @@ -797,7 +799,7 @@ u32 dapl_ib_query_hca(DAPL_HCA *hca_ptr, tduffy> return DAT_SUCCESS; tduffy> } tduffy> tduffy> -u32 dapl_ib_completion_poll(DAPL_HCA *hca_ptr, DAPL_EVD *evd_ptr, tduffy> +u32 dapl_ib_completion_poll(struct dapl_hca *hca_ptr, struct dapl_evd *evd_ptr, tduffy> struct ib_wc *cqe_ptr) tduffy> { tduffy> int ib_status; tduffy> @@ -810,7 +812,7 @@ u32 dapl_ib_completion_poll(DAPL_HCA *hc tduffy> return dapl_ib_status_convert(ib_status); tduffy> } tduffy> tduffy> -u32 dapl_ib_completion_notify(DAPL_EVD *evd_ptr) tduffy> +u32 dapl_ib_completion_notify(struct dapl_evd *evd_ptr) tduffy> { tduffy> int ib_status; tduffy> tduffy> Index: linux-kernel-resolv/dat-provider/dapl_ep_modify.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_ep_modify.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_ep_modify.c (working copy) tduffy> @@ -56,12 +56,12 @@ static inline u32 tduffy> dapl_ep_modify_validate_parameters(DAT_EP_HANDLE ep_handle, tduffy> enum dat_ep_param_mask mask, tduffy> const struct dat_ep_param *ep_param, tduffy> - DAPL_IA ** ia_ptr, tduffy> - DAPL_EP ** ep_ptr, tduffy> + struct dapl_ia **ia_ptr, tduffy> + struct dapl_ep **ep_ptr, tduffy> struct dat_ep_attr *ep_attr_ptr) tduffy> { tduffy> - DAPL_IA *ia; tduffy> - DAPL_EP *ep; tduffy> + struct dapl_ia *ia; tduffy> + struct dapl_ep *ep; tduffy> struct dat_ep_attr ep_attr; tduffy> struct dat_ep_attr ep_attr_limit; tduffy> struct dat_ep_attr ep_attr_request; tduffy> @@ -76,7 +76,7 @@ dapl_ep_modify_validate_parameters(DAT_E tduffy> goto bail; tduffy> } tduffy> tduffy> - ep = (DAPL_EP *) ep_handle; tduffy> + ep = (struct dapl_ep *)ep_handle; tduffy> ia = ep->header.owner_ia; tduffy> tduffy> /* tduffy> @@ -139,7 +139,7 @@ dapl_ep_modify_validate_parameters(DAT_E tduffy> if (mask & DAT_EP_FIELD_RECV_EVD_HANDLE) { tduffy> if (ep_param->recv_evd_handle != NULL && tduffy> (DAPL_BAD_HANDLE(ep_param->recv_evd_handle, DAPL_MAGIC_EVD) tduffy> - || !((DAPL_EVD *) ep_param->recv_evd_handle)-> tduffy> + || !((struct dapl_evd *)ep_param->recv_evd_handle)-> tduffy> evd_flags & DAT_EVD_DTO_FLAG)) { tduffy> dat_status = tduffy> DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); tduffy> @@ -161,7 +161,7 @@ dapl_ep_modify_validate_parameters(DAT_E tduffy> if (ep_param->connect_evd_handle != NULL && tduffy> DAPL_BAD_HANDLE(ep_param->connect_evd_handle, tduffy> DAPL_MAGIC_EVD) tduffy> - && !(((DAPL_EVD *) ep_param->connect_evd_handle)-> tduffy> + && !(((struct dapl_evd *)ep_param->connect_evd_handle)-> tduffy> evd_flags & DAT_EVD_CONNECTION_FLAG)) { tduffy> dat_status = tduffy> DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); tduffy> @@ -298,13 +298,13 @@ dapl_ep_modify_validate_parameters(DAT_E tduffy> u32 dapl_ep_modify(DAT_EP_HANDLE ep_handle, enum dat_ep_param_mask mask, tduffy> const struct dat_ep_param *ep_param) tduffy> { tduffy> - DAPL_IA *ia; tduffy> - DAPL_EP *ep1, *ep2; tduffy> + struct dapl_ia *ia; tduffy> + struct dapl_ep *ep1, *ep2; tduffy> struct dat_ep_attr ep_attr1, ep_attr2; tduffy> - DAPL_EP new_ep, copy_of_old_ep; tduffy> - DAPL_EP alloc_ep; /* Holder for resources. */ tduffy> - DAPL_PZ *tmp_pz; tduffy> - DAPL_EVD *tmp_evd; tduffy> + struct dapl_ep new_ep, copy_of_old_ep; tduffy> + struct dapl_ep alloc_ep; /* Holder for resources. */ tduffy> + struct dapl_pz *tmp_pz; tduffy> + struct dapl_evd *tmp_evd; tduffy> u32 dat_status; tduffy> tduffy> /* Flag indicating we've allocated a new one of these. */ tduffy> @@ -325,9 +325,8 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand tduffy> mask, tduffy> ep_param, tduffy> &ia, &ep1, &ep_attr1); tduffy> - if (DAT_SUCCESS != dat_status) { tduffy> + if (DAT_SUCCESS != dat_status) tduffy> goto bail; tduffy> - } tduffy> tduffy> /* tduffy> * Setup the alloc_ep with the appropriate parameters (primarily tduffy> @@ -335,23 +334,17 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand tduffy> */ tduffy> alloc_ep = *ep1; tduffy> alloc_ep.param.ep_attr = ep_attr1; tduffy> - if (mask & DAT_EP_FIELD_PZ_HANDLE) { tduffy> + if (mask & DAT_EP_FIELD_PZ_HANDLE) tduffy> alloc_ep.param.pz_handle = ep_param->pz_handle; tduffy> - } tduffy> tduffy> - if (mask & DAT_EP_FIELD_RECV_EVD_HANDLE) { tduffy> + if (mask & DAT_EP_FIELD_RECV_EVD_HANDLE) tduffy> alloc_ep.param.recv_evd_handle = ep_param->recv_evd_handle; tduffy> - } tduffy> tduffy> - if (mask & DAT_EP_FIELD_REQUEST_EVD_HANDLE) { tduffy> - alloc_ep.param.request_evd_handle = tduffy> - ep_param->request_evd_handle; tduffy> - } tduffy> + if (mask & DAT_EP_FIELD_REQUEST_EVD_HANDLE) tduffy> + alloc_ep.param.request_evd_handle = ep_param->request_evd_handle; tduffy> tduffy> - if (mask & DAT_EP_FIELD_CONNECT_EVD_HANDLE) { tduffy> - alloc_ep.param.connect_evd_handle = tduffy> - ep_param->connect_evd_handle; tduffy> - } tduffy> + if (mask & DAT_EP_FIELD_CONNECT_EVD_HANDLE) tduffy> + alloc_ep.param.connect_evd_handle = ep_param->connect_evd_handle; tduffy> tduffy> /* tduffy> * Allocate everything that might be needed. tduffy> @@ -364,16 +357,14 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand tduffy> */ tduffy> dat_status = dapl_cb_create(&alloc_ep.req_buffer, ep1, /* For pointer in buffer bool. */ tduffy> ep_attr1.max_request_dtos); tduffy> - if (DAT_SUCCESS != dat_status) { tduffy> + if (DAT_SUCCESS != dat_status) tduffy> goto bail; tduffy> - } tduffy> rqst_cb_allocated = TRUE; tduffy> tduffy> dat_status = dapl_cb_create(&alloc_ep.recv_buffer, ep1, /* For pointer in buffer bool. */ tduffy> ep_attr1.max_recv_dtos); tduffy> - if (DAT_SUCCESS != dat_status) { tduffy> + if (DAT_SUCCESS != dat_status) tduffy> goto bail; tduffy> - } tduffy> recv_cb_allocated = TRUE; tduffy> tduffy> alloc_ep.send_iov_num = ep_attr1.max_request_iov; tduffy> @@ -397,9 +388,8 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand tduffy> recv_iov_allocated = TRUE; tduffy> tduffy> dat_status = dapl_ib_qp_alloc(ia, &alloc_ep, ep1); tduffy> - if (dat_status != DAT_SUCCESS) { tduffy> + if (dat_status != DAT_SUCCESS) tduffy> goto bail; tduffy> - } tduffy> qp_allocated = TRUE; tduffy> tduffy> /* tduffy> @@ -527,92 +517,79 @@ u32 dapl_ep_modify(DAT_EP_HANDLE ep_hand tduffy> * and then decrementing old ones (so if they're the same tduffy> * the refcount never drops to zero). tduffy> */ tduffy> - tmp_pz = (DAPL_PZ *) new_ep.param.pz_handle; tduffy> - if (NULL != tmp_pz) { tduffy> + tmp_pz = (struct dapl_pz *)new_ep.param.pz_handle; tduffy> + if (tmp_pz) tduffy> atomic_inc(&tmp_pz->pz_ref_count); tduffy> - } tduffy> tduffy> - tmp_evd = (DAPL_EVD *) new_ep.param.recv_evd_handle; tduffy> - if (NULL != tmp_evd) { tduffy> + tmp_evd = (struct dapl_evd *)new_ep.param.recv_evd_handle; tduffy> + if (tmp_evd) tduffy> atomic_inc(&tmp_evd->evd_ref_count); tduffy> - } tduffy> tduffy> - tmp_evd = (DAPL_EVD *) new_ep.param.request_evd_handle; tduffy> - if (NULL != tmp_evd) { tduffy> + tmp_evd = (struct dapl_evd *)new_ep.param.request_evd_handle; tduffy> + if (tmp_evd) tduffy> atomic_inc(&tmp_evd->evd_ref_count); tduffy> - } tduffy> tduffy> - tmp_evd = (DAPL_EVD *) new_ep.param.connect_evd_handle; tduffy> - if (NULL != tmp_evd) { tduffy> + tmp_evd = (struct dapl_evd *)new_ep.param.connect_evd_handle; tduffy> + if (tmp_evd) tduffy> atomic_inc(&tmp_evd->evd_ref_count); tduffy> - } tduffy> tduffy> /* decreament the old reference counts */ tduffy> - tmp_pz = (DAPL_PZ *) copy_of_old_ep.param.pz_handle; tduffy> - if (NULL != tmp_pz) { tduffy> + tmp_pz = (struct dapl_pz *)copy_of_old_ep.param.pz_handle; tduffy> + if (tmp_pz) tduffy> atomic_dec(&tmp_pz->pz_ref_count); tduffy> - } tduffy> tduffy> - tmp_evd = (DAPL_EVD *) copy_of_old_ep.param.recv_evd_handle; tduffy> - if (NULL != tmp_evd) { tduffy> + tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.recv_evd_handle; tduffy> + if (tmp_evd) tduffy> atomic_dec(&tmp_evd->evd_ref_count); tduffy> - } tduffy> tduffy> - tmp_evd = (DAPL_EVD *) copy_of_old_ep.param.request_evd_handle; tduffy> - if (NULL != tmp_evd) { tduffy> + tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.request_evd_handle; tduffy> + if (tmp_evd) tduffy> atomic_dec(&tmp_evd->evd_ref_count); tduffy> - } tduffy> tduffy> - tmp_evd = (DAPL_EVD *) copy_of_old_ep.param.connect_evd_handle; tduffy> - if (NULL != tmp_evd) { tduffy> + tmp_evd = (struct dapl_evd *)copy_of_old_ep.param.connect_evd_handle; tduffy> + if (tmp_evd) tduffy> atomic_dec(&tmp_evd->evd_ref_count); tduffy> - } tduffy> tduffy> - bail: tduffy> +bail: tduffy> if (qp_allocated) { tduffy> u32 local_dat_status; tduffy> - if (dat_status != DAT_SUCCESS || !qp_used) { tduffy> + if (dat_status != DAT_SUCCESS || !qp_used) tduffy> local_dat_status = dapl_ib_qp_free(ia, &alloc_ep); tduffy> - } else { tduffy> + else tduffy> local_dat_status = dapl_ib_qp_free(ia, ©_of_old_ep); tduffy> - } tduffy> - if (local_dat_status != DAT_SUCCESS) { tduffy> + if (local_dat_status != DAT_SUCCESS) tduffy> dapl_dbg_log(DAPL_DBG_TYPE_WARN, tduffy> "ep_modify: Failed to free QP; status %x\n", tduffy> local_dat_status); tduffy> - } tduffy> } tduffy> tduffy> if (rqst_cb_allocated) { tduffy> - if (dat_status != DAT_SUCCESS || !rqst_cb_used) { tduffy> + if (dat_status != DAT_SUCCESS || !rqst_cb_used) tduffy> dapl_cb_free(&alloc_ep.req_buffer); tduffy> - } else { tduffy> + else tduffy> dapl_cb_free(©_of_old_ep.req_buffer); tduffy> - } tduffy> } tduffy> tduffy> if (recv_cb_allocated) { tduffy> - if (dat_status != DAT_SUCCESS || !recv_cb_used) { tduffy> + if (dat_status != DAT_SUCCESS || !recv_cb_used) tduffy> dapl_cb_free(&alloc_ep.recv_buffer); tduffy> - } else { tduffy> + else tduffy> dapl_cb_free(©_of_old_ep.recv_buffer); tduffy> - } tduffy> } tduffy> tduffy> if (rqst_iov_allocated) { tduffy> - if (dat_status != DAT_SUCCESS || !rqst_iov_used) { tduffy> + if (dat_status != DAT_SUCCESS || !rqst_iov_used) tduffy> kfree(alloc_ep.send_iov); tduffy> - } else { tduffy> + else tduffy> kfree(copy_of_old_ep.send_iov); tduffy> - } tduffy> } tduffy> tduffy> if (recv_iov_allocated) { tduffy> - if (dat_status != DAT_SUCCESS || !recv_iov_used) { tduffy> + if (dat_status != DAT_SUCCESS || !recv_iov_used) tduffy> kfree(alloc_ep.recv_iov); tduffy> - } else { tduffy> + else tduffy> kfree(copy_of_old_ep.recv_iov); tduffy> - } tduffy> } tduffy> + tduffy> return dat_status; tduffy> } tduffy> Index: linux-kernel-resolv/dat-provider/dapl_openib_qp.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_openib_qp.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_openib_qp.c (working copy) tduffy> @@ -44,7 +44,8 @@ tduffy> #include "dapl_evd_util.h" tduffy> #include "dapl_openib_util.h" tduffy> tduffy> -struct ib_cq * dapl_get_dto_cq(DAPL_IA * ia_ptr, DAPL_EVD * recv_evd_ptr); tduffy> +struct ib_cq *dapl_get_dto_cq(struct dapl_ia *ia_ptr, tduffy> + struct dapl_evd *recv_evd_ptr); tduffy> tduffy> /* tduffy> * dapl_ib_qp_alloc tduffy> @@ -66,7 +67,8 @@ struct ib_cq * dapl_get_dto_cq(DAPL_IA * tduffy> * DAT_INSUFFICIENT_RESOURCES tduffy> * tduffy> */ tduffy> -u32 dapl_ib_qp_alloc(DAPL_IA *ia_ptr, DAPL_EP *ep_ptr, DAPL_EP *ep_ctx_ptr) tduffy> +u32 dapl_ib_qp_alloc(struct dapl_ia *ia_ptr, struct dapl_ep *ep_ptr, tduffy> + struct dapl_ep *ep_ctx_ptr) tduffy> { tduffy> struct dat_ep_attr *attr; tduffy> int ib_status; tduffy> @@ -80,15 +82,15 @@ u32 dapl_ib_qp_alloc(DAPL_IA *ia_ptr, DA tduffy> tduffy> dapl_os_assert(ep_ptr->param.pz_handle != NULL); tduffy> ib_pd_handle = tduffy> - (struct ib_pd *)((DAPL_PZ *) ep_ptr->param.pz_handle)->pd_handle; tduffy> + (struct ib_pd *)((struct dapl_pz *)ep_ptr->param.pz_handle)->pd_handle; tduffy> ib_hca_handle = ia_ptr->hca_ptr->ib_hca_handle; tduffy> tduffy> cq_recv = NULL; tduffy> cq_send = NULL; tduffy> cq_recv = dapl_get_dto_cq(ia_ptr, tduffy> - (DAPL_EVD *) ep_ptr->param.recv_evd_handle); tduffy> + (struct dapl_evd *)ep_ptr->param.recv_evd_handle); tduffy> cq_send = dapl_get_dto_cq(ia_ptr, tduffy> - (DAPL_EVD *) ep_ptr->param. tduffy> + (struct dapl_evd *)ep_ptr->param. tduffy> request_evd_handle); tduffy> tduffy> qp_attr.send_cq = cq_send; tduffy> @@ -151,7 +153,7 @@ u32 dapl_ib_qp_alloc(DAPL_IA *ia_ptr, DA tduffy> * Status of destroy operation. tduffy> * tduffy> */ tduffy> -u32 dapl_ib_qp_free(DAPL_IA *ia_ptr, DAPL_EP *ep_ptr) tduffy> +u32 dapl_ib_qp_free(struct dapl_ia *ia_ptr, struct dapl_ep *ep_ptr) tduffy> { tduffy> int ib_status; tduffy> tduffy> @@ -178,7 +180,7 @@ u32 dapl_ib_qp_free(DAPL_IA *ia_ptr, DAP tduffy> * Returns tduffy> * A valid CQ handle tduffy> */ tduffy> -struct ib_cq * dapl_get_dto_cq(DAPL_IA * ia_ptr, DAPL_EVD * evd_ptr) tduffy> +struct ib_cq *dapl_get_dto_cq(struct dapl_ia *ia_ptr, struct dapl_evd *evd_ptr) tduffy> { tduffy> struct ib_cq *cq_handle; tduffy> int cq_entries; tduffy> @@ -232,7 +234,7 @@ struct ib_cq * dapl_get_dto_cq(DAPL_IA * tduffy> * DAT_INVALID_PARAMETER tduffy> * tduffy> */ tduffy> -u32 dapl_ib_qp_modify(DAPL_IA *ia_ptr, DAPL_EP *ep_ptr, tduffy> +u32 dapl_ib_qp_modify(struct dapl_ia *ia_ptr, struct dapl_ep *ep_ptr, tduffy> struct dat_ep_attr *ep_attr) tduffy> { tduffy> struct ib_qp_attr qp_attr; tduffy> Index: linux-kernel-resolv/dat-provider/dapl_evd_kcreate.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_evd_kcreate.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_evd_kcreate.c (working copy) tduffy> @@ -53,9 +53,9 @@ u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_ha tduffy> const struct dat_upcall_object *upcall, tduffy> enum dat_evd_flags evd_flags, DAT_EVD_HANDLE *evd_handle) tduffy> { tduffy> - DAPL_IA *ia_ptr; tduffy> - DAPL_EVD *evd_ptr; tduffy> - DAPL_CNO *cno_ptr; tduffy> + struct dapl_ia *ia_ptr; tduffy> + struct dapl_evd *evd_ptr; tduffy> + struct dapl_cno *cno_ptr; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> DAT_EVD_HANDLE evdhandle = (DAT_EVD_HANDLE) NULL; tduffy> tduffy> @@ -64,7 +64,7 @@ u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_ha tduffy> ia_handle, tduffy> evd_min_qlen, upcall_policy, evd_flags, evd_handle); tduffy> tduffy> - ia_ptr = (DAPL_IA *) ia_handle; tduffy> + ia_ptr = (struct dapl_ia *)ia_handle; tduffy> cno_ptr = NULL; tduffy> evd_ptr = NULL; tduffy> *evd_handle = NULL; tduffy> @@ -94,38 +94,31 @@ u32 dapl_evd_kcreate(DAT_IA_HANDLE ia_ha tduffy> goto bail; tduffy> } tduffy> tduffy> - cno_ptr = dapl_cno_alloc (ia_ptr, upcall); tduffy> + cno_ptr = dapl_cno_alloc(ia_ptr, upcall); tduffy> tduffy> - if (!cno_ptr) tduffy> - { tduffy> + if (!cno_ptr) { tduffy> dat_status = DAT_INSUFFICIENT_RESOURCES; tduffy> goto bail; tduffy> } tduffy> tduffy> - cno_ptr->cno_upcall_policy = upcall_policy; tduffy> - tduffy> + cno_ptr->cno_upcall_policy = upcall_policy; tduffy> tduffy> dat_status = dapl_evd_internal_create(ia_ptr, tduffy> cno_ptr, tduffy> evd_min_qlen, tduffy> evd_flags, &evd_ptr); tduffy> - if (dat_status != DAT_SUCCESS) { tduffy> + if (dat_status != DAT_SUCCESS) tduffy> goto bail; tduffy> - } tduffy> tduffy> evdhandle = *evd_handle = (DAT_EVD_HANDLE) evd_ptr; tduffy> tduffy> - bail: tduffy> +bail: tduffy> if (dat_status != DAT_SUCCESS && cno_ptr != NULL) tduffy> - { tduffy> dapl_cno_dealloc (cno_ptr); tduffy> - } tduffy> tduffy> - if (dat_status != DAT_SUCCESS) { tduffy> - if (evd_ptr) { tduffy> + if (dat_status != DAT_SUCCESS) tduffy> + if (evd_ptr) tduffy> dapl_evd_free(evd_ptr); tduffy> - } tduffy> - } tduffy> tduffy> return dat_status; tduffy> } tduffy> Index: linux-kernel-resolv/dat-provider/dapl_cno_util.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_cno_util.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_cno_util.c (working copy) tduffy> @@ -38,9 +38,10 @@ tduffy> /* tduffy> * alloc and initialize an EVD struct tduffy> */ tduffy> -DAPL_CNO *dapl_cno_alloc(DAPL_IA *ia_ptr, const struct dat_upcall_object *upcall) tduffy> +struct dapl_cno *dapl_cno_alloc(struct dapl_ia *ia_ptr, tduffy> + const struct dat_upcall_object *upcall) tduffy> { tduffy> - DAPL_CNO *cno_ptr; tduffy> + struct dapl_cno *cno_ptr; tduffy> tduffy> cno_ptr = kmalloc(sizeof *cno_ptr, GFP_ATOMIC); tduffy> if (!cno_ptr) tduffy> @@ -71,7 +72,7 @@ DAPL_CNO *dapl_cno_alloc(DAPL_IA *ia_ptr tduffy> /* tduffy> * Free the passed in CNO structure. tduffy> */ tduffy> -void dapl_cno_dealloc(DAPL_CNO *cno_ptr) tduffy> +void dapl_cno_dealloc(struct dapl_cno *cno_ptr) tduffy> { tduffy> dapl_os_assert(cno_ptr->header.magic == DAPL_MAGIC_CNO); tduffy> tduffy> @@ -98,7 +99,7 @@ void dapl_cno_dealloc(DAPL_CNO *cno_ptr) tduffy> * Returns: tduffy> * None tduffy> */ tduffy> -void dapl_cno_trigger(DAPL_CNO *cno_ptr, DAPL_EVD *evd_ptr) tduffy> +void dapl_cno_trigger(struct dapl_cno *cno_ptr, struct dapl_evd *evd_ptr) tduffy> { tduffy> u32 dat_status = DAT_SUCCESS; tduffy> struct dat_event event; tduffy> Index: linux-kernel-resolv/dat-provider/dapl_timer_util.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_timer_util.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_timer_util.c (working copy) tduffy> @@ -53,7 +53,7 @@ tduffy> #include "dapl_timer_util.h" tduffy> tduffy> struct dapl_timer_head { tduffy> - DAPL_LLIST_HEAD timer_list_head; tduffy> + struct dapl_llist_entry *timer_list_head; tduffy> spinlock_t lock; tduffy> unsigned long flags; tduffy> DAPL_OS_WAIT_OBJECT wait_object; tduffy> @@ -129,7 +129,8 @@ u32 dapl_timer_set(struct dapl_timer_ent tduffy> if (dapl_llist_is_empty(&g_dapl_timer_head.timer_list_head)) { tduffy> /* Case 1: add entry to head of list */ tduffy> dapl_llist_add_head(&g_dapl_timer_head.timer_list_head, tduffy> - (DAPL_LLIST_ENTRY *) & timer->list_entry, tduffy> + (struct dapl_llist_entry *) tduffy> + &timer->list_entry, tduffy> timer); tduffy> wakeup_tmo_thread = TRUE; tduffy> } else { tduffy> @@ -139,42 +140,41 @@ u32 dapl_timer_set(struct dapl_timer_ent tduffy> if (timer->expires < list_ptr->expires) { tduffy> /* Case 2: add entry to head of list */ tduffy> dapl_llist_add_head(&g_dapl_timer_head.timer_list_head, tduffy> - (DAPL_LLIST_ENTRY *) & timer-> tduffy> - list_entry, timer); tduffy> + (struct dapl_llist_entry *) tduffy> + &timer->list_entry, timer); tduffy> wakeup_tmo_thread = TRUE; tduffy> } else { tduffy> /* Case 3: figure out where entry goes in sorted list */ tduffy> list_ptr = tduffy> dapl_llist_next_entry(&g_dapl_timer_head. tduffy> timer_list_head, tduffy> - (DAPL_LLIST_ENTRY *) & tduffy> - list_ptr->list_entry); tduffy> + (struct dapl_llist_entry *) tduffy> + &list_ptr->list_entry); tduffy> tduffy> while (list_ptr != NULL) { tduffy> if (timer->expires < list_ptr->expires) { tduffy> dapl_llist_add_entry(&g_dapl_timer_head. tduffy> timer_list_head, tduffy> - (DAPL_LLIST_ENTRY tduffy> - *) & list_ptr-> tduffy> - list_entry, tduffy> - (DAPL_LLIST_ENTRY tduffy> - *) & timer-> tduffy> - list_entry, timer); tduffy> + (struct dapl_llist_entry *) tduffy> + &list_ptr->list_entry, tduffy> + (struct dapl_llist_entry *) tduffy> + &timer->list_entry, tduffy> + timer); tduffy> break; tduffy> tduffy> } tduffy> list_ptr = tduffy> dapl_llist_next_entry(&g_dapl_timer_head. tduffy> timer_list_head, tduffy> - (DAPL_LLIST_ENTRY *) & tduffy> - list_ptr->list_entry); tduffy> + (struct dapl_llist_entry *) tduffy> + &list_ptr->list_entry); tduffy> } tduffy> if (list_ptr == NULL) { tduffy> /* entry goes to the end of the list */ tduffy> dapl_llist_add_tail(&g_dapl_timer_head. tduffy> timer_list_head, tduffy> - (DAPL_LLIST_ENTRY *) & tduffy> - timer->list_entry, timer); tduffy> + (struct dapl_llist_entry *) tduffy> + &timer->list_entry, timer); tduffy> } tduffy> } tduffy> tduffy> @@ -209,8 +209,8 @@ void dapl_timer_cancel(struct dapl_timer tduffy> if (!dapl_llist_is_empty(&g_dapl_timer_head.timer_list_head) && tduffy> timer->list_entry.list_head == &g_dapl_timer_head.timer_list_head) { tduffy> dapl_llist_remove_entry(&g_dapl_timer_head.timer_list_head, tduffy> - (DAPL_LLIST_ENTRY *) & timer-> tduffy> - list_entry); tduffy> + (struct dapl_llist_entry *) tduffy> + &timer->list_entry); tduffy> } tduffy> /* tduffy> * If this was the first entry on the queue we could awaken the tduffy> Index: linux-kernel-resolv/dat-provider/dapl_openib_util.h tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_openib_util.h (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_openib_util.h (working copy) tduffy> @@ -62,15 +62,16 @@ u32 dapl_ib_enum_hcas(const char *vendor tduffy> tduffy> u32 dapl_ib_get_instance_data(DAPL_HCA_NAME hca_name, char *instance); tduffy> tduffy> -u32 dapl_ib_open_hca(char *namestr, DAPL_HCA *hca_ptr); tduffy> +u32 dapl_ib_open_hca(char *namestr, struct dapl_hca *hca_ptr); tduffy> tduffy> -u32 dapl_ib_close_hca(DAPL_HCA *hca_ptr); tduffy> +u32 dapl_ib_close_hca(struct dapl_hca *hca_ptr); tduffy> tduffy> -u32 dapl_ib_qp_alloc(DAPL_IA *ia_ptr, DAPL_EP *ep_ptr, DAPL_EP *ep_ctx_ptr); tduffy> +u32 dapl_ib_qp_alloc(struct dapl_ia *ia_ptr, struct dapl_ep *ep_ptr, tduffy> + struct dapl_ep *ep_ctx_ptr); tduffy> tduffy> -u32 dapl_ib_qp_free(DAPL_IA *ia_ptr, DAPL_EP *ep_ptr); tduffy> +u32 dapl_ib_qp_free(struct dapl_ia *ia_ptr, struct dapl_ep *ep_ptr); tduffy> tduffy> -u32 dapl_ib_qp_modify(DAPL_IA *ia_ptr, DAPL_EP *ep_ptr, tduffy> +u32 dapl_ib_qp_modify(struct dapl_ia *ia_ptr, struct dapl_ep *ep_ptr, tduffy> struct dat_ep_attr *ep_attr); tduffy> tduffy> u32 dapl_ib_connect(DAT_EP_HANDLE ep_handle, tduffy> @@ -78,12 +79,12 @@ u32 dapl_ib_connect(DAT_EP_HANDLE ep_han tduffy> DAT_CONN_QUAL remote_conn_qual, tduffy> int private_data_size, void *private_data); tduffy> tduffy> -u32 dapl_ib_disconnect(DAPL_EP *ep_ptr, enum dat_close_flags close_flags); tduffy> +u32 dapl_ib_disconnect(struct dapl_ep *ep_ptr, enum dat_close_flags close_flags); tduffy> tduffy> -u32 dapl_ib_setup_conn_listener(DAPL_IA *ia_ptr, u64 ServiceID, tduffy> - DAPL_SP *sp_ptr); tduffy> +u32 dapl_ib_setup_conn_listener(struct dapl_ia *ia_ptr, u64 ServiceID, tduffy> + struct dapl_sp *sp_ptr); tduffy> tduffy> -u32 dapl_ib_remove_conn_listener(DAPL_IA *ia_ptr, DAPL_SP *sp_ptr); tduffy> +u32 dapl_ib_remove_conn_listener(struct dapl_ia *ia_ptr, struct dapl_sp *sp_ptr); tduffy> tduffy> u32 dapl_ib_accept_connection(DAT_CR_HANDLE cr_handle, tduffy> DAT_EP_HANDLE ep_handle, tduffy> @@ -92,61 +93,63 @@ u32 dapl_ib_accept_connection(DAT_CR_HAN tduffy> tduffy> u32 dapl_ib_reject_connection(struct dapl_cm_id *cm_handle); tduffy> tduffy> -u32 dapl_ib_setup_async_callback(DAPL_IA *ia_ptr, tduffy> +u32 dapl_ib_setup_async_callback(struct dapl_ia *ia_ptr, tduffy> DAPL_ASYNC_HANDLER_TYPE handler_type, tduffy> ib_async_handler_t callback, void *context); tduffy> tduffy> -u32 dapl_ib_cq_alloc(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr, int *cqlen, tduffy> - ib_comp_handler callback); tduffy> +u32 dapl_ib_cq_alloc(struct dapl_ia *ia_ptr, struct dapl_evd *evd_ptr, tduffy> + int *cqlen, ib_comp_handler callback); tduffy> tduffy> -u32 dapl_ib_cq_free(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr); tduffy> +u32 dapl_ib_cq_free(struct dapl_ia *ia_ptr, struct dapl_evd *evd_ptr); tduffy> tduffy> -u32 dapl_set_cq_notify(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr); tduffy> +u32 dapl_set_cq_notify(struct dapl_ia *ia_ptr, struct dapl_evd *evd_ptr); tduffy> tduffy> -u32 dapl_ib_cq_resize(DAPL_IA *ia_ptr, DAPL_EVD *evd_ptr, int *cqlen); tduffy> +u32 dapl_ib_cq_resize(struct dapl_ia *ia_ptr, struct dapl_evd *evd_ptr, tduffy> + int *cqlen); tduffy> tduffy> -u32 dapl_ib_pd_alloc(DAPL_IA *ia_ptr, DAPL_PZ *pz); tduffy> +u32 dapl_ib_pd_alloc(struct dapl_ia *ia_ptr, struct dapl_pz *pz); tduffy> tduffy> -u32 dapl_ib_pd_free(DAPL_PZ *pz); tduffy> +u32 dapl_ib_pd_free(struct dapl_pz *pz); tduffy> tduffy> -u32 dapl_ib_mr_register(DAPL_IA *ia_ptr, DAPL_LMR *lmr, void *virt_addr, tduffy> - u64 length, enum dat_mem_priv_flags privileges); tduffy> +u32 dapl_ib_mr_register(struct dapl_ia *ia_ptr, struct dapl_lmr *lmr, tduffy> + void *virt_addr, u64 length, tduffy> + enum dat_mem_priv_flags privileges); tduffy> tduffy> -u32 dapl_ib_mr_register_physical(DAPL_IA *ia_ptr, DAPL_LMR *lmr, tduffy> +u32 dapl_ib_mr_register_physical(struct dapl_ia *ia_ptr, struct dapl_lmr *lmr, tduffy> void *phys_addr, u64 length, tduffy> enum dat_mem_priv_flags privileges); tduffy> tduffy> -u32 dapl_ib_mr_deregister(DAPL_LMR *lmr); tduffy> +u32 dapl_ib_mr_deregister(struct dapl_lmr *lmr); tduffy> tduffy> -u32 dapl_ib_mr_register_shared(DAPL_IA *ia_ptr, DAPL_LMR *lmr, tduffy> +u32 dapl_ib_mr_register_shared(struct dapl_ia *ia_ptr, struct dapl_lmr *lmr, tduffy> enum dat_mem_priv_flags privileges); tduffy> tduffy> -u32 dapl_ib_mw_alloc(DAPL_RMR *rmr); tduffy> +u32 dapl_ib_mw_alloc(struct dapl_rmr *rmr); tduffy> tduffy> -u32 dapl_ib_mw_free(DAPL_RMR *rmr); tduffy> +u32 dapl_ib_mw_free(struct dapl_rmr *rmr); tduffy> tduffy> -u32 dapl_ib_mw_bind(DAPL_RMR *rmr, DAPL_LMR *lmr, DAPL_EP *ep, tduffy> - DAPL_COOKIE *cookie, u64 virtual_address, tduffy> - u64 length, enum dat_mem_priv_flags mem_priv, tduffy> - boolean_t is_signaled); tduffy> +u32 dapl_ib_mw_bind(struct dapl_rmr *rmr, struct dapl_lmr *lmr, tduffy> + struct dapl_ep *ep, struct dapl_cookie *cookie, tduffy> + u64 virtual_address, u64 length, tduffy> + enum dat_mem_priv_flags mem_priv, boolean_t is_signaled); tduffy> tduffy> -u32 dapl_ib_mw_unbind(DAPL_RMR *rmr, DAPL_EP *ep, DAPL_COOKIE *cookie, tduffy> - boolean_t is_signaled); tduffy> +u32 dapl_ib_mw_unbind(struct dapl_rmr *rmr, struct dapl_ep *ep, tduffy> + struct dapl_cookie *cookie, boolean_t is_signaled); tduffy> tduffy> -u32 dapl_ib_query_hca(DAPL_HCA *hca_ptr, struct dat_ia_attr *ia_attr, tduffy> +u32 dapl_ib_query_hca(struct dapl_hca *hca_ptr, struct dat_ia_attr *ia_attr, tduffy> struct dat_ep_attr *ep_attr, tduffy> struct sockaddr_in6 *ip_addr); tduffy> tduffy> -u32 dapl_ib_completion_poll(DAPL_HCA *hca_ptr, DAPL_EVD *evd_ptr, tduffy> +u32 dapl_ib_completion_poll(struct dapl_hca *hca_ptr, struct dapl_evd *evd_ptr, tduffy> struct ib_wc *cqe_ptr); tduffy> tduffy> -u32 dapl_ib_completion_notify(DAPL_EVD *evd_ptr); tduffy> +u32 dapl_ib_completion_notify(struct dapl_evd *evd_ptr); tduffy> tduffy> enum dat_dto_completion_status dapl_ib_get_dto_status(struct ib_wc *cqe_ptr); tduffy> tduffy> -void dapl_ib_reinit_ep(DAPL_EP * ep_ptr); tduffy> +void dapl_ib_reinit_ep(struct dapl_ep * ep_ptr); tduffy> tduffy> -void dapl_ib_disconnect_clean(DAPL_EP * ep_ptr, boolean_t passive); tduffy> +void dapl_ib_disconnect_clean(struct dapl_ep * ep_ptr, boolean_t passive); tduffy> tduffy> u32 dapl_ib_get_async_event(struct ib_event *cause_ptr, tduffy> enum dat_event_number *async_event); tduffy> Index: linux-kernel-resolv/dat-provider/dapl_evd_kquery.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_evd_kquery.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_evd_kquery.c (working copy) tduffy> @@ -40,7 +40,7 @@ tduffy> tduffy> u32 dapl_evd_kquery(DAT_EVD_HANDLE evd_handle, struct dat_evd_param *evd_param) tduffy> { tduffy> - DAPL_EVD *evd_ptr; tduffy> + struct dapl_evd *evd_ptr; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> if (NULL == evd_param) { tduffy> @@ -58,7 +58,7 @@ u32 dapl_evd_kquery(DAT_EVD_HANDLE evd_h tduffy> goto bail; tduffy> } tduffy> tduffy> - evd_ptr = (DAPL_EVD *) evd_handle; tduffy> + evd_ptr = (struct dapl_evd *)evd_handle; tduffy> /* tduffy> * We may be racing against the thread safe modify tduffy> * calls here (dat_evd_{enable,disable,{set,clear}_unwaitable}). tduffy> Index: linux-kernel-resolv/dat-provider/dapl_ia_close.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_ia_close.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_ia_close.c (working copy) tduffy> @@ -59,13 +59,13 @@ tduffy> */ tduffy> u32 dapl_ia_close(DAT_IA_HANDLE ia_handle, enum dat_close_flags ia_flags) tduffy> { tduffy> - DAPL_IA *ia_ptr; tduffy> + struct dapl_ia *ia_ptr; tduffy> u32 dat_status; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> "dapl_ia_close (%p, %d)\n", ia_handle, ia_flags); tduffy> tduffy> - ia_ptr = (DAPL_IA *) ia_handle; tduffy> + ia_ptr = (struct dapl_ia *)ia_handle; tduffy> tduffy> if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { tduffy> dat_status = tduffy> Index: linux-kernel-resolv/dat-provider/dapl_evd_qp_async_error_callb.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_evd_qp_async_error_callb.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_evd_qp_async_error_callb.c (working copy) tduffy> @@ -54,13 +54,13 @@ tduffy> tduffy> void dapl_evd_qp_async_error_callback(struct ib_event *cause, void *context) tduffy> { tduffy> - DAPL_EP *ep_ptr; tduffy> - DAPL_EVD *async_evd; tduffy> + struct dapl_ep *ep_ptr; tduffy> + struct dapl_evd *async_evd; tduffy> enum dat_event_number async_event; tduffy> u32 dat_status; tduffy> tduffy> - ep_ptr = (DAPL_EP *) context; tduffy> - async_evd = (DAPL_EVD *) ep_ptr->header.owner_ia->async_error_evd; tduffy> + ep_ptr = (struct dapl_ep *)context; tduffy> + async_evd = (struct dapl_evd *)ep_ptr->header.owner_ia->async_error_evd; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK | DAPL_DBG_TYPE_EXCEPTION, tduffy> "--> dapl_evd_qp_async_error_callback: ep %p qp %p (%x) state %d\n", tduffy> Index: linux-kernel-resolv/dat-provider/dapl_cno_util.h tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_cno_util.h (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_cno_util.h (working copy) tduffy> @@ -36,16 +36,11 @@ tduffy> tduffy> #include "dapl.h" tduffy> tduffy> -DAPL_CNO * tduffy> -dapl_cno_alloc(DAPL_IA *ia_ptr, const struct dat_upcall_object *upcall); tduffy> +struct dapl_cno *dapl_cno_alloc(struct dapl_ia *ia_ptr, tduffy> + const struct dat_upcall_object *upcall); tduffy> tduffy> -void tduffy> -dapl_cno_dealloc ( tduffy> - DAPL_CNO *cno_ptr) ; tduffy> +void dapl_cno_dealloc(struct dapl_cno *cno_ptr); tduffy> tduffy> -void tduffy> -dapl_cno_trigger( tduffy> - DAPL_CNO *cno_ptr, tduffy> - DAPL_EVD *evd_ptr); tduffy> +void dapl_cno_trigger(struct dapl_cno *cno_ptr, struct dapl_evd *evd_ptr); tduffy> tduffy> #endif tduffy> Index: linux-kernel-resolv/dat-provider/dapl_srq_set_lw.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_srq_set_lw.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_srq_set_lw.c (working copy) tduffy> @@ -65,7 +65,7 @@ tduffy> tduffy> u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, int low_watermark) tduffy> { tduffy> - DAPL_SRQ *srq_ptr; tduffy> + struct dapl_srq *srq_ptr; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_set_lw (%p, %d)\n", tduffy> @@ -77,7 +77,7 @@ u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_h tduffy> goto bail; tduffy> } tduffy> tduffy> - srq_ptr = (DAPL_SRQ *) srq_handle; tduffy> + srq_ptr = (struct dapl_srq *)srq_handle; tduffy> tduffy> /* XXX Put implementation here XXX */ tduffy> tduffy> Index: linux-kernel-resolv/dat-provider/dapl_evd_un_async_error_callb.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_evd_un_async_error_callb.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_evd_un_async_error_callb.c (working copy) tduffy> @@ -46,7 +46,7 @@ tduffy> tduffy> void dapl_evd_un_async_error_callback(struct ib_event *cause, void *context) tduffy> { tduffy> - DAPL_EVD *async_evd; tduffy> + struct dapl_evd *async_evd; tduffy> enum dat_event_number async_event; tduffy> u32 dat_status; tduffy> tduffy> @@ -59,7 +59,7 @@ void dapl_evd_un_async_error_callback(st tduffy> return; tduffy> } tduffy> tduffy> - async_evd = (DAPL_EVD *) context; tduffy> + async_evd = (struct dapl_evd *)context; tduffy> tduffy> dat_status = dapl_ib_get_async_event(cause, &async_event); tduffy> if (dat_status == DAT_SUCCESS) { tduffy> Index: linux-kernel-resolv/dat-provider/dapl_cr_accept.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_cr_accept.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_cr_accept.c (working copy) tduffy> @@ -42,9 +42,9 @@ tduffy> u32 dapl_cr_accept(DAT_CR_HANDLE cr_handle, DAT_EP_HANDLE ep_handle, tduffy> int private_data_size, const void *private_data) tduffy> { tduffy> - DAPL_EP *ep_ptr; tduffy> + struct dapl_ep *ep_ptr; tduffy> u32 dat_status; tduffy> - DAPL_CR *cr_ptr; tduffy> + struct dapl_cr *cr_ptr; tduffy> enum dat_ep_state entry_ep_state; tduffy> DAT_EP_HANDLE entry_ep_handle; tduffy> tduffy> @@ -58,7 +58,7 @@ u32 dapl_cr_accept(DAT_CR_HANDLE cr_hand tduffy> goto bail; tduffy> } tduffy> tduffy> - cr_ptr = (DAPL_CR *) cr_handle; tduffy> + cr_ptr = (struct dapl_cr *)cr_handle; tduffy> tduffy> /* tduffy> * Return an error if we have an ep_handle and the CR already has an tduffy> @@ -91,21 +91,21 @@ u32 dapl_cr_accept(DAT_CR_HANDLE cr_hand tduffy> */ tduffy> if (ep_handle == NULL) { tduffy> ep_handle = cr_ptr->param.local_ep_handle; tduffy> - if ((((DAPL_EP *) ep_handle)->param.ep_state != tduffy> + if ((((struct dapl_ep *)ep_handle)->param.ep_state != tduffy> DAT_EP_STATE_TENTATIVE_CONNECTION_PENDING) tduffy> - && (((DAPL_EP *) ep_handle)->param.ep_state != tduffy> + && (((struct dapl_ep *)ep_handle)->param.ep_state != tduffy> DAT_EP_STATE_PASSIVE_CONNECTION_PENDING)) { tduffy> return DAT_INVALID_STATE; tduffy> } tduffy> } else { tduffy> /* ensure this EP isn't connected or in use */ tduffy> - if (((DAPL_EP *) ep_handle)->param.ep_state != tduffy> + if (((struct dapl_ep *)ep_handle)->param.ep_state != tduffy> DAT_EP_STATE_UNCONNECTED) { tduffy> return DAT_INVALID_STATE; tduffy> } tduffy> } tduffy> tduffy> - ep_ptr = (DAPL_EP *) ep_handle; tduffy> + ep_ptr = (struct dapl_ep *)ep_handle; tduffy> tduffy> /* tduffy> * Verify the attributes of the EP handle before we connect it. Test tduffy> @@ -124,7 +124,7 @@ u32 dapl_cr_accept(DAT_CR_HANDLE cr_hand tduffy> /* test connect handle */ tduffy> || ep_ptr->param.connect_evd_handle == NULL tduffy> || DAPL_BAD_HANDLE(ep_ptr->param.connect_evd_handle, DAPL_MAGIC_EVD) tduffy> - || !(((DAPL_EVD *) ep_ptr->param.connect_evd_handle)-> tduffy> + || !(((struct dapl_evd *)ep_ptr->param.connect_evd_handle)-> tduffy> evd_flags & DAT_EVD_CONNECTION_FLAG) tduffy> /* test optional completion handles */ tduffy> || (ep_ptr->param.recv_evd_handle != DAT_HANDLE_NULL && tduffy> Index: linux-kernel-resolv/dat-provider/dapl_rsp_create.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_rsp_create.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_rsp_create.c (working copy) tduffy> @@ -72,14 +72,14 @@ u32 dapl_rsp_create(DAT_IA_HANDLE ia_han tduffy> DAT_EP_HANDLE ep_handle, DAT_EVD_HANDLE evd_handle, tduffy> DAT_RSP_HANDLE *rsp_handle) tduffy> { tduffy> - DAPL_IA *ia_ptr; tduffy> - DAPL_SP *sp_ptr; tduffy> - DAPL_EVD *evd_ptr; tduffy> - DAPL_EP *ep_ptr; tduffy> + struct dapl_ia *ia_ptr; tduffy> + struct dapl_sp *sp_ptr; tduffy> + struct dapl_evd *evd_ptr; tduffy> + struct dapl_ep *ep_ptr; tduffy> boolean_t sp_found; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> - ia_ptr = (DAPL_IA *) ia_handle; tduffy> + ia_ptr = (struct dapl_ia *)ia_handle; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_CM, tduffy> ">>> dapl_rsp_free conn_qual: %x EP: %p\n", tduffy> @@ -106,14 +106,14 @@ u32 dapl_rsp_create(DAT_IA_HANDLE ia_han tduffy> goto bail; tduffy> } tduffy> tduffy> - ep_ptr = (DAPL_EP *) ep_handle; tduffy> + ep_ptr = (struct dapl_ep *)ep_handle; tduffy> if (ep_ptr->param.ep_state != DAT_EP_STATE_UNCONNECTED) { tduffy> dat_status = tduffy> DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep_ptr)); tduffy> goto bail; tduffy> } tduffy> tduffy> - evd_ptr = (DAPL_EVD *) evd_handle; tduffy> + evd_ptr = (struct dapl_evd *)evd_handle; tduffy> if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { tduffy> dat_status = tduffy> DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); tduffy> @@ -147,7 +147,7 @@ u32 dapl_rsp_create(DAT_IA_HANDLE ia_han tduffy> /* tduffy> * Take a reference on the EVD handle tduffy> */ tduffy> - atomic_inc(&((DAPL_EVD *) evd_handle)->evd_ref_count); tduffy> + atomic_inc(&((struct dapl_evd *)evd_handle)->evd_ref_count); tduffy> tduffy> /* tduffy> * Update the EP state indicating the provider now owns it tduffy> @@ -174,7 +174,7 @@ u32 dapl_rsp_create(DAT_IA_HANDLE ia_han tduffy> * wrong! Decrements the EVD refcount & release it. Set tduffy> * the state to FREE, so we know the call failed. tduffy> */ tduffy> - atomic_dec(&((DAPL_EVD *) evd_handle)-> tduffy> + atomic_dec(&((struct dapl_evd *)evd_handle)-> tduffy> evd_ref_count); tduffy> sp_ptr->evd_handle = NULL; tduffy> sp_ptr->state = DAPL_SP_STATE_FREE; tduffy> Index: linux-kernel-resolv/dat-provider/dapl_srq_post_recv.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_srq_post_recv.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_srq_post_recv.c (working copy) tduffy> @@ -71,8 +71,8 @@ u32 dapl_srq_post_recv(DAT_SRQ_HANDLE sr tduffy> struct dat_lmr_triplet *local_iov, tduffy> DAT_DTO_COOKIE user_cookie) tduffy> { tduffy> - DAPL_SRQ *srq_ptr; tduffy> - DAPL_COOKIE *cookie; tduffy> + struct dapl_srq *srq_ptr; tduffy> + struct dapl_cookie *cookie; tduffy> u32 dat_status; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> @@ -85,7 +85,7 @@ u32 dapl_srq_post_recv(DAT_SRQ_HANDLE sr tduffy> goto bail; tduffy> } tduffy> tduffy> - srq_ptr = (DAPL_SRQ *) srq_handle; tduffy> + srq_ptr = (struct dapl_srq *)srq_handle; tduffy> tduffy> /* tduffy> * Synchronization ok since this buffer is only used for receive tduffy> Index: linux-kernel-resolv/dat-provider/dapl_lmr_free.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_lmr_free.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_lmr_free.c (working copy) tduffy> @@ -59,7 +59,7 @@ tduffy> tduffy> u32 dapl_lmr_free(DAT_LMR_HANDLE lmr_handle) tduffy> { tduffy> - DAPL_LMR *lmr; tduffy> + struct dapl_lmr *lmr; tduffy> u32 dat_status; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_free (%p)\n", lmr_handle); tduffy> @@ -70,36 +70,34 @@ u32 dapl_lmr_free(DAT_LMR_HANDLE lmr_han tduffy> goto bail; tduffy> } tduffy> tduffy> - lmr = (DAPL_LMR *) lmr_handle; tduffy> + lmr = (struct dapl_lmr *)lmr_handle; tduffy> tduffy> switch (lmr->param.mem_type) { tduffy> case DAT_MEM_TYPE_PHYSICAL: tduffy> case DAT_MEM_TYPE_VIRTUAL: tduffy> case DAT_MEM_TYPE_LMR: tduffy> { tduffy> - DAPL_PZ *pz; tduffy> + struct dapl_pz *pz; tduffy> tduffy> - if (0 != atomic_read(&lmr->lmr_ref_count)) { tduffy> + if (0 != atomic_read(&lmr->lmr_ref_count)) tduffy> return DAT_INVALID_STATE; tduffy> - } tduffy> tduffy> dat_status = tduffy> dapl_hash_remove(lmr->header.owner_ia->hca_ptr-> tduffy> lmr_hash_table, tduffy> lmr->param.lmr_context, NULL); tduffy> - if (dat_status != DAT_SUCCESS) { tduffy> + if (dat_status != DAT_SUCCESS) tduffy> goto bail; tduffy> - } tduffy> tduffy> dat_status = dapl_ib_mr_deregister(lmr); tduffy> tduffy> if (dat_status == DAT_SUCCESS) { tduffy> tduffy> - pz = (DAPL_PZ *) lmr->param.pz_handle; tduffy> + pz = (struct dapl_pz *)lmr->param.pz_handle; tduffy> atomic_dec(&pz->pz_ref_count); tduffy> tduffy> dapl_lmr_dealloc(lmr); tduffy> - } else { tduffy> + } else tduffy> /* tduffy> * Deregister failed; put it back in the tduffy> * hash table. tduffy> @@ -107,7 +105,6 @@ u32 dapl_lmr_free(DAT_LMR_HANDLE lmr_han tduffy> dapl_hash_insert(lmr->header.owner_ia-> tduffy> hca_ptr->lmr_hash_table, tduffy> lmr->param.lmr_context, lmr); tduffy> - } tduffy> tduffy> break; tduffy> } tduffy> @@ -124,6 +121,6 @@ u32 dapl_lmr_free(DAT_LMR_HANDLE lmr_han tduffy> break; tduffy> } tduffy> } tduffy> - bail: tduffy> +bail: tduffy> return dat_status; tduffy> } tduffy> Index: linux-kernel-resolv/dat-provider/dapl_get_handle_type.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_get_handle_type.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_get_handle_type.c (working copy) tduffy> @@ -59,9 +59,9 @@ tduffy> u32 dapl_get_handle_type(DAT_HANDLE dat_handle, enum dat_handle_type *type) tduffy> { tduffy> u32 dat_status = DAT_SUCCESS; tduffy> - DAPL_HEADER *header; tduffy> + struct dapl_header *header; tduffy> tduffy> - header = (DAPL_HEADER *) dat_handle; tduffy> + header = (struct dapl_header *)dat_handle; tduffy> if (((header) == NULL) || tduffy> ((unsigned long)(header) & 3) || tduffy> (header->magic != DAPL_MAGIC_IA && tduffy> Index: linux-kernel-resolv/dat-provider/dapl_rmr_util.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_rmr_util.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_rmr_util.c (working copy) tduffy> @@ -28,9 +28,9 @@ tduffy> #include "dapl_rmr_util.h" tduffy> #include "dapl_ia_util.h" tduffy> tduffy> -DAPL_RMR *dapl_rmr_alloc(DAPL_PZ * pz) tduffy> +struct dapl_rmr *dapl_rmr_alloc(struct dapl_pz *pz) tduffy> { tduffy> - DAPL_RMR *rmr; tduffy> + struct dapl_rmr *rmr; tduffy> tduffy> /* Allocate LMR */ tduffy> rmr = kmalloc(sizeof *rmr, GFP_ATOMIC); tduffy> @@ -70,9 +70,10 @@ DAPL_RMR *dapl_rmr_alloc(DAPL_PZ * pz) tduffy> return rmr; tduffy> } tduffy> tduffy> -void dapl_rmr_dealloc(DAPL_RMR * rmr) tduffy> +void dapl_rmr_dealloc(struct dapl_rmr *rmr) tduffy> { tduffy> - rmr->header.magic = DAPL_MAGIC_INVALID; /* reset magic to prevent reuse */ tduffy> + /* reset magic to prevent reuse */ tduffy> + rmr->header.magic = DAPL_MAGIC_INVALID; tduffy> tduffy> dapl_ia_unlink_rmr(rmr->header.owner_ia, rmr); tduffy> /* no need to destroy rmr->header.lock */ tduffy> Index: linux-kernel-resolv/dat-provider/dapl_ep_util.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_ep_util.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_ep_util.c (working copy) tduffy> @@ -55,9 +55,10 @@ tduffy> * none tduffy> * tduffy> */ tduffy> -DAPL_EP *dapl_ep_alloc(DAPL_IA * ia_ptr, const struct dat_ep_attr *ep_attr) tduffy> +struct dapl_ep *dapl_ep_alloc(struct dapl_ia *ia_ptr, tduffy> + const struct dat_ep_attr *ep_attr) tduffy> { tduffy> - DAPL_EP *ep_ptr; tduffy> + struct dapl_ep *ep_ptr; tduffy> tduffy> /* Allocate EP */ tduffy> ep_ptr = kmalloc(sizeof *ep_ptr + sizeof (struct sockaddr), GFP_ATOMIC); tduffy> @@ -94,8 +95,7 @@ DAPL_EP *dapl_ep_alloc(DAPL_IA * ia_ptr, tduffy> (ep_ptr->param.ep_attr.max_request_dtos / 256) + 1; tduffy> ep_ptr->param.ep_attr.max_recv_dtos = tduffy> (ep_ptr->param.ep_attr.max_recv_dtos / 256) + 1; tduffy> - } tduffy> - else tduffy> + } else tduffy> ep_ptr->param.ep_attr = *ep_attr; tduffy> tduffy> ep_ptr->qp_handle = NULL; tduffy> @@ -139,7 +139,7 @@ DAPL_EP *dapl_ep_alloc(DAPL_IA * ia_ptr, tduffy> goto bail; tduffy> } tduffy> tduffy> - bail: tduffy> +bail: tduffy> return ep_ptr; tduffy> } tduffy> tduffy> @@ -158,26 +158,24 @@ DAPL_EP *dapl_ep_alloc(DAPL_IA * ia_ptr, tduffy> * none tduffy> * tduffy> */ tduffy> -void dapl_ep_dealloc(DAPL_EP * ep_ptr) tduffy> +void dapl_ep_dealloc(struct dapl_ep *ep_ptr) tduffy> { tduffy> dapl_os_assert(ep_ptr->header.magic == DAPL_MAGIC_EP); tduffy> tduffy> - ep_ptr->header.magic = DAPL_MAGIC_INVALID; /* reset magic to prevent reuse */ tduffy> + /* reset magic to prevent reuse */ tduffy> + ep_ptr->header.magic = DAPL_MAGIC_INVALID; tduffy> tduffy> dapl_cb_free(&ep_ptr->req_buffer); tduffy> dapl_cb_free(&ep_ptr->recv_buffer); tduffy> tduffy> - if (NULL != ep_ptr->recv_iov) { tduffy> + if (ep_ptr->recv_iov) tduffy> kfree(ep_ptr->recv_iov); tduffy> - } tduffy> tduffy> - if (NULL != ep_ptr->send_iov) { tduffy> + if (ep_ptr->send_iov) tduffy> kfree(ep_ptr->send_iov); tduffy> - } tduffy> tduffy> - if (NULL != ep_ptr->cxn_timer) { tduffy> + if (ep_ptr->cxn_timer) tduffy> kfree(ep_ptr->cxn_timer); tduffy> - } tduffy> tduffy> kfree(ep_ptr); tduffy> } tduffy> @@ -194,9 +192,8 @@ u32 dapl_ep_check_recv_completion_flags( tduffy> * an EVD completes, even though the dapl handler is invoked. tduffy> */ tduffy> tduffy> - if (flags & DAT_COMPLETION_SUPPRESS_FLAG) { tduffy> + if (flags & DAT_COMPLETION_SUPPRESS_FLAG) tduffy> return DAT_INVALID_PARAMETER; tduffy> - } tduffy> tduffy> return DAT_SUCCESS; tduffy> } tduffy> @@ -208,8 +205,8 @@ u32 dapl_ep_post_send_req(DAT_EP_HANDLE tduffy> enum dat_completion_flags completion_flags, tduffy> DAPL_DTO_TYPE dto_type, enum ib_wr_opcode op_type) tduffy> { tduffy> - DAPL_EP *ep_ptr; tduffy> - DAPL_COOKIE *cookie; tduffy> + struct dapl_ep *ep_ptr; tduffy> + struct dapl_cookie *cookie; tduffy> u32 dat_status; tduffy> tduffy> if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) { tduffy> @@ -218,7 +215,7 @@ u32 dapl_ep_post_send_req(DAT_EP_HANDLE tduffy> goto bail; tduffy> } tduffy> tduffy> - ep_ptr = (DAPL_EP *) ep_handle; tduffy> + ep_ptr = (struct dapl_ep *)ep_handle; tduffy> tduffy> /* tduffy> * Synchronization ok since this buffer is only used for send tduffy> @@ -226,9 +223,8 @@ u32 dapl_ep_post_send_req(DAT_EP_HANDLE tduffy> */ tduffy> dat_status = dapl_dto_cookie_alloc(&ep_ptr->req_buffer, dto_type, tduffy> user_cookie, &cookie); tduffy> - if (dat_status != DAT_SUCCESS) { tduffy> + if (dat_status != DAT_SUCCESS) tduffy> goto bail; tduffy> - } tduffy> tduffy> /* tduffy> * Take reference before posting to avoid race conditions with tduffy> @@ -247,7 +243,7 @@ u32 dapl_ep_post_send_req(DAT_EP_HANDLE tduffy> dapl_cookie_dealloc(&ep_ptr->req_buffer, cookie); tduffy> } tduffy> tduffy> - bail: tduffy> +bail: tduffy> return dat_status; tduffy> } tduffy> tduffy> @@ -259,11 +255,11 @@ u32 dapl_ep_post_send_req(DAT_EP_HANDLE tduffy> */ tduffy> void dapl_ep_timeout(void *arg) tduffy> { tduffy> - DAPL_EP *ep_ptr; tduffy> + struct dapl_ep *ep_ptr; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_CM, "--> dapl_ep_timeout! ep %lx\n", arg); tduffy> tduffy> - ep_ptr = (DAPL_EP *) arg; tduffy> + ep_ptr = (struct dapl_ep *)arg; tduffy> tduffy> /* reset the EP state */ tduffy> ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; tduffy> @@ -271,7 +267,7 @@ void dapl_ep_timeout(void *arg) tduffy> /* Clean up the EP and put the underlying QP into the ERROR state. */ tduffy> dapl_ib_disconnect_clean(ep_ptr, TRUE); tduffy> tduffy> - (void)dapl_evd_post_connection_event((DAPL_EVD *) ep_ptr->param. tduffy> + (void)dapl_evd_post_connection_event((struct dapl_evd *)ep_ptr->param. tduffy> connect_evd_handle, tduffy> DAT_CONNECTION_EVENT_TIMED_OUT, tduffy> (DAT_HANDLE) ep_ptr, 0, NULL); tduffy> @@ -283,7 +279,7 @@ void dapl_ep_timeout(void *arg) tduffy> * Return the INVALID_STATE connection subtype associated with an tduffy> * INVALID_STATE on an EP. Strictly for error reporting. tduffy> */ tduffy> -enum dat_return_subtype dapl_ep_state_subtype(DAPL_EP * ep_ptr) tduffy> +enum dat_return_subtype dapl_ep_state_subtype(struct dapl_ep *ep_ptr) tduffy> { tduffy> enum dat_return_subtype dat_status; tduffy> tduffy> Index: linux-kernel-resolv/dat-provider/dapl_rmr_util.h tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_rmr_util.h (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_rmr_util.h (working copy) tduffy> @@ -39,15 +39,9 @@ tduffy> tduffy> #include "dapl_mr_util.h" tduffy> tduffy> -/********************************************************************* tduffy> - * * tduffy> - * Function Prototypes * tduffy> - * * tduffy> - *********************************************************************/ tduffy> +extern struct dapl_rmr *dapl_rmr_alloc(struct dapl_pz *pz); tduffy> tduffy> -extern DAPL_RMR *dapl_rmr_alloc(DAPL_PZ * pz); tduffy> - tduffy> -extern void dapl_rmr_dealloc(DAPL_RMR * rmr); tduffy> +extern void dapl_rmr_dealloc(struct dapl_rmr *rmr); tduffy> tduffy> static inline boolean_t tduffy> dapl_rmr_validate_completion_flag(enum dat_completion_flags mask, tduffy> Index: linux-kernel-resolv/dat-provider/dapl_lmr_sync_rdma_write.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_lmr_sync_rdma_write.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_lmr_sync_rdma_write.c (working copy) tduffy> @@ -45,14 +45,14 @@ u32 dapl_lmr_sync_rdma_write(DAT_IA_HAND tduffy> const struct dat_lmr_triplet *local_segments, tduffy> u64 num_segments) tduffy> { tduffy> - DAPL_IA *ia_ptr; tduffy> + struct dapl_ia *ia_ptr; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> "dat_lmr_sync_rdma_write (%p, %p, %ld)\n", tduffy> ia_handle, local_segments, num_segments); tduffy> tduffy> - ia_ptr = (DAPL_IA *) ia_handle; tduffy> + ia_ptr = (struct dapl_ia *)ia_handle; tduffy> tduffy> if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { tduffy> dat_status = tduffy> Index: linux-kernel-resolv/dat-provider/dapl_rsp_free.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_rsp_free.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_rsp_free.c (working copy) tduffy> @@ -60,12 +60,12 @@ tduffy> */ tduffy> u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_handle) tduffy> { tduffy> - DAPL_IA *ia_ptr; tduffy> - DAPL_SP *sp_ptr; tduffy> - DAPL_EP *ep_ptr; tduffy> + struct dapl_ia *ia_ptr; tduffy> + struct dapl_sp *sp_ptr; tduffy> + struct dapl_ep *ep_ptr; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> - sp_ptr = (DAPL_SP *) rsp_handle; tduffy> + sp_ptr = (struct dapl_sp *)rsp_handle; tduffy> /* tduffy> * Verify handle tduffy> */ tduffy> @@ -76,7 +76,7 @@ u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_han tduffy> goto bail; tduffy> } tduffy> tduffy> - /* ia_ptr = (DAPL_IA *)sp_ptr->header.owner_ia; */ tduffy> + /* ia_ptr = (struct dapl_ia *)sp_ptr->header.owner_ia; */ tduffy> ia_ptr = sp_ptr->header.owner_ia; tduffy> tduffy> /* tduffy> @@ -90,17 +90,16 @@ u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_han tduffy> * Make sure we don't leave a dangling EP. If the state is still tduffy> * RESERVED then the RSP still owns it. tduffy> */ tduffy> - ep_ptr = (DAPL_EP *) sp_ptr->ep_handle; tduffy> - if (ep_ptr != NULL && ep_ptr->param.ep_state == DAT_EP_STATE_RESERVED) { tduffy> + ep_ptr = (struct dapl_ep *)sp_ptr->ep_handle; tduffy> + if (ep_ptr != NULL && ep_ptr->param.ep_state == DAT_EP_STATE_RESERVED) tduffy> ep_ptr->param.ep_state = DAT_EP_STATE_UNCONNECTED; tduffy> - } tduffy> sp_ptr->ep_handle = NULL; tduffy> tduffy> /* Release reference on EVD. If an error was encountered in a previous tduffy> * free the evd_handle will be NULL tduffy> */ tduffy> if (sp_ptr->evd_handle) { tduffy> - atomic_dec(&((DAPL_EVD *) sp_ptr->evd_handle)-> tduffy> + atomic_dec(&((struct dapl_evd *)sp_ptr->evd_handle)-> tduffy> evd_ref_count); tduffy> sp_ptr->evd_handle = NULL; tduffy> } tduffy> @@ -138,6 +137,6 @@ u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_han tduffy> sp_ptr->header.flags); tduffy> } tduffy> tduffy> - bail: tduffy> +bail: tduffy> return dat_status; tduffy> } tduffy> Index: linux-kernel-resolv/dat-provider/dapl_srq_free.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_srq_free.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_srq_free.c (working copy) tduffy> @@ -61,14 +61,14 @@ tduffy> */ tduffy> u32 dapl_srq_free(DAT_SRQ_HANDLE srq_handle) tduffy> { tduffy> - DAPL_SRQ *srq_ptr; tduffy> - DAPL_IA *ia_ptr; tduffy> + struct dapl_srq *srq_ptr; tduffy> + struct dapl_ia *ia_ptr; tduffy> struct dat_srq_param *param; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_free (%p)\n", srq_handle); tduffy> tduffy> - srq_ptr = (DAPL_SRQ *) srq_handle; tduffy> + srq_ptr = (struct dapl_srq *)srq_handle; tduffy> param = &srq_ptr->param; tduffy> tduffy> /* tduffy> Index: linux-kernel-resolv/dat-provider/dapl_ep_util.h tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_ep_util.h (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_ep_util.h (working copy) tduffy> @@ -42,9 +42,10 @@ tduffy> tduffy> /* function prototypes */ tduffy> tduffy> -extern DAPL_EP *dapl_ep_alloc(DAPL_IA * ia, const struct dat_ep_attr *ep_attr); tduffy> +extern struct dapl_ep *dapl_ep_alloc(struct dapl_ia *ia, tduffy> + const struct dat_ep_attr *ep_attr); tduffy> tduffy> -extern void dapl_ep_dealloc(DAPL_EP * ep_ptr); tduffy> +extern void dapl_ep_dealloc(struct dapl_ep *ep_ptr); tduffy> tduffy> extern u32 dapl_ep_check_recv_completion_flags(enum dat_completion_flags flags); tduffy> tduffy> @@ -58,6 +59,6 @@ extern u32 dapl_ep_post_send_req(DAT_EP_ tduffy> tduffy> void dapl_ep_timeout(void *arg); tduffy> tduffy> -enum dat_return_subtype dapl_ep_state_subtype(DAPL_EP * ep_ptr); tduffy> +enum dat_return_subtype dapl_ep_state_subtype(struct dapl_ep *ep_ptr); tduffy> tduffy> #endif /* DAPL_EP_UTIL_H */ tduffy> Index: linux-kernel-resolv/dat-provider/dapl_ep_create_with_srq.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_ep_create_with_srq.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_ep_create_with_srq.c (working copy) tduffy> @@ -77,8 +77,8 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL tduffy> const struct dat_ep_attr *ep_attr, tduffy> DAT_EP_HANDLE *ep_handle) tduffy> { tduffy> - DAPL_IA *ia_ptr; tduffy> - DAPL_EP *ep_ptr; tduffy> + struct dapl_ia *ia_ptr; tduffy> + struct dapl_ep *ep_ptr; tduffy> struct dat_ep_attr *ep_attr_limit; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> @@ -90,7 +90,7 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL tduffy> request_evd_handle, tduffy> connect_evd_handle, srq_handle, ep_attr, ep_handle); tduffy> tduffy> - ia_ptr = (DAPL_IA *) ia_handle; tduffy> + ia_ptr = (struct dapl_ia *)ia_handle; tduffy> tduffy> /* tduffy> * Verify parameters tduffy> @@ -116,7 +116,7 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL tduffy> /* If connect handle is !NULL verify handle is good */ tduffy> if (connect_evd_handle != DAT_HANDLE_NULL && tduffy> (DAPL_BAD_HANDLE(connect_evd_handle, DAPL_MAGIC_EVD) || tduffy> - !(((DAPL_EVD *) connect_evd_handle)-> tduffy> + !(((struct dapl_evd *)connect_evd_handle)-> tduffy> evd_flags & DAT_EVD_CONNECTION_FLAG))) { tduffy> dat_status = tduffy> DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CONN); tduffy> @@ -125,7 +125,7 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL tduffy> /* If recv_evd is !NULL, verify handle is good and flags are valid */ tduffy> if (recv_evd_handle != DAT_HANDLE_NULL && tduffy> (DAPL_BAD_HANDLE(recv_evd_handle, DAPL_MAGIC_EVD) || tduffy> - !(((DAPL_EVD *) recv_evd_handle)->evd_flags & DAT_EVD_DTO_FLAG))) { tduffy> + !(((struct dapl_evd *)recv_evd_handle)->evd_flags & DAT_EVD_DTO_FLAG))) { tduffy> dat_status = tduffy> DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_RECV); tduffy> goto bail; tduffy> @@ -134,7 +134,7 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL tduffy> /* If req_evd is !NULL, verify handle is good and flags are valid */ tduffy> if (request_evd_handle != DAT_HANDLE_NULL && tduffy> (DAPL_BAD_HANDLE(request_evd_handle, DAPL_MAGIC_EVD) || tduffy> - !(((DAPL_EVD *) request_evd_handle)-> tduffy> + !(((struct dapl_evd *)request_evd_handle)-> tduffy> evd_flags & DAT_EVD_DTO_FLAG))) { tduffy> dat_status = tduffy> DAT_ERROR(DAT_INVALID_HANDLE, tduffy> @@ -253,7 +253,7 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL tduffy> */ tduffy> if (pz_handle != DAT_HANDLE_NULL) { tduffy> /* Take a reference on the PZ handle */ tduffy> - atomic_inc(&((DAPL_PZ *) pz_handle)->pz_ref_count); tduffy> + atomic_inc(&((struct dapl_pz *)pz_handle)->pz_ref_count); tduffy> tduffy> /* tduffy> * Get a QP from the IB provider tduffy> @@ -261,14 +261,12 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL tduffy> dat_status = dapl_ib_qp_alloc(ia_ptr, ep_ptr, ep_ptr); tduffy> tduffy> if (dat_status != DAT_SUCCESS) { tduffy> - atomic_dec(&((DAPL_PZ *) pz_handle)-> tduffy> - pz_ref_count); tduffy> + atomic_dec(&((struct dapl_pz *)pz_handle)->pz_ref_count); tduffy> dapl_ep_dealloc(ep_ptr); tduffy> goto bail; tduffy> } tduffy> - } else { tduffy> + } else tduffy> ep_ptr->qp_state = DAPL_QP_STATE_UNATTACHED; tduffy> - } tduffy> tduffy> /* tduffy> * Update ref counts. See the spec where the endpoint marks tduffy> @@ -278,16 +276,14 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL tduffy> * tduffy> * N.B. This should really be done by a util routine. tduffy> */ tduffy> - atomic_inc(&((DAPL_EVD *) connect_evd_handle)->evd_ref_count); tduffy> + atomic_inc(&((struct dapl_evd *)connect_evd_handle)->evd_ref_count); tduffy> /* Optional handles */ tduffy> - if (recv_evd_handle != DAT_HANDLE_NULL) { tduffy> - atomic_inc(&((DAPL_EVD *) recv_evd_handle)-> tduffy> + if (recv_evd_handle != DAT_HANDLE_NULL) tduffy> + atomic_inc(&((struct dapl_evd *)recv_evd_handle)-> tduffy> evd_ref_count); tduffy> - } tduffy> - if (request_evd_handle != DAT_HANDLE_NULL) { tduffy> - atomic_inc(&((DAPL_EVD *) request_evd_handle)-> tduffy> + if (request_evd_handle != DAT_HANDLE_NULL) tduffy> + atomic_inc(&((struct dapl_evd *)request_evd_handle)-> tduffy> evd_ref_count); tduffy> - } tduffy> tduffy> /* Link it onto the IA */ tduffy> dapl_ia_link_ep(ia_ptr, ep_ptr); tduffy> @@ -296,6 +292,6 @@ u32 dapl_ep_create_with_srq(DAT_IA_HANDL tduffy> tduffy> #endif /* notdef */ tduffy> tduffy> - bail: tduffy> +bail: tduffy> return dat_status; tduffy> } tduffy> Index: linux-kernel-resolv/dat-provider/dapl_ia_open.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_ia_open.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_ia_open.c (working copy) tduffy> @@ -57,12 +57,9 @@ u32 dapl_ia_open(const char *name, int a tduffy> { tduffy> u32 dat_status = DAT_SUCCESS; tduffy> struct dat_provider *provider; tduffy> - DAPL_HCA *hca_ptr; tduffy> - DAPL_IA *ia_ptr; tduffy> - DAPL_EVD *evd_ptr; tduffy> - tduffy> - hca_ptr = NULL; tduffy> - ia_ptr = NULL; tduffy> + struct dapl_hca *hca_ptr = NULL; tduffy> + struct dapl_ia *ia_ptr = NULL; tduffy> + struct dapl_evd *evd_ptr; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> "dapl_ia_open (%s, %d, %p, %p)\n", tduffy> @@ -87,7 +84,7 @@ u32 dapl_ia_open(const char *name, int a tduffy> /* initialize the caller's param */ tduffy> *ia_handle_ptr = DAT_HANDLE_NULL; tduffy> tduffy> - hca_ptr = (DAPL_HCA *) provider->extension; tduffy> + hca_ptr = (struct dapl_hca *)provider->extension; tduffy> atomic_inc(&hca_ptr->handle_ref_count); tduffy> tduffy> /* Allocate and initialize ia structure */ tduffy> @@ -103,7 +100,7 @@ u32 dapl_ia_open(const char *name, int a tduffy> * use the one passed in (if non-NULL) or create one tduffy> */ tduffy> tduffy> - evd_ptr = (DAPL_EVD *) * async_evd_handle_ptr; tduffy> + evd_ptr = (struct dapl_evd *)*async_evd_handle_ptr; tduffy> if (evd_ptr) { tduffy> if (DAPL_BAD_HANDLE(evd_ptr, DAPL_MAGIC_EVD) || tduffy> !(evd_ptr->evd_flags & DAT_EVD_ASYNC_FLAG)) { tduffy> Index: linux-kernel-resolv/dat-provider/dapl_evd_util.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_evd_util.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_evd_util.c (working copy) tduffy> @@ -41,7 +41,7 @@ tduffy> tduffy> static inline void dapl_evd_eh_print_cqe(struct ib_wc *cqe); tduffy> tduffy> -u32 dapl_evd_event_alloc(DAPL_EVD *evd_ptr, int qlen); tduffy> +u32 dapl_evd_event_alloc(struct dapl_evd *evd_ptr, int qlen); tduffy> tduffy> /* tduffy> * actually create the evd. this is called after all parameter checking tduffy> @@ -49,11 +49,11 @@ u32 dapl_evd_event_alloc(DAPL_EVD *evd_p tduffy> * to create the default async evd. tduffy> */ tduffy> tduffy> -u32 dapl_evd_internal_create(DAPL_IA *ia_ptr, DAPL_CNO *cno_ptr, tduffy> +u32 dapl_evd_internal_create(struct dapl_ia *ia_ptr, struct dapl_cno *cno_ptr, tduffy> int min_qlen, enum dat_evd_flags evd_flags, tduffy> - DAPL_EVD **evd_ptr_ptr) tduffy> + struct dapl_evd **evd_ptr_ptr) tduffy> { tduffy> - DAPL_EVD *evd_ptr; tduffy> + struct dapl_evd *evd_ptr; tduffy> int cq_len; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> @@ -117,10 +117,11 @@ bail: tduffy> /* tduffy> * alloc and initialize an EVD struct tduffy> */ tduffy> -DAPL_EVD *dapl_evd_alloc(DAPL_IA * ia_ptr, DAPL_CNO * cno_ptr, tduffy> - enum dat_evd_flags evd_flags, int qlen) tduffy> +struct dapl_evd *dapl_evd_alloc(struct dapl_ia *ia_ptr, tduffy> + struct dapl_cno *cno_ptr, tduffy> + enum dat_evd_flags evd_flags, int qlen) tduffy> { tduffy> - DAPL_EVD *evd_ptr; tduffy> + struct dapl_evd *evd_ptr; tduffy> tduffy> /* Allocate EVD */ tduffy> evd_ptr = kmalloc(sizeof *evd_ptr, GFP_ATOMIC); tduffy> @@ -161,7 +162,7 @@ bail: tduffy> /* tduffy> * alloc events into an EVD. tduffy> */ tduffy> -u32 dapl_evd_event_alloc(DAPL_EVD *evd_ptr, int qlen) tduffy> +u32 dapl_evd_event_alloc(struct dapl_evd *evd_ptr, int qlen) tduffy> { tduffy> struct dat_event *event_ptr; tduffy> int i; tduffy> @@ -201,7 +202,7 @@ bail: tduffy> /* tduffy> * realloc events into an EVD. tduffy> */ tduffy> -u32 dapl_evd_event_realloc(DAPL_EVD *evd_ptr, int qlen) tduffy> +u32 dapl_evd_event_realloc(struct dapl_evd *evd_ptr, int qlen) tduffy> { tduffy> struct dat_event *events; tduffy> int i, old_qlen, diff; tduffy> @@ -251,10 +252,10 @@ bail: tduffy> * will clean up all of the internal data structures and report the tduffy> * error. tduffy> */ tduffy> -u32 dapl_evd_dealloc(DAPL_EVD *evd_ptr) tduffy> +u32 dapl_evd_dealloc(struct dapl_evd *evd_ptr) tduffy> { tduffy> u32 dat_status = DAT_SUCCESS; tduffy> - DAPL_IA *ia_ptr; tduffy> + struct dapl_ia *ia_ptr; tduffy> tduffy> dapl_os_assert(evd_ptr->header.magic == DAPL_MAGIC_EVD); tduffy> dapl_os_assert(atomic_read(&evd_ptr->evd_ref_count) == 0); tduffy> @@ -363,7 +364,7 @@ void dapl_evd_eh_print_cqe(struct ib_wc tduffy> * that the lock is held. tduffy> */ tduffy> tduffy> -static struct dat_event *dapl_evd_get_event(DAPL_EVD * evd_ptr) tduffy> +static struct dat_event *dapl_evd_get_event(struct dapl_evd *evd_ptr) tduffy> { tduffy> struct dat_event *event; tduffy> tduffy> @@ -389,11 +390,11 @@ static struct dat_event *dapl_evd_get_ev tduffy> * entry to this function. tduffy> */ tduffy> tduffy> -static void dapl_evd_post_event(DAPL_EVD *evd_ptr, tduffy> +static void dapl_evd_post_event(struct dapl_evd *evd_ptr, tduffy> const struct dat_event *event_ptr) tduffy> { tduffy> u32 dat_status; tduffy> - DAPL_CNO * cno_to_trigger = NULL; tduffy> + struct dapl_cno *cno_to_trigger = NULL; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_EVD, tduffy> "dapl_evd_post_event: Called with event # %x\n", tduffy> @@ -420,10 +421,10 @@ static void dapl_evd_post_event(DAPL_EVD tduffy> /* tduffy> * format an overflow event for posting tduffy> */ tduffy> -static void tduffy> -dapl_evd_format_overflow_event(DAPL_EVD * evd_ptr, struct dat_event * event_ptr) tduffy> +static void dapl_evd_format_overflow_event(struct dapl_evd *evd_ptr, tduffy> + struct dat_event *event_ptr) tduffy> { tduffy> - DAPL_IA *ia_ptr; tduffy> + struct dapl_ia *ia_ptr; tduffy> tduffy> ia_ptr = evd_ptr->header.owner_ia; tduffy> tduffy> @@ -436,9 +437,8 @@ dapl_evd_format_overflow_event(DAPL_EVD tduffy> /* tduffy> * post an overflow event tduffy> */ tduffy> -static void tduffy> -dapl_evd_post_overflow_event(DAPL_EVD * async_evd_ptr, tduffy> - DAPL_EVD * overflow_evd_ptr) tduffy> +static void dapl_evd_post_overflow_event(struct dapl_evd *async_evd_ptr, tduffy> + struct dapl_evd *overflow_evd_ptr) tduffy> { tduffy> struct dat_event *overflow_event; tduffy> tduffy> @@ -464,7 +464,7 @@ dapl_evd_post_overflow_event(DAPL_EVD * tduffy> return; tduffy> } tduffy> tduffy> -static struct dat_event *dapl_evd_get_and_init_event(DAPL_EVD *evd_ptr, tduffy> +static struct dat_event *dapl_evd_get_and_init_event(struct dapl_evd *evd_ptr, tduffy> enum dat_event_number evno) tduffy> { tduffy> struct dat_event *event_ptr; tduffy> @@ -481,7 +481,7 @@ static struct dat_event *dapl_evd_get_an tduffy> return event_ptr; tduffy> } tduffy> tduffy> -u32 dapl_evd_post_cr_arrival_event(DAPL_EVD *evd_ptr, tduffy> +u32 dapl_evd_post_cr_arrival_event(struct dapl_evd *evd_ptr, tduffy> DAT_SP_HANDLE sp_handle, tduffy> struct sockaddr *ia_address_ptr, tduffy> DAT_CONN_QUAL conn_qual, tduffy> @@ -511,7 +511,7 @@ u32 dapl_evd_post_cr_arrival_event(DAPL_ tduffy> return DAT_SUCCESS; tduffy> } tduffy> tduffy> -u32 dapl_evd_post_connection_event(DAPL_EVD *evd_ptr, tduffy> +u32 dapl_evd_post_connection_event(struct dapl_evd *evd_ptr, tduffy> enum dat_event_number event_number, tduffy> DAT_EP_HANDLE ep_handle, tduffy> int private_data_size, tduffy> @@ -540,9 +540,9 @@ u32 dapl_evd_post_connection_event(DAPL_ tduffy> return DAT_SUCCESS; tduffy> } tduffy> tduffy> -u32 dapl_evd_post_async_error_event(DAPL_EVD *evd_ptr, tduffy> +u32 dapl_evd_post_async_error_event(struct dapl_evd *evd_ptr, tduffy> enum dat_event_number event_number, tduffy> - DAPL_IA *ia_ptr) tduffy> + struct dapl_ia *ia_ptr) tduffy> { tduffy> struct dat_event *event_ptr; tduffy> event_ptr = dapl_evd_get_and_init_event(evd_ptr, event_number); tduffy> @@ -564,7 +564,7 @@ u32 dapl_evd_post_async_error_event(DAPL tduffy> return DAT_SUCCESS; tduffy> } tduffy> tduffy> -u32 dapl_evd_post_software_event(DAPL_EVD *evd_ptr, tduffy> +u32 dapl_evd_post_software_event(struct dapl_evd *evd_ptr, tduffy> enum dat_event_number event_number, tduffy> void *pointer) tduffy> { tduffy> @@ -604,12 +604,12 @@ u32 dapl_evd_post_software_event(DAPL_EV tduffy> * none tduffy> * tduffy> */ tduffy> -static void tduffy> -dapl_evd_cqe_to_event(DAPL_EVD * evd_ptr, struct ib_wc *cqe_ptr, tduffy> - struct dat_event *event_ptr) tduffy> +static void dapl_evd_cqe_to_event(struct dapl_evd *evd_ptr, tduffy> + struct ib_wc *cqe_ptr, tduffy> + struct dat_event *event_ptr) tduffy> { tduffy> - DAPL_EP *ep_ptr; tduffy> - DAPL_COOKIE *cookie; tduffy> + struct dapl_ep *ep_ptr; tduffy> + struct dapl_cookie *cookie; tduffy> enum dat_dto_completion_status dto_status; tduffy> tduffy> /* tduffy> @@ -618,7 +618,7 @@ dapl_evd_cqe_to_event(DAPL_EVD * evd_ptr tduffy> */ tduffy> dto_status = dapl_ib_get_dto_status(cqe_ptr); tduffy> tduffy> - cookie = (DAPL_COOKIE *) (unsigned long) cqe_ptr->wr_id; tduffy> + cookie = (struct dapl_cookie *) (unsigned long) cqe_ptr->wr_id; tduffy> dapl_os_assert((NULL != cookie)); tduffy> tduffy> ep_ptr = cookie->ep; tduffy> @@ -633,7 +633,7 @@ dapl_evd_cqe_to_event(DAPL_EVD * evd_ptr tduffy> switch (cookie->type) { tduffy> case DAPL_COOKIE_TYPE_DTO: tduffy> { tduffy> - DAPL_COOKIE_BUFFER *buffer; tduffy> + struct dapl_cookie_buffer *buffer; tduffy> tduffy> if (DAPL_DTO_TYPE_RECV == cookie->val.dto.type) { tduffy> atomic_dec(&ep_ptr->recv_count); tduffy> @@ -740,7 +740,7 @@ dapl_evd_cqe_to_event(DAPL_EVD * evd_ptr tduffy> * Status of operation tduffy> * tduffy> */ tduffy> -u32 dapl_evd_cq_poll_to_event(DAPL_EVD *evd_ptr, struct dat_event *event) tduffy> +u32 dapl_evd_cq_poll_to_event(struct dapl_evd *evd_ptr, struct dat_event *event) tduffy> { tduffy> u32 dat_status; tduffy> struct ib_wc cur_cqe; tduffy> Index: linux-kernel-resolv/dat-provider/dapl_ep_disconnect.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_ep_disconnect.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_ep_disconnect.c (working copy) tduffy> @@ -66,16 +66,16 @@ tduffy> u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_handle, tduffy> enum dat_close_flags disconnect_flags) tduffy> { tduffy> - DAPL_EP *ep_ptr; tduffy> - DAPL_EVD *evd_ptr; tduffy> - DAPL_CR *cr_ptr; tduffy> + struct dapl_ep *ep_ptr; tduffy> + struct dapl_evd *evd_ptr; tduffy> + struct dapl_cr *cr_ptr; tduffy> u32 dat_status; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_API | DAPL_DBG_TYPE_CM, tduffy> "dapl_ep_disconnect (%p, %x)\n", tduffy> ep_handle, disconnect_flags); tduffy> tduffy> - ep_ptr = (DAPL_EP *) ep_handle; tduffy> + ep_ptr = (struct dapl_ep *)ep_handle; tduffy> tduffy> /* tduffy> * Verify parameter & state tduffy> @@ -144,7 +144,7 @@ u32 dapl_ep_disconnect(DAT_EP_HANDLE ep_ tduffy> /* clean up connection state */ tduffy> dapl_sp_remove_ep(ep_ptr); tduffy> tduffy> - evd_ptr = (DAPL_EVD *) ep_ptr->param.connect_evd_handle; tduffy> + evd_ptr = (struct dapl_evd *)ep_ptr->param.connect_evd_handle; tduffy> dapl_evd_post_connection_event(evd_ptr, tduffy> DAT_CONNECTION_EVENT_DISCONNECTED, tduffy> (DAT_HANDLE) ep_ptr, 0, NULL); tduffy> Index: linux-kernel-resolv/dat-provider/dapl_sp_util.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_sp_util.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_sp_util.c (working copy) tduffy> @@ -59,13 +59,13 @@ tduffy> * pointer to sp info struct tduffy> * tduffy> */ tduffy> -DAPL_SP *dapl_sp_alloc(DAPL_IA * ia_ptr, boolean_t is_psp) tduffy> +struct dapl_sp *dapl_sp_alloc(struct dapl_ia *ia_ptr, boolean_t is_psp) tduffy> { tduffy> - DAPL_SP *sp_ptr; tduffy> + struct dapl_sp *sp_ptr; tduffy> tduffy> /* Allocate EP */ tduffy> sp_ptr = kmalloc(sizeof *sp_ptr, GFP_ATOMIC); tduffy> - if (!sp_ptr) tduffy> + if (!sp_ptr) tduffy> return NULL; tduffy> tduffy> /* zero the structure */ tduffy> @@ -111,7 +111,7 @@ DAPL_SP *dapl_sp_alloc(DAPL_IA * ia_ptr, tduffy> * none tduffy> * tduffy> */ tduffy> -void dapl_sp_dealloc(DAPL_SP * sp_ptr) tduffy> +void dapl_sp_dealloc(struct dapl_sp *sp_ptr) tduffy> { tduffy> dapl_os_assert(sp_ptr->header.magic == DAPL_MAGIC_PSP || tduffy> sp_ptr->header.magic == DAPL_MAGIC_RSP); tduffy> @@ -138,7 +138,7 @@ void dapl_sp_dealloc(DAPL_SP * sp_ptr) tduffy> * none tduffy> * tduffy> */ tduffy> -void dapl_sp_link_cr(DAPL_SP *sp_ptr, DAPL_CR *cr_ptr) tduffy> +void dapl_sp_link_cr(struct dapl_sp *sp_ptr, struct dapl_cr *cr_ptr) tduffy> { tduffy> spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); tduffy> dapl_llist_add_tail(&sp_ptr->cr_list_head, tduffy> @@ -166,14 +166,15 @@ void dapl_sp_link_cr(DAPL_SP *sp_ptr, DA tduffy> * cr_ptr Pointer to matching DAPL_CR tduffy> * tduffy> */ tduffy> -DAPL_CR *dapl_sp_search_cr(DAPL_SP * sp_ptr, struct dapl_cm_id *ib_cm_handle) tduffy> +struct dapl_cr *dapl_sp_search_cr(struct dapl_sp *sp_ptr, tduffy> + struct dapl_cm_id *ib_cm_handle) tduffy> { tduffy> - DAPL_CR *cr_ptr; tduffy> + struct dapl_cr *cr_ptr; tduffy> tduffy> if (dapl_llist_is_empty(&sp_ptr->cr_list_head)) tduffy> return NULL; tduffy> tduffy> - cr_ptr = (DAPL_CR *) dapl_llist_peek_head(&sp_ptr->cr_list_head); tduffy> + cr_ptr = (struct dapl_cr *)dapl_llist_peek_head(&sp_ptr->cr_list_head); tduffy> tduffy> do { tduffy> if (cr_ptr->ib_cm_handle == ib_cm_handle) tduffy> @@ -203,7 +204,7 @@ DAPL_CR *dapl_sp_search_cr(DAPL_SP * sp_ tduffy> * void tduffy> * tduffy> */ tduffy> -void dapl_sp_remove_cr(DAPL_SP * sp_ptr, DAPL_CR * cr_ptr) tduffy> +void dapl_sp_remove_cr(struct dapl_sp *sp_ptr, struct dapl_cr *cr_ptr) tduffy> { tduffy> if (dapl_llist_is_empty(&sp_ptr->cr_list_head)) { tduffy> dapl_dbg_log(DAPL_DBG_TYPE_ERR, tduffy> @@ -233,10 +234,10 @@ void dapl_sp_remove_cr(DAPL_SP * sp_ptr, tduffy> * void tduffy> * tduffy> */ tduffy> -void dapl_sp_remove_ep(DAPL_EP * ep_ptr) tduffy> +void dapl_sp_remove_ep(struct dapl_ep *ep_ptr) tduffy> { tduffy> - DAPL_SP *sp_ptr; tduffy> - DAPL_CR *cr_ptr; tduffy> + struct dapl_sp *sp_ptr; tduffy> + struct dapl_cr *cr_ptr; tduffy> tduffy> cr_ptr = ep_ptr->cr_ptr; tduffy> tduffy> @@ -253,7 +254,5 @@ void dapl_sp_remove_ep(DAPL_EP * ep_ptr) tduffy> tduffy> /* free memory outside of the lock */ tduffy> dapl_cr_free(cr_ptr); tduffy> - tduffy> - return; tduffy> } tduffy> } tduffy> Index: linux-kernel-resolv/dat-provider/dapl_evd_util.h tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_evd_util.h (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_evd_util.h (working copy) tduffy> @@ -40,16 +40,17 @@ tduffy> tduffy> #include "dapl.h" tduffy> tduffy> -u32 dapl_evd_internal_create(DAPL_IA *ia_ptr, DAPL_CNO *cno_ptr, tduffy> +u32 dapl_evd_internal_create(struct dapl_ia *ia_ptr, struct dapl_cno *cno_ptr, tduffy> int min_qlen, enum dat_evd_flags evd_flags, tduffy> - DAPL_EVD **evd_ptr_ptr); tduffy> + struct dapl_evd **evd_ptr_ptr); tduffy> tduffy> -DAPL_EVD *dapl_evd_alloc(DAPL_IA * ia_ptr, DAPL_CNO * cno_ptr, tduffy> - enum dat_evd_flags evd_flags, int qlen); tduffy> +struct dapl_evd *dapl_evd_alloc(struct dapl_ia *ia_ptr, tduffy> + struct dapl_cno *cno_ptr, tduffy> + enum dat_evd_flags evd_flags, int qlen); tduffy> tduffy> -u32 dapl_evd_dealloc(DAPL_EVD *evd_ptr); tduffy> +u32 dapl_evd_dealloc(struct dapl_evd *evd_ptr); tduffy> tduffy> -u32 dapl_evd_event_realloc(DAPL_EVD *evd_ptr, int qlen); tduffy> +u32 dapl_evd_event_realloc(struct dapl_evd *evd_ptr, int qlen); tduffy> tduffy> /* tduffy> * Each of these functions will retrieve a free event from tduffy> @@ -62,23 +63,23 @@ u32 dapl_evd_event_realloc(DAPL_EVD *evd tduffy> * DAT_SUCCESS otherwise. tduffy> */ tduffy> tduffy> -u32 dapl_evd_post_cr_arrival_event(DAPL_EVD *evd_ptr, tduffy> +u32 dapl_evd_post_cr_arrival_event(struct dapl_evd *evd_ptr, tduffy> DAT_SP_HANDLE sp_handle, tduffy> struct sockaddr *ia_address_ptr, tduffy> DAT_CONN_QUAL conn_qual, tduffy> DAT_CR_HANDLE cr_handle); tduffy> tduffy> -u32 dapl_evd_post_connection_event(DAPL_EVD *evd_ptr, tduffy> +u32 dapl_evd_post_connection_event(struct dapl_evd *evd_ptr, tduffy> enum dat_event_number event_number, tduffy> DAT_EP_HANDLE ep_handle, tduffy> int private_data_size, tduffy> void *private_data); tduffy> tduffy> -u32 dapl_evd_post_async_error_event(DAPL_EVD *evd_ptr, tduffy> +u32 dapl_evd_post_async_error_event(struct dapl_evd *evd_ptr, tduffy> enum dat_event_number event_number, tduffy> - DAPL_IA *ia_ptr); tduffy> + struct dapl_ia *ia_ptr); tduffy> tduffy> -u32 dapl_evd_post_software_event(DAPL_EVD *evd_ptr, tduffy> +u32 dapl_evd_post_software_event(struct dapl_evd *evd_ptr, tduffy> enum dat_event_number event_number, tduffy> void *pointer); tduffy> tduffy> @@ -105,7 +106,7 @@ extern void dapl_evd_cq_async_error_call tduffy> extern void dapl_evd_qp_async_error_callback(struct ib_event *cause_ptr, tduffy> void *context); tduffy> tduffy> -extern u32 dapl_evd_cq_poll_to_event(DAPL_EVD *evd_ptr, tduffy> +extern u32 dapl_evd_cq_poll_to_event(struct dapl_evd *evd_ptr, tduffy> struct dat_event *event); tduffy> tduffy> #endif tduffy> Index: linux-kernel-resolv/dat-provider/dapl_hash.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_hash.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_hash.c (working copy) tduffy> @@ -161,7 +161,7 @@ dapl_hash_rehash(DAPL_HASH_ELEM * elemen tduffy> * TRUE on success tduffy> */ tduffy> static boolean_t tduffy> -dapl_hash_add(DAPL_HASH_TABLEP p_table, DAPL_HASH_KEY key, void *datum, tduffy> +dapl_hash_add(struct dapl_hash_table *table, DAPL_HASH_KEY key, void *datum, tduffy> boolean_t allow_dup, boolean_t *report_dup) tduffy> { tduffy> void *olddatum; tduffy> @@ -184,7 +184,7 @@ dapl_hash_add(DAPL_HASH_TABLEP p_table, tduffy> return FALSE; tduffy> } tduffy> tduffy> - DAPL_HASHLOOKUP(p_table, key, olddatum, found); tduffy> + DAPL_HASHLOOKUP(table, key, olddatum, found); tduffy> if (found) { tduffy> /* tduffy> * key exists already tduffy> @@ -201,16 +201,16 @@ dapl_hash_add(DAPL_HASH_TABLEP p_table, tduffy> } tduffy> } tduffy> tduffy> - hashValue = DAPL_DOHASH(key, p_table->tbl_size); tduffy> + hashValue = DAPL_DOHASH(key, table->tbl_size); tduffy> key = save_key; tduffy> - if (NO_DATUM(p_table->table[hashValue].datum)) { tduffy> + if (NO_DATUM(table->table[hashValue].datum)) { tduffy> /* tduffy> * Empty head - just fill it in tduffy> */ tduffy> - p_table->table[hashValue].key = key; tduffy> - p_table->table[hashValue].datum = datum; tduffy> - p_table->table[hashValue].next_element = NULL; tduffy> - p_table->num_entries++; tduffy> + table->table[hashValue].key = key; tduffy> + table->table[hashValue].datum = datum; tduffy> + table->table[hashValue].next_element = NULL; tduffy> + table->num_entries++; tduffy> status = TRUE; tduffy> } else { tduffy> DAPL_HASH_ELEM *newelement = kmalloc(sizeof *newelement, tduffy> @@ -223,14 +223,14 @@ dapl_hash_add(DAPL_HASH_TABLEP p_table, tduffy> newelement->key = key; tduffy> newelement->datum = datum; tduffy> newelement->next_element = NULL; tduffy> - for (lastelement = &p_table->table[hashValue]; tduffy> + for (lastelement = &table->table[hashValue]; tduffy> lastelement->next_element; tduffy> lastelement = lastelement->next_element) { tduffy> /* Walk to the end of the chain */ tduffy> chain_len++; tduffy> } tduffy> lastelement->next_element = newelement; tduffy> - p_table->num_entries++; tduffy> + table->num_entries++; tduffy> status = TRUE; tduffy> } else tduffy> status = FALSE; tduffy> @@ -241,14 +241,14 @@ dapl_hash_add(DAPL_HASH_TABLEP p_table, tduffy> * length. tduffy> */ tduffy> chain_len++; tduffy> - p_table->hash_tbl_inserts++; tduffy> - p_table->hash_tbl_total += p_table->num_entries; tduffy> - p_table->hash_chn_total += chain_len; tduffy> - if (p_table->num_entries > p_table->hash_tbl_max) { tduffy> - p_table->hash_tbl_max = p_table->num_entries; tduffy> + table->hash_tbl_inserts++; tduffy> + table->hash_tbl_total += table->num_entries; tduffy> + table->hash_chn_total += chain_len; tduffy> + if (table->num_entries > table->hash_tbl_max) { tduffy> + table->hash_tbl_max = table->num_entries; tduffy> } tduffy> - if (chain_len > p_table->hash_chn_max) { tduffy> - p_table->hash_chn_max = chain_len; tduffy> + if (chain_len > table->hash_chn_max) { tduffy> + table->hash_chn_max = chain_len; tduffy> } tduffy> tduffy> return status; tduffy> @@ -319,9 +319,9 @@ dapl_hash_delete_element(DAPL_HASH_ELEM tduffy> /* tduffy> * Create a new hash table with at least 'table_size' hash buckets. tduffy> */ tduffy> -u32 dapl_hash_create(int table_size, DAPL_HASH_TABLE **pp_table) tduffy> +u32 dapl_hash_create(int table_size, struct dapl_hash_table **pp_table) tduffy> { tduffy> - DAPL_HASH_TABLE *p_table; tduffy> + struct dapl_hash_table *p_table; tduffy> int table_length = table_size * sizeof (struct DAPL_HASH_ELEM); tduffy> u32 dat_status = DAT_SUCCESS; tduffy> int i; tduffy> @@ -356,14 +356,14 @@ u32 dapl_hash_create(int table_size, DAP tduffy> tduffy> *pp_table = p_table; tduffy> tduffy> - bail: tduffy> +bail: tduffy> return DAT_SUCCESS; tduffy> } tduffy> tduffy> /* tduffy> * Destroy a hash table tduffy> */ tduffy> -u32 dapl_hash_free(DAPL_HASH_TABLE *p_table) tduffy> +u32 dapl_hash_free(struct dapl_hash_table *p_table) tduffy> { tduffy> dapl_os_assert(p_table && p_table->table); tduffy> tduffy> @@ -378,7 +378,7 @@ u32 dapl_hash_free(DAPL_HASH_TABLE *p_ta tduffy> * Returns the number of elements stored in the table tduffy> */ tduffy> tduffy> -u32 dapl_hash_size(DAPL_HASH_TABLE *p_table, int *p_size) tduffy> +u32 dapl_hash_size(struct dapl_hash_table *p_table, int *p_size) tduffy> { tduffy> dapl_os_assert(p_table && p_size); tduffy> tduffy> @@ -392,7 +392,7 @@ u32 dapl_hash_size(DAPL_HASH_TABLE *p_ta tduffy> * Duplicates are not expected, and return in error, having done nothing. tduffy> */ tduffy> tduffy> -u32 dapl_hash_insert(DAPL_HASH_TABLE *p_table, DAPL_HASH_KEY key, tduffy> +u32 dapl_hash_insert(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, tduffy> DAPL_HASH_DATA data) tduffy> { tduffy> u32 dat_status = DAT_SUCCESS; tduffy> @@ -415,7 +415,7 @@ u32 dapl_hash_insert(DAPL_HASH_TABLE *p_ tduffy> * data is returned in the DAPL_HASH_DATA tduffy> * pointer if that pointer is not NULL. tduffy> */ tduffy> -u32 dapl_hash_search(DAPL_HASH_TABLE *p_table, DAPL_HASH_KEY key, tduffy> +u32 dapl_hash_search(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, tduffy> DAPL_HASH_DATA *p_data) tduffy> { tduffy> u32 dat_status; tduffy> @@ -439,7 +439,7 @@ u32 dapl_hash_search(DAPL_HASH_TABLE *p_ tduffy> return dat_status; tduffy> } tduffy> tduffy> -u32 dapl_hash_remove(DAPL_HASH_TABLE *p_table, DAPL_HASH_KEY key, tduffy> +u32 dapl_hash_remove(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, tduffy> DAPL_HASH_DATA *p_data) tduffy> { tduffy> u32 dat_status; tduffy> Index: linux-kernel-resolv/dat-provider/dapl_sp_util.h tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_sp_util.h (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_sp_util.h (working copy) tduffy> @@ -37,16 +37,17 @@ tduffy> #ifndef DAPL_PSP_UTIL_H tduffy> #define DAPL_PSP_UTIL_H tduffy> tduffy> -DAPL_SP *dapl_sp_alloc(DAPL_IA * ia_ptr, boolean_t is_psp); tduffy> +struct dapl_sp *dapl_sp_alloc(struct dapl_ia *ia_ptr, boolean_t is_psp); tduffy> tduffy> -void dapl_sp_dealloc(DAPL_SP * sp_ptr); tduffy> +void dapl_sp_dealloc(struct dapl_sp *sp_ptr); tduffy> tduffy> -void dapl_sp_link_cr(DAPL_SP * sp_ptr, DAPL_CR * cr_ptr); tduffy> +void dapl_sp_link_cr(struct dapl_sp *sp_ptr, struct dapl_cr *cr_ptr); tduffy> tduffy> -DAPL_CR *dapl_sp_search_cr(DAPL_SP * sp_ptr, struct dapl_cm_id *ib_cm_handle); tduffy> +struct dapl_cr *dapl_sp_search_cr(struct dapl_sp *sp_ptr, tduffy> + struct dapl_cm_id *ib_cm_handle); tduffy> tduffy> -void dapl_sp_remove_cr(DAPL_SP * sp_ptr, DAPL_CR * cr_ptr); tduffy> +void dapl_sp_remove_cr(struct dapl_sp *sp_ptr, struct dapl_cr *cr_ptr); tduffy> tduffy> -void dapl_sp_remove_ep(DAPL_EP * ep_ptr); tduffy> +void dapl_sp_remove_ep(struct dapl_ep *ep_ptr); tduffy> tduffy> -#endif /* DAPL_PSP_UTIL_H */ tduffy> +#endif /* DAPL_PSP_UTIL_H */ tduffy> Index: linux-kernel-resolv/dat-provider/dapl_ep_dup_connect.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_ep_dup_connect.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_ep_dup_connect.c (working copy) tduffy> @@ -73,12 +73,12 @@ u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep tduffy> DAT_TIMEOUT timeout, int private_data_size, tduffy> const void *private_data, enum dat_qos qos) tduffy> { tduffy> - DAPL_EP *ep_dup_ptr; tduffy> + struct dapl_ep *ep_dup_ptr; tduffy> struct sockaddr *remote_ia_address_ptr; tduffy> DAT_CONN_QUAL remote_conn_qual; tduffy> u32 dat_status; tduffy> tduffy> - ep_dup_ptr = (DAPL_EP *) ep_dup_handle; tduffy> + ep_dup_ptr = (struct dapl_ep *)ep_dup_handle; tduffy> tduffy> /* tduffy> * Verify the dup handle, which must be connected. All other tduffy> Index: linux-kernel-resolv/dat-provider/dapl.h tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl.h (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl.h (working copy) tduffy> @@ -123,7 +123,7 @@ typedef enum dapl_evd_completion { tduffy> #define DAPL_BAD_HANDLE(h, magicNum) ( \ tduffy> ((h) == NULL) || \ tduffy> ((unsigned long)(h) & 3) || \ tduffy> - (((DAPL_HEADER *)(h))->magic != (magicNum))) tduffy> + (((struct dapl_header *)(h))->magic != (magicNum))) tduffy> tduffy> #define DAPL_MIN(a, b) ((a < b) ? (a) : (b)) tduffy> #define DAPL_MAX(a, b) ((a > b) ? (a) : (b)) tduffy> @@ -158,35 +158,9 @@ typedef struct ib_hca_transport { tduffy> u16 lid; tduffy> } ib_hca_transport_t; tduffy> tduffy> -typedef struct dapl_llist_entry DAPL_LLIST_ENTRY; tduffy> -typedef DAPL_LLIST_ENTRY *DAPL_LLIST_HEAD; tduffy> -typedef struct dapl_ring_buffer DAPL_RING_BUFFER; tduffy> -typedef struct dapl_cookie_buffer DAPL_COOKIE_BUFFER; tduffy> - tduffy> -typedef struct dapl_hash_table DAPL_HASH_TABLE; tduffy> -typedef struct dapl_hash_table *DAPL_HASH_TABLEP; tduffy> typedef u64 DAPL_HASH_KEY; tduffy> typedef void *DAPL_HASH_DATA; tduffy> tduffy> -typedef struct dapl_hca DAPL_HCA; tduffy> - tduffy> -typedef struct dapl_header DAPL_HEADER; tduffy> - tduffy> -typedef struct dapl_ia DAPL_IA; tduffy> -typedef struct dapl_cno DAPL_CNO; tduffy> -typedef struct dapl_evd DAPL_EVD; tduffy> -typedef struct dapl_ep DAPL_EP; tduffy> -typedef struct dapl_srq DAPL_SRQ; tduffy> -typedef struct dapl_pz DAPL_PZ; tduffy> -typedef struct dapl_lmr DAPL_LMR; tduffy> -typedef struct dapl_rmr DAPL_RMR; tduffy> -typedef struct dapl_sp DAPL_SP; tduffy> -typedef struct dapl_cr DAPL_CR; tduffy> - tduffy> -typedef struct dapl_cookie DAPL_COOKIE; tduffy> -typedef struct dapl_dto_cookie DAPL_DTO_COOKIE; tduffy> -typedef struct dapl_rmr_cookie DAPL_RMR_COOKIE; tduffy> - tduffy> /********************************************************************* tduffy> * * tduffy> * Structures * tduffy> @@ -197,7 +171,7 @@ struct dapl_llist_entry { tduffy> struct dapl_llist_entry *flink; tduffy> struct dapl_llist_entry *blink; tduffy> void *data; tduffy> - DAPL_LLIST_HEAD *list_head; /* for consistency checking */ tduffy> + struct dapl_llist_entry **list_head; /* for consistency checking */ tduffy> }; tduffy> tduffy> struct dapl_ring_buffer { tduffy> @@ -208,7 +182,7 @@ struct dapl_ring_buffer { tduffy> }; tduffy> tduffy> struct dapl_cookie_buffer { tduffy> - DAPL_COOKIE *pool; tduffy> + struct dapl_cookie *pool; tduffy> int pool_size; tduffy> atomic_t head; tduffy> atomic_t tail; tduffy> @@ -217,7 +191,7 @@ struct dapl_cookie_buffer { tduffy> struct dapl_hca { tduffy> spinlock_t lock; tduffy> unsigned long flags; tduffy> - DAPL_LLIST_HEAD ia_list_head; /* list of all open IAs */ tduffy> + struct dapl_llist_entry *ia_list_head; /* list of all open IAs */ tduffy> atomic_t handle_ref_count; /* count of ia_opens on handle */ tduffy> struct sockaddr_in6 hca_address; /* local address of HCA */ tduffy> char *name; /* provider name */ tduffy> @@ -225,7 +199,7 @@ struct dapl_hca { tduffy> u8 port_num; /* number of physical port */ tduffy> ib_hca_transport_t ib_trans; /* Values specific transport API */ tduffy> /* Memory Subsystem Support */ tduffy> - DAPL_HASH_TABLE *lmr_hash_table; tduffy> + struct dapl_hash_table *lmr_hash_table; tduffy> /* Limits & useful HCA attributes */ tduffy> struct dat_ia_attr ia_attr; tduffy> struct dat_ep_attr ep_attr; tduffy> @@ -236,8 +210,8 @@ struct dapl_header { tduffy> struct dat_provider *provider; /* required by DAT - must be first */ tduffy> DAPL_MAGIC magic; /* magic number for verification */ tduffy> enum dat_handle_type handle_type; /* struct type */ tduffy> - DAPL_IA *owner_ia; /* ia which owns this stuct */ tduffy> - DAPL_LLIST_ENTRY ia_list_entry; /* link entry on ia struct */ tduffy> + struct dapl_ia *owner_ia; /* ia which owns this struct */ tduffy> + struct dapl_llist_entry ia_list_entry; /* link entry on ia struct */ tduffy> DAT_CONTEXT user_context; /* user context - opaque to DAPL */ tduffy> spinlock_t lock; /* lock - in header for easier macros */ tduffy> unsigned long flags; /* saved lock flag values */ tduffy> @@ -245,32 +219,32 @@ struct dapl_header { tduffy> tduffy> /* DAPL_IA maps to DAT_IA_HANDLE */ tduffy> struct dapl_ia { tduffy> - DAPL_HEADER header; tduffy> - DAPL_HCA *hca_ptr; tduffy> - DAPL_EVD *async_error_evd; tduffy> + struct dapl_header header; /* WARNING: must be first */ tduffy> + struct dapl_hca *hca_ptr; tduffy> + struct dapl_evd *async_error_evd; tduffy> boolean_t cleanup_async_error_evd; tduffy> tduffy> - DAPL_LLIST_ENTRY hca_ia_list_entry; /* HCAs list of IAs */ tduffy> - DAPL_LLIST_HEAD ep_list_head; /* EP queue */ tduffy> - DAPL_LLIST_HEAD lmr_list_head; /* LMR queue */ tduffy> - DAPL_LLIST_HEAD rmr_list_head; /* RMR queue */ tduffy> - DAPL_LLIST_HEAD pz_list_head; /* PZ queue */ tduffy> - DAPL_LLIST_HEAD evd_list_head; /* EVD queue */ tduffy> - DAPL_LLIST_HEAD psp_list_head; /* PSP queue */ tduffy> - DAPL_LLIST_HEAD rsp_list_head; /* RSP queue */ tduffy> - DAPL_LLIST_HEAD srq_list_head; /* SRQ queue */ tduffy> + struct dapl_llist_entry hca_ia_list_entry; /* HCAs list of IAs */ tduffy> + struct dapl_llist_entry *ep_list_head; /* EP queue */ tduffy> + struct dapl_llist_entry *lmr_list_head; /* LMR queue */ tduffy> + struct dapl_llist_entry *rmr_list_head; /* RMR queue */ tduffy> + struct dapl_llist_entry *pz_list_head; /* PZ queue */ tduffy> + struct dapl_llist_entry *evd_list_head; /* EVD queue */ tduffy> + struct dapl_llist_entry *psp_list_head; /* PSP queue */ tduffy> + struct dapl_llist_entry *rsp_list_head; /* RSP queue */ tduffy> + struct dapl_llist_entry *srq_list_head; /* SRQ queue */ tduffy> }; tduffy> tduffy> /* DAPL_CNO maps to DAT_CNO_HANDLE */ tduffy> struct dapl_cno { tduffy> - DAPL_HEADER header; tduffy> + struct dapl_header header; /* WARNING: must be first */ tduffy> struct dat_upcall_object cno_upcall; tduffy> enum dat_upcall_policy cno_upcall_policy; tduffy> }; tduffy> tduffy> /* DAPL_EVD maps to DAT_EVD_HANDLE */ tduffy> struct dapl_evd { tduffy> - DAPL_HEADER header; tduffy> + struct dapl_header header; /* WARNING: must be first */ tduffy> DAPL_EVD_STATE evd_state; tduffy> enum dat_evd_flags evd_flags; tduffy> tduffy> @@ -292,22 +266,22 @@ struct dapl_evd { tduffy> /* the actual events */ tduffy> int qlen; tduffy> struct dat_event *events; tduffy> - DAPL_RING_BUFFER free_event_queue; tduffy> - DAPL_RING_BUFFER pending_event_queue; tduffy> - DAPL_CNO *cno_ptr; tduffy> + struct dapl_ring_buffer free_event_queue; tduffy> + struct dapl_ring_buffer pending_event_queue; tduffy> + struct dapl_cno *cno_ptr; tduffy> }; tduffy> tduffy> /* uDAPL timer entry, used to queue timeouts */ tduffy> struct dapl_timer_entry { tduffy> - DAPL_LLIST_ENTRY list_entry; /* link entry on ia struct */ tduffy> + struct dapl_llist_entry list_entry; /* link entry on ia struct */ tduffy> DAPL_OS_TIMEVAL expires; tduffy> void (*function) (void *); tduffy> void *data; tduffy> }; tduffy> tduffy> -/* DAPL_EP maps to DAT_EP_HANDLE */ tduffy> +/* struct dapl_ep maps to DAT_EP_HANDLE */ tduffy> struct dapl_ep { tduffy> - DAPL_HEADER header; tduffy> + struct dapl_header header; /* WARNING: must be first */ tduffy> /* What the DAT Consumer asked for */ tduffy> struct dat_ep_param param; tduffy> tduffy> @@ -336,8 +310,8 @@ struct dapl_ep { tduffy> atomic_t req_count; tduffy> atomic_t recv_count; tduffy> tduffy> - DAPL_COOKIE_BUFFER req_buffer; tduffy> - DAPL_COOKIE_BUFFER recv_buffer; tduffy> + struct dapl_cookie_buffer req_buffer; tduffy> + struct dapl_cookie_buffer recv_buffer; tduffy> tduffy> struct ib_sge *recv_iov; tduffy> int recv_iov_num; tduffy> @@ -346,37 +320,37 @@ struct dapl_ep { tduffy> int send_iov_num; tduffy> }; tduffy> tduffy> -/* DAPL_SRQ maps to DAT_SRQ_HANDLE */ tduffy> +/* struct dapl_srq maps to DAT_SRQ_HANDLE */ tduffy> struct dapl_srq { tduffy> - DAPL_HEADER header; tduffy> + struct dapl_header header; /* WARNING: must be first */ tduffy> struct dat_srq_param param; tduffy> atomic_t srq_ref_count; tduffy> - DAPL_COOKIE_BUFFER recv_buffer; tduffy> + struct dapl_cookie_buffer recv_buffer; tduffy> atomic_t recv_count; tduffy> }; tduffy> tduffy> -/* DAPL_PZ maps to DAT_PZ_HANDLE */ tduffy> +/* struct dapl_pz maps to DAT_PZ_HANDLE */ tduffy> struct dapl_pz { tduffy> - DAPL_HEADER header; tduffy> + struct dapl_header header; /* WARNING: must be first */ tduffy> struct ib_pd *pd_handle; tduffy> atomic_t pz_ref_count; tduffy> }; tduffy> tduffy> -/* DAPL_LMR maps to DAT_LMR_HANDLE */ tduffy> +/* struct dapl_lmr maps to DAT_LMR_HANDLE */ tduffy> struct dapl_lmr { tduffy> - DAPL_HEADER header; tduffy> + struct dapl_header header; /* WARNING: must be first */ tduffy> struct dat_lmr_param param; tduffy> struct ib_mr *mr_handle; tduffy> atomic_t lmr_ref_count; tduffy> }; tduffy> tduffy> -/* DAPL_RMR maps to DAT_RMR_HANDLE */ tduffy> +/* struct dapl_rmr maps to DAT_RMR_HANDLE */ tduffy> struct dapl_rmr { tduffy> - DAPL_HEADER header; tduffy> + struct dapl_header header; /* WARNING: must be first */ tduffy> struct dat_rmr_param param; tduffy> - DAPL_EP *ep; tduffy> - DAPL_PZ *pz; tduffy> - DAPL_LMR *lmr; tduffy> + struct dapl_ep *ep; tduffy> + struct dapl_pz *pz; tduffy> + struct dapl_lmr *lmr; tduffy> struct ib_mw *mw_handle; tduffy> }; tduffy> tduffy> @@ -389,9 +363,9 @@ typedef enum dapl_sp_state { tduffy> DAPL_SP_STATE_RSP_PENDING tduffy> } DAPL_SP_STATE; tduffy> tduffy> -/* DAPL_SP maps to DAT_PSP_HANDLE and DAT_RSP_HANDLE */ tduffy> +/* struct dapl_sp maps to DAT_PSP_HANDLE and DAT_RSP_HANDLE */ tduffy> struct dapl_sp { tduffy> - DAPL_HEADER header; tduffy> + struct dapl_header header; /* WARNING: must be first */ tduffy> DAPL_SP_STATE state; /* type and queue of the SP */ tduffy> tduffy> /* PSP/RSP PARAM fields */ tduffy> @@ -404,13 +378,13 @@ struct dapl_sp { tduffy> /* maintenence fields */ tduffy> boolean_t listening; /* PSP is registered & active */ tduffy> struct ib_cm_id *cm_srvc_handle; /* Used by CM */ tduffy> - DAPL_LLIST_HEAD cr_list_head; /* CR pending queue */ tduffy> + struct dapl_llist_entry *cr_list_head; /* CR pending queue */ tduffy> int cr_list_count; /* count of CRs on queue */ tduffy> }; tduffy> tduffy> -/* DAPL_CR maps to DAT_CR_HANDLE */ tduffy> +/* struct dapl_cr maps to DAT_CR_HANDLE */ tduffy> struct dapl_cr { tduffy> - DAPL_HEADER header; tduffy> + struct dapl_header header; /* WARNING: must be first */ tduffy> tduffy> /* for convenience the data is kept as a struct dat_cr_param. tduffy> * however, the "local_endpoint" field is always NULL tduffy> @@ -430,7 +404,7 @@ struct dapl_cr { tduffy> * Need to be able to associate the CR back to the PSP for tduffy> * dapl_cr_reject. tduffy> */ tduffy> - DAPL_SP *sp_ptr; tduffy> + struct dapl_sp *sp_ptr; tduffy> }; tduffy> tduffy> typedef enum dapl_dto_type { tduffy> @@ -446,27 +420,27 @@ typedef enum dapl_cookie_type { tduffy> DAPL_COOKIE_TYPE_RMR, tduffy> } DAPL_COOKIE_TYPE; tduffy> tduffy> -/* DAPL_DTO_COOKIE used as context for DTO WQEs */ tduffy> +/* struct dapl_dto_cookie used as context for DTO WQEs */ tduffy> struct dapl_dto_cookie { tduffy> DAPL_DTO_TYPE type; tduffy> DAT_DTO_COOKIE cookie; tduffy> int size; /* used for SEND and RDMA write */ tduffy> }; tduffy> tduffy> -/* DAPL_RMR_COOKIE used as context for bind WQEs */ tduffy> +/* struct dapl_rmr_cookie used as context for bind WQEs */ tduffy> struct dapl_rmr_cookie { tduffy> - DAPL_RMR *rmr; tduffy> + struct dapl_rmr *rmr; tduffy> DAT_RMR_COOKIE cookie; tduffy> }; tduffy> tduffy> -/* DAPL_COOKIE used as context for WQEs */ tduffy> +/* struct dapl_cookie used as context for WQEs */ tduffy> struct dapl_cookie { tduffy> DAPL_COOKIE_TYPE type; /* Must be first, to define struct. */ tduffy> - DAPL_EP *ep; tduffy> + struct dapl_ep *ep; tduffy> int index; tduffy> union { tduffy> - DAPL_DTO_COOKIE dto; tduffy> - DAPL_RMR_COOKIE rmr; tduffy> + struct dapl_dto_cookie dto; tduffy> + struct dapl_rmr_cookie rmr; tduffy> } val; tduffy> }; tduffy> tduffy> @@ -737,34 +711,35 @@ extern u32 dapl_srq_set_lw(DAT_SRQ_HANDL tduffy> * DAPL internal utility function prototpyes tduffy> */ tduffy> tduffy> -extern void dapl_llist_init_head(DAPL_LLIST_HEAD * head); tduffy> +extern void dapl_llist_init_head(struct dapl_llist_entry **head); tduffy> tduffy> -extern void dapl_llist_init_entry(DAPL_LLIST_ENTRY * entry); tduffy> +extern void dapl_llist_init_entry(struct dapl_llist_entry *entry); tduffy> tduffy> -extern boolean_t dapl_llist_is_empty(DAPL_LLIST_HEAD * head); tduffy> +extern boolean_t dapl_llist_is_empty(struct dapl_llist_entry **head); tduffy> tduffy> -extern void dapl_llist_add_head(DAPL_LLIST_HEAD * head, tduffy> - DAPL_LLIST_ENTRY * entry, void *data); tduffy> +extern void dapl_llist_add_head(struct dapl_llist_entry **head, tduffy> + struct dapl_llist_entry *entry, void *data); tduffy> tduffy> -extern void dapl_llist_add_tail(DAPL_LLIST_HEAD * head, tduffy> - DAPL_LLIST_ENTRY * entry, void *data); tduffy> +extern void dapl_llist_add_tail(struct dapl_llist_entry **head, tduffy> + struct dapl_llist_entry *entry, void *data); tduffy> tduffy> -extern void dapl_llist_add_entry(DAPL_LLIST_HEAD * head, tduffy> - DAPL_LLIST_ENTRY * entry, tduffy> - DAPL_LLIST_ENTRY * new_entry, void *data); tduffy> +extern void dapl_llist_add_entry(struct dapl_llist_entry **head, tduffy> + struct dapl_llist_entry *entry, tduffy> + struct dapl_llist_entry *new_entry, tduffy> + void *data); tduffy> tduffy> -extern void *dapl_llist_remove_head(DAPL_LLIST_HEAD * head); tduffy> +extern void *dapl_llist_remove_head(struct dapl_llist_entry **head); tduffy> tduffy> -extern void *dapl_llist_remove_tail(DAPL_LLIST_HEAD * head); tduffy> +extern void *dapl_llist_remove_tail(struct dapl_llist_entry **head); tduffy> tduffy> -extern void *dapl_llist_remove_entry(DAPL_LLIST_HEAD * head, tduffy> - DAPL_LLIST_ENTRY * entry); tduffy> +extern void *dapl_llist_remove_entry(struct dapl_llist_entry **head, tduffy> + struct dapl_llist_entry *entry); tduffy> tduffy> -extern void *dapl_llist_peek_head(DAPL_LLIST_HEAD * head); tduffy> +extern void *dapl_llist_peek_head(struct dapl_llist_entry **head); tduffy> tduffy> -extern void *dapl_llist_next_entry(DAPL_LLIST_HEAD * head, tduffy> - DAPL_LLIST_ENTRY * cur_ent); tduffy> +extern void *dapl_llist_next_entry(struct dapl_llist_entry **head, tduffy> + struct dapl_llist_entry *cur_ent); tduffy> tduffy> -extern void dapl_llist_debug_print_list(DAPL_LLIST_HEAD * head); tduffy> +extern void dapl_llist_debug_print_list(struct dapl_llist_entry **head); tduffy> tduffy> #endif tduffy> Index: linux-kernel-resolv/dat-provider/dapl_psp_query.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_psp_query.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_psp_query.c (working copy) tduffy> @@ -40,11 +40,11 @@ tduffy> tduffy> u32 dapl_psp_query(DAT_PSP_HANDLE psp_handle, struct dat_psp_param *psp_param) tduffy> { tduffy> - DAPL_SP *sp_ptr; tduffy> + struct dapl_sp *sp_ptr; tduffy> u32 status; tduffy> tduffy> if (DAPL_BAD_HANDLE(psp_handle, DAPL_MAGIC_PSP) || tduffy> - ((DAPL_SP *) psp_handle)->listening != TRUE) { tduffy> + ((struct dapl_sp *)psp_handle)->listening != TRUE) { tduffy> status = tduffy> DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PSP); tduffy> goto bail; tduffy> @@ -55,7 +55,7 @@ u32 dapl_psp_query(DAT_PSP_HANDLE psp_ha tduffy> goto bail; tduffy> } tduffy> tduffy> - sp_ptr = (DAPL_SP *) psp_handle; tduffy> + sp_ptr = (struct dapl_sp *)psp_handle; tduffy> tduffy> psp_param->ia_handle = sp_ptr->ia_handle; tduffy> psp_param->conn_qual = sp_ptr->conn_qual; tduffy> @@ -64,6 +64,6 @@ u32 dapl_psp_query(DAT_PSP_HANDLE psp_ha tduffy> tduffy> status = DAT_SUCCESS; tduffy> tduffy> - bail: tduffy> +bail: tduffy> return status; tduffy> } tduffy> Index: linux-kernel-resolv/dat-provider/dapl_llist.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_llist.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_llist.c (working copy) tduffy> @@ -57,7 +57,7 @@ tduffy> * tduffy> * Purpose: initialize a linked list head tduffy> */ tduffy> -void dapl_llist_init_head(DAPL_LLIST_HEAD * head) tduffy> +void dapl_llist_init_head(struct dapl_llist_entry **head) tduffy> { tduffy> *head = NULL; tduffy> } tduffy> @@ -67,7 +67,7 @@ void dapl_llist_init_head(DAPL_LLIST_HEA tduffy> * tduffy> * Purpose: initialize a linked list entry tduffy> */ tduffy> -void dapl_llist_init_entry(DAPL_LLIST_ENTRY * entry) tduffy> +void dapl_llist_init_entry(struct dapl_llist_entry *entry) tduffy> { tduffy> entry->blink = NULL; tduffy> entry->flink = NULL; tduffy> @@ -80,7 +80,7 @@ void dapl_llist_init_entry(DAPL_LLIST_EN tduffy> * tduffy> * Purpose: returns TRUE if the linked list is empty tduffy> */ tduffy> -boolean_t dapl_llist_is_empty(DAPL_LLIST_HEAD * head) tduffy> +boolean_t dapl_llist_is_empty(struct dapl_llist_entry **head) tduffy> { tduffy> return (*head == NULL); tduffy> } tduffy> @@ -91,10 +91,10 @@ boolean_t dapl_llist_is_empty(DAPL_LLIST tduffy> * Purpose: Add an entry to the head of a linked list tduffy> */ tduffy> void tduffy> -dapl_llist_add_head(DAPL_LLIST_HEAD * head, tduffy> - DAPL_LLIST_ENTRY * entry, void *data) tduffy> +dapl_llist_add_head(struct dapl_llist_entry **head, tduffy> + struct dapl_llist_entry *entry, void *data) tduffy> { tduffy> - DAPL_LLIST_ENTRY *first; tduffy> + struct dapl_llist_entry *first; tduffy> tduffy> /* deal with empty list */ tduffy> if (dapl_llist_is_empty(head)) { tduffy> @@ -119,10 +119,10 @@ dapl_llist_add_head(DAPL_LLIST_HEAD * he tduffy> * Purpose: Add an entry to the tail of a linked list tduffy> */ tduffy> void tduffy> -dapl_llist_add_tail(DAPL_LLIST_HEAD * head, tduffy> - DAPL_LLIST_ENTRY * entry, void *data) tduffy> +dapl_llist_add_tail(struct dapl_llist_entry **head, tduffy> + struct dapl_llist_entry *entry, void *data) tduffy> { tduffy> - DAPL_LLIST_ENTRY *last; tduffy> + struct dapl_llist_entry *last; tduffy> tduffy> /* deal with empty list */ tduffy> if (dapl_llist_is_empty(head)) { tduffy> @@ -147,12 +147,12 @@ dapl_llist_add_tail(DAPL_LLIST_HEAD * he tduffy> * not verify the list or the validity of the entries passed in. tduffy> */ tduffy> void tduffy> -dapl_llist_add_entry(DAPL_LLIST_HEAD * head, tduffy> - DAPL_LLIST_ENTRY * entry, tduffy> - DAPL_LLIST_ENTRY * new_entry, void *data) tduffy> +dapl_llist_add_entry(struct dapl_llist_entry **head, tduffy> + struct dapl_llist_entry *entry, tduffy> + struct dapl_llist_entry *new_entry, void *data) tduffy> { tduffy> - DAPL_LLIST_ENTRY *first; tduffy> - DAPL_LLIST_ENTRY *prev; tduffy> + struct dapl_llist_entry *first; tduffy> + struct dapl_llist_entry *prev; tduffy> tduffy> /* get the head pointer */ tduffy> first = *head; tduffy> @@ -180,9 +180,9 @@ dapl_llist_add_entry(DAPL_LLIST_HEAD * h tduffy> * tduffy> * Purpose: Remove the first entry of a linked list tduffy> */ tduffy> -void *dapl_llist_remove_head(DAPL_LLIST_HEAD * head) tduffy> +void *dapl_llist_remove_head(struct dapl_llist_entry **head) tduffy> { tduffy> - DAPL_LLIST_ENTRY *first; tduffy> + struct dapl_llist_entry *first; tduffy> tduffy> dapl_os_assert(!dapl_llist_is_empty(head)); tduffy> first = *head; tduffy> @@ -206,9 +206,9 @@ void *dapl_llist_remove_head(DAPL_LLIST_ tduffy> * tduffy> * Purpose: Remove the last entry of a linked list tduffy> */ tduffy> -void *dapl_llist_remove_tail(DAPL_LLIST_HEAD * head) tduffy> +void *dapl_llist_remove_tail(struct dapl_llist_entry **head) tduffy> { tduffy> - DAPL_LLIST_ENTRY *last; tduffy> + struct dapl_llist_entry *last; tduffy> tduffy> dapl_os_assert(!dapl_llist_is_empty(head)); tduffy> last = (*head)->blink; tduffy> @@ -232,9 +232,10 @@ void *dapl_llist_remove_tail(DAPL_LLIST_ tduffy> * tduffy> * Purpose: Remove the specified entry from a linked list tduffy> */ tduffy> -void *dapl_llist_remove_entry(DAPL_LLIST_HEAD * head, DAPL_LLIST_ENTRY * entry) tduffy> +void *dapl_llist_remove_entry(struct dapl_llist_entry **head, tduffy> + struct dapl_llist_entry *entry) tduffy> { tduffy> - DAPL_LLIST_ENTRY *first; tduffy> + struct dapl_llist_entry *first; tduffy> tduffy> dapl_os_assert(!dapl_llist_is_empty(head)); tduffy> first = *head; tduffy> @@ -263,9 +264,9 @@ void *dapl_llist_remove_entry(DAPL_LLIST tduffy> * dapl_llist_peek_head tduffy> */ tduffy> tduffy> -void *dapl_llist_peek_head(DAPL_LLIST_HEAD * head) tduffy> +void *dapl_llist_peek_head(struct dapl_llist_entry **head) tduffy> { tduffy> - DAPL_LLIST_ENTRY *first; tduffy> + struct dapl_llist_entry *first; tduffy> tduffy> dapl_os_assert(!dapl_llist_is_empty(head)); tduffy> first = *head; tduffy> @@ -279,9 +280,10 @@ void *dapl_llist_peek_head(DAPL_LLIST_HE tduffy> * head tduffy> */ tduffy> tduffy> -void *dapl_llist_next_entry(DAPL_LLIST_HEAD * head, DAPL_LLIST_ENTRY * cur_ent) tduffy> +void *dapl_llist_next_entry(struct dapl_llist_entry **head, tduffy> + struct dapl_llist_entry *cur_ent) tduffy> { tduffy> - DAPL_LLIST_ENTRY *next; tduffy> + struct dapl_llist_entry *next; tduffy> tduffy> dapl_os_assert(!dapl_llist_is_empty(head)); tduffy> if (cur_ent == NULL) { tduffy> @@ -301,10 +303,10 @@ void *dapl_llist_next_entry(DAPL_LLIST_H tduffy> * tduffy> * Purpose: Prints the linked list for debugging tduffy> */ tduffy> -void dapl_llist_debug_print_list(DAPL_LLIST_HEAD * head) tduffy> +void dapl_llist_debug_print_list(struct dapl_llist_entry **head) tduffy> { tduffy> - DAPL_LLIST_ENTRY *entry; tduffy> - DAPL_LLIST_ENTRY *first; tduffy> + struct dapl_llist_entry *entry; tduffy> + struct dapl_llist_entry *first; tduffy> first = *head; tduffy> if (!first) { tduffy> dapl_dbg_log(DAPL_DBG_TYPE_UTIL, "EMPTY_LIST\n"); tduffy> Index: linux-kernel-resolv/dat-provider/dapl_hash.h tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_hash.h (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_hash.h (working copy) tduffy> @@ -67,19 +67,19 @@ tduffy> * * tduffy> *********************************************************************/ tduffy> tduffy> -extern u32 dapl_hash_create(int capacity, DAPL_HASH_TABLE **pp_table); tduffy> +extern u32 dapl_hash_create(int capacity, struct dapl_hash_table **pp_table); tduffy> tduffy> -extern u32 dapl_hash_free(DAPL_HASH_TABLE *p_table); tduffy> +extern u32 dapl_hash_free(struct dapl_hash_table *p_table); tduffy> tduffy> -extern u32 dapl_hash_size(DAPL_HASH_TABLE *p_table, int *p_size); tduffy> +extern u32 dapl_hash_size(struct dapl_hash_table *p_table, int *p_size); tduffy> tduffy> -extern u32 dapl_hash_insert(DAPL_HASH_TABLE *p_table, DAPL_HASH_KEY key, tduffy> +extern u32 dapl_hash_insert(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, tduffy> DAPL_HASH_DATA data); tduffy> tduffy> -extern u32 dapl_hash_search(DAPL_HASH_TABLE *p_table, DAPL_HASH_KEY key, tduffy> +extern u32 dapl_hash_search(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, tduffy> DAPL_HASH_DATA *p_data); tduffy> tduffy> -extern u32 dapl_hash_remove(DAPL_HASH_TABLE *p_table, DAPL_HASH_KEY key, tduffy> +extern u32 dapl_hash_remove(struct dapl_hash_table *p_table, DAPL_HASH_KEY key, tduffy> DAPL_HASH_DATA *p_data); tduffy> tduffy> #endif /* DAPL_HASH_H */ tduffy> Index: linux-kernel-resolv/dat-provider/dapl_lmr_kcreate.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_lmr_kcreate.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_lmr_kcreate.c (working copy) tduffy> @@ -37,8 +37,8 @@ tduffy> #include "dapl_lmr_util.h" tduffy> #include "dapl_openib_util.h" tduffy> tduffy> -static inline u32 dapl_lmr_create_virtual(DAPL_IA *ia, void *virt_addr, tduffy> - u64 length, DAPL_PZ *pz, tduffy> +static inline u32 dapl_lmr_create_virtual(struct dapl_ia *ia, void *virt_addr, tduffy> + u64 length, struct dapl_pz *pz, tduffy> enum dat_mem_priv_flags privileges, tduffy> DAT_LMR_HANDLE *lmr_handle, tduffy> DAT_LMR_CONTEXT *lmr_context, tduffy> @@ -46,7 +46,7 @@ static inline u32 dapl_lmr_create_virtua tduffy> u64 *registered_length, tduffy> u64 *registered_address) tduffy> { tduffy> - DAPL_LMR *lmr; tduffy> + struct dapl_lmr *lmr; tduffy> DAT_REGION_DESCRIPTION reg_desc; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> @@ -114,9 +114,9 @@ bail: tduffy> tduffy> } tduffy> tduffy> -static inline u32 dapl_lmr_create_physical(DAPL_IA *ia, tduffy> +static inline u32 dapl_lmr_create_physical(struct dapl_ia *ia, tduffy> DAT_REGION_DESCRIPTION phys_addr, tduffy> - u64 page_count, DAPL_PZ *pz, tduffy> + u64 page_count, struct dapl_pz *pz, tduffy> enum dat_mem_priv_flags privileges, tduffy> DAT_LMR_HANDLE *lmr_handle, tduffy> DAT_LMR_CONTEXT *lmr_context, tduffy> @@ -124,7 +124,7 @@ static inline u32 dapl_lmr_create_physic tduffy> u64 *registered_length, tduffy> u64 *registered_address) tduffy> { tduffy> - DAPL_LMR *lmr; tduffy> + struct dapl_lmr *lmr; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> u64 *array = phys_addr.for_array; tduffy> tduffy> @@ -190,8 +190,9 @@ bail: tduffy> return dat_status; tduffy> } tduffy> tduffy> -static inline u32 dapl_lmr_create_lmr(DAPL_IA *ia, DAPL_LMR *original_lmr, tduffy> - DAPL_PZ *pz, tduffy> +static inline u32 dapl_lmr_create_lmr(struct dapl_ia *ia, tduffy> + struct dapl_lmr *original_lmr, tduffy> + struct dapl_pz *pz, tduffy> enum dat_mem_priv_flags privileges, tduffy> DAT_LMR_HANDLE *lmr_handle, tduffy> DAT_LMR_CONTEXT *lmr_context, tduffy> @@ -199,7 +200,7 @@ static inline u32 dapl_lmr_create_lmr(DA tduffy> u64 *registered_length, tduffy> u64 *registered_address) tduffy> { tduffy> - DAPL_LMR *lmr; tduffy> + struct dapl_lmr *lmr; tduffy> DAT_REGION_DESCRIPTION reg_desc; tduffy> u32 dat_status; tduffy> tduffy> @@ -309,8 +310,8 @@ u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_ha tduffy> enum dat_mem_optimize_flags optimization, DAT_LMR_HANDLE *lmr_handle, tduffy> DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address) tduffy> { tduffy> - DAPL_IA *ia; tduffy> - DAPL_PZ *pz; tduffy> + struct dapl_ia *ia; tduffy> + struct dapl_pz *pz; tduffy> u32 dat_status; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> @@ -328,8 +329,8 @@ u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_ha tduffy> goto bail; tduffy> } tduffy> tduffy> - ia = (DAPL_IA *) ia_handle; tduffy> - pz = (DAPL_PZ *) pz_handle; tduffy> + ia = (struct dapl_ia *)ia_handle; tduffy> + pz = (struct dapl_pz *)pz_handle; tduffy> tduffy> switch (mem_type) { tduffy> case DAT_MEM_TYPE_VIRTUAL: tduffy> @@ -358,7 +359,7 @@ u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_ha tduffy> } tduffy> case DAT_MEM_TYPE_LMR: tduffy> { tduffy> - DAPL_LMR *lmr; tduffy> + struct dapl_lmr *lmr; tduffy> tduffy> if (DAPL_BAD_HANDLE tduffy> (region_description.for_lmr_handle, tduffy> @@ -369,7 +370,7 @@ u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_ha tduffy> goto bail; tduffy> } tduffy> tduffy> - lmr = (DAPL_LMR *) region_description.for_lmr_handle; tduffy> + lmr = (struct dapl_lmr *)region_description.for_lmr_handle; tduffy> tduffy> dat_status = tduffy> dapl_lmr_create_lmr(ia, lmr, pz, privileges, tduffy> @@ -394,6 +395,6 @@ u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_ha tduffy> } tduffy> } tduffy> tduffy> - bail: tduffy> +bail: tduffy> return dat_status; tduffy> } tduffy> Index: linux-kernel-resolv/dat-provider/dapl_ep_post_recv.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_ep_post_recv.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_ep_post_recv.c (working copy) tduffy> @@ -71,8 +71,8 @@ u32 dapl_ep_post_recv(DAT_EP_HANDLE ep_h tduffy> DAT_DTO_COOKIE user_cookie, tduffy> enum dat_completion_flags completion_flags) tduffy> { tduffy> - DAPL_EP *ep_ptr; tduffy> - DAPL_COOKIE *cookie; tduffy> + struct dapl_ep *ep_ptr; tduffy> + struct dapl_cookie *cookie; tduffy> u32 dat_status; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> @@ -87,7 +87,7 @@ u32 dapl_ep_post_recv(DAT_EP_HANDLE ep_h tduffy> goto bail; tduffy> } tduffy> tduffy> - ep_ptr = (DAPL_EP *) ep_handle; tduffy> + ep_ptr = (struct dapl_ep *)ep_handle; tduffy> tduffy> /* tduffy> * Synchronization ok since this buffer is only used for receive tduffy> Index: linux-kernel-resolv/dat-provider/dapl_cr_query.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_cr_query.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_cr_query.c (working copy) tduffy> @@ -36,7 +36,7 @@ tduffy> tduffy> u32 dapl_cr_query(DAT_CR_HANDLE cr_handle, struct dat_cr_param *cr_param) tduffy> { tduffy> - DAPL_CR *cr_ptr; tduffy> + struct dapl_cr *cr_ptr; tduffy> u32 status; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> @@ -53,7 +53,7 @@ u32 dapl_cr_query(DAT_CR_HANDLE cr_handl tduffy> goto bail; tduffy> } tduffy> tduffy> - cr_ptr = (DAPL_CR *) cr_handle; tduffy> + cr_ptr = (struct dapl_cr *)cr_handle; tduffy> tduffy> /* obtain the remote IP address */ tduffy> (void) dapl_ib_cm_remote_addr((DAT_HANDLE) cr_handle, tduffy> @@ -64,6 +64,6 @@ u32 dapl_cr_query(DAT_CR_HANDLE cr_handl tduffy> tduffy> status = DAT_SUCCESS; tduffy> tduffy> - bail: tduffy> +bail: tduffy> return status; tduffy> } tduffy> Index: linux-kernel-resolv/dat-provider/dapl_rsp_query.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_rsp_query.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_rsp_query.c (working copy) tduffy> @@ -40,7 +40,7 @@ tduffy> tduffy> u32 dapl_rsp_query(DAT_RSP_HANDLE rsp_handle, struct dat_rsp_param *rsp_param) tduffy> { tduffy> - DAPL_SP *sp_ptr; tduffy> + struct dapl_sp *sp_ptr; tduffy> u32 status; tduffy> tduffy> if (DAPL_BAD_HANDLE(rsp_handle, DAPL_MAGIC_RSP)) { tduffy> @@ -53,7 +53,7 @@ u32 dapl_rsp_query(DAT_RSP_HANDLE rsp_ha tduffy> goto bail; tduffy> } tduffy> tduffy> - sp_ptr = (DAPL_SP *) rsp_handle; tduffy> + sp_ptr = (struct dapl_sp *)rsp_handle; tduffy> tduffy> /* tduffy> * Fill in the RSP params tduffy> @@ -65,6 +65,6 @@ u32 dapl_rsp_query(DAT_RSP_HANDLE rsp_ha tduffy> tduffy> status = DAT_SUCCESS; tduffy> tduffy> - bail: tduffy> +bail: tduffy> return status; tduffy> } tduffy> Index: linux-kernel-resolv/dat-provider/dapl_pz_free.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_pz_free.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_pz_free.c (working copy) tduffy> @@ -56,7 +56,7 @@ tduffy> */ tduffy> u32 dapl_pz_free(DAT_PZ_HANDLE pz_handle) tduffy> { tduffy> - DAPL_PZ *pz; tduffy> + struct dapl_pz *pz; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_pz_free (%p)\n", pz_handle); tduffy> @@ -67,7 +67,7 @@ u32 dapl_pz_free(DAT_PZ_HANDLE pz_handle tduffy> goto bail; tduffy> } tduffy> tduffy> - pz = (DAPL_PZ *) pz_handle; tduffy> + pz = (struct dapl_pz *)pz_handle; tduffy> tduffy> if (0 != atomic_read(&pz->pz_ref_count)) { tduffy> dat_status = tduffy> Index: linux-kernel-resolv/dat-provider/dapl_ep_set_watermark.c tduffy> =================================================================== tduffy> --- linux-kernel-resolv/dat-provider/dapl_ep_set_watermark.c (revision 2431) tduffy> +++ linux-kernel-resolv/dat-provider/dapl_ep_set_watermark.c (working copy) tduffy> @@ -63,13 +63,13 @@ u32 dapl_ep_set_watermark(DAT_EP_HANDLE tduffy> int soft_high_watermark, tduffy> int hard_high_watermark) tduffy> { tduffy> - DAPL_EP *ep_ptr; tduffy> + struct dapl_ep *ep_ptr; tduffy> u32 dat_status = DAT_SUCCESS; tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_ep_set_watermark (%p, %d, %d)\n", tduffy> ep_handle, soft_high_watermark, hard_high_watermark); tduffy> tduffy> - ep_ptr = (DAPL_EP *) ep_handle; tduffy> + ep_ptr = (struct dapl_ep *)ep_handle; tduffy> tduffy> /* tduffy> * Verify parameter & state tduffy> From sean.hefty at intel.com Fri May 20 21:23:58 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Fri, 20 May 2005 21:23:58 -0700 Subject: [openib-general] in need of a simple ulp In-Reply-To: <428E876B.8070704@linuxmachines.com> Message-ID: >> In this case you won't know what's on a cqe until you poll the >> completion. > >The cmpost test only poll's on the completion on the client side. I'll >change it to run ib_poll_cq() on the server also. Now I'm so confused >I'm not sure why I wanted this in the first place though :) Both the server and client sides should call the internal routine poll_cqs(), which will remove the CQ entries. >Another thing I've noticed is that I can't seem to get comp_handler() or >event_handler() to run. You pass them into ib_create_cq(), but perhaps >this isn't an implemented feature or some other things must be set for >this to work. Cmpost is brain-dead in this respect. The event_handler would be for things like asynchronous errors, which cmpost doesn't care about. The comp_handler is useful for most apps, but you need to arm the CQ for events by calling ib_req_notify_cq() or ib_req_ncomp_notif() first. Once a completion occurs, its comp_handler would be invoked, where an app would typically call ib_poll_cq() - ib_req_notify_cq() - ib_poll_cq(). The second poll operation is needed to avoid a race condition where a completion could occur after the last poll, but before the CQ can be rearmed. - Sean From tduffy at sun.com Fri May 20 22:59:46 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 20 May 2005 22:59:46 -0700 Subject: [openib-general] kDAPL needs less files Message-ID: <1116655186.16690.15.camel@duffman> I would be happy to submit patches, but I am not sure that would be the best way to do this. I think kdapl (dat-provider) should have less .c files, that all the functionality of a given subsystem (ia, ep, ep, etc.) should all be in one .c file instead of lots of little files. There are ones with only one function in them. So, James, I don't know if you agree, but if you do, how would you like to proceed? Thanks, -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From mst at mellanox.co.il Sat May 21 22:19:51 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Sun, 22 May 2005 08:19:51 +0300 Subject: [openib-general] Re: perftest on i386 In-Reply-To: <20050520145944.GZ15688@aon.at> References: <20050518013821.GO1078@esmail.cup.hp.com> <20050518135130.GE25713@mellanox.co.il> <20050518180620.GA6004@esmail.cup.hp.com> <20050518182448.GC7553@aon.at> <20050518184436.GA27680@mellanox.co.il> <20050518185631.GE7553@aon.at> <20050520143558.GY15688@aon.at> <20050520145944.GZ15688@aon.at> Message-ID: <20050522051951.GA8522@mellanox.co.il> Quoting r. Bernhard Fischer : > Subject: perftest on i386 > > Hi, > > On i386 UP, i see odd results during runs. The machine is basically > idle. > > The worst latency is always somewhere around exchange 12-80. > Is this expected and if not, does somebody have an idea what may cause > this? > > Thank you, scheduler moving one of the tasks (server or client) around. If you have an SMP machine, you'll get more stable results using taskset. -- MST - Michael S. Tsirkin From mst at mellanox.co.il Sat May 21 22:33:51 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Sun, 22 May 2005 08:33:51 +0300 Subject: [openib-general] Re: performance counters in /sys In-Reply-To: <20050520164843.GC14411@esmail.cup.hp.com> References: <35EA21F54A45CB47B879F21A91F4862F581222@taurus.voltaire.com> <1116539301.4479.255.camel@hal.voltaire.com> <428DC946.5080207@hp.com> <20050520161146.GB14411@esmail.cup.hp.com> <20050520164843.GC14411@esmail.cup.hp.com> Message-ID: <20050522053351.GB8522@mellanox.co.il> Quoting r. Grant Grundler : > Subject: Re: performance counters in /sys > > On Fri, May 20, 2005 at 10:20:29AM -0600, Ronald G. Minnich wrote: > > yes, that is the accepted thing nowadays. And, boy, is it SLOW. SLOW, > > SLOW, SLOW. > > Open file, read ONE value, close file. Takes a long time. So we've seen > > that for a number of values you really can't sample at interesting rates. > > Exactly. That's Mark's gripe too. > > > Of course, you can speed it up a bit -- keep your files open, right? > > open once, read, fseek 0, read, ... > > Except then, if you need to unload the module, well, you can't: you have > > the /sys files open and the module can't really be unloaded. > > That's good, IMHO. It will also prevent hotplug from working. So no, I dont think its good. > Unloading the HCA driver will disable the device. > On ia64 platforms (and probably PPC too) will lead to hard fail on the > next MMIO read since the device won't respond to MMIO space accesses. > > "lsof /sys/class/infiniband" would help track down the monitoring tool. Who would do MMIO reads if the driver is unloaded? -- MST - Michael S. Tsirkin From mst at mellanox.co.il Sun May 22 02:34:58 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Sun, 22 May 2005 12:34:58 +0300 Subject: diff-perftest-08 [was: Re: [openib-general] Re: Re: diff-perftest-07 replace pp_get_local_lid()] In-Reply-To: <20050520143558.GY15688@aon.at> References: <20050518013821.GO1078@esmail.cup.hp.com> <20050518135130.GE25713@mellanox.co.il> <20050518180620.GA6004@esmail.cup.hp.com> <20050518182448.GC7553@aon.at> <20050518184436.GA27680@mellanox.co.il> <20050518185631.GE7553@aon.at> <20050520143558.GY15688@aon.at> Message-ID: <20050522093458.GA9249@mellanox.co.il> Quoting r. Bernhard Fischer : > Subject: diff-perftest-08 [was: Re: [openib-general] Re: Re: diff-perftest-07 replace pp_get_local_lid()] > > Looking at the code, 1 is the default and not 4096 as advertised (see > attached patch to reflect default of 1 in usage). > > o Change usage() to match the actual default of size. > o sparse warning amout non-ANSI function decl of get_cpu_mhz(), > whitespace in printf while at it. > o Check arguments -s and -t > o Trim some whitespace > Signed-off-by: Bernhard Fischer I applied these. > o Add printing Size, TX-depth and Iters to print_report(). > Without a short description of the workload, it is bit hard to > interpret the results if one cannot see the command-line used to run > the tests. Why not print out the command line? Further, I'd like to avoid passing the context to print_report. But I wouldnt object to -V (--verbose) flag: when set, print all the parameters out, and do it in main, right after they are parsed; further it might be a good idea to print the connection data only when this flag is set. > o Remove odd 'write(sockfd,"foo"...' (What was that good for?) Its actually useful for debug (when there's a bug and server/client get out of sync, instead of blocking forever, one side gets woken with a wrong message so it can detect this and exit). Anyway, eventually we have to replace all this code: Arbitrarily reuse exch_dest for this purpose. */ if (servername) { rem_dest = pp_client_exch_dest(sockfd, &my_dest); } else { rem_dest = pp_server_exch_dest(sockfd, &my_dest); } write(sockfd, "done", sizeof "done"); with each side sending a message saying the qp is ready to receive, and waiting for the other side to do so. > I can specify -s 940 as the maximum, using 941 gives me "cannot create > QP" in rdma_lat.c. Is this expected? > What are the contraints and limits > for size here? Right. Thats because the test currently uses inline data unconditionally. And *that* is because the maximum legal inline data size cant be queried from the device yet. Need to fix that in uverbs: there does not seem to exist a way now to query hca. > > Quick unrelated question: > Is there any limit on the number of cqe's i can pass to mthca_create_cq? > I initially tried 40960 or 4096 but that did not work. I'm now using 64 > but would like to use more than those in the future.. > > I would expect 40960 or 4096 to work. Roland? What am I missing? -- MST - Michael S. Tsirkin From mst at mellanox.co.il Sun May 22 02:40:37 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Sun, 22 May 2005 12:40:37 +0300 Subject: [openib-general] libibverbs and max inline data Message-ID: <20050522094037.GC9249@mellanox.co.il> Hi! I tried to figure out how much inline data a qp can support, to know what max_inline_data value to put in the qp. However, there doesnt seem to exist a way to do that, currently. Roland, what is the right way to do that? Maybe the driver should be changed to trim max_inline_data to the maximum possible value? Thanks, -- MST - Michael S. Tsirkin From amced at doneasy.com Sun May 22 05:07:02 2005 From: amced at doneasy.com (Lottie Colvin) Date: Sun, 22 May 2005 11:07:02 -0100 Subject: [openib-general] Your account #0M7434 Message-ID: Hello, We tried contacting you awhile ago about your low interest morta(ge rate.= You have qualified for the lowest rate in years... You could get over $380,000 for as little as $500 a month! Ba(d credit? Doesn't matter, low rates are fixed no matter what! To get a free, no obli,gation consultation click below: http://www.j0ys.net/sign.asp Best Regards, Nichole Cortez to be remov(ed: http://www.j0ys.net/gone.asp this process takes one week, so please be patient. we do our best to take your email/s off but you have to fill out a rem/ove or else you will continue to recieve email/s. From halr at voltaire.com Sun May 22 08:49:45 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 22 May 2005 11:49:45 -0400 Subject: [openib-general] Re: [openib-commits] r2426 - gen2/users/jlentini/linux-kernel/dat-provider In-Reply-To: <20050520235642.GE15688@aon.at> References: <20050520220508.CB9A12283D9@openib.ca.sandia.gov> <20050520235642.GE15688@aon.at> Message-ID: <1116776985.4481.5185.camel@hal.voltaire.com> On Fri, 2005-05-20 at 19:56, Bernhard Fischer wrote: > > /* maintenence fields */ > maintenance. > also in: > users/jlentini/userspace/dapl/include/dapl.h > > boolean_t listening; /* PSP is registered & active */ > >- ib_cm_srvc_handle_t cm_srvc_handle; /* Used by CM */ > >+ struct ib_cm_id *cm_srvc_handle; /* Used by CM */ > > DAPL_LLIST_HEAD cr_list_head; /* CR pending queue */ > > int cr_list_count; /* count of CRs on queue */ > > }; > >--- gen2/users/jlentini/linux-kernel/dat-provider/dapl_evd_util.c 2005-05-20 17:22:17 UTC (rev 2425) > >+++ gen2/users/jlentini/linux-kernel/dat-provider/dapl_evd_util.c 2005-05-20 22:05:07 UTC (rev 2426) > >@@ -331,22 +322,22 @@ > > dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK, > > "\t dapl_evd_dto_callback : CQE \n"); > > dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK, > >- "\t\t work_req_id %lli\n", DAPL_GET_CQE_WRID(cqe_ptr)); > >- if (DAPL_GET_CQE_STATUS(cqe_ptr) == 0) { > >- if (DAPL_GET_CQE_OPTYPE(cqe_ptr) == OP_RECEIVE) { > >+ "\t\t work_req_id %lli\n", cqe_ptr->wr_id); > [[:space:]] --------------^ ? > >+ if (cqe_ptr->status == 0) { > >+ if (cqe_ptr->opcode == IB_WC_RECV) { > > dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK, > >- "\t\t op_type: OP_RECEIVE\n"); > >+ "\t\t op_type: IB_WC_RECV\n"); > ditto > > dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK, > > "\t\t bytes_num %d\n", > ditto > >- DAPL_GET_CQE_BYTESNUM(cqe_ptr)); > >+ cqe_ptr->byte_len); > > } else { > > dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK, > > "\t\t op_type: %s\n", > ditto > >- optable[DAPL_GET_CQE_OPTYPE(cqe_ptr)]); > >+ optable[cqe_ptr->opcode]); > > } > > } > > dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK, > >- "\t\t status %d\n", DAPL_GET_CQE_STATUS(cqe_ptr)); > >+ "\t\t status %d\n", cqe_ptr->status); > ditto > > dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK, > > "\t >>>>>>>>>>>>>>>>>>>>>>><<<<<<<<<<<<<<<<<<<\n"); > fooooooooooooooooooooooooooooooooooooooooooobar! > merge errors? I'm not following what you mean here. > also in: > gen2/branches/shaharf-ibat/src/userspace/management/libibmad/src/rpc.c > gen2/ulps/iser/make.conf > gen2/users/jlentini/linux-kernel/dat-provider/dapl_evd_util.c (multiple > times) > gen2/users/jlentini/linux-kernel/dat-provider/dapl_rsp_free.c (multiple > gen2/users/jlentini/linux-kernel/dat-provider/dapl_rsp_create.c > times) > gen2/users/jlentini/linux-kernel/dat-provider/dapl_openib_cm.c (multiple > times) > gen2/users/jlentini/userspace/dapl/common/dapl_evd_util.c (ditto) > gen2/users/jlentini/userspace/dapl/common/dapl_rsp_free.c (ditto) > gen2/users/jlentini/userspace/dapl/common/dapl_rsp_create.c (ditto) > gen2/trunk/src/userspace/management/libibmad/src/rpc.c What is the problem with these ? > > #endif > >@@ -667,21 +657,21 @@ > > > > #ifdef DAPL_DBG > > /* Current gen2 mthca is not setting the opcode in seccesful cqe */ > >- /* The opcode will be OP_SEND or OP_RECEIVE acording the is_send bit */ > >+ /* The opcode will be IB_WC_SEND or IB_WC_RECV acording the is_send bit */ > "acording" -enoparse: s/acord/accord/g > > also in: > gen2/branches/shaharf-ibat/src/userspace/management/osm/opensm/osm_state_mgr.c > gen2/branches/roland-uverbs/src/userspace/management/osm/opensm/osm_sa_mcmember_record.c The above 2 don't matter (not worth the effort to change them). > gen2/utils/src/linux-user/ibdm/datamodel/ibdm.i > gen2/utils/src/linux-user/IBMgtSim/src/ibdm.i > gen2/utils/src/linux-user/IBMgtSim/utils/RunSimTest > there: callabcks: -enoparse; These files are owned by Eitan from Mellanox. > continue with acord: > gen2/users/jlentini/linux-kernel/dat-provider/dapl_evd_util.c > gen2/trunk/src/userspace/management/osm/opensm/osm_state_mgr.c Fixed (osm_state_mgr.c) > "calla" -enoparse: s/calla/call > gen2/utils/src/linux-user/IBMgtSim/utils/RunSimTest This file is owned by Eitan from Mellanox. > gen2/ulps/iser/include/iser_api.h Fixed. > gen2/trunk/src/userspace/management/osm/libvendor/osm_vendor_mlx_ts.c > gen2/trunk/src/userspace/management/osm/libvendor/osm_vendor_ts.c > gen2/trunk/src/userspace/management/osm/libvendor/osm_vendor_mlx_ts_anafa.c Fixed. > [If needed (i.e. if i'm aware of any request to do so) i'll expand those..] > > > /* We can not check the following assert for now */ > [skip] > >@@ -184,19 +184,8 @@ > > } > > } > > /* force the EP into error state to force flush all posted DTOs. */ > >- { > >- struct dat_ep_attr ep_attr; > >- struct dat_named_attr ep_state; > >+ (void) dapl_modify_qp_state_to_error(ep_ptr->qp_handle); > > sun will hopefuly fix all of > egrep -ri "(^[[:space:]]*\(void\))" > openib.gen2/upstream/gen2/trunk/|egrep -v "svn-(text|base)" > so i won't comment on that single occurance above ;) From roland at topspin.com Sun May 22 12:30:06 2005 From: roland at topspin.com (Roland Dreier) Date: Sun, 22 May 2005 12:30:06 -0700 Subject: [openib-general] Re: libibverbs and max inline data References: <20050522094037.GC9249@mellanox.co.il> Message-ID: <52y8a75a9t.fsf@topspin.com> Michael> Hi! I tried to figure out how much inline data a qp can Michael> support, to know what max_inline_data value to put in the Michael> qp. However, there doesnt seem to exist a way to do Michael> that, currently. I think you're right. Michael> Maybe the driver should be changed to trim Michael> max_inline_data to the maximum possible value? Yes, that would probably make sense. In a sense that's what the kernel driver already does -- it always clips the max_inline_data request from the consumer down to 0. Independent of that we could add a field to the device properties struct (and implement ibv_query_device() for userspace). - R. From roland at topspin.com Sun May 22 12:30:07 2005 From: roland at topspin.com (Roland Dreier) Date: Sun, 22 May 2005 12:30:07 -0700 Subject: [openib-general] [PATCH] auto*() && ./configure -!ac_default_prefix References: <20050520215957.GD15688@aon.at> Message-ID: <52sm0f5a9s.fsf@topspin.com> Bernhard> * trunk/src/userspace/lib{mthca,ibcm}/configure.in: if Bernhard> prefix != ac_default_prefix use prefix for checks. Bernhard> fixup for e.g. SuSE when giving a non standart --prefix; I don't think we want to do this. I don't know of other packages that fiddle with things this way. It seems perfectly fine to pass the appropriate CPPFLAGS and/or LDFLAGS when running configure. In fact that has the advantage of working with --includedir or --libdir as well. Bernhard> Should be fixed upstream, non-debian will need a while Bernhard> to pick those up, though; therefore, please Bernhard> consider.. :-/ I don't understand this last bit. Who is the upstream? - R. From Mark.Seger at hp.com Sun May 22 15:36:47 2005 From: Mark.Seger at hp.com (Mark Seger) Date: Sun, 22 May 2005 18:36:47 -0400 Subject: [openib-general] Re: performance counters in /sys In-Reply-To: <20050522053351.GB8522@mellanox.co.il> References: <35EA21F54A45CB47B879F21A91F4862F581222@taurus.voltaire.com> <1116539301.4479.255.camel@hal.voltaire.com> <428DC946.5080207@hp.com> <20050520161146.GB14411@esmail.cup.hp.com> <20050520164843.GC14411@esmail.cup.hp.com> <20050522053351.GB8522@mellanox.co.il> Message-ID: <4291097F.8070404@hp.com> Michael S. Tsirkin wrote: >Quoting r. Grant Grundler : > > >>Subject: Re: performance counters in /sys >> >>On Fri, May 20, 2005 at 10:20:29AM -0600, Ronald G. Minnich wrote: >> >> >>>yes, that is the accepted thing nowadays. And, boy, is it SLOW. SLOW, >>>SLOW, SLOW. >>>Open file, read ONE value, close file. Takes a long time. So we've seen >>>that for a number of values you really can't sample at interesting rates. >>> >>> >>Exactly. That's Mark's gripe too. >> >> I guess the thing that has me mystified about all this is I can certainly appreciate the potential 'goodness' of having 1 var/file for user oriented access but perhaps one of the better examples of why this is just a bad idea for programmatic access is the individual process stats. Is the implication of this that some day those too would be moved to /sys as one stat per file?!? Can you imaging trying to run top or ps if that were to happen? -mark From iod00d at hp.com Sun May 22 20:58:30 2005 From: iod00d at hp.com (Grant Grundler) Date: Sun, 22 May 2005 20:58:30 -0700 Subject: [openib-general] Re: performance counters in /sys In-Reply-To: <20050522053351.GB8522@mellanox.co.il> References: <35EA21F54A45CB47B879F21A91F4862F581222@taurus.voltaire.com> <1116539301.4479.255.camel@hal.voltaire.com> <428DC946.5080207@hp.com> <20050520161146.GB14411@esmail.cup.hp.com> <20050520164843.GC14411@esmail.cup.hp.com> <20050522053351.GB8522@mellanox.co.il> Message-ID: <20050523035830.GB22945@esmail.cup.hp.com> On Sun, May 22, 2005 at 08:33:51AM +0300, Michael S. Tsirkin wrote: > > That's good, IMHO. > > It will also prevent hotplug from working. So no, I dont think its good. I agree that hotplug should work. I'm just pointing out that keeping the file open should prevent the driver from unloading. > > Unloading the HCA driver will disable the device. > > On ia64 platforms (and probably PPC too) will lead to hard fail on the > > next MMIO read since the device won't respond to MMIO space accesses. > > > > "lsof /sys/class/infiniband" would help track down the monitoring tool. > > Who would do MMIO reads if the driver is unloaded? No one if the /sys file was closed like we want them too. I'm just pointing out the counters must point at an enabled device for as long as any one of the /sys files was open. thanks, grant From mst at mellanox.co.il Sun May 22 22:19:19 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Mon, 23 May 2005 08:19:19 +0300 Subject: [openib-general] Re: libibverbs and max inline data In-Reply-To: <52y8a75a9t.fsf@topspin.com> References: <20050522094037.GC9249@mellanox.co.il> <52y8a75a9t.fsf@topspin.com> Message-ID: <20050523051919.GA17349@mellanox.co.il> Quoting r. Roland Dreier : > Subject: Re: libibverbs and max inline data > > Michael> Hi! I tried to figure out how much inline data a qp can > Michael> support, to know what max_inline_data value to put in the > Michael> qp. However, there doesnt seem to exist a way to do > Michael> that, currently. > > I think you're right. > > Michael> Maybe the driver should be changed to trim > Michael> max_inline_data to the maximum possible value? > > Yes, that would probably make sense. In a sense that's what the > kernel driver already does -- it always clips the max_inline_data > request from the consumer down to 0. Right. So the driver could check the context to know what to return: 0 or the actual value. > Independent of that we could add a field to the device properties > struct (and implement ibv_query_device() for userspace). > > - R. -- MST - Michael S. Tsirkin From mst at mellanox.co.il Sun May 22 22:28:28 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Mon, 23 May 2005 08:28:28 +0300 Subject: [openib-general] Re: performance counters in /sys In-Reply-To: <4291097F.8070404@hp.com> References: <35EA21F54A45CB47B879F21A91F4862F581222@taurus.voltaire.com> <1116539301.4479.255.camel@hal.voltaire.com> <428DC946.5080207@hp.com> <20050520161146.GB14411@esmail.cup.hp.com> <20050520164843.GC14411@esmail.cup.hp.com> <20050522053351.GB8522@mellanox.co.il> <4291097F.8070404@hp.com> Message-ID: <20050523052827.GB17349@mellanox.co.il> Quoting r. Mark Seger : > > Quoting r. Grant Grundler : > > > > >On Fri, May 20, 2005 at 10:20:29AM -0600, Ronald G. Minnich wrote: > > > > > > > > > > yes, that is the accepted thing nowadays. And, boy, is it SLOW. SLOW, > > > > SLOW, SLOW. > > > > Open file, read ONE value, close file. Takes a long time. So we've seen > > > > that for a number of values you really can't sample at interesting > > > > rates. > > > > > > > > > > Exactly. That's Mark's gripe too. > > I guess the thing that has me mystified about all this is I can > certainly appreciate the potential 'goodness' of having 1 var/file for > user oriented access but perhaps one of the better examples of why this > is just a bad idea for programmatic access is the individual process > stats. Is the implication of this that some day those too would be > moved to /sys as one stat per file?!? Can you imaging trying to run top > or ps if that were to happen? > > -mark Does someone care enough to post a benchmark? -- MST - Michael S. Tsirkin From roland at topspin.com Mon May 23 07:14:30 2005 From: roland at topspin.com (Roland Dreier) Date: Mon, 23 May 2005 07:14:30 -0700 Subject: [openib-general] Re: [PATCH] user_mad: Support RMPP on send side References: <1116538456.4479.236.camel@hal.voltaire.com> Message-ID: <52hdgu2fnd.fsf@topspin.com> Looks OK to commit to me. - R. From roland at topspin.com Mon May 23 07:29:25 2005 From: roland at topspin.com (Roland Dreier) Date: Mon, 23 May 2005 07:29:25 -0700 Subject: [openib-general] Re: [RFC] [PATCH] user_mad: Support RMPP on send side References: <1115845653.4490.188.camel@localhost.localdomain> <5264xgf85t.fsf@topspin.com> <1116595380.4478.65.camel@hal.voltaire.com> Message-ID: <52br722eyi.fsf@topspin.com> Hal> It appears to me that there is nothing that enforces the man Hal> page behavior above in Linux so this is a "convention". Is Hal> this something we can take advantage of or do we need to Hal> require the additional call to get the buffer length ? Something I was thinking a little while ago was that the character device interface for userspace MADs might have been a mistake. Having a new datagram socket family would solve this problem quite nicely -- a consumer could pass MSG_PEEK and/or MSG_TRUNC to recv() to get the true length of a datagram. However it's a little tricky to see a really clean way for userspace to use this to handle arbitrary sized RMPP receives. - R. From roland at topspin.com Mon May 23 07:29:27 2005 From: roland at topspin.com (Roland Dreier) Date: Mon, 23 May 2005 07:29:27 -0700 Subject: [openib-general] Re: registering read-only memory References: <20050519085208.GC4717@mellanox.co.il> <52ll6bb6ig.fsf@topspin.com> <20050519154938.GA5854@mellanox.co.il> Message-ID: <5264xa2eyg.fsf@topspin.com> Michael> 2. ibv_reg_mr fails. Why is that? Michael> Returns NULL, naturally. ... hmm, I'll take a look. - R. From jlentini at netapp.com Mon May 23 07:32:12 2005 From: jlentini at netapp.com (James Lentini) Date: Mon, 23 May 2005 10:32:12 -0400 (EDT) Subject: [openib-general] Re: [openib-commits] r2426 - gen2/users/jlentini/linux-kernel/dat-provider In-Reply-To: <20050520235642.GE15688@aon.at> References: <20050520220508.CB9A12283D9@openib.ca.sandia.gov> <20050520235642.GE15688@aon.at> Message-ID: Bernhard, Thank you for pointing these items out. Comments below: On Sat, 21 May 2005, Bernhard Fischer wrote: > On Fri, May 20, 2005 at 03:05:08PM -0700, jlentini at openib.org wrote: >> Author: jlentini >> Date: 2005-05-20 15:05:07 -0700 (Fri, 20 May 2005) >> New Revision: 2426 > > Just tersely parsing, no mind to delve into any API implications > whatsoever. [Late, no motivation to think, nor even read, not even > speaking of writing anything publically. I'm over quota with my > telephone-IP provider since 11th, which nag's our scanty pocket-money > crucially, etc] > >> Modified: gen2/users/jlentini/linux-kernel/dat-provider/dapl.h >> =================================================================== >> --- gen2/users/jlentini/linux-kernel/dat-provider/dapl.h 2005-05-20 17:22:17 UTC (rev 2425) >> +++ gen2/users/jlentini/linux-kernel/dat-provider/dapl.h 2005-05-20 22:05:07 UTC (rev 2426) >> @@ -144,6 +140,24 @@ >> * * >> *********************************************************************/ >> >> +typedef void (*ib_async_handler_t) (struct ib_event *, void *); >> + >> +/* >> + * ib_hca_transport_t structure. Define fields specific to this >> + * provider implementation necessary to maintain the HCA >> + * >> + * NOTE: null_ib_hca_handle is strictly internal to IB. DAPL allows a >> + * NULL DTO EVD handle, but IB does not. Create a CQ under the >> + * hood and make sure an error is generated if the user every > "every tries to post"? -enoparse; please be more specific -- simpler. > 'ever' would still be obfuscated. IOW please rephrase and put > abbreviations into the wiki. Agreed. I'll rephrase that. I can update the WIKI with DAPL acronyms. >> + * tries to post, by setting the WQ length to 0 in ep_create >> + * and/or ep_modify. >> + */ >> @@ -394,7 +403,7 @@ >> >> /* maintenence fields */ > maintenance. > also in: > users/jlentini/userspace/dapl/include/dapl.h >> boolean_t listening; /* PSP is registered & active */ >> - ib_cm_srvc_handle_t cm_srvc_handle; /* Used by CM */ >> + struct ib_cm_id *cm_srvc_handle; /* Used by CM */ >> DAPL_LLIST_HEAD cr_list_head; /* CR pending queue */ >> int cr_list_count; /* count of CRs on queue */ >> }; >> --- gen2/users/jlentini/linux-kernel/dat-provider/dapl_evd_util.c 2005-05-20 17:22:17 UTC (rev 2425) >> +++ gen2/users/jlentini/linux-kernel/dat-provider/dapl_evd_util.c 2005-05-20 22:05:07 UTC (rev 2426) >> @@ -331,22 +322,22 @@ >> dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK, >> "\t dapl_evd_dto_callback : CQE \n"); >> dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK, >> - "\t\t work_req_id %lli\n", DAPL_GET_CQE_WRID(cqe_ptr)); >> - if (DAPL_GET_CQE_STATUS(cqe_ptr) == 0) { >> - if (DAPL_GET_CQE_OPTYPE(cqe_ptr) == OP_RECEIVE) { >> + "\t\t work_req_id %lli\n", cqe_ptr->wr_id); > [[:space:]] --------------^ ? Which space are you referring to? The one between \t\t and work_req_id? >> + if (cqe_ptr->status == 0) { >> + if (cqe_ptr->opcode == IB_WC_RECV) { >> dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK, >> - "\t\t op_type: OP_RECEIVE\n"); >> + "\t\t op_type: IB_WC_RECV\n"); > ditto >> dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK, >> "\t\t bytes_num %d\n", > ditto >> - DAPL_GET_CQE_BYTESNUM(cqe_ptr)); >> + cqe_ptr->byte_len); >> } else { >> dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK, >> "\t\t op_type: %s\n", > ditto >> - optable[DAPL_GET_CQE_OPTYPE(cqe_ptr)]); >> + optable[cqe_ptr->opcode]); >> } >> } >> dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK, >> - "\t\t status %d\n", DAPL_GET_CQE_STATUS(cqe_ptr)); >> + "\t\t status %d\n", cqe_ptr->status); > ditto >> dapl_dbg_log(DAPL_DBG_TYPE_CALLBACK, >> "\t >>>>>>>>>>>>>>>>>>>>>>><<<<<<<<<<<<<<<<<<<\n"); > fooooooooooooooooooooooooooooooooooooooooooobar! > merge errors? No that text, ">>>>>>>>>>>>>>>>>>>>>>><<<<<<<<<<<<<<<<<<<" is actually part of a debugging print out. I agree that these debug statements are of debatable value. If a high volume of events are being generated, the amount of data produced by these is difficult to use. > also in: > gen2/branches/shaharf-ibat/src/userspace/management/libibmad/src/rpc.c > gen2/ulps/iser/make.conf > gen2/users/jlentini/linux-kernel/dat-provider/dapl_evd_util.c (multiple > times) > gen2/users/jlentini/linux-kernel/dat-provider/dapl_rsp_free.c (multiple > gen2/users/jlentini/linux-kernel/dat-provider/dapl_rsp_create.c > times) > gen2/users/jlentini/linux-kernel/dat-provider/dapl_openib_cm.c (multiple > times) > gen2/users/jlentini/userspace/dapl/common/dapl_evd_util.c (ditto) > gen2/users/jlentini/userspace/dapl/common/dapl_rsp_free.c (ditto) > gen2/users/jlentini/userspace/dapl/common/dapl_rsp_create.c (ditto) > gen2/trunk/src/userspace/management/libibmad/src/rpc.c > >> #endif >> @@ -667,21 +657,21 @@ >> >> #ifdef DAPL_DBG >> /* Current gen2 mthca is not setting the opcode in seccesful cqe */ >> - /* The opcode will be OP_SEND or OP_RECEIVE acording the is_send bit */ >> + /* The opcode will be IB_WC_SEND or IB_WC_RECV acording the is_send bit */ > "acording" -enoparse: s/acord/accord/g > > also in: > gen2/branches/shaharf-ibat/src/userspace/management/osm/opensm/osm_state_mgr.c > gen2/branches/roland-uverbs/src/userspace/management/osm/opensm/osm_sa_mcmember_record.c > gen2/utils/src/linux-user/ibdm/datamodel/ibdm.i > gen2/utils/src/linux-user/IBMgtSim/src/ibdm.i > gen2/utils/src/linux-user/IBMgtSim/utils/RunSimTest > there: callabcks: -enoparse; > continue with acord: > gen2/users/jlentini/linux-kernel/dat-provider/dapl_evd_util.c > gen2/trunk/src/userspace/management/osm/opensm/osm_state_mgr.c I've fixed my error: gen2/users/jlentini/linux-kernel/dat-provider/dapl_evd_util.c The others belong to Roland, Shahar, and Eitan. > > "calla" -enoparse: s/calla/call > gen2/utils/src/linux-user/IBMgtSim/utils/RunSimTest > gen2/ulps/iser/include/iser_api.h > gen2/trunk/src/userspace/management/osm/libvendor/osm_vendor_mlx_ts.c > gen2/trunk/src/userspace/management/osm/libvendor/osm_vendor_ts.c > gen2/trunk/src/userspace/management/osm/libvendor/osm_vendor_mlx_ts_anafa.c > > [If needed (i.e. if i'm aware of any request to do so) i'll expand those..] Ditto. > >> /* We can not check the following assert for now */ > [skip] >> @@ -184,19 +184,8 @@ >> } >> } >> /* force the EP into error state to force flush all posted DTOs. */ >> - { >> - struct dat_ep_attr ep_attr; >> - struct dat_named_attr ep_state; >> + (void) dapl_modify_qp_state_to_error(ep_ptr->qp_handle); > > sun will hopefuly fix all of > egrep -ri "(^[[:space:]]*\(void\))" > openib.gen2/upstream/gen2/trunk/|egrep -v "svn-(text|base)" > so i won't comment on that single occurance above ;) I didn't realize the convention was to not have spaces before a cast. This isn't in Documentation/CodingStyle. I know I've been adding these here and there. I'll fix these as I see them. From jlentini at netapp.com Mon May 23 07:53:43 2005 From: jlentini at netapp.com (James Lentini) Date: Mon, 23 May 2005 10:53:43 -0400 (EDT) Subject: [openib-general] Re: kDAPL needs less files In-Reply-To: <1116655186.16690.15.camel@duffman> References: <1116655186.16690.15.camel@duffman> Message-ID: I agree Tom. I think the easiest way to do this is one subsystem at a time. To avoid us duplicating work, how responding on this thread with the subsystem(s) you would like to merge. On a somewhat related topic, the DAPL linked list implementation should be removed in favor of the Linux implementation. That would allow us to remove the dapl_llist.c file. james On Fri, 20 May 2005, Tom Duffy wrote: tduffy> I would be happy to submit patches, but I am not sure that would be the tduffy> best way to do this. I think kdapl (dat-provider) should have less .c tduffy> files, that all the functionality of a given subsystem (ia, ep, ep, tduffy> etc.) should all be in one .c file instead of lots of little files. tduffy> There are ones with only one function in them. tduffy> tduffy> So, James, I don't know if you agree, but if you do, how would you like tduffy> to proceed? tduffy> tduffy> Thanks, tduffy> tduffy> -tduffy tduffy> From jlentini at netapp.com Mon May 23 08:07:52 2005 From: jlentini at netapp.com (James Lentini) Date: Mon, 23 May 2005 11:07:52 -0400 (EDT) Subject: [openib-general] Re: libibverbs and max inline data In-Reply-To: <52y8a75a9t.fsf@topspin.com> References: <20050522094037.GC9249@mellanox.co.il> <52y8a75a9t.fsf@topspin.com> Message-ID: On Sun, 22 May 2005, Roland Dreier wrote: > Michael> Hi! I tried to figure out how much inline data a qp can > Michael> support, to know what max_inline_data value to put in the > Michael> qp. However, there doesnt seem to exist a way to do > Michael> that, currently. > > I think you're right. > > Michael> Maybe the driver should be changed to trim > Michael> max_inline_data to the maximum possible value? > > Yes, that would probably make sense. In a sense that's what the > kernel driver already does -- it always clips the max_inline_data > request from the consumer down to 0. ib[v]_create_qp should return an error when an unsupported inline data size is requested. > Independent of that we could add a field to the device properties > struct (and implement ibv_query_device() for userspace). > > - R. > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From rminnich at lanl.gov Mon May 23 08:15:38 2005 From: rminnich at lanl.gov (Ronald G. Minnich) Date: Mon, 23 May 2005 09:15:38 -0600 (MDT) Subject: [openib-general] Re: performance counters in /sys In-Reply-To: <20050523052827.GB17349@mellanox.co.il> References: <35EA21F54A45CB47B879F21A91F4862F581222@taurus.voltaire.com> <1116539301.4479.255.camel@hal.voltaire.com> <428DC946.5080207@hp.com> <20050520161146.GB14411@esmail.cup.hp.com> <20050520164843.GC14411@esmail.cup.hp.com> <20050522053351.GB8522@mellanox.co.il> <4291097F.8070404@hp.com> <20050523052827.GB17349@mellanox.co.il> Message-ID: On Mon, 23 May 2005, Michael S. Tsirkin wrote: > > I guess the thing that has me mystified about all this is I can > > certainly appreciate the potential 'goodness' of having 1 var/file for > > user oriented access but perhaps one of the better examples of why this > > is just a bad idea for programmatic access is the individual process > > stats. Is the implication of this that some day those too would be > > moved to /sys as one stat per file?!? Can you imaging trying to run top > > or ps if that were to happen? > Does someone care enough to post a benchmark? I have one I will try to post, but ... I did create a test program to see if I could run supermon in user mode with no kernel module. Supermon kernel module presents one file in /proc/sys that provides an s-expression containing info on avenrun, enet stats, disk, etc. There is an additional file presnted with an s-expression containing descriptors of what the data means, whether it is bounded, and if bounded, what the min/max is. The module works correctly in the presence of hotplug (s-expressions are your friend). On Linux, we can hand out S-expressions at 12 Khz; on Plan 9, at 30 Khz -- i.e., 12,000 and 30.000 samples/second. Obviously, you don't run that fast, the reason to know the max is to give you an idea of the impact of reading the file at, e.g., 100hz; in the case of supermon, the impact is in essence zero. My simple test program opened the files that the supermon kernel module provides, and on a request for data, did a read() and lseek(0), concatenating the 'single value per file' into an s-expression. Performance was pretty bad, I figure that you really can't sample at more than 1/50 hz (once every 50 seconds) without impacting the cpu performance in a measurable way. Plus, keeeping the files open means you can't unload modules. So, I went to open/read/close for each file, to allow module unloading to work, and of course performance was even worse. One value per file is a fine rule if you don't need high performance data collection. In supermon, we can sample at 100hz with no impact on the apps, and have actually observed mpi collectives in progress in that way. It's pretty neat. One stat per file is not a good idea if you need low-impact, high performance data collection. ron From halr at voltaire.com Mon May 23 08:31:27 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 23 May 2005 11:31:27 -0400 Subject: [openib-general] Re: [RFC] [PATCH] user_mad: Support RMPP on send side In-Reply-To: <52br722eyi.fsf@topspin.com> References: <1115845653.4490.188.camel@localhost.localdomain> <5264xgf85t.fsf@topspin.com> <1116595380.4478.65.camel@hal.voltaire.com> <52br722eyi.fsf@topspin.com> Message-ID: <1116862286.4481.5670.camel@hal.voltaire.com> On Mon, 2005-05-23 at 10:29, Roland Dreier wrote: > Hal> It appears to me that there is nothing that enforces the man > Hal> page behavior above in Linux so this is a "convention". Is > Hal> this something we can take advantage of or do we need to > Hal> require the additional call to get the buffer length ? > > Something I was thinking a little while ago was that the character > device interface for userspace MADs might have been a mistake. Having > a new datagram socket family would solve this problem quite nicely -- > a consumer could pass MSG_PEEK and/or MSG_TRUNC to recv() to get the > true length of a datagram. One could always add an ioctl to the character device to do this if it is unacceptable to change the convention for this character device. Changing over to a socket based interface appears to be a longer term approach. Does this have other advantages over the character device based approach ? Are sockets more hot swap friendly ? > However it's a little tricky to see a > really clean way for userspace to use this to handle arbitrary sized > RMPP receives. Not sure what you are referring to here. Wouldn't the returned size be used for the size on the subsequent read ? Isn't the issue still the same whether it is socket or character device based: how is a read with a too short length handled ? -- Hal From mst at mellanox.co.il Mon May 23 08:45:58 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Mon, 23 May 2005 18:45:58 +0300 Subject: [openib-general] Re: performance counters in /sys In-Reply-To: References: <35EA21F54A45CB47B879F21A91F4862F581222@taurus.voltaire.com> <1116539301.4479.255.camel@hal.voltaire.com> <428DC946.5080207@hp.com> <20050520161146.GB14411@esmail.cup.hp.com> <20050520164843.GC14411@esmail.cup.hp.com> <20050522053351.GB8522@mellanox.co.il> <4291097F.8070404@hp.com> <20050523052827.GB17349@mellanox.co.il> Message-ID: <20050523154558.GJ9732@mellanox.co.il> Quoting r. Ronald G. Minnich : > Subject: Re: performance counters in /sys > > > > On Mon, 23 May 2005, Michael S. Tsirkin wrote: > > > > I guess the thing that has me mystified about all this is I can > > > certainly appreciate the potential 'goodness' of having 1 var/file for > > > user oriented access but perhaps one of the better examples of why this > > > is just a bad idea for programmatic access is the individual process > > > stats. Is the implication of this that some day those too would be > > > moved to /sys as one stat per file?!? Can you imaging trying to run top > > > or ps if that were to happen? > > > > Does someone care enough to post a benchmark? > > I have one I will try to post Good, it shall then be possible to profile it and look for a solution. > But ... I did create a test program to see > if I could run supermon in user mode with no kernel module. Supermon > kernel module presents one file in /proc/sys that provides an s-expression > containing info on avenrun, enet stats, disk, etc. There is an additional > file presnted with an s-expression containing descriptors of what the data > means, whether it is bounded, and if bounded, what the min/max is. The > module works correctly in the presence of hotplug (s-expressions are your > friend). > > On Linux, we can hand out S-expressions at 12 Khz; on Plan 9, at 30 Khz -- > i.e., 12,000 and 30.000 samples/second. Obviously, you don't run that > fast, the reason to know the max is to give you an idea of the impact of > reading the file at, e.g., 100hz; in the case of supermon, the impact is > in essence zero. > > My simple test program opened the files that the supermon kernel module > provides, and on a request for data, did a read() and lseek(0), Did you try pread(2)? > concatenating the 'single value per file' into an s-expression. > > Performance was pretty bad, I figure that you really can't sample at more > than 1/50 hz (once every 50 seconds) without impacting the cpu performance > in a measurable way. Plus, keeeping the files open means you can't unload > modules. > > So, I went to open/read/close for each file, to allow module unloading to > work, and of course performance was even worse. By how much? > One value per file is a fine rule if you don't need high performance data > collection. In supermon, we can sample at 100hz with no impact on the > apps, and have actually observed mpi collectives in progress in that way. > It's pretty neat. > > One stat per file is not a good idea if you need low-impact, high > performance data collection. > > ron > Its not immediately obvious from the description above that its the "one value per file" that is the performance bottleneck, is it? For example, I'm not familiar with the s-expression format. Is it possible the extra reformatting/string copying from the read buffuer to s-expression eats the CPU? Well, I guess when we see the benchmark it shall be possible to judge where does the time go. If all of sysfs has a speed problem, the right way seems to solve it and not to add more /proc files as a work-around, as supermon module seems to do. thanks, -- MST - Michael S. Tsirkin From mst at mellanox.co.il Mon May 23 08:48:58 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Mon, 23 May 2005 18:48:58 +0300 Subject: [openib-general] Re: Re: libibverbs and max inline data In-Reply-To: References: <20050522094037.GC9249@mellanox.co.il> <52y8a75a9t.fsf@topspin.com> Message-ID: <20050523154858.GK9732@mellanox.co.il> Quoting r. James Lentini : > Subject: Re: Re: libibverbs and max inline data > > > On Sun, 22 May 2005, Roland Dreier wrote: > > > Michael> Hi! I tried to figure out how much inline data a qp can > > Michael> support, to know what max_inline_data value to put in the > > Michael> qp. However, there doesnt seem to exist a way to do > > Michael> that, currently. > > > >I think you're right. > > > > Michael> Maybe the driver should be changed to trim > > Michael> max_inline_data to the maximum possible value? > > > >Yes, that would probably make sense. In a sense that's what the > >kernel driver already does -- it always clips the max_inline_data > >request from the consumer down to 0. > > ib[v]_create_qp should return an error when an unsupported inline data > size is requested. Why is it important? The user could always check the max_inline_data returned and close the qp if thats too small, but I dont see many users doing this - inline data is mainly a tuning opportunity. > >Independent of that we could add a field to the device properties > >struct (and implement ibv_query_device() for userspace). > > > >- R. -- MST - Michael S. Tsirkin From mst at mellanox.co.il Mon May 23 08:49:38 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Mon, 23 May 2005 18:49:38 +0300 Subject: [openib-general] Re: registering read-only memory In-Reply-To: <5264xa2eyg.fsf@topspin.com> References: <20050519085208.GC4717@mellanox.co.il> <52ll6bb6ig.fsf@topspin.com> <20050519154938.GA5854@mellanox.co.il> <5264xa2eyg.fsf@topspin.com> Message-ID: <20050523154938.GL9732@mellanox.co.il> Quoting r. Roland Dreier : > Subject: Re: registering read-only memory > > Michael> 2. ibv_reg_mr fails. Why is that? > > Michael> Returns NULL, naturally. > > ... hmm, I'll take a look. > > - R. > What about the patch that I posted? -- MST - Michael S. Tsirkin From roland at topspin.com Mon May 23 08:54:56 2005 From: roland at topspin.com (Roland Dreier) Date: Mon, 23 May 2005 08:54:56 -0700 Subject: [openib-general] Re: registering read-only memory In-Reply-To: <20050523154938.GL9732@mellanox.co.il> (Michael S. Tsirkin's message of "Mon, 23 May 2005 18:49:38 +0300") References: <20050519085208.GC4717@mellanox.co.il> <52ll6bb6ig.fsf@topspin.com> <20050519154938.GA5854@mellanox.co.il> <5264xa2eyg.fsf@topspin.com> <20050523154938.GL9732@mellanox.co.il> Message-ID: <52y8a6ym27.fsf@topspin.com> Michael> What about the patch that I posted? Adding the "const" makes sense but adding "volatile" doesn't seem good to me. I think it's better for the tiny subset of people who really need "volatile" to add casts. - R. From halr at voltaire.com Mon May 23 08:53:19 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 23 May 2005 11:53:19 -0400 Subject: [openib-general] Re: [PATCH] user_mad: Support RMPP on send side In-Reply-To: <52hdgu2fnd.fsf@topspin.com> References: <1116538456.4479.236.camel@hal.voltaire.com> <52hdgu2fnd.fsf@topspin.com> Message-ID: <1116863599.4481.5714.camel@hal.voltaire.com> On Mon, 2005-05-23 at 10:14, Roland Dreier wrote: > Looks OK to commit to me. Thanks. Applied. -- Hal From mst at mellanox.co.il Mon May 23 09:01:58 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Mon, 23 May 2005 19:01:58 +0300 Subject: [openib-general] Re: registering read-only memory In-Reply-To: <52y8a6ym27.fsf@topspin.com> References: <20050519085208.GC4717@mellanox.co.il> <52ll6bb6ig.fsf@topspin.com> <20050519154938.GA5854@mellanox.co.il> <5264xa2eyg.fsf@topspin.com> <20050523154938.GL9732@mellanox.co.il> <52y8a6ym27.fsf@topspin.com> Message-ID: <20050523160158.GA18721@mellanox.co.il> Quoting r. Roland Dreier : > Subject: Re: registering read-only memory > > Michael> What about the patch that I posted? > > Adding the "const" makes sense but adding "volatile" doesn't seem good > to me. Could you explain why? Since the library never accesses the data, it has no effect except turning off a warning. Volatile says like "this buffer can change without the application writing there", and thats what we have with hardware writing there, no? > I think it's better for the tiny subset of people who really > need "volatile" to add casts. > > - R. > IMO, you cant always do this. If you have a non-volatile variable and cast it to a volatile pointer, compiler is permitted to detect this and ignore the volatile attribute, making the code that polls data incorrect (I'm not claiming gcc is smart enough to do this, though). -- MST - Michael S. Tsirkin From mst at mellanox.co.il Mon May 23 09:05:50 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Mon, 23 May 2005 19:05:50 +0300 Subject: [openib-general] .max_recv_wr = 0 Message-ID: <20050523160550.GB18721@mellanox.co.il> Hi, Roland! I have a test that sets .max_recv_wr = 0 and crashes in mthca_poll_cq, when a send completion arrives. Here's a backtrace of a crash: #0 0x00002aaaab22ad90 in mthca_poll_cq (ibcq=Variable "ibcq" is not available. ) at src/cq.c:317 #1 0x000000000040259d in main (argc=Variable "argc" is not available.) at verbs.h:543 (gdb) frame 0 which is: wc->wr_id = (*cur_qp)->wrid[wqe_index + (*cur_qp)->rq.max]; I suspect rq.max isnt initialised to 0. -- MST - Michael S. Tsirkin From jlentini at netapp.com Mon May 23 09:08:32 2005 From: jlentini at netapp.com (James Lentini) Date: Mon, 23 May 2005 12:08:32 -0400 (EDT) Subject: [openib-general] Re: Re: libibverbs and max inline data In-Reply-To: <20050523154858.GK9732@mellanox.co.il> References: <20050522094037.GC9249@mellanox.co.il> <52y8a75a9t.fsf@topspin.com> <20050523154858.GK9732@mellanox.co.il> Message-ID: On Mon, 23 May 2005, Michael S. Tsirkin wrote: > Quoting r. James Lentini : >> Subject: Re: Re: libibverbs and max inline data >> >> >> On Sun, 22 May 2005, Roland Dreier wrote: >> >>> Michael> Hi! I tried to figure out how much inline data a qp can >>> Michael> support, to know what max_inline_data value to put in the >>> Michael> qp. However, there doesnt seem to exist a way to do >>> Michael> that, currently. >>> >>> I think you're right. >>> >>> Michael> Maybe the driver should be changed to trim >>> Michael> max_inline_data to the maximum possible value? >>> >>> Yes, that would probably make sense. In a sense that's what the >>> kernel driver already does -- it always clips the max_inline_data >>> request from the consumer down to 0. >> >> ib[v]_create_qp should return an error when an unsupported inline data >> size is requested. > > Why is it important? The user could always check the max_inline_data > returned and close the qp if thats too small, but I dont see > many users doing this - inline data is mainly a tuning opportunity. Once there is a way to programmatically determine the max_inline_data value, your proposal would be feasible. Even if it were possible, I still believe that the driver should validate this value. Why have the user specify the ib[v]_qp_cap values if they are going to be ignored? As a rule, the driver should return an error when the user requests a capability/resource that the hardware cannot support. kDAPL was bitten by this last week. The call to ib_create_qp succeed but the value kDAPL was placing in the ib_qp_init_attr's max_inline_data field was too large. As a result, a subsequent qp call failed. If ib_create_qp had returned an error to begin with, it would have been easier to determine the problem. >>> Independent of that we could add a field to the device properties >>> struct (and implement ibv_query_device() for userspace). >>> >>> - R. > > -- > MST - Michael S. Tsirkin > From roland at topspin.com Mon May 23 09:18:12 2005 From: roland at topspin.com (Roland Dreier) Date: Mon, 23 May 2005 09:18:12 -0700 Subject: [openib-general] Re: libibverbs and max inline data In-Reply-To: (James Lentini's message of "Mon, 23 May 2005 12:08:32 -0400 (EDT)") References: <20050522094037.GC9249@mellanox.co.il> <52y8a75a9t.fsf@topspin.com> <20050523154858.GK9732@mellanox.co.il> Message-ID: <52oeb1zzjv.fsf@topspin.com> James> Once there is a way to programmatically determine the James> max_inline_data value, your proposal would be James> feasible. Even if it were possible, I still believe that James> the driver should validate this value. I think you may be misunderstanding the suggestion. The idea is that the driver takes the input value of max_inline_data as a hint, and tries to allocate a QP that supports that value. However, if the value is too large, the driver reduces the value down to the largest value that it can support. In any case, the true amount of inline data supported is returned to the consumer in the qp_cap structure. - R. From mst at mellanox.co.il Mon May 23 09:25:59 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Mon, 23 May 2005 19:25:59 +0300 Subject: [openib-general] Re: Re: libibverbs and max inline data In-Reply-To: References: <20050522094037.GC9249@mellanox.co.il> <52y8a75a9t.fsf@topspin.com> <20050523154858.GK9732@mellanox.co.il> Message-ID: <20050523162559.GC18721@mellanox.co.il> Quoting r. James Lentini : > Subject: Re: Re: libibverbs and max inline data > > > > On Mon, 23 May 2005, Michael S. Tsirkin wrote: > > >Quoting r. James Lentini : > >>Subject: Re: Re: libibverbs and max inline data > >> > >> > >>On Sun, 22 May 2005, Roland Dreier wrote: > >> > >>> Michael> Hi! I tried to figure out how much inline data a qp can > >>> Michael> support, to know what max_inline_data value to put in the > >>> Michael> qp. However, there doesnt seem to exist a way to do > >>> Michael> that, currently. > >>> > >>>I think you're right. > >>> > >>> Michael> Maybe the driver should be changed to trim > >>> Michael> max_inline_data to the maximum possible value? > >>> > >>>Yes, that would probably make sense. In a sense that's what the > >>>kernel driver already does -- it always clips the max_inline_data > >>>request from the consumer down to 0. > >> > >>ib[v]_create_qp should return an error when an unsupported inline data > >>size is requested. > > > >Why is it important? The user could always check the max_inline_data > >returned and close the qp if thats too small, but I dont see > >many users doing this - inline data is mainly a tuning opportunity. > > Once there is a way to programmatically determine the max_inline_data > value, your proposal would be feasible. There is a way to do it - the actual values are returned in the attr that you pass to create_qp. > Even if it were possible, I > still believe that the driver should validate this value. > > Why have the user specify the ib[v]_qp_cap values if they are going to > be ignored? max_inline_data is special in that its a tuning opportunity. So its mostly harmless for the driver to ignore. > As a rule, the driver should return an error when the user requests a > capability/resource that the hardware cannot support. > > kDAPL was bitten by this last week. The call to ib_create_qp succeed > but the value kDAPL was placing in the ib_qp_init_attr's > max_inline_data field was too large. It was just a bug in the mthca kernel driver. You shouldnt work around such bugs in an application. > As a result, a subsequent qp call failed. > If ib_create_qp had returned an error to begin with, it would > have been easier to determine the problem. It seems wrong for the application to hard-code the max_inline_data value that suits a particular hardware. Better for the app to tell the driver what it needs and the driver will give it what hardware can support. > >>>Independent of that we could add a field to the device properties > >>>struct (and implement ibv_query_device() for userspace). ibv_query_device would be useful for other things as well (e.g. getting the max MTU). -- MST - Michael S. Tsirkin From mshefty at ichips.intel.com Mon May 23 09:27:47 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Mon, 23 May 2005 09:27:47 -0700 Subject: [openib-general] Re: performance counters in /sys In-Reply-To: <4291097F.8070404@hp.com> References: <35EA21F54A45CB47B879F21A91F4862F581222@taurus.voltaire.com> <1116539301.4479.255.camel@hal.voltaire.com> <428DC946.5080207@hp.com> <20050520161146.GB14411@esmail.cup.hp.com> <20050520164843.GC14411@esmail.cup.hp.com> <20050522053351.GB8522@mellanox.co.il> <4291097F.8070404@hp.com> Message-ID: <42920483.3040900@ichips.intel.com> Mark Seger wrote: > I guess the thing that has me mystified about all this is I can > certainly appreciate the potential 'goodness' of having 1 var/file for > user oriented access but perhaps one of the better examples of why this > is just a bad idea for programmatic access is the individual process > stats. Is the implication of this that some day those too would be > moved to /sys as one stat per file?!? Can you imaging trying to run top > or ps if that were to happen? Are there any performance counters that aren't available through the PMA MADs? If not, is there any reason why the PMA interface shouldn't be used for programmatic access? - Sean From halr at voltaire.com Mon May 23 09:30:11 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 23 May 2005 12:30:11 -0400 Subject: [openib-general] [PATCH] userspace/management changes to support send side RMPP Message-ID: <1116865811.4481.5750.camel@hal.voltaire.com> userspace/management changes to support send side RMPP (needs change to linux-kernel/infiniband/core/user_mad.c) ABI_VERSION is now 3 RMPP is enabled in build SA GetTable is now supported properly (within current RMPP limitations) Signed-off-by: Hal Rosenstock Modified: gen2/trunk/src/userspace/management/diags/ibping/src/ibping.c =================================================================== --- gen2/trunk/src/userspace/management/diags/ibping/src/ibping.c 2005-05-23 16:12:49 UTC (rev 2455) +++ gen2/trunk/src/userspace/management/diags/ibping/src/ibping.c 2005-05-23 16:33:46 UTC (rev 2456) @@ -295,7 +295,7 @@ madrpc_init(ca, ca_port, mgmt_classes, 3); if (server) { - if (mad_register_server(ping_class, 0, oui) < 0) + if (mad_register_server(ping_class, 0, 0, oui) < 0) PANIC("can't serve class %d", ping_class); get_host_and_domain(host_and_domain, sizeof host_and_domain); @@ -305,7 +305,7 @@ exit(0); } - if (mad_register_client(ping_class) < 0) + if (mad_register_client(ping_class, 0) < 0) PANIC("can't register to ping class %d", ping_class); if (ib_resolve_portid_str(&portid, argv[0], dest_type, sm_id) < 0) Modified: gen2/trunk/src/userspace/management/diags/ibsysstat/src/ibsysstat.c =================================================================== --- gen2/trunk/src/userspace/management/diags/ibsysstat/src/ibsysstat.c 2005-05-23 16:12:49 UTC (rev 2455) +++ gen2/trunk/src/userspace/management/diags/ibsysstat/src/ibsysstat.c 2005-05-23 16:33:46 UTC (rev 2456) @@ -269,7 +269,7 @@ main(int argc, char **argv) { int mgmt_classes[3] = {IB_SMI_CLASS, IB_SMI_DIRECT_CLASS, IB_SA_CLASS}; - int ping_class = IB_VENDOR_OPENIB_SYSSTAT_CLASS; + int sysstat_class = IB_VENDOR_OPENIB_SYSSTAT_CLASS; ib_portid_t portid = {0}; ib_portid_t *sm_id = 0, sm_portid = {0}; int timeout = 0, udebug = 0, server = 0; @@ -336,8 +336,8 @@ madrpc_init(ca, ca_port, mgmt_classes, 3); if (server) { - if (mad_register_server(ping_class, 0, oui) < 0) - PANIC("can't serve class %d", ping_class); + if (mad_register_server(sysstat_class, 0, 0, oui) < 0) + PANIC("can't serve class %d", sysstat_class); host_ncpu = build_cpuinfo(); @@ -346,8 +346,8 @@ exit(0); } - if (mad_register_client(ping_class) < 0) - PANIC("can't register to ping class %d", ping_class); + if (mad_register_client(sysstat_class, 0) < 0) + PANIC("can't register to ping class %d", sysstat_class); if (ib_resolve_portid_str(&portid, argv[0], dest_type, sm_id) < 0) PANIC("can't resolve destination port %s", argv[0]); Modified: gen2/trunk/src/userspace/management/diags/smpdump/src/smpdump.c =================================================================== --- gen2/trunk/src/userspace/management/diags/smpdump/src/smpdump.c 2005-05-23 16:12:49 UTC (rev 2455) +++ gen2/trunk/src/userspace/management/diags/smpdump/src/smpdump.c 2005-05-23 16:33:46 UTC (rev 2456) @@ -282,7 +282,7 @@ if ((portid = umad_open_port(dev_name, dev_port)) < 0) PANIC("can't open UMAD port (%s:%d)", dev_name, dev_port); - if ((mad_agent = umad_register(portid, mgmt_class, 1, 0)) < 0) + if ((mad_agent = umad_register(portid, mgmt_class, 1, 0, 0)) < 0) PANIC("Couldn't register agent for SMPs"); if (!(umad = umad_alloc(1))) @@ -298,7 +298,7 @@ if (debug > 1) xdump(stderr, "before send:\n", smp, 256); - if (umad_send(portid, mad_agent, umad, timeout_ms) < 0) + if (umad_send(portid, mad_agent, umad, IB_MAD_SIZE, timeout_ms) < 0) PANIC("send failed"); if (umad_recv(portid, umad, -1) != mad_agent) Modified: gen2/trunk/src/userspace/management/libibmad/include/infiniband/mad.h =================================================================== --- gen2/trunk/src/userspace/management/libibmad/include/infiniband/mad.h 2005-05-23 16:12:49 UTC (rev 2455) +++ gen2/trunk/src/userspace/management/libibmad/include/infiniband/mad.h 2005-05-23 16:33:46 UTC (rev 2456) @@ -630,9 +630,9 @@ int mad_build_pkt(void *umad, ib_rpc_t *rpc, ib_portid_t *dport, ib_rmpp_hdr_t *rmpp, void *data); /* register.c */ -int mad_register_client(int mgmt); -int mad_register_server(int mgmt, uint32_t method_mask[4], - uint32_t class_oui); +int mad_register_client(int mgmt, uint8_t rmpp_version); +int mad_register_server(int mgmt, uint8_t rmpp_version, + uint32_t method_mask[4], uint32_t class_oui); int mad_class_agent(int mgmt); int mad_agent_class(int agent); Modified: gen2/trunk/src/userspace/management/libibmad/src/mad.c =================================================================== --- gen2/trunk/src/userspace/management/libibmad/src/mad.c 2005-05-23 16:12:49 UTC (rev 2455) +++ gen2/trunk/src/userspace/management/libibmad/src/mad.c 2005-05-23 16:33:46 UTC (rev 2456) @@ -147,7 +147,7 @@ mad_set_field(buf, 0, IB_DRSMP_DRDLID_F, drpath->drdlid ? drpath->drdlid : 0xffff); mad_set_field(buf, 0, IB_DRSMP_DRSLID_F, drpath->drslid ? drpath->drslid : 0xffff); - /* bytes 128 - 256 - by default should be zero due memset*/ + /* bytes 128 - 256 - by default should be zero due to memset*/ if (is_resp) mad_set_array(buf, 0, IB_DRSMP_RPATH_F, drpath->p); else Modified: gen2/trunk/src/userspace/management/libibmad/src/register.c =================================================================== --- gen2/trunk/src/userspace/management/libibmad/src/register.c 2005-05-23 16:12:49 UTC (rev 2455) +++ gen2/trunk/src/userspace/management/libibmad/src/register.c 2005-05-23 16:33:46 UTC (rev 2456) @@ -116,7 +116,7 @@ } int -mad_register_client(int mgmt) +mad_register_client(int mgmt, uint8_t rmpp_version) { int vers, agent; @@ -124,7 +124,7 @@ DEBUG("Unknown class %d mgmt_class", mgmt); return -1; } - if ((agent = umad_register(madrpc_portid(), mgmt, vers, 0)) < 0) { + if ((agent = umad_register(madrpc_portid(), mgmt, vers, rmpp_version, 0)) < 0) { DEBUG("Can't register agent for class %d", mgmt); return -1; } @@ -141,9 +141,11 @@ } int -mad_register_server(int mgmt, uint32_t method_mask[4], uint32_t class_oui) +mad_register_server(int mgmt, uint8_t rmpp_version, + uint32_t method_mask[4], uint32_t class_oui) { - uint32_t class_method_mask[4] = {0xffffffff, 0xffffffff, 0xffffffff, 0xffffffff}; + uint32_t class_method_mask[4] = {0xffffffff, 0xffffffff, + 0xffffffff, 0xffffffff}; uint8_t oui[3]; int agent, vers, mad_portid; @@ -165,11 +167,11 @@ oui[0] = (class_oui >> 16) & 0xff; oui[1] = (class_oui >> 8) & 0xff; oui[2] = class_oui & 0xff; - if ((agent = umad_register_oui(mad_portid, mgmt, oui, class_method_mask)) < 0) { + if ((agent = umad_register_oui(mad_portid, mgmt, rmpp_version, oui, class_method_mask)) < 0) { DEBUG("Can't register agent for class %d", mgmt); return -1; } - } else if ((agent = umad_register(mad_portid, mgmt, vers, class_method_mask)) < 0) { + } else if ((agent = umad_register(mad_portid, mgmt, vers, rmpp_version, class_method_mask)) < 0) { DEBUG("Can't register agent for class %d", mgmt); return -1; } Modified: gen2/trunk/src/userspace/management/libibmad/src/rpc.c =================================================================== --- gen2/trunk/src/userspace/management/libibmad/src/rpc.c 2005-05-23 16:12:49 UTC (rev 2455) +++ gen2/trunk/src/userspace/management/libibmad/src/rpc.c 2005-05-23 16:33:46 UTC (rev 2456) @@ -112,8 +112,8 @@ timeout = def_madrpc_timeout; if (ibdebug > 1) { - WARN(">>> sending: len %d pktsz %d", len, umad_size()); - xdump(stderr, "send buf\n", umad, umad_size()); + WARN(">>> sending: len %d pktsz %d", len, umad_size() + IB_MAD_SIZE); + xdump(stderr, "send buf\n", umad, umad_size() + IB_MAD_SIZE); } if (save_mad) { @@ -126,7 +126,7 @@ if (retries) ERRS("retry %d (timeout %d ms)", retries+1, timeout); - if (umad_send(mad_portid, agentid, umad, timeout) < 0) { + if (umad_send(mad_portid, agentid, umad, IB_MAD_SIZE, timeout) < 0) { WARN("send failed; %m"); return -1; } @@ -157,7 +157,7 @@ uint8_t pktbuf[1024], *mad; void *umad = pktbuf; - memset(pktbuf, 0, umad_size()); + memset(pktbuf, 0, umad_size() + IB_MAD_SIZE); if ((len = mad_build_pkt(umad, rpc, dport, 0, payload)) < 0) return 0; @@ -192,7 +192,7 @@ uint8_t pktbuf[1024], *mad; void *umad = pktbuf; - memset(pktbuf, 0, umad_size()); + memset(pktbuf, 0, umad_size() + IB_MAD_SIZE); DEBUG("rmpp %p data %p", rmpp, data); @@ -263,7 +263,7 @@ while (num_classes--) { int mgmt = *mgmt_classes++; - if (mad_register_client(mgmt) < 0) + if (mad_register_client(mgmt, 0) < 0) PANIC("client_register for mgmt %d failed", mgmt); } } Modified: gen2/trunk/src/userspace/management/libibmad/src/serv.c =================================================================== --- gen2/trunk/src/userspace/management/libibmad/src/serv.c 2005-05-23 16:12:49 UTC (rev 2455) +++ gen2/trunk/src/userspace/management/libibmad/src/serv.c 2005-05-23 16:33:46 UTC (rev 2456) @@ -71,7 +71,7 @@ } if (umad_send(madrpc_portid(), mad_class_agent(rpc->mgtclass), - umad, rpc->timeout) < 0) { + umad, IB_MAD_SIZE, rpc->timeout) < 0) { WARN("send failed; %m"); return -1; } @@ -145,7 +145,7 @@ xdump(stderr, "mad respond pkt\n", mad, IB_MAD_SIZE); if (umad_send(madrpc_portid(), mad_class_agent(rpc.mgtclass), umad, - rpc.timeout) < 0) { + IB_MAD_SIZE, rpc.timeout) < 0) { DEBUG("send failed; %m"); return -1; } @@ -156,7 +156,7 @@ void * mad_receive(void *umad, int timeout) { - void *mad = umad ? umad : umad_alloc(1); + void *mad = umad ? umad : umad_alloc(1, umad_size() + IB_MAD_SIZE); int agent; if ((agent = umad_recv(madrpc_portid(), mad, timeout)) < 0) { @@ -172,7 +172,7 @@ void * mad_alloc(void) { - return umad_alloc(1); + return umad_alloc(1, umad_size() + IB_MAD_SIZE); } void Modified: gen2/trunk/src/userspace/management/libibumad/include/infiniband/umad.h =================================================================== --- gen2/trunk/src/userspace/management/libibumad/include/infiniband/umad.h 2005-05-23 16:12:49 UTC (rev 2455) +++ gen2/trunk/src/userspace/management/libibumad/include/infiniband/umad.h 2005-05-23 16:33:46 UTC (rev 2456) @@ -63,7 +63,7 @@ uint32_t flow_label; } ib_mad_addr_t; -#define IB_UMAD_ABI_VERSION 2 +#define IB_UMAD_ABI_VERSION 3 #define IB_UMAD_ABI_DIR "/sys/class/infiniband_mad" #define IB_UMAD_ABI_FILE "abi_version" @@ -160,14 +160,15 @@ int umad_set_addr(void *umad, int dlid, int dqp, int sl, int qkey); int umad_set_pkey(void *umad, int pkey); -int umad_send(int portid, int agentid, void *umad, int timeout_ms); +int umad_send(int portid, int agentid, void *umad, int length, + int timeout_ms); int umad_recv(int portid, void *umad, int timeout_ms); int umad_poll(int portid, int timeout_ms); int umad_register(int portid, int mgmt_class, int mgmt_version, - uint32_t method_mask[4]); -int umad_register_oui(int portid, int mgmt_class, uint8_t oui[3], - uint32_t method_mask[4]); + uint8_t rmpp_version, uint32_t method_mask[4]); +int umad_register_oui(int portid, int mgmt_class, uint8_t rmpp_version, + uint8_t oui[3], uint32_t method_mask[4]); int umad_unregister(int portid, int agentid); int umad_debug(int level); @@ -177,9 +178,9 @@ #include static inline void * -umad_alloc(int num) /* alloc array of umad buffers */ +umad_alloc(int num, int size) /* alloc array of umad buffers */ { - return calloc(num, umad_size()); + return calloc(num, size); } static inline void Modified: gen2/trunk/src/userspace/management/libibumad/src/umad.c =================================================================== --- gen2/trunk/src/userspace/management/libibumad/src/umad.c 2005-05-23 16:12:49 UTC (rev 2455) +++ gen2/trunk/src/userspace/management/libibumad/src/umad.c 2005-05-23 16:33:46 UTC (rev 2456) @@ -50,12 +50,14 @@ #include "umad.h" +#define IB_OPENIB_OUI (0x001405) + typedef struct ib_user_mad { - uint8_t data[256]; uint32_t agent_id; uint32_t status; uint32_t timeout_ms; ib_mad_addr_t addr; + uint8_t data[0]; } ib_user_mad_t; typedef struct ib_user_mad_reg_req { @@ -65,6 +67,7 @@ uint8_t mgmt_class; uint8_t mgmt_class_version; uint8_t oui[3]; + uint8_t rmpp_version; } ib_user_mad_reg_req_t; #define TRACE if (umaddebug) WARN @@ -233,7 +236,7 @@ return 1; } - if (*port > 0) { /* user wants user gets */ + if (*port > 0) { /* user wants user gets */ if (*port > ca.numports) return -1; if (!ca.ports[*port]) @@ -300,7 +303,8 @@ if (port_type > 0) { if (best_port) *best_port = port; - DEBUG("found ca %s with active port %d", names[caidx], port); + DEBUG("found ca %s with active port %d", + names[caidx], port); return (char *)(names + caidx); } @@ -332,18 +336,22 @@ int r, i, ret; int portnum; - strncpy(ca->ca_name, ca_name, sizeof(ca->ca_name)); + strncpy(ca->ca_name, ca_name, sizeof ca->ca_name); - snprintf(dir_name, sizeof dir_name - 1, "%s/%s", SYS_INFINIBAND, ca->ca_name); + snprintf(dir_name, sizeof dir_name - 1, "%s/%s", SYS_INFINIBAND, + ca->ca_name); dir_name[sizeof dir_name - 1] = 0; if ((r = sys_read_uint(dir_name, SYS_NODE_TYPE, &ca->node_type)) < 0) return r; - if ((r = sys_read_string(dir_name, SYS_CA_FW_VERS, ca->fw_ver, sizeof ca->fw_ver)) < 0) + if ((r = sys_read_string(dir_name, SYS_CA_FW_VERS, ca->fw_ver, + sizeof ca->fw_ver)) < 0) return r; - if ((r = sys_read_string(dir_name, SYS_CA_HW_VERS, ca->hw_ver, sizeof ca->hw_ver)) < 0) + if ((r = sys_read_string(dir_name, SYS_CA_HW_VERS, ca->hw_ver, + sizeof ca->hw_ver)) < 0) return r; - if ((r = sys_read_string(dir_name, SYS_CA_TYPE, ca->ca_type, sizeof ca->ca_type)) < 0) + if ((r = sys_read_string(dir_name, SYS_CA_TYPE, ca->ca_type, + sizeof ca->ca_type)) < 0) return r; if ((r = sys_read_guid(dir_name, SYS_CA_NODE_GUID, &ca->node_guid)) < 0) return r; @@ -367,14 +375,16 @@ memset(ca->ports, 0, sizeof ca->ports); for (i = 0; i < r; i++) { portnum = 0; - if (!strcmp(".", namelist[i]->d_name) || !strcmp("..", namelist[i]->d_name)) + if (!strcmp(".", namelist[i]->d_name) || + !strcmp("..", namelist[i]->d_name)) continue; if (strcmp("0", namelist[i]->d_name) && - ((portnum = atoi(namelist[i]->d_name)) <= 0 || portnum >= UMAD_CA_MAX_PORTS)) { + ((portnum = atoi(namelist[i]->d_name)) <= 0 || + portnum >= UMAD_CA_MAX_PORTS)) { ret = -EIO; goto clean; } - if (!(ca->ports[portnum] = calloc(1, sizeof (*ca->ports[portnum])))) { + if (!(ca->ports[portnum] = calloc(1, sizeof(*ca->ports[portnum])))) { ret = -ENOMEM; goto clean; } @@ -489,7 +499,8 @@ if (!strcmp(namelist[i]->d_name, ".") || !strcmp(namelist[i]->d_name, "..")) { } else - strncpy(cas[j++], namelist[i]->d_name, UMAD_CA_NAME_LEN); + strncpy(cas[j++], namelist[i]->d_name, + UMAD_CA_NAME_LEN); free(namelist[i]); } DEBUG("return %d cas", j); @@ -679,7 +690,8 @@ { struct ib_user_mad *mad = umad; - TRACE("umad %p dlid %d dqp %d sl %d, qkey %x", umad, dlid, dqp, sl, qkey); + TRACE("umad %p dlid %d dqp %d sl %d, qkey %x", + umad, dlid, dqp, sl, qkey); mad->addr.qpn = htonl(dqp); mad->addr.lid = htons(dlid); mad->addr.qkey = htonl(qkey); @@ -704,7 +716,7 @@ } int -umad_send(int portid, int agentid, void *umad, int timeout_ms) +umad_send(int portid, int agentid, void *umad, int length, int timeout_ms) { struct ib_user_mad *mad = umad; Port *port; @@ -720,7 +732,7 @@ if (umaddebug > 1) umad_dump(mad); - if (write(port->dev_fd, mad, sizeof (*mad)) == sizeof (*mad)) + if (write(port->dev_fd, mad, length + sizeof *mad) == length + sizeof *mad) return 0; DEBUG("send error: %m"); @@ -758,7 +770,7 @@ if (timeout_ms && (n = dev_poll(port->dev_fd, timeout_ms)) < 0) return n; - if ((n = read(port->dev_fd, umad, sizeof (*mad))) == sizeof (*mad)) { + if ((n = read(port->dev_fd, umad, sizeof *mad + 256)) == sizeof *mad + 256) { DEBUG("mad received by agent %d", mad->agent_id); return mad->agent_id; } @@ -766,7 +778,7 @@ if (n == -EWOULDBLOCK) return n; - DEBUG("read returned %d != sizeof umad %d (%m)", n, sizeof (*mad)); + DEBUG("read returned %d != sizeof umad %d (%m)", n, sizeof *mad); return -EIO; } @@ -783,14 +795,14 @@ } int -umad_register_oui(int portid, int mgmt_class, uint8_t oui[3], - uint32_t method_mask[4]) +umad_register_oui(int portid, int mgmt_class, uint8_t rmpp_version, + uint8_t oui[3], uint32_t method_mask[4]) { struct ib_user_mad_reg_req req; Port *port; - TRACE("portid %d mgmt_class %u oui 0x%x%x%x method_mask %p", - portid, mgmt_class, (int)oui[0], (int)oui[1], + TRACE("portid %d mgmt_class %u rmpp_version %d oui 0x%x%x%x method_mask %p", + portid, mgmt_class, (int)rmpp_version, (int)oui[0], (int)oui[1], (int)oui[2], method_mask); if (!(port = port_get(portid))) @@ -805,6 +817,7 @@ req.mgmt_class = mgmt_class; req.mgmt_class_version = 1; memcpy(req.oui, oui, sizeof req.oui); + req.rmpp_version = rmpp_version; if ((void *)method_mask != 0) memcpy(req.method_mask, method_mask, sizeof req.method_mask); @@ -817,21 +830,22 @@ return req.id; /* return agentid */ } - DEBUG("portid %d registering qp %d class %s version %d out 0x%x failed: %m", + DEBUG("portid %d registering qp %d class %s version %d oui 0x%x failed: %m", portid, req.qpn, req.mgmt_class, req.mgmt_class_version, oui); return -EPERM; } int umad_register(int portid, int mgmt_class, int mgmt_version, - uint32_t method_mask[4]) + uint8_t rmpp_version, uint32_t method_mask[4]) { struct ib_user_mad_reg_req req; Port *port; + uint32_t oui = htonl(IB_OPENIB_OUI); int qp; - TRACE("portid %d mgmt_class %u mgmt_version %u method_mask %p", - portid, mgmt_class, mgmt_version, method_mask); + TRACE("portid %d mgmt_class %u mgmt_version %u rmpp_version %d method_mask %p", + portid, mgmt_class, mgmt_version, rmpp_version, method_mask); if (!(port = port_get(portid))) return -EINVAL; @@ -839,12 +853,15 @@ req.qpn = qp = (mgmt_class == 0x1 || mgmt_class == 0x81) ? 0 : 1; req.mgmt_class = mgmt_class; req.mgmt_class_version = mgmt_version; + req.rmpp_version = rmpp_version; if ((void *)method_mask != 0) memcpy(req.method_mask, method_mask, sizeof req.method_mask); else memset(req.method_mask, 0, sizeof req.method_mask); + memcpy(&req.oui, (char *)&oui + 1, sizeof req.oui); + if (!ioctl(port->dev_fd, IB_USER_MAD_REGISTER_AGENT, (void *)&req)) { DEBUG("portid %d registered to use agent %d qp %d", portid, req.id, qp); Modified: gen2/trunk/src/userspace/management/osm/libvendor/Makefile.am =================================================================== --- gen2/trunk/src/userspace/management/osm/libvendor/Makefile.am 2005-05-23 16:12:49 UTC (rev 2455) +++ gen2/trunk/src/userspace/management/osm/libvendor/Makefile.am 2005-05-23 16:33:46 UTC (rev 2456) @@ -9,7 +9,7 @@ lib_LTLIBRARIES = libosmvendor.la -libosmvendor_la_CFLAGS = -Wall -DOSM_VENDOR_INTF_OPENIB +libosmvendor_la_CFLAGS = -Wall -DOSM_VENDOR_INTF_OPENIB -DVENDOR_RMPP_SUPPORT if HAVE_LD_VERSION_SCRIPT libosmvendor_version_script = -Wl,--version-script=$(srcdir)/libosmvendor.map Modified: gen2/trunk/src/userspace/management/osm/libvendor/osm_vendor_ibumad.c =================================================================== --- gen2/trunk/src/userspace/management/osm/libvendor/osm_vendor_ibumad.c 2005-05-23 16:12:49 UTC (rev 2455) +++ gen2/trunk/src/userspace/management/osm/libvendor/osm_vendor_ibumad.c 2005-05-23 16:33:46 UTC (rev 2456) @@ -224,7 +224,8 @@ pthread_sigmask(SIG_BLOCK, &sigs, NULL); for (;;) { - if (!umad && !(umad = umad_alloc(1))) { + if (!umad && + !(umad = umad_alloc(1, umad_size() + MAD_BLOCK_SIZE))) { osm_log(p_ur->p_log, OSM_LOG_ERROR, "umad_receiver: can't alloc umad\n"); break; @@ -236,7 +237,8 @@ break; } - if (mad_agent >= UMAD_CA_MAX_AGENTS || !(p_bind = p_vend->agents[mad_agent])) { + if (mad_agent >= UMAD_CA_MAX_AGENTS || + !(p_bind = p_vend->agents[mad_agent])) { osm_log(p_ur->p_log, OSM_LOG_ERROR, "umad_receiver: bad mad agent %d - dropping\n", mad_agent); continue; @@ -249,7 +251,8 @@ mad->mgmt_class == IB_MCLASS_SUBN_LID || mad->mgmt_class == IB_MCLASS_SUBN_DIR); - if (!(madw_p = osm_mad_pool_get(p_bind->p_mad_pool, (osm_bind_handle_t)p_bind, + if (!(madw_p = osm_mad_pool_get(p_bind->p_mad_pool, + (osm_bind_handle_t)p_bind, MAD_BLOCK_SIZE, &osm_addr))) { osm_log( p_vend->p_log, OSM_LOG_ERROR, "umad_receiver: " "request for a new madw failed -- dropping packet\n" ); @@ -271,7 +274,8 @@ if (mad->mgmt_class != IB_MCLASS_SUBN_DIR) { /* LID routed */ osm_log( p_vend->p_log, OSM_LOG_ERROR, - "umad_receiver: class 0x%x LID 0x%x\n", mad->mgmt_class, ib_addr->lid); + "umad_receiver: class 0x%x LID 0x%x\n", + mad->mgmt_class, ib_addr->lid); } else { /* Direct routed SMP */ ib_smp_t *smp = (ib_smp_t *)mad; @@ -280,8 +284,10 @@ char line[BUF_SIZE]; osm_log( p_vend->p_log, OSM_LOG_ERROR, - "umad_receiver: DR SMP hop ptr %d hop count %d DR SLID 0x%x DR DLID 0x%x\n", smp->hop_ptr, smp->hop_count, smp->dr_slid, smp->dr_dlid); - if( osm_log_is_active( p_vend->p_log, OSM_LOG_ERROR) ) + "umad_receiver: DR SMP hop ptr %d hop count %d DR SLID 0x%x DR DLID 0x%x\n", + smp->hop_ptr, smp->hop_count, smp->dr_slid, smp->dr_dlid); + if( osm_log_is_active( p_vend->p_log, + OSM_LOG_ERROR) ) { sprintf( buf, "Initial path = "); for ( i = 0; i < smp->hop_count; i++ ) { @@ -622,6 +628,7 @@ osm_umad_bind_info_t *p_bind = 0; uint32_t method_mask[4]; int umad_port_id; + uint8_t rmpp_version; OSM_LOG_ENTER( p_vend->p_log, osm_vendor_bind ); @@ -657,10 +664,20 @@ memset(method_mask, 0xff, sizeof method_mask); /* accept all methods */ +#ifndef VENDOR_RMPP_SUPPORT + rmpp_version = 0; +#else + /* If SA class, set rmpp_version */ + if (p_user_bind->mad_class == IB_MCLASS_SUBN_ADM) + rmpp_version = 1; + else + rmpp_version = 0; +#endif + if ((p_bind->agent_id = umad_register(p_vend->umad_port_id, p_user_bind->mad_class, p_user_bind->class_version, - method_mask)) < 0) { + rmpp_version, method_mask)) < 0) { osm_log( p_vend->p_log, OSM_LOG_ERROR, "osm_vendor_bind: " "Unable to register class %u version %u.\n", p_user_bind->mad_class, p_user_bind->class_version); @@ -681,10 +698,12 @@ p_vend->agents[p_bind->agent_id] = p_bind; - if (p_user_bind->mad_class == 0x81) { - if ((p_bind->agent_id1 = umad_register(p_vend->umad_port_id, 1, + /* If Subn Directed Route class, register Subn LID routed class */ + if (p_user_bind->mad_class == IB_MCLASS_SUBN_DIR) { + if ((p_bind->agent_id1 = umad_register(p_vend->umad_port_id, + IB_MCLASS_SUBN_LID, p_user_bind->class_version, - method_mask)) < 0) { + 0, method_mask)) < 0) { osm_log( p_vend->p_log, OSM_LOG_ERROR, "osm_vendor_bind: " "Unable to register class 1 version %u.\n", p_user_bind->class_version); @@ -724,13 +743,12 @@ OSM_LOG_ENTER( p_vend->p_log, osm_vendor_get ); +osm_log( p_vend->p_log, OSM_LOG_DEBUG, "osm_vendor_get: " + "Acquiring UMAD for p_madw = %p, size = %u.\n", p_vw, mad_size ); CL_ASSERT( p_vw ); - CL_ASSERT( mad_size <= MAD_BLOCK_SIZE ); + p_vw->size = mad_size; + p_vw->umad = umad_alloc(1, mad_size + umad_size()); - /* IB_UMAD assumes it is 256 - we must follow */ - p_vw->size = MAD_BLOCK_SIZE; - p_vw->umad = umad_alloc(1); - /* track locally */ p_vw->h_bind = h_bind; @@ -786,7 +804,7 @@ osm_vend_wrap_t* const p_vw = osm_madw_get_vend_ptr( p_madw ); osm_mad_addr_t* const p_mad_addr = osm_madw_get_mad_addr_ptr( p_madw ); ib_mad_t* const p_mad = osm_madw_get_mad_ptr( p_madw ); - ib_sa_mad_t* const p_sa = (ib_sa_mad_t *)p_vw->umad; + ib_sa_mad_t* const p_sa = (ib_sa_mad_t *)p_mad; int ret = -1; #ifndef VENDOR_RMPP_SUPPORT uint32_t paylen = 0; @@ -821,6 +839,7 @@ p_sa->rmpp_status = 0; #ifdef VENDOR_RMPP_SUPPORT } + osm_log( p_vend->p_log, OSM_LOG_ERROR, "osm_vendor_send: RMPP %d length %d\n", ib_rmpp_is_flag_set((ib_rmpp_mad_t *)p_sa, IB_RMPP_FLAG_ACTIVE), p_madw->mad_size); #else } else { p_sa->rmpp_version = 1; @@ -838,7 +857,7 @@ if (resp_expected) put_madw(p_vend, p_madw, &p_mad->trans_id); - if ((ret = umad_send(p_bind->port_id, p_bind->agent_id, p_vw->umad, resp_expected ? p_vend->timeout : 0)) < 0) { + if ((ret = umad_send(p_bind->port_id, p_bind->agent_id, p_vw->umad, p_madw->mad_size, resp_expected ? p_vend->timeout : 0)) < 0) { osm_log( p_vend->p_log, OSM_LOG_ERROR, "osm_vendor_send: Send failed %d (%m).\n", ret); (*p_bind->send_err_callback)(p_bind->client_context, p_madw); /* cb frees madw */ if (resp_expected) @@ -850,7 +869,7 @@ osm_mad_pool_put(p_bind->p_mad_pool, p_madw); osm_log(p_vend->p_log, OSM_LOG_DEBUG, "__osm_mtl_send_callback: " - "Completed Sending Request MADW:%p.\n", p_madw); + "Completed Sending Request MADW: %p.\n", p_madw); Exit: OSM_LOG_EXIT( p_vend->p_log ); return( ret ); Modified: gen2/trunk/src/userspace/management/osm/opensm/Makefile.am =================================================================== --- gen2/trunk/src/userspace/management/osm/opensm/Makefile.am 2005-05-23 16:12:49 UTC (rev 2455) +++ gen2/trunk/src/userspace/management/osm/opensm/Makefile.am 2005-05-23 16:33:46 UTC (rev 2456) @@ -47,7 +47,7 @@ osm_ucast_mgr.c osm_ucast_updn.c \ osm_vl15intf.c osm_vl_arb_rcv.c\ osm_vl_arb_rcv_ctrl.c -opensm_CFLAGS = -Wall -DOSM_VENDOR_INTF_OPENIB +opensm_CFLAGS = -Wall -DOSM_VENDOR_INTF_OPENIB -DVENDOR_RMPP_SUPPORT opensm_LDADD = $(libdir)/libibumad.la \ $(libdir)/libibcommon.la \ $(libdir)/libosmcomp.la \ From halr at voltaire.com Mon May 23 09:42:04 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 23 May 2005 12:42:04 -0400 Subject: [openib-general] Re: performance counters in /sys In-Reply-To: <42920483.3040900@ichips.intel.com> References: <35EA21F54A45CB47B879F21A91F4862F581222@taurus.voltaire.com> <1116539301.4479.255.camel@hal.voltaire.com> <428DC946.5080207@hp.com> <20050520161146.GB14411@esmail.cup.hp.com> <20050520164843.GC14411@esmail.cup.hp.com> <20050522053351.GB8522@mellanox.co.il> <4291097F.8070404@hp.com> <42920483.3040900@ichips.intel.com> Message-ID: <1116866524.4481.5774.camel@hal.voltaire.com> On Mon, 2005-05-23 at 12:27, Sean Hefty wrote: > Are there any performance counters that aren't available through the PMA > MADs? If not, is there any reason why the PMA interface shouldn't be used > for programmatic access? All the counters found in: /sys/class/infiniband/mthca0/ports/1/counters excessive_buffer_overrun_errors port_rcv_remote_physical_errors link_downed port_rcv_switch_relay_errors link_error_recovery port_xmit_constraint_errors local_link_integrity_errors port_xmit_data port_rcv_constraint_errors port_xmit_discards port_rcv_data port_xmit_packets port_rcv_errors symbol_error port_rcv_packets VL15_dropped are available via the PMA (and via the perfquery tool): /usr/local/ib/bin/perfquery 1 1 # Port counters: Lid 0x1 port 1 PortSelect:......................1 CounterSelect:...................0x0000 SymbolErrors:....................10344 LinkRecovers:....................255 LinkDowned:......................4 RcvErrors:.......................0 RcvRemotePhysErrors:.............0 RcvSwRelayErrors:................0 XmtDiscards:.....................19 XmtConstraintErrors:.............0 RcvConstraintErrors:.............0 LinkIntegrityErrors:.............0 ExcBufOverrunErrors:.............0 VL15Dropped:.....................0 XmtBytes:........................126990 RcvBytes:........................126952 XmtPkts:.........................1791 RcvBytes:........................1790 One advantage is that all counters are retrieved with one MAD. -- Hal From jlentini at netapp.com Mon May 23 10:41:30 2005 From: jlentini at netapp.com (James Lentini) Date: Mon, 23 May 2005 13:41:30 -0400 (EDT) Subject: [openib-general] Re: libibverbs and max inline data In-Reply-To: <52oeb1zzjv.fsf@topspin.com> References: <20050522094037.GC9249@mellanox.co.il> <52y8a75a9t.fsf@topspin.com> <20050523154858.GK9732@mellanox.co.il> <52oeb1zzjv.fsf@topspin.com> Message-ID: On Mon, 23 May 2005, Roland Dreier wrote: > James> Once there is a way to programmatically determine the > James> max_inline_data value, your proposal would be > James> feasible. Even if it were possible, I still believe that > James> the driver should validate this value. > > I think you may be misunderstanding the suggestion. The idea is that > the driver takes the input value of max_inline_data as a hint, and > tries to allocate a QP that supports that value. However, if the > value is too large, the driver reduces the value down to the largest > value that it can support. In any case, the true amount of inline > data supported is returned to the consumer in the qp_cap structure. Is ib_create_qp supposed to fail if an ib_qp_cap value is invalid? From jlentini at netapp.com Mon May 23 10:48:19 2005 From: jlentini at netapp.com (James Lentini) Date: Mon, 23 May 2005 13:48:19 -0400 (EDT) Subject: [openib-general] Re: libibverbs and max inline data In-Reply-To: <52oeb1zzjv.fsf@topspin.com> References: <20050522094037.GC9249@mellanox.co.il> <52y8a75a9t.fsf@topspin.com> <20050523154858.GK9732@mellanox.co.il> <52oeb1zzjv.fsf@topspin.com> Message-ID: On Mon, 23 May 2005, Roland Dreier wrote: > James> Once there is a way to programmatically determine the > James> max_inline_data value, your proposal would be > James> feasible. Even if it were possible, I still believe that > James> the driver should validate this value. > > I think you may be misunderstanding the suggestion. The idea is that > the driver takes the input value of max_inline_data as a hint, and > tries to allocate a QP that supports that value. However, if the > value is too large, the driver reduces the value down to the largest > value that it can support. I don't think this is true: > In any case, the true amount of inline data supported is returned to > the consumer in the qp_cap structure. When I look at mthca_create_qp, the last part of the function is: if (err) { kfree(qp); return ERR_PTR(err); } init_attr->cap.max_inline_data = 0; init_attr->cap.max_send_wr = qp->sq.max; init_attr->cap.max_recv_wr = qp->rq.max; init_attr->cap.max_send_sge = qp->sq.max_gs; init_attr->cap.max_recv_sge = qp->rq.max_gs; return &qp->ibqp; } So if there was an error, the function return without updating the user's cap values. Is this a bug? From mshefty at ichips.intel.com Mon May 23 10:51:17 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Mon, 23 May 2005 10:51:17 -0700 Subject: [openib-general] Re: Re: libibverbs and max inline data In-Reply-To: <20050523154858.GK9732@mellanox.co.il> References: <20050522094037.GC9249@mellanox.co.il> <52y8a75a9t.fsf@topspin.com> <20050523154858.GK9732@mellanox.co.il> Message-ID: <42921815.1030308@ichips.intel.com> Michael S. Tsirkin wrote: >>ib[v]_create_qp should return an error when an unsupported inline data >>size is requested. > > Why is it important? The user could always check the max_inline_data > returned and close the qp if thats too small, but I dont see > many users doing this - inline data is mainly a tuning opportunity. The user could also try to create the QP again if the request fails. IMO, a return value of success should be sufficient for an app to determine if a request was successful. Automatically reducing the inline data size requested by the user, but indicating success is similar to allocating a QP smaller than what was requested, but still indicating success. - Sean From jlentini at netapp.com Mon May 23 10:57:54 2005 From: jlentini at netapp.com (James Lentini) Date: Mon, 23 May 2005 13:57:54 -0400 (EDT) Subject: [openib-general] Re: Re: libibverbs and max inline data In-Reply-To: <20050523162559.GC18721@mellanox.co.il> References: <20050522094037.GC9249@mellanox.co.il> <52y8a75a9t.fsf@topspin.com> <20050523154858.GK9732@mellanox.co.il> <20050523162559.GC18721@mellanox.co.il> Message-ID: On Mon, 23 May 2005, Michael S. Tsirkin wrote: > Quoting r. James Lentini : >> Subject: Re: Re: libibverbs and max inline data >> >> >> >> On Mon, 23 May 2005, Michael S. Tsirkin wrote: >> >>> Quoting r. James Lentini : >>>> Subject: Re: Re: libibverbs and max inline data >>>> >>>> >>>> On Sun, 22 May 2005, Roland Dreier wrote: >>>> >>>>> Michael> Hi! I tried to figure out how much inline data a qp can >>>>> Michael> support, to know what max_inline_data value to put in the >>>>> Michael> qp. However, there doesnt seem to exist a way to do >>>>> Michael> that, currently. >>>>> >>>>> I think you're right. >>>>> >>>>> Michael> Maybe the driver should be changed to trim >>>>> Michael> max_inline_data to the maximum possible value? >>>>> >>>>> Yes, that would probably make sense. In a sense that's what the >>>>> kernel driver already does -- it always clips the max_inline_data >>>>> request from the consumer down to 0. >>>> >>>> ib[v]_create_qp should return an error when an unsupported inline data >>>> size is requested. >>> >>> Why is it important? The user could always check the max_inline_data >>> returned and close the qp if thats too small, but I dont see >>> many users doing this - inline data is mainly a tuning opportunity. >> >> Once there is a way to programmatically determine the max_inline_data >> value, your proposal would be feasible. > > There is a way to do it - the actual values are returned in the attr that you > pass to create_qp. See my previous email to Roland. I believe that mthca_create_qp does not update the attr if an error occurs. > >> Even if it were possible, I >> still believe that the driver should validate this value. >> >> Why have the user specify the ib[v]_qp_cap values if they are going to >> be ignored? > > max_inline_data is special in that its a tuning opportunity. So its > mostly harmless for the driver to ignore. The driver doesn't ignore it though. Take a look at mthca_set_qp_size. If you set max_inline_data to a value such that qp->sq.max_gs will be dev->limits.max_sg, then mthca_set_qp_size will fail with error EINVAL. This will result in the failure of ib_create_qp. >> As a rule, the driver should return an error when the user requests a >> capability/resource that the hardware cannot support. >> >> kDAPL was bitten by this last week. The call to ib_create_qp succeed >> but the value kDAPL was placing in the ib_qp_init_attr's >> max_inline_data field was too large. > > It was just a bug in the mthca kernel driver. > You shouldnt work around such bugs in an application. I'm sorry. My memory was wrong here. ib_create_qp did fail when the max_inline_data field was too large. > >> As a result, a subsequent qp call failed. >> If ib_create_qp had returned an error to begin with, it would >> have been easier to determine the problem. > > It seems wrong for the application to hard-code the max_inline_data > value that suits a particular hardware. Better for the app to tell > the driver what it needs and the driver will give it what hardware > can support. > >>>>> Independent of that we could add a field to the device properties >>>>> struct (and implement ibv_query_device() for userspace). > > ibv_query_device would be useful for other things as well (e.g. > getting the max MTU). > > -- > MST - Michael S. Tsirkin > From Mark.Seger at hp.com Mon May 23 11:49:17 2005 From: Mark.Seger at hp.com (Mark Seger) Date: Mon, 23 May 2005 14:49:17 -0400 Subject: [openib-general] Re: performance counters in /sys In-Reply-To: <1116866524.4481.5774.camel@hal.voltaire.com> References: <35EA21F54A45CB47B879F21A91F4862F581222@taurus.voltaire.com> <1116539301.4479.255.camel@hal.voltaire.com> <428DC946.5080207@hp.com> <20050520161146.GB14411@esmail.cup.hp.com> <20050520164843.GC14411@esmail.cup.hp.com> <20050522053351.GB8522@mellanox.co.il> <4291097F.8070404@hp.com> <42920483.3040900@ichips.intel.com> <1116866524.4481.5774.camel@hal.voltaire.com> Message-ID: <429225AD.4080808@hp.com> Since I was the one who originally created this topic I'd like to restate what I said that got this all started. I'm trying to do relatively lightweight monitoring of lots of system performance counters (on the order of 100-200 or more) across a number of subsystems using standard interfaces. While I don't feel the need to be able to take 10Ks of samples/sec I would like to at least run efficiently at 1-10/sec range. I also want to avoid writing custom kernel code and/or talking directly to hardware. As I said in my base note I'm currently reading from /proc while some sets of counters are better organized than others, I can still access them relatively efficiently. While I could certainly "get by" reading one variable per file, I do worry about the overhead as the sampling frequency goes down. This will also be a problem as the number of counters and devices grow. The suggestion about using perfquery would certainly work, but I'd also be concerned about the overhead in running it at smaller sampling intervals. I certainly understand the desire to move to sysfs and that /usr/src/linux/Documentation/filesystems/sysfs.txt states that "Mixing types, expressing multiple lines of data, and doing fancy formatting of data is heavily frowned upon. Doing these things may get you publically humiliated and your code rewritten without notice." However, I don't read this to mean you must only have one data item per file. For example, I took a look at /sys/block/hda/stat because one of the types of data I collect is disk stats and I was wondering how sysfs dealt with them. Sure enough, they're all in one file per disk as shown below: dl380-2: cat /sys/block/hda/stat 0 0 0 0 0 0 0 0 0 0 0 Also note some of these count bytes, some sectors and other jiffies, so even the units need not be identical. -mark Hal Rosenstock wrote: >On Mon, 2005-05-23 at 12:27, Sean Hefty wrote: > > >>Are there any performance counters that aren't available through the PMA >>MADs? If not, is there any reason why the PMA interface shouldn't be used >>for programmatic access? >> >> > >All the counters found in: >/sys/class/infiniband/mthca0/ports/1/counters >excessive_buffer_overrun_errors port_rcv_remote_physical_errors >link_downed port_rcv_switch_relay_errors >link_error_recovery port_xmit_constraint_errors >local_link_integrity_errors port_xmit_data >port_rcv_constraint_errors port_xmit_discards >port_rcv_data port_xmit_packets >port_rcv_errors symbol_error >port_rcv_packets VL15_dropped > >are available via the PMA (and via the perfquery tool): >/usr/local/ib/bin/perfquery 1 1 ># Port counters: Lid 0x1 port 1 >PortSelect:......................1 >CounterSelect:...................0x0000 >SymbolErrors:....................10344 >LinkRecovers:....................255 >LinkDowned:......................4 >RcvErrors:.......................0 >RcvRemotePhysErrors:.............0 >RcvSwRelayErrors:................0 >XmtDiscards:.....................19 >XmtConstraintErrors:.............0 >RcvConstraintErrors:.............0 >LinkIntegrityErrors:.............0 >ExcBufOverrunErrors:.............0 >VL15Dropped:.....................0 >XmtBytes:........................126990 >RcvBytes:........................126952 >XmtPkts:.........................1791 >RcvBytes:........................1790 > >One advantage is that all counters are retrieved with one MAD. > >-- Hal > > From mshefty at ichips.intel.com Mon May 23 12:28:33 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Mon, 23 May 2005 12:28:33 -0700 Subject: [openib-general] Re: performance counters in /sys In-Reply-To: <429225AD.4080808@hp.com> References: <35EA21F54A45CB47B879F21A91F4862F581222@taurus.voltaire.com> <1116539301.4479.255.camel@hal.voltaire.com> <428DC946.5080207@hp.com> <20050520161146.GB14411@esmail.cup.hp.com> <20050520164843.GC14411@esmail.cup.hp.com> <20050522053351.GB8522@mellanox.co.il> <4291097F.8070404@hp.com> <42920483.3040900@ichips.intel.com> <1116866524.4481.5774.camel@hal.voltaire.com> <429225AD.4080808@hp.com> Message-ID: <42922EE1.3050803@ichips.intel.com> Mark Seger wrote: > As I said in my base note I'm currently reading from /proc while some > sets of counters are better organized than others, I can still access > them relatively efficiently. While I could certainly "get by" reading > one variable per file, I do worry about the overhead as the sampling > frequency goes down. This will also be a problem as the number of > counters and devices grow. The suggestion about using perfquery would > certainly work, but I'd also be concerned about the overhead in running > it at smaller sampling intervals. I could be mistaken here, but I believe that these are hardware based counters that are accessed by generating a Performance Management MAD to retrieve that desired value(s). (It looks like one MAD is generated for each individual counter that is read when accessed through /proc.) I'm not sure that you can get much more efficient that using something similar to perfquery, which retrieves all values using a single MAD, and uses the standard IB method of reading the counters. Additionally, you should be able to access these counters from across the fabric. - Sean From roland at topspin.com Mon May 23 12:57:23 2005 From: roland at topspin.com (Roland Dreier) Date: Mon, 23 May 2005 12:57:23 -0700 Subject: [openib-general] Re: libibverbs and max inline data In-Reply-To: (James Lentini's message of "Mon, 23 May 2005 13:48:19 -0400 (EDT)") References: <20050522094037.GC9249@mellanox.co.il> <52y8a75a9t.fsf@topspin.com> <20050523154858.GK9732@mellanox.co.il> <52oeb1zzjv.fsf@topspin.com> Message-ID: <52hdgtzpek.fsf@topspin.com> Roland> I think you may be misunderstanding the suggestion. The Roland> idea is that the driver takes the input value of Roland> max_inline_data as a hint, and tries to allocate a QP that Roland> supports that value. However, if the value is too large, Roland> the driver reduces the value down to the largest value Roland> that it can support. James> I don't think this is true: Right, it isn't true yet. I was trying to explain Michael's suggestion for improving the interface. James> So if there was an error, the function return without James> updating the user's cap values. Is this a bug? No. What should it return? No QP has been created, so it doesn't make sense to return the capacity of the QP. - R. From jlentini at netapp.com Mon May 23 14:03:18 2005 From: jlentini at netapp.com (James Lentini) Date: Mon, 23 May 2005 17:03:18 -0400 (EDT) Subject: [openib-general] Re: libibverbs and max inline data In-Reply-To: <52hdgtzpek.fsf@topspin.com> References: <20050522094037.GC9249@mellanox.co.il> <52y8a75a9t.fsf@topspin.com> <20050523154858.GK9732@mellanox.co.il> <52oeb1zzjv.fsf@topspin.com> <52hdgtzpek.fsf@topspin.com> Message-ID: On Mon, 23 May 2005, Roland Dreier wrote: > Roland> I think you may be misunderstanding the suggestion. The > Roland> idea is that the driver takes the input value of > Roland> max_inline_data as a hint, and tries to allocate a QP that > Roland> supports that value. However, if the value is too large, > Roland> the driver reduces the value down to the largest value > Roland> that it can support. > > James> I don't think this is true: > > Right, it isn't true yet. I was trying to explain Michael's > suggestion for improving the interface. Ok. > James> So if there was an error, the function return without > James> updating the user's cap values. Is this a bug? > > No. What should it return? No QP has been created, so it doesn't > make sense to return the capacity of the QP. It's fine. I was thinking you wanted to use the ib_qp_init_attr to return valid capability values to the user on failure. Reading back through the thread, I don't know how I got that impression. From sean.hefty at intel.com Mon May 23 14:55:12 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Mon, 23 May 2005 14:55:12 -0700 Subject: [openib-general] [PATCH] [CM] add support to repeat MRA messages Message-ID: The following patch adds support to repeat MRA messages in response to receiving a duplicate REQ, REP, or LAP. Signed-off-by: Sean Hefty Index: cm.c =================================================================== --- cm.c (revision 2417) +++ cm.c (working copy) @@ -143,6 +143,7 @@ struct cm_id_private { u8 local_ack_timeout; u8 retry_count; u8 rnr_retry_count; + u8 service_timeout; struct list_head work_list; atomic_t work_count; @@ -292,12 +293,10 @@ static void cm_free_id(u32 local_id) spin_unlock_irqrestore(&cm.lock, flags); } -static struct cm_id_private * cm_acquire_id(u32 local_id, u32 remote_id) +static struct cm_id_private * cm_get_id(u32 local_id, u32 remote_id) { struct cm_id_private *cm_id_priv; - unsigned long flags; - spin_lock_irqsave(&cm.lock, flags); cm_id_priv = idr_find(&cm.local_id_table, (int) local_id); if (cm_id_priv) { if (cm_id_priv->id.remote_id == remote_id) @@ -305,6 +304,17 @@ static struct cm_id_private * cm_acquire else cm_id_priv = NULL; } + + return cm_id_priv; +} + +static struct cm_id_private * cm_acquire_id(u32 local_id, u32 remote_id) +{ + struct cm_id_private *cm_id_priv; + unsigned long flags; + + spin_lock_irqsave(&cm.lock, flags); + cm_id_priv = cm_get_id(local_id, remote_id); spin_unlock_irqrestore(&cm.lock, flags); return cm_id_priv; @@ -1029,18 +1039,175 @@ static void cm_process_work(struct cm_id ib_destroy_cm_id(&cm_id_priv->id); } +static void cm_format_mra(struct cm_mra_msg *mra_msg, + struct cm_id_private *cm_id_priv, + enum cm_msg_response msg_mraed, u8 service_timeout, + const void *private_data, u8 private_data_len) +{ + /* todo: TID should match REQ or LAP */ + if (msg_mraed == CM_MSG_RESPONSE_OTHER) + cm_format_mad_hdr(&mra_msg->hdr, CM_MRA_ATTR_ID, + cm_form_tid(cm_id_priv, CM_MSG_SEQUENCE_LAP)); + else + cm_format_mad_hdr(&mra_msg->hdr, CM_MRA_ATTR_ID, + cm_form_tid(cm_id_priv, CM_MSG_SEQUENCE_REQ)); + + cm_mra_set_msg_mraed(mra_msg, msg_mraed); + + mra_msg->local_comm_id = cm_id_priv->id.local_id; + mra_msg->remote_comm_id = cm_id_priv->id.remote_id; + cm_mra_set_service_timeout(mra_msg, service_timeout); + + if (private_data && private_data_len) + memcpy(mra_msg->private_data, private_data, private_data_len); +} + +static void cm_format_rej(struct cm_rej_msg *rej_msg, + struct cm_id_private *cm_id_priv, + enum ib_cm_rej_reason reason, + void *ari, + u8 ari_length, + const void *private_data, + u8 private_data_len) +{ + /* todo: TID should match received REQ */ + cm_format_mad_hdr(&rej_msg->hdr, CM_REJ_ATTR_ID, + cm_form_tid(cm_id_priv, CM_MSG_SEQUENCE_REQ)); + + rej_msg->remote_comm_id = cm_id_priv->id.remote_id; + + switch(cm_id_priv->id.state) { + case IB_CM_REQ_RCVD: + rej_msg->local_comm_id = 0; + cm_rej_set_msg_rejected(rej_msg, CM_MSG_RESPONSE_REQ); + break; + case IB_CM_MRA_REQ_SENT: + rej_msg->local_comm_id = cm_id_priv->id.local_id; + cm_rej_set_msg_rejected(rej_msg, CM_MSG_RESPONSE_REQ); + break; + case IB_CM_REP_RCVD: + case IB_CM_MRA_REP_SENT: + rej_msg->local_comm_id = cm_id_priv->id.local_id; + cm_rej_set_msg_rejected(rej_msg, CM_MSG_RESPONSE_REP); + break; + default: + rej_msg->local_comm_id = cm_id_priv->id.local_id; + cm_rej_set_msg_rejected(rej_msg, CM_MSG_RESPONSE_OTHER); + break; + } + + rej_msg->reason = reason; + if (ari && ari_length) { + cm_rej_set_reject_info_len(rej_msg, ari_length); + memcpy(rej_msg->ari, ari, ari_length); + } + + if (private_data && private_data_len) + memcpy(rej_msg->private_data, private_data, private_data_len); +} + +static void cm_dup_req_handler(struct cm_work *work, + struct cm_id_private *cm_id_priv) +{ + struct ib_mad_send_buf *msg = NULL; + struct ib_send_wr *bad_send_wr; + unsigned long flags; + int ret; + + /* Quick state check to discard duplicate REQs. */ + if (cm_id_priv->id.state == IB_CM_REQ_RCVD) + return; + + ret = cm_alloc_response_msg(work->port, work->mad_recv_wc, &msg); + if (ret) + return; + + spin_lock_irqsave(&cm_id_priv->lock, flags); + switch (cm_id_priv->id.state) { + case IB_CM_MRA_REQ_SENT: + cm_format_mra((struct cm_mra_msg *) msg->mad, cm_id_priv, + CM_MSG_RESPONSE_REQ, cm_id_priv->service_timeout, + cm_id_priv->private_data, + cm_id_priv->private_data_len); + break; + case IB_CM_TIMEWAIT: + cm_format_rej((struct cm_rej_msg *) msg->mad, cm_id_priv, + IB_CM_REJ_STALE_CONN, NULL, 0, NULL, 0); + break; + default: + goto unlock; + } + spin_unlock_irqrestore(&cm_id_priv->lock, flags); + + ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent, &msg->send_wr, + &bad_send_wr); + if (ret) + goto free; + return; + +unlock: spin_unlock_irqrestore(&cm_id_priv->lock, flags); +free: cm_free_msg(msg); +} + +static struct cm_id_private * cm_match_req(struct cm_work *work, + struct cm_id_private *cm_id_priv) +{ + struct cm_id_private *listen_cm_id_priv, *cur_cm_id_priv; + struct cm_timewait_info *timewait_info; + struct cm_req_msg *req_msg; + unsigned long flags; + + req_msg = (struct cm_req_msg *)work->mad_recv_wc->recv_buf.mad; + + /* Check for duplicate REQ and stale connections. */ + spin_lock_irqsave(&cm.lock, flags); + timewait_info = cm_insert_remote_id(cm_id_priv->timewait_info); + if (!timewait_info) + timewait_info = cm_insert_remote_qpn(cm_id_priv->timewait_info); + + if (timewait_info) { + cur_cm_id_priv = cm_get_id(timewait_info->work.local_id, + timewait_info->work.remote_id); + spin_unlock_irqrestore(&cm.lock, flags); + if (cur_cm_id_priv) { + cm_dup_req_handler(work, cur_cm_id_priv); + cm_deref_id(cur_cm_id_priv); + } else + cm_issue_rej(work->port, work->mad_recv_wc, + IB_CM_REJ_STALE_CONN, CM_MSG_RESPONSE_REQ, + NULL, 0); + goto error; + } + + /* Find matching listen request. */ + listen_cm_id_priv = cm_find_listen(req_msg->service_id); + if (!listen_cm_id_priv) { + spin_unlock_irqrestore(&cm.lock, flags); + cm_issue_rej(work->port, work->mad_recv_wc, + IB_CM_REJ_INVALID_SERVICE_ID, CM_MSG_RESPONSE_REQ, + NULL, 0); + goto error; + } + atomic_inc(&listen_cm_id_priv->refcount); + atomic_inc(&cm_id_priv->refcount); + cm_id_priv->id.state = IB_CM_REQ_RCVD; + atomic_inc(&cm_id_priv->work_count); + spin_unlock_irqrestore(&cm.lock, flags); + return listen_cm_id_priv; + +error: cm_cleanup_timewait(cm_id_priv->timewait_info); + return NULL; +} + static int cm_req_handler(struct cm_work *work) { struct ib_cm_id *cm_id; struct cm_id_private *cm_id_priv, *listen_cm_id_priv; struct cm_req_msg *req_msg; - unsigned long flags; int ret; req_msg = (struct cm_req_msg *)work->mad_recv_wc->recv_buf.mad; - /* todo: Check for peer-to-peer connection. */ - cm_id = ib_create_cm_id(NULL, NULL); if (IS_ERR(cm_id)) return PTR_ERR(cm_id); @@ -1059,37 +1226,11 @@ static int cm_req_handler(struct cm_work cm_id_priv->timewait_info->remote_ca_guid = req_msg->local_ca_guid; cm_id_priv->timewait_info->remote_qpn = cm_req_get_local_qpn(req_msg); - spin_lock_irqsave(&cm.lock, flags); - /* Check for duplicate REQ. */ - if (cm_insert_remote_id(cm_id_priv->timewait_info)) { - spin_unlock_irqrestore(&cm.lock, flags); - ret = -EINVAL; - goto error2; - } - /* Check for a stale connection. */ - if (cm_insert_remote_qpn(cm_id_priv->timewait_info)) { - spin_unlock_irqrestore(&cm.lock, flags); - cm_issue_rej(work->port, work->mad_recv_wc, - IB_CM_REJ_STALE_CONN, CM_MSG_RESPONSE_REQ, - NULL, 0); - ret = -EINVAL; - goto error2; - } - /* Find matching listen request. */ - listen_cm_id_priv = cm_find_listen(req_msg->service_id); + listen_cm_id_priv = cm_match_req(work, cm_id_priv); if (!listen_cm_id_priv) { - spin_unlock_irqrestore(&cm.lock, flags); - cm_issue_rej(work->port, work->mad_recv_wc, - IB_CM_REJ_INVALID_SERVICE_ID, CM_MSG_RESPONSE_REQ, - NULL, 0); ret = -EINVAL; goto error2; } - atomic_inc(&listen_cm_id_priv->refcount); - atomic_inc(&cm_id_priv->refcount); - cm_id_priv->id.state = IB_CM_REQ_RCVD; - atomic_inc(&cm_id_priv->work_count); - spin_unlock_irqrestore(&cm.lock, flags); cm_id_priv->id.cm_handler = listen_cm_id_priv->id.cm_handler; cm_id_priv->id.context = listen_cm_id_priv->id.context; @@ -1125,8 +1266,8 @@ static int cm_req_handler(struct cm_work error3: atomic_dec(&cm_id_priv->refcount); cm_deref_id(listen_cm_id_priv); -error2: cm_cleanup_timewait(cm_id_priv->timewait_info); - kfree(cm_id_priv->timewait_info); + cm_cleanup_timewait(cm_id_priv->timewait_info); +error2: kfree(cm_id_priv->timewait_info); error1: ib_destroy_cm_id(&cm_id_priv->id); return ret; } @@ -1291,7 +1432,7 @@ static void cm_dup_rep_handler(struct cm { struct cm_id_private *cm_id_priv; struct cm_rep_msg *rep_msg; - struct ib_mad_send_buf *msg; + struct ib_mad_send_buf *msg = NULL; struct ib_send_wr *bad_send_wr; unsigned long flags; int ret; @@ -1304,26 +1445,31 @@ static void cm_dup_rep_handler(struct cm ret = cm_alloc_response_msg(work->port, work->mad_recv_wc, &msg); if (ret) - goto out; + goto deref; spin_lock_irqsave(&cm_id_priv->lock, flags); - if (cm_id_priv->id.state != IB_CM_ESTABLISHED || - cm_id_priv->id.lap_state != IB_CM_LAP_IDLE) { - spin_unlock_irqrestore(&cm_id_priv->lock, flags); - goto error; - } - cm_format_rtu((struct cm_rtu_msg *) msg->mad, cm_id_priv, - cm_id_priv->private_data, cm_id_priv->private_data_len); + if (cm_id_priv->id.state == IB_CM_ESTABLISHED) + cm_format_rtu((struct cm_rtu_msg *) msg->mad, cm_id_priv, + cm_id_priv->private_data, + cm_id_priv->private_data_len); + else if (cm_id_priv->id.state == IB_CM_MRA_REP_SENT) + cm_format_mra((struct cm_mra_msg *) msg->mad, cm_id_priv, + CM_MSG_RESPONSE_REP, cm_id_priv->service_timeout, + cm_id_priv->private_data, + cm_id_priv->private_data_len); + else + goto unlock; spin_unlock_irqrestore(&cm_id_priv->lock, flags); ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent, &msg->send_wr, &bad_send_wr); if (ret) - goto error; - goto out; + goto free; + goto deref; -error: cm_free_msg(msg); -out: cm_deref_id(cm_id_priv); +unlock: spin_unlock_irqrestore(&cm_id_priv->lock, flags); +free: cm_free_msg(msg); +deref: cm_deref_id(cm_id_priv); } static int cm_rep_handler(struct cm_work *work) @@ -1593,7 +1739,7 @@ static int cm_dreq_handler(struct cm_wor { struct cm_id_private *cm_id_priv; struct cm_dreq_msg *dreq_msg; - struct ib_mad_send_buf *msg; + struct ib_mad_send_buf *msg = NULL; struct ib_send_wr *bad_send_wr; unsigned long flags; int ret; @@ -1689,50 +1835,6 @@ out: return -EINVAL; } -static void cm_format_rej(struct cm_rej_msg *rej_msg, - struct cm_id_private *cm_id_priv, - enum ib_cm_rej_reason reason, - void *ari, - u8 ari_length, - const void *private_data, - u8 private_data_len) -{ - /* todo: TID should match received REQ */ - cm_format_mad_hdr(&rej_msg->hdr, CM_REJ_ATTR_ID, - cm_form_tid(cm_id_priv, CM_MSG_SEQUENCE_REQ)); - - rej_msg->remote_comm_id = cm_id_priv->id.remote_id; - - switch(cm_id_priv->id.state) { - case IB_CM_REQ_RCVD: - rej_msg->local_comm_id = 0; - cm_rej_set_msg_rejected(rej_msg, CM_MSG_RESPONSE_REQ); - break; - case IB_CM_MRA_REQ_SENT: - rej_msg->local_comm_id = cm_id_priv->id.local_id; - cm_rej_set_msg_rejected(rej_msg, CM_MSG_RESPONSE_REQ); - break; - case IB_CM_REP_RCVD: - case IB_CM_MRA_REP_SENT: - rej_msg->local_comm_id = cm_id_priv->id.local_id; - cm_rej_set_msg_rejected(rej_msg, CM_MSG_RESPONSE_REP); - break; - default: - rej_msg->local_comm_id = cm_id_priv->id.local_id; - cm_rej_set_msg_rejected(rej_msg, CM_MSG_RESPONSE_OTHER); - break; - } - - rej_msg->reason = reason; - if (ari && ari_length) { - cm_rej_set_reject_info_len(rej_msg, ari_length); - memcpy(rej_msg->ari, ari, ari_length); - } - - if (private_data && private_data_len) - memcpy(rej_msg->private_data, private_data, private_data_len); -} - int ib_send_cm_rej(struct ib_cm_id *cm_id, enum ib_cm_rej_reason reason, void *ari, @@ -1895,48 +1997,6 @@ out: return -EINVAL; } -static void cm_format_mra(struct cm_mra_msg *mra_msg, - struct cm_id_private *cm_id_priv, - u8 service_timeout, - const void *private_data, - u8 private_data_len) -{ - enum cm_msg_sequence msg_sequence; - unsigned long flags; - enum cm_msg_response msg_mraed; - - spin_lock_irqsave(&cm_id_priv->lock, flags); - switch(cm_id_priv->id.state) { - case IB_CM_REQ_RCVD: - msg_sequence = CM_MSG_SEQUENCE_REQ; - msg_mraed = CM_MSG_RESPONSE_REQ; - break; - case IB_CM_REP_RCVD: - msg_sequence = CM_MSG_SEQUENCE_REQ; - msg_mraed = CM_MSG_RESPONSE_REP; - break; - case IB_CM_ESTABLISHED: - msg_sequence = CM_MSG_SEQUENCE_LAP; - msg_mraed = CM_MSG_RESPONSE_OTHER; - break; - default: - spin_unlock_irqrestore(&cm_id_priv->lock, flags); - return; - } - spin_unlock_irqrestore(&cm_id_priv->lock, flags); - /* todo: TID should matched REQ or LAP */ - cm_format_mad_hdr(&mra_msg->hdr, CM_MRA_ATTR_ID, - cm_form_tid(cm_id_priv, msg_sequence)); - cm_mra_set_msg_mraed(mra_msg, msg_mraed); - - mra_msg->local_comm_id = cm_id_priv->id.local_id; - mra_msg->remote_comm_id = cm_id_priv->id.remote_id; - cm_mra_set_service_timeout(mra_msg, service_timeout); - - if (private_data && private_data_len) - memcpy(mra_msg->private_data, private_data, private_data_len); -} - int ib_send_cm_mra(struct ib_cm_id *cm_id, u8 service_timeout, const void *private_data, @@ -1954,44 +2014,50 @@ int ib_send_cm_mra(struct ib_cm_id *cm_i cm_id_priv = container_of(cm_id, struct cm_id_private, id); ret = cm_alloc_msg(cm_id_priv, &msg); if (ret) - goto out; - - cm_format_mra((struct cm_mra_msg *) msg->mad, cm_id_priv, - service_timeout, private_data, private_data_len); + return ret; spin_lock_irqsave(&cm_id_priv->lock, flags); - if (cm_id_priv->id.state == IB_CM_REQ_RCVD || - cm_id_priv->id.state == IB_CM_REP_RCVD || - (cm_id_priv->id.state == IB_CM_ESTABLISHED && - cm_id_priv->id.lap_state == IB_CM_LAP_RCVD)) { - + switch(cm_id_priv->id.state) { + case IB_CM_REQ_RCVD: + cm_format_mra((struct cm_mra_msg *) msg->mad, cm_id_priv, + CM_MSG_RESPONSE_REQ, service_timeout, + private_data, private_data_len); ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent, &msg->send_wr, &bad_send_wr); - } else { - ret = -EINVAL; - } - - if (ret) { - spin_unlock_irqrestore(&cm_id_priv->lock, flags); - cm_free_msg(msg); - goto out; - } - - switch (cm_id_priv->id.state) { - case IB_CM_REQ_RCVD: + if (ret) + goto error; cm_id->state = IB_CM_MRA_REQ_SENT; break; case IB_CM_REP_RCVD: + cm_format_mra((struct cm_mra_msg *) msg->mad, cm_id_priv, + CM_MSG_RESPONSE_REP, service_timeout, + private_data, private_data_len); + ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent, + &msg->send_wr, &bad_send_wr); + if (ret) + goto error; cm_id->state = IB_CM_MRA_REP_SENT; break; case IB_CM_ESTABLISHED: - cm_id_priv->id.lap_state = IB_CM_MRA_LAP_SENT; + cm_format_mra((struct cm_mra_msg *) msg->mad, cm_id_priv, + CM_MSG_RESPONSE_OTHER, service_timeout, + private_data, private_data_len); + ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent, + &msg->send_wr, &bad_send_wr); + if (ret) + goto error; + cm_id->lap_state = IB_CM_MRA_LAP_SENT; break; default: - break; + ret = -EINVAL; + goto error; } + cm_id_priv->service_timeout = service_timeout; spin_unlock_irqrestore(&cm_id_priv->lock, flags); -out: + return 0; + +error: spin_unlock_irqrestore(&cm_id_priv->lock, flags); + cm_free_msg(msg); return ret; } EXPORT_SYMBOL(ib_send_cm_mra); @@ -2177,6 +2243,8 @@ static int cm_lap_handler(struct cm_work struct cm_id_private *cm_id_priv; struct cm_lap_msg *lap_msg; struct ib_cm_lap_event_param *param; + struct ib_mad_send_buf *msg = NULL; + struct ib_send_wr *bad_send_wr; unsigned long flags; int ret; @@ -2193,11 +2261,31 @@ static int cm_lap_handler(struct cm_work work->cm_event.private_data = &lap_msg->private_data; spin_lock_irqsave(&cm_id_priv->lock, flags); - if (cm_id_priv->id.state != IB_CM_ESTABLISHED && - cm_id_priv->id.lap_state != IB_CM_LAP_IDLE) { + if (cm_id_priv->id.state != IB_CM_ESTABLISHED) + goto unlock; + + switch (cm_id_priv->id.lap_state) { + case IB_CM_LAP_IDLE: + break; + case IB_CM_MRA_LAP_SENT: + if (cm_alloc_response_msg(work->port, work->mad_recv_wc, &msg)) + goto unlock; + + cm_format_mra((struct cm_mra_msg *) msg->mad, cm_id_priv, + CM_MSG_RESPONSE_OTHER, + cm_id_priv->service_timeout, + cm_id_priv->private_data, + cm_id_priv->private_data_len); spin_unlock_irqrestore(&cm_id_priv->lock, flags); - goto out; + + if (ib_post_send_mad(cm_id_priv->av.port->mad_agent, + &msg->send_wr, &bad_send_wr)) + cm_free_msg(msg); + goto deref; + default: + goto unlock; } + cm_id_priv->id.lap_state = IB_CM_LAP_RCVD; ret = atomic_inc_and_test(&cm_id_priv->work_count); if (!ret) @@ -2209,8 +2297,9 @@ static int cm_lap_handler(struct cm_work else cm_deref_id(cm_id_priv); return 0; -out: - cm_deref_id(cm_id_priv); + +unlock: spin_unlock_irqrestore(&cm_id_priv->lock, flags); +deref: cm_deref_id(cm_id_priv); return -EINVAL; } From tduffy at sun.com Mon May 23 15:57:08 2005 From: tduffy at sun.com (Tom Duffy) Date: Mon, 23 May 2005 15:57:08 -0700 Subject: [openib-general] [PATCH][RFC] kDAPL: use linux native timers instead of own Message-ID: <1116889028.19768.6.camel@duffman> This patch removes the kDAPL timer implementation and uses the Linux native timer class. Please look over and comment. Signed-off-by: Tom Duffy Index: linux-kernel-timer/test/dapltest/test/dapl_performance_util.c =================================================================== --- linux-kernel-timer/test/dapltest/test/dapl_performance_util.c (revision 2457) +++ linux-kernel-timer/test/dapltest/test/dapl_performance_util.c (working copy) @@ -538,7 +538,7 @@ DT_performance_wait ( pre_ts = DT_Mdep_GetTimeStamp (); ret = DT_Tdep_evd_wait ( evd_handle, - DAT_TIMEOUT_INFINITE, + MAX_SCHEDULE_TIMEOUT, &event); post_ts = DT_Mdep_GetTimeStamp (); Index: linux-kernel-timer/test/dapltest/test/dapl_performance_client.c =================================================================== --- linux-kernel-timer/test/dapltest/test/dapl_performance_client.c (revision 2457) +++ linux-kernel-timer/test/dapltest/test/dapl_performance_client.c (working copy) @@ -105,7 +105,7 @@ retry: ret = dat_ep_connect (test_ptr->ep_context.ep_handle, test_ptr->remote_ia_addr, test_ptr->ep_context.port, - DAT_TIMEOUT_INFINITE, + MAX_SCHEDULE_TIMEOUT, 0, (void *) 0, /* no private data */ test_ptr->cmd->qos, Index: linux-kernel-timer/test/dapltest/test/dapl_server.c =================================================================== --- linux-kernel-timer/test/dapltest/test/dapl_server.c (revision 2457) +++ linux-kernel-timer/test/dapltest/test/dapl_server.c (working copy) @@ -584,7 +584,7 @@ DT_cs_Server (Params_t * params_ptr) DT_Mdep_wait_object_wait ( &pt_ptr->synch_wait_object, - DAT_TIMEOUT_INFINITE); + MAX_SCHEDULE_TIMEOUT); /* Send the Server_Info */ DT_Tdep_PT_Debug (1,(phead,"%s: Send Server_Info\n", module)); Index: linux-kernel-timer/test/dapltest/test/dapl_transaction_test.c =================================================================== --- linux-kernel-timer/test/dapltest/test/dapl_transaction_test.c (revision 2457) +++ linux-kernel-timer/test/dapltest/test/dapl_transaction_test.c (working copy) @@ -714,7 +714,7 @@ retry: ret = dat_ep_connect (test_ptr->ep_context[i].ep_handle, test_ptr->remote_ia_addr, test_ptr->ep_context[i].ia_port, - DAT_TIMEOUT_INFINITE, + MAX_SCHEDULE_TIMEOUT, strlen (private_data_str), private_data_str, pt_ptr->Params.ReliabilityLevel, DAT_CONNECT_DEFAULT_FLAG); Index: linux-kernel-timer/test/dapltest/test/dapl_test_util.c =================================================================== --- linux-kernel-timer/test/dapltest/test/dapl_test_util.c (revision 2457) +++ linux-kernel-timer/test/dapltest/test/dapl_test_util.c (working copy) @@ -261,7 +261,7 @@ DT_cr_event_wait ( DT_Tdep_Print_Head *p u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); + ret = DT_Tdep_evd_wait (evd_handle, MAX_SCHEDULE_TIMEOUT, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (CR) failed: %s\n", @@ -316,7 +316,7 @@ DT_conn_event_wait (DT_Tdep_Print_Head * u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); + ret = DT_Tdep_evd_wait (evd_handle, MAX_SCHEDULE_TIMEOUT, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (CONN) failed: %s\n", @@ -369,7 +369,7 @@ DT_disco_event_wait ( DT_Tdep_Print_Head u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); + ret = DT_Tdep_evd_wait (evd_handle, MAX_SCHEDULE_TIMEOUT, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (DISCONN) failed: %s\n", @@ -492,7 +492,7 @@ DT_dto_event_wait (DT_Tdep_Print_Head *p u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); + ret = DT_Tdep_evd_wait (evd_handle, MAX_SCHEDULE_TIMEOUT, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (DTO) failed: %s\n", @@ -536,7 +536,7 @@ DT_rmr_event_wait (DT_Tdep_Print_Head *p u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); + ret = DT_Tdep_evd_wait (evd_handle, MAX_SCHEDULE_TIMEOUT, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (RMR) failed: %s\n", Index: linux-kernel-timer/test/dapltest/test/dapl_client.c =================================================================== --- linux-kernel-timer/test/dapltest/test/dapl_client.c (revision 2457) +++ linux-kernel-timer/test/dapltest/test/dapl_client.c (working copy) @@ -256,7 +256,7 @@ retry: ret = dat_ep_connect (ep_handle, server_netaddr, SERVER_PORT_NUMBER, - DAT_TIMEOUT_INFINITE, + MAX_SCHEDULE_TIMEOUT, 0, (void *) 0, /* no private data */ params_ptr->ReliabilityLevel, DAT_CONNECT_DEFAULT_FLAG); Index: linux-kernel-timer/test/dapltest/include/dapl_tdep.h =================================================================== --- linux-kernel-timer/test/dapltest/include/dapl_tdep.h (revision 2457) +++ linux-kernel-timer/test/dapltest/include/dapl_tdep.h (working copy) @@ -56,7 +56,7 @@ DT_Tdep_evd_free (DAT_EVD_HANDLE evd_ha u32 DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_handle, - DAT_TIMEOUT timeout, + unsigned long timeout, struct dat_event *event); u32 DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_handle, Index: linux-kernel-timer/test/dapltest/include/dapl_proto.h =================================================================== --- linux-kernel-timer/test/dapltest/include/dapl_proto.h (revision 2457) +++ linux-kernel-timer/test/dapltest/include/dapl_proto.h (working copy) @@ -189,7 +189,7 @@ void DT_Mdep_Thread_EXIT ( void * thr int DT_Mdep_wait_object_init ( DT_WAIT_OBJECT *wait_obj); int DT_Mdep_wait_object_wait ( DT_WAIT_OBJECT *wait_obj, - int timeout_val); + unsigned long timeout_val); int DT_Mdep_wait_object_wakeup ( DT_WAIT_OBJECT *wait_obj); int DT_Mdep_wait_object_destroy ( DT_WAIT_OBJECT *wait_obj); Index: linux-kernel-timer/test/dapltest/include/dapl_transaction_test.h =================================================================== --- linux-kernel-timer/test/dapltest/include/dapl_transaction_test.h (revision 2457) +++ linux-kernel-timer/test/dapltest/include/dapl_transaction_test.h (working copy) @@ -79,7 +79,7 @@ typedef struct Transaction_Cmd_t *cmd; struct sockaddr * remote_ia_addr; DAT_CONN_QUAL base_port; - DAT_TIMEOUT time_out; + unsigned long time_out; int evd_length; Thread *thread; Index: linux-kernel-timer/test/dapltest/mdep/linux/dapl_mdep_user.c =================================================================== --- linux-kernel-timer/test/dapltest/mdep/linux/dapl_mdep_user.c (revision 2457) +++ linux-kernel-timer/test/dapltest/mdep/linux/dapl_mdep_user.c (working copy) @@ -459,7 +459,7 @@ DT_Mdep_wait_object_init ( int DT_Mdep_wait_object_wait ( DT_WAIT_OBJECT *wait_obj, - int timeout_val) + unsigned long timeout_val) { int dat_status; int pthread_status; Index: linux-kernel-timer/test/dapltest/mdep/linux/dapl_mdep_kernel.c =================================================================== --- linux-kernel-timer/test/dapltest/mdep/linux/dapl_mdep_kernel.c (revision 2457) +++ linux-kernel-timer/test/dapltest/mdep/linux/dapl_mdep_kernel.c (working copy) @@ -313,13 +313,13 @@ DT_Mdep_wait_object_init ( int DT_Mdep_wait_object_wait ( DT_WAIT_OBJECT *wait_obj, - int timeout_val) + unsigned long timeout_val) { int expire; int dat_status; dat_status = DAT_SUCCESS; - if ( DAT_TIMEOUT_INFINITE == timeout_val) + if ( MAX_SCHEDULE_TIMEOUT == timeout_val) { interruptible_sleep_on (wait_obj); } Index: linux-kernel-timer/test/dapltest/kdapl/kdapl_tdep_evd.c =================================================================== --- linux-kernel-timer/test/dapltest/kdapl/kdapl_tdep_evd.c (revision 2457) +++ linux-kernel-timer/test/dapltest/kdapl/kdapl_tdep_evd.c (working copy) @@ -214,7 +214,7 @@ DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_ u32 DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_handle, - DAT_TIMEOUT timeout, + unsigned long timeout, struct dat_event *dat_event) { u32 dat_status; @@ -240,7 +240,7 @@ DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_han return dat_status; } /* wait for an event */ - if (timeout == DAT_TIMEOUT_INFINITE) + if (timeout == MAX_SCHEDULE_TIMEOUT) { rc = wait_event_interruptible (evd_ptr->wait_object, evd_ptr->event_next != NULL); Index: linux-kernel-timer/dat-provider/dapl_ep_connect.c =================================================================== --- linux-kernel-timer/dat-provider/dapl_ep_connect.c (revision 2457) +++ linux-kernel-timer/dat-provider/dapl_ep_connect.c (working copy) @@ -35,7 +35,6 @@ #include "dapl_ep_util.h" #include "dapl_openib_util.h" #include "dapl_evd_util.h" -#include "dapl_timer_util.h" /* * Request a connection be established between the local Endpoint @@ -44,7 +43,7 @@ */ u32 dapl_ep_connect(DAT_EP_HANDLE ep_handle, struct sockaddr *remote_ia_address, - DAT_CONN_QUAL remote_conn_qual, DAT_TIMEOUT timeout, + DAT_CONN_QUAL remote_conn_qual, unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos qos, enum dat_connect_flags connect_flags) { @@ -278,23 +277,17 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); if (ep_ptr->param.ep_state == DAT_EP_STATE_ACTIVE_CONNECTION_PENDING - && timeout != DAT_TIMEOUT_INFINITE) { - ep_ptr->cxn_timer = kmalloc(sizeof *ep_ptr->cxn_timer, - GFP_ATOMIC); - if (!ep_ptr->cxn_timer) { - dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, - DAT_RESOURCE_MEMORY); - goto bail; - } - - dapl_timer_set(ep_ptr->cxn_timer, - dapl_ep_timeout, ep_ptr, timeout); + && timeout != MAX_SCHEDULE_TIMEOUT) { + ep_ptr->cxn_timer.expires = jiffies + timeout; + ep_ptr->cxn_timer.data = (unsigned long)ep_ptr; + ep_ptr->cxn_timer.function = dapl_ep_timeout; + add_timer(&ep_ptr->cxn_timer); } spin_unlock_irqrestore(&ep_ptr->header.lock, ep_ptr->header.flags); } - bail: +bail: dapl_dbg_log(DAPL_DBG_TYPE_RTN | DAPL_DBG_TYPE_CM, "dapl_ep_connect () returns 0x%x\n", dat_status); Index: linux-kernel-timer/dat-provider/dapl_module.c =================================================================== --- linux-kernel-timer/dat-provider/dapl_module.c (revision 2457) +++ linux-kernel-timer/dat-provider/dapl_module.c (working copy) @@ -36,7 +36,6 @@ #include "dapl_mr_util.h" #include "dapl_util.h" #include "dapl_openib_util.h" -#include "dapl_timer_util.h" MODULE_LICENSE("Dual BSD/GPL"); MODULE_DESCRIPTION("DAT Provider for InfiniBand"); @@ -57,8 +56,6 @@ static int __init dapl_init(void) dapl_dbg_log(DAPL_DBG_TYPE_UTIL, "provider started\n"); - dapl_timer_init(); - dat_status = dapl_provider_list_create(); if (DAT_SUCCESS != dat_status) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, Index: linux-kernel-timer/dat-provider/dapl_evd_connection_callb.c =================================================================== --- linux-kernel-timer/dat-provider/dapl_evd_connection_callb.c (revision 2457) +++ linux-kernel-timer/dat-provider/dapl_evd_connection_callb.c (working copy) @@ -32,7 +32,6 @@ #include "dapl.h" #include "dapl_evd_util.h" #include "dapl_ep_util.h" -#include "dapl_timer_util.h" /* * dapl_evd_connection_callback @@ -90,11 +89,7 @@ void dapl_evd_connection_callback(struct /* * If a connection timer has been set up on this EP, cancel it now */ - if (ep_ptr->cxn_timer != NULL) { - dapl_timer_cancel(ep_ptr->cxn_timer); - kfree(ep_ptr->cxn_timer); - ep_ptr->cxn_timer = NULL; - } + del_timer(&ep_ptr->cxn_timer); switch (event) { case DAT_CONNECTION_EVENT_ESTABLISHED: Index: linux-kernel-timer/dat-provider/Makefile =================================================================== --- linux-kernel-timer/dat-provider/Makefile (revision 2457) +++ linux-kernel-timer/dat-provider/Makefile (working copy) @@ -101,7 +101,6 @@ PROVIDER_MODULES := \ dapl_srq_resize \ dapl_srq_set_lw \ dapl_srq_util \ - dapl_timer_util \ dapl_util PROVIDER_OBJS := $(foreach s, $(PROVIDER_MODULES), $(s).o) Index: linux-kernel-timer/dat-provider/dapl_ep_free.c =================================================================== --- linux-kernel-timer/dat-provider/dapl_ep_free.c (revision 2457) +++ linux-kernel-timer/dat-provider/dapl_ep_free.c (working copy) @@ -34,7 +34,6 @@ #include "dapl_ep_util.h" #include "dapl_openib_util.h" #include "dapl_ring_buffer_util.h" -#include "dapl_timer_util.h" /* * dapl_ep_free @@ -107,11 +106,7 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle */ spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); - if (ep_ptr->cxn_timer != NULL) { - dapl_timer_cancel(ep_ptr->cxn_timer); - kfree(ep_ptr->cxn_timer); - ep_ptr->cxn_timer = NULL; - } + del_timer(&ep_ptr->cxn_timer); /* Remove the EP from the IA */ dapl_ia_unlink_ep(ia_ptr, ep_ptr); Index: linux-kernel-timer/dat-provider/dapl_timer_util.c =================================================================== --- linux-kernel-timer/dat-provider/dapl_timer_util.c (revision 2457) +++ linux-kernel-timer/dat-provider/dapl_timer_util.c (working copy) @@ -1,313 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * Routines to add and cancel timer records. A timer record - * is put on the global timer queue. If the timer thread is - * not running, start it. The timer thread will sleep - * until a timer event or until a process wakes it up - * to notice a new timer is available; we use a DAPL_WAIT_OBJ - * for synchronization. - * - * If a timer is cancelled, it is simlpy removed from the - * queue. The timer may wakeup and notice there is no timer - * record to awaken at this time, so it will reset for the - * next entry. When there are no timer records to manage, - * the timer thread just sleeps until awakened. - * - * This file also contains the timer handler thread, - * embodied in dapl_timer_thread(). - * - * $Id$ - */ - -#include "dapl.h" -#include "dapl_timer_util.h" - -struct dapl_timer_head { - struct dapl_llist_entry *timer_list_head; - spinlock_t lock; - unsigned long flags; - DAPL_OS_WAIT_OBJECT wait_object; - int thread_id; -}; - -static struct dapl_timer_head g_dapl_timer_head; - -static int dapl_timer_thread(void *arg); - -void dapl_timer_init(void) -{ - /* - * Set up the timer thread elements. The timer thread isn't - * started until it is actually needed - */ - g_dapl_timer_head.timer_list_head = NULL; - spin_lock_init(&g_dapl_timer_head.lock); - dapl_os_wait_object_init(&g_dapl_timer_head.wait_object); - g_dapl_timer_head.thread_id = -1; -} - -/* - * dapl_timer_set - * - * Set a timer. The timer will invoke the specified function - * after a number of useconds expires. - * - * Input: - * timer User provided timer structure - * func Function to invoke when timer expires - * data Argument passed to func() - * expires microseconds until timer fires - * - * Returns: - * no return value - * - */ -u32 dapl_timer_set(struct dapl_timer_entry *timer, void (*func) (void *), - void *data, DAPL_OS_TIMEVAL expires) -{ - struct dapl_timer_entry *list_ptr; - DAPL_OS_TIMEVAL cur_time; - boolean_t wakeup_tmo_thread; - - /* - * Start the timer thread the first time we need a timer - */ - if ( -1 == g_dapl_timer_head.thread_id ) { - g_dapl_timer_head.thread_id = - kernel_thread(dapl_timer_thread, &g_dapl_timer_head, 0); - } - - dapl_llist_init_entry(&timer->list_entry); - wakeup_tmo_thread = FALSE; - dapl_os_get_time(&cur_time); - timer->expires = cur_time + expires; /* calculate future time */ - timer->function = func; - timer->data = data; - - /* - * Put the element on the queue: sorted by wakeup time, eariliest - * first. - */ - spin_lock_irqsave(&g_dapl_timer_head.lock, g_dapl_timer_head.flags); - /* - * Deal with 3 cases due to our list structure: - * 1) list is empty: become the list head - * 2) New timer is sooner than list head: become the list head - * 3) otherwise, sort the timer into the list, no need to wake - * the timer thread up - */ - if (dapl_llist_is_empty(&g_dapl_timer_head.timer_list_head)) { - /* Case 1: add entry to head of list */ - dapl_llist_add_head(&g_dapl_timer_head.timer_list_head, - (struct dapl_llist_entry *) - &timer->list_entry, - timer); - wakeup_tmo_thread = TRUE; - } else { - list_ptr = (struct dapl_timer_entry *) - dapl_llist_peek_head(&g_dapl_timer_head.timer_list_head); - - if (timer->expires < list_ptr->expires) { - /* Case 2: add entry to head of list */ - dapl_llist_add_head(&g_dapl_timer_head.timer_list_head, - (struct dapl_llist_entry *) - &timer->list_entry, timer); - wakeup_tmo_thread = TRUE; - } else { - /* Case 3: figure out where entry goes in sorted list */ - list_ptr = - dapl_llist_next_entry(&g_dapl_timer_head. - timer_list_head, - (struct dapl_llist_entry *) - &list_ptr->list_entry); - - while (list_ptr != NULL) { - if (timer->expires < list_ptr->expires) { - dapl_llist_add_entry(&g_dapl_timer_head. - timer_list_head, - (struct dapl_llist_entry *) - &list_ptr->list_entry, - (struct dapl_llist_entry *) - &timer->list_entry, - timer); - break; - - } - list_ptr = - dapl_llist_next_entry(&g_dapl_timer_head. - timer_list_head, - (struct dapl_llist_entry *) - &list_ptr->list_entry); - } - if (list_ptr == NULL) { - /* entry goes to the end of the list */ - dapl_llist_add_tail(&g_dapl_timer_head. - timer_list_head, - (struct dapl_llist_entry *) - &timer->list_entry, timer); - } - } - - } - spin_unlock_irqrestore(&g_dapl_timer_head.lock, - g_dapl_timer_head.flags); - - if (wakeup_tmo_thread == TRUE) { - dapl_os_wait_object_wakeup(&g_dapl_timer_head.wait_object); - } - - return DAT_SUCCESS; -} - -/* - * dapl_timer_cancel - * - * Cancel a timer. Simply deletes the timer with no function invocations - * - * Input: - * timer User provided timer structure - * - * Returns: - * no return value - */ -void dapl_timer_cancel(struct dapl_timer_entry *timer) -{ - spin_lock_irqsave(&g_dapl_timer_head.lock, g_dapl_timer_head.flags); - /* - * make sure the entry has not been removed by another thread - */ - if (!dapl_llist_is_empty(&g_dapl_timer_head.timer_list_head) && - timer->list_entry.list_head == &g_dapl_timer_head.timer_list_head) { - dapl_llist_remove_entry(&g_dapl_timer_head.timer_list_head, - (struct dapl_llist_entry *) - &timer->list_entry); - } - /* - * If this was the first entry on the queue we could awaken the - * thread and have it reset the list; but it will just wake up - * and find that the timer entry has been removed, then go back - * to sleep, so don't bother. - */ - spin_unlock_irqrestore(&g_dapl_timer_head.lock, - g_dapl_timer_head.flags); -} - -/* - * dapl_timer_thread - * - * Core worker thread dealing with all timers. Basic algorithm: - * - Sleep until work shows up - * - Take first element of sorted timer list and wake - * invoke the callback if expired - * - Sleep for the timeout period if not expired - * - * Input: - * timer_head Timer head structure to manage timer lists - * - * Returns: - * int - */ -int dapl_timer_thread(void *arg) -{ - struct dapl_timer_entry *list_ptr; - DAPL_OS_TIMEVAL cur_time; - struct dapl_timer_head *timer_head; - u32 dat_status; - - timer_head = arg; - - for (;;) { - if (dapl_llist_is_empty(&timer_head->timer_list_head)) { - dat_status = - dapl_os_wait_object_wait(&timer_head->wait_object, - DAT_TIMEOUT_INFINITE); - } - - /* - * Lock policy: - * While this thread is accessing the timer list, it holds the - * lock. Otherwise, it doesn't. - */ - spin_lock_irqsave(&timer_head->lock, timer_head->flags); - while (!dapl_llist_is_empty(&timer_head->timer_list_head)) { - list_ptr = (struct dapl_timer_entry *) - dapl_llist_peek_head(&g_dapl_timer_head. - timer_list_head); - dapl_os_get_time(&cur_time); - - if (list_ptr->expires <= cur_time) { - /* - * Remove the entry from the list. Sort out how much - * time we need to sleep for the next one - */ - list_ptr = - dapl_llist_remove_head(&timer_head-> - timer_list_head); - spin_unlock_irqrestore(&timer_head->lock, - timer_head->flags); - - /* - * Invoke the user callback - */ - list_ptr->function(list_ptr->data); - /* timer structure was allocated by caller, we don't - * free it here. - */ - - /* reacquire the lock */ - spin_lock_irqsave(&timer_head->lock, - timer_head->flags); - } else { - spin_unlock_irqrestore(&timer_head->lock, - timer_head->flags); - dat_status = - dapl_os_wait_object_wait(&timer_head-> - wait_object, - (DAT_TIMEOUT) - (list_ptr-> - expires - - cur_time)); - spin_lock_irqsave(&timer_head->lock, - timer_head->flags); - } - } - /* - * release the lock before going back to the top to sleep - */ - spin_unlock_irqrestore(&timer_head->lock, timer_head->flags); - - if (dat_status == DAT_INTERNAL_ERROR) { - /* - * XXX What do we do here? - */ - } - } /* for (;;) */ - - return 0; -} Index: linux-kernel-timer/dat-provider/dapl_timer_util.h =================================================================== --- linux-kernel-timer/dat-provider/dapl_timer_util.h (revision 2457) +++ linux-kernel-timer/dat-provider/dapl_timer_util.h (working copy) @@ -1,42 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#ifndef DAPL_TIMER_UTIL_H -#define DAPL_TIMER_UTIL_H - -void dapl_timer_init(void); - -u32 dapl_timer_set(struct dapl_timer_entry *timer, void (*func) (void *), - void *data, DAPL_OS_TIMEVAL expires); - -void dapl_timer_cancel(struct dapl_timer_entry *timer); - -#endif /* DAPL_TIMER_UTIL_H */ Index: linux-kernel-timer/dat-provider/dapl_ep_util.c =================================================================== --- linux-kernel-timer/dat-provider/dapl_ep_util.c (revision 2457) +++ linux-kernel-timer/dat-provider/dapl_ep_util.c (working copy) @@ -75,6 +75,7 @@ struct dapl_ep *dapl_ep_alloc(struct dap dapl_llist_init_entry(&ep_ptr->header.ia_list_entry); spin_lock_init(&ep_ptr->header.lock); + init_timer(&ep_ptr->cxn_timer); /* * Initialize the body @@ -169,8 +170,7 @@ void dapl_ep_dealloc(struct dapl_ep *ep_ if (ep_ptr->send_iov) kfree(ep_ptr->send_iov); - if (ep_ptr->cxn_timer) - kfree(ep_ptr->cxn_timer); + del_timer(&ep_ptr->cxn_timer); kfree(ep_ptr); } @@ -248,7 +248,7 @@ bail: * If this routine is invoked before a connection occurs, generate an * event */ -void dapl_ep_timeout(void *arg) +void dapl_ep_timeout(unsigned long arg) { struct dapl_ep *ep_ptr; Index: linux-kernel-timer/dat-provider/dapl_ep_util.h =================================================================== --- linux-kernel-timer/dat-provider/dapl_ep_util.h (revision 2457) +++ linux-kernel-timer/dat-provider/dapl_ep_util.h (working copy) @@ -52,7 +52,7 @@ extern u32 dapl_ep_post_send_req(DAT_EP_ DAPL_DTO_TYPE dto_type, enum ib_wr_opcode op_type); -void dapl_ep_timeout(void *arg); +void dapl_ep_timeout(unsigned long arg); enum dat_return_subtype dapl_ep_state_subtype(struct dapl_ep *ep_ptr); Index: linux-kernel-timer/dat-provider/dapl_util.c =================================================================== --- linux-kernel-timer/dat-provider/dapl_util.c (revision 2457) +++ linux-kernel-timer/dat-provider/dapl_util.c (working copy) @@ -33,104 +33,6 @@ #include "dapl_module.h" #include "dapl_util.h" -/* - * dapl_os_get_time - * - * Return 64 bit value of current time in microseconds. - * - * Input: - * loc User location to place current time - * - * Returns: - * DAT_SUCCESS - */ - -u32 dapl_os_get_time(DAPL_OS_TIMEVAL *loc) -{ - struct timeval tv; - - do_gettimeofday(&tv); - *loc = ((u64) (tv.tv_sec) * 1000000L) + (u64) tv.tv_usec; - - return DAT_SUCCESS; -} - -/* - * Wait object routines - */ - -/* - * dapl_os_wait_object_init - * - * Initialize a wait object - * - * Input: - * wait_obj - * - * Returns: - * DAT_SUCCESS - * DAT_INTERNAL_ERROR - */ -u32 dapl_os_wait_object_init(DAPL_OS_WAIT_OBJECT *wait_obj) -{ - init_waitqueue_head(&wait_obj->wait_queue); - - return DAT_SUCCESS; -} - -/* Wait on the supplied wait object, up to the specified time_out. - * A timeout of DAT_TIMEOUT_INFINITE will wait indefinitely. - * Timeout should be specified in micro seconds. - * - * Functional returns: - * DAT_SUCCESS -- another thread invoked dapl_os_wait object_wakeup - * DAT_INVALID_STATE -- someone else is already waiting in this wait - * object. - * only one waiter is allowed at a time. - * DAT_ABORT -- another thread invoked dapl_os_wait_object_destroy - * DAT_TIMEOUT -- the specified time limit was reached. - */ - -u32 dapl_os_wait_object_wait(DAPL_OS_WAIT_OBJECT *wait_obj, - DAT_TIMEOUT timeout_val) -{ - u32 dat_status = DAT_SUCCESS; - - if (DAT_TIMEOUT_INFINITE == timeout_val) { - interruptible_sleep_on(&wait_obj->wait_queue); - } else { - int expire; - - expire = timeout_val * HZ / 1000000; - - while (expire) { - current->state = TASK_INTERRUPTIBLE; - expire = schedule_timeout(expire); - } - dat_status = DAT_ERROR(DAT_TIMEOUT_EXPIRED, 0); - } - return dat_status; -} - -/* - * dapl_os_wait_object_wakeup - * - * Wakeup a thread waiting on a wait object - * - * Input: - * wait_obj - * - * Returns: - * DAT_SUCCESS - * DAT_INTERNAL_ERROR - */ -u32 dapl_os_wait_object_wakeup(DAPL_OS_WAIT_OBJECT *wait_obj) -{ - wake_up_interruptible(&wait_obj->wait_queue); - - return DAT_SUCCESS; -} - #ifdef DAPL_DBG void dapl_dbg_log(DAPL_DBG_TYPE type, const char *fmt, ...) Index: linux-kernel-timer/dat-provider/dapl_util.h =================================================================== --- linux-kernel-timer/dat-provider/dapl_util.h (revision 2457) +++ linux-kernel-timer/dat-provider/dapl_util.h (working copy) @@ -107,24 +107,6 @@ static inline int dapl_os_atomic_assign( */ /* - * The wait object invariant: Presuming a call to dapl_os_wait_object_wait - * occurs at some point, there will be at least one wakeup after each call - * to dapl_os_wait_object_signal. I.e. Signals are not ignored, though - * they may be coallesced. - */ -typedef struct dapl_os_wait_object { - wait_queue_head_t wait_queue; -} DAPL_OS_WAIT_OBJECT; - -/* function prototypes */ -u32 dapl_os_wait_object_init(DAPL_OS_WAIT_OBJECT *wait_obj); - -u32 dapl_os_wait_object_wait(DAPL_OS_WAIT_OBJECT *wait_obj, - DAT_TIMEOUT timeout_val); - -u32 dapl_os_wait_object_wakeup(DAPL_OS_WAIT_OBJECT *wait_obj); - -/* * Memory Functions */ @@ -154,15 +136,6 @@ static inline char *dapl_os_strdup(const } /* - * Timer Functions - */ - -typedef u64 DAPL_OS_TIMEVAL; - -/* timer function prototype */ -u32 dapl_os_get_time(DAPL_OS_TIMEVAL *); - -/* * *printf format helper. We use the C string constant concatenation * ability to define 64 bit formats, which unfortunatly are non standard * in the C compiler world. Index: linux-kernel-timer/dat-provider/dapl_ep_dup_connect.c =================================================================== --- linux-kernel-timer/dat-provider/dapl_ep_dup_connect.c (revision 2457) +++ linux-kernel-timer/dat-provider/dapl_ep_dup_connect.c (working copy) @@ -32,7 +32,6 @@ #include "dapl.h" #include "dapl_ep_util.h" #include "dapl_openib_util.h" -#include "dapl_timer_util.h" /* * dapl_ep_dup_connect @@ -63,7 +62,7 @@ * DAT_MODEL_NOT_SUPPORTED */ u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, DAT_EP_HANDLE ep_dup_handle, - DAT_TIMEOUT timeout, int private_data_size, + unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos qos) { struct dapl_ep *ep_dup_ptr; @@ -114,6 +113,6 @@ u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep private_data_size, private_data, qos, DAT_CONNECT_DEFAULT_FLAG); - bail: +bail: return dat_status; } Index: linux-kernel-timer/dat-provider/dapl.h =================================================================== --- linux-kernel-timer/dat-provider/dapl.h (revision 2457) +++ linux-kernel-timer/dat-provider/dapl.h (working copy) @@ -237,14 +237,6 @@ struct dapl_evd { struct dapl_cno *cno_ptr; }; -/* DAPL timer entry, used to queue timeouts */ -struct dapl_timer_entry { - struct dapl_llist_entry list_entry; /* link entry on ia struct */ - DAPL_OS_TIMEVAL expires; - void (*function) (void *); - void *data; -}; - /* struct dapl_ep maps to DAT_EP_HANDLE */ struct dapl_ep { struct dapl_header header; @@ -266,8 +258,8 @@ struct dapl_ep { /* For passive connections we maintain a back pointer to the CR */ void *cr_ptr; - /* pointer to connection timer, if set */ - struct dapl_timer_entry *cxn_timer; + /* connection timer */ + struct timer_list cxn_timer; /* private data container */ char private_data[DAPL_MAX_PRIVATE_DATA_SIZE]; @@ -500,14 +492,15 @@ extern u32 dapl_ep_modify(DAT_EP_HANDLE extern u32 dapl_ep_connect(DAT_EP_HANDLE ep_handle, struct sockaddr *remote_ia_address, DAT_CONN_QUAL remote_conn_qual, - DAT_TIMEOUT timeout, + unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos quality_of_service, enum dat_connect_flags connect_flags); extern u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, - DAT_EP_HANDLE ep_dup_handle, DAT_TIMEOUT timeout, + DAT_EP_HANDLE ep_dup_handle, + unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos quality_of_service); Index: linux-kernel-timer/patches/alt_dat_provider_makefile =================================================================== --- linux-kernel-timer/patches/alt_dat_provider_makefile (revision 2457) +++ linux-kernel-timer/patches/alt_dat_provider_makefile (working copy) @@ -95,7 +95,6 @@ PROVIDER_MODULES := \ dapl_srq_resize \ dapl_srq_set_lw \ dapl_srq_util \ - dapl_timer_util \ dapl_util PROVIDER_OBJS := $(foreach s, $(PROVIDER_MODULES), $(s).o) @@ -103,7 +102,7 @@ PROVIDER_OBJS := $(foreach s, $(PROVIDER KDAPL_DEBUG = 1 ifeq (1,$(KDAPL_DEBUG)) EXTRA_CFLAGS += -O0 -g - EXTRA_CFLAGS += -DDAPL_DBG # -DDAPL_DBG_IO_TRC + EXTRA_CFLAGS += -DDAPL_DBG endif EXTRA_CFLAGS += -DDAPL_ATS -Idrivers/infiniband/include -I$(obj)/../dat -I$(obj)/../patches/ @@ -121,4 +120,4 @@ default: endif clean: - rm -f *.o *.ko + rm -rf *.o *.ko .*.cmd *.mod.* .tmp_versions .*.d .*.o.tmp cscope.out Index: linux-kernel-timer/dat/dat.h =================================================================== --- linux-kernel-timer/dat/dat.h (revision 2457) +++ linux-kernel-timer/dat/dat.h (working copy) @@ -383,11 +383,6 @@ enum dat_completion_flags { DAT_COMPLETION_BARRIER_FENCE_FLAG = 0x08, }; -typedef u32 DAT_TIMEOUT; /* microseconds */ - -/* timeout = infinity */ -#define DAT_TIMEOUT_INFINITE ((DAT_TIMEOUT) ~0) - /* dat handles */ typedef void *DAT_HANDLE; typedef DAT_HANDLE DAT_CR_HANDLE; @@ -1139,12 +1134,12 @@ typedef u32 (*DAT_EP_MODIFY_FUNC)(DAT_EP const struct dat_ep_param *); typedef u32 (*DAT_EP_CONNECT_FUNC)(DAT_EP_HANDLE, struct sockaddr *, - DAT_CONN_QUAL, DAT_TIMEOUT, int, + DAT_CONN_QUAL, unsigned long timeout, int, const void *, enum dat_qos, enum dat_connect_flags); typedef u32 (*DAT_EP_DUP_CONNECT_FUNC)(DAT_EP_HANDLE, DAT_EP_HANDLE, - DAT_TIMEOUT, int, const void *, + unsigned long timeout, int, const void *, enum dat_qos); typedef u32 (*DAT_EP_DISCONNECT_FUNC)(DAT_EP_HANDLE, enum dat_close_flags); @@ -1493,7 +1488,7 @@ static inline u32 dat_evd_resize(DAT_EVD } static inline u32 dat_ep_connect(DAT_EP_HANDLE ep, struct sockaddr *ia_addr, - DAT_CONN_QUAL conn_qual, DAT_TIMEOUT timeout, + DAT_CONN_QUAL conn_qual, unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos qos, enum dat_connect_flags flags) @@ -1533,7 +1528,8 @@ static inline u32 dat_ep_disconnect(DAT_ } static inline u32 dat_ep_dup_connect(DAT_EP_HANDLE ep, DAT_EP_HANDLE dup_ep, - DAT_TIMEOUT timeout, int private_data_size, + unsigned long timeout, + int private_data_size, const void *private_data, enum dat_qos qos) { return DAT_CALL_PROVIDER_FUNC(ep_dup_connect_func, ep, dup_ep, timeout, From libor at topspin.com Mon May 23 16:04:25 2005 From: libor at topspin.com (Libor Michalek) Date: Mon, 23 May 2005 16:04:25 -0700 Subject: [openib-general] Re: Possible minor change to ucm.c In-Reply-To: <1116600579.4479.19.camel@hal.voltaire.com>; from halr@voltaire.com on Fri, May 20, 2005 at 10:49:40AM -0400 References: <1116600579.4479.19.camel@hal.voltaire.com> Message-ID: <20050523160425.A17273@topspin.com> On Fri, May 20, 2005 at 10:49:40AM -0400, Hal Rosenstock wrote: > Hi Libor, > > A possible minor change to ucm.c::ib_ucm_event_handler line 440 would > be: > > uevent->cm_id = (id == IB_UCM_CM_ID_INVALID ? > cm_id : NULL); > rather than: > uevent->cm_id = ((event->event == IB_CM_REQ_RECEIVED || > event->event == IB_CM_SIDR_REQ_RECEIVED ) ? > cm_id : NULL); Why make the change, since they have the same effect? The only way an invalid 'id' will make it that far in the code is during those two events. -Libor From sean.hefty at intel.com Mon May 23 15:44:33 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Mon, 23 May 2005 15:44:33 -0700 Subject: [openib-general] [PATCH] [MAD] optimize cancel MAD Message-ID: This patch optimizes the cancel MAD path. It eliminates searching the timeout list when setting a MAD's timeout to zero, removes duplicates checks to insert a work item to process the timeout, and avoids resending the MAD as it is canceled. Signed-off-by: Sean Hefty Index: mad.c =================================================================== --- mad.c (revision 2413) +++ mad.c (working copy) @@ -1794,14 +1794,18 @@ static void wait_for_response(struct ib_ delay = mad_send_wr->timeout; mad_send_wr->timeout += jiffies; - list_for_each_prev(list_item, &mad_agent_priv->wait_list) { - temp_mad_send_wr = list_entry(list_item, - struct ib_mad_send_wr_private, - agent_list); - if (time_after(mad_send_wr->timeout, - temp_mad_send_wr->timeout)) - break; + if (delay) { + list_for_each_prev(list_item, &mad_agent_priv->wait_list) { + temp_mad_send_wr = list_entry(list_item, + struct ib_mad_send_wr_private, + agent_list); + if (time_after(mad_send_wr->timeout, + temp_mad_send_wr->timeout)) + break; + } } + else + list_item = &mad_agent_priv->wait_list; list_add(&mad_send_wr->agent_list, list_item); /* Reschedule a work item if we have a shorter timeout */ @@ -1817,7 +1821,6 @@ void ib_reset_mad_timeout(struct ib_mad_ { mad_send_wr->timeout = msecs_to_jiffies(timeout_ms); wait_for_response(mad_send_wr); - adjust_timeout(mad_send_wr->mad_agent_priv); } /* @@ -2266,7 +2269,8 @@ static void timeout_sends(void *data) } list_del(&mad_send_wr->agent_list); - if (!retry_send(mad_send_wr)) + if (mad_send_wr->status == IB_WC_SUCCESS && + !retry_send(mad_send_wr)) continue; spin_unlock_irqrestore(&mad_agent_priv->lock, flags); From kashyapv at us.ibm.com Mon May 23 22:14:11 2005 From: kashyapv at us.ibm.com (Vivek Kashyap) Date: Mon, 23 May 2005 22:14:11 -0700 (PDT) Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <521x83azr0.fsf@topspin.com> Message-ID: On Thu, 19 May 2005, Roland Dreier wrote: > Vivek> The draft does allow for a negotiation per connection for > Vivek> the implementations that wish to take advantage of > Vivek> it. However, an implementation can by default choose to use > Vivek> a 'connected-mode MTU' e.g. 32K always. It can then, for > Vivek> every connection choose to, negotiate to this value and if > Vivek> it is not workable fall back to the UD mode and deny the > Vivek> connection mode. The ARP entries hold the connected mode > Vivek> flags thereby keeping track of the mode to use per > Vivek> destination. > > But this means that the MTU of the link will be different for UD > destinations (including multicast) and RC destinations, right? Or am > I missing something? > > - R. > I should say it depends. One can utilise a setup that sets the MTU to 2044 or whatever is the UD MTU on the subnet for all modes. The connection will advertise the maximum receive MTU as this value. Consider a proof of concept implementation: 1. MUST support IPoIB-UD. Call its MTU UD-MTU. 2. The connection mode is setup to utilise UD-MTU 3. The node therefore has only one MTU to send 4. Since the node advertises UD-MTU as the maximum receive MTU it will not receive a connection packet greater than that value. This is simple enough -- only need to add support for connected mode address resolution and connection setup. Such a solution provides APM and reliable connection if that is chosen but not the large MTU. If the large MTU is desired then force there is a split between UD and connected mode MTU. Vivek From tduffy at sun.com Mon May 23 22:25:49 2005 From: tduffy at sun.com (Tom Duffy) Date: Mon, 23 May 2005 22:25:49 -0700 Subject: [openib-general] [PATCH] kDAPL: cleanup some C++ style comments Message-ID: <1116912349.14182.1.camel@duffman> Signed-off-by: Tom Duffy Index: linux-kernel-new/dat-provider/dapl_openib_cm.c =================================================================== --- linux-kernel-new/dat-provider/dapl_openib_cm.c (revision 2462) +++ linux-kernel-new/dat-provider/dapl_openib_cm.c (working copy) @@ -573,12 +573,14 @@ void dapl_ib_reinit_ep(struct dapl_ep *e ia_ptr = ep_ptr->header.owner_ia; - //*** What if QP is connected? Do we need to destroy the cm_id - //*** here as well, send a DREQ, REJ? What cm_id state are we in? - //*** we may not care if we just destroy the cm_id - //*** ib_destroy_cm_id(ep_ptr->cm_handle); - //*** supporting this call may require setting the cm_handle to - //*** something like NULL wherever it is destroyed + /* + * What if QP is connected? Do we need to destroy the cm_id + * here as well, send a DREQ, REJ? What cm_id state are we in? + * we may not care if we just destroy the cm_id + * ib_destroy_cm_id(ep_ptr->cm_handle); + * supporting this call may require setting the cm_handle to + * something like NULL wherever it is destroyed + */ ib_status = dapl_modify_qp_state_to_reset(ep_ptr->qp_handle); if (ib_status < 0) { @@ -660,8 +662,10 @@ u32 dapl_ib_setup_conn_listener(struct d */ u32 dapl_ib_remove_conn_listener(struct dapl_ia *ia_ptr, struct dapl_sp *sp_ptr) { - //*** This will hang if called from CM thread context... - //*** Move back to using WQ... + /* + * This will hang if called from CM thread context... + * Move back to using WQ... + */ if (sp_ptr->cm_srvc_handle != NULL) { ib_destroy_cm_id(sp_ptr->cm_srvc_handle); sp_ptr->cm_srvc_handle = NULL; From mst at mellanox.co.il Tue May 24 00:06:36 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Tue, 24 May 2005 10:06:36 +0300 Subject: [openib-general] Re: Re: performance counters in /sys In-Reply-To: <429225AD.4080808@hp.com> References: <1116539301.4479.255.camel@hal.voltaire.com> <428DC946.5080207@hp.com> <20050520161146.GB14411@esmail.cup.hp.com> <20050520164843.GC14411@esmail.cup.hp.com> <20050522053351.GB8522@mellanox.co.il> <4291097F.8070404@hp.com> <42920483.3040900@ichips.intel.com> <1116866524.4481.5774.camel@hal.voltaire.com> <429225AD.4080808@hp.com> Message-ID: <20050524070636.GD18721@mellanox.co.il> Quoting r. Mark Seger : > Subject: Re: Re: performance counters in /sys > > Since I was the one who originally created this topic I'd like to > restate what I said that got this all started. I'm trying to do > relatively lightweight monitoring of lots of system performance counters > (on the order of 100-200 or more) across a number of subsystems using > standard interfaces. While I don't feel the need to be able to take > 10Ks of samples/sec I would like to at least run efficiently at 1-10/sec > range. I also want to avoid writing custom kernel code and/or talking > directly to hardware. > > As I said in my base note I'm currently reading from /proc while some > sets of counters are better organized than others, I can still access > them relatively efficiently. While I could certainly "get by" reading > one variable per file, I do worry about the overhead as the sampling > frequency goes down. This will also be a problem as the number of > counters and devices grow. The suggestion about using perfquery would > certainly work, but I'd also be concerned about the overhead in running > it at smaller sampling intervals. > > I certainly understand the desire to move to sysfs and that > /usr/src/linux/Documentation/filesystems/sysfs.txt states that "Mixing > types, expressing multiple lines of data, and doing fancy formatting of > data is heavily frowned upon. Doing these things may get you publically > humiliated and your code rewritten without notice." However, I don't > read this to mean you must only have one data item per file. For > example, I took a look at /sys/block/hda/stat because one of the types > of data I collect is disk stats and I was wondering how sysfs dealt with > them. Sure enough, they're all in one file per disk as shown below: > > dl380-2: cat /sys/block/hda/stat > 0 0 0 0 0 0 0 > 0 0 0 0 > > Also note some of these count bytes, some sectors and other jiffies, so > even the units need not be identical. > > -mark So, what about an actual benchmark for the two approaches that were proposed (perfquery / sysfs)? -- MST - Michael S. Tsirkin From lenber.cong at intel.com Tue May 24 00:13:21 2005 From: lenber.cong at intel.com (Cong, Lenber) Date: Tue, 24 May 2005 15:13:21 +0800 Subject: [openib-general] [HELP] Encounter Kernel Panic when Add Mellanox HCA Supporting on 2.6.9 Kernel Message-ID: Dear All I tried to enable Mellanox HCA card on RHEL4. By now the drivers of Mellanox for 2.6 kernel are not ready yet, then I used the backport patch for 2.6.9 delivered by Woody in 4/16. I patched the kernel and re-compiled it with modulizing Infiniband support, and everything is OK. But when I rebooted the machine with the new kernel, a kernel panic error was encountered during initializing hardware. Here is the error message of rebooting: " Starting udev: Initializing hardware ... Storage, network, audio Kernel Panic - not syncing driver/infiniband/hw/mthca/mthca_allocator.c: 46: spin_is_locked on uninitialized spinlock: f70f7dac " If I disabled the option "Mellanox HCA support" when configuring kernel, system can be rebooted successfully. But if the option is selected, the error above will occur. (The machine I used is a Langley server with two 2.4GHz Xeon CPUs. The OS installed on it is RedHatEL4.) Does anyone happen to encounter such error? Or did I make any mistake during the process? Would you provide some comments/suggestion to me? It will be appreciated very much. Thanks. Lenber Sincerely Lenber -------------- next part -------------- An HTML attachment was scrubbed... URL: From mst at mellanox.co.il Tue May 24 05:49:54 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Tue, 24 May 2005 15:49:54 +0300 Subject: [openib-general] [PATCH] dont create physical mr for user context Message-ID: <20050524124954.GB27428@mellanox.co.il> It seems that mthca creates a physical MR for each PD, which would grant a userspace process access to all physical memory if it can guess the key. Fix this by adding a "privileged" flag, set only for kernel-level consumers. Signed-off-by: Michael S. Tsirkin Index: mthca_main.c =================================================================== --- mthca_main.c (revision 2437) +++ mthca_main.c (working copy) @@ -664,6 +664,8 @@ static int __devinit mthca_setup_hca(str goto err_pd_table_free; } + dev->driver_pd.privileged = 1; + err = mthca_pd_alloc(dev, &dev->driver_pd); if (err) { mthca_err(dev, "Failed to create driver PD, " Index: mthca_provider.c =================================================================== --- mthca_provider.c (revision 2437) +++ mthca_provider.c (working copy) @@ -379,6 +379,8 @@ static struct ib_pd *mthca_alloc_pd(stru if (!pd) return ERR_PTR(-ENOMEM); + pd->privileged = !context; + err = mthca_pd_alloc(to_mdev(ibdev), pd); if (err) { kfree(pd); Index: mthca_provider.h =================================================================== --- mthca_provider.h (revision 2437) +++ mthca_provider.h (working copy) @@ -91,6 +91,7 @@ struct mthca_pd { u32 pd_num; atomic_t sqp_count; struct mthca_mr ntmr; + int privileged; }; struct mthca_eq { Index: mthca_pd.c =================================================================== --- mthca_pd.c (revision 2437) +++ mthca_pd.c (working copy) @@ -48,6 +48,9 @@ int mthca_pd_alloc(struct mthca_dev *dev if (pd->pd_num == -1) return -ENOMEM; + if (!pd->privileged) + return 0; + err = mthca_mr_alloc_notrans(dev, pd->pd_num, MTHCA_MPT_FLAG_LOCAL_READ | MTHCA_MPT_FLAG_LOCAL_WRITE, @@ -61,7 +64,8 @@ int mthca_pd_alloc(struct mthca_dev *dev void mthca_pd_free(struct mthca_dev *dev, struct mthca_pd *pd) { might_sleep(); - mthca_free_mr(dev, &pd->ntmr); + if (pd->privileged) + mthca_free_mr(dev, &pd->ntmr); mthca_free(&dev->pd_table.alloc, pd->pd_num); } -- MST - Michael S. Tsirkin From roland at topspin.com Tue May 24 06:44:16 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 24 May 2005 06:44:16 -0700 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: (Vivek Kashyap's message of "Mon, 23 May 2005 22:14:11 -0700 (PDT)") References: Message-ID: <524qcsyc0f.fsf@topspin.com> Vivek> I should say it depends. One can utilise a setup that sets Vivek> the MTU to 2044 or whatever is the UD MTU on the subnet for Vivek> all modes. The connection will advertise the maximum Vivek> receive MTU as this value. OK, but that's throwing away the main advantage of connected mode IPoIB, right? For full connected mode support, do you see an alternative to handling per-destination MTUs? - R. From MAILER-DAEMON at aol.com Tue May 24 07:27:12 2005 From: MAILER-DAEMON at aol.com (Mail Delivery Subsystem) Date: Tue, 24 May 2005 10:27:12 -0400 (EDT) Subject: [openib-general] Returned mail: User unknown Message-ID: <200505241427.KAK18350@rly-nc06.mx.aol.com> The original message was received at Tue, 24 May 2005 10:27:01 -0400 (EDT) from cpe-143-238-211-158.wa.bigpond.net.au [143.238.211.158] *** ATTENTION *** Your e-mail is being returned to you because there was a problem with its delivery. The address which was undeliverable is listed in the section labeled: "----- The following addresses had permanent fatal errors -----". The reason your mail is being returned to you is listed in the section labeled: "----- Transcript of Session Follows -----". The line beginning with "<<<" describes the specific reason your e-mail could not be delivered. The next line contains a second error message which is a general translation for other e-mail servers. Please direct further questions regarding this message to your e-mail administrator. --AOL Postmaster ----- The following addresses had permanent fatal errors ----- ----- Transcript of session follows ----- ... while talking to air-nc02.mail.aol.com.: >>> RCPT To: <<< 550 MAILBOX NOT FOUND 550 ... User unknown -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: text/rfc822-headers Size: 672 bytes Desc: not available URL: From robert.j.woodruff at intel.com Tue May 24 08:26:03 2005 From: robert.j.woodruff at intel.com (Bob Woodruff) Date: Tue, 24 May 2005 08:26:03 -0700 Subject: [openib-general] [HELP] Encounter Kernel Panic when Add MellanoxHCA Supporting on 2.6.9 Kernel In-Reply-To: Message-ID: Lenber wrote, > Here is the error message of rebooting: " > Starting udev: > Initializing hardware ... Storage, network, audio Kernel Panic - not > syncing driver/infiniband/hw/mthca/mthca_allocator.c: 46: spin_is_locked on > uninitialized spinlock: f70f7dac These patches were based on the code that is in 2.6.12-rc. I have to admit I did not try them on IA32, I only tested on Itanium and EM64T. I can also try to set up some IA32 machines today or I can send you newer patches based on SVN2425 that I have tested on EM64T on Redhat EL4.0. Roland, has anything been fixed since the 2.6.12 drop in mthca that could account for this panic ? woody From mshefty at ichips.intel.com Tue May 24 08:54:01 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Tue, 24 May 2005 08:54:01 -0700 Subject: [openib-general] Re: Re: performance counters in /sys In-Reply-To: <20050524070636.GD18721@mellanox.co.il> References: <1116539301.4479.255.camel@hal.voltaire.com> <428DC946.5080207@hp.com> <20050520161146.GB14411@esmail.cup.hp.com> <20050520164843.GC14411@esmail.cup.hp.com> <20050522053351.GB8522@mellanox.co.il> <4291097F.8070404@hp.com> <42920483.3040900@ichips.intel.com> <1116866524.4481.5774.camel@hal.voltaire.com> <429225AD.4080808@hp.com> <20050524070636.GD18721@mellanox.co.il> Message-ID: <42934E19.4010702@ichips.intel.com> Michael S. Tsirkin wrote: > So, what about an actual benchmark for the two approaches that were proposed > (perfquery / sysfs)? A benchmark would be nice to see if sysfs could be used, but I'm not sure that it's needed in order to compare the performance between the two. How would sysfs counters get populated if not via a PM MAD? - Sean From ardavis at ichips.intel.com Tue May 24 09:38:15 2005 From: ardavis at ichips.intel.com (Arlin Davis) Date: Tue, 24 May 2005 09:38:15 -0700 Subject: [openib-general] ibv_reg_mr limits? In-Reply-To: <52r7g5kcji.fsf@topspin.com> References: <428935D0.1010508@ichips.intel.com> <5264xhlwln.fsf@topspin.com> <20050517162638.GA7553@aon.at> <52r7g5kcji.fsf@topspin.com> Message-ID: <42935877.3030608@ichips.intel.com> Roland Dreier wrote: > Bernhard> Any timeframe on when this will be changed? > >I hope to fix this up soonish but there are a lot of small things like >this to clean up. > > - R. > > > is soonish up yet? :-) From mshefty at ichips.intel.com Tue May 24 09:46:40 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Tue, 24 May 2005 09:46:40 -0700 Subject: [openib-general] Re: Latest CM and retransmissions In-Reply-To: <1116952775.4490.53.camel@hal.voltaire.com> References: <1116952775.4490.53.camel@hal.voltaire.com> Message-ID: <42935A70.4030003@ichips.intel.com> Hal Rosenstock wrote: > Hi Sean, > > With the latest CM changes (not sure when I last picked these up), I am > now seeing the following sequences. The times between the CM MADs are on > the order of 10s of usec. These sequences all refer to the same > connection. It seems that the retransmissions are occuring despite the > expected MAD being received. This was a bug in the MAD code, where a request was resent when it was canceled. The bug entered when ib_modify_mad() was added, and a fix went in yesterday. It shouldn't affect the operation any, just results in a lot of unnecessary MADs... - Sean From kashyapv at us.ibm.com Tue May 24 09:57:12 2005 From: kashyapv at us.ibm.com (Vivek Kashyap) Date: Tue, 24 May 2005 09:57:12 -0700 (PDT) Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <524qcsyc0f.fsf@topspin.com> Message-ID: On Tue, 24 May 2005, Roland Dreier wrote: > Vivek> I should say it depends. One can utilise a setup that sets > Vivek> the MTU to 2044 or whatever is the UD MTU on the subnet for > Vivek> all modes. The connection will advertise the maximum > Vivek> receive MTU as this value. > > OK, but that's throwing away the main advantage of connected mode > IPoIB, right? I was addressing the case of a minimal implementation. For sucn an an initial implementation one can go as above. However, as I noted in the previous mail, if a large MTU is to be supported then one has to split the CM and UD MTUs since the UD MTU is limited to 2 or 4K. It is the limitation of the media that we get large MTU with connected mode whereas we get multicast (absolutely necessary for for address resolution) only with UD unless we want to do multicast emulation. > > For full connected mode support, do you see an alternative to handling > per-destination MTUs? ..how about the following: To send a node must be aware of the remote node's maximum MTU. The the node can determine the minimum of these maximums. Therefore, one can go with a two way split -- for UD use the UD MTU. For any RC communication use this minimum (or less depending on the packet size). Therfore, one can decrement the RC MTU if needed at every new RC connection that is made - i.e. one does recognise the receive MTU from the peer but uses it to decrement the overall RC MTU if required. It will not be optimal for all cases since some could do with larger MTU but will interoperate. One can also administratively set a large enough 'minimum RC MTU' on all the nodes in the subnet. Vivek > > - R. > > From ftillier at silverstorm.com Tue May 24 10:11:56 2005 From: ftillier at silverstorm.com (Fab Tillier) Date: Tue, 24 May 2005 10:11:56 -0700 Subject: [openib-general] How about ib_send_page() ? In-Reply-To: Message-ID: <001401c56083$b27dfa70$9b5aa8c0@infiniconsys.com> > From: Vivek Kashyap [mailto:kashyapv at us.ibm.com] > Sent: Tuesday, May 24, 2005 9:57 AM > > On Tue, 24 May 2005, Roland Dreier wrote: > > > Vivek> I should say it depends. One can utilise a setup that sets > > Vivek> the MTU to 2044 or whatever is the UD MTU on the subnet for > > Vivek> all modes. The connection will advertise the maximum > > Vivek> receive MTU as this value. > > > > OK, but that's throwing away the main advantage of connected mode > > IPoIB, right? > > I was addressing the case of a minimal implementation. For sucn an an > initial implementation one can go as above. However, as I noted > in the previous mail, if a large MTU is to be supported then one has to > split the CM and UD MTUs since the UD MTU is limited to 2 or 4K. > It is the limitation of the media that we get large MTU with connected > mode > whereas we get multicast (absolutely necessary for for address resolution) > only with UD unless we want to do multicast emulation. Ok, so this question is from a noob, but here goes anyway. Why can't IPoIB advertise a larger MTU than the UD MTU, and then just fragment large IP packets up if they need to go over the IB UD transport? Is there any reason this couldn't work? If it does, it allows IPoIB to expose a single MTU to the OS, and take care of the rest under the covers. Just a thought. - Fab From halr at voltaire.com Tue May 24 10:10:11 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 24 May 2005 13:10:11 -0400 Subject: [openib-general] Re: Latest CM and retransmissions In-Reply-To: <42935A70.4030003@ichips.intel.com> References: <1116952775.4490.53.camel@hal.voltaire.com> <42935A70.4030003@ichips.intel.com> Message-ID: <1116954610.4490.73.camel@hal.voltaire.com> On Tue, 2005-05-24 at 12:46, Sean Hefty wrote: > Hal Rosenstock wrote: > > Hi Sean, > > > > With the latest CM changes (not sure when I last picked these up), I am > > now seeing the following sequences. The times between the CM MADs are on > > the order of 10s of usec. These sequences all refer to the same > > connection. It seems that the retransmissions are occuring despite the > > expected MAD being received. > > This was a bug in the MAD code, where a request was resent when it was > canceled. The bug entered when ib_modify_mad() was added, and a fix went in > yesterday. It shouldn't affect the operation any, just results in a lot of > unnecessary MADs... One of my machines used did not have this fix. That fixed the connection setup side but not teardown. I still see: DREQ --------> <------- DREP DREQ --------> DREQ --------> -- Hal From halr at voltaire.com Tue May 24 09:39:35 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 24 May 2005 12:39:35 -0400 Subject: [openib-general] Latest CM and retransmissions Message-ID: <1116952775.4490.53.camel@hal.voltaire.com> Hi Sean, With the latest CM changes (not sure when I last picked these up), I am now seeing the following sequences. The times between the CM MADs are on the order of 10s of usec. These sequences all refer to the same connection. It seems that the retransmissions are occuring despite the expected MAD being received. 1. Connection setup 1 REQ --------> <------- REP RTU --------> <------- REP RTU --------> 2. Connection setup 2 REQ --------> <------- REP REQ --------> RTU --------> <------- REJ 3. Connection setup 3 REQ --------> <------- REP RTU --------> <------- REJ RTU --------> 4. Connection teardown DREQ --------> <------- DREP DREQ --------> DREQ --------> -- Hal From kashyapv at us.ibm.com Tue May 24 10:43:28 2005 From: kashyapv at us.ibm.com (Vivek Kashyap) Date: Tue, 24 May 2005 10:43:28 -0700 (PDT) Subject: [openib-general] How about ib_send_page() ? In-Reply-To: <1116521932.4479.27.camel@hal.voltaire.com> Message-ID: On 19 May 2005, Hal Rosenstock wrote: > Hi Vivek, > > On Thu, 2005-05-19 at 12:41, Vivek Kashyap wrote: > > > > > > > > > > The most interesting optimization available is implementing the IPoIB > > > connected mode draft, although I don't think it's as easy as Vivek > > > indicated -- for example, I'm not sure how to deal with having > > > different MTUs depending on the destination. > > > > > > > > The draft does allow for a negotiation per connection for the implementations > > that wish to take advantage of it. However, an implementation can by default > > choose to use a 'connected-mode MTU' e.g. 32K always. It can then, for every > > connection choose to, negotiate to this value and if it is not workable fall > > back to the UD mode and deny the connection mode. The ARP entries hold the > > connected mode flags thereby keeping track of the mode to use per destination. > > Sounds like there should be an "agreement" on a default connected mode > MTU or else this will drop down to UD. Do you mean an administrative CM MTU set across the subnet? It is possible, though one must be aware of any conflicts depending on the implementations. See an alternative suggestion I just noted in response to Roland's question. > > I have a couple of clarification questions on 5.1 Per-Connection MTU: > > 1. I presume the Receive MTU is in the first 2 bytes of the private data > in the CM messages. Is that correct ? Yes..though I'll relook and make the placement clearer in the next version of the draft. > > 2. Also, CM REQ is mentioned for the requester receive MTU. Wouldn't CM > REP carry the granted receive MTU which is constrained to be the > requested MTU or less ? So 2 things on this: No, we don't need a two way handshake. Each side just advertises the maximum MTU it can handle. It is like TCP MSS. The > The I-D says "The private data field MUST carry" the receive MTU. Does > that include RTUs ? No, I'd think that we can limit it to REQ. It has been a while since I went over this -- I'll go over the spec again and post an update if needed. Vivek > > Thanks. > > -- Hal > > > From roland at topspin.com Tue May 24 11:08:03 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 24 May 2005 11:08:03 -0700 Subject: [openib-general] ibv_reg_mr limits? In-Reply-To: <42935877.3030608@ichips.intel.com> (Arlin Davis's message of "Tue, 24 May 2005 09:38:15 -0700") References: <428935D0.1010508@ichips.intel.com> <5264xhlwln.fsf@topspin.com> <20050517162638.GA7553@aon.at> <52r7g5kcji.fsf@topspin.com> <42935877.3030608@ichips.intel.com> Message-ID: <52ll64wl8c.fsf@topspin.com> Arlin> is soonish up yet? :-) no, not really. sorry, I've been really busy lately, and it's been impossible to take a day so I can sit down and fix this. - R. From roland at topspin.com Tue May 24 11:11:06 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 24 May 2005 11:11:06 -0700 Subject: [openib-general] [HELP] Encounter Kernel Panic when Add MellanoxHCA Supporting on 2.6.9 Kernel In-Reply-To: (Bob Woodruff's message of "Tue, 24 May 2005 08:26:03 -0700") References: Message-ID: <52ekbwwl39.fsf@topspin.com> I just tried the latest svn on 2.6.11 with CONFIG_DEBUG_SPINLOCK turned on, and I didn't see any problems. The message driver/infiniband/hw/mthca/mthca_allocator.c: 46: spin_is_locked on uninitialized spinlock: f70f7dac is coming from CHECK_LOCK, which is turned on with CONFIG_DEBUG_SPINLOCK. However there should be more traceback information printed to the console as well... did that get dumped as well? Bob> Roland, has anything been fixed since the 2.6.12 drop in Bob> mthca that could account for this panic ? Not that I know of... - R. From roland at topspin.com Tue May 24 11:26:51 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 24 May 2005 11:26:51 -0700 Subject: [openib-general] Re: [PATCH] dont create physical mr for user context In-Reply-To: <20050524124954.GB27428@mellanox.co.il> (Michael S. Tsirkin's message of "Tue, 24 May 2005 15:49:54 +0300") References: <20050524124954.GB27428@mellanox.co.il> Message-ID: <527jhowkd0.fsf@topspin.com> Thanks, excellent catch. I changed your patch slightly as below to make "privileged" a parameter to mthca_pd_alloc() so that it's harder for someone to leave it uninitialized by accident. - R. --- infiniband/hw/mthca/mthca_dev.h (revision 2468) +++ infiniband/hw/mthca/mthca_dev.h (working copy) @@ -384,7 +384,7 @@ void mthca_unregister_device(struct mthc int mthca_uar_alloc(struct mthca_dev *dev, struct mthca_uar *uar); void mthca_uar_free(struct mthca_dev *dev, struct mthca_uar *uar); -int mthca_pd_alloc(struct mthca_dev *dev, struct mthca_pd *pd); +int mthca_pd_alloc(struct mthca_dev *dev, int privileged, struct mthca_pd *pd); void mthca_pd_free(struct mthca_dev *dev, struct mthca_pd *pd); int mthca_mr_alloc_notrans(struct mthca_dev *dev, u32 pd, --- infiniband/hw/mthca/mthca_main.c (revision 2468) +++ infiniband/hw/mthca/mthca_main.c (working copy) @@ -664,7 +664,7 @@ static int __devinit mthca_setup_hca(str goto err_pd_table_free; } - err = mthca_pd_alloc(dev, &dev->driver_pd); + err = mthca_pd_alloc(dev, 1, &dev->driver_pd); if (err) { mthca_err(dev, "Failed to create driver PD, " "aborting.\n"); --- infiniband/hw/mthca/mthca_provider.c (revision 2468) +++ infiniband/hw/mthca/mthca_provider.c (working copy) @@ -379,7 +379,7 @@ static struct ib_pd *mthca_alloc_pd(stru if (!pd) return ERR_PTR(-ENOMEM); - err = mthca_pd_alloc(to_mdev(ibdev), pd); + err = mthca_pd_alloc(to_mdev(ibdev), !context, pd); if (err) { kfree(pd); return ERR_PTR(err); --- infiniband/hw/mthca/mthca_provider.h (revision 2468) +++ infiniband/hw/mthca/mthca_provider.h (working copy) @@ -91,6 +91,7 @@ struct mthca_pd { u32 pd_num; atomic_t sqp_count; struct mthca_mr ntmr; + int privileged; }; struct mthca_eq { --- infiniband/hw/mthca/mthca_pd.c (revision 2468) +++ infiniband/hw/mthca/mthca_pd.c (working copy) @@ -37,23 +37,27 @@ #include "mthca_dev.h" -int mthca_pd_alloc(struct mthca_dev *dev, struct mthca_pd *pd) +int mthca_pd_alloc(struct mthca_dev *dev, int privileged, struct mthca_pd *pd) { - int err; + int err = 0; might_sleep(); + pd->privileged = privileged; + atomic_set(&pd->sqp_count, 0); pd->pd_num = mthca_alloc(&dev->pd_table.alloc); if (pd->pd_num == -1) return -ENOMEM; - err = mthca_mr_alloc_notrans(dev, pd->pd_num, - MTHCA_MPT_FLAG_LOCAL_READ | - MTHCA_MPT_FLAG_LOCAL_WRITE, - &pd->ntmr); - if (err) - mthca_free(&dev->pd_table.alloc, pd->pd_num); + if (privileged) { + err = mthca_mr_alloc_notrans(dev, pd->pd_num, + MTHCA_MPT_FLAG_LOCAL_READ | + MTHCA_MPT_FLAG_LOCAL_WRITE, + &pd->ntmr); + if (err) + mthca_free(&dev->pd_table.alloc, pd->pd_num); + } return err; } @@ -61,7 +65,8 @@ int mthca_pd_alloc(struct mthca_dev *dev void mthca_pd_free(struct mthca_dev *dev, struct mthca_pd *pd) { might_sleep(); - mthca_free_mr(dev, &pd->ntmr); + if (pd->privileged) + mthca_free_mr(dev, &pd->ntmr); mthca_free(&dev->pd_table.alloc, pd->pd_num); } From halr at voltaire.com Tue May 24 11:32:16 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 24 May 2005 14:32:16 -0400 Subject: [openib-general] [PATCH] [kdapl CM] Fix endian conversions of service ID Message-ID: <1116959536.4490.166.camel@hal.voltaire.com> [kdapl CM] Fix endian conversions of service ID Problem pointed out by James Lentini Signed-off-by: Hal Rosenstock Index: dapl_openib_cm.c =================================================================== -- dapl_openib_cm.c (revision 2468) +++ dapl_openib_cm.c (working copy) @@ -309,7 +309,7 @@ if (conn->dapl_path.mtu > IB_MTU_1024) conn->dapl_path.mtu = IB_MTU_1024; - conn->param.service_id = be64_to_cpu(conn->service_id); + conn->param.service_id = conn->service_id; conn->param.primary_path = &conn->dapl_path; conn->param.alternate_path = NULL; @@ -445,8 +445,7 @@ conn->param.local_cm_response_timeout = DAPL_OPENIB_CM_RESPONSE_TIMEOUT; conn->param.max_cm_retries = DAPL_OPENIB_MAX_CM_RETRIES; - memcpy(&conn->service_id, &remote_conn_qual, sizeof conn->service_id); - + conn->service_id = be64_to_cpu(remote_conn_qual); conn->remote_ia_address = remote_ia_address; conn->dapl_comp.fn = &dapl_rt_comp_handler; conn->dapl_comp.context = conn; @@ -627,7 +626,7 @@ } status = ib_cm_listen(sp_ptr->cm_srvc_handle, - be64_to_cpu(sp_ptr->conn_qual), 0); + cpu_to_be64(sp_ptr->conn_qual), 0); if (status) { ib_destroy_cm_id(sp_ptr->cm_srvc_handle); sp_ptr->cm_srvc_handle = NULL; From jlentini at netapp.com Tue May 24 11:40:47 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 24 May 2005 14:40:47 -0400 (EDT) Subject: [openib-general] Re: [PATCH][RFC] kDAPL: use linux native timers instead of own In-Reply-To: <1116889028.19768.6.camel@duffman> References: <1116889028.19768.6.camel@duffman> Message-ID: Tom, This looks very good. A few comments: - Do we need to convert the timeout from milliseconds to jiffies? DAT timeout values are in milliseconds and kernel timers are in jiffies. I think we need to convert from milliseconds to jiffies before setting up a timer. - Is it safe to use del_timer()? Currently, kDAPL assumes that any function may be called from an interrupt (hence the use of spin_lock_irqsave(), etc). This precludes us from using del_timer_sync(), but I think there are scenarios in which we could get into trouble by using del_timer(). For example, suppose we are on an SMP machine and the timeout occurs. While dapl_ep_timeout() is running on one processor, the kDAPL consumer calls dat_ep_free() on another processor. Now one processor would free the DAPL_EP structure's memory while it was being used by dapl_ep_timeout() and its callees. Could this be fixed by simply moving line 260 of dapl_ep_util.c to the end of dapl_ep_timeout and updating dapl_ep_free to return if the ep state is DAT_EP_STATE_ACTIVE_CONNECTION_PENDING? - Could we use cm timeouts? The timers are only used to implement 2 DAT API calls: dat_ep_connect and dat_ep_dup_connect. Both use a timer to implement the same feature (cancel connection attempts after some period of time) and both eventually call ib_send_cm_req(). ib_send_cm_req() allows the consumer to specify timeouts. What happens when these timeouts expire? Could DAPL use these instead of kernel timers? I don't see an ib_cm_event_type that corresponds to a timeout. This implies that the CM user's ib_cm_handler is not invoked when one of these timeout values expires. If so kDAPL wouldn't be able to make use of them. Does anyone know how these work? james On Mon, 23 May 2005, Tom Duffy wrote: tduffy> This patch removes the kDAPL timer implementation and uses the Linux tduffy> native timer class. Please look over and comment. tduffy> tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> Index: linux-kernel-timer/test/dapltest/test/dapl_performance_util.c tduffy> =================================================================== tduffy> --- linux-kernel-timer/test/dapltest/test/dapl_performance_util.c (revision 2457) tduffy> +++ linux-kernel-timer/test/dapltest/test/dapl_performance_util.c (working copy) tduffy> @@ -538,7 +538,7 @@ DT_performance_wait ( tduffy> pre_ts = DT_Mdep_GetTimeStamp (); tduffy> tduffy> ret = DT_Tdep_evd_wait ( evd_handle, tduffy> - DAT_TIMEOUT_INFINITE, tduffy> + MAX_SCHEDULE_TIMEOUT, tduffy> &event); tduffy> tduffy> post_ts = DT_Mdep_GetTimeStamp (); tduffy> Index: linux-kernel-timer/test/dapltest/test/dapl_performance_client.c tduffy> =================================================================== tduffy> --- linux-kernel-timer/test/dapltest/test/dapl_performance_client.c (revision 2457) tduffy> +++ linux-kernel-timer/test/dapltest/test/dapl_performance_client.c (working copy) tduffy> @@ -105,7 +105,7 @@ retry: tduffy> ret = dat_ep_connect (test_ptr->ep_context.ep_handle, tduffy> test_ptr->remote_ia_addr, tduffy> test_ptr->ep_context.port, tduffy> - DAT_TIMEOUT_INFINITE, tduffy> + MAX_SCHEDULE_TIMEOUT, tduffy> 0, tduffy> (void *) 0, /* no private data */ tduffy> test_ptr->cmd->qos, tduffy> Index: linux-kernel-timer/test/dapltest/test/dapl_server.c tduffy> =================================================================== tduffy> --- linux-kernel-timer/test/dapltest/test/dapl_server.c (revision 2457) tduffy> +++ linux-kernel-timer/test/dapltest/test/dapl_server.c (working copy) tduffy> @@ -584,7 +584,7 @@ DT_cs_Server (Params_t * params_ptr) tduffy> tduffy> DT_Mdep_wait_object_wait ( tduffy> &pt_ptr->synch_wait_object, tduffy> - DAT_TIMEOUT_INFINITE); tduffy> + MAX_SCHEDULE_TIMEOUT); tduffy> tduffy> /* Send the Server_Info */ tduffy> DT_Tdep_PT_Debug (1,(phead,"%s: Send Server_Info\n", module)); tduffy> Index: linux-kernel-timer/test/dapltest/test/dapl_transaction_test.c tduffy> =================================================================== tduffy> --- linux-kernel-timer/test/dapltest/test/dapl_transaction_test.c (revision 2457) tduffy> +++ linux-kernel-timer/test/dapltest/test/dapl_transaction_test.c (working copy) tduffy> @@ -714,7 +714,7 @@ retry: tduffy> ret = dat_ep_connect (test_ptr->ep_context[i].ep_handle, tduffy> test_ptr->remote_ia_addr, tduffy> test_ptr->ep_context[i].ia_port, tduffy> - DAT_TIMEOUT_INFINITE, tduffy> + MAX_SCHEDULE_TIMEOUT, tduffy> strlen (private_data_str), private_data_str, tduffy> pt_ptr->Params.ReliabilityLevel, tduffy> DAT_CONNECT_DEFAULT_FLAG); tduffy> Index: linux-kernel-timer/test/dapltest/test/dapl_test_util.c tduffy> =================================================================== tduffy> --- linux-kernel-timer/test/dapltest/test/dapl_test_util.c (revision 2457) tduffy> +++ linux-kernel-timer/test/dapltest/test/dapl_test_util.c (working copy) tduffy> @@ -261,7 +261,7 @@ DT_cr_event_wait ( DT_Tdep_Print_Head *p tduffy> u32 ret; tduffy> struct dat_event event; tduffy> tduffy> - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); tduffy> + ret = DT_Tdep_evd_wait (evd_handle, MAX_SCHEDULE_TIMEOUT, &event); tduffy> if (ret != DAT_SUCCESS) tduffy> { tduffy> DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (CR) failed: %s\n", tduffy> @@ -316,7 +316,7 @@ DT_conn_event_wait (DT_Tdep_Print_Head * tduffy> u32 ret; tduffy> struct dat_event event; tduffy> tduffy> - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); tduffy> + ret = DT_Tdep_evd_wait (evd_handle, MAX_SCHEDULE_TIMEOUT, &event); tduffy> if (ret != DAT_SUCCESS) tduffy> { tduffy> DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (CONN) failed: %s\n", tduffy> @@ -369,7 +369,7 @@ DT_disco_event_wait ( DT_Tdep_Print_Head tduffy> u32 ret; tduffy> struct dat_event event; tduffy> tduffy> - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); tduffy> + ret = DT_Tdep_evd_wait (evd_handle, MAX_SCHEDULE_TIMEOUT, &event); tduffy> if (ret != DAT_SUCCESS) tduffy> { tduffy> DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (DISCONN) failed: %s\n", tduffy> @@ -492,7 +492,7 @@ DT_dto_event_wait (DT_Tdep_Print_Head *p tduffy> u32 ret; tduffy> struct dat_event event; tduffy> tduffy> - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); tduffy> + ret = DT_Tdep_evd_wait (evd_handle, MAX_SCHEDULE_TIMEOUT, &event); tduffy> if (ret != DAT_SUCCESS) tduffy> { tduffy> DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (DTO) failed: %s\n", tduffy> @@ -536,7 +536,7 @@ DT_rmr_event_wait (DT_Tdep_Print_Head *p tduffy> u32 ret; tduffy> struct dat_event event; tduffy> tduffy> - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); tduffy> + ret = DT_Tdep_evd_wait (evd_handle, MAX_SCHEDULE_TIMEOUT, &event); tduffy> if (ret != DAT_SUCCESS) tduffy> { tduffy> DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (RMR) failed: %s\n", tduffy> Index: linux-kernel-timer/test/dapltest/test/dapl_client.c tduffy> =================================================================== tduffy> --- linux-kernel-timer/test/dapltest/test/dapl_client.c (revision 2457) tduffy> +++ linux-kernel-timer/test/dapltest/test/dapl_client.c (working copy) tduffy> @@ -256,7 +256,7 @@ retry: tduffy> ret = dat_ep_connect (ep_handle, tduffy> server_netaddr, tduffy> SERVER_PORT_NUMBER, tduffy> - DAT_TIMEOUT_INFINITE, tduffy> + MAX_SCHEDULE_TIMEOUT, tduffy> 0, (void *) 0, /* no private data */ tduffy> params_ptr->ReliabilityLevel, tduffy> DAT_CONNECT_DEFAULT_FLAG); tduffy> Index: linux-kernel-timer/test/dapltest/include/dapl_tdep.h tduffy> =================================================================== tduffy> --- linux-kernel-timer/test/dapltest/include/dapl_tdep.h (revision 2457) tduffy> +++ linux-kernel-timer/test/dapltest/include/dapl_tdep.h (working copy) tduffy> @@ -56,7 +56,7 @@ DT_Tdep_evd_free (DAT_EVD_HANDLE evd_ha tduffy> tduffy> u32 tduffy> DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_handle, tduffy> - DAT_TIMEOUT timeout, tduffy> + unsigned long timeout, tduffy> struct dat_event *event); tduffy> u32 tduffy> DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_handle, tduffy> Index: linux-kernel-timer/test/dapltest/include/dapl_proto.h tduffy> =================================================================== tduffy> --- linux-kernel-timer/test/dapltest/include/dapl_proto.h (revision 2457) tduffy> +++ linux-kernel-timer/test/dapltest/include/dapl_proto.h (working copy) tduffy> @@ -189,7 +189,7 @@ void DT_Mdep_Thread_EXIT ( void * thr tduffy> int DT_Mdep_wait_object_init ( DT_WAIT_OBJECT *wait_obj); tduffy> int DT_Mdep_wait_object_wait ( tduffy> DT_WAIT_OBJECT *wait_obj, tduffy> - int timeout_val); tduffy> + unsigned long timeout_val); tduffy> int DT_Mdep_wait_object_wakeup ( DT_WAIT_OBJECT *wait_obj); tduffy> int DT_Mdep_wait_object_destroy ( DT_WAIT_OBJECT *wait_obj); tduffy> tduffy> Index: linux-kernel-timer/test/dapltest/include/dapl_transaction_test.h tduffy> =================================================================== tduffy> --- linux-kernel-timer/test/dapltest/include/dapl_transaction_test.h (revision 2457) tduffy> +++ linux-kernel-timer/test/dapltest/include/dapl_transaction_test.h (working copy) tduffy> @@ -79,7 +79,7 @@ typedef struct tduffy> Transaction_Cmd_t *cmd; tduffy> struct sockaddr * remote_ia_addr; tduffy> DAT_CONN_QUAL base_port; tduffy> - DAT_TIMEOUT time_out; tduffy> + unsigned long time_out; tduffy> int evd_length; tduffy> Thread *thread; tduffy> tduffy> Index: linux-kernel-timer/test/dapltest/mdep/linux/dapl_mdep_user.c tduffy> =================================================================== tduffy> --- linux-kernel-timer/test/dapltest/mdep/linux/dapl_mdep_user.c (revision 2457) tduffy> +++ linux-kernel-timer/test/dapltest/mdep/linux/dapl_mdep_user.c (working copy) tduffy> @@ -459,7 +459,7 @@ DT_Mdep_wait_object_init ( tduffy> int tduffy> DT_Mdep_wait_object_wait ( tduffy> DT_WAIT_OBJECT *wait_obj, tduffy> - int timeout_val) tduffy> + unsigned long timeout_val) tduffy> { tduffy> int dat_status; tduffy> int pthread_status; tduffy> Index: linux-kernel-timer/test/dapltest/mdep/linux/dapl_mdep_kernel.c tduffy> =================================================================== tduffy> --- linux-kernel-timer/test/dapltest/mdep/linux/dapl_mdep_kernel.c (revision 2457) tduffy> +++ linux-kernel-timer/test/dapltest/mdep/linux/dapl_mdep_kernel.c (working copy) tduffy> @@ -313,13 +313,13 @@ DT_Mdep_wait_object_init ( tduffy> int tduffy> DT_Mdep_wait_object_wait ( tduffy> DT_WAIT_OBJECT *wait_obj, tduffy> - int timeout_val) tduffy> + unsigned long timeout_val) tduffy> { tduffy> int expire; tduffy> int dat_status; tduffy> tduffy> dat_status = DAT_SUCCESS; tduffy> - if ( DAT_TIMEOUT_INFINITE == timeout_val) tduffy> + if ( MAX_SCHEDULE_TIMEOUT == timeout_val) tduffy> { tduffy> interruptible_sleep_on (wait_obj); tduffy> } tduffy> Index: linux-kernel-timer/test/dapltest/kdapl/kdapl_tdep_evd.c tduffy> =================================================================== tduffy> --- linux-kernel-timer/test/dapltest/kdapl/kdapl_tdep_evd.c (revision 2457) tduffy> +++ linux-kernel-timer/test/dapltest/kdapl/kdapl_tdep_evd.c (working copy) tduffy> @@ -214,7 +214,7 @@ DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_ tduffy> tduffy> u32 tduffy> DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_handle, tduffy> - DAT_TIMEOUT timeout, tduffy> + unsigned long timeout, tduffy> struct dat_event *dat_event) tduffy> { tduffy> u32 dat_status; tduffy> @@ -240,7 +240,7 @@ DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_han tduffy> return dat_status; tduffy> } tduffy> /* wait for an event */ tduffy> - if (timeout == DAT_TIMEOUT_INFINITE) tduffy> + if (timeout == MAX_SCHEDULE_TIMEOUT) tduffy> { tduffy> rc = wait_event_interruptible (evd_ptr->wait_object, tduffy> evd_ptr->event_next != NULL); tduffy> Index: linux-kernel-timer/dat-provider/dapl_ep_connect.c tduffy> =================================================================== tduffy> --- linux-kernel-timer/dat-provider/dapl_ep_connect.c (revision 2457) tduffy> +++ linux-kernel-timer/dat-provider/dapl_ep_connect.c (working copy) tduffy> @@ -35,7 +35,6 @@ tduffy> #include "dapl_ep_util.h" tduffy> #include "dapl_openib_util.h" tduffy> #include "dapl_evd_util.h" tduffy> -#include "dapl_timer_util.h" tduffy> tduffy> /* tduffy> * Request a connection be established between the local Endpoint tduffy> @@ -44,7 +43,7 @@ tduffy> */ tduffy> u32 dapl_ep_connect(DAT_EP_HANDLE ep_handle, tduffy> struct sockaddr *remote_ia_address, tduffy> - DAT_CONN_QUAL remote_conn_qual, DAT_TIMEOUT timeout, tduffy> + DAT_CONN_QUAL remote_conn_qual, unsigned long timeout, tduffy> int private_data_size, const void *private_data, tduffy> enum dat_qos qos, enum dat_connect_flags connect_flags) tduffy> { tduffy> @@ -278,23 +277,17 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han tduffy> spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); tduffy> if (ep_ptr->param.ep_state == tduffy> DAT_EP_STATE_ACTIVE_CONNECTION_PENDING tduffy> - && timeout != DAT_TIMEOUT_INFINITE) { tduffy> - ep_ptr->cxn_timer = kmalloc(sizeof *ep_ptr->cxn_timer, tduffy> - GFP_ATOMIC); tduffy> - if (!ep_ptr->cxn_timer) { tduffy> - dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, tduffy> - DAT_RESOURCE_MEMORY); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - dapl_timer_set(ep_ptr->cxn_timer, tduffy> - dapl_ep_timeout, ep_ptr, timeout); tduffy> + && timeout != MAX_SCHEDULE_TIMEOUT) { tduffy> + ep_ptr->cxn_timer.expires = jiffies + timeout; tduffy> + ep_ptr->cxn_timer.data = (unsigned long)ep_ptr; tduffy> + ep_ptr->cxn_timer.function = dapl_ep_timeout; tduffy> + add_timer(&ep_ptr->cxn_timer); tduffy> } tduffy> spin_unlock_irqrestore(&ep_ptr->header.lock, tduffy> ep_ptr->header.flags); tduffy> } tduffy> tduffy> - bail: tduffy> +bail: tduffy> dapl_dbg_log(DAPL_DBG_TYPE_RTN | DAPL_DBG_TYPE_CM, tduffy> "dapl_ep_connect () returns 0x%x\n", dat_status); tduffy> tduffy> Index: linux-kernel-timer/dat-provider/dapl_module.c tduffy> =================================================================== tduffy> --- linux-kernel-timer/dat-provider/dapl_module.c (revision 2457) tduffy> +++ linux-kernel-timer/dat-provider/dapl_module.c (working copy) tduffy> @@ -36,7 +36,6 @@ tduffy> #include "dapl_mr_util.h" tduffy> #include "dapl_util.h" tduffy> #include "dapl_openib_util.h" tduffy> -#include "dapl_timer_util.h" tduffy> tduffy> MODULE_LICENSE("Dual BSD/GPL"); tduffy> MODULE_DESCRIPTION("DAT Provider for InfiniBand"); tduffy> @@ -57,8 +56,6 @@ static int __init dapl_init(void) tduffy> tduffy> dapl_dbg_log(DAPL_DBG_TYPE_UTIL, "provider started\n"); tduffy> tduffy> - dapl_timer_init(); tduffy> - tduffy> dat_status = dapl_provider_list_create(); tduffy> if (DAT_SUCCESS != dat_status) { tduffy> dapl_dbg_log(DAPL_DBG_TYPE_ERR, tduffy> Index: linux-kernel-timer/dat-provider/dapl_evd_connection_callb.c tduffy> =================================================================== tduffy> --- linux-kernel-timer/dat-provider/dapl_evd_connection_callb.c (revision 2457) tduffy> +++ linux-kernel-timer/dat-provider/dapl_evd_connection_callb.c (working copy) tduffy> @@ -32,7 +32,6 @@ tduffy> #include "dapl.h" tduffy> #include "dapl_evd_util.h" tduffy> #include "dapl_ep_util.h" tduffy> -#include "dapl_timer_util.h" tduffy> tduffy> /* tduffy> * dapl_evd_connection_callback tduffy> @@ -90,11 +89,7 @@ void dapl_evd_connection_callback(struct tduffy> /* tduffy> * If a connection timer has been set up on this EP, cancel it now tduffy> */ tduffy> - if (ep_ptr->cxn_timer != NULL) { tduffy> - dapl_timer_cancel(ep_ptr->cxn_timer); tduffy> - kfree(ep_ptr->cxn_timer); tduffy> - ep_ptr->cxn_timer = NULL; tduffy> - } tduffy> + del_timer(&ep_ptr->cxn_timer); tduffy> tduffy> switch (event) { tduffy> case DAT_CONNECTION_EVENT_ESTABLISHED: tduffy> Index: linux-kernel-timer/dat-provider/Makefile tduffy> =================================================================== tduffy> --- linux-kernel-timer/dat-provider/Makefile (revision 2457) tduffy> +++ linux-kernel-timer/dat-provider/Makefile (working copy) tduffy> @@ -101,7 +101,6 @@ PROVIDER_MODULES := \ tduffy> dapl_srq_resize \ tduffy> dapl_srq_set_lw \ tduffy> dapl_srq_util \ tduffy> - dapl_timer_util \ tduffy> dapl_util tduffy> tduffy> PROVIDER_OBJS := $(foreach s, $(PROVIDER_MODULES), $(s).o) tduffy> Index: linux-kernel-timer/dat-provider/dapl_ep_free.c tduffy> =================================================================== tduffy> --- linux-kernel-timer/dat-provider/dapl_ep_free.c (revision 2457) tduffy> +++ linux-kernel-timer/dat-provider/dapl_ep_free.c (working copy) tduffy> @@ -34,7 +34,6 @@ tduffy> #include "dapl_ep_util.h" tduffy> #include "dapl_openib_util.h" tduffy> #include "dapl_ring_buffer_util.h" tduffy> -#include "dapl_timer_util.h" tduffy> tduffy> /* tduffy> * dapl_ep_free tduffy> @@ -107,11 +106,7 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle tduffy> */ tduffy> spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); tduffy> tduffy> - if (ep_ptr->cxn_timer != NULL) { tduffy> - dapl_timer_cancel(ep_ptr->cxn_timer); tduffy> - kfree(ep_ptr->cxn_timer); tduffy> - ep_ptr->cxn_timer = NULL; tduffy> - } tduffy> + del_timer(&ep_ptr->cxn_timer); tduffy> tduffy> /* Remove the EP from the IA */ tduffy> dapl_ia_unlink_ep(ia_ptr, ep_ptr); tduffy> Index: linux-kernel-timer/dat-provider/dapl_timer_util.c tduffy> =================================================================== tduffy> --- linux-kernel-timer/dat-provider/dapl_timer_util.c (revision 2457) tduffy> +++ linux-kernel-timer/dat-provider/dapl_timer_util.c (working copy) tduffy> @@ -1,313 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/* tduffy> - * Routines to add and cancel timer records. A timer record tduffy> - * is put on the global timer queue. If the timer thread is tduffy> - * not running, start it. The timer thread will sleep tduffy> - * until a timer event or until a process wakes it up tduffy> - * to notice a new timer is available; we use a DAPL_WAIT_OBJ tduffy> - * for synchronization. tduffy> - * tduffy> - * If a timer is cancelled, it is simlpy removed from the tduffy> - * queue. The timer may wakeup and notice there is no timer tduffy> - * record to awaken at this time, so it will reset for the tduffy> - * next entry. When there are no timer records to manage, tduffy> - * the timer thread just sleeps until awakened. tduffy> - * tduffy> - * This file also contains the timer handler thread, tduffy> - * embodied in dapl_timer_thread(). tduffy> - * tduffy> - * $Id$ tduffy> - */ tduffy> - tduffy> -#include "dapl.h" tduffy> -#include "dapl_timer_util.h" tduffy> - tduffy> -struct dapl_timer_head { tduffy> - struct dapl_llist_entry *timer_list_head; tduffy> - spinlock_t lock; tduffy> - unsigned long flags; tduffy> - DAPL_OS_WAIT_OBJECT wait_object; tduffy> - int thread_id; tduffy> -}; tduffy> - tduffy> -static struct dapl_timer_head g_dapl_timer_head; tduffy> - tduffy> -static int dapl_timer_thread(void *arg); tduffy> - tduffy> -void dapl_timer_init(void) tduffy> -{ tduffy> - /* tduffy> - * Set up the timer thread elements. The timer thread isn't tduffy> - * started until it is actually needed tduffy> - */ tduffy> - g_dapl_timer_head.timer_list_head = NULL; tduffy> - spin_lock_init(&g_dapl_timer_head.lock); tduffy> - dapl_os_wait_object_init(&g_dapl_timer_head.wait_object); tduffy> - g_dapl_timer_head.thread_id = -1; tduffy> -} tduffy> - tduffy> -/* tduffy> - * dapl_timer_set tduffy> - * tduffy> - * Set a timer. The timer will invoke the specified function tduffy> - * after a number of useconds expires. tduffy> - * tduffy> - * Input: tduffy> - * timer User provided timer structure tduffy> - * func Function to invoke when timer expires tduffy> - * data Argument passed to func() tduffy> - * expires microseconds until timer fires tduffy> - * tduffy> - * Returns: tduffy> - * no return value tduffy> - * tduffy> - */ tduffy> -u32 dapl_timer_set(struct dapl_timer_entry *timer, void (*func) (void *), tduffy> - void *data, DAPL_OS_TIMEVAL expires) tduffy> -{ tduffy> - struct dapl_timer_entry *list_ptr; tduffy> - DAPL_OS_TIMEVAL cur_time; tduffy> - boolean_t wakeup_tmo_thread; tduffy> - tduffy> - /* tduffy> - * Start the timer thread the first time we need a timer tduffy> - */ tduffy> - if ( -1 == g_dapl_timer_head.thread_id ) { tduffy> - g_dapl_timer_head.thread_id = tduffy> - kernel_thread(dapl_timer_thread, &g_dapl_timer_head, 0); tduffy> - } tduffy> - tduffy> - dapl_llist_init_entry(&timer->list_entry); tduffy> - wakeup_tmo_thread = FALSE; tduffy> - dapl_os_get_time(&cur_time); tduffy> - timer->expires = cur_time + expires; /* calculate future time */ tduffy> - timer->function = func; tduffy> - timer->data = data; tduffy> - tduffy> - /* tduffy> - * Put the element on the queue: sorted by wakeup time, eariliest tduffy> - * first. tduffy> - */ tduffy> - spin_lock_irqsave(&g_dapl_timer_head.lock, g_dapl_timer_head.flags); tduffy> - /* tduffy> - * Deal with 3 cases due to our list structure: tduffy> - * 1) list is empty: become the list head tduffy> - * 2) New timer is sooner than list head: become the list head tduffy> - * 3) otherwise, sort the timer into the list, no need to wake tduffy> - * the timer thread up tduffy> - */ tduffy> - if (dapl_llist_is_empty(&g_dapl_timer_head.timer_list_head)) { tduffy> - /* Case 1: add entry to head of list */ tduffy> - dapl_llist_add_head(&g_dapl_timer_head.timer_list_head, tduffy> - (struct dapl_llist_entry *) tduffy> - &timer->list_entry, tduffy> - timer); tduffy> - wakeup_tmo_thread = TRUE; tduffy> - } else { tduffy> - list_ptr = (struct dapl_timer_entry *) tduffy> - dapl_llist_peek_head(&g_dapl_timer_head.timer_list_head); tduffy> - tduffy> - if (timer->expires < list_ptr->expires) { tduffy> - /* Case 2: add entry to head of list */ tduffy> - dapl_llist_add_head(&g_dapl_timer_head.timer_list_head, tduffy> - (struct dapl_llist_entry *) tduffy> - &timer->list_entry, timer); tduffy> - wakeup_tmo_thread = TRUE; tduffy> - } else { tduffy> - /* Case 3: figure out where entry goes in sorted list */ tduffy> - list_ptr = tduffy> - dapl_llist_next_entry(&g_dapl_timer_head. tduffy> - timer_list_head, tduffy> - (struct dapl_llist_entry *) tduffy> - &list_ptr->list_entry); tduffy> - tduffy> - while (list_ptr != NULL) { tduffy> - if (timer->expires < list_ptr->expires) { tduffy> - dapl_llist_add_entry(&g_dapl_timer_head. tduffy> - timer_list_head, tduffy> - (struct dapl_llist_entry *) tduffy> - &list_ptr->list_entry, tduffy> - (struct dapl_llist_entry *) tduffy> - &timer->list_entry, tduffy> - timer); tduffy> - break; tduffy> - tduffy> - } tduffy> - list_ptr = tduffy> - dapl_llist_next_entry(&g_dapl_timer_head. tduffy> - timer_list_head, tduffy> - (struct dapl_llist_entry *) tduffy> - &list_ptr->list_entry); tduffy> - } tduffy> - if (list_ptr == NULL) { tduffy> - /* entry goes to the end of the list */ tduffy> - dapl_llist_add_tail(&g_dapl_timer_head. tduffy> - timer_list_head, tduffy> - (struct dapl_llist_entry *) tduffy> - &timer->list_entry, timer); tduffy> - } tduffy> - } tduffy> - tduffy> - } tduffy> - spin_unlock_irqrestore(&g_dapl_timer_head.lock, tduffy> - g_dapl_timer_head.flags); tduffy> - tduffy> - if (wakeup_tmo_thread == TRUE) { tduffy> - dapl_os_wait_object_wakeup(&g_dapl_timer_head.wait_object); tduffy> - } tduffy> - tduffy> - return DAT_SUCCESS; tduffy> -} tduffy> - tduffy> -/* tduffy> - * dapl_timer_cancel tduffy> - * tduffy> - * Cancel a timer. Simply deletes the timer with no function invocations tduffy> - * tduffy> - * Input: tduffy> - * timer User provided timer structure tduffy> - * tduffy> - * Returns: tduffy> - * no return value tduffy> - */ tduffy> -void dapl_timer_cancel(struct dapl_timer_entry *timer) tduffy> -{ tduffy> - spin_lock_irqsave(&g_dapl_timer_head.lock, g_dapl_timer_head.flags); tduffy> - /* tduffy> - * make sure the entry has not been removed by another thread tduffy> - */ tduffy> - if (!dapl_llist_is_empty(&g_dapl_timer_head.timer_list_head) && tduffy> - timer->list_entry.list_head == &g_dapl_timer_head.timer_list_head) { tduffy> - dapl_llist_remove_entry(&g_dapl_timer_head.timer_list_head, tduffy> - (struct dapl_llist_entry *) tduffy> - &timer->list_entry); tduffy> - } tduffy> - /* tduffy> - * If this was the first entry on the queue we could awaken the tduffy> - * thread and have it reset the list; but it will just wake up tduffy> - * and find that the timer entry has been removed, then go back tduffy> - * to sleep, so don't bother. tduffy> - */ tduffy> - spin_unlock_irqrestore(&g_dapl_timer_head.lock, tduffy> - g_dapl_timer_head.flags); tduffy> -} tduffy> - tduffy> -/* tduffy> - * dapl_timer_thread tduffy> - * tduffy> - * Core worker thread dealing with all timers. Basic algorithm: tduffy> - * - Sleep until work shows up tduffy> - * - Take first element of sorted timer list and wake tduffy> - * invoke the callback if expired tduffy> - * - Sleep for the timeout period if not expired tduffy> - * tduffy> - * Input: tduffy> - * timer_head Timer head structure to manage timer lists tduffy> - * tduffy> - * Returns: tduffy> - * int tduffy> - */ tduffy> -int dapl_timer_thread(void *arg) tduffy> -{ tduffy> - struct dapl_timer_entry *list_ptr; tduffy> - DAPL_OS_TIMEVAL cur_time; tduffy> - struct dapl_timer_head *timer_head; tduffy> - u32 dat_status; tduffy> - tduffy> - timer_head = arg; tduffy> - tduffy> - for (;;) { tduffy> - if (dapl_llist_is_empty(&timer_head->timer_list_head)) { tduffy> - dat_status = tduffy> - dapl_os_wait_object_wait(&timer_head->wait_object, tduffy> - DAT_TIMEOUT_INFINITE); tduffy> - } tduffy> - tduffy> - /* tduffy> - * Lock policy: tduffy> - * While this thread is accessing the timer list, it holds the tduffy> - * lock. Otherwise, it doesn't. tduffy> - */ tduffy> - spin_lock_irqsave(&timer_head->lock, timer_head->flags); tduffy> - while (!dapl_llist_is_empty(&timer_head->timer_list_head)) { tduffy> - list_ptr = (struct dapl_timer_entry *) tduffy> - dapl_llist_peek_head(&g_dapl_timer_head. tduffy> - timer_list_head); tduffy> - dapl_os_get_time(&cur_time); tduffy> - tduffy> - if (list_ptr->expires <= cur_time) { tduffy> - /* tduffy> - * Remove the entry from the list. Sort out how much tduffy> - * time we need to sleep for the next one tduffy> - */ tduffy> - list_ptr = tduffy> - dapl_llist_remove_head(&timer_head-> tduffy> - timer_list_head); tduffy> - spin_unlock_irqrestore(&timer_head->lock, tduffy> - timer_head->flags); tduffy> - tduffy> - /* tduffy> - * Invoke the user callback tduffy> - */ tduffy> - list_ptr->function(list_ptr->data); tduffy> - /* timer structure was allocated by caller, we don't tduffy> - * free it here. tduffy> - */ tduffy> - tduffy> - /* reacquire the lock */ tduffy> - spin_lock_irqsave(&timer_head->lock, tduffy> - timer_head->flags); tduffy> - } else { tduffy> - spin_unlock_irqrestore(&timer_head->lock, tduffy> - timer_head->flags); tduffy> - dat_status = tduffy> - dapl_os_wait_object_wait(&timer_head-> tduffy> - wait_object, tduffy> - (DAT_TIMEOUT) tduffy> - (list_ptr-> tduffy> - expires - tduffy> - cur_time)); tduffy> - spin_lock_irqsave(&timer_head->lock, tduffy> - timer_head->flags); tduffy> - } tduffy> - } tduffy> - /* tduffy> - * release the lock before going back to the top to sleep tduffy> - */ tduffy> - spin_unlock_irqrestore(&timer_head->lock, timer_head->flags); tduffy> - tduffy> - if (dat_status == DAT_INTERNAL_ERROR) { tduffy> - /* tduffy> - * XXX What do we do here? tduffy> - */ tduffy> - } tduffy> - } /* for (;;) */ tduffy> - tduffy> - return 0; tduffy> -} tduffy> Index: linux-kernel-timer/dat-provider/dapl_timer_util.h tduffy> =================================================================== tduffy> --- linux-kernel-timer/dat-provider/dapl_timer_util.h (revision 2457) tduffy> +++ linux-kernel-timer/dat-provider/dapl_timer_util.h (working copy) tduffy> @@ -1,42 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/* tduffy> - * $Id$ tduffy> - */ tduffy> - tduffy> -#ifndef DAPL_TIMER_UTIL_H tduffy> -#define DAPL_TIMER_UTIL_H tduffy> - tduffy> -void dapl_timer_init(void); tduffy> - tduffy> -u32 dapl_timer_set(struct dapl_timer_entry *timer, void (*func) (void *), tduffy> - void *data, DAPL_OS_TIMEVAL expires); tduffy> - tduffy> -void dapl_timer_cancel(struct dapl_timer_entry *timer); tduffy> - tduffy> -#endif /* DAPL_TIMER_UTIL_H */ tduffy> Index: linux-kernel-timer/dat-provider/dapl_ep_util.c tduffy> =================================================================== tduffy> --- linux-kernel-timer/dat-provider/dapl_ep_util.c (revision 2457) tduffy> +++ linux-kernel-timer/dat-provider/dapl_ep_util.c (working copy) tduffy> @@ -75,6 +75,7 @@ struct dapl_ep *dapl_ep_alloc(struct dap tduffy> tduffy> dapl_llist_init_entry(&ep_ptr->header.ia_list_entry); tduffy> spin_lock_init(&ep_ptr->header.lock); tduffy> + init_timer(&ep_ptr->cxn_timer); tduffy> tduffy> /* tduffy> * Initialize the body tduffy> @@ -169,8 +170,7 @@ void dapl_ep_dealloc(struct dapl_ep *ep_ tduffy> if (ep_ptr->send_iov) tduffy> kfree(ep_ptr->send_iov); tduffy> tduffy> - if (ep_ptr->cxn_timer) tduffy> - kfree(ep_ptr->cxn_timer); tduffy> + del_timer(&ep_ptr->cxn_timer); tduffy> tduffy> kfree(ep_ptr); tduffy> } tduffy> @@ -248,7 +248,7 @@ bail: tduffy> * If this routine is invoked before a connection occurs, generate an tduffy> * event tduffy> */ tduffy> -void dapl_ep_timeout(void *arg) tduffy> +void dapl_ep_timeout(unsigned long arg) tduffy> { tduffy> struct dapl_ep *ep_ptr; tduffy> tduffy> Index: linux-kernel-timer/dat-provider/dapl_ep_util.h tduffy> =================================================================== tduffy> --- linux-kernel-timer/dat-provider/dapl_ep_util.h (revision 2457) tduffy> +++ linux-kernel-timer/dat-provider/dapl_ep_util.h (working copy) tduffy> @@ -52,7 +52,7 @@ extern u32 dapl_ep_post_send_req(DAT_EP_ tduffy> DAPL_DTO_TYPE dto_type, tduffy> enum ib_wr_opcode op_type); tduffy> tduffy> -void dapl_ep_timeout(void *arg); tduffy> +void dapl_ep_timeout(unsigned long arg); tduffy> tduffy> enum dat_return_subtype dapl_ep_state_subtype(struct dapl_ep *ep_ptr); tduffy> tduffy> Index: linux-kernel-timer/dat-provider/dapl_util.c tduffy> =================================================================== tduffy> --- linux-kernel-timer/dat-provider/dapl_util.c (revision 2457) tduffy> +++ linux-kernel-timer/dat-provider/dapl_util.c (working copy) tduffy> @@ -33,104 +33,6 @@ tduffy> #include "dapl_module.h" tduffy> #include "dapl_util.h" tduffy> tduffy> -/* tduffy> - * dapl_os_get_time tduffy> - * tduffy> - * Return 64 bit value of current time in microseconds. tduffy> - * tduffy> - * Input: tduffy> - * loc User location to place current time tduffy> - * tduffy> - * Returns: tduffy> - * DAT_SUCCESS tduffy> - */ tduffy> - tduffy> -u32 dapl_os_get_time(DAPL_OS_TIMEVAL *loc) tduffy> -{ tduffy> - struct timeval tv; tduffy> - tduffy> - do_gettimeofday(&tv); tduffy> - *loc = ((u64) (tv.tv_sec) * 1000000L) + (u64) tv.tv_usec; tduffy> - tduffy> - return DAT_SUCCESS; tduffy> -} tduffy> - tduffy> -/* tduffy> - * Wait object routines tduffy> - */ tduffy> - tduffy> -/* tduffy> - * dapl_os_wait_object_init tduffy> - * tduffy> - * Initialize a wait object tduffy> - * tduffy> - * Input: tduffy> - * wait_obj tduffy> - * tduffy> - * Returns: tduffy> - * DAT_SUCCESS tduffy> - * DAT_INTERNAL_ERROR tduffy> - */ tduffy> -u32 dapl_os_wait_object_init(DAPL_OS_WAIT_OBJECT *wait_obj) tduffy> -{ tduffy> - init_waitqueue_head(&wait_obj->wait_queue); tduffy> - tduffy> - return DAT_SUCCESS; tduffy> -} tduffy> - tduffy> -/* Wait on the supplied wait object, up to the specified time_out. tduffy> - * A timeout of DAT_TIMEOUT_INFINITE will wait indefinitely. tduffy> - * Timeout should be specified in micro seconds. tduffy> - * tduffy> - * Functional returns: tduffy> - * DAT_SUCCESS -- another thread invoked dapl_os_wait object_wakeup tduffy> - * DAT_INVALID_STATE -- someone else is already waiting in this wait tduffy> - * object. tduffy> - * only one waiter is allowed at a time. tduffy> - * DAT_ABORT -- another thread invoked dapl_os_wait_object_destroy tduffy> - * DAT_TIMEOUT -- the specified time limit was reached. tduffy> - */ tduffy> - tduffy> -u32 dapl_os_wait_object_wait(DAPL_OS_WAIT_OBJECT *wait_obj, tduffy> - DAT_TIMEOUT timeout_val) tduffy> -{ tduffy> - u32 dat_status = DAT_SUCCESS; tduffy> - tduffy> - if (DAT_TIMEOUT_INFINITE == timeout_val) { tduffy> - interruptible_sleep_on(&wait_obj->wait_queue); tduffy> - } else { tduffy> - int expire; tduffy> - tduffy> - expire = timeout_val * HZ / 1000000; tduffy> - tduffy> - while (expire) { tduffy> - current->state = TASK_INTERRUPTIBLE; tduffy> - expire = schedule_timeout(expire); tduffy> - } tduffy> - dat_status = DAT_ERROR(DAT_TIMEOUT_EXPIRED, 0); tduffy> - } tduffy> - return dat_status; tduffy> -} tduffy> - tduffy> -/* tduffy> - * dapl_os_wait_object_wakeup tduffy> - * tduffy> - * Wakeup a thread waiting on a wait object tduffy> - * tduffy> - * Input: tduffy> - * wait_obj tduffy> - * tduffy> - * Returns: tduffy> - * DAT_SUCCESS tduffy> - * DAT_INTERNAL_ERROR tduffy> - */ tduffy> -u32 dapl_os_wait_object_wakeup(DAPL_OS_WAIT_OBJECT *wait_obj) tduffy> -{ tduffy> - wake_up_interruptible(&wait_obj->wait_queue); tduffy> - tduffy> - return DAT_SUCCESS; tduffy> -} tduffy> - tduffy> #ifdef DAPL_DBG tduffy> tduffy> void dapl_dbg_log(DAPL_DBG_TYPE type, const char *fmt, ...) tduffy> Index: linux-kernel-timer/dat-provider/dapl_util.h tduffy> =================================================================== tduffy> --- linux-kernel-timer/dat-provider/dapl_util.h (revision 2457) tduffy> +++ linux-kernel-timer/dat-provider/dapl_util.h (working copy) tduffy> @@ -107,24 +107,6 @@ static inline int dapl_os_atomic_assign( tduffy> */ tduffy> tduffy> /* tduffy> - * The wait object invariant: Presuming a call to dapl_os_wait_object_wait tduffy> - * occurs at some point, there will be at least one wakeup after each call tduffy> - * to dapl_os_wait_object_signal. I.e. Signals are not ignored, though tduffy> - * they may be coallesced. tduffy> - */ tduffy> -typedef struct dapl_os_wait_object { tduffy> - wait_queue_head_t wait_queue; tduffy> -} DAPL_OS_WAIT_OBJECT; tduffy> - tduffy> -/* function prototypes */ tduffy> -u32 dapl_os_wait_object_init(DAPL_OS_WAIT_OBJECT *wait_obj); tduffy> - tduffy> -u32 dapl_os_wait_object_wait(DAPL_OS_WAIT_OBJECT *wait_obj, tduffy> - DAT_TIMEOUT timeout_val); tduffy> - tduffy> -u32 dapl_os_wait_object_wakeup(DAPL_OS_WAIT_OBJECT *wait_obj); tduffy> - tduffy> -/* tduffy> * Memory Functions tduffy> */ tduffy> tduffy> @@ -154,15 +136,6 @@ static inline char *dapl_os_strdup(const tduffy> } tduffy> tduffy> /* tduffy> - * Timer Functions tduffy> - */ tduffy> - tduffy> -typedef u64 DAPL_OS_TIMEVAL; tduffy> - tduffy> -/* timer function prototype */ tduffy> -u32 dapl_os_get_time(DAPL_OS_TIMEVAL *); tduffy> - tduffy> -/* tduffy> * *printf format helper. We use the C string constant concatenation tduffy> * ability to define 64 bit formats, which unfortunatly are non standard tduffy> * in the C compiler world. tduffy> Index: linux-kernel-timer/dat-provider/dapl_ep_dup_connect.c tduffy> =================================================================== tduffy> --- linux-kernel-timer/dat-provider/dapl_ep_dup_connect.c (revision 2457) tduffy> +++ linux-kernel-timer/dat-provider/dapl_ep_dup_connect.c (working copy) tduffy> @@ -32,7 +32,6 @@ tduffy> #include "dapl.h" tduffy> #include "dapl_ep_util.h" tduffy> #include "dapl_openib_util.h" tduffy> -#include "dapl_timer_util.h" tduffy> tduffy> /* tduffy> * dapl_ep_dup_connect tduffy> @@ -63,7 +62,7 @@ tduffy> * DAT_MODEL_NOT_SUPPORTED tduffy> */ tduffy> u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, DAT_EP_HANDLE ep_dup_handle, tduffy> - DAT_TIMEOUT timeout, int private_data_size, tduffy> + unsigned long timeout, int private_data_size, tduffy> const void *private_data, enum dat_qos qos) tduffy> { tduffy> struct dapl_ep *ep_dup_ptr; tduffy> @@ -114,6 +113,6 @@ u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep tduffy> private_data_size, tduffy> private_data, tduffy> qos, DAT_CONNECT_DEFAULT_FLAG); tduffy> - bail: tduffy> +bail: tduffy> return dat_status; tduffy> } tduffy> Index: linux-kernel-timer/dat-provider/dapl.h tduffy> =================================================================== tduffy> --- linux-kernel-timer/dat-provider/dapl.h (revision 2457) tduffy> +++ linux-kernel-timer/dat-provider/dapl.h (working copy) tduffy> @@ -237,14 +237,6 @@ struct dapl_evd { tduffy> struct dapl_cno *cno_ptr; tduffy> }; tduffy> tduffy> -/* DAPL timer entry, used to queue timeouts */ tduffy> -struct dapl_timer_entry { tduffy> - struct dapl_llist_entry list_entry; /* link entry on ia struct */ tduffy> - DAPL_OS_TIMEVAL expires; tduffy> - void (*function) (void *); tduffy> - void *data; tduffy> -}; tduffy> - tduffy> /* struct dapl_ep maps to DAT_EP_HANDLE */ tduffy> struct dapl_ep { tduffy> struct dapl_header header; tduffy> @@ -266,8 +258,8 @@ struct dapl_ep { tduffy> /* For passive connections we maintain a back pointer to the CR */ tduffy> void *cr_ptr; tduffy> tduffy> - /* pointer to connection timer, if set */ tduffy> - struct dapl_timer_entry *cxn_timer; tduffy> + /* connection timer */ tduffy> + struct timer_list cxn_timer; tduffy> tduffy> /* private data container */ tduffy> char private_data[DAPL_MAX_PRIVATE_DATA_SIZE]; tduffy> @@ -500,14 +492,15 @@ extern u32 dapl_ep_modify(DAT_EP_HANDLE tduffy> extern u32 dapl_ep_connect(DAT_EP_HANDLE ep_handle, tduffy> struct sockaddr *remote_ia_address, tduffy> DAT_CONN_QUAL remote_conn_qual, tduffy> - DAT_TIMEOUT timeout, tduffy> + unsigned long timeout, tduffy> int private_data_size, tduffy> const void *private_data, tduffy> enum dat_qos quality_of_service, tduffy> enum dat_connect_flags connect_flags); tduffy> tduffy> extern u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, tduffy> - DAT_EP_HANDLE ep_dup_handle, DAT_TIMEOUT timeout, tduffy> + DAT_EP_HANDLE ep_dup_handle, tduffy> + unsigned long timeout, tduffy> int private_data_size, tduffy> const void *private_data, tduffy> enum dat_qos quality_of_service); tduffy> Index: linux-kernel-timer/patches/alt_dat_provider_makefile tduffy> =================================================================== tduffy> --- linux-kernel-timer/patches/alt_dat_provider_makefile (revision 2457) tduffy> +++ linux-kernel-timer/patches/alt_dat_provider_makefile (working copy) tduffy> @@ -95,7 +95,6 @@ PROVIDER_MODULES := \ tduffy> dapl_srq_resize \ tduffy> dapl_srq_set_lw \ tduffy> dapl_srq_util \ tduffy> - dapl_timer_util \ tduffy> dapl_util tduffy> tduffy> PROVIDER_OBJS := $(foreach s, $(PROVIDER_MODULES), $(s).o) tduffy> @@ -103,7 +102,7 @@ PROVIDER_OBJS := $(foreach s, $(PROVIDER tduffy> KDAPL_DEBUG = 1 tduffy> ifeq (1,$(KDAPL_DEBUG)) tduffy> EXTRA_CFLAGS += -O0 -g tduffy> - EXTRA_CFLAGS += -DDAPL_DBG # -DDAPL_DBG_IO_TRC tduffy> + EXTRA_CFLAGS += -DDAPL_DBG tduffy> endif tduffy> tduffy> EXTRA_CFLAGS += -DDAPL_ATS -Idrivers/infiniband/include -I$(obj)/../dat -I$(obj)/../patches/ tduffy> @@ -121,4 +120,4 @@ default: tduffy> endif tduffy> tduffy> clean: tduffy> - rm -f *.o *.ko tduffy> + rm -rf *.o *.ko .*.cmd *.mod.* .tmp_versions .*.d .*.o.tmp cscope.out tduffy> Index: linux-kernel-timer/dat/dat.h tduffy> =================================================================== tduffy> --- linux-kernel-timer/dat/dat.h (revision 2457) tduffy> +++ linux-kernel-timer/dat/dat.h (working copy) tduffy> @@ -383,11 +383,6 @@ enum dat_completion_flags { tduffy> DAT_COMPLETION_BARRIER_FENCE_FLAG = 0x08, tduffy> }; tduffy> tduffy> -typedef u32 DAT_TIMEOUT; /* microseconds */ tduffy> - tduffy> -/* timeout = infinity */ tduffy> -#define DAT_TIMEOUT_INFINITE ((DAT_TIMEOUT) ~0) tduffy> - tduffy> /* dat handles */ tduffy> typedef void *DAT_HANDLE; tduffy> typedef DAT_HANDLE DAT_CR_HANDLE; tduffy> @@ -1139,12 +1134,12 @@ typedef u32 (*DAT_EP_MODIFY_FUNC)(DAT_EP tduffy> const struct dat_ep_param *); tduffy> tduffy> typedef u32 (*DAT_EP_CONNECT_FUNC)(DAT_EP_HANDLE, struct sockaddr *, tduffy> - DAT_CONN_QUAL, DAT_TIMEOUT, int, tduffy> + DAT_CONN_QUAL, unsigned long timeout, int, tduffy> const void *, enum dat_qos, tduffy> enum dat_connect_flags); tduffy> tduffy> typedef u32 (*DAT_EP_DUP_CONNECT_FUNC)(DAT_EP_HANDLE, DAT_EP_HANDLE, tduffy> - DAT_TIMEOUT, int, const void *, tduffy> + unsigned long timeout, int, const void *, tduffy> enum dat_qos); tduffy> tduffy> typedef u32 (*DAT_EP_DISCONNECT_FUNC)(DAT_EP_HANDLE, enum dat_close_flags); tduffy> @@ -1493,7 +1488,7 @@ static inline u32 dat_evd_resize(DAT_EVD tduffy> } tduffy> tduffy> static inline u32 dat_ep_connect(DAT_EP_HANDLE ep, struct sockaddr *ia_addr, tduffy> - DAT_CONN_QUAL conn_qual, DAT_TIMEOUT timeout, tduffy> + DAT_CONN_QUAL conn_qual, unsigned long timeout, tduffy> int private_data_size, tduffy> const void *private_data, enum dat_qos qos, tduffy> enum dat_connect_flags flags) tduffy> @@ -1533,7 +1528,8 @@ static inline u32 dat_ep_disconnect(DAT_ tduffy> } tduffy> tduffy> static inline u32 dat_ep_dup_connect(DAT_EP_HANDLE ep, DAT_EP_HANDLE dup_ep, tduffy> - DAT_TIMEOUT timeout, int private_data_size, tduffy> + unsigned long timeout, tduffy> + int private_data_size, tduffy> const void *private_data, enum dat_qos qos) tduffy> { tduffy> return DAT_CALL_PROVIDER_FUNC(ep_dup_connect_func, ep, dup_ep, timeout, tduffy> From jlentini at netapp.com Tue May 24 11:43:06 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 24 May 2005 14:43:06 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: cleanup some C++ style comments In-Reply-To: <1116912349.14182.1.camel@duffman> References: <1116912349.14182.1.camel@duffman> Message-ID: Committed in revision 2472. On Mon, 23 May 2005, Tom Duffy wrote: tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> Index: linux-kernel-new/dat-provider/dapl_openib_cm.c tduffy> =================================================================== tduffy> --- linux-kernel-new/dat-provider/dapl_openib_cm.c (revision 2462) tduffy> +++ linux-kernel-new/dat-provider/dapl_openib_cm.c (working copy) tduffy> @@ -573,12 +573,14 @@ void dapl_ib_reinit_ep(struct dapl_ep *e tduffy> tduffy> ia_ptr = ep_ptr->header.owner_ia; tduffy> tduffy> - //*** What if QP is connected? Do we need to destroy the cm_id tduffy> - //*** here as well, send a DREQ, REJ? What cm_id state are we in? tduffy> - //*** we may not care if we just destroy the cm_id tduffy> - //*** ib_destroy_cm_id(ep_ptr->cm_handle); tduffy> - //*** supporting this call may require setting the cm_handle to tduffy> - //*** something like NULL wherever it is destroyed tduffy> + /* tduffy> + * What if QP is connected? Do we need to destroy the cm_id tduffy> + * here as well, send a DREQ, REJ? What cm_id state are we in? tduffy> + * we may not care if we just destroy the cm_id tduffy> + * ib_destroy_cm_id(ep_ptr->cm_handle); tduffy> + * supporting this call may require setting the cm_handle to tduffy> + * something like NULL wherever it is destroyed tduffy> + */ tduffy> tduffy> ib_status = dapl_modify_qp_state_to_reset(ep_ptr->qp_handle); tduffy> if (ib_status < 0) { tduffy> @@ -660,8 +662,10 @@ u32 dapl_ib_setup_conn_listener(struct d tduffy> */ tduffy> u32 dapl_ib_remove_conn_listener(struct dapl_ia *ia_ptr, struct dapl_sp *sp_ptr) tduffy> { tduffy> - //*** This will hang if called from CM thread context... tduffy> - //*** Move back to using WQ... tduffy> + /* tduffy> + * This will hang if called from CM thread context... tduffy> + * Move back to using WQ... tduffy> + */ tduffy> if (sp_ptr->cm_srvc_handle != NULL) { tduffy> ib_destroy_cm_id(sp_ptr->cm_srvc_handle); tduffy> sp_ptr->cm_srvc_handle = NULL; tduffy> From mshefty at ichips.intel.com Tue May 24 11:50:03 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Tue, 24 May 2005 11:50:03 -0700 Subject: [openib-general] Re: [PATCH][RFC] kDAPL: use linux native timers instead of own In-Reply-To: References: <1116889028.19768.6.camel@duffman> Message-ID: <4293775B.4080303@ichips.intel.com> James Lentini wrote: > - Could we use cm timeouts? > > The timers are only used to implement 2 DAT API calls: > dat_ep_connect and dat_ep_dup_connect. Both use a timer to > implement the same feature (cancel connection attempts > after some period of time) and both eventually call > ib_send_cm_req(). ib_send_cm_req() allows the consumer to specify > timeouts. What happens when these timeouts expire? Could DAPL use > these instead of kernel timers? > > I don't see an ib_cm_event_type that corresponds to a timeout. This > implies that the CM user's ib_cm_handler is not invoked when one of > these timeout values expires. If so kDAPL wouldn't be able to make > use of them. Does anyone know how these work? The CM will invoke the user's ib_cm_handler with an "ERROR" status if a timeout occurs. For example, a user calls ib_send_cm_req(). If a REP is not received within the timeout/retry window, the REQ will fail with IB_CM_REQ_ERROR. You shouldn't need any timers to cancel the connection requests. - Sean From jlentini at netapp.com Tue May 24 12:00:13 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 24 May 2005 15:00:13 -0400 (EDT) Subject: [openib-general] Re: [PATCH][RFC] kDAPL: use linux native timers instead of own In-Reply-To: <4293775B.4080303@ichips.intel.com> References: <1116889028.19768.6.camel@duffman> <4293775B.4080303@ichips.intel.com> Message-ID: On Tue, 24 May 2005, Sean Hefty wrote: > James Lentini wrote: > > - Could we use cm timeouts? >> >> The timers are only used to implement 2 DAT API calls: dat_ep_connect >> and dat_ep_dup_connect. Both use a timer to implement the same feature >> (cancel connection attempts after some period of time) and both >> eventually call ib_send_cm_req(). ib_send_cm_req() allows the consumer >> to specify timeouts. What happens when these timeouts expire? Could DAPL >> use these instead of kernel timers? >> I don't see an ib_cm_event_type that corresponds to a timeout. This >> implies that the CM user's ib_cm_handler is not invoked when one of these >> timeout values expires. If so kDAPL wouldn't be able to make >> use of them. Does anyone know how these work? > > The CM will invoke the user's ib_cm_handler with an "ERROR" status if a > timeout occurs. > > For example, a user calls ib_send_cm_req(). If a REP is not received within > the timeout/retry window, the REQ will fail with IB_CM_REQ_ERROR. You > shouldn't need any timers to cancel the connection requests. Excellent. I guess the original DAPL timer wasn't necessary for gen2. Sean, what unit of time are the timeouts expected in? Tom, are you willing to rework the patch to use the CM timeout values? From mshefty at ichips.intel.com Tue May 24 12:05:51 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Tue, 24 May 2005 12:05:51 -0700 Subject: [openib-general] Re: [PATCH][RFC] kDAPL: use linux native timers instead of own In-Reply-To: References: <1116889028.19768.6.camel@duffman> <4293775B.4080303@ichips.intel.com> Message-ID: <42937B0F.8040707@ichips.intel.com> James Lentini wrote: > Sean, what unit of time are the timeouts expected in? > > Tom, are you willing to rework the patch to use the CM timeout values? The timeouts are in the intuitive format of 4.096 us x 2 ^ (timeout). Basically, the CM takes the timeout values return in a path record. The CM converts the provided value into ms using something similar to: 1 << (timeout - 8) - Sean From roland at topspin.com Tue May 24 12:07:24 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 24 May 2005 12:07:24 -0700 Subject: [openib-general] Re: registering read-only memory In-Reply-To: <20050519085208.GC4717@mellanox.co.il> (Michael S. Tsirkin's message of "Thu, 19 May 2005 11:52:08 +0300") References: <20050519085208.GC4717@mellanox.co.il> Message-ID: <52ekbwv3wz.fsf@topspin.com> > const char foo[]="Michael Tsirkin"; > ibv_reg_mr(ctx->pd, foo, strlen(foo), 0); This seemed to work fine on my system (I got a non-null ibv_mr back with valid-looking L_Key/R_Key). - R. From mshefty at ichips.intel.com Tue May 24 12:14:41 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Tue, 24 May 2005 12:14:41 -0700 Subject: [openib-general] Re: Latest CM and retransmissions In-Reply-To: <1116954610.4490.73.camel@hal.voltaire.com> References: <1116952775.4490.53.camel@hal.voltaire.com> <42935A70.4030003@ichips.intel.com> <1116954610.4490.73.camel@hal.voltaire.com> Message-ID: <42937D21.7090209@ichips.intel.com> Hal Rosenstock wrote: > That fixed the connection setup side but not teardown. I still see: > > DREQ --------> > <------- DREP > DREQ --------> > DREQ --------> I ran some more tests, and I can't reproduce this. Can you describe your test setup wrt the CM more? - Sean From jlentini at netapp.com Tue May 24 12:20:49 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 24 May 2005 15:20:49 -0400 (EDT) Subject: [openib-general] Re: [PATCH] [kdapl CM] Fix endian conversions of service ID In-Reply-To: <1116959536.4490.166.camel@hal.voltaire.com> References: <1116959536.4490.166.camel@hal.voltaire.com> Message-ID: halr> [kdapl CM] Fix endian conversions of service ID halr> Problem pointed out by James Lentini halr> halr> Signed-off-by: Hal Rosenstock halr> halr> Index: dapl_openib_cm.c halr> =================================================================== halr> -- dapl_openib_cm.c (revision 2468) halr> +++ dapl_openib_cm.c (working copy) halr> @@ -309,7 +309,7 @@ halr> if (conn->dapl_path.mtu > IB_MTU_1024) halr> conn->dapl_path.mtu = IB_MTU_1024; halr> halr> - conn->param.service_id = be64_to_cpu(conn->service_id); halr> + conn->param.service_id = conn->service_id; With the change to dapl_ib_connect below, the conn->service_id is in CPU byte order at this point. The conn->param is a ib_cm_req_param structure. The comment describing this structure's service_id field says that it should be in network (big endian) byte order. So... halr> conn->param.primary_path = &conn->dapl_path; halr> conn->param.alternate_path = NULL; halr> halr> @@ -445,8 +445,7 @@ halr> conn->param.local_cm_response_timeout = halr> DAPL_OPENIB_CM_RESPONSE_TIMEOUT; halr> conn->param.max_cm_retries = DAPL_OPENIB_MAX_CM_RETRIES; halr> halr> - memcpy(&conn->service_id, &remote_conn_qual, sizeof halr> conn->service_id); halr> - halr> + conn->service_id = be64_to_cpu(remote_conn_qual); ...that makes me think we should change the line above to conn->service_id = remote_conn_qual; and require that consumer's specify their connection qualifier values in network byte order here and ... halr> conn->remote_ia_address = remote_ia_address; halr> conn->dapl_comp.fn = &dapl_rt_comp_handler; halr> conn->dapl_comp.context = conn; halr> @@ -627,7 +626,7 @@ halr> } halr> halr> status = ib_cm_listen(sp_ptr->cm_srvc_handle, halr> - be64_to_cpu(sp_ptr->conn_qual), 0); halr> + cpu_to_be64(sp_ptr->conn_qual), 0); ... do the same here. What do you think? halr> if (status) { halr> ib_destroy_cm_id(sp_ptr->cm_srvc_handle); halr> sp_ptr->cm_srvc_handle = NULL; halr> halr> halr> From halr at voltaire.com Tue May 24 12:20:21 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 24 May 2005 15:20:21 -0400 Subject: [openib-general] Re: Latest CM and retransmissions In-Reply-To: <42937D21.7090209@ichips.intel.com> References: <1116952775.4490.53.camel@hal.voltaire.com> <42935A70.4030003@ichips.intel.com> <1116954610.4490.73.camel@hal.voltaire.com> <42937D21.7090209@ichips.intel.com> Message-ID: <1116962382.4490.184.camel@hal.voltaire.com> On Tue, 2005-05-24 at 15:14, Sean Hefty wrote: > Hal Rosenstock wrote: > > That fixed the connection setup side but not teardown. I still see: > > > > DREQ --------> > > <------- DREP > > DREQ --------> > > DREQ --------> > > I ran some more tests, and I can't reproduce this. Can you describe your > test setup wrt the CM more? It's just running the kdapl quit test between 2 machines. -- Hal From mshefty at ichips.intel.com Tue May 24 12:28:30 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Tue, 24 May 2005 12:28:30 -0700 Subject: [openib-general] Re: Latest CM and retransmissions In-Reply-To: <1116962382.4490.184.camel@hal.voltaire.com> References: <1116952775.4490.53.camel@hal.voltaire.com> <42935A70.4030003@ichips.intel.com> <1116954610.4490.73.camel@hal.voltaire.com> <42937D21.7090209@ichips.intel.com> <1116962382.4490.184.camel@hal.voltaire.com> Message-ID: <4293805E.8050100@ichips.intel.com> Hal Rosenstock wrote: >>>DREQ --------> >>> <------- DREP >>>DREQ --------> >>>DREQ --------> >> >>I ran some more tests, and I can't reproduce this. Can you describe your >>test setup wrt the CM more? > > > It's just running the kdapl quit test between 2 machines. Can you describe what this test does? This result can occur if the DREP is lost and the sending cm_id exits timewait before the DREQ is received a second time. How many QPs are connecting at once? Just one? - Sean From jlentini at netapp.com Tue May 24 12:30:27 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 24 May 2005 15:30:27 -0400 (EDT) Subject: [openib-general] Re: [PATCH][RFC] kDAPL: use linux native timers instead of own In-Reply-To: <42937B0F.8040707@ichips.intel.com> References: <1116889028.19768.6.camel@duffman> <4293775B.4080303@ichips.intel.com> <42937B0F.8040707@ichips.intel.com> Message-ID: On Tue, 24 May 2005, Sean Hefty wrote: > James Lentini wrote: >> Sean, what unit of time are the timeouts expected in? >> >> Tom, are you willing to rework the patch to use the CM timeout values? > > The timeouts are in the intuitive format of 4.096 us x 2 ^ (timeout). Out of curiosity, why the strange format? Is it an IB spec thing? > Basically, the CM takes the timeout values return in a path record. The CM > converts the provided value into ms using something similar to: > > 1 << (timeout - 8) > > - Sean > From halr at voltaire.com Tue May 24 12:26:54 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 24 May 2005 15:26:54 -0400 Subject: [openib-general] Re: [PATCH] [kdapl CM] Fix endian conversions of service ID In-Reply-To: References: <1116959536.4490.166.camel@hal.voltaire.com> Message-ID: <1116962814.4490.192.camel@hal.voltaire.com> On Tue, 2005-05-24 at 15:20, James Lentini wrote: > halr> [kdapl CM] Fix endian conversions of service ID > halr> Problem pointed out by James Lentini > halr> > halr> Signed-off-by: Hal Rosenstock > halr> > halr> Index: dapl_openib_cm.c > halr> =================================================================== > halr> -- dapl_openib_cm.c (revision 2468) > halr> +++ dapl_openib_cm.c (working copy) > halr> @@ -309,7 +309,7 @@ > halr> if (conn->dapl_path.mtu > IB_MTU_1024) > halr> conn->dapl_path.mtu = IB_MTU_1024; > halr> > halr> - conn->param.service_id = be64_to_cpu(conn->service_id); > halr> + conn->param.service_id = conn->service_id; > > With the change to dapl_ib_connect below, the conn->service_id is in > CPU byte order at this point. The conn->param is a ib_cm_req_param > structure. The comment describing this structure's service_id field > says that it should be in network (big endian) byte order. > > So... > > halr> conn->param.primary_path = &conn->dapl_path; > halr> conn->param.alternate_path = NULL; > halr> > halr> @@ -445,8 +445,7 @@ > halr> conn->param.local_cm_response_timeout = > halr> DAPL_OPENIB_CM_RESPONSE_TIMEOUT; > halr> conn->param.max_cm_retries = DAPL_OPENIB_MAX_CM_RETRIES; > halr> > halr> - memcpy(&conn->service_id, &remote_conn_qual, sizeof > halr> conn->service_id); > halr> - > halr> + conn->service_id = be64_to_cpu(remote_conn_qual); > > ...that makes me think we should change the line above to > > conn->service_id = remote_conn_qual; > > and require that consumer's specify their connection qualifier values > in network byte order here and ... I think the convention OpenIB has been using is to supply parameters in CPU endian but it can work either way. > halr> conn->remote_ia_address = remote_ia_address; > halr> conn->dapl_comp.fn = &dapl_rt_comp_handler; > halr> conn->dapl_comp.context = conn; > halr> @@ -627,7 +626,7 @@ > halr> } > halr> > halr> status = ib_cm_listen(sp_ptr->cm_srvc_handle, > halr> - be64_to_cpu(sp_ptr->conn_qual), 0); > halr> + cpu_to_be64(sp_ptr->conn_qual), 0); > > ... do the same here. What do you think? From mshefty at ichips.intel.com Tue May 24 12:32:51 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Tue, 24 May 2005 12:32:51 -0700 Subject: [openib-general] Re: [PATCH][RFC] kDAPL: use linux native timers instead of own In-Reply-To: References: <1116889028.19768.6.camel@duffman> <4293775B.4080303@ichips.intel.com> <42937B0F.8040707@ichips.intel.com> Message-ID: <42938163.9060605@ichips.intel.com> James Lentini wrote: > > > On Tue, 24 May 2005, Sean Hefty wrote: > >> James Lentini wrote: >> >>> Sean, what unit of time are the timeouts expected in? >>> >>> Tom, are you willing to rework the patch to use the CM timeout values? >> >> >> The timeouts are in the intuitive format of 4.096 us x 2 ^ (timeout). > > > Out of curiosity, why the strange format? Is it an IB spec thing? These are the timeouts specified in the IB spec. The CM is anticipating that a client will have obtained a path record from the SM, which will have the timeouts in this format. - Sean From halr at voltaire.com Tue May 24 12:39:10 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 24 May 2005 15:39:10 -0400 Subject: [openib-general] Re: Latest CM and retransmissions In-Reply-To: <4293805E.8050100@ichips.intel.com> References: <1116952775.4490.53.camel@hal.voltaire.com> <42935A70.4030003@ichips.intel.com> <1116954610.4490.73.camel@hal.voltaire.com> <42937D21.7090209@ichips.intel.com> <1116962382.4490.184.camel@hal.voltaire.com> <4293805E.8050100@ichips.intel.com> Message-ID: <1116963550.4490.198.camel@hal.voltaire.com> On Tue, 2005-05-24 at 15:28, Sean Hefty wrote: > Hal Rosenstock wrote: > >>>DREQ --------> > >>> <------- DREP > >>>DREQ --------> > >>>DREQ --------> > >> > >>I ran some more tests, and I can't reproduce this. Can you describe your > >>test setup wrt the CM more? > > > > > > It's just running the kdapl quit test between 2 machines. > > Can you describe what this test does? This result can occur if the DREP is > lost and the sending cm_id exits timewait before the DREQ is received a > second time. I don't think the DREP is lost and is coming back quickly (177.436 usec after the DREQ). The connection was set up for 4 retries. I see the original one followed by the DREP followed by 4 more DREQs (for that same connection). > How many QPs are connecting at once? Just one? Just one in this test invocation. -- Hal From sean.hefty at intel.com Tue May 24 12:46:16 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Tue, 24 May 2005 12:46:16 -0700 Subject: [openib-general] [PATCH] [CM] repeat private data with MRA, RTU, DREP messages Message-ID: The following patch finishes adding in support to repeat user-specified private data in MRA, RTU, and DREP messages. Signed-off-by: Sean Hefty Index: cm.c =================================================================== --- cm.c (revision 2460) +++ cm.c (working copy) @@ -219,6 +219,32 @@ static void cm_free_msg(struct ib_mad_se ib_free_send_mad(msg); } +static void * cm_copy_private_data(const void *private_data, + u8 private_data_len) +{ + void *data; + + if (!private_data || !private_data_len) + return NULL; + + data = kmalloc(private_data_len, GFP_KERNEL); + if (!data) + return ERR_PTR(-ENOMEM); + + memcpy(data, private_data, private_data_len); + return data; +} + +static void cm_set_private_data(struct cm_id_private *cm_id_priv, + void *private_data, u8 private_data_len) +{ + if (cm_id_priv->private_data && cm_id_priv->private_data_len) + kfree(cm_id_priv->private_data); + + cm_id_priv->private_data = private_data; + cm_id_priv->private_data_len = private_data_len; +} + static void cm_set_ah_attr(struct ib_ah_attr *ah_attr, u8 port_num, u16 dlid, u8 sl, u16 src_path_bits) { @@ -672,6 +698,8 @@ retest: wait_event(cm_id_priv->wait, !atomic_read(&cm_id_priv->refcount)); while ((work = cm_dequeue_work(cm_id_priv)) != NULL) cm_free_work(work); + if (cm_id_priv->private_data && cm_id_priv->private_data_len) + kfree(cm_id_priv->private_data); kfree(cm_id_priv); } EXPORT_SYMBOL(ib_destroy_cm_id); @@ -1373,6 +1401,7 @@ int ib_send_cm_rtu(struct ib_cm_id *cm_i struct ib_mad_send_buf *msg; struct ib_send_wr *bad_send_wr; unsigned long flags; + void *data; int ret; if (private_data && private_data_len > IB_CM_RTU_PRIVATE_DATA_SIZE) @@ -1381,7 +1410,13 @@ int ib_send_cm_rtu(struct ib_cm_id *cm_i cm_id_priv = container_of(cm_id, struct cm_id_private, id); ret = cm_alloc_msg(cm_id_priv, &msg); if (ret) - goto out; + return ret; + + data = cm_copy_private_data(private_data, private_data_len); + if (IS_ERR(data)) { + ret = PTR_ERR(data); + goto error1; + } cm_format_rtu((struct cm_rtu_msg *) msg->mad, cm_id_priv, private_data, private_data_len); @@ -1394,15 +1429,17 @@ int ib_send_cm_rtu(struct ib_cm_id *cm_i else ret = -EINVAL; - if (ret) { - spin_unlock_irqrestore(&cm_id_priv->lock, flags); - cm_free_msg(msg); - goto out; - } + if (ret) + goto error2; cm_id->state = IB_CM_ESTABLISHED; + cm_set_private_data(cm_id_priv, data, private_data_len); spin_unlock_irqrestore(&cm_id_priv->lock, flags); -out: + return 0; + +error2: spin_unlock_irqrestore(&cm_id_priv->lock, flags); + kfree(data); +error1: cm_free_msg(msg); return ret; } EXPORT_SYMBOL(ib_send_cm_rtu); @@ -1706,31 +1743,51 @@ int ib_send_cm_drep(struct ib_cm_id *cm_ struct ib_mad_send_buf *msg; struct ib_send_wr *bad_send_wr; unsigned long flags; - int msg_ret, ret; + void *data; + int ret; if (private_data && private_data_len > IB_CM_DREP_PRIVATE_DATA_SIZE) return -EINVAL; cm_id_priv = container_of(cm_id, struct cm_id_private, id); - msg_ret = cm_alloc_msg(cm_id_priv, &msg); - if (!msg_ret) - cm_format_drep((struct cm_drep_msg *) msg->mad, cm_id_priv, - private_data, private_data_len); + ret = cm_alloc_msg(cm_id_priv, &msg); + if (ret) + goto error1; + + data = cm_copy_private_data(private_data, private_data_len); + if (IS_ERR(data)) { + ret = PTR_ERR(data); + goto error2; + } + + cm_format_drep((struct cm_drep_msg *) msg->mad, cm_id_priv, + private_data, private_data_len); spin_lock_irqsave(&cm_id_priv->lock, flags); if (cm_id->state != IB_CM_DREQ_RCVD) { spin_unlock_irqrestore(&cm_id_priv->lock, flags); - ret = -EINVAL; - goto out; + cm_free_msg(msg); + kfree(data); + return -EINVAL; } - ret = msg_ret ? msg_ret : - ib_post_send_mad(cm_id_priv->av.port->mad_agent, - &msg->send_wr, &bad_send_wr); + + ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent, &msg->send_wr, + &bad_send_wr); + if (ret) + cm_free_msg(msg); + + cm_set_private_data(cm_id_priv, data, private_data_len); cm_enter_timewait(cm_id_priv); spin_unlock_irqrestore(&cm_id_priv->lock, flags); -out: - if (!msg_ret && ret) - cm_free_msg(msg); + return ret; + +error2: + cm_free_msg(msg); +error1: + spin_lock_irqsave(&cm_id_priv->lock, flags); + if (cm_id->state == IB_CM_DREQ_RCVD) + cm_enter_timewait(cm_id_priv); + spin_unlock_irqrestore(&cm_id_priv->lock, flags); return ret; } EXPORT_SYMBOL(ib_send_cm_drep); @@ -2005,6 +2062,7 @@ int ib_send_cm_mra(struct ib_cm_id *cm_i struct cm_id_private *cm_id_priv; struct ib_mad_send_buf *msg; struct ib_send_wr *bad_send_wr; + void *data; unsigned long flags; int ret; @@ -2016,6 +2074,12 @@ int ib_send_cm_mra(struct ib_cm_id *cm_i if (ret) return ret; + data = cm_copy_private_data(private_data, private_data_len); + if (IS_ERR(data)) { + ret = PTR_ERR(data); + goto error1; + } + spin_lock_irqsave(&cm_id_priv->lock, flags); switch(cm_id_priv->id.state) { case IB_CM_REQ_RCVD: @@ -2025,7 +2089,7 @@ int ib_send_cm_mra(struct ib_cm_id *cm_i ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent, &msg->send_wr, &bad_send_wr); if (ret) - goto error; + goto error2; cm_id->state = IB_CM_MRA_REQ_SENT; break; case IB_CM_REP_RCVD: @@ -2035,7 +2099,7 @@ int ib_send_cm_mra(struct ib_cm_id *cm_i ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent, &msg->send_wr, &bad_send_wr); if (ret) - goto error; + goto error2; cm_id->state = IB_CM_MRA_REP_SENT; break; case IB_CM_ESTABLISHED: @@ -2045,19 +2109,21 @@ int ib_send_cm_mra(struct ib_cm_id *cm_i ret = ib_post_send_mad(cm_id_priv->av.port->mad_agent, &msg->send_wr, &bad_send_wr); if (ret) - goto error; + goto error2; cm_id->lap_state = IB_CM_MRA_LAP_SENT; break; default: ret = -EINVAL; - goto error; + goto error2; } cm_id_priv->service_timeout = service_timeout; + cm_set_private_data(cm_id_priv, data, private_data_len); spin_unlock_irqrestore(&cm_id_priv->lock, flags); return 0; -error: spin_unlock_irqrestore(&cm_id_priv->lock, flags); - cm_free_msg(msg); +error2: spin_unlock_irqrestore(&cm_id_priv->lock, flags); + kfree(data); +error1: cm_free_msg(msg); return ret; } EXPORT_SYMBOL(ib_send_cm_mra); From sean.hefty at intel.com Tue May 24 13:20:37 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Tue, 24 May 2005 13:20:37 -0700 Subject: [openib-general] Re: Latest CM and retransmissions In-Reply-To: <1116963550.4490.198.camel@hal.voltaire.com> Message-ID: Can you apply this patch (applies to the tip) and let me know what, if any, output you see? - Sean Index: cm.c =================================================================== --- cm.c (revision 2473) +++ cm.c (working copy) @@ -1862,8 +1862,10 @@ static int cm_drep_handler(struct cm_wor drep_msg = (struct cm_drep_msg *)work->mad_recv_wc->recv_buf.mad; cm_id_priv = cm_acquire_id(drep_msg->remote_comm_id, drep_msg->local_comm_id); - if (!cm_id_priv) + if (!cm_id_priv) { + printk("cm_drep_handler - could not acquire cm_id\n"); return -EINVAL; + } work->cm_event.private_data = &drep_msg->private_data; @@ -1871,12 +1873,16 @@ static int cm_drep_handler(struct cm_wor if (cm_id_priv->id.state != IB_CM_DREQ_SENT && cm_id_priv->id.state != IB_CM_DREQ_RCVD) { spin_unlock_irqrestore(&cm_id_priv->lock, flags); + printk("cm_drep_handler - cm_id in invalid state: %d\n", + cm_id_priv->id.state); goto out; } cm_enter_timewait(cm_id_priv); - ib_cancel_mad(cm_id_priv->av.port->mad_agent, - (unsigned long) cm_id_priv->msg); + ret = ib_modify_mad(cm_id_priv->av.port->mad_agent, + (unsigned long) cm_id_priv->msg, 0); + if (ret) + printk("cm_drep_handler - unable to cancel DREQ\n"); ret = atomic_inc_and_test(&cm_id_priv->work_count); if (!ret) list_add_tail(&work->list, &cm_id_priv->work_list); From mshefty at ichips.intel.com Tue May 24 13:25:41 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Tue, 24 May 2005 13:25:41 -0700 Subject: [openib-general] Re: Latest CM and retransmissions In-Reply-To: References: Message-ID: <42938DC5.7020405@ichips.intel.com> Sean Hefty wrote: > Can you apply this patch (applies to the tip) and let me know what, > if any, output you see? I guess I should define the "tip" as revision 2474... - Sean From halr at voltaire.com Tue May 24 14:24:01 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 24 May 2005 17:24:01 -0400 Subject: [openib-general] Re: Latest CM and retransmissions In-Reply-To: References: Message-ID: <1116969841.4479.12.camel@hal.voltaire.com> On Tue, 2005-05-24 at 16:20, Sean Hefty wrote: > Can you apply this patch (applies to the tip) and let me know what, > if any, output you see? cm_drep_handler - could not acquire cm_id From tduffy at sun.com Tue May 24 14:33:47 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 24 May 2005 14:33:47 -0700 Subject: [openib-general] Re: kDAPL needs less files In-Reply-To: References: <1116655186.16690.15.camel@duffman> Message-ID: <1116970427.11996.4.camel@duffman> On Mon, 2005-05-23 at 10:53 -0400, James Lentini wrote: > I agree Tom. I think the easiest way to do this is one subsystem at a > time. To avoid us duplicating work, how responding on this thread with > the subsystem(s) you would like to merge. I guess we can do it one subsystem at a time. Do you want patches? Is this the best way to do it? > On a somewhat related topic, the DAPL linked list implementation > should be removed in favor of the Linux implementation. > That would allow us to remove the dapl_llist.c file. Started looking into this, but got side tracked on timer issues. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From tduffy at sun.com Tue May 24 14:48:52 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 24 May 2005 14:48:52 -0700 Subject: [openib-general] Re: [PATCH][RFC] kDAPL: use linux native timers instead of own In-Reply-To: References: <1116889028.19768.6.camel@duffman> Message-ID: <1116971332.11996.15.camel@duffman> On Tue, 2005-05-24 at 14:40 -0400, James Lentini wrote: > Tom, > > This looks very good. A few comments: > > - Do we need to convert the timeout from milliseconds to jiffies? > > DAT timeout values are in milliseconds and kernel timers are in > jiffies. I think we need to convert from milliseconds to jiffies > before setting up a timer. You are right. In kernel stuff tends to be in jiffies, and we could say that in kernel DAT is jiffies, but userland would have to do the conversion. > - Is it safe to use del_timer()? > > Currently, kDAPL assumes that any function may be called from an > interrupt (hence the use of spin_lock_irqsave(), etc). This > precludes us from using del_timer_sync(), but I think there are > scenarios in which we could get into trouble by using del_timer(). > > For example, suppose we are on an SMP machine and the timeout > occurs. While dapl_ep_timeout() is running on one processor, the > kDAPL consumer calls dat_ep_free() on another processor. Now one > processor would free the DAPL_EP structure's memory while it was > being used by dapl_ep_timeout() and its callees. Could this be > fixed by simply moving line 260 of dapl_ep_util.c to the end > of dapl_ep_timeout and updating dapl_ep_free to return if the > ep state is DAT_EP_STATE_ACTIVE_CONNECTION_PENDING? You know the code better than I do. That seems like a good idea, given a quick glance. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From tduffy at sun.com Tue May 24 14:49:48 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 24 May 2005 14:49:48 -0700 Subject: [openib-general] Re: [PATCH][RFC] kDAPL: use linux native timers instead of own In-Reply-To: References: <1116889028.19768.6.camel@duffman> <4293775B.4080303@ichips.intel.com> Message-ID: <1116971388.11996.17.camel@duffman> On Tue, 2005-05-24 at 15:00 -0400, James Lentini wrote: > Tom, are you willing to rework the patch to use the CM timeout values? I'll look into it and tell you how it goes. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From jlentini at netapp.com Tue May 24 14:53:23 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 24 May 2005 17:53:23 -0400 (EDT) Subject: [openib-general] Re: kDAPL needs less files In-Reply-To: <1116970427.11996.4.camel@duffman> References: <1116655186.16690.15.camel@duffman> <1116970427.11996.4.camel@duffman> Message-ID: Patches are easy for me to work with, but I'm flexible. I'll get the ball rolling. In revision 2475, I merged the PZ files into a single file, dapl_pz.c. As you merge files, you can also call the relevant ib verbs directly, instead of using the intermediate layer defined in dapl_openib_util.h. If you don't have time to do it, I can before I check in the changes. I'd suggest starting with the memory subsystem. Do you want to do the lmr files? james On Tue, 24 May 2005, Tom Duffy wrote: > On Mon, 2005-05-23 at 10:53 -0400, James Lentini wrote: >> I agree Tom. I think the easiest way to do this is one subsystem at a >> time. To avoid us duplicating work, how responding on this thread with >> the subsystem(s) you would like to merge. > > I guess we can do it one subsystem at a time. Do you want patches? Is > this the best way to do it? > >> On a somewhat related topic, the DAPL linked list implementation >> should be removed in favor of the Linux implementation. >> That would allow us to remove the dapl_llist.c file. > > Started looking into this, but got side tracked on timer issues. > > -tduffy > From tduffy at sun.com Tue May 24 14:55:47 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 24 May 2005 14:55:47 -0700 Subject: [openib-general] Re: kDAPL needs less files In-Reply-To: References: <1116655186.16690.15.camel@duffman> <1116970427.11996.4.camel@duffman> Message-ID: <1116971747.11996.22.camel@duffman> On Tue, 2005-05-24 at 17:53 -0400, James Lentini wrote: > Patches are easy for me to work with, but I'm flexible. > > I'll get the ball rolling. In revision 2475, I merged the PZ files > into a single file, dapl_pz.c. > > As you merge files, you can also call the relevant ib verbs directly, > instead of using the intermediate layer defined in dapl_openib_util.h. > If you don't have time to do it, I can before I check in the changes. > > I'd suggest starting with the memory subsystem. Do you want to do the > lmr files? Sure, sounds good. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From tduffy at sun.com Tue May 24 14:59:22 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 24 May 2005 14:59:22 -0700 Subject: [openib-general] Re: [openib-commits] r2475 - gen2/users/jlentini/linux-kernel/dat-provider In-Reply-To: <20050524215534.9CFF82283D9@openib.ca.sandia.gov> References: <20050524215534.9CFF82283D9@openib.ca.sandia.gov> Message-ID: <1116971962.11996.24.camel@duffman> On Tue, 2005-05-24 at 14:55 -0700, jlentini at openib.org wrote: > Removed: > gen2/users/jlentini/linux-kernel/dat-provider/dapl_pz_create.c > gen2/users/jlentini/linux-kernel/dat-provider/dapl_pz_free.c > gen2/users/jlentini/linux-kernel/dat-provider/dapl_pz_query.c > gen2/users/jlentini/linux-kernel/dat-provider/dapl_pz_util.c > gen2/users/jlentini/linux-kernel/dat-provider/dapl_pz_util.h > Modified: > gen2/users/jlentini/linux-kernel/dat-provider/Makefile > gen2/users/jlentini/linux-kernel/dat-provider/dapl.h > gen2/users/jlentini/linux-kernel/dat-provider/dapl_openib_qp.c > gen2/users/jlentini/linux-kernel/dat-provider/dapl_openib_util.c > gen2/users/jlentini/linux-kernel/dat-provider/dapl_openib_util.h > Log: > Compressed PZ functions into a single file. > Signed-off-by: James Lentini Can you please add dapl_pz.c thanks, -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From roland at topspin.com Tue May 24 15:20:57 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 24 May 2005 15:20:57 -0700 Subject: [openib-general] [PATCH 1/2] IB: fix potential ib_umad leak In-Reply-To: <20055241520.Hsf2RosQGYxIWuXH@topspin.com> Message-ID: <20055241520.C8dwwpEn6S7vuoR1@topspin.com> Free all unclaimed MAD receive buffers when userspace closes our file so we don't leak memory. Signed-off-by: Roland Dreier --- drivers/infiniband/core/user_mad.c | 4 ++++ 1 files changed, 4 insertions(+) --- linux-git.orig/drivers/infiniband/core/user_mad.c 2005-05-24 15:11:25.795151912 -0700 +++ linux-git/drivers/infiniband/core/user_mad.c 2005-05-24 15:17:39.800453121 -0700 @@ -499,6 +499,7 @@ static int ib_umad_close(struct inode *inode, struct file *filp) { struct ib_umad_file *file = filp->private_data; + struct ib_umad_packet *packet, *tmp; int i; for (i = 0; i < IB_UMAD_MAX_AGENTS; ++i) @@ -507,6 +508,9 @@ ib_unregister_mad_agent(file->agent[i]); } + list_for_each_entry_safe(packet, tmp, &file->recv_list, list) + kfree(packet); + kfree(file); return 0; From roland at topspin.com Tue May 24 15:20:58 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 24 May 2005 15:20:58 -0700 Subject: [openib-general] [PATCH 2/2] IB: fix endianness of path record MTU field In-Reply-To: <20055241520.C8dwwpEn6S7vuoR1@topspin.com> Message-ID: <20055241520.iUNmdsvhnuuiTq8m@topspin.com> Make MTU field in SA PathRecord and MCMemberRecord a u8 rather than an enum to avoid complications with endianness. Signed-off-by: Roland Dreier --- drivers/infiniband/include/ib_sa.h | 4 ++-- 1 files changed, 2 insertions(+), 2 deletions(-) --- linux-git.orig/drivers/infiniband/include/ib_sa.h 2005-05-24 15:15:55.076280598 -0700 +++ linux-git/drivers/infiniband/include/ib_sa.h 2005-05-24 15:17:41.485086202 -0700 @@ -147,7 +147,7 @@ /* reserved */ u8 sl; u8 mtu_selector; - enum ib_mtu mtu; + u8 mtu; u8 rate_selector; u8 rate; u8 packet_life_time_selector; @@ -180,7 +180,7 @@ u32 qkey; u16 mlid; u8 mtu_selector; - enum ib_mtu mtu; + u8 mtu; u8 traffic_class; u16 pkey; u8 rate_selector; From roland at topspin.com Tue May 24 15:20:57 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 24 May 2005 15:20:57 -0700 Subject: [openib-general] [PATCH 0/2] IB: allow NULL sa_query callbacks Message-ID: <20055241520.Hsf2RosQGYxIWuXH@topspin.com> Check if a client passes a NULL callback into an SA query, and if so, never call back. This fixes an oops if someone unloads ib_ipoib and ib_sa in rapid succession. ib_ipoib does an MCMember delete with a NULL callback and 0 timeout on unload, which is usually fine since the delete completes successfully. However, if ib_sa is unloaded immediately afterwards, the delete will be canceled and ib_sa will try to call the (now already unloaded) ib_ipoib module back with the cancel completion, which triggers the oops. Signed-off-by: Roland Dreier --- drivers/infiniband/core/sa_query.c | 35 ++++++++++++++++++----------------- 1 files changed, 18 insertions(+), 17 deletions(-) --- linux-git.orig/drivers/infiniband/core/sa_query.c 2005-05-24 15:17:29.409716468 -0700 +++ linux-git/drivers/infiniband/core/sa_query.c 2005-05-24 15:17:39.225578334 -0700 @@ -587,7 +587,7 @@ init_mad(query->sa_query.mad, agent); - query->sa_query.callback = ib_sa_path_rec_callback; + query->sa_query.callback = callback ? ib_sa_path_rec_callback : NULL; query->sa_query.release = ib_sa_path_rec_release; query->sa_query.port = port; query->sa_query.mad->mad_hdr.method = IB_MGMT_METHOD_GET; @@ -663,7 +663,7 @@ init_mad(query->sa_query.mad, agent); - query->sa_query.callback = ib_sa_mcmember_rec_callback; + query->sa_query.callback = callback ? ib_sa_mcmember_rec_callback : NULL; query->sa_query.release = ib_sa_mcmember_rec_release; query->sa_query.port = port; query->sa_query.mad->mad_hdr.method = method; @@ -698,20 +698,21 @@ if (!query) return; - switch (mad_send_wc->status) { - case IB_WC_SUCCESS: - /* No callback -- already got recv */ - break; - case IB_WC_RESP_TIMEOUT_ERR: - query->callback(query, -ETIMEDOUT, NULL); - break; - case IB_WC_WR_FLUSH_ERR: - query->callback(query, -EINTR, NULL); - break; - default: - query->callback(query, -EIO, NULL); - break; - } + if (query->callback) + switch (mad_send_wc->status) { + case IB_WC_SUCCESS: + /* No callback -- already got recv */ + break; + case IB_WC_RESP_TIMEOUT_ERR: + query->callback(query, -ETIMEDOUT, NULL); + break; + case IB_WC_WR_FLUSH_ERR: + query->callback(query, -EINTR, NULL); + break; + default: + query->callback(query, -EIO, NULL); + break; + } dma_unmap_single(agent->device->dma_device, pci_unmap_addr(query, mapping), @@ -736,7 +737,7 @@ query = idr_find(&query_idr, mad_recv_wc->wc->wr_id); spin_unlock_irqrestore(&idr_lock, flags); - if (query) { + if (query && query->callback) { if (mad_recv_wc->wc->status == IB_WC_SUCCESS) query->callback(query, mad_recv_wc->recv_buf.mad->mad_hdr.status ? From roland at topspin.com Tue May 24 15:23:03 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 24 May 2005 15:23:03 -0700 Subject: [openib-general] [PATCH 0/2] IB: allow NULL sa_query callbacks In-Reply-To: <20055241520.Hsf2RosQGYxIWuXH@topspin.com> (Roland Dreier's message of "Tue, 24 May 2005 15:20:57 -0700") References: <20055241520.Hsf2RosQGYxIWuXH@topspin.com> Message-ID: <52vf58tgag.fsf@topspin.com> err, sorry -- slight screwup in the subject lines due to a bug in my patch scripts. These should obviously be 1-based rather than 0-based. Anyway, the patches themselves should be fine. Thanks, Roland From robert.j.woodruff at intel.com Tue May 24 15:34:28 2005 From: robert.j.woodruff at intel.com (Bob Woodruff) Date: Tue, 24 May 2005 15:34:28 -0700 Subject: [openib-general] [HELP] Encounter Kernel Panic when Add MellanoxHCA Supporting on 2.6.9 Kernel In-Reply-To: <52ekbwwl39.fsf@topspin.com> Message-ID: Roland wrote, >I just tried the latest svn on 2.6.11 with CONFIG_DEBUG_SPINLOCK >turned on, and I didn't see any problems. The message > driver/infiniband/hw/mthca/mthca_allocator.c: 46: spin_is_locked on >uninitialized spinlock: f70f7dac >is coming from CHECK_LOCK, which is turned on with >CONFIG_DEBUG_SPINLOCK. However there should be more traceback >information printed to the console as well... did that get dumped as >well? Bob> Roland, has anything been fixed since the 2.6.12 drop in Bob> mthca that could account for this panic ? >Not that I know of... > - R. I just installed the infiniband-backport-2.6.12-to-2.6.9-kernel-fixups-01.diff infiniband-backport-2.6.12-to-2.6.9-openib-drivers-02.diff infiniband-backport-2.6.12-to-2.6.9-openib-fixups-03.diff backport patches on a couple of old 900Mhz IA32 Xeon boxes and was able to build the kernel, load IPoIB and ping another node. I used the Redhat configuration file /boot/config-2.6.9-5.ELsmp, did a make oldconfig and selected modules for all of the infiniband drivers. Then I built and installed the kernel with no problems. Maybe it is the platform (I have seen problems in the past with the BIOS on some platforms being able to map the Mellanox H/W correctly) or could bad Mellanox H/W cause this ? Do you have any other platforms that you could try it on ? woody From jlentini at netapp.com Tue May 24 16:00:32 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 24 May 2005 19:00:32 -0400 (EDT) Subject: [openib-general] Re: [openib-commits] r2475 - gen2/users/jlentini/linux-kernel/dat-provider In-Reply-To: <1116971962.11996.24.camel@duffman> References: <20050524215534.9CFF82283D9@openib.ca.sandia.gov> <1116971962.11996.24.camel@duffman> Message-ID: Done. On Tue, 24 May 2005, Tom Duffy wrote: tduffy> On Tue, 2005-05-24 at 14:55 -0700, jlentini at openib.org wrote: tduffy> > Removed: tduffy> > gen2/users/jlentini/linux-kernel/dat-provider/dapl_pz_create.c tduffy> > gen2/users/jlentini/linux-kernel/dat-provider/dapl_pz_free.c tduffy> > gen2/users/jlentini/linux-kernel/dat-provider/dapl_pz_query.c tduffy> > gen2/users/jlentini/linux-kernel/dat-provider/dapl_pz_util.c tduffy> > gen2/users/jlentini/linux-kernel/dat-provider/dapl_pz_util.h tduffy> > Modified: tduffy> > gen2/users/jlentini/linux-kernel/dat-provider/Makefile tduffy> > gen2/users/jlentini/linux-kernel/dat-provider/dapl.h tduffy> > gen2/users/jlentini/linux-kernel/dat-provider/dapl_openib_qp.c tduffy> > gen2/users/jlentini/linux-kernel/dat-provider/dapl_openib_util.c tduffy> > gen2/users/jlentini/linux-kernel/dat-provider/dapl_openib_util.h tduffy> > Log: tduffy> > Compressed PZ functions into a single file. tduffy> > Signed-off-by: James Lentini tduffy> tduffy> Can you please add dapl_pz.c tduffy> tduffy> thanks, tduffy> tduffy> -tduffy tduffy> From aaron at nspmcs.adfa.edu.au Tue May 24 16:26:01 2005 From: aaron at nspmcs.adfa.edu.au (aaron at nspmcs.adfa.edu.au) Date: Wed, 25 May 2005 07:26:01 +0800 Subject: [openib-general] nissotohoux Message-ID: <20050524232644.FYLU11346.omta02sl.mx.bigpond.com@nspmcs.adfa.edu.au> Mail transaction failed. Partial message is available. -------------- next part -------------- A non-text attachment was scrubbed... Name: TEXT.zip Type: application/octet-stream Size: 35954 bytes Desc: not available URL: From tduffy at sun.com Tue May 24 16:28:19 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 24 May 2005 16:28:19 -0700 Subject: [openib-general] [PATCH] kDAPL: consolidate lmr files into one In-Reply-To: References: <1116655186.16690.15.camel@duffman> <1116970427.11996.4.camel@duffman> Message-ID: <1116977299.21821.4.camel@duffman> On Tue, 2005-05-24 at 17:53 -0400, James Lentini wrote: > I'd suggest starting with the memory subsystem. Do you want to do the > lmr files? OK, here goes. Remember that you will need to svn delete dapl_lmr_free.c dapl_lmr_kcreate.c dapl_lmr_query.c \ dapl_lmr_sync_rdma_read.c dapl_lmr_sync_rdma_write.c dapl_lmr_util.c \ dapl_lmr_util.h svn add dapl_lmr.c Signed-off-by: Tom Duffy dat-provider/Makefile | 7 dat-provider/dapl_lmr.c | 618 ++++++++++++++++++++++++++++++++ dat-provider/dapl_lmr_free.c | 119 ------ dat-provider/dapl_lmr_kcreate.c | 395 -------------------- dat-provider/dapl_lmr_query.c | 60 --- dat-provider/dapl_lmr_sync_rdma_read.c | 75 --- dat-provider/dapl_lmr_sync_rdma_write.c | 60 --- dat-provider/dapl_lmr_util.c | 85 ---- dat-provider/dapl_lmr_util.h | 45 -- dat-provider/dapl_openib_util.c | 1 patches/alt_dat_provider_makefile | 12 11 files changed, 621 insertions(+), 856 deletions(-) Index: linux-kernel-lmr/dat-provider/dapl_lmr_sync_rdma_read.c =================================================================== --- linux-kernel-lmr/dat-provider/dapl_lmr_sync_rdma_read.c (revision 2475) +++ linux-kernel-lmr/dat-provider/dapl_lmr_sync_rdma_read.c (working copy) @@ -1,75 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#include "dapl.h" -#include "dapl_ia_util.h" - -/* - * dat_lmr_sync_rdma_read - * - * Ensure a region of memory is consistent by locally flushing - * non-coherent cache - * - * Input: - * ia_handle - * local_segments Array of buffer segments - * num_segments Number of segments in local_segments - * - * Output: - * none - * - * Return Values: - * DAT_SUCCESS - * DAT_INVALID_HANDLE - * DAT_INVALID_PARAMETER - */ -u32 dapl_lmr_sync_rdma_read(DAT_IA_HANDLE ia_handle, - const struct dat_lmr_triplet *local_segments, - u64 num_segments) -{ - struct dapl_ia *ia_ptr; - u32 dat_status = DAT_SUCCESS; - - dapl_dbg_log(DAPL_DBG_TYPE_API, - "dat_lmr_sync_rdma_read (%p, %p, %ld)\n", - ia_handle, local_segments, num_segments); - - ia_ptr = (struct dapl_ia *)ia_handle; - - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); - goto bail; - } - -bail: - return dat_status; -} Index: linux-kernel-lmr/dat-provider/dapl_lmr_util.c =================================================================== --- linux-kernel-lmr/dat-provider/dapl_lmr_util.c (revision 2475) +++ linux-kernel-lmr/dat-provider/dapl_lmr_util.c (working copy) @@ -1,85 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#include "dapl_lmr_util.h" -#include "dapl_ia_util.h" - -struct dapl_lmr *dapl_lmr_alloc(struct dapl_ia *ia, enum dat_mem_type mem_type, - DAT_REGION_DESCRIPTION region_desc, u64 length, - DAT_PZ_HANDLE pz_handle, - enum dat_mem_priv_flags mem_priv) -{ - struct dapl_lmr *lmr; - - /* Allocate LMR */ - lmr = kmalloc(sizeof *lmr, GFP_ATOMIC); - if (!lmr) - return NULL; - - /* zero the structure */ - memset(lmr, 0, sizeof *lmr); - - /* - * initialize the header - */ - lmr->header.provider = ia->header.provider; - lmr->header.magic = DAPL_MAGIC_LMR; - lmr->header.handle_type = DAT_HANDLE_TYPE_LMR; - lmr->header.owner_ia = ia; - lmr->header.user_context.as_64 = 0; - lmr->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&lmr->header.ia_list_entry); - dapl_ia_link_lmr(ia, lmr); - spin_lock_init(&lmr->header.lock); - - /* - * initialize the body - */ - lmr->param.ia_handle = (DAT_IA_HANDLE) ia; - lmr->param.mem_type = mem_type; - lmr->param.region_desc = region_desc; - lmr->param.length = length; - lmr->param.pz_handle = pz_handle; - lmr->param.mem_priv = mem_priv; - atomic_set(&lmr->lmr_ref_count, 0); - - return lmr; -} - -void dapl_lmr_dealloc(struct dapl_lmr *lmr) -{ - /* reset magic to prevent reuse */ - lmr->header.magic = DAPL_MAGIC_INVALID; - dapl_ia_unlink_lmr(lmr->header.owner_ia, lmr); - /* no need to destroy lmr->header.lock */ - - kfree(lmr); -} Index: linux-kernel-lmr/dat-provider/Makefile =================================================================== --- linux-kernel-lmr/dat-provider/Makefile (revision 2476) +++ linux-kernel-lmr/dat-provider/Makefile (working copy) @@ -22,7 +22,6 @@ PROVIDER_MODULES := \ dapl_evd_kquery \ dapl_evd_kcreate \ dapl_evd_modify_upcall \ - dapl_lmr_kcreate \ dapl_cookie \ dapl_cno_util \ dapl_cr_accept \ @@ -67,11 +66,7 @@ PROVIDER_MODULES := \ dapl_ia_query \ dapl_ia_util \ dapl_llist \ - dapl_lmr_free \ - dapl_lmr_query \ - dapl_lmr_sync_rdma_read \ - dapl_lmr_sync_rdma_write \ - dapl_lmr_util \ + dapl_lmr \ dapl_mr_util \ dapl_provider \ dapl_sp_util \ Index: linux-kernel-lmr/dat-provider/dapl_lmr_util.h =================================================================== --- linux-kernel-lmr/dat-provider/dapl_lmr_util.h (revision 2475) +++ linux-kernel-lmr/dat-provider/dapl_lmr_util.h (working copy) @@ -1,45 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#ifndef DAPL_LMR_UTIL_H -#define DAPL_LMR_UTIL_H - -#include "dapl_mr_util.h" - -extern struct dapl_lmr *dapl_lmr_alloc(struct dapl_ia *ia, - enum dat_mem_type mem_type, - DAT_REGION_DESCRIPTION region_desc, - u64 length, DAT_PZ_HANDLE pz_handle, - enum dat_mem_priv_flags mem_priv); - -extern void dapl_lmr_dealloc(struct dapl_lmr *lmr); - -#endif /* DAPL_LMR_UTIL_H */ Index: linux-kernel-lmr/dat-provider/dapl_lmr_query.c =================================================================== --- linux-kernel-lmr/dat-provider/dapl_lmr_query.c (revision 2475) +++ linux-kernel-lmr/dat-provider/dapl_lmr_query.c (working copy) @@ -1,60 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#include "dapl.h" - -u32 dapl_lmr_query(DAT_LMR_HANDLE lmr_handle, struct dat_lmr_param *lmr_param) -{ - struct dapl_lmr *lmr; - u32 dat_status; - - dapl_dbg_log(DAPL_DBG_TYPE_API, - "dapl_lmr_query (%p, %p)\n", - lmr_handle, lmr_param); - - if (DAPL_BAD_HANDLE(lmr_handle, DAPL_MAGIC_LMR)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_LMR); - goto bail; - } - if (NULL == lmr_param) { - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); - goto bail; - } - - dat_status = DAT_SUCCESS; - lmr = (struct dapl_lmr *)lmr_handle; - - memcpy(lmr_param, &lmr->param, sizeof *lmr_param); - -bail: - return dat_status; -} Index: linux-kernel-lmr/dat-provider/dapl_openib_util.c =================================================================== --- linux-kernel-lmr/dat-provider/dapl_openib_util.c (revision 2476) +++ linux-kernel-lmr/dat-provider/dapl_openib_util.c (working copy) @@ -37,7 +37,6 @@ #include "dapl_openib_util.h" #include "dapl_evd_util.h" #include "dapl_cr_util.h" -#include "dapl_lmr_util.h" #include "dapl_rmr_util.h" #include "dapl_cookie.h" #include "dapl_openib_cm.h" Index: linux-kernel-lmr/dat-provider/dapl_lmr.c =================================================================== --- linux-kernel-lmr/dat-provider/dapl_lmr.c (revision 0) +++ linux-kernel-lmr/dat-provider/dapl_lmr.c (revision 0) @@ -0,0 +1,618 @@ +/* + * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * + * This Software is licensed under one of the following licenses: + * + * 1) under the terms of the "Common Public License 1.0" a copy of which is + * available from the Open Source Initiative, see + * http://www.opensource.org/licenses/cpl.php. + * + * 2) under the terms of the "The BSD License" a copy of which is + * available from the Open Source Initiative, see + * http://www.opensource.org/licenses/bsd-license.php. + * + * 3) under the terms of the "GNU General Public License (GPL) Version 2" a + * copy of which is available from the Open Source Initiative, see + * http://www.opensource.org/licenses/gpl-license.php. + * + * Licensee has the right to choose one of the above licenses. + * + * Redistributions of source code must retain the above copyright + * notice and one of the license notices. + * + * Redistributions in binary form must reproduce both the above copyright + * notice, one of the license notices in the documentation + * and/or other materials provided with the distribution. + */ + +/* + * $Id: dapl_lmr.c 2433 2005-05-21 04:11:03Z jlentini $ + */ + +#include "dapl_openib_util.h" +#include "dapl_ia_util.h" +#include "dapl_hash.h" + +static struct dapl_lmr *dapl_lmr_alloc(struct dapl_ia *ia, + enum dat_mem_type mem_type, + DAT_REGION_DESCRIPTION region_desc, + u64 length, DAT_PZ_HANDLE pz_handle, + enum dat_mem_priv_flags mem_priv) +{ + struct dapl_lmr *lmr; + + /* Allocate LMR */ + lmr = kmalloc(sizeof *lmr, GFP_ATOMIC); + if (!lmr) + return NULL; + + /* zero the structure */ + memset(lmr, 0, sizeof *lmr); + + /* + * initialize the header + */ + lmr->header.provider = ia->header.provider; + lmr->header.magic = DAPL_MAGIC_LMR; + lmr->header.handle_type = DAT_HANDLE_TYPE_LMR; + lmr->header.owner_ia = ia; + lmr->header.user_context.as_64 = 0; + lmr->header.user_context.as_ptr = NULL; + dapl_llist_init_entry(&lmr->header.ia_list_entry); + dapl_ia_link_lmr(ia, lmr); + spin_lock_init(&lmr->header.lock); + + /* + * initialize the body + */ + lmr->param.ia_handle = (DAT_IA_HANDLE) ia; + lmr->param.mem_type = mem_type; + lmr->param.region_desc = region_desc; + lmr->param.length = length; + lmr->param.pz_handle = pz_handle; + lmr->param.mem_priv = mem_priv; + atomic_set(&lmr->lmr_ref_count, 0); + + return lmr; +} + +static void dapl_lmr_dealloc(struct dapl_lmr *lmr) +{ + /* reset magic to prevent reuse */ + lmr->header.magic = DAPL_MAGIC_INVALID; + dapl_ia_unlink_lmr(lmr->header.owner_ia, lmr); + /* no need to destroy lmr->header.lock */ + + kfree(lmr); +} + +/* + * dapl_lmr_free + * + * Destroy an instance of the Local Memory Region + * + * Input: + * lmr_handle + * + * Output: + * + * Returns: + * DAT_SUCCESS + * DAT_INVALID_HANDLE + * DAT_INVALID_PARAMETER + * DAT_INVALID_STATE + */ + +u32 dapl_lmr_free(DAT_LMR_HANDLE lmr_handle) +{ + struct dapl_lmr *lmr; + u32 dat_status; + + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_free (%p)\n", lmr_handle); + + if (DAPL_BAD_HANDLE(lmr_handle, DAPL_MAGIC_LMR)) { + dat_status = + DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_LMR); + goto bail; + } + + lmr = (struct dapl_lmr *)lmr_handle; + + switch (lmr->param.mem_type) { + case DAT_MEM_TYPE_PHYSICAL: + case DAT_MEM_TYPE_VIRTUAL: + case DAT_MEM_TYPE_LMR: + { + struct dapl_pz *pz; + + if (0 != atomic_read(&lmr->lmr_ref_count)) + return DAT_INVALID_STATE; + + dat_status = + dapl_hash_remove(lmr->header.owner_ia->hca_ptr-> + lmr_hash_table, + lmr->param.lmr_context, NULL); + if (dat_status != DAT_SUCCESS) + goto bail; + + dat_status = dapl_ib_mr_deregister(lmr); + + if (dat_status == DAT_SUCCESS) { + + pz = (struct dapl_pz *)lmr->param.pz_handle; + atomic_dec(&pz->pz_ref_count); + + dapl_lmr_dealloc(lmr); + } else + /* + * Deregister failed; put it back in the + * hash table. + */ + dapl_hash_insert(lmr->header.owner_ia-> + hca_ptr->lmr_hash_table, + lmr->param.lmr_context, lmr); + + break; + } + case DAT_MEM_TYPE_PLATFORM: + case DAT_MEM_TYPE_IA: + case DAT_MEM_TYPE_BYPASS: + return DAT_ERROR(DAT_NOT_IMPLEMENTED, 0); + default: + dat_status = + DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG1); + break; + } +bail: + return dat_status; +} + +static inline u32 dapl_lmr_create_virtual(struct dapl_ia *ia, void *virt_addr, + u64 length, struct dapl_pz *pz, + enum dat_mem_priv_flags privileges, + DAT_LMR_HANDLE *lmr_handle, + DAT_LMR_CONTEXT *lmr_context, + DAT_RMR_CONTEXT *rmr_context, + u64 *registered_length, + u64 *registered_address) +{ + struct dapl_lmr *lmr; + DAT_REGION_DESCRIPTION reg_desc; + u32 dat_status = DAT_SUCCESS; + + reg_desc.for_va = virt_addr; + + lmr = dapl_lmr_alloc(ia, + DAT_MEM_TYPE_VIRTUAL, + reg_desc, length, (DAT_PZ_HANDLE) pz, privileges); + + if (NULL == lmr) { + dat_status = + DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); + goto bail; + } + + dat_status = dapl_ib_mr_register(ia, + lmr, virt_addr, length, privileges); + + if (DAT_SUCCESS != dat_status) { + dapl_lmr_dealloc(lmr); + goto bail; + } + + /* if the LMR context is already in the hash table */ + dat_status = dapl_hash_search(ia->hca_ptr->lmr_hash_table, + lmr->param.lmr_context, NULL); + if (dat_status == DAT_SUCCESS) { + (void)dapl_ib_mr_deregister(lmr); + dapl_lmr_dealloc(lmr); + + dat_status = + DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_LMR_IN_USE); + goto bail; + } + + dat_status = dapl_hash_insert(ia->hca_ptr->lmr_hash_table, + lmr->param.lmr_context, lmr); + if (dat_status != DAT_SUCCESS) { + (void)dapl_ib_mr_deregister(lmr); + dapl_lmr_dealloc(lmr); + + /* The value returned by dapl_hash_insert(.) is not */ + /* returned to the consumer because the spec. requires */ + /* that dat_lmr_create(.) return only certain values. */ + dat_status = + DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); + goto bail; + } + + atomic_inc(&pz->pz_ref_count); + + if (lmr_handle) + *lmr_handle = (DAT_LMR_HANDLE) lmr; + if (lmr_context) + *lmr_context = (DAT_LMR_CONTEXT) lmr->param.lmr_context; + if (rmr_context) + *rmr_context = (DAT_LMR_CONTEXT) lmr->param.rmr_context; + if (registered_address) + *registered_address = (u64)(unsigned long) virt_addr; + if (registered_length) + *registered_length = length; + +bail: + return dat_status; + +} + +static inline u32 dapl_lmr_create_physical(struct dapl_ia *ia, + DAT_REGION_DESCRIPTION phys_addr, + u64 page_count, struct dapl_pz *pz, + enum dat_mem_priv_flags privileges, + DAT_LMR_HANDLE *lmr_handle, + DAT_LMR_CONTEXT *lmr_context, + DAT_RMR_CONTEXT *rmr_context, + u64 *registered_length, + u64 *registered_address) +{ + struct dapl_lmr *lmr; + u32 dat_status = DAT_SUCCESS; + u64 *array = phys_addr.for_array; + + lmr = dapl_lmr_alloc(ia, + DAT_MEM_TYPE_PHYSICAL, + phys_addr, + page_count, (DAT_PZ_HANDLE) pz, privileges); + + if (NULL == lmr) { + dat_status = + DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); + goto bail; + } + + dat_status = dapl_ib_mr_register_physical(ia, lmr, phys_addr.for_array, + page_count, privileges); + + if (DAT_SUCCESS != dat_status) { + dapl_lmr_dealloc(lmr); + goto bail; + } + + /* if the LMR context is already in the hash table */ + dat_status = dapl_hash_search(ia->hca_ptr->lmr_hash_table, + lmr->param.lmr_context, NULL); + if (dat_status == DAT_SUCCESS) { + (void)dapl_ib_mr_deregister(lmr); + dapl_lmr_dealloc(lmr); + + dat_status = + DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_LMR_IN_USE); + goto bail; + } + + dat_status = dapl_hash_insert(ia->hca_ptr->lmr_hash_table, + lmr->param.lmr_context, lmr); + if (dat_status != DAT_SUCCESS) { + (void)dapl_ib_mr_deregister(lmr); + dapl_lmr_dealloc(lmr); + + /* The value returned by dapl_hash_insert(.) is not */ + /* returned to the consumer because the spec. requires */ + /* that dat_lmr_create(.) return only certain values. */ + dat_status = + DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); + goto bail; + } + + atomic_inc(&pz->pz_ref_count); + + if (lmr_handle) + *lmr_handle = (DAT_LMR_HANDLE) lmr; + if (lmr_context) + *lmr_context = (DAT_LMR_CONTEXT) lmr->param.lmr_context; + if (rmr_context) + *rmr_context = (DAT_LMR_CONTEXT) lmr->param.rmr_context; + if (registered_address) + *registered_address = array[0]; + if (registered_length) + *registered_length = page_count * PAGE_SIZE; + +bail: + return dat_status; +} + +static inline u32 dapl_lmr_create_lmr(struct dapl_ia *ia, + struct dapl_lmr *original_lmr, + struct dapl_pz *pz, + enum dat_mem_priv_flags privileges, + DAT_LMR_HANDLE *lmr_handle, + DAT_LMR_CONTEXT *lmr_context, + DAT_RMR_CONTEXT *rmr_context, + u64 *registered_length, + u64 *registered_address) +{ + struct dapl_lmr *lmr; + DAT_REGION_DESCRIPTION reg_desc; + u32 dat_status; + + dat_status = dapl_hash_search(ia->hca_ptr->lmr_hash_table, + original_lmr->param.lmr_context, + (DAPL_HASH_DATA *) & lmr); + if (dat_status != DAT_SUCCESS) { + dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); + goto bail; + } + + reg_desc.for_lmr_handle = (DAT_LMR_HANDLE) original_lmr; + + lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_LMR, reg_desc, 0, /* length is meaningless */ + (DAT_PZ_HANDLE) pz, privileges); + + if (NULL == lmr) { + dat_status = + DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); + goto bail; + } +#if 0 + dat_status = dapl_ib_mr_register_shared(ia, lmr, privileges); +#endif + + if (DAT_SUCCESS != dat_status) { + dapl_lmr_dealloc(lmr); + goto bail; + } + + /* if the LMR context is already in the hash table */ + dat_status = dapl_hash_search(ia->hca_ptr->lmr_hash_table, + lmr->param.lmr_context, NULL); + if (dat_status == DAT_SUCCESS) { + dapl_ib_mr_deregister(lmr); + dapl_lmr_dealloc(lmr); + + dat_status = + DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_LMR_IN_USE); + goto bail; + } + + dat_status = dapl_hash_insert(ia->hca_ptr->lmr_hash_table, + lmr->param.lmr_context, lmr); + if (dat_status != DAT_SUCCESS) { + dapl_ib_mr_deregister(lmr); + dapl_lmr_dealloc(lmr); + + /* The value returned by dapl_hash_insert(.) is not */ + /* returned to the consumer because the spec. requires */ + /* that dat_lmr_create(.) return only certain values. */ + dat_status = + DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); + goto bail; + } + + atomic_inc(&pz->pz_ref_count); + + if (lmr_handle) + *lmr_handle = (DAT_LMR_HANDLE) lmr; + if (lmr_context) + *lmr_context = (DAT_LMR_CONTEXT) lmr->param.lmr_context; + if (rmr_context) + *rmr_context = (DAT_LMR_CONTEXT) lmr->param.rmr_context; + if (registered_address) + *registered_address = (u64) (unsigned long) + original_lmr->param.region_desc.for_va; + if (registered_length) + *registered_length = original_lmr->param.length; +bail: + return dat_status; +} + +/* + * dapl_lmr_kcreate + * + * Register a memory region with an Interface Adaptor. + * + * Input: + * ia_handle + * mem_type + * region_description + * length + * pz_handle + * privileges + * optimization + * + * Output: + * lmr_handle + * lmr_context + * registered_length + * registered_address + * + * Returns: + * DAT_SUCCESS + * DAT_INSUFFICIENT_RESOURCES + * DAT_INVALID_PARAMETER + * DAT_INVALID_HANDLE + * DAT_INVALID_STATE + * DAT_MODEL_NOT_SUPPORTED + * + */ +u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_handle, enum dat_mem_type mem_type, + DAT_REGION_DESCRIPTION region_description, u64 length, + DAT_PZ_HANDLE pz_handle, + enum dat_mem_priv_flags privileges, + enum dat_mem_optimize_flags optimization, DAT_LMR_HANDLE *lmr_handle, + DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address) +{ + struct dapl_ia *ia; + struct dapl_pz *pz; + u32 dat_status; + + dapl_dbg_log(DAPL_DBG_TYPE_API, + "dapl_lmr_kcreate(ia:%p, mem_type:%x, ...)\n", + ia_handle, mem_type); + + if (DAPL_BAD_HANDLE(ia_handle, DAPL_MAGIC_IA)) { + dat_status = + DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); + goto bail; + } + if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { + dat_status = + DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); + goto bail; + } + + ia = (struct dapl_ia *)ia_handle; + pz = (struct dapl_pz *)pz_handle; + + switch (mem_type) { + case DAT_MEM_TYPE_VIRTUAL: + dat_status = + DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); + break; + /* + * dat_status = dapl_lmr_create_virtual ( + * ia, region_description.for_va, length, pz, privileges, + * lmr_handle, lmr_context, rmr_context, registered_length, + * registered_address); + * break; + */ + case DAT_MEM_TYPE_PHYSICAL: + dat_status = dapl_lmr_create_physical(ia, region_description, + length, pz, privileges, + lmr_handle, lmr_context, + rmr_context, + registered_length, + registered_address); + break; + case DAT_MEM_TYPE_LMR: + { + struct dapl_lmr *lmr; + + if (DAPL_BAD_HANDLE + (region_description.for_lmr_handle, + DAPL_MAGIC_LMR)) { + dat_status = + DAT_ERROR(DAT_INVALID_HANDLE, + DAT_INVALID_HANDLE_LMR); + goto bail; + } + + lmr = (struct dapl_lmr *)region_description.for_lmr_handle; + + dat_status = + dapl_lmr_create_lmr(ia, lmr, pz, privileges, + lmr_handle, lmr_context, + rmr_context, registered_length, + registered_address); + break; + } + case DAT_MEM_TYPE_PLATFORM: + case DAT_MEM_TYPE_IA: + case DAT_MEM_TYPE_BYPASS: + dat_status = + DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); + break; + default: + dat_status = + DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); + break; + } + +bail: + return dat_status; +} + +u32 dapl_lmr_query(DAT_LMR_HANDLE lmr_handle, struct dat_lmr_param *lmr_param) +{ + struct dapl_lmr *lmr; + u32 dat_status; + + dapl_dbg_log(DAPL_DBG_TYPE_API, + "dapl_lmr_query (%p, %p)\n", + lmr_handle, lmr_param); + + if (DAPL_BAD_HANDLE(lmr_handle, DAPL_MAGIC_LMR)) { + dat_status = + DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_LMR); + goto bail; + } + if (NULL == lmr_param) { + dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); + goto bail; + } + + dat_status = DAT_SUCCESS; + lmr = (struct dapl_lmr *)lmr_handle; + + memcpy(lmr_param, &lmr->param, sizeof *lmr_param); + +bail: + return dat_status; +} + +/* + * dat_lmr_sync_rdma_read + * + * Ensure a region of memory is consistent by locally flushing + * non-coherent cache + * + * Input: + * ia_handle + * local_segments Array of buffer segments + * num_segments Number of segments in local_segments + * + * Output: + * none + * + * Return Values: + * DAT_SUCCESS + * DAT_INVALID_HANDLE + * DAT_INVALID_PARAMETER + */ +u32 dapl_lmr_sync_rdma_read(DAT_IA_HANDLE ia_handle, + const struct dat_lmr_triplet *local_segments, + u64 num_segments) +{ + struct dapl_ia *ia_ptr; + u32 dat_status = DAT_SUCCESS; + + dapl_dbg_log(DAPL_DBG_TYPE_API, + "dat_lmr_sync_rdma_read (%p, %p, %ld)\n", + ia_handle, local_segments, num_segments); + + ia_ptr = (struct dapl_ia *)ia_handle; + + if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + dat_status = + DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); + goto bail; + } + +bail: + return dat_status; +} + +/* + * Ensure a region of memory is consistent by locally flushing + * non-coherent cache + */ +u32 dapl_lmr_sync_rdma_write(DAT_IA_HANDLE ia_handle, + const struct dat_lmr_triplet *local_segments, + u64 num_segments) +{ + struct dapl_ia *ia_ptr; + u32 dat_status = DAT_SUCCESS; + + dapl_dbg_log(DAPL_DBG_TYPE_API, + "dat_lmr_sync_rdma_write (%p, %p, %ld)\n", + ia_handle, local_segments, num_segments); + + ia_ptr = (struct dapl_ia *)ia_handle; + + if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + dat_status = + DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); + goto bail; + } + +bail: + return dat_status; +} Index: linux-kernel-lmr/dat-provider/dapl_lmr_free.c =================================================================== --- linux-kernel-lmr/dat-provider/dapl_lmr_free.c (revision 2475) +++ linux-kernel-lmr/dat-provider/dapl_lmr_free.c (working copy) @@ -1,119 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#include "dapl_lmr_util.h" -#include "dapl_openib_util.h" -#include "dapl_ia_util.h" - -/* - * dapl_lmr_free - * - * Destroy an instance of the Local Memory Region - * - * Input: - * lmr_handle - * - * Output: - * - * Returns: - * DAT_SUCCESS - * DAT_INVALID_HANDLE - * DAT_INVALID_PARAMETER - * DAT_INVALID_STATE - */ - -u32 dapl_lmr_free(DAT_LMR_HANDLE lmr_handle) -{ - struct dapl_lmr *lmr; - u32 dat_status; - - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_free (%p)\n", lmr_handle); - - if (DAPL_BAD_HANDLE(lmr_handle, DAPL_MAGIC_LMR)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_LMR); - goto bail; - } - - lmr = (struct dapl_lmr *)lmr_handle; - - switch (lmr->param.mem_type) { - case DAT_MEM_TYPE_PHYSICAL: - case DAT_MEM_TYPE_VIRTUAL: - case DAT_MEM_TYPE_LMR: - { - struct dapl_pz *pz; - - if (0 != atomic_read(&lmr->lmr_ref_count)) - return DAT_INVALID_STATE; - - dat_status = - dapl_hash_remove(lmr->header.owner_ia->hca_ptr-> - lmr_hash_table, - lmr->param.lmr_context, NULL); - if (dat_status != DAT_SUCCESS) - goto bail; - - dat_status = dapl_ib_mr_deregister(lmr); - - if (dat_status == DAT_SUCCESS) { - - pz = (struct dapl_pz *)lmr->param.pz_handle; - atomic_dec(&pz->pz_ref_count); - - dapl_lmr_dealloc(lmr); - } else - /* - * Deregister failed; put it back in the - * hash table. - */ - dapl_hash_insert(lmr->header.owner_ia-> - hca_ptr->lmr_hash_table, - lmr->param.lmr_context, lmr); - - break; - } - case DAT_MEM_TYPE_PLATFORM: - case DAT_MEM_TYPE_IA: - case DAT_MEM_TYPE_BYPASS: - { - return DAT_ERROR(DAT_NOT_IMPLEMENTED, 0); - } - default: - { - dat_status = - DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG1); - break; - } - } -bail: - return dat_status; -} Index: linux-kernel-lmr/dat-provider/dapl_lmr_sync_rdma_write.c =================================================================== --- linux-kernel-lmr/dat-provider/dapl_lmr_sync_rdma_write.c (revision 2475) +++ linux-kernel-lmr/dat-provider/dapl_lmr_sync_rdma_write.c (working copy) @@ -1,60 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#include "dapl.h" -#include "dapl_ia_util.h" - -/* - * Ensure a region of memory is consistent by locally flushing - * non-coherent cache - */ -u32 dapl_lmr_sync_rdma_write(DAT_IA_HANDLE ia_handle, - const struct dat_lmr_triplet *local_segments, - u64 num_segments) -{ - struct dapl_ia *ia_ptr; - u32 dat_status = DAT_SUCCESS; - - dapl_dbg_log(DAPL_DBG_TYPE_API, - "dat_lmr_sync_rdma_write (%p, %p, %ld)\n", - ia_handle, local_segments, num_segments); - - ia_ptr = (struct dapl_ia *)ia_handle; - - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); - goto bail; - } - -bail: - return dat_status; -} Index: linux-kernel-lmr/dat-provider/dapl_lmr_kcreate.c =================================================================== --- linux-kernel-lmr/dat-provider/dapl_lmr_kcreate.c (revision 2475) +++ linux-kernel-lmr/dat-provider/dapl_lmr_kcreate.c (working copy) @@ -1,395 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#include "dapl_lmr_util.h" -#include "dapl_openib_util.h" - -static inline u32 dapl_lmr_create_virtual(struct dapl_ia *ia, void *virt_addr, - u64 length, struct dapl_pz *pz, - enum dat_mem_priv_flags privileges, - DAT_LMR_HANDLE *lmr_handle, - DAT_LMR_CONTEXT *lmr_context, - DAT_RMR_CONTEXT *rmr_context, - u64 *registered_length, - u64 *registered_address) -{ - struct dapl_lmr *lmr; - DAT_REGION_DESCRIPTION reg_desc; - u32 dat_status = DAT_SUCCESS; - - reg_desc.for_va = virt_addr; - - lmr = dapl_lmr_alloc(ia, - DAT_MEM_TYPE_VIRTUAL, - reg_desc, length, (DAT_PZ_HANDLE) pz, privileges); - - if (NULL == lmr) { - dat_status = - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); - goto bail; - } - - dat_status = dapl_ib_mr_register(ia, - lmr, virt_addr, length, privileges); - - if (DAT_SUCCESS != dat_status) { - dapl_lmr_dealloc(lmr); - goto bail; - } - - /* if the LMR context is already in the hash table */ - dat_status = dapl_hash_search(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, NULL); - if (dat_status == DAT_SUCCESS) { - (void)dapl_ib_mr_deregister(lmr); - dapl_lmr_dealloc(lmr); - - dat_status = - DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_LMR_IN_USE); - goto bail; - } - - dat_status = dapl_hash_insert(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, lmr); - if (dat_status != DAT_SUCCESS) { - (void)dapl_ib_mr_deregister(lmr); - dapl_lmr_dealloc(lmr); - - /* The value returned by dapl_hash_insert(.) is not */ - /* returned to the consumer because the spec. requires */ - /* that dat_lmr_create(.) return only certain values. */ - dat_status = - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); - goto bail; - } - - atomic_inc(&pz->pz_ref_count); - - if (lmr_handle) - *lmr_handle = (DAT_LMR_HANDLE) lmr; - if (lmr_context) - *lmr_context = (DAT_LMR_CONTEXT) lmr->param.lmr_context; - if (rmr_context) - *rmr_context = (DAT_LMR_CONTEXT) lmr->param.rmr_context; - if (registered_address) - *registered_address = (u64)(unsigned long) virt_addr; - if (registered_length) - *registered_length = length; - -bail: - return dat_status; - -} - -static inline u32 dapl_lmr_create_physical(struct dapl_ia *ia, - DAT_REGION_DESCRIPTION phys_addr, - u64 page_count, struct dapl_pz *pz, - enum dat_mem_priv_flags privileges, - DAT_LMR_HANDLE *lmr_handle, - DAT_LMR_CONTEXT *lmr_context, - DAT_RMR_CONTEXT *rmr_context, - u64 *registered_length, - u64 *registered_address) -{ - struct dapl_lmr *lmr; - u32 dat_status = DAT_SUCCESS; - u64 *array = phys_addr.for_array; - - lmr = dapl_lmr_alloc(ia, - DAT_MEM_TYPE_PHYSICAL, - phys_addr, - page_count, (DAT_PZ_HANDLE) pz, privileges); - - if (NULL == lmr) { - dat_status = - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); - goto bail; - } - - dat_status = dapl_ib_mr_register_physical(ia, lmr, phys_addr.for_array, - page_count, privileges); - - if (DAT_SUCCESS != dat_status) { - dapl_lmr_dealloc(lmr); - goto bail; - } - - /* if the LMR context is already in the hash table */ - dat_status = dapl_hash_search(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, NULL); - if (dat_status == DAT_SUCCESS) { - (void)dapl_ib_mr_deregister(lmr); - dapl_lmr_dealloc(lmr); - - dat_status = - DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_LMR_IN_USE); - goto bail; - } - - dat_status = dapl_hash_insert(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, lmr); - if (dat_status != DAT_SUCCESS) { - (void)dapl_ib_mr_deregister(lmr); - dapl_lmr_dealloc(lmr); - - /* The value returned by dapl_hash_insert(.) is not */ - /* returned to the consumer because the spec. requires */ - /* that dat_lmr_create(.) return only certain values. */ - dat_status = - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); - goto bail; - } - - atomic_inc(&pz->pz_ref_count); - - if (lmr_handle) - *lmr_handle = (DAT_LMR_HANDLE) lmr; - if (lmr_context) - *lmr_context = (DAT_LMR_CONTEXT) lmr->param.lmr_context; - if (rmr_context) - *rmr_context = (DAT_LMR_CONTEXT) lmr->param.rmr_context; - if (registered_address) - *registered_address = array[0]; - if (registered_length) - *registered_length = page_count * PAGE_SIZE; - -bail: - return dat_status; -} - -static inline u32 dapl_lmr_create_lmr(struct dapl_ia *ia, - struct dapl_lmr *original_lmr, - struct dapl_pz *pz, - enum dat_mem_priv_flags privileges, - DAT_LMR_HANDLE *lmr_handle, - DAT_LMR_CONTEXT *lmr_context, - DAT_RMR_CONTEXT *rmr_context, - u64 *registered_length, - u64 *registered_address) -{ - struct dapl_lmr *lmr; - DAT_REGION_DESCRIPTION reg_desc; - u32 dat_status; - - dat_status = dapl_hash_search(ia->hca_ptr->lmr_hash_table, - original_lmr->param.lmr_context, - (DAPL_HASH_DATA *) & lmr); - if (dat_status != DAT_SUCCESS) { - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); - goto bail; - } - - reg_desc.for_lmr_handle = (DAT_LMR_HANDLE) original_lmr; - - lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_LMR, reg_desc, 0, /* length is meaningless */ - (DAT_PZ_HANDLE) pz, privileges); - - if (NULL == lmr) { - dat_status = - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); - goto bail; - } -#if 0 - dat_status = dapl_ib_mr_register_shared(ia, lmr, privileges); -#endif - - if (DAT_SUCCESS != dat_status) { - dapl_lmr_dealloc(lmr); - goto bail; - } - - /* if the LMR context is already in the hash table */ - dat_status = dapl_hash_search(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, NULL); - if (dat_status == DAT_SUCCESS) { - dapl_ib_mr_deregister(lmr); - dapl_lmr_dealloc(lmr); - - dat_status = - DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_LMR_IN_USE); - goto bail; - } - - dat_status = dapl_hash_insert(ia->hca_ptr->lmr_hash_table, - lmr->param.lmr_context, lmr); - if (dat_status != DAT_SUCCESS) { - dapl_ib_mr_deregister(lmr); - dapl_lmr_dealloc(lmr); - - /* The value returned by dapl_hash_insert(.) is not */ - /* returned to the consumer because the spec. requires */ - /* that dat_lmr_create(.) return only certain values. */ - dat_status = - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); - goto bail; - } - - atomic_inc(&pz->pz_ref_count); - - if (lmr_handle) - *lmr_handle = (DAT_LMR_HANDLE) lmr; - if (lmr_context) - *lmr_context = (DAT_LMR_CONTEXT) lmr->param.lmr_context; - if (rmr_context) - *rmr_context = (DAT_LMR_CONTEXT) lmr->param.rmr_context; - if (registered_address) - *registered_address = (u64) (unsigned long) - original_lmr->param.region_desc.for_va; - if (registered_length) - *registered_length = original_lmr->param.length; -bail: - return dat_status; -} - -/* - * dapl_lmr_kcreate - * - * Register a memory region with an Interface Adaptor. - * - * Input: - * ia_handle - * mem_type - * region_description - * length - * pz_handle - * privileges - * optimization - * - * Output: - * lmr_handle - * lmr_context - * registered_length - * registered_address - * - * Returns: - * DAT_SUCCESS - * DAT_INSUFFICIENT_RESOURCES - * DAT_INVALID_PARAMETER - * DAT_INVALID_HANDLE - * DAT_INVALID_STATE - * DAT_MODEL_NOT_SUPPORTED - * - */ -u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_handle, enum dat_mem_type mem_type, - DAT_REGION_DESCRIPTION region_description, u64 length, - DAT_PZ_HANDLE pz_handle, - enum dat_mem_priv_flags privileges, - enum dat_mem_optimize_flags optimization, DAT_LMR_HANDLE *lmr_handle, - DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address) -{ - struct dapl_ia *ia; - struct dapl_pz *pz; - u32 dat_status; - - dapl_dbg_log(DAPL_DBG_TYPE_API, - "dapl_lmr_kcreate(ia:%p, mem_type:%x, ...)\n", - ia_handle, mem_type); - - if (DAPL_BAD_HANDLE(ia_handle, DAPL_MAGIC_IA)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); - goto bail; - } - if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); - goto bail; - } - - ia = (struct dapl_ia *)ia_handle; - pz = (struct dapl_pz *)pz_handle; - - switch (mem_type) { - case DAT_MEM_TYPE_VIRTUAL: - { - dat_status = - DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); - break; - /* - dat_status = dapl_lmr_create_virtual ( - ia, region_description.for_va, length, pz, privileges, - lmr_handle, lmr_context, rmr_context, registered_length, - registered_address); - break; - */ - } - case DAT_MEM_TYPE_PHYSICAL: - { - dat_status = - dapl_lmr_create_physical(ia, region_description, - length, pz, privileges, - lmr_handle, lmr_context, - rmr_context, - registered_length, - registered_address); - break; - } - case DAT_MEM_TYPE_LMR: - { - struct dapl_lmr *lmr; - - if (DAPL_BAD_HANDLE - (region_description.for_lmr_handle, - DAPL_MAGIC_LMR)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, - DAT_INVALID_HANDLE_LMR); - goto bail; - } - - lmr = (struct dapl_lmr *)region_description.for_lmr_handle; - - dat_status = - dapl_lmr_create_lmr(ia, lmr, pz, privileges, - lmr_handle, lmr_context, - rmr_context, registered_length, - registered_address); - break; - } - case DAT_MEM_TYPE_PLATFORM: - case DAT_MEM_TYPE_IA: - case DAT_MEM_TYPE_BYPASS: - { - dat_status = - DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); - break; - } - default: - { - dat_status = - DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); - break; - } - } - -bail: - return dat_status; -} Index: linux-kernel-lmr/patches/alt_dat_provider_makefile =================================================================== --- linux-kernel-lmr/patches/alt_dat_provider_makefile (revision 2475) +++ linux-kernel-lmr/patches/alt_dat_provider_makefile (working copy) @@ -16,7 +16,6 @@ PROVIDER_MODULES := \ dapl_evd_kquery \ dapl_evd_kcreate \ dapl_evd_modify_upcall \ - dapl_lmr_kcreate \ dapl_cookie \ dapl_cno_util \ dapl_cr_accept \ @@ -61,11 +60,7 @@ PROVIDER_MODULES := \ dapl_ia_query \ dapl_ia_util \ dapl_llist \ - dapl_lmr_free \ - dapl_lmr_query \ - dapl_lmr_sync_rdma_read \ - dapl_lmr_sync_rdma_write \ - dapl_lmr_util \ + dapl_lmr \ dapl_mr_util \ dapl_provider \ dapl_sp_util \ @@ -73,10 +68,7 @@ PROVIDER_MODULES := \ dapl_psp_create_any \ dapl_psp_free \ dapl_psp_query \ - dapl_pz_create \ - dapl_pz_free \ - dapl_pz_query \ - dapl_pz_util \ + dapl_pz \ dapl_ring_buffer_util \ dapl_rmr_bind \ dapl_rmr_create \ From mshefty at ichips.intel.com Tue May 24 16:30:23 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Tue, 24 May 2005 16:30:23 -0700 Subject: [openib-general] Re: Latest CM and retransmissions In-Reply-To: <1116969841.4479.12.camel@hal.voltaire.com> References: <1116969841.4479.12.camel@hal.voltaire.com> Message-ID: <4293B90F.5050700@ichips.intel.com> Hal Rosenstock wrote: > On Tue, 2005-05-24 at 16:20, Sean Hefty wrote: > >>Can you apply this patch (applies to the tip) and let me know what, >>if any, output you see? > > > cm_drep_handler - could not acquire cm_id This explains the repeated DREQs then... The DREP is not able to match with an existing connection with matching remote and local IDs. I have no idea why the DREP is not matching however. I have tried multiple times to reproduce this, but have been unsuccessful. Let me work on a couple of CM enhancements to madeye to see if we can identify the reason the DREP doesn't match. - Sean From eli at mellanox.co.il Tue May 24 23:54:00 2005 From: eli at mellanox.co.il (Eli Cohen) Date: Wed, 25 May 2005 09:54:00 +0300 Subject: [openib-general] RE: registering read-only memory Message-ID: <506C3D7B14CDD411A52C00025558DED605B1A87D@mtlex01.yok.mtl.com> Please try: const char *foo="Michael Tsirkin"; as opposed to const char foo[]="Michael Tsirkin"; -----Original Message----- From: Roland Dreier [mailto:roland at topspin.com] Sent: Tuesday, May 24, 2005 10:07 PM To: Michael S. Tsirkin Cc: openib-general at openib.org; Eli Cohen Subject: Re: registering read-only memory > const char foo[]="Michael Tsirkin"; > ibv_reg_mr(ctx->pd, foo, strlen(foo), 0); This seemed to work fine on my system (I got a non-null ibv_mr back with valid-looking L_Key/R_Key). - R. -------------- next part -------------- An HTML attachment was scrubbed... URL: From blist at aon.at Wed May 25 01:23:58 2005 From: blist at aon.at (Bernhard Fischer) Date: Wed, 25 May 2005 10:23:58 +0200 Subject: [openib-general] Re: [openib-commits] r2426 - gen2/users/jlentini/linux-kernel/dat-provider In-Reply-To: References: <20050520220508.CB9A12283D9@openib.ca.sandia.gov> <20050520235642.GE15688@aon.at> Message-ID: <20050525082358.GB23121@aon.at> On Mon, May 23, 2005 at 10:32:12AM -0400, James Lentini wrote: > >Bernhard, > >Thank you for pointing these items out. Comments below: > >On Sat, 21 May 2005, Bernhard Fischer wrote: > >>On Fri, May 20, 2005 at 03:05:08PM -0700, jlentini at openib.org wrote: >>>Author: jlentini >>>Date: 2005-05-20 15:05:07 -0700 (Fri, 20 May 2005) >>>New Revision: 2426 >>>- if (DAPL_GET_CQE_OPTYPE(cqe_ptr) == OP_RECEIVE) { >>>+ "\t\t work_req_id %lli\n", cqe_ptr->wr_id); >>[[:space:]] --------------^ ? > >Which space are you referring to? The one between \t\t and >work_req_id? Yes, i ment the one between \t\t and work_req_id. I now see that those are part of the formatting, so i retract this comment. Sorry. >>>@@ -667,21 +657,21 @@ >>> >>>#ifdef DAPL_DBG >>> /* Current gen2 mthca is not setting the opcode in >>> seccesful cqe */ s/secces/succes/g >>>- /* The opcode will be OP_SEND or OP_RECEIVE acording >>>the is_send bit */ >>>+ /* The opcode will be IB_WC_SEND or IB_WC_RECV >>>acording the is_send bit */ >>"acording" -enoparse: s/acord/accord/g >> >>also in: >>gen2/branches/shaharf-ibat/src/userspace/management/osm/opensm/osm_state_mgr.c >>gen2/branches/roland-uverbs/src/userspace/management/osm/opensm/osm_sa_mcmember_record.c >>gen2/utils/src/linux-user/ibdm/datamodel/ibdm.i >>gen2/utils/src/linux-user/IBMgtSim/src/ibdm.i >>gen2/utils/src/linux-user/IBMgtSim/utils/RunSimTest >>there: callabcks: -enoparse; >>continue with acord: >>gen2/users/jlentini/linux-kernel/dat-provider/dapl_evd_util.c >>gen2/trunk/src/userspace/management/osm/opensm/osm_state_mgr.c > >I've fixed my error: > >gen2/users/jlentini/linux-kernel/dat-provider/dapl_evd_util.c Ok, thank you. > >The others belong to Roland, Shahar, and Eitan. >>>- { >>>- struct dat_ep_attr ep_attr; >>>- struct dat_named_attr ep_state; >>>+ (void) dapl_modify_qp_state_to_error(ep_ptr->qp_handle); >> >>sun will hopefuly fix all of >>egrep -ri "(^[[:space:]]*\(void\))" >>openib.gen2/upstream/gen2/trunk/|egrep -v "svn-(text|base)" >>so i won't comment on that single occurance above ;) > >I didn't realize the convention was to not have spaces before a cast. >This isn't in Documentation/CodingStyle. I know I've been adding these >here and there. I'll fix these as I see them. I ment that Tom Duffy and you will take care of converting a couple of those into void functions, so the casts will no longer be needed. It's not worth the effort to remove the space after the cast. In contrast, personally, i find that the space makes to code easier to read. cheers, Bernhard From lenber.cong at intel.com Wed May 25 03:02:04 2005 From: lenber.cong at intel.com (Cong, Lenber) Date: Wed, 25 May 2005 18:02:04 +0800 Subject: [openib-general] [HELP] Encounter Kernel Panic when Add MellanoxHCA Supporting on 2.6.9 Kernel Message-ID: I tried the patches (2.6.12-to-2.6.9, not svn backport) on an EM64T desktop (without HCA card). The kernel can be installed successfully. I still can't reboot the kernel on Xeon SMP server, even with the new patches (svn backport). The same error was encountered. Then I disabled the option CONFIG_DEBUG_SPINLOCK. The error message disappeared, but the kernel still can't be booted. Can I assume it is the problem of HCA card? Or the issue is relative with the SMP platform? So strange.. Thanks - Lenber -----Original Message----- From: Woodruff, Robert J Sent: 2005年5月25日 6:34 To: Cong, Lenber; openib-general at openib.org Cc: 'Roland Dreier' Subject: RE: [openib-general] [HELP] Encounter Kernel Panic when Add MellanoxHCA Supporting on 2.6.9 Kernel Roland wrote, >I just tried the latest svn on 2.6.11 with CONFIG_DEBUG_SPINLOCK >turned on, and I didn't see any problems. The message > driver/infiniband/hw/mthca/mthca_allocator.c: 46: spin_is_locked on >uninitialized spinlock: f70f7dac >is coming from CHECK_LOCK, which is turned on with >CONFIG_DEBUG_SPINLOCK. However there should be more traceback >information printed to the console as well... did that get dumped as >well? Bob> Roland, has anything been fixed since the 2.6.12 drop in Bob> mthca that could account for this panic ? >Not that I know of... > - R. I just installed the infiniband-backport-2.6.12-to-2.6.9-kernel-fixups-01.diff infiniband-backport-2.6.12-to-2.6.9-openib-drivers-02.diff infiniband-backport-2.6.12-to-2.6.9-openib-fixups-03.diff backport patches on a couple of old 900Mhz IA32 Xeon boxes and was able to build the kernel, load IPoIB and ping another node. I used the Redhat configuration file /boot/config-2.6.9-5.ELsmp, did a make oldconfig and selected modules for all of the infiniband drivers. Then I built and installed the kernel with no problems. Maybe it is the platform (I have seen problems in the past with the BIOS on some platforms being able to map the Mellanox H/W correctly) or could bad Mellanox H/W cause this ? Do you have any other platforms that you could try it on ? woody From mst at mellanox.co.il Wed May 25 06:49:34 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 25 May 2005 16:49:34 +0300 Subject: [openib-general] [PATCH] pingpong test: zero-initialize all attributes Message-ID: <20050525134934.GJ27428@mellanox.co.il> Some address handle attributes (notably static rate flow control) were uninitialized. Fix this by initializing all fields to 0. (Other examples may need fixing, too). Signed-off-by: Michael S. Tsirkin Index: pingpong.c =================================================================== --- pingpong.c (revision 2437) +++ pingpong.c (working copy) @@ -360,19 +360,19 @@ static int pp_post_send(struct pingpong_ static int pp_connect_ctx(struct pingpong_context *ctx, int port, int my_psn, struct pingpong_dest *dest) { - struct ibv_qp_attr attr; - - attr.qp_state = IBV_QPS_RTR; - attr.path_mtu = IBV_MTU_1024; - attr.dest_qp_num = dest->qpn; - attr.rq_psn = dest->psn; - attr.max_dest_rd_atomic = 1; - attr.min_rnr_timer = 12; - attr.ah_attr.is_global = 0; - attr.ah_attr.dlid = dest->lid; - attr.ah_attr.sl = 0; - attr.ah_attr.src_path_bits = 0; - attr.ah_attr.port_num = port; + struct ibv_qp_attr attr = { + .qp_state = IBV_QPS_RTR; + .path_mtu = IBV_MTU_1024; + .dest_qp_num = dest->qpn; + .rq_psn = dest->psn; + .max_dest_rd_atomic = 1; + .min_rnr_timer = 12; + .ah_attr.is_global = 0; + .ah_attr.dlid = dest->lid; + .ah_attr.sl = 0; + .ah_attr.src_path_bits = 0; + .ah_attr.port_num = port; + }; if (ibv_modify_qp(ctx->qp, &attr, IBV_QP_STATE | IBV_QP_AV | -- MST - Michael S. Tsirkin From halr at voltaire.com Wed May 25 07:53:29 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 25 May 2005 10:53:29 -0400 Subject: [openib-general] RE: OpenSM Routing Scalability Proposal In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C3017C@mtlex01.yok.mtl.com> References: <506C3D7B14CDD411A52C00025558DED607C3017C@mtlex01.yok.mtl.com> Message-ID: <1117032809.7724.24.camel@hal.voltaire.com> Hi Eitan, On Tue, 2005-05-17 at 10:53, Eitan Zahavi wrote: > Hi All, > > This is an updated proposal document for your comments. I finally got a chance to read this. Some comments below. > The main change is in describing the need for preserving enough data > to enable incremental routing algorithm. I think incremental can help but presents some new issues. > So the actual proposal is to implement the algorithm described in > section 4.3. 4.1 (min hop) and 4.2 (up/down) are already implemented, right ? > EZ <> It seems like there are 2 parts to 4.3: 1. Min hop table per leaf switch rather than per LID What are the savings for this ? Seems like in terms of memory, this is something like a divisor of L times the number of LIDs per HCA port. Of course, switch port 0s on non leaf switches need to be accomodated. 2. Incremental routing (5) a. Subcase of 5 where there is no other link between 2 adjacent switches. Is another way of stating this, examine next hop switches to see if there is a path between the 2 original switches and keep expanding the depth until 1 is found ? Couldn't this be worse from a compute standpoint than rerouting everything depending on the topology (the likelihood of another path between the 2 original switches) ? b. 5 asks "How do we support topology changes line moving an HCA from one Switch to another?" Also, what about a link moving from one switch to another ? It seems that link down is handled, but nothing is done on a link up. Doesn't there need to be incremental defined for links being added ? c. Also, with incremental routing, it's unclear to me how the paths found would compare with the ones which would be determined from the full algorithm (from scratch). Also, would there be some point at which the full routing would be retriggered ? d. Clearly, there are end node responsibilities here as well (whether this is done incrementally or fully or something else). 3. Persistency (6) a. Full LFT storage (6.1) This presumes that the determination of a topology change upon discovery is cheaper computationally than running the routing. Has this been proven ? (I hope this is the case). b. Root nodes storage (6.2) Are the root nodes determined by the routing or supplied to the routing ? Are they different for unicast and multicast ? -- Hal From jlentini at netapp.com Wed May 25 08:33:34 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 25 May 2005 11:33:34 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: consolidate lmr files into one In-Reply-To: <1116977299.21821.4.camel@duffman> References: <1116655186.16690.15.camel@duffman> <1116970427.11996.4.camel@duffman> <1116977299.21821.4.camel@duffman> Message-ID: Committed in revision 2483. On Tue, 24 May 2005, Tom Duffy wrote: tduffy> On Tue, 2005-05-24 at 17:53 -0400, James Lentini wrote: tduffy> > I'd suggest starting with the memory subsystem. Do you want to do the tduffy> > lmr files? tduffy> tduffy> OK, here goes. Remember that you will need to tduffy> tduffy> svn delete dapl_lmr_free.c dapl_lmr_kcreate.c dapl_lmr_query.c \ tduffy> dapl_lmr_sync_rdma_read.c dapl_lmr_sync_rdma_write.c dapl_lmr_util.c \ tduffy> dapl_lmr_util.h tduffy> tduffy> svn add dapl_lmr.c tduffy> tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> dat-provider/Makefile | 7 tduffy> dat-provider/dapl_lmr.c | 618 ++++++++++++++++++++++++++++++++ tduffy> dat-provider/dapl_lmr_free.c | 119 ------ tduffy> dat-provider/dapl_lmr_kcreate.c | 395 -------------------- tduffy> dat-provider/dapl_lmr_query.c | 60 --- tduffy> dat-provider/dapl_lmr_sync_rdma_read.c | 75 --- tduffy> dat-provider/dapl_lmr_sync_rdma_write.c | 60 --- tduffy> dat-provider/dapl_lmr_util.c | 85 ---- tduffy> dat-provider/dapl_lmr_util.h | 45 -- tduffy> dat-provider/dapl_openib_util.c | 1 tduffy> patches/alt_dat_provider_makefile | 12 tduffy> 11 files changed, 621 insertions(+), 856 deletions(-) tduffy> tduffy> Index: linux-kernel-lmr/dat-provider/dapl_lmr_sync_rdma_read.c tduffy> =================================================================== tduffy> --- linux-kernel-lmr/dat-provider/dapl_lmr_sync_rdma_read.c (revision 2475) tduffy> +++ linux-kernel-lmr/dat-provider/dapl_lmr_sync_rdma_read.c (working copy) tduffy> @@ -1,75 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/* tduffy> - * $Id$ tduffy> - */ tduffy> - tduffy> -#include "dapl.h" tduffy> -#include "dapl_ia_util.h" tduffy> - tduffy> -/* tduffy> - * dat_lmr_sync_rdma_read tduffy> - * tduffy> - * Ensure a region of memory is consistent by locally flushing tduffy> - * non-coherent cache tduffy> - * tduffy> - * Input: tduffy> - * ia_handle tduffy> - * local_segments Array of buffer segments tduffy> - * num_segments Number of segments in local_segments tduffy> - * tduffy> - * Output: tduffy> - * none tduffy> - * tduffy> - * Return Values: tduffy> - * DAT_SUCCESS tduffy> - * DAT_INVALID_HANDLE tduffy> - * DAT_INVALID_PARAMETER tduffy> - */ tduffy> -u32 dapl_lmr_sync_rdma_read(DAT_IA_HANDLE ia_handle, tduffy> - const struct dat_lmr_triplet *local_segments, tduffy> - u64 num_segments) tduffy> -{ tduffy> - struct dapl_ia *ia_ptr; tduffy> - u32 dat_status = DAT_SUCCESS; tduffy> - tduffy> - dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> - "dat_lmr_sync_rdma_read (%p, %p, %ld)\n", tduffy> - ia_handle, local_segments, num_segments); tduffy> - tduffy> - ia_ptr = (struct dapl_ia *)ia_handle; tduffy> - tduffy> - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> -bail: tduffy> - return dat_status; tduffy> -} tduffy> Index: linux-kernel-lmr/dat-provider/dapl_lmr_util.c tduffy> =================================================================== tduffy> --- linux-kernel-lmr/dat-provider/dapl_lmr_util.c (revision 2475) tduffy> +++ linux-kernel-lmr/dat-provider/dapl_lmr_util.c (working copy) tduffy> @@ -1,85 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/* tduffy> - * $Id$ tduffy> - */ tduffy> - tduffy> -#include "dapl_lmr_util.h" tduffy> -#include "dapl_ia_util.h" tduffy> - tduffy> -struct dapl_lmr *dapl_lmr_alloc(struct dapl_ia *ia, enum dat_mem_type mem_type, tduffy> - DAT_REGION_DESCRIPTION region_desc, u64 length, tduffy> - DAT_PZ_HANDLE pz_handle, tduffy> - enum dat_mem_priv_flags mem_priv) tduffy> -{ tduffy> - struct dapl_lmr *lmr; tduffy> - tduffy> - /* Allocate LMR */ tduffy> - lmr = kmalloc(sizeof *lmr, GFP_ATOMIC); tduffy> - if (!lmr) tduffy> - return NULL; tduffy> - tduffy> - /* zero the structure */ tduffy> - memset(lmr, 0, sizeof *lmr); tduffy> - tduffy> - /* tduffy> - * initialize the header tduffy> - */ tduffy> - lmr->header.provider = ia->header.provider; tduffy> - lmr->header.magic = DAPL_MAGIC_LMR; tduffy> - lmr->header.handle_type = DAT_HANDLE_TYPE_LMR; tduffy> - lmr->header.owner_ia = ia; tduffy> - lmr->header.user_context.as_64 = 0; tduffy> - lmr->header.user_context.as_ptr = NULL; tduffy> - dapl_llist_init_entry(&lmr->header.ia_list_entry); tduffy> - dapl_ia_link_lmr(ia, lmr); tduffy> - spin_lock_init(&lmr->header.lock); tduffy> - tduffy> - /* tduffy> - * initialize the body tduffy> - */ tduffy> - lmr->param.ia_handle = (DAT_IA_HANDLE) ia; tduffy> - lmr->param.mem_type = mem_type; tduffy> - lmr->param.region_desc = region_desc; tduffy> - lmr->param.length = length; tduffy> - lmr->param.pz_handle = pz_handle; tduffy> - lmr->param.mem_priv = mem_priv; tduffy> - atomic_set(&lmr->lmr_ref_count, 0); tduffy> - tduffy> - return lmr; tduffy> -} tduffy> - tduffy> -void dapl_lmr_dealloc(struct dapl_lmr *lmr) tduffy> -{ tduffy> - /* reset magic to prevent reuse */ tduffy> - lmr->header.magic = DAPL_MAGIC_INVALID; tduffy> - dapl_ia_unlink_lmr(lmr->header.owner_ia, lmr); tduffy> - /* no need to destroy lmr->header.lock */ tduffy> - tduffy> - kfree(lmr); tduffy> -} tduffy> Index: linux-kernel-lmr/dat-provider/Makefile tduffy> =================================================================== tduffy> --- linux-kernel-lmr/dat-provider/Makefile (revision 2476) tduffy> +++ linux-kernel-lmr/dat-provider/Makefile (working copy) tduffy> @@ -22,7 +22,6 @@ PROVIDER_MODULES := \ tduffy> dapl_evd_kquery \ tduffy> dapl_evd_kcreate \ tduffy> dapl_evd_modify_upcall \ tduffy> - dapl_lmr_kcreate \ tduffy> dapl_cookie \ tduffy> dapl_cno_util \ tduffy> dapl_cr_accept \ tduffy> @@ -67,11 +66,7 @@ PROVIDER_MODULES := \ tduffy> dapl_ia_query \ tduffy> dapl_ia_util \ tduffy> dapl_llist \ tduffy> - dapl_lmr_free \ tduffy> - dapl_lmr_query \ tduffy> - dapl_lmr_sync_rdma_read \ tduffy> - dapl_lmr_sync_rdma_write \ tduffy> - dapl_lmr_util \ tduffy> + dapl_lmr \ tduffy> dapl_mr_util \ tduffy> dapl_provider \ tduffy> dapl_sp_util \ tduffy> Index: linux-kernel-lmr/dat-provider/dapl_lmr_util.h tduffy> =================================================================== tduffy> --- linux-kernel-lmr/dat-provider/dapl_lmr_util.h (revision 2475) tduffy> +++ linux-kernel-lmr/dat-provider/dapl_lmr_util.h (working copy) tduffy> @@ -1,45 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/* tduffy> - * $Id$ tduffy> - */ tduffy> - tduffy> -#ifndef DAPL_LMR_UTIL_H tduffy> -#define DAPL_LMR_UTIL_H tduffy> - tduffy> -#include "dapl_mr_util.h" tduffy> - tduffy> -extern struct dapl_lmr *dapl_lmr_alloc(struct dapl_ia *ia, tduffy> - enum dat_mem_type mem_type, tduffy> - DAT_REGION_DESCRIPTION region_desc, tduffy> - u64 length, DAT_PZ_HANDLE pz_handle, tduffy> - enum dat_mem_priv_flags mem_priv); tduffy> - tduffy> -extern void dapl_lmr_dealloc(struct dapl_lmr *lmr); tduffy> - tduffy> -#endif /* DAPL_LMR_UTIL_H */ tduffy> Index: linux-kernel-lmr/dat-provider/dapl_lmr_query.c tduffy> =================================================================== tduffy> --- linux-kernel-lmr/dat-provider/dapl_lmr_query.c (revision 2475) tduffy> +++ linux-kernel-lmr/dat-provider/dapl_lmr_query.c (working copy) tduffy> @@ -1,60 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/* tduffy> - * $Id$ tduffy> - */ tduffy> - tduffy> -#include "dapl.h" tduffy> - tduffy> -u32 dapl_lmr_query(DAT_LMR_HANDLE lmr_handle, struct dat_lmr_param *lmr_param) tduffy> -{ tduffy> - struct dapl_lmr *lmr; tduffy> - u32 dat_status; tduffy> - tduffy> - dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> - "dapl_lmr_query (%p, %p)\n", tduffy> - lmr_handle, lmr_param); tduffy> - tduffy> - if (DAPL_BAD_HANDLE(lmr_handle, DAPL_MAGIC_LMR)) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_LMR); tduffy> - goto bail; tduffy> - } tduffy> - if (NULL == lmr_param) { tduffy> - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - dat_status = DAT_SUCCESS; tduffy> - lmr = (struct dapl_lmr *)lmr_handle; tduffy> - tduffy> - memcpy(lmr_param, &lmr->param, sizeof *lmr_param); tduffy> - tduffy> -bail: tduffy> - return dat_status; tduffy> -} tduffy> Index: linux-kernel-lmr/dat-provider/dapl_openib_util.c tduffy> =================================================================== tduffy> --- linux-kernel-lmr/dat-provider/dapl_openib_util.c (revision 2476) tduffy> +++ linux-kernel-lmr/dat-provider/dapl_openib_util.c (working copy) tduffy> @@ -37,7 +37,6 @@ tduffy> #include "dapl_openib_util.h" tduffy> #include "dapl_evd_util.h" tduffy> #include "dapl_cr_util.h" tduffy> -#include "dapl_lmr_util.h" tduffy> #include "dapl_rmr_util.h" tduffy> #include "dapl_cookie.h" tduffy> #include "dapl_openib_cm.h" tduffy> Index: linux-kernel-lmr/dat-provider/dapl_lmr.c tduffy> =================================================================== tduffy> --- linux-kernel-lmr/dat-provider/dapl_lmr.c (revision 0) tduffy> +++ linux-kernel-lmr/dat-provider/dapl_lmr.c (revision 0) tduffy> @@ -0,0 +1,618 @@ tduffy> +/* tduffy> + * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * tduffy> + * This Software is licensed under one of the following licenses: tduffy> + * tduffy> + * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> + * available from the Open Source Initiative, see tduffy> + * http://www.opensource.org/licenses/cpl.php. tduffy> + * tduffy> + * 2) under the terms of the "The BSD License" a copy of which is tduffy> + * available from the Open Source Initiative, see tduffy> + * http://www.opensource.org/licenses/bsd-license.php. tduffy> + * tduffy> + * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> + * copy of which is available from the Open Source Initiative, see tduffy> + * http://www.opensource.org/licenses/gpl-license.php. tduffy> + * tduffy> + * Licensee has the right to choose one of the above licenses. tduffy> + * tduffy> + * Redistributions of source code must retain the above copyright tduffy> + * notice and one of the license notices. tduffy> + * tduffy> + * Redistributions in binary form must reproduce both the above copyright tduffy> + * notice, one of the license notices in the documentation tduffy> + * and/or other materials provided with the distribution. tduffy> + */ tduffy> + tduffy> +/* tduffy> + * $Id: dapl_lmr.c 2433 2005-05-21 04:11:03Z jlentini $ tduffy> + */ tduffy> + tduffy> +#include "dapl_openib_util.h" tduffy> +#include "dapl_ia_util.h" tduffy> +#include "dapl_hash.h" tduffy> + tduffy> +static struct dapl_lmr *dapl_lmr_alloc(struct dapl_ia *ia, tduffy> + enum dat_mem_type mem_type, tduffy> + DAT_REGION_DESCRIPTION region_desc, tduffy> + u64 length, DAT_PZ_HANDLE pz_handle, tduffy> + enum dat_mem_priv_flags mem_priv) tduffy> +{ tduffy> + struct dapl_lmr *lmr; tduffy> + tduffy> + /* Allocate LMR */ tduffy> + lmr = kmalloc(sizeof *lmr, GFP_ATOMIC); tduffy> + if (!lmr) tduffy> + return NULL; tduffy> + tduffy> + /* zero the structure */ tduffy> + memset(lmr, 0, sizeof *lmr); tduffy> + tduffy> + /* tduffy> + * initialize the header tduffy> + */ tduffy> + lmr->header.provider = ia->header.provider; tduffy> + lmr->header.magic = DAPL_MAGIC_LMR; tduffy> + lmr->header.handle_type = DAT_HANDLE_TYPE_LMR; tduffy> + lmr->header.owner_ia = ia; tduffy> + lmr->header.user_context.as_64 = 0; tduffy> + lmr->header.user_context.as_ptr = NULL; tduffy> + dapl_llist_init_entry(&lmr->header.ia_list_entry); tduffy> + dapl_ia_link_lmr(ia, lmr); tduffy> + spin_lock_init(&lmr->header.lock); tduffy> + tduffy> + /* tduffy> + * initialize the body tduffy> + */ tduffy> + lmr->param.ia_handle = (DAT_IA_HANDLE) ia; tduffy> + lmr->param.mem_type = mem_type; tduffy> + lmr->param.region_desc = region_desc; tduffy> + lmr->param.length = length; tduffy> + lmr->param.pz_handle = pz_handle; tduffy> + lmr->param.mem_priv = mem_priv; tduffy> + atomic_set(&lmr->lmr_ref_count, 0); tduffy> + tduffy> + return lmr; tduffy> +} tduffy> + tduffy> +static void dapl_lmr_dealloc(struct dapl_lmr *lmr) tduffy> +{ tduffy> + /* reset magic to prevent reuse */ tduffy> + lmr->header.magic = DAPL_MAGIC_INVALID; tduffy> + dapl_ia_unlink_lmr(lmr->header.owner_ia, lmr); tduffy> + /* no need to destroy lmr->header.lock */ tduffy> + tduffy> + kfree(lmr); tduffy> +} tduffy> + tduffy> +/* tduffy> + * dapl_lmr_free tduffy> + * tduffy> + * Destroy an instance of the Local Memory Region tduffy> + * tduffy> + * Input: tduffy> + * lmr_handle tduffy> + * tduffy> + * Output: tduffy> + * tduffy> + * Returns: tduffy> + * DAT_SUCCESS tduffy> + * DAT_INVALID_HANDLE tduffy> + * DAT_INVALID_PARAMETER tduffy> + * DAT_INVALID_STATE tduffy> + */ tduffy> + tduffy> +u32 dapl_lmr_free(DAT_LMR_HANDLE lmr_handle) tduffy> +{ tduffy> + struct dapl_lmr *lmr; tduffy> + u32 dat_status; tduffy> + tduffy> + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_free (%p)\n", lmr_handle); tduffy> + tduffy> + if (DAPL_BAD_HANDLE(lmr_handle, DAPL_MAGIC_LMR)) { tduffy> + dat_status = tduffy> + DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_LMR); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + lmr = (struct dapl_lmr *)lmr_handle; tduffy> + tduffy> + switch (lmr->param.mem_type) { tduffy> + case DAT_MEM_TYPE_PHYSICAL: tduffy> + case DAT_MEM_TYPE_VIRTUAL: tduffy> + case DAT_MEM_TYPE_LMR: tduffy> + { tduffy> + struct dapl_pz *pz; tduffy> + tduffy> + if (0 != atomic_read(&lmr->lmr_ref_count)) tduffy> + return DAT_INVALID_STATE; tduffy> + tduffy> + dat_status = tduffy> + dapl_hash_remove(lmr->header.owner_ia->hca_ptr-> tduffy> + lmr_hash_table, tduffy> + lmr->param.lmr_context, NULL); tduffy> + if (dat_status != DAT_SUCCESS) tduffy> + goto bail; tduffy> + tduffy> + dat_status = dapl_ib_mr_deregister(lmr); tduffy> + tduffy> + if (dat_status == DAT_SUCCESS) { tduffy> + tduffy> + pz = (struct dapl_pz *)lmr->param.pz_handle; tduffy> + atomic_dec(&pz->pz_ref_count); tduffy> + tduffy> + dapl_lmr_dealloc(lmr); tduffy> + } else tduffy> + /* tduffy> + * Deregister failed; put it back in the tduffy> + * hash table. tduffy> + */ tduffy> + dapl_hash_insert(lmr->header.owner_ia-> tduffy> + hca_ptr->lmr_hash_table, tduffy> + lmr->param.lmr_context, lmr); tduffy> + tduffy> + break; tduffy> + } tduffy> + case DAT_MEM_TYPE_PLATFORM: tduffy> + case DAT_MEM_TYPE_IA: tduffy> + case DAT_MEM_TYPE_BYPASS: tduffy> + return DAT_ERROR(DAT_NOT_IMPLEMENTED, 0); tduffy> + default: tduffy> + dat_status = tduffy> + DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG1); tduffy> + break; tduffy> + } tduffy> +bail: tduffy> + return dat_status; tduffy> +} tduffy> + tduffy> +static inline u32 dapl_lmr_create_virtual(struct dapl_ia *ia, void *virt_addr, tduffy> + u64 length, struct dapl_pz *pz, tduffy> + enum dat_mem_priv_flags privileges, tduffy> + DAT_LMR_HANDLE *lmr_handle, tduffy> + DAT_LMR_CONTEXT *lmr_context, tduffy> + DAT_RMR_CONTEXT *rmr_context, tduffy> + u64 *registered_length, tduffy> + u64 *registered_address) tduffy> +{ tduffy> + struct dapl_lmr *lmr; tduffy> + DAT_REGION_DESCRIPTION reg_desc; tduffy> + u32 dat_status = DAT_SUCCESS; tduffy> + tduffy> + reg_desc.for_va = virt_addr; tduffy> + tduffy> + lmr = dapl_lmr_alloc(ia, tduffy> + DAT_MEM_TYPE_VIRTUAL, tduffy> + reg_desc, length, (DAT_PZ_HANDLE) pz, privileges); tduffy> + tduffy> + if (NULL == lmr) { tduffy> + dat_status = tduffy> + DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + dat_status = dapl_ib_mr_register(ia, tduffy> + lmr, virt_addr, length, privileges); tduffy> + tduffy> + if (DAT_SUCCESS != dat_status) { tduffy> + dapl_lmr_dealloc(lmr); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + /* if the LMR context is already in the hash table */ tduffy> + dat_status = dapl_hash_search(ia->hca_ptr->lmr_hash_table, tduffy> + lmr->param.lmr_context, NULL); tduffy> + if (dat_status == DAT_SUCCESS) { tduffy> + (void)dapl_ib_mr_deregister(lmr); tduffy> + dapl_lmr_dealloc(lmr); tduffy> + tduffy> + dat_status = tduffy> + DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_LMR_IN_USE); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + dat_status = dapl_hash_insert(ia->hca_ptr->lmr_hash_table, tduffy> + lmr->param.lmr_context, lmr); tduffy> + if (dat_status != DAT_SUCCESS) { tduffy> + (void)dapl_ib_mr_deregister(lmr); tduffy> + dapl_lmr_dealloc(lmr); tduffy> + tduffy> + /* The value returned by dapl_hash_insert(.) is not */ tduffy> + /* returned to the consumer because the spec. requires */ tduffy> + /* that dat_lmr_create(.) return only certain values. */ tduffy> + dat_status = tduffy> + DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + atomic_inc(&pz->pz_ref_count); tduffy> + tduffy> + if (lmr_handle) tduffy> + *lmr_handle = (DAT_LMR_HANDLE) lmr; tduffy> + if (lmr_context) tduffy> + *lmr_context = (DAT_LMR_CONTEXT) lmr->param.lmr_context; tduffy> + if (rmr_context) tduffy> + *rmr_context = (DAT_LMR_CONTEXT) lmr->param.rmr_context; tduffy> + if (registered_address) tduffy> + *registered_address = (u64)(unsigned long) virt_addr; tduffy> + if (registered_length) tduffy> + *registered_length = length; tduffy> + tduffy> +bail: tduffy> + return dat_status; tduffy> + tduffy> +} tduffy> + tduffy> +static inline u32 dapl_lmr_create_physical(struct dapl_ia *ia, tduffy> + DAT_REGION_DESCRIPTION phys_addr, tduffy> + u64 page_count, struct dapl_pz *pz, tduffy> + enum dat_mem_priv_flags privileges, tduffy> + DAT_LMR_HANDLE *lmr_handle, tduffy> + DAT_LMR_CONTEXT *lmr_context, tduffy> + DAT_RMR_CONTEXT *rmr_context, tduffy> + u64 *registered_length, tduffy> + u64 *registered_address) tduffy> +{ tduffy> + struct dapl_lmr *lmr; tduffy> + u32 dat_status = DAT_SUCCESS; tduffy> + u64 *array = phys_addr.for_array; tduffy> + tduffy> + lmr = dapl_lmr_alloc(ia, tduffy> + DAT_MEM_TYPE_PHYSICAL, tduffy> + phys_addr, tduffy> + page_count, (DAT_PZ_HANDLE) pz, privileges); tduffy> + tduffy> + if (NULL == lmr) { tduffy> + dat_status = tduffy> + DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + dat_status = dapl_ib_mr_register_physical(ia, lmr, phys_addr.for_array, tduffy> + page_count, privileges); tduffy> + tduffy> + if (DAT_SUCCESS != dat_status) { tduffy> + dapl_lmr_dealloc(lmr); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + /* if the LMR context is already in the hash table */ tduffy> + dat_status = dapl_hash_search(ia->hca_ptr->lmr_hash_table, tduffy> + lmr->param.lmr_context, NULL); tduffy> + if (dat_status == DAT_SUCCESS) { tduffy> + (void)dapl_ib_mr_deregister(lmr); tduffy> + dapl_lmr_dealloc(lmr); tduffy> + tduffy> + dat_status = tduffy> + DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_LMR_IN_USE); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + dat_status = dapl_hash_insert(ia->hca_ptr->lmr_hash_table, tduffy> + lmr->param.lmr_context, lmr); tduffy> + if (dat_status != DAT_SUCCESS) { tduffy> + (void)dapl_ib_mr_deregister(lmr); tduffy> + dapl_lmr_dealloc(lmr); tduffy> + tduffy> + /* The value returned by dapl_hash_insert(.) is not */ tduffy> + /* returned to the consumer because the spec. requires */ tduffy> + /* that dat_lmr_create(.) return only certain values. */ tduffy> + dat_status = tduffy> + DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + atomic_inc(&pz->pz_ref_count); tduffy> + tduffy> + if (lmr_handle) tduffy> + *lmr_handle = (DAT_LMR_HANDLE) lmr; tduffy> + if (lmr_context) tduffy> + *lmr_context = (DAT_LMR_CONTEXT) lmr->param.lmr_context; tduffy> + if (rmr_context) tduffy> + *rmr_context = (DAT_LMR_CONTEXT) lmr->param.rmr_context; tduffy> + if (registered_address) tduffy> + *registered_address = array[0]; tduffy> + if (registered_length) tduffy> + *registered_length = page_count * PAGE_SIZE; tduffy> + tduffy> +bail: tduffy> + return dat_status; tduffy> +} tduffy> + tduffy> +static inline u32 dapl_lmr_create_lmr(struct dapl_ia *ia, tduffy> + struct dapl_lmr *original_lmr, tduffy> + struct dapl_pz *pz, tduffy> + enum dat_mem_priv_flags privileges, tduffy> + DAT_LMR_HANDLE *lmr_handle, tduffy> + DAT_LMR_CONTEXT *lmr_context, tduffy> + DAT_RMR_CONTEXT *rmr_context, tduffy> + u64 *registered_length, tduffy> + u64 *registered_address) tduffy> +{ tduffy> + struct dapl_lmr *lmr; tduffy> + DAT_REGION_DESCRIPTION reg_desc; tduffy> + u32 dat_status; tduffy> + tduffy> + dat_status = dapl_hash_search(ia->hca_ptr->lmr_hash_table, tduffy> + original_lmr->param.lmr_context, tduffy> + (DAPL_HASH_DATA *) & lmr); tduffy> + if (dat_status != DAT_SUCCESS) { tduffy> + dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + reg_desc.for_lmr_handle = (DAT_LMR_HANDLE) original_lmr; tduffy> + tduffy> + lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_LMR, reg_desc, 0, /* length is meaningless */ tduffy> + (DAT_PZ_HANDLE) pz, privileges); tduffy> + tduffy> + if (NULL == lmr) { tduffy> + dat_status = tduffy> + DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> + goto bail; tduffy> + } tduffy> +#if 0 tduffy> + dat_status = dapl_ib_mr_register_shared(ia, lmr, privileges); tduffy> +#endif tduffy> + tduffy> + if (DAT_SUCCESS != dat_status) { tduffy> + dapl_lmr_dealloc(lmr); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + /* if the LMR context is already in the hash table */ tduffy> + dat_status = dapl_hash_search(ia->hca_ptr->lmr_hash_table, tduffy> + lmr->param.lmr_context, NULL); tduffy> + if (dat_status == DAT_SUCCESS) { tduffy> + dapl_ib_mr_deregister(lmr); tduffy> + dapl_lmr_dealloc(lmr); tduffy> + tduffy> + dat_status = tduffy> + DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_LMR_IN_USE); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + dat_status = dapl_hash_insert(ia->hca_ptr->lmr_hash_table, tduffy> + lmr->param.lmr_context, lmr); tduffy> + if (dat_status != DAT_SUCCESS) { tduffy> + dapl_ib_mr_deregister(lmr); tduffy> + dapl_lmr_dealloc(lmr); tduffy> + tduffy> + /* The value returned by dapl_hash_insert(.) is not */ tduffy> + /* returned to the consumer because the spec. requires */ tduffy> + /* that dat_lmr_create(.) return only certain values. */ tduffy> + dat_status = tduffy> + DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + atomic_inc(&pz->pz_ref_count); tduffy> + tduffy> + if (lmr_handle) tduffy> + *lmr_handle = (DAT_LMR_HANDLE) lmr; tduffy> + if (lmr_context) tduffy> + *lmr_context = (DAT_LMR_CONTEXT) lmr->param.lmr_context; tduffy> + if (rmr_context) tduffy> + *rmr_context = (DAT_LMR_CONTEXT) lmr->param.rmr_context; tduffy> + if (registered_address) tduffy> + *registered_address = (u64) (unsigned long) tduffy> + original_lmr->param.region_desc.for_va; tduffy> + if (registered_length) tduffy> + *registered_length = original_lmr->param.length; tduffy> +bail: tduffy> + return dat_status; tduffy> +} tduffy> + tduffy> +/* tduffy> + * dapl_lmr_kcreate tduffy> + * tduffy> + * Register a memory region with an Interface Adaptor. tduffy> + * tduffy> + * Input: tduffy> + * ia_handle tduffy> + * mem_type tduffy> + * region_description tduffy> + * length tduffy> + * pz_handle tduffy> + * privileges tduffy> + * optimization tduffy> + * tduffy> + * Output: tduffy> + * lmr_handle tduffy> + * lmr_context tduffy> + * registered_length tduffy> + * registered_address tduffy> + * tduffy> + * Returns: tduffy> + * DAT_SUCCESS tduffy> + * DAT_INSUFFICIENT_RESOURCES tduffy> + * DAT_INVALID_PARAMETER tduffy> + * DAT_INVALID_HANDLE tduffy> + * DAT_INVALID_STATE tduffy> + * DAT_MODEL_NOT_SUPPORTED tduffy> + * tduffy> + */ tduffy> +u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_handle, enum dat_mem_type mem_type, tduffy> + DAT_REGION_DESCRIPTION region_description, u64 length, tduffy> + DAT_PZ_HANDLE pz_handle, tduffy> + enum dat_mem_priv_flags privileges, tduffy> + enum dat_mem_optimize_flags optimization, DAT_LMR_HANDLE *lmr_handle, tduffy> + DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address) tduffy> +{ tduffy> + struct dapl_ia *ia; tduffy> + struct dapl_pz *pz; tduffy> + u32 dat_status; tduffy> + tduffy> + dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> + "dapl_lmr_kcreate(ia:%p, mem_type:%x, ...)\n", tduffy> + ia_handle, mem_type); tduffy> + tduffy> + if (DAPL_BAD_HANDLE(ia_handle, DAPL_MAGIC_IA)) { tduffy> + dat_status = tduffy> + DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); tduffy> + goto bail; tduffy> + } tduffy> + if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { tduffy> + dat_status = tduffy> + DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + ia = (struct dapl_ia *)ia_handle; tduffy> + pz = (struct dapl_pz *)pz_handle; tduffy> + tduffy> + switch (mem_type) { tduffy> + case DAT_MEM_TYPE_VIRTUAL: tduffy> + dat_status = tduffy> + DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); tduffy> + break; tduffy> + /* tduffy> + * dat_status = dapl_lmr_create_virtual ( tduffy> + * ia, region_description.for_va, length, pz, privileges, tduffy> + * lmr_handle, lmr_context, rmr_context, registered_length, tduffy> + * registered_address); tduffy> + * break; tduffy> + */ tduffy> + case DAT_MEM_TYPE_PHYSICAL: tduffy> + dat_status = dapl_lmr_create_physical(ia, region_description, tduffy> + length, pz, privileges, tduffy> + lmr_handle, lmr_context, tduffy> + rmr_context, tduffy> + registered_length, tduffy> + registered_address); tduffy> + break; tduffy> + case DAT_MEM_TYPE_LMR: tduffy> + { tduffy> + struct dapl_lmr *lmr; tduffy> + tduffy> + if (DAPL_BAD_HANDLE tduffy> + (region_description.for_lmr_handle, tduffy> + DAPL_MAGIC_LMR)) { tduffy> + dat_status = tduffy> + DAT_ERROR(DAT_INVALID_HANDLE, tduffy> + DAT_INVALID_HANDLE_LMR); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + lmr = (struct dapl_lmr *)region_description.for_lmr_handle; tduffy> + tduffy> + dat_status = tduffy> + dapl_lmr_create_lmr(ia, lmr, pz, privileges, tduffy> + lmr_handle, lmr_context, tduffy> + rmr_context, registered_length, tduffy> + registered_address); tduffy> + break; tduffy> + } tduffy> + case DAT_MEM_TYPE_PLATFORM: tduffy> + case DAT_MEM_TYPE_IA: tduffy> + case DAT_MEM_TYPE_BYPASS: tduffy> + dat_status = tduffy> + DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); tduffy> + break; tduffy> + default: tduffy> + dat_status = tduffy> + DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); tduffy> + break; tduffy> + } tduffy> + tduffy> +bail: tduffy> + return dat_status; tduffy> +} tduffy> + tduffy> +u32 dapl_lmr_query(DAT_LMR_HANDLE lmr_handle, struct dat_lmr_param *lmr_param) tduffy> +{ tduffy> + struct dapl_lmr *lmr; tduffy> + u32 dat_status; tduffy> + tduffy> + dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> + "dapl_lmr_query (%p, %p)\n", tduffy> + lmr_handle, lmr_param); tduffy> + tduffy> + if (DAPL_BAD_HANDLE(lmr_handle, DAPL_MAGIC_LMR)) { tduffy> + dat_status = tduffy> + DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_LMR); tduffy> + goto bail; tduffy> + } tduffy> + if (NULL == lmr_param) { tduffy> + dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + dat_status = DAT_SUCCESS; tduffy> + lmr = (struct dapl_lmr *)lmr_handle; tduffy> + tduffy> + memcpy(lmr_param, &lmr->param, sizeof *lmr_param); tduffy> + tduffy> +bail: tduffy> + return dat_status; tduffy> +} tduffy> + tduffy> +/* tduffy> + * dat_lmr_sync_rdma_read tduffy> + * tduffy> + * Ensure a region of memory is consistent by locally flushing tduffy> + * non-coherent cache tduffy> + * tduffy> + * Input: tduffy> + * ia_handle tduffy> + * local_segments Array of buffer segments tduffy> + * num_segments Number of segments in local_segments tduffy> + * tduffy> + * Output: tduffy> + * none tduffy> + * tduffy> + * Return Values: tduffy> + * DAT_SUCCESS tduffy> + * DAT_INVALID_HANDLE tduffy> + * DAT_INVALID_PARAMETER tduffy> + */ tduffy> +u32 dapl_lmr_sync_rdma_read(DAT_IA_HANDLE ia_handle, tduffy> + const struct dat_lmr_triplet *local_segments, tduffy> + u64 num_segments) tduffy> +{ tduffy> + struct dapl_ia *ia_ptr; tduffy> + u32 dat_status = DAT_SUCCESS; tduffy> + tduffy> + dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> + "dat_lmr_sync_rdma_read (%p, %p, %ld)\n", tduffy> + ia_handle, local_segments, num_segments); tduffy> + tduffy> + ia_ptr = (struct dapl_ia *)ia_handle; tduffy> + tduffy> + if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { tduffy> + dat_status = tduffy> + DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> +bail: tduffy> + return dat_status; tduffy> +} tduffy> + tduffy> +/* tduffy> + * Ensure a region of memory is consistent by locally flushing tduffy> + * non-coherent cache tduffy> + */ tduffy> +u32 dapl_lmr_sync_rdma_write(DAT_IA_HANDLE ia_handle, tduffy> + const struct dat_lmr_triplet *local_segments, tduffy> + u64 num_segments) tduffy> +{ tduffy> + struct dapl_ia *ia_ptr; tduffy> + u32 dat_status = DAT_SUCCESS; tduffy> + tduffy> + dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> + "dat_lmr_sync_rdma_write (%p, %p, %ld)\n", tduffy> + ia_handle, local_segments, num_segments); tduffy> + tduffy> + ia_ptr = (struct dapl_ia *)ia_handle; tduffy> + tduffy> + if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { tduffy> + dat_status = tduffy> + DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> +bail: tduffy> + return dat_status; tduffy> +} tduffy> Index: linux-kernel-lmr/dat-provider/dapl_lmr_free.c tduffy> =================================================================== tduffy> --- linux-kernel-lmr/dat-provider/dapl_lmr_free.c (revision 2475) tduffy> +++ linux-kernel-lmr/dat-provider/dapl_lmr_free.c (working copy) tduffy> @@ -1,119 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/* tduffy> - * $Id$ tduffy> - */ tduffy> - tduffy> -#include "dapl_lmr_util.h" tduffy> -#include "dapl_openib_util.h" tduffy> -#include "dapl_ia_util.h" tduffy> - tduffy> -/* tduffy> - * dapl_lmr_free tduffy> - * tduffy> - * Destroy an instance of the Local Memory Region tduffy> - * tduffy> - * Input: tduffy> - * lmr_handle tduffy> - * tduffy> - * Output: tduffy> - * tduffy> - * Returns: tduffy> - * DAT_SUCCESS tduffy> - * DAT_INVALID_HANDLE tduffy> - * DAT_INVALID_PARAMETER tduffy> - * DAT_INVALID_STATE tduffy> - */ tduffy> - tduffy> -u32 dapl_lmr_free(DAT_LMR_HANDLE lmr_handle) tduffy> -{ tduffy> - struct dapl_lmr *lmr; tduffy> - u32 dat_status; tduffy> - tduffy> - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_lmr_free (%p)\n", lmr_handle); tduffy> - tduffy> - if (DAPL_BAD_HANDLE(lmr_handle, DAPL_MAGIC_LMR)) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_LMR); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - lmr = (struct dapl_lmr *)lmr_handle; tduffy> - tduffy> - switch (lmr->param.mem_type) { tduffy> - case DAT_MEM_TYPE_PHYSICAL: tduffy> - case DAT_MEM_TYPE_VIRTUAL: tduffy> - case DAT_MEM_TYPE_LMR: tduffy> - { tduffy> - struct dapl_pz *pz; tduffy> - tduffy> - if (0 != atomic_read(&lmr->lmr_ref_count)) tduffy> - return DAT_INVALID_STATE; tduffy> - tduffy> - dat_status = tduffy> - dapl_hash_remove(lmr->header.owner_ia->hca_ptr-> tduffy> - lmr_hash_table, tduffy> - lmr->param.lmr_context, NULL); tduffy> - if (dat_status != DAT_SUCCESS) tduffy> - goto bail; tduffy> - tduffy> - dat_status = dapl_ib_mr_deregister(lmr); tduffy> - tduffy> - if (dat_status == DAT_SUCCESS) { tduffy> - tduffy> - pz = (struct dapl_pz *)lmr->param.pz_handle; tduffy> - atomic_dec(&pz->pz_ref_count); tduffy> - tduffy> - dapl_lmr_dealloc(lmr); tduffy> - } else tduffy> - /* tduffy> - * Deregister failed; put it back in the tduffy> - * hash table. tduffy> - */ tduffy> - dapl_hash_insert(lmr->header.owner_ia-> tduffy> - hca_ptr->lmr_hash_table, tduffy> - lmr->param.lmr_context, lmr); tduffy> - tduffy> - break; tduffy> - } tduffy> - case DAT_MEM_TYPE_PLATFORM: tduffy> - case DAT_MEM_TYPE_IA: tduffy> - case DAT_MEM_TYPE_BYPASS: tduffy> - { tduffy> - return DAT_ERROR(DAT_NOT_IMPLEMENTED, 0); tduffy> - } tduffy> - default: tduffy> - { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG1); tduffy> - break; tduffy> - } tduffy> - } tduffy> -bail: tduffy> - return dat_status; tduffy> -} tduffy> Index: linux-kernel-lmr/dat-provider/dapl_lmr_sync_rdma_write.c tduffy> =================================================================== tduffy> --- linux-kernel-lmr/dat-provider/dapl_lmr_sync_rdma_write.c (revision 2475) tduffy> +++ linux-kernel-lmr/dat-provider/dapl_lmr_sync_rdma_write.c (working copy) tduffy> @@ -1,60 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/* tduffy> - * $Id$ tduffy> - */ tduffy> - tduffy> -#include "dapl.h" tduffy> -#include "dapl_ia_util.h" tduffy> - tduffy> -/* tduffy> - * Ensure a region of memory is consistent by locally flushing tduffy> - * non-coherent cache tduffy> - */ tduffy> -u32 dapl_lmr_sync_rdma_write(DAT_IA_HANDLE ia_handle, tduffy> - const struct dat_lmr_triplet *local_segments, tduffy> - u64 num_segments) tduffy> -{ tduffy> - struct dapl_ia *ia_ptr; tduffy> - u32 dat_status = DAT_SUCCESS; tduffy> - tduffy> - dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> - "dat_lmr_sync_rdma_write (%p, %p, %ld)\n", tduffy> - ia_handle, local_segments, num_segments); tduffy> - tduffy> - ia_ptr = (struct dapl_ia *)ia_handle; tduffy> - tduffy> - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> -bail: tduffy> - return dat_status; tduffy> -} tduffy> Index: linux-kernel-lmr/dat-provider/dapl_lmr_kcreate.c tduffy> =================================================================== tduffy> --- linux-kernel-lmr/dat-provider/dapl_lmr_kcreate.c (revision 2475) tduffy> +++ linux-kernel-lmr/dat-provider/dapl_lmr_kcreate.c (working copy) tduffy> @@ -1,395 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/* tduffy> - * $Id$ tduffy> - */ tduffy> - tduffy> -#include "dapl_lmr_util.h" tduffy> -#include "dapl_openib_util.h" tduffy> - tduffy> -static inline u32 dapl_lmr_create_virtual(struct dapl_ia *ia, void *virt_addr, tduffy> - u64 length, struct dapl_pz *pz, tduffy> - enum dat_mem_priv_flags privileges, tduffy> - DAT_LMR_HANDLE *lmr_handle, tduffy> - DAT_LMR_CONTEXT *lmr_context, tduffy> - DAT_RMR_CONTEXT *rmr_context, tduffy> - u64 *registered_length, tduffy> - u64 *registered_address) tduffy> -{ tduffy> - struct dapl_lmr *lmr; tduffy> - DAT_REGION_DESCRIPTION reg_desc; tduffy> - u32 dat_status = DAT_SUCCESS; tduffy> - tduffy> - reg_desc.for_va = virt_addr; tduffy> - tduffy> - lmr = dapl_lmr_alloc(ia, tduffy> - DAT_MEM_TYPE_VIRTUAL, tduffy> - reg_desc, length, (DAT_PZ_HANDLE) pz, privileges); tduffy> - tduffy> - if (NULL == lmr) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - dat_status = dapl_ib_mr_register(ia, tduffy> - lmr, virt_addr, length, privileges); tduffy> - tduffy> - if (DAT_SUCCESS != dat_status) { tduffy> - dapl_lmr_dealloc(lmr); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - /* if the LMR context is already in the hash table */ tduffy> - dat_status = dapl_hash_search(ia->hca_ptr->lmr_hash_table, tduffy> - lmr->param.lmr_context, NULL); tduffy> - if (dat_status == DAT_SUCCESS) { tduffy> - (void)dapl_ib_mr_deregister(lmr); tduffy> - dapl_lmr_dealloc(lmr); tduffy> - tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_LMR_IN_USE); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - dat_status = dapl_hash_insert(ia->hca_ptr->lmr_hash_table, tduffy> - lmr->param.lmr_context, lmr); tduffy> - if (dat_status != DAT_SUCCESS) { tduffy> - (void)dapl_ib_mr_deregister(lmr); tduffy> - dapl_lmr_dealloc(lmr); tduffy> - tduffy> - /* The value returned by dapl_hash_insert(.) is not */ tduffy> - /* returned to the consumer because the spec. requires */ tduffy> - /* that dat_lmr_create(.) return only certain values. */ tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - atomic_inc(&pz->pz_ref_count); tduffy> - tduffy> - if (lmr_handle) tduffy> - *lmr_handle = (DAT_LMR_HANDLE) lmr; tduffy> - if (lmr_context) tduffy> - *lmr_context = (DAT_LMR_CONTEXT) lmr->param.lmr_context; tduffy> - if (rmr_context) tduffy> - *rmr_context = (DAT_LMR_CONTEXT) lmr->param.rmr_context; tduffy> - if (registered_address) tduffy> - *registered_address = (u64)(unsigned long) virt_addr; tduffy> - if (registered_length) tduffy> - *registered_length = length; tduffy> - tduffy> -bail: tduffy> - return dat_status; tduffy> - tduffy> -} tduffy> - tduffy> -static inline u32 dapl_lmr_create_physical(struct dapl_ia *ia, tduffy> - DAT_REGION_DESCRIPTION phys_addr, tduffy> - u64 page_count, struct dapl_pz *pz, tduffy> - enum dat_mem_priv_flags privileges, tduffy> - DAT_LMR_HANDLE *lmr_handle, tduffy> - DAT_LMR_CONTEXT *lmr_context, tduffy> - DAT_RMR_CONTEXT *rmr_context, tduffy> - u64 *registered_length, tduffy> - u64 *registered_address) tduffy> -{ tduffy> - struct dapl_lmr *lmr; tduffy> - u32 dat_status = DAT_SUCCESS; tduffy> - u64 *array = phys_addr.for_array; tduffy> - tduffy> - lmr = dapl_lmr_alloc(ia, tduffy> - DAT_MEM_TYPE_PHYSICAL, tduffy> - phys_addr, tduffy> - page_count, (DAT_PZ_HANDLE) pz, privileges); tduffy> - tduffy> - if (NULL == lmr) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - dat_status = dapl_ib_mr_register_physical(ia, lmr, phys_addr.for_array, tduffy> - page_count, privileges); tduffy> - tduffy> - if (DAT_SUCCESS != dat_status) { tduffy> - dapl_lmr_dealloc(lmr); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - /* if the LMR context is already in the hash table */ tduffy> - dat_status = dapl_hash_search(ia->hca_ptr->lmr_hash_table, tduffy> - lmr->param.lmr_context, NULL); tduffy> - if (dat_status == DAT_SUCCESS) { tduffy> - (void)dapl_ib_mr_deregister(lmr); tduffy> - dapl_lmr_dealloc(lmr); tduffy> - tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_LMR_IN_USE); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - dat_status = dapl_hash_insert(ia->hca_ptr->lmr_hash_table, tduffy> - lmr->param.lmr_context, lmr); tduffy> - if (dat_status != DAT_SUCCESS) { tduffy> - (void)dapl_ib_mr_deregister(lmr); tduffy> - dapl_lmr_dealloc(lmr); tduffy> - tduffy> - /* The value returned by dapl_hash_insert(.) is not */ tduffy> - /* returned to the consumer because the spec. requires */ tduffy> - /* that dat_lmr_create(.) return only certain values. */ tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - atomic_inc(&pz->pz_ref_count); tduffy> - tduffy> - if (lmr_handle) tduffy> - *lmr_handle = (DAT_LMR_HANDLE) lmr; tduffy> - if (lmr_context) tduffy> - *lmr_context = (DAT_LMR_CONTEXT) lmr->param.lmr_context; tduffy> - if (rmr_context) tduffy> - *rmr_context = (DAT_LMR_CONTEXT) lmr->param.rmr_context; tduffy> - if (registered_address) tduffy> - *registered_address = array[0]; tduffy> - if (registered_length) tduffy> - *registered_length = page_count * PAGE_SIZE; tduffy> - tduffy> -bail: tduffy> - return dat_status; tduffy> -} tduffy> - tduffy> -static inline u32 dapl_lmr_create_lmr(struct dapl_ia *ia, tduffy> - struct dapl_lmr *original_lmr, tduffy> - struct dapl_pz *pz, tduffy> - enum dat_mem_priv_flags privileges, tduffy> - DAT_LMR_HANDLE *lmr_handle, tduffy> - DAT_LMR_CONTEXT *lmr_context, tduffy> - DAT_RMR_CONTEXT *rmr_context, tduffy> - u64 *registered_length, tduffy> - u64 *registered_address) tduffy> -{ tduffy> - struct dapl_lmr *lmr; tduffy> - DAT_REGION_DESCRIPTION reg_desc; tduffy> - u32 dat_status; tduffy> - tduffy> - dat_status = dapl_hash_search(ia->hca_ptr->lmr_hash_table, tduffy> - original_lmr->param.lmr_context, tduffy> - (DAPL_HASH_DATA *) & lmr); tduffy> - if (dat_status != DAT_SUCCESS) { tduffy> - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - reg_desc.for_lmr_handle = (DAT_LMR_HANDLE) original_lmr; tduffy> - tduffy> - lmr = dapl_lmr_alloc(ia, DAT_MEM_TYPE_LMR, reg_desc, 0, /* length is meaningless */ tduffy> - (DAT_PZ_HANDLE) pz, privileges); tduffy> - tduffy> - if (NULL == lmr) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> - goto bail; tduffy> - } tduffy> -#if 0 tduffy> - dat_status = dapl_ib_mr_register_shared(ia, lmr, privileges); tduffy> -#endif tduffy> - tduffy> - if (DAT_SUCCESS != dat_status) { tduffy> - dapl_lmr_dealloc(lmr); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - /* if the LMR context is already in the hash table */ tduffy> - dat_status = dapl_hash_search(ia->hca_ptr->lmr_hash_table, tduffy> - lmr->param.lmr_context, NULL); tduffy> - if (dat_status == DAT_SUCCESS) { tduffy> - dapl_ib_mr_deregister(lmr); tduffy> - dapl_lmr_dealloc(lmr); tduffy> - tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_LMR_IN_USE); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - dat_status = dapl_hash_insert(ia->hca_ptr->lmr_hash_table, tduffy> - lmr->param.lmr_context, lmr); tduffy> - if (dat_status != DAT_SUCCESS) { tduffy> - dapl_ib_mr_deregister(lmr); tduffy> - dapl_lmr_dealloc(lmr); tduffy> - tduffy> - /* The value returned by dapl_hash_insert(.) is not */ tduffy> - /* returned to the consumer because the spec. requires */ tduffy> - /* that dat_lmr_create(.) return only certain values. */ tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - atomic_inc(&pz->pz_ref_count); tduffy> - tduffy> - if (lmr_handle) tduffy> - *lmr_handle = (DAT_LMR_HANDLE) lmr; tduffy> - if (lmr_context) tduffy> - *lmr_context = (DAT_LMR_CONTEXT) lmr->param.lmr_context; tduffy> - if (rmr_context) tduffy> - *rmr_context = (DAT_LMR_CONTEXT) lmr->param.rmr_context; tduffy> - if (registered_address) tduffy> - *registered_address = (u64) (unsigned long) tduffy> - original_lmr->param.region_desc.for_va; tduffy> - if (registered_length) tduffy> - *registered_length = original_lmr->param.length; tduffy> -bail: tduffy> - return dat_status; tduffy> -} tduffy> - tduffy> -/* tduffy> - * dapl_lmr_kcreate tduffy> - * tduffy> - * Register a memory region with an Interface Adaptor. tduffy> - * tduffy> - * Input: tduffy> - * ia_handle tduffy> - * mem_type tduffy> - * region_description tduffy> - * length tduffy> - * pz_handle tduffy> - * privileges tduffy> - * optimization tduffy> - * tduffy> - * Output: tduffy> - * lmr_handle tduffy> - * lmr_context tduffy> - * registered_length tduffy> - * registered_address tduffy> - * tduffy> - * Returns: tduffy> - * DAT_SUCCESS tduffy> - * DAT_INSUFFICIENT_RESOURCES tduffy> - * DAT_INVALID_PARAMETER tduffy> - * DAT_INVALID_HANDLE tduffy> - * DAT_INVALID_STATE tduffy> - * DAT_MODEL_NOT_SUPPORTED tduffy> - * tduffy> - */ tduffy> -u32 dapl_lmr_kcreate(DAT_IA_HANDLE ia_handle, enum dat_mem_type mem_type, tduffy> - DAT_REGION_DESCRIPTION region_description, u64 length, tduffy> - DAT_PZ_HANDLE pz_handle, tduffy> - enum dat_mem_priv_flags privileges, tduffy> - enum dat_mem_optimize_flags optimization, DAT_LMR_HANDLE *lmr_handle, tduffy> - DAT_LMR_CONTEXT *lmr_context, DAT_RMR_CONTEXT *rmr_context, u64 *registered_length, u64 *registered_address) tduffy> -{ tduffy> - struct dapl_ia *ia; tduffy> - struct dapl_pz *pz; tduffy> - u32 dat_status; tduffy> - tduffy> - dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> - "dapl_lmr_kcreate(ia:%p, mem_type:%x, ...)\n", tduffy> - ia_handle, mem_type); tduffy> - tduffy> - if (DAPL_BAD_HANDLE(ia_handle, DAPL_MAGIC_IA)) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); tduffy> - goto bail; tduffy> - } tduffy> - if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - ia = (struct dapl_ia *)ia_handle; tduffy> - pz = (struct dapl_pz *)pz_handle; tduffy> - tduffy> - switch (mem_type) { tduffy> - case DAT_MEM_TYPE_VIRTUAL: tduffy> - { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); tduffy> - break; tduffy> - /* tduffy> - dat_status = dapl_lmr_create_virtual ( tduffy> - ia, region_description.for_va, length, pz, privileges, tduffy> - lmr_handle, lmr_context, rmr_context, registered_length, tduffy> - registered_address); tduffy> - break; tduffy> - */ tduffy> - } tduffy> - case DAT_MEM_TYPE_PHYSICAL: tduffy> - { tduffy> - dat_status = tduffy> - dapl_lmr_create_physical(ia, region_description, tduffy> - length, pz, privileges, tduffy> - lmr_handle, lmr_context, tduffy> - rmr_context, tduffy> - registered_length, tduffy> - registered_address); tduffy> - break; tduffy> - } tduffy> - case DAT_MEM_TYPE_LMR: tduffy> - { tduffy> - struct dapl_lmr *lmr; tduffy> - tduffy> - if (DAPL_BAD_HANDLE tduffy> - (region_description.for_lmr_handle, tduffy> - DAPL_MAGIC_LMR)) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, tduffy> - DAT_INVALID_HANDLE_LMR); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - lmr = (struct dapl_lmr *)region_description.for_lmr_handle; tduffy> - tduffy> - dat_status = tduffy> - dapl_lmr_create_lmr(ia, lmr, pz, privileges, tduffy> - lmr_handle, lmr_context, tduffy> - rmr_context, registered_length, tduffy> - registered_address); tduffy> - break; tduffy> - } tduffy> - case DAT_MEM_TYPE_PLATFORM: tduffy> - case DAT_MEM_TYPE_IA: tduffy> - case DAT_MEM_TYPE_BYPASS: tduffy> - { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); tduffy> - break; tduffy> - } tduffy> - default: tduffy> - { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); tduffy> - break; tduffy> - } tduffy> - } tduffy> - tduffy> -bail: tduffy> - return dat_status; tduffy> -} tduffy> Index: linux-kernel-lmr/patches/alt_dat_provider_makefile tduffy> =================================================================== tduffy> --- linux-kernel-lmr/patches/alt_dat_provider_makefile (revision 2475) tduffy> +++ linux-kernel-lmr/patches/alt_dat_provider_makefile (working copy) tduffy> @@ -16,7 +16,6 @@ PROVIDER_MODULES := \ tduffy> dapl_evd_kquery \ tduffy> dapl_evd_kcreate \ tduffy> dapl_evd_modify_upcall \ tduffy> - dapl_lmr_kcreate \ tduffy> dapl_cookie \ tduffy> dapl_cno_util \ tduffy> dapl_cr_accept \ tduffy> @@ -61,11 +60,7 @@ PROVIDER_MODULES := \ tduffy> dapl_ia_query \ tduffy> dapl_ia_util \ tduffy> dapl_llist \ tduffy> - dapl_lmr_free \ tduffy> - dapl_lmr_query \ tduffy> - dapl_lmr_sync_rdma_read \ tduffy> - dapl_lmr_sync_rdma_write \ tduffy> - dapl_lmr_util \ tduffy> + dapl_lmr \ tduffy> dapl_mr_util \ tduffy> dapl_provider \ tduffy> dapl_sp_util \ tduffy> @@ -73,10 +68,7 @@ PROVIDER_MODULES := \ tduffy> dapl_psp_create_any \ tduffy> dapl_psp_free \ tduffy> dapl_psp_query \ tduffy> - dapl_pz_create \ tduffy> - dapl_pz_free \ tduffy> - dapl_pz_query \ tduffy> - dapl_pz_util \ tduffy> + dapl_pz \ tduffy> dapl_ring_buffer_util \ tduffy> dapl_rmr_bind \ tduffy> dapl_rmr_create \ tduffy> From roland at topspin.com Wed May 25 08:36:11 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 25 May 2005 08:36:11 -0700 Subject: [openib-general] [HELP] Encounter Kernel Panic when Add MellanoxHCA Supporting on 2.6.9 Kernel In-Reply-To: (Lenber Cong's message of "Wed, 25 May 2005 18:02:04 +0800") References: Message-ID: <52d5rftj10.fsf@topspin.com> Lenber> Can I assume it is the problem of HCA card? Or the issue Lenber> is relative with the SMP platform? So strange.. It's possible it's the HCA but I'm not sure what could be wrong. With CONFIG_DEBUG_SPINLOCK can you get more of the traceback? The BUG() should be producing a full stack trace. Thanks, Roland From jlentini at netapp.com Wed May 25 08:38:56 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 25 May 2005 11:38:56 -0400 (EDT) Subject: [openib-general] Re: [PATCH] [kdapl CM] Fix endian conversions of service ID In-Reply-To: <1116962814.4490.192.camel@hal.voltaire.com> References: <1116959536.4490.166.camel@hal.voltaire.com> <1116962814.4490.192.camel@hal.voltaire.com> Message-ID: halr> On Tue, 2005-05-24 at 15:20, James Lentini wrote: halr> > halr> [kdapl CM] Fix endian conversions of service ID halr> > halr> Problem pointed out by James Lentini halr> > halr> halr> > halr> Signed-off-by: Hal Rosenstock halr> > halr> halr> > halr> Index: dapl_openib_cm.c halr> > halr> =================================================================== halr> > halr> -- dapl_openib_cm.c (revision 2468) halr> > halr> +++ dapl_openib_cm.c (working copy) halr> > halr> @@ -309,7 +309,7 @@ halr> > halr> if (conn->dapl_path.mtu > IB_MTU_1024) halr> > halr> conn->dapl_path.mtu = IB_MTU_1024; halr> > halr> halr> > halr> - conn->param.service_id = be64_to_cpu(conn->service_id); halr> > halr> + conn->param.service_id = conn->service_id; halr> > halr> > With the change to dapl_ib_connect below, the conn->service_id is in halr> > CPU byte order at this point. The conn->param is a ib_cm_req_param halr> > structure. The comment describing this structure's service_id field halr> > says that it should be in network (big endian) byte order. halr> > halr> > So... halr> > halr> > halr> conn->param.primary_path = &conn->dapl_path; halr> > halr> conn->param.alternate_path = NULL; halr> > halr> halr> > halr> @@ -445,8 +445,7 @@ halr> > halr> conn->param.local_cm_response_timeout = halr> > halr> DAPL_OPENIB_CM_RESPONSE_TIMEOUT; halr> > halr> conn->param.max_cm_retries = DAPL_OPENIB_MAX_CM_RETRIES; halr> > halr> halr> > halr> - memcpy(&conn->service_id, &remote_conn_qual, sizeof halr> > halr> conn->service_id); halr> > halr> - halr> > halr> + conn->service_id = be64_to_cpu(remote_conn_qual); halr> > halr> > ...that makes me think we should change the line above to halr> > halr> > conn->service_id = remote_conn_qual; halr> > halr> > and require that consumer's specify their connection qualifier values halr> > in network byte order here and ... halr> halr> I think the convention OpenIB has been using is to supply parameters in halr> CPU endian but it can work either way. The comments in ib_cm.h say that service id parameters should be in network byte order. Are these incorrect? halr> halr> > halr> conn->remote_ia_address = remote_ia_address; halr> > halr> conn->dapl_comp.fn = &dapl_rt_comp_handler; halr> > halr> conn->dapl_comp.context = conn; halr> > halr> @@ -627,7 +626,7 @@ halr> > halr> } halr> > halr> halr> > halr> status = ib_cm_listen(sp_ptr->cm_srvc_handle, halr> > halr> - be64_to_cpu(sp_ptr->conn_qual), 0); halr> > halr> + cpu_to_be64(sp_ptr->conn_qual), 0); halr> > halr> > ... do the same here. What do you think? halr> halr> From halr at voltaire.com Wed May 25 08:41:51 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 25 May 2005 11:41:51 -0400 Subject: [openib-general] Re: [PATCH] [kdapl CM] Fix endian conversions of service ID In-Reply-To: References: <1116959536.4490.166.camel@hal.voltaire.com> <1116962814.4490.192.camel@hal.voltaire.com> Message-ID: <1117035711.7724.30.camel@hal.voltaire.com> On Wed, 2005-05-25 at 11:38, James Lentini wrote: > halr> On Tue, 2005-05-24 at 15:20, James Lentini wrote: > halr> > halr> [kdapl CM] Fix endian conversions of service ID > halr> > halr> Problem pointed out by James Lentini > halr> > halr> > halr> > halr> Signed-off-by: Hal Rosenstock > halr> > halr> > halr> > halr> Index: dapl_openib_cm.c > halr> > halr> =================================================================== > halr> > halr> -- dapl_openib_cm.c (revision 2468) > halr> > halr> +++ dapl_openib_cm.c (working copy) > halr> > halr> @@ -309,7 +309,7 @@ > halr> > halr> if (conn->dapl_path.mtu > IB_MTU_1024) > halr> > halr> conn->dapl_path.mtu = IB_MTU_1024; > halr> > halr> > halr> > halr> - conn->param.service_id = be64_to_cpu(conn->service_id); > halr> > halr> + conn->param.service_id = conn->service_id; > halr> > > halr> > With the change to dapl_ib_connect below, the conn->service_id is in > halr> > CPU byte order at this point. The conn->param is a ib_cm_req_param > halr> > structure. The comment describing this structure's service_id field > halr> > says that it should be in network (big endian) byte order. > halr> > > halr> > So... > halr> > > halr> > halr> conn->param.primary_path = &conn->dapl_path; > halr> > halr> conn->param.alternate_path = NULL; > halr> > halr> > halr> > halr> @@ -445,8 +445,7 @@ > halr> > halr> conn->param.local_cm_response_timeout = > halr> > halr> DAPL_OPENIB_CM_RESPONSE_TIMEOUT; > halr> > halr> conn->param.max_cm_retries = DAPL_OPENIB_MAX_CM_RETRIES; > halr> > halr> > halr> > halr> - memcpy(&conn->service_id, &remote_conn_qual, sizeof > halr> > halr> conn->service_id); > halr> > halr> - > halr> > halr> + conn->service_id = be64_to_cpu(remote_conn_qual); > halr> > > halr> > ...that makes me think we should change the line above to > halr> > > halr> > conn->service_id = remote_conn_qual; > halr> > > halr> > and require that consumer's specify their connection qualifier values > halr> > in network byte order here and ... > halr> > halr> I think the convention OpenIB has been using is to supply parameters in > halr> CPU endian but it can work either way. > > The comments in ib_cm.h say that service id parameters should be in > network byte order. Are these incorrect? No. I was referring to your alternative of where this is performed. > halr> > halr> > halr> conn->remote_ia_address = remote_ia_address; > halr> > halr> conn->dapl_comp.fn = &dapl_rt_comp_handler; > halr> > halr> conn->dapl_comp.context = conn; > halr> > halr> @@ -627,7 +626,7 @@ > halr> > halr> } > halr> > halr> > halr> > halr> status = ib_cm_listen(sp_ptr->cm_srvc_handle, > halr> > halr> - be64_to_cpu(sp_ptr->conn_qual), 0); > halr> > halr> + cpu_to_be64(sp_ptr->conn_qual), 0); > halr> > > halr> > ... do the same here. What do you think? > halr> > halr> From mshefty at ichips.intel.com Wed May 25 09:30:45 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Wed, 25 May 2005 09:30:45 -0700 Subject: [openib-general] Re: Latest CM and retransmissions In-Reply-To: <1116954610.4490.73.camel@hal.voltaire.com> References: <1116952775.4490.53.camel@hal.voltaire.com> <42935A70.4030003@ichips.intel.com> <1116954610.4490.73.camel@hal.voltaire.com> Message-ID: <4294A835.8000802@ichips.intel.com> Hal Rosenstock wrote: > DREQ --------> > <------- DREP > DREQ --------> > DREQ --------> Do you know if the code that issues the DREQ destroys the cm_id immediately afterwards? - Sean From halr at voltaire.com Wed May 25 09:57:41 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 25 May 2005 12:57:41 -0400 Subject: [openib-general] Re: Latest CM and retransmissions In-Reply-To: <4294A835.8000802@ichips.intel.com> References: <1116952775.4490.53.camel@hal.voltaire.com> <42935A70.4030003@ichips.intel.com> <1116954610.4490.73.camel@hal.voltaire.com> <4294A835.8000802@ichips.intel.com> Message-ID: <1117040260.7724.52.camel@hal.voltaire.com> On Wed, 2005-05-25 at 12:30, Sean Hefty wrote: > Hal Rosenstock wrote: > > DREQ --------> > > <------- DREP > > DREQ --------> > > DREQ --------> > > Do you know if the code that issues the DREQ destroys the cm_id immediately > afterwards? It likely is. I was going to ask about this yesterday. -- Hal From mshefty at ichips.intel.com Wed May 25 10:18:45 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Wed, 25 May 2005 10:18:45 -0700 Subject: [openib-general] OOPS: ib_mad crashery on bootup In-Reply-To: <1116530995.980.9.camel@duffman> References: <1116530995.980.9.camel@duffman> Message-ID: <4294B375.8070808@ichips.intel.com> I've been able to hit an issue in the MAD layer that might be related. I still do not know the root cause, however. - Sean cmpost: starting client cmpost: connecting cmpost: connect time: 4479000 us cmpost: completing data transfers cmpost: waiting to disconnect cmpost: test complete Unable to handle kernel paging request at virtual address 6b6b6b83 printing eip: f8ca36e2 *pde = 00000000 Oops: 0000 [#1] SMP Modules linked in: ib_cmpost ib_cm ib_sa ib_mthca ib_mad ib_core edd st sr_mod ide_cd cdrom thermal processor fan button battery ac e100 mii e1000 hw_random uhci_hcd usbcore evdev reiserfs aic7xxx sd_mod scsi_mod CPU: 0 EIP: 0060:[pg0+948262626/1069220864] Not tainted VLI EIP: 0060:[] Not tainted VLI EFLAGS: 00010292 (2.6.9) EIP is at ib_mad_send_done_handler+0x12/0x120 [ib_mad] eax: dfda185c ebx: c5cf4790 ecx: c967855c edx: d92e9f24 esi: 6b6b6b6b edi: d92e9f24 ebp: c5cf4790 esp: d92e9f04 ds: 007b es: 007b ss: 0068 Process ib_mad1 (pid: 4493, threadinfo=d92e8000 task=f7738230) Stack: 00000001 d92e9f24 d810795c f6c6f5bc dfda185c d92e9f24 dfda185c f8ca3979 c5cf4790 00000000 00000000 00000000 d92e9f58 c011e4f9 00000000 00000402 00000006 00000000 0001f341 f7612c30 dfda18cc f7612c30 dfda18d0 c01305f8 Call Trace: [pg0+948263289/1069220864] ib_mad_completion_handler+0x89/0xa0 [ib_mad] [] ib_mad_completion_handler+0x89/0xa0 [ib_mad] [__wake_up+41/64] __wake_up+0x29/0x40 [] __wake_up+0x29/0x40 [worker_thread+424/560] worker_thread+0x1a8/0x230 [] worker_thread+0x1a8/0x230 [pg0+948263152/1069220864] ib_mad_completion_handler+0x0/0xa0 [ib_mad] [] ib_mad_completion_handler+0x0/0xa0 [ib_mad] [default_wake_function+0/16] default_wake_function+0x0/0x10 [] default_wake_function+0x0/0x10 [default_wake_function+0/16] default_wake_function+0x0/0x10 [] default_wake_function+0x0/0x10 [worker_thread+0/560] worker_thread+0x0/0x230 [] worker_thread+0x0/0x230 [kthread+136/176] kthread+0x88/0xb0 [] kthread+0x88/0xb0 [kthread+0/176] kthread+0x0/0xb0 [] kthread+0x0/0xb0 [kernel_thread_helper+5/16] kernel_thread_helper+0x5/0x10 [] kernel_thread_helper+0x5/0x10 Code: 00 6a 00 e8 11 ae 47 c7 5d eb 98 8d b4 26 00 00 00 00 8d bc 27 00 00 00 00 55 57 56 53 83 ec 0c 89 54 24 04 8b 1a 89 dd 8b 73 08 <8b> 46 18 89 04 24 eb 50 8d b6 00 00 00 00 8b 54 24 04 89 d8 e8 From mshefty at ichips.intel.com Wed May 25 10:39:59 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Wed, 25 May 2005 10:39:59 -0700 Subject: [openib-general] Re: Latest CM and retransmissions In-Reply-To: <1117040260.7724.52.camel@hal.voltaire.com> References: <1116952775.4490.53.camel@hal.voltaire.com> <42935A70.4030003@ichips.intel.com> <1116954610.4490.73.camel@hal.voltaire.com> <4294A835.8000802@ichips.intel.com> <1117040260.7724.52.camel@hal.voltaire.com> Message-ID: <4294B86F.4070101@ichips.intel.com> Hal Rosenstock wrote: > On Wed, 2005-05-25 at 12:30, Sean Hefty wrote: > >>Hal Rosenstock wrote: >> >>>DREQ --------> >>> <------- DREP >>>DREQ --------> >>>DREQ --------> >> >>Do you know if the code that issues the DREQ destroys the cm_id immediately >>afterwards? > > It likely is. I was going to ask about this yesterday. If a client destroys the cm_id immediately after sending a DREQ (before the DREP is received), the CM will transition the cm_id directly into the timewait state. I've just committed a change to the CM to cancel the DREQ if the cm_id is destroyed. Note that this won't result in the DREP matching with the DREQ, since the cm_id has been destroyed, but should prevent the DREQ from being resent, if this is indeed what is happening. Can you pull the CM from 2485 and retest - Sean From ftillier at silverstorm.com Wed May 25 10:51:19 2005 From: ftillier at silverstorm.com (Fab Tillier) Date: Wed, 25 May 2005 10:51:19 -0700 Subject: [openib-general] Re: Latest CM and retransmissions In-Reply-To: <4294B86F.4070101@ichips.intel.com> Message-ID: <001601c56152$5c0940d0$9b5aa8c0@infiniconsys.com> > From: Sean Hefty [mailto:mshefty at ichips.intel.com] > Sent: Wednesday, May 25, 2005 10:40 AM > > Hal Rosenstock wrote: > > On Wed, 2005-05-25 at 12:30, Sean Hefty wrote: > > > >>Hal Rosenstock wrote: > >> > >>>DREQ --------> > >>> <------- DREP > >>>DREQ --------> > >>>DREQ --------> > >> > >>Do you know if the code that issues the DREQ destroys the cm_id > >>immediately afterwards? > > > > It likely is. I was going to ask about this yesterday. > > If a client destroys the cm_id immediately after sending a DREQ (before > the > DREP is received), the CM will transition the cm_id directly into the > timewait state. I've just committed a change to the CM to cancel the DREQ > if the cm_id is destroyed. Note that this won't result in the DREP > matching > with the DREQ, since the cm_id has been destroyed, but should prevent the > DREQ from being resent, if this is indeed what is happening. Can you pull > the CM from 2485 and retest Why not just delay the transition into timewait until the DREP is received or the DREQ times out? - Fab From jlentini at netapp.com Wed May 25 10:53:37 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 25 May 2005 13:53:37 -0400 (EDT) Subject: [openib-general] Re: [PATCH] [kdapl CM] Fix endian conversions of service ID In-Reply-To: <1117035711.7724.30.camel@hal.voltaire.com> References: <1116959536.4490.166.camel@hal.voltaire.com> <1116962814.4490.192.camel@hal.voltaire.com> <1117035711.7724.30.camel@hal.voltaire.com> Message-ID: I committed the fix for this in revision 2486. I made one small simplification: I used the dapl_cm_id's param.service_id field throughout. On Wed, 25 May 2005, Hal Rosenstock wrote: > On Wed, 2005-05-25 at 11:38, James Lentini wrote: >> halr> On Tue, 2005-05-24 at 15:20, James Lentini wrote: >> halr> > halr> [kdapl CM] Fix endian conversions of service ID >> halr> > halr> Problem pointed out by James Lentini >> halr> > halr> >> halr> > halr> Signed-off-by: Hal Rosenstock >> halr> > halr> >> halr> > halr> Index: dapl_openib_cm.c >> halr> > halr> =================================================================== >> halr> > halr> -- dapl_openib_cm.c (revision 2468) >> halr> > halr> +++ dapl_openib_cm.c (working copy) >> halr> > halr> @@ -309,7 +309,7 @@ >> halr> > halr> if (conn->dapl_path.mtu > IB_MTU_1024) >> halr> > halr> conn->dapl_path.mtu = IB_MTU_1024; >> halr> > halr> >> halr> > halr> - conn->param.service_id = be64_to_cpu(conn->service_id); >> halr> > halr> + conn->param.service_id = conn->service_id; >> halr> > >> halr> > With the change to dapl_ib_connect below, the conn->service_id is in >> halr> > CPU byte order at this point. The conn->param is a ib_cm_req_param >> halr> > structure. The comment describing this structure's service_id field >> halr> > says that it should be in network (big endian) byte order. >> halr> > >> halr> > So... >> halr> > >> halr> > halr> conn->param.primary_path = &conn->dapl_path; >> halr> > halr> conn->param.alternate_path = NULL; >> halr> > halr> >> halr> > halr> @@ -445,8 +445,7 @@ >> halr> > halr> conn->param.local_cm_response_timeout = >> halr> > halr> DAPL_OPENIB_CM_RESPONSE_TIMEOUT; >> halr> > halr> conn->param.max_cm_retries = DAPL_OPENIB_MAX_CM_RETRIES; >> halr> > halr> >> halr> > halr> - memcpy(&conn->service_id, &remote_conn_qual, sizeof >> halr> > halr> conn->service_id); >> halr> > halr> - >> halr> > halr> + conn->service_id = be64_to_cpu(remote_conn_qual); >> halr> > >> halr> > ...that makes me think we should change the line above to >> halr> > >> halr> > conn->service_id = remote_conn_qual; >> halr> > >> halr> > and require that consumer's specify their connection qualifier values >> halr> > in network byte order here and ... >> halr> >> halr> I think the convention OpenIB has been using is to supply parameters in >> halr> CPU endian but it can work either way. >> >> The comments in ib_cm.h say that service id parameters should be in >> network byte order. Are these incorrect? > > No. I was referring to your alternative of where this is performed. > >> halr> >> halr> > halr> conn->remote_ia_address = remote_ia_address; >> halr> > halr> conn->dapl_comp.fn = &dapl_rt_comp_handler; >> halr> > halr> conn->dapl_comp.context = conn; >> halr> > halr> @@ -627,7 +626,7 @@ >> halr> > halr> } >> halr> > halr> >> halr> > halr> status = ib_cm_listen(sp_ptr->cm_srvc_handle, >> halr> > halr> - be64_to_cpu(sp_ptr->conn_qual), 0); >> halr> > halr> + cpu_to_be64(sp_ptr->conn_qual), 0); >> halr> > >> halr> > ... do the same here. What do you think? >> halr> >> halr> > From tduffy at sun.com Wed May 25 10:54:18 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 25 May 2005 10:54:18 -0700 Subject: [openib-general] [PATCH] kDAPL: consolidate rmr files into one Message-ID: <1117043658.29341.6.camel@duffman> Signed-off-by: Tom Duffy Index: linux-kernel-rmr/dat-provider/dapl_rmr_create.c =================================================================== --- linux-kernel-rmr/dat-provider/dapl_rmr_create.c (revision 2483) +++ linux-kernel-rmr/dat-provider/dapl_rmr_create.c (working copy) @@ -1,89 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#include "dapl_rmr_util.h" -#include "dapl_openib_util.h" - -/* - * dapl_rmr_create - * - * Create a remote memory region for the specified protection zone - * - * Input: - * pz_handle - * - * Output: - * rmr_handle - * - * Returns: - * DAT_SUCCESS - * DAT_INSUFFICIENT_RESOURCES - * DAT_INVALID_PARAMETER - */ -u32 dapl_rmr_create(DAT_PZ_HANDLE pz_handle, DAT_RMR_HANDLE *rmr_handle) -{ - struct dapl_pz *pz; - struct dapl_rmr *rmr; - u32 dat_status = DAT_SUCCESS; - - if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); - goto bail; - } - - pz = (struct dapl_pz *)pz_handle; - - rmr = dapl_rmr_alloc(pz); - - if (rmr == NULL) { - dat_status = - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); - goto bail; - } - - dat_status = dapl_ib_mw_alloc(rmr); - - if (dat_status != DAT_SUCCESS) { - dapl_rmr_dealloc(rmr); - dat_status = - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, - DAT_RESOURCE_MEMORY_REGION); - goto bail; - } - - atomic_inc(&pz->pz_ref_count); - - *rmr_handle = rmr; - -bail: - return dat_status; -} Index: linux-kernel-rmr/dat-provider/Makefile =================================================================== --- linux-kernel-rmr/dat-provider/Makefile (revision 2483) +++ linux-kernel-rmr/dat-provider/Makefile (working copy) @@ -76,11 +61,7 @@ PROVIDER_MODULES := \ dapl_psp_query \ dapl_pz \ dapl_ring_buffer_util \ - dapl_rmr_bind \ - dapl_rmr_create \ - dapl_rmr_free \ - dapl_rmr_query \ - dapl_rmr_util \ + dapl_rmr \ dapl_rsp_create \ dapl_rsp_free \ dapl_rsp_query \ @@ -98,5 +79,25 @@ PROVIDER_MODULES := \ Index: linux-kernel-rmr/dat-provider/dapl_rmr_free.c =================================================================== --- linux-kernel-rmr/dat-provider/dapl_rmr_free.c (revision 2483) +++ linux-kernel-rmr/dat-provider/dapl_rmr_free.c (working copy) @@ -1,84 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#include "dapl_rmr_util.h" -#include "dapl_openib_util.h" -#include "dapl_ia_util.h" - -/* - * dapl_rmr_free - * - * Destroy an instance of the Remote Memory Region - * - * Input: - * rmr_handle - * - * Output: - * none - * - * Returns: - * DAT_SUCCESS - * DAT_INVALID_PARAMETER - */ -u32 dapl_rmr_free(DAT_RMR_HANDLE rmr_handle) -{ - struct dapl_rmr *rmr; - u32 dat_status = DAT_SUCCESS; - - if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); - goto bail; - } - - rmr = (struct dapl_rmr *)rmr_handle; - - /* - * If the user did not perform an unbind op, release - * counts here. - */ - if (rmr->param.lmr_triplet.virtual_address != 0) { - atomic_dec(&rmr->lmr->lmr_ref_count); - rmr->param.lmr_triplet.virtual_address = 0; - } - - dat_status = dapl_ib_mw_free(rmr); - - if (dat_status != DAT_SUCCESS) - goto bail; - - atomic_dec(&rmr->pz->pz_ref_count); - - dapl_rmr_dealloc(rmr); - -bail: - return dat_status; -} Index: linux-kernel-rmr/dat-provider/dapl_rmr_query.c =================================================================== --- linux-kernel-rmr/dat-provider/dapl_rmr_query.c (revision 2483) +++ linux-kernel-rmr/dat-provider/dapl_rmr_query.c (working copy) @@ -1,55 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#include "dapl.h" - -u32 dapl_rmr_query(DAT_RMR_HANDLE rmr_handle, struct dat_rmr_param *rmr_param) -{ - struct dapl_rmr *rmr; - u32 dat_status = DAT_SUCCESS; - - if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); - goto bail; - } - if (NULL == rmr_param) { - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); - goto bail; - } - - rmr = (struct dapl_rmr *)rmr_handle; - - memcpy(rmr_param, &rmr->param, sizeof *rmr_param); - -bail: - return dat_status; -} Index: linux-kernel-rmr/dat-provider/dapl_rmr_bind.c =================================================================== --- linux-kernel-rmr/dat-provider/dapl_rmr_bind.c (revision 2483) +++ linux-kernel-rmr/dat-provider/dapl_rmr_bind.c (working copy) @@ -1,273 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#include "dapl.h" -#include "dapl_rmr_util.h" -#include "dapl_ep_util.h" -#include "dapl_cookie.h" -#include "dapl_openib_util.h" - -/********************************************************************* - * * - * Function Definitions * - * * - *********************************************************************/ - -static inline u32 dapl_rmr_bind_fuse(struct dapl_rmr *rmr, - const struct dat_lmr_triplet *lmr_triplet, - enum dat_mem_priv_flags mem_priv, - struct dapl_ep *ep_ptr, - DAT_RMR_COOKIE user_cookie, - enum dat_completion_flags completion_flags, - DAT_RMR_CONTEXT *rmr_context) -{ - struct dapl_lmr *lmr; - struct dapl_cookie *cookie; - u32 status; - boolean_t is_signaled; - - status = dapl_hash_search(rmr->header.owner_ia->hca_ptr->lmr_hash_table, - lmr_triplet->lmr_context, - (DAPL_HASH_DATA *) &lmr); - if (DAT_SUCCESS != status) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); - goto bail; - } - - /* if the ep in unconnected return an error. IB requires that the */ - /* QP be connected to change a memory window binding since: */ - /* */ - /* - memory window bind operations are WQEs placed on a QP's */ - /* send queue */ - /* */ - /* - QP's only process WQEs on the send queue when the QP is in */ - /* the RTS state */ - if (DAT_EP_STATE_CONNECTED != ep_ptr->param.ep_state) { - status = - DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep_ptr)); - goto bail; - } - - if (FALSE == - dapl_mr_bounds_check(dapl_mr_get_address - (lmr->param.region_desc, lmr->param.mem_type), - lmr->param.length, - lmr_triplet->virtual_address, - lmr_triplet->segment_length)) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); - goto bail; - } - - /* If the LMR, RMR, and EP are not in the same PZ, there is an error */ - if ((ep_ptr->param.pz_handle != lmr->param.pz_handle) || - (ep_ptr->param.pz_handle != rmr->param.pz_handle)) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); - goto bail; - } - - if (!dapl_rmr_validate_completion_flag(DAT_COMPLETION_SUPPRESS_FLAG, - ep_ptr->param.ep_attr. - request_completion_flags, - completion_flags) || - !dapl_rmr_validate_completion_flag(DAT_COMPLETION_UNSIGNALLED_FLAG, - ep_ptr->param.ep_attr. - request_completion_flags, - completion_flags) || - !dapl_rmr_validate_completion_flag( - DAT_COMPLETION_BARRIER_FENCE_FLAG, - ep_ptr->param.ep_attr.request_completion_flags, - completion_flags)) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); - goto bail; - } - - status = dapl_rmr_cookie_alloc(&ep_ptr->req_buffer, rmr, user_cookie, - &cookie); - if (DAT_SUCCESS != status) - goto bail; - - is_signaled = (completion_flags & DAT_COMPLETION_SUPPRESS_FLAG) ? - FALSE : TRUE; - - /* - * Take reference before posting to avoid race conditions with - * completions - */ - atomic_inc(&ep_ptr->req_count); - - status = dapl_ib_mw_bind(rmr, lmr, ep_ptr, cookie, - lmr_triplet->virtual_address, - lmr_triplet->segment_length, - mem_priv, is_signaled); - if (DAT_SUCCESS != status) { - atomic_dec(&ep_ptr->req_count); - dapl_cookie_dealloc(&ep_ptr->req_buffer, cookie); - goto bail; - } - - atomic_inc(&lmr->lmr_ref_count); - - /* if the RMR was previously bound */ - if (NULL != rmr->lmr) - atomic_dec(&rmr->lmr->lmr_ref_count); - - rmr->param.mem_priv = mem_priv; - rmr->param.lmr_triplet = *lmr_triplet; - rmr->ep = ep_ptr; - rmr->lmr = lmr; - - if (NULL != rmr_context) - *rmr_context = rmr->param.rmr_context; - -bail: - return status; -} - -static inline u32 dapl_rmr_bind_unfuse(struct dapl_rmr *rmr, - struct dapl_ep *ep_ptr, - DAT_RMR_COOKIE user_cookie, - enum dat_completion_flags cflags) -{ - struct dapl_cookie *cookie; - u32 status = DAT_SUCCESS; - boolean_t is_signaled; - - /* - * if the ep in unconnected return an error. IB requires that the - * QP be connected to change a memory window binding since: - * - * - memory window bind operations are WQEs placed on a QP's - * send queue - * - * - QP's only process WQEs on the send queue when the QP is in - * the RTS state - */ - if (DAT_EP_STATE_CONNECTED != ep_ptr->param.ep_state) { - status = - DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep_ptr)); - goto bail1; - } - - /* If the RMR and EP are not in the same PZ, there is an error */ - if (ep_ptr->param.pz_handle != rmr->param.pz_handle) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); - goto bail1; - } - - if (!dapl_rmr_validate_completion_flag(DAT_COMPLETION_SUPPRESS_FLAG, - ep_ptr->param.ep_attr. - request_completion_flags, - cflags) || - !dapl_rmr_validate_completion_flag(DAT_COMPLETION_UNSIGNALLED_FLAG, - ep_ptr->param.ep_attr. - request_completion_flags, - cflags) || - !dapl_rmr_validate_completion_flag( - DAT_COMPLETION_BARRIER_FENCE_FLAG, - ep_ptr->param.ep_attr.request_completion_flags, cflags)) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); - goto bail1; - } - - status = dapl_rmr_cookie_alloc(&ep_ptr->req_buffer, rmr, user_cookie, - &cookie); - if (DAT_SUCCESS != status) - goto bail1; - - is_signaled = (cflags & DAT_COMPLETION_UNSIGNALLED_FLAG) ? FALSE : TRUE; - - /* - * Take reference before posting to avoid race conditions with - * completions - */ - atomic_inc(&ep_ptr->req_count); - - status = dapl_ib_mw_unbind(rmr, ep_ptr, cookie, is_signaled); - if (DAT_SUCCESS != status) { - atomic_dec(&ep_ptr->req_count); - dapl_cookie_dealloc(&ep_ptr->req_buffer, cookie); - goto bail1; - } - - /* if the RMR was previously bound */ - if (NULL != rmr->lmr) - atomic_dec(&rmr->lmr->lmr_ref_count); - - rmr->param.mem_priv = DAT_MEM_PRIV_NONE_FLAG; - rmr->param.lmr_triplet.lmr_context = 0; - rmr->param.lmr_triplet.virtual_address = 0; - rmr->param.lmr_triplet.segment_length = 0; - rmr->ep = ep_ptr; - rmr->lmr = NULL; - -bail1: - return status; -} - -/* - * dapl_rmr_bind - * - * DAPL Requirements Version xxx, 6.6.4.4 - * - * Bind the RMR to the specified memory region within the LMR and - * provide a new rmr_context value. - * - * Input: - * Output: - */ -u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_handle, - const struct dat_lmr_triplet *lmr_triplet, - enum dat_mem_priv_flags mem_priv, DAT_EP_HANDLE ep_handle, - DAT_RMR_COOKIE user_cookie, - enum dat_completion_flags completion_flags, - DAT_RMR_CONTEXT *rmr_context) -{ - struct dapl_rmr *rmr; - struct dapl_ep *ep_ptr; - - if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) - return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); - - if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) - return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); - - rmr = (struct dapl_rmr *)rmr_handle; - ep_ptr = (struct dapl_ep *)ep_handle; - - /* if the rmr should be bound */ - if (0 != lmr_triplet->segment_length) - return dapl_rmr_bind_fuse(rmr, lmr_triplet, mem_priv, ep_ptr, - user_cookie, completion_flags, - rmr_context); - else /* the rmr should be unbound */ - return dapl_rmr_bind_unfuse(rmr, ep_ptr, user_cookie, - completion_flags); -} Index: linux-kernel-rmr/dat-provider/dapl_openib_util.c =================================================================== --- linux-kernel-rmr/dat-provider/dapl_openib_util.c (revision 2483) +++ linux-kernel-rmr/dat-provider/dapl_openib_util.c (working copy) @@ -37,7 +37,6 @@ #include "dapl_openib_util.h" #include "dapl_evd_util.h" #include "dapl_cr_util.h" -#include "dapl_rmr_util.h" #include "dapl_cookie.h" #include "dapl_openib_cm.h" Index: linux-kernel-rmr/dat-provider/dapl_rmr_util.c =================================================================== --- linux-kernel-rmr/dat-provider/dapl_rmr_util.c (revision 2483) +++ linux-kernel-rmr/dat-provider/dapl_rmr_util.c (working copy) @@ -1,82 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -#include "dapl_rmr_util.h" -#include "dapl_ia_util.h" - -struct dapl_rmr *dapl_rmr_alloc(struct dapl_pz *pz) -{ - struct dapl_rmr *rmr; - - /* Allocate LMR */ - rmr = kmalloc(sizeof *rmr, GFP_ATOMIC); - if (!rmr) - return NULL; - - /* zero the structure */ - memset(rmr, 0, sizeof *rmr); - - /* - * initialize the header - */ - rmr->header.provider = pz->header.provider; - rmr->header.magic = DAPL_MAGIC_RMR; - rmr->header.handle_type = DAT_HANDLE_TYPE_RMR; - rmr->header.owner_ia = pz->header.owner_ia; - rmr->header.user_context.as_64 = 0; - rmr->header.user_context.as_ptr = NULL; - dapl_llist_init_entry(&rmr->header.ia_list_entry); - dapl_ia_link_rmr(rmr->header.owner_ia, rmr); - spin_lock_init(&rmr->header.lock); - - /* - * initialize the body - */ - rmr->param.ia_handle = (DAT_IA_HANDLE) pz->header.owner_ia; - rmr->param.pz_handle = (DAT_PZ_HANDLE) pz; - rmr->param.lmr_triplet.lmr_context = 0; - rmr->param.lmr_triplet.pad = 0; - rmr->param.lmr_triplet.virtual_address = 0; - rmr->param.lmr_triplet.segment_length = 0; - - rmr->param.mem_priv = 0; - rmr->pz = pz; - rmr->lmr = NULL; - - return rmr; -} - -void dapl_rmr_dealloc(struct dapl_rmr *rmr) -{ - /* reset magic to prevent reuse */ - rmr->header.magic = DAPL_MAGIC_INVALID; - - dapl_ia_unlink_rmr(rmr->header.owner_ia, rmr); - /* no need to destroy rmr->header.lock */ - - kfree(rmr); -} Index: linux-kernel-rmr/dat-provider/dapl_rmr.c =================================================================== --- linux-kernel-rmr/dat-provider/dapl_rmr.c (revision 0) +++ linux-kernel-rmr/dat-provider/dapl_rmr.c (revision 0) @@ -0,0 +1,461 @@ +/* + * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * + * This Software is licensed under one of the following licenses: + * + * 1) under the terms of the "Common Public License 1.0" a copy of which is + * available from the Open Source Initiative, see + * http://www.opensource.org/licenses/cpl.php. + * + * 2) under the terms of the "The BSD License" a copy of which is + * available from the Open Source Initiative, see + * http://www.opensource.org/licenses/bsd-license.php. + * + * 3) under the terms of the "GNU General Public License (GPL) Version 2" a + * copy of which is available from the Open Source Initiative, see + * http://www.opensource.org/licenses/gpl-license.php. + * + * Licensee has the right to choose one of the above licenses. + * + * Redistributions of source code must retain the above copyright + * notice and one of the license notices. + * + * Redistributions in binary form must reproduce both the above copyright + * notice, one of the license notices in the documentation + * and/or other materials provided with the distribution. + */ + +/* + * $Id$ + */ + +#include "dapl.h" +#include "dapl_ep_util.h" +#include "dapl_ia_util.h" +#include "dapl_mr_util.h" +#include "dapl_hash.h" +#include "dapl_cookie.h" +#include "dapl_openib_util.h" + +static inline boolean_t +dapl_rmr_validate_completion_flag(enum dat_completion_flags mask, + enum dat_completion_flags allow, + enum dat_completion_flags request) +{ + if ((mask & request) && !(mask & allow)) + return FALSE; + else + return TRUE; +} + + +static struct dapl_rmr *dapl_rmr_alloc(struct dapl_pz *pz) +{ + struct dapl_rmr *rmr; + + /* Allocate LMR */ + rmr = kmalloc(sizeof *rmr, GFP_ATOMIC); + if (!rmr) + return NULL; + + /* zero the structure */ + memset(rmr, 0, sizeof *rmr); + + /* + * initialize the header + */ + rmr->header.provider = pz->header.provider; + rmr->header.magic = DAPL_MAGIC_RMR; + rmr->header.handle_type = DAT_HANDLE_TYPE_RMR; + rmr->header.owner_ia = pz->header.owner_ia; + rmr->header.user_context.as_64 = 0; + rmr->header.user_context.as_ptr = NULL; + dapl_llist_init_entry(&rmr->header.ia_list_entry); + dapl_ia_link_rmr(rmr->header.owner_ia, rmr); + spin_lock_init(&rmr->header.lock); + + /* + * initialize the body + */ + rmr->param.ia_handle = (DAT_IA_HANDLE)pz->header.owner_ia; + rmr->param.pz_handle = (DAT_PZ_HANDLE)pz; + rmr->param.lmr_triplet.lmr_context = 0; + rmr->param.lmr_triplet.pad = 0; + rmr->param.lmr_triplet.virtual_address = 0; + rmr->param.lmr_triplet.segment_length = 0; + + rmr->param.mem_priv = 0; + rmr->pz = pz; + rmr->lmr = NULL; + + return rmr; +} + +static void dapl_rmr_dealloc(struct dapl_rmr *rmr) +{ + /* reset magic to prevent reuse */ + rmr->header.magic = DAPL_MAGIC_INVALID; + + dapl_ia_unlink_rmr(rmr->header.owner_ia, rmr); + /* no need to destroy rmr->header.lock */ + + kfree(rmr); +} + +static inline u32 dapl_rmr_bind_fuse(struct dapl_rmr *rmr, + const struct dat_lmr_triplet *lmr_triplet, + enum dat_mem_priv_flags mem_priv, + struct dapl_ep *ep_ptr, + DAT_RMR_COOKIE user_cookie, + enum dat_completion_flags completion_flags, + DAT_RMR_CONTEXT *rmr_context) +{ + struct dapl_lmr *lmr; + struct dapl_cookie *cookie; + u32 status; + boolean_t is_signaled; + + status = dapl_hash_search(rmr->header.owner_ia->hca_ptr->lmr_hash_table, + lmr_triplet->lmr_context, + (DAPL_HASH_DATA *) &lmr); + if (DAT_SUCCESS != status) { + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); + goto bail; + } + + /* + * if the ep in unconnected return an error. IB requires that the + * QP be connected to change a memory window binding since: + * + * - memory window bind operations are WQEs placed on a QP's + * send queue + * + * - QP's only process WQEs on the send queue when the QP is in + * the RTS state + */ + if (DAT_EP_STATE_CONNECTED != ep_ptr->param.ep_state) { + status = + DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep_ptr)); + goto bail; + } + + if (FALSE == + dapl_mr_bounds_check(dapl_mr_get_address + (lmr->param.region_desc, lmr->param.mem_type), + lmr->param.length, + lmr_triplet->virtual_address, + lmr_triplet->segment_length)) { + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); + goto bail; + } + + /* If the LMR, RMR, and EP are not in the same PZ, there is an error */ + if ((ep_ptr->param.pz_handle != lmr->param.pz_handle) || + (ep_ptr->param.pz_handle != rmr->param.pz_handle)) { + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); + goto bail; + } + + if (!dapl_rmr_validate_completion_flag(DAT_COMPLETION_SUPPRESS_FLAG, + ep_ptr->param.ep_attr. + request_completion_flags, + completion_flags) || + !dapl_rmr_validate_completion_flag(DAT_COMPLETION_UNSIGNALLED_FLAG, + ep_ptr->param.ep_attr. + request_completion_flags, + completion_flags) || + !dapl_rmr_validate_completion_flag( + DAT_COMPLETION_BARRIER_FENCE_FLAG, + ep_ptr->param.ep_attr.request_completion_flags, + completion_flags)) { + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); + goto bail; + } + + status = dapl_rmr_cookie_alloc(&ep_ptr->req_buffer, rmr, user_cookie, + &cookie); + if (DAT_SUCCESS != status) + goto bail; + + is_signaled = (completion_flags & DAT_COMPLETION_SUPPRESS_FLAG) ? + FALSE : TRUE; + + /* + * Take reference before posting to avoid race conditions with + * completions + */ + atomic_inc(&ep_ptr->req_count); + + status = dapl_ib_mw_bind(rmr, lmr, ep_ptr, cookie, + lmr_triplet->virtual_address, + lmr_triplet->segment_length, + mem_priv, is_signaled); + if (DAT_SUCCESS != status) { + atomic_dec(&ep_ptr->req_count); + dapl_cookie_dealloc(&ep_ptr->req_buffer, cookie); + goto bail; + } + + atomic_inc(&lmr->lmr_ref_count); + + /* if the RMR was previously bound */ + if (NULL != rmr->lmr) + atomic_dec(&rmr->lmr->lmr_ref_count); + + rmr->param.mem_priv = mem_priv; + rmr->param.lmr_triplet = *lmr_triplet; + rmr->ep = ep_ptr; + rmr->lmr = lmr; + + if (NULL != rmr_context) + *rmr_context = rmr->param.rmr_context; + +bail: + return status; +} + +static inline u32 dapl_rmr_bind_unfuse(struct dapl_rmr *rmr, + struct dapl_ep *ep_ptr, + DAT_RMR_COOKIE user_cookie, + enum dat_completion_flags cflags) +{ + struct dapl_cookie *cookie; + u32 status = DAT_SUCCESS; + boolean_t is_signaled; + + /* + * if the ep in unconnected return an error. IB requires that the + * QP be connected to change a memory window binding since: + * + * - memory window bind operations are WQEs placed on a QP's + * send queue + * + * - QP's only process WQEs on the send queue when the QP is in + * the RTS state + */ + if (DAT_EP_STATE_CONNECTED != ep_ptr->param.ep_state) { + status = DAT_ERROR(DAT_INVALID_STATE, + dapl_ep_state_subtype(ep_ptr)); + goto bail1; + } + + /* If the RMR and EP are not in the same PZ, there is an error */ + if (ep_ptr->param.pz_handle != rmr->param.pz_handle) { + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); + goto bail1; + } + + if (!dapl_rmr_validate_completion_flag(DAT_COMPLETION_SUPPRESS_FLAG, + ep_ptr->param.ep_attr. + request_completion_flags, + cflags) || + !dapl_rmr_validate_completion_flag(DAT_COMPLETION_UNSIGNALLED_FLAG, + ep_ptr->param.ep_attr. + request_completion_flags, + cflags) || + !dapl_rmr_validate_completion_flag( + DAT_COMPLETION_BARRIER_FENCE_FLAG, + ep_ptr->param.ep_attr.request_completion_flags, cflags)) { + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); + goto bail1; + } + + status = dapl_rmr_cookie_alloc(&ep_ptr->req_buffer, rmr, user_cookie, + &cookie); + if (DAT_SUCCESS != status) + goto bail1; + + is_signaled = (cflags & DAT_COMPLETION_UNSIGNALLED_FLAG) ? FALSE : TRUE; + + /* + * Take reference before posting to avoid race conditions with + * completions + */ + atomic_inc(&ep_ptr->req_count); + + status = dapl_ib_mw_unbind(rmr, ep_ptr, cookie, is_signaled); + if (DAT_SUCCESS != status) { + atomic_dec(&ep_ptr->req_count); + dapl_cookie_dealloc(&ep_ptr->req_buffer, cookie); + goto bail1; + } + + /* if the RMR was previously bound */ + if (NULL != rmr->lmr) + atomic_dec(&rmr->lmr->lmr_ref_count); + + rmr->param.mem_priv = DAT_MEM_PRIV_NONE_FLAG; + rmr->param.lmr_triplet.lmr_context = 0; + rmr->param.lmr_triplet.virtual_address = 0; + rmr->param.lmr_triplet.segment_length = 0; + rmr->ep = ep_ptr; + rmr->lmr = NULL; + +bail1: + return status; +} + +/* + * dapl_rmr_bind + * + * DAPL Requirements Version xxx, 6.6.4.4 + * + * Bind the RMR to the specified memory region within the LMR and + * provide a new rmr_context value. + * + * Input: + * Output: + */ +u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_handle, + const struct dat_lmr_triplet *lmr_triplet, + enum dat_mem_priv_flags mem_priv, DAT_EP_HANDLE ep_handle, + DAT_RMR_COOKIE user_cookie, + enum dat_completion_flags completion_flags, + DAT_RMR_CONTEXT *rmr_context) +{ + struct dapl_rmr *rmr; + struct dapl_ep *ep_ptr; + + if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) + return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); + + if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) + return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); + + rmr = (struct dapl_rmr *)rmr_handle; + ep_ptr = (struct dapl_ep *)ep_handle; + + /* if the rmr should be bound */ + if (0 != lmr_triplet->segment_length) + return dapl_rmr_bind_fuse(rmr, lmr_triplet, mem_priv, ep_ptr, + user_cookie, completion_flags, + rmr_context); + else /* the rmr should be unbound */ + return dapl_rmr_bind_unfuse(rmr, ep_ptr, user_cookie, + completion_flags); +} + +/* + * dapl_rmr_create + * + * Create a remote memory region for the specified protection zone + * + * Input: + * pz_handle + * + * Output: + * rmr_handle + * + * Returns: + * DAT_SUCCESS + * DAT_INSUFFICIENT_RESOURCES + * DAT_INVALID_PARAMETER + */ +u32 dapl_rmr_create(DAT_PZ_HANDLE pz_handle, DAT_RMR_HANDLE *rmr_handle) +{ + struct dapl_pz *pz; + struct dapl_rmr *rmr; + u32 status = DAT_SUCCESS; + + if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); + goto bail; + } + + pz = (struct dapl_pz *)pz_handle; + + rmr = dapl_rmr_alloc(pz); + + if (rmr == NULL) { + status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, + DAT_RESOURCE_MEMORY); + goto bail; + } + + status = dapl_ib_mw_alloc(rmr); + + if (status != DAT_SUCCESS) { + dapl_rmr_dealloc(rmr); + status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, + DAT_RESOURCE_MEMORY_REGION); + goto bail; + } + + atomic_inc(&pz->pz_ref_count); + + *rmr_handle = rmr; + +bail: + return status; +} + +/* + * dapl_rmr_free + * + * Destroy an instance of the Remote Memory Region + * + * Input: + * rmr_handle + * + * Output: + * none + * + * Returns: + * DAT_SUCCESS + * DAT_INVALID_PARAMETER + */ +u32 dapl_rmr_free(DAT_RMR_HANDLE rmr_handle) +{ + struct dapl_rmr *rmr; + u32 status = DAT_SUCCESS; + + if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) { + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); + goto bail; + } + + rmr = (struct dapl_rmr *)rmr_handle; + + /* + * If the user did not perform an unbind op, release + * counts here. + */ + if (rmr->param.lmr_triplet.virtual_address != 0) { + atomic_dec(&rmr->lmr->lmr_ref_count); + rmr->param.lmr_triplet.virtual_address = 0; + } + + status = dapl_ib_mw_free(rmr); + + if (status != DAT_SUCCESS) + goto bail; + + atomic_dec(&rmr->pz->pz_ref_count); + + dapl_rmr_dealloc(rmr); + +bail: + return status; +} + +u32 dapl_rmr_query(DAT_RMR_HANDLE rmr_handle, struct dat_rmr_param *rmr_param) +{ + struct dapl_rmr *rmr; + u32 status = DAT_SUCCESS; + + if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) { + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); + goto bail; + } + if (NULL == rmr_param) { + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); + goto bail; + } + + rmr = (struct dapl_rmr *)rmr_handle; + + memcpy(rmr_param, &rmr->param, sizeof *rmr_param); + +bail: + return status; +} Index: linux-kernel-rmr/dat-provider/dapl_rmr_util.h =================================================================== --- linux-kernel-rmr/dat-provider/dapl_rmr_util.h (revision 2483) +++ linux-kernel-rmr/dat-provider/dapl_rmr_util.h (working copy) @@ -1,52 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#ifndef DAPL_RMR_UTIL_H -#define DAPL_RMR_UTIL_H - -#include "dapl_mr_util.h" - -extern struct dapl_rmr *dapl_rmr_alloc(struct dapl_pz *pz); - -extern void dapl_rmr_dealloc(struct dapl_rmr *rmr); - -static inline boolean_t -dapl_rmr_validate_completion_flag(enum dat_completion_flags mask, - enum dat_completion_flags allow, - enum dat_completion_flags request) -{ - if ((mask & request) && !(mask & allow)) - return FALSE; - else - return TRUE; -} - -#endif /* DAPL_RMR_UTIL_H */ Index: linux-kernel-rmr/patches/alt_dat_provider_makefile =================================================================== --- linux-kernel-rmr/patches/alt_dat_provider_makefile (revision 2483) +++ linux-kernel-rmr/patches/alt_dat_provider_makefile (working copy) @@ -16,7 +16,6 @@ PROVIDER_MODULES := \ dapl_evd_kquery \ dapl_evd_kcreate \ dapl_evd_modify_upcall \ - dapl_lmr_kcreate \ dapl_cookie \ dapl_cno_util \ dapl_cr_accept \ @@ -61,11 +60,7 @@ PROVIDER_MODULES := \ dapl_ia_query \ dapl_ia_util \ dapl_llist \ - dapl_lmr_free \ - dapl_lmr_query \ - dapl_lmr_sync_rdma_read \ - dapl_lmr_sync_rdma_write \ - dapl_lmr_util \ + dapl_lmr \ dapl_mr_util \ dapl_provider \ dapl_sp_util \ @@ -73,16 +68,9 @@ PROVIDER_MODULES := \ dapl_psp_create_any \ dapl_psp_free \ dapl_psp_query \ - dapl_pz_create \ - dapl_pz_free \ - dapl_pz_query \ - dapl_pz_util \ + dapl_pz \ dapl_ring_buffer_util \ - dapl_rmr_bind \ - dapl_rmr_create \ - dapl_rmr_free \ - dapl_rmr_query \ - dapl_rmr_util \ + dapl_rmr \ dapl_rsp_create \ dapl_rsp_free \ dapl_rsp_query \ @@ -103,7 +91,7 @@ PROVIDER_OBJS := $(foreach s, $(PROVIDER KDAPL_DEBUG = 1 ifeq (1,$(KDAPL_DEBUG)) EXTRA_CFLAGS += -O0 -g - EXTRA_CFLAGS += -DDAPL_DBG # -DDAPL_DBG_IO_TRC + EXTRA_CFLAGS += -DDAPL_DBG endif EXTRA_CFLAGS += -DDAPL_ATS -Idrivers/infiniband/include -I$(obj)/../dat -I$(obj)/../patches/ @@ -121,4 +109,4 @@ default: endif clean: - rm -f *.o *.ko + rm -f *.o *.ko .*.d .*.o.tmp *.mod.* .tmp_versions .*.cmd Index: linux-kernel-rmr/patches/Makefile =================================================================== --- linux-kernel-rmr/patches/Makefile (revision 2483) +++ linux-kernel-rmr/patches/Makefile (working copy) @@ -20,4 +20,4 @@ default: endif clean: - rm -f *.o *.ko + rm -rf *.o *.ko .*.d .*.o.tmp *.mod.* .tmp_versions .*.cmd From mshefty at ichips.intel.com Wed May 25 11:03:02 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Wed, 25 May 2005 11:03:02 -0700 Subject: [openib-general] Re: Latest CM and retransmissions In-Reply-To: <001601c56152$5c0940d0$9b5aa8c0@infiniconsys.com> References: <001601c56152$5c0940d0$9b5aa8c0@infiniconsys.com> Message-ID: <4294BDD6.8080908@ichips.intel.com> Fab Tillier wrote: > Why not just delay the transition into timewait until the DREP is received > or the DREQ times out? By the user destroying the cm_id, they're indicating that they don't care about the DREP. My thought was to let the destroy occur immediately, rather than delaying the process until the DREQ timed out, which could be on the order of a few seconds. - Sean From eitan at mellanox.co.il Wed May 25 11:45:21 2005 From: eitan at mellanox.co.il (Eitan Zahavi) Date: Wed, 25 May 2005 21:45:21 +0300 Subject: [openib-general] RE: OpenSM Routing Scalability Proposal Message-ID: <506C3D7B14CDD411A52C00025558DED607C3020A@mtlex01.yok.mtl.com> Hi Hal, All your points are valid. Especially the ones regarding the incremental algorithm for routing. Please see below. Eitan Zahavi Design Technology Director Mellanox Technologies LTD Tel:+972-4-9097208 Fax:+972-4-9593245 P.O. Box 586 Yokneam 20692 ISRAEL > -----Original Message----- > From: Hal Rosenstock [mailto:halr at voltaire.com] > Sent: Wednesday, May 25, 2005 5:53 PM > To: Eitan Zahavi > Cc: 'openib-general at openib.org' > Subject: Re: [openib-general] RE: OpenSM Routing Scalability Proposal > > Hi Eitan, > > On Tue, 2005-05-17 at 10:53, Eitan Zahavi wrote: > > Hi All, > > > > This is an updated proposal document for your comments. > > I finally got a chance to read this. Some comments below. > > > The main change is in describing the need for preserving enough data > > to enable incremental routing algorithm. > > I think incremental can help but presents some new issues. [EZ] Yes very true. I do not have a full algorithm in place that will cover all possible cases. > > > So the actual proposal is to implement the algorithm described in > > section 4.3. > > 4.1 (min hop) and 4.2 (up/down) are already implemented, right ? [EZ] No they are not. The current implementation uses MinHop tables etc. > > > EZ <> > > It seems like there are 2 parts to 4.3: > 1. Min hop table per leaf switch rather than per LID > What are the savings for this ? Seems like in terms of memory, this is > something like a divisor of L times the number of LIDs per HCA port. [EZ] Yes. > Of course, switch port 0s on non leaf switches need to be accomodated. [EZ] True. > > 2. Incremental routing (5) > a. Subcase of 5 where there is no other link between 2 adjacent > switches. Is another way of stating this, examine next hop switches to > see if there is a path between the 2 original switches and keep > expanding the depth until 1 is found ? Couldn't this be worse from a > compute standpoint than rerouting everything depending on the topology > (the likelihood of another path between the 2 original switches) ? [EZ] If one knows which ports have changed this will be faster then full recalc. > > b. 5 asks "How do we support topology changes line moving an HCA from > one Switch to another?" Also, what about a link moving from one switch > to another ? It seems that link down is handled, but nothing is done on > a link up. Doesn't there need to be incremental defined for links being > added ? [EZ] Yes. This is not even close to full algorithm. > > c. Also, with incremental routing, it's unclear to me how the paths > found would compare with the ones which would be determined from the > full algorithm (from scratch). Also, would there be some point at which > the full routing would be retriggered ? [EZ] Good point. > > d. Clearly, there are end node responsibilities here as well (whether > this is done incrementally or fully or something else). [EZ] Not sure what you mean. > > 3. Persistency (6) > a. Full LFT storage (6.1) This presumes that the determination of a > topology change upon discovery is cheaper computationally than running > the routing. Has this been proven ? (I hope this is the case). [EZ] Comparing two graphs is O(Links). > b. Root nodes storage (6.2) Are the root nodes determined by the routing > or supplied to the routing ? Are they different for unicast and > multicast ? [EZ] Both is true. If not provided by human extracted using heuristics. > > -- Hal -------------- next part -------------- An HTML attachment was scrubbed... URL: From halr at voltaire.com Wed May 25 12:04:22 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 25 May 2005 15:04:22 -0400 Subject: [openib-general] Re: Latest CM and retransmissions In-Reply-To: <4294B86F.4070101@ichips.intel.com> References: <1116952775.4490.53.camel@hal.voltaire.com> <42935A70.4030003@ichips.intel.com> <1116954610.4490.73.camel@hal.voltaire.com> <4294A835.8000802@ichips.intel.com> <1117040260.7724.52.camel@hal.voltaire.com> <4294B86F.4070101@ichips.intel.com> Message-ID: <1117047861.7724.345.camel@hal.voltaire.com> On Wed, 2005-05-25 at 13:39, Sean Hefty wrote: > >>Do you know if the code that issues the DREQ destroys the cm_id immediately > >>afterwards? > > > > It likely is. I was going to ask about this yesterday. > > If a client destroys the cm_id immediately after sending a DREQ (before the > DREP is received), the CM will transition the cm_id directly into the > timewait state. I've just committed a change to the CM to cancel the DREQ > if the cm_id is destroyed. Note that this won't result in the DREP matching > with the DREQ, since the cm_id has been destroyed, but should prevent the > DREQ from being resent, if this is indeed what is happening. Can you pull > the CM from 2485 and retest Yes, that's better :-) Only 1 DREQ/DREP. Thanks. -- Hal From tduffy at sun.com Wed May 25 12:10:12 2005 From: tduffy at sun.com (Tom Duffy) Date: Wed, 25 May 2005 12:10:12 -0700 Subject: [openib-general] [Fwd: RE: Use of dapl_ring_buffer in KDAPL !] Message-ID: <1117048212.12010.3.camel@duffman> Moving this discussion on list since more people might know of a way to do this with already existing linux primitives. -tduffy -------------- next part -------------- An embedded message was scrubbed... From: James Lentini Subject: RE: Use of dapl_ring_buffer in KDAPL ! Date: Wed, 25 May 2005 15:05:42 -0400 (EDT) Size: 3630 URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From halr at voltaire.com Wed May 25 12:29:47 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 25 May 2005 15:29:47 -0400 Subject: [openib-general] RE: OpenSM Routing Scalability Proposal In-Reply-To: <506C3D7B14CDD411A52C00025558DED607C3020A@mtlex01.yok.mtl.com> References: <506C3D7B14CDD411A52C00025558DED607C3020A@mtlex01.yok.mtl.com> Message-ID: <1117049387.7724.393.camel@hal.voltaire.com> On Wed, 2005-05-25 at 14:45, Eitan Zahavi wrote: > Hi Hal, > > All your points are valid. Especially the ones regarding the > incremental algorithm for routing. Please see below. One other question: Is there an impact of LMC > 0 on this ? > Eitan Zahavi > Design Technology Director > Mellanox Technologies LTD > Tel:+972-4-9097208 > Fax:+972-4-9593245 > P.O. Box 586 Yokneam 20692 ISRAEL > > > > --Original Message-- > > From: Hal Rosenstock [mailto:halr at voltaire.com] > > Sent: Wednesday, May 25, 2005 5:53 PM > > To: Eitan Zahavi > > Cc: 'openib-general at openib.org' > > Subject: Re: [openib-general] RE: OpenSM Routing Scalability > Proposal > > > > Hi Eitan, > > > > On Tue, 2005-05-17 at 10:53, Eitan Zahavi wrote: > > > Hi All, > > > > > > This is an updated proposal document for your comments. > > > > I finally got a chance to read this. Some comments below. > > > > > The main change is in describing the need for preserving enough > data > > > to enable incremental routing algorithm. > > > > I think incremental can help but presents some new issues. > [EZ] Yes very true. I do not have a full algorithm in place that will > cover all possible cases. > > > > > So the actual proposal is to implement the algorithm described in > > > section 4.3. > > > > 4.1 (min hop) and 4.2 (up/down) are already implemented, right ? > [EZ] No they are not. The current implementation uses MinHop tables > etc. > I'm not sure I'm following you. Are you saying min hop is implemented and up/down isn't (just analyzed) ? > > > EZ <> > > > > It seems like there are 2 parts to 4.3: > > 1. Min hop table per leaf switch rather than per LID > > What are the savings for this ? Seems like in terms of memory, this > is > > something like a divisor of L times the number of LIDs per HCA port. > [EZ] Yes. > > Of course, switch port 0s on non leaf switches need to be > accomodated. > [EZ] True. > > > > 2. Incremental routing (5) > > a. Subcase of 5 where there is no other link between 2 adjacent > > switches. Is another way of stating this, examine next hop switches > to > > see if there is a path between the 2 original switches and keep > > expanding the depth until 1 is found ? Couldn't this be worse from a > > compute standpoint than rerouting everything depending on the > topology > > (the likelihood of another path between the 2 original switches) ? > [EZ] If one knows which ports have changed this will be faster then > full recalc. Sure but if the depth keeps expanding because no path is found between the switches which lost a trunk link between them, then isn't the calculation done on an ever expanding horizon of switches ? That was the case I was referring to. > > b. 5 asks "How do we support topology changes line moving an HCA > from > > one Switch to another?" Also, what about a link moving from one > switch > > to another ? It seems that link down is handled, but nothing is done > on > > a link up. Doesn't there need to be incremental defined for links > being > > added ? > [EZ] Yes. This is not even close to full algorithm. > > > > c. Also, with incremental routing, it's unclear to me how the paths > > found would compare with the ones which would be determined from the > > full algorithm (from scratch). Also, would there be some point at > which > > the full routing would be retriggered ? > [EZ] Good point. > > > > d. Clearly, there are end node responsibilities here as well > (whether > > this is done incrementally or fully or something else). > [EZ] Not sure what you mean. I'm referring to path changes and their implications on connections. > > 3. Persistency (6) > > a. Full LFT storage (6.1) This presumes that the determination of a > > topology change upon discovery is cheaper computationally than > running > > the routing. Has this been proven ? (I hope this is the case). > [EZ] Comparing two graphs is O(Links). OK. I hope not twice the memory is needed for this. > > b. Root nodes storage (6.2) Are the root nodes determined by the > routing > > or supplied to the routing ? Are they different for unicast and > > multicast ? > [EZ] Both is true. If not provided by human extracted using > heuristics. By both is true, do you mean that the root nodes can either be determined by the routing or supplied to the routing ? Are the unicast and multicast roots the same or different ? -- Hal From eitan at mellanox.co.il Wed May 25 12:49:10 2005 From: eitan at mellanox.co.il (Eitan Zahavi) Date: Wed, 25 May 2005 22:49:10 +0300 Subject: [openib-general] RE: OpenSM Routing Scalability Proposal Message-ID: <506C3D7B14CDD411A52C00025558DED607C3020E@mtlex01.yok.mtl.com> > > On Wed, 2005-05-25 at 14:45, Eitan Zahavi wrote: > > Hi Hal, > > > > All your points are valid. Especially the ones regarding the > > incremental algorithm for routing. Please see below. > > One other question: > Is there an impact of LMC > 0 on this ? [EZ] If LMC > 0 then the proposed algorithm for calculating min hop tables (step1) is going to be even faster then today implementation. > > > > > > > Hi Eitan, > > > > > > On Tue, 2005-05-17 at 10:53, Eitan Zahavi wrote: > > > > Hi All, > > > > > > > > This is an updated proposal document for your comments. > > > > > > I finally got a chance to read this. Some comments below. > > > > > > > The main change is in describing the need for preserving enough > > data > > > > to enable incremental routing algorithm. > > > > > > I think incremental can help but presents some new issues. > > [EZ] Yes very true. I do not have a full algorithm in place that will > > cover all possible cases. > > > > > > > So the actual proposal is to implement the algorithm described in > > > > section 4.3. > > > > > > 4.1 (min hop) and 4.2 (up/down) are already implemented, right ? > > [EZ] No they are not. The current implementation uses MinHop tables > > etc. > > > > I'm not sure I'm following you. Are you saying min hop is implemented > and up/down isn't (just analyzed) ? [EZ] No - I use the term min hop for the first stage of the routing. Today this first stage generate a different kind of table then the proposed and it does so using a different traversal algorithm. > > > > > EZ <> > > > > > > It seems like there are 2 parts to 4.3: > > > 1. Min hop table per leaf switch rather than per LID > > > What are the savings for this ? Seems like in terms of memory, this > > is > > > something like a divisor of L times the number of LIDs per HCA port. > > [EZ] Yes. > > > Of course, switch port 0s on non leaf switches need to be > > accomodated. > > [EZ] True. > > > > > > 2. Incremental routing (5) > > > a. Subcase of 5 where there is no other link between 2 adjacent > > > switches. Is another way of stating this, examine next hop switches > > to > > > see if there is a path between the 2 original switches and keep > > > expanding the depth until 1 is found ? Couldn't this be worse from a > > > compute standpoint than rerouting everything depending on the > > topology > > > (the likelihood of another path between the 2 original switches) ? > > [EZ] If one knows which ports have changed this will be faster then > > full recalc. > > Sure but if the depth keeps expanding because no path is found between > the switches which lost a trunk link between them, then isn't the > calculation done on an ever expanding horizon of switches ? That was the > case I was referring to. [EZ] But not all switches needs to be recomputed. > > > > b. 5 asks "How do we support topology changes line moving an HCA > > from > > > one Switch to another?" Also, what about a link moving from one > > switch > > > to another ? It seems that link down is handled, but nothing is done > > on > > > a link up. Doesn't there need to be incremental defined for links > > being > > > added ? > > [EZ] Yes. This is not even close to full algorithm. > > > > > > c. Also, with incremental routing, it's unclear to me how the paths > > > found would compare with the ones which would be determined from the > > > full algorithm (from scratch). Also, would there be some point at > > which > > > the full routing would be retriggered ? > > [EZ] Good point. > > > > > > d. Clearly, there are end node responsibilities here as well > > (whether > > > this is done incrementally or fully or something else). > > [EZ] Not sure what you mean. > > I'm referring to path changes and their implications on connections. [EZ] I assume the QP has already timed out. > > > > 3. Persistency (6) > > > a. Full LFT storage (6.1) This presumes that the determination of a > > > topology change upon discovery is cheaper computationally than > > running > > > the routing. Has this been proven ? (I hope this is the case). > > [EZ] Comparing two graphs is O(Links). > > OK. I hope not twice the memory is needed for this. [EZ] The memory involved with keeping the connectivity is small compared to the routing data and various other tables (PKey SL2VL...) > > > > b. Root nodes storage (6.2) Are the root nodes determined by the > > routing > > > or supplied to the routing ? Are they different for unicast and > > > multicast ? > > [EZ] Both is true. If not provided by human extracted using > > heuristics. > > By both is true, do you mean that the root nodes can either be > determined by the routing or supplied to the routing ? [EZ] OpenSM can take roots from file or calculate them using some heuristics. > > Are the unicast and multicast roots the same or different ? [EZ] Multicast roots are calculated only. If you have a small group then the roots will be the lowest level in the tree that fits all the members. > > -- Hal -------------- next part -------------- An HTML attachment was scrubbed... URL: From halr at voltaire.com Wed May 25 12:50:46 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 25 May 2005 15:50:46 -0400 Subject: [openib-general] [PATCH] [TRIVIAL] [kdapl CM] Change messages to be consistent with routine names Message-ID: <1117050380.7724.410.camel@hal.voltaire.com> [kdapl CM] Change messages to be consistent with routine names Signed-off-by: Hal Rosenstock Index: dapl_openib_cm.c =================================================================== --- dapl_openib_cm.c (revision 2486) +++ dapl_openib_cm.c (working copy) @@ -78,7 +78,7 @@ int status; if (conn->ep->qp_handle == NULL) { - dapl_dbg_log(DAPL_DBG_TYPE_ERR, " do_rep_recv: invalid qp " + dapl_dbg_log(DAPL_DBG_TYPE_ERR, " dapl_rep_recv: invalid qp " "handle\n"); goto disc; } @@ -86,7 +86,7 @@ /* First, transition QP to RTR */ status = dapl_modify_qp_state_to_rtr(conn->cm_id, conn->ep->qp_handle); if (status) { - dapl_dbg_log(DAPL_DBG_TYPE_ERR, " do_rep_recv: could not " + dapl_dbg_log(DAPL_DBG_TYPE_ERR, " dapl_rep_recv: could not " "modify QP state to RTR status %d\n", status); goto disc; } @@ -94,14 +94,14 @@ /* Now, transition QP to RTS */ status = dapl_modify_qp_state_to_rts(conn->cm_id, conn->ep->qp_handle); if (status) { - dapl_dbg_log(DAPL_DBG_TYPE_ERR, " do_rep_recv: could not " + dapl_dbg_log(DAPL_DBG_TYPE_ERR, " dapl_rep_recv: could not " "modify QP state to RTS status %d\n", status); goto disc; } status = ib_send_cm_rtu(conn->cm_id, NULL, 0); if (status) { - dapl_dbg_log(DAPL_DBG_TYPE_ERR, " do_rep_recv: ib_send_cm_rtu " + dapl_dbg_log(DAPL_DBG_TYPE_ERR, " dapl_rep_recv: ib_send_cm_rtu " "failed: %d\n", status); goto disc; } @@ -181,7 +181,7 @@ status = dapl_modify_qp_state_to_rts(conn->cm_id, conn->ep->qp_handle); if (status) { - dapl_dbg_log(DAPL_DBG_TYPE_ERR, " do_rtu_recv: could not " + dapl_dbg_log(DAPL_DBG_TYPE_ERR, " dapl_rtu_recv: could not " "modify QP state to RTS status %d\n", status); goto reject; } From roland at topspin.com Wed May 25 13:11:05 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 25 May 2005 13:11:05 -0700 Subject: [openib-general] Re: [PATCH] pingpong test: zero-initialize all attributes In-Reply-To: <20050525134934.GJ27428@mellanox.co.il> (Michael S. Tsirkin's message of "Wed, 25 May 2005 16:49:34 +0300") References: <20050525134934.GJ27428@mellanox.co.il> Message-ID: <524qcrqd5y.fsf@topspin.com> Thanks, good catch. I fixed your patch so that it compiles (including with gcc-2.95) and committed it. - R. From mst at mellanox.co.il Wed May 25 13:21:28 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Wed, 25 May 2005 23:21:28 +0300 Subject: [openib-general] Re: Re: [PATCH] (repost) sdp: replace mlock with get_user_pages In-Reply-To: <20050513100149.B24123@topspin.com> References: <20050505110158.GH30653@mellanox.co.il> <20050506175415.I26823@topspin.com> <20050510143232.GF2474@mellanox.co.il> <20050512151051.C23734@topspin.com> <52fywst4z3.fsf@topspin.com> <20050513135145.GA320@mellanox.co.il> <20050513100149.B24123@topspin.com> Message-ID: <20050525202128.GA7463@mellanox.co.il> Quoting r. Libor Michalek : > Subject: Re: Re: [PATCH] (repost) sdp: replace mlock with get_user_pages > > On Fri, May 13, 2005 at 04:51:45PM +0300, Michael S. Tsirkin wrote: > > Quoting r. Roland Dreier : > > > > > > Libor> Always call aio_complete() immediately when > > > Libor> iocb_complete() is called, and only spawn the work thread > > > Libor> to unlock the memory after the aio_complete() call. The > > > Libor> patch is below. > > > > Libor, I dont think its a good idea - this will break other assumptions, > > like the assumption that the task mm isnt destroyed before we unlock > > the memory. > > That's a good point. > > > > Another alternative would be to always complete aios asynchronously, > > > which should preserve the order. I guess this would hurt latency for > > > small ios... > > > > To avoid hurting latency, lets count the number of outstanding > > asynchronous AIOs, and if there are asynchronous AIOs complete > > all of them asynchronously. > > > > Does this make sense? > > Yes, except that the current iocb code does not reference individual > sockets anywhere, and do_iocb_complete would have to be the function > which decremented the per connection counter of outstanding AIOs. Also, > since we don't have to do get_user_pages a second time on the send > path, this would only need to be done on the recv path. > > -Libor > I thought about this some more: what if we set users to 1 before releasing the irq spinlock, and call sdp_conn_unlock in thread context after completing the aio iocb? Any synchronous transfer would then wait till socket is unloacked. -- MST - Michael S. Tsirkin From jlentini at netapp.com Wed May 25 14:13:11 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 25 May 2005 17:13:11 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: consolidate rmr files into one In-Reply-To: <1117043658.29341.6.camel@duffman> References: <1117043658.29341.6.camel@duffman> Message-ID: Committed in revision 2488. On Wed, 25 May 2005, Tom Duffy wrote: tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> Index: linux-kernel-rmr/dat-provider/dapl_rmr_create.c tduffy> =================================================================== tduffy> --- linux-kernel-rmr/dat-provider/dapl_rmr_create.c (revision 2483) tduffy> +++ linux-kernel-rmr/dat-provider/dapl_rmr_create.c (working copy) tduffy> @@ -1,89 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/* tduffy> - * $Id$ tduffy> - */ tduffy> - tduffy> -#include "dapl_rmr_util.h" tduffy> -#include "dapl_openib_util.h" tduffy> - tduffy> -/* tduffy> - * dapl_rmr_create tduffy> - * tduffy> - * Create a remote memory region for the specified protection zone tduffy> - * tduffy> - * Input: tduffy> - * pz_handle tduffy> - * tduffy> - * Output: tduffy> - * rmr_handle tduffy> - * tduffy> - * Returns: tduffy> - * DAT_SUCCESS tduffy> - * DAT_INSUFFICIENT_RESOURCES tduffy> - * DAT_INVALID_PARAMETER tduffy> - */ tduffy> -u32 dapl_rmr_create(DAT_PZ_HANDLE pz_handle, DAT_RMR_HANDLE *rmr_handle) tduffy> -{ tduffy> - struct dapl_pz *pz; tduffy> - struct dapl_rmr *rmr; tduffy> - u32 dat_status = DAT_SUCCESS; tduffy> - tduffy> - if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - pz = (struct dapl_pz *)pz_handle; tduffy> - tduffy> - rmr = dapl_rmr_alloc(pz); tduffy> - tduffy> - if (rmr == NULL) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - dat_status = dapl_ib_mw_alloc(rmr); tduffy> - tduffy> - if (dat_status != DAT_SUCCESS) { tduffy> - dapl_rmr_dealloc(rmr); tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, tduffy> - DAT_RESOURCE_MEMORY_REGION); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - atomic_inc(&pz->pz_ref_count); tduffy> - tduffy> - *rmr_handle = rmr; tduffy> - tduffy> -bail: tduffy> - return dat_status; tduffy> -} tduffy> Index: linux-kernel-rmr/dat-provider/Makefile tduffy> =================================================================== tduffy> --- linux-kernel-rmr/dat-provider/Makefile (revision 2483) tduffy> +++ linux-kernel-rmr/dat-provider/Makefile (working copy) tduffy> @@ -76,11 +61,7 @@ PROVIDER_MODULES := \ tduffy> dapl_psp_query \ tduffy> dapl_pz \ tduffy> dapl_ring_buffer_util \ tduffy> - dapl_rmr_bind \ tduffy> - dapl_rmr_create \ tduffy> - dapl_rmr_free \ tduffy> - dapl_rmr_query \ tduffy> - dapl_rmr_util \ tduffy> + dapl_rmr \ tduffy> dapl_rsp_create \ tduffy> dapl_rsp_free \ tduffy> dapl_rsp_query \ tduffy> @@ -98,5 +79,25 @@ PROVIDER_MODULES := \ tduffy> Index: linux-kernel-rmr/dat-provider/dapl_rmr_free.c tduffy> =================================================================== tduffy> --- linux-kernel-rmr/dat-provider/dapl_rmr_free.c (revision 2483) tduffy> +++ linux-kernel-rmr/dat-provider/dapl_rmr_free.c (working copy) tduffy> @@ -1,84 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/* tduffy> - * $Id$ tduffy> - */ tduffy> - tduffy> -#include "dapl_rmr_util.h" tduffy> -#include "dapl_openib_util.h" tduffy> -#include "dapl_ia_util.h" tduffy> - tduffy> -/* tduffy> - * dapl_rmr_free tduffy> - * tduffy> - * Destroy an instance of the Remote Memory Region tduffy> - * tduffy> - * Input: tduffy> - * rmr_handle tduffy> - * tduffy> - * Output: tduffy> - * none tduffy> - * tduffy> - * Returns: tduffy> - * DAT_SUCCESS tduffy> - * DAT_INVALID_PARAMETER tduffy> - */ tduffy> -u32 dapl_rmr_free(DAT_RMR_HANDLE rmr_handle) tduffy> -{ tduffy> - struct dapl_rmr *rmr; tduffy> - u32 dat_status = DAT_SUCCESS; tduffy> - tduffy> - if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - rmr = (struct dapl_rmr *)rmr_handle; tduffy> - tduffy> - /* tduffy> - * If the user did not perform an unbind op, release tduffy> - * counts here. tduffy> - */ tduffy> - if (rmr->param.lmr_triplet.virtual_address != 0) { tduffy> - atomic_dec(&rmr->lmr->lmr_ref_count); tduffy> - rmr->param.lmr_triplet.virtual_address = 0; tduffy> - } tduffy> - tduffy> - dat_status = dapl_ib_mw_free(rmr); tduffy> - tduffy> - if (dat_status != DAT_SUCCESS) tduffy> - goto bail; tduffy> - tduffy> - atomic_dec(&rmr->pz->pz_ref_count); tduffy> - tduffy> - dapl_rmr_dealloc(rmr); tduffy> - tduffy> -bail: tduffy> - return dat_status; tduffy> -} tduffy> Index: linux-kernel-rmr/dat-provider/dapl_rmr_query.c tduffy> =================================================================== tduffy> --- linux-kernel-rmr/dat-provider/dapl_rmr_query.c (revision 2483) tduffy> +++ linux-kernel-rmr/dat-provider/dapl_rmr_query.c (working copy) tduffy> @@ -1,55 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/* tduffy> - * $Id$ tduffy> - */ tduffy> - tduffy> -#include "dapl.h" tduffy> - tduffy> -u32 dapl_rmr_query(DAT_RMR_HANDLE rmr_handle, struct dat_rmr_param *rmr_param) tduffy> -{ tduffy> - struct dapl_rmr *rmr; tduffy> - u32 dat_status = DAT_SUCCESS; tduffy> - tduffy> - if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); tduffy> - goto bail; tduffy> - } tduffy> - if (NULL == rmr_param) { tduffy> - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - rmr = (struct dapl_rmr *)rmr_handle; tduffy> - tduffy> - memcpy(rmr_param, &rmr->param, sizeof *rmr_param); tduffy> - tduffy> -bail: tduffy> - return dat_status; tduffy> -} tduffy> Index: linux-kernel-rmr/dat-provider/dapl_rmr_bind.c tduffy> =================================================================== tduffy> --- linux-kernel-rmr/dat-provider/dapl_rmr_bind.c (revision 2483) tduffy> +++ linux-kernel-rmr/dat-provider/dapl_rmr_bind.c (working copy) tduffy> @@ -1,273 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/* tduffy> - * $Id$ tduffy> - */ tduffy> - tduffy> -#include "dapl.h" tduffy> -#include "dapl_rmr_util.h" tduffy> -#include "dapl_ep_util.h" tduffy> -#include "dapl_cookie.h" tduffy> -#include "dapl_openib_util.h" tduffy> - tduffy> -/********************************************************************* tduffy> - * * tduffy> - * Function Definitions * tduffy> - * * tduffy> - *********************************************************************/ tduffy> - tduffy> -static inline u32 dapl_rmr_bind_fuse(struct dapl_rmr *rmr, tduffy> - const struct dat_lmr_triplet *lmr_triplet, tduffy> - enum dat_mem_priv_flags mem_priv, tduffy> - struct dapl_ep *ep_ptr, tduffy> - DAT_RMR_COOKIE user_cookie, tduffy> - enum dat_completion_flags completion_flags, tduffy> - DAT_RMR_CONTEXT *rmr_context) tduffy> -{ tduffy> - struct dapl_lmr *lmr; tduffy> - struct dapl_cookie *cookie; tduffy> - u32 status; tduffy> - boolean_t is_signaled; tduffy> - tduffy> - status = dapl_hash_search(rmr->header.owner_ia->hca_ptr->lmr_hash_table, tduffy> - lmr_triplet->lmr_context, tduffy> - (DAPL_HASH_DATA *) &lmr); tduffy> - if (DAT_SUCCESS != status) { tduffy> - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - /* if the ep in unconnected return an error. IB requires that the */ tduffy> - /* QP be connected to change a memory window binding since: */ tduffy> - /* */ tduffy> - /* - memory window bind operations are WQEs placed on a QP's */ tduffy> - /* send queue */ tduffy> - /* */ tduffy> - /* - QP's only process WQEs on the send queue when the QP is in */ tduffy> - /* the RTS state */ tduffy> - if (DAT_EP_STATE_CONNECTED != ep_ptr->param.ep_state) { tduffy> - status = tduffy> - DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep_ptr)); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - if (FALSE == tduffy> - dapl_mr_bounds_check(dapl_mr_get_address tduffy> - (lmr->param.region_desc, lmr->param.mem_type), tduffy> - lmr->param.length, tduffy> - lmr_triplet->virtual_address, tduffy> - lmr_triplet->segment_length)) { tduffy> - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - /* If the LMR, RMR, and EP are not in the same PZ, there is an error */ tduffy> - if ((ep_ptr->param.pz_handle != lmr->param.pz_handle) || tduffy> - (ep_ptr->param.pz_handle != rmr->param.pz_handle)) { tduffy> - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - if (!dapl_rmr_validate_completion_flag(DAT_COMPLETION_SUPPRESS_FLAG, tduffy> - ep_ptr->param.ep_attr. tduffy> - request_completion_flags, tduffy> - completion_flags) || tduffy> - !dapl_rmr_validate_completion_flag(DAT_COMPLETION_UNSIGNALLED_FLAG, tduffy> - ep_ptr->param.ep_attr. tduffy> - request_completion_flags, tduffy> - completion_flags) || tduffy> - !dapl_rmr_validate_completion_flag( tduffy> - DAT_COMPLETION_BARRIER_FENCE_FLAG, tduffy> - ep_ptr->param.ep_attr.request_completion_flags, tduffy> - completion_flags)) { tduffy> - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - status = dapl_rmr_cookie_alloc(&ep_ptr->req_buffer, rmr, user_cookie, tduffy> - &cookie); tduffy> - if (DAT_SUCCESS != status) tduffy> - goto bail; tduffy> - tduffy> - is_signaled = (completion_flags & DAT_COMPLETION_SUPPRESS_FLAG) ? tduffy> - FALSE : TRUE; tduffy> - tduffy> - /* tduffy> - * Take reference before posting to avoid race conditions with tduffy> - * completions tduffy> - */ tduffy> - atomic_inc(&ep_ptr->req_count); tduffy> - tduffy> - status = dapl_ib_mw_bind(rmr, lmr, ep_ptr, cookie, tduffy> - lmr_triplet->virtual_address, tduffy> - lmr_triplet->segment_length, tduffy> - mem_priv, is_signaled); tduffy> - if (DAT_SUCCESS != status) { tduffy> - atomic_dec(&ep_ptr->req_count); tduffy> - dapl_cookie_dealloc(&ep_ptr->req_buffer, cookie); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - atomic_inc(&lmr->lmr_ref_count); tduffy> - tduffy> - /* if the RMR was previously bound */ tduffy> - if (NULL != rmr->lmr) tduffy> - atomic_dec(&rmr->lmr->lmr_ref_count); tduffy> - tduffy> - rmr->param.mem_priv = mem_priv; tduffy> - rmr->param.lmr_triplet = *lmr_triplet; tduffy> - rmr->ep = ep_ptr; tduffy> - rmr->lmr = lmr; tduffy> - tduffy> - if (NULL != rmr_context) tduffy> - *rmr_context = rmr->param.rmr_context; tduffy> - tduffy> -bail: tduffy> - return status; tduffy> -} tduffy> - tduffy> -static inline u32 dapl_rmr_bind_unfuse(struct dapl_rmr *rmr, tduffy> - struct dapl_ep *ep_ptr, tduffy> - DAT_RMR_COOKIE user_cookie, tduffy> - enum dat_completion_flags cflags) tduffy> -{ tduffy> - struct dapl_cookie *cookie; tduffy> - u32 status = DAT_SUCCESS; tduffy> - boolean_t is_signaled; tduffy> - tduffy> - /* tduffy> - * if the ep in unconnected return an error. IB requires that the tduffy> - * QP be connected to change a memory window binding since: tduffy> - * tduffy> - * - memory window bind operations are WQEs placed on a QP's tduffy> - * send queue tduffy> - * tduffy> - * - QP's only process WQEs on the send queue when the QP is in tduffy> - * the RTS state tduffy> - */ tduffy> - if (DAT_EP_STATE_CONNECTED != ep_ptr->param.ep_state) { tduffy> - status = tduffy> - DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep_ptr)); tduffy> - goto bail1; tduffy> - } tduffy> - tduffy> - /* If the RMR and EP are not in the same PZ, there is an error */ tduffy> - if (ep_ptr->param.pz_handle != rmr->param.pz_handle) { tduffy> - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); tduffy> - goto bail1; tduffy> - } tduffy> - tduffy> - if (!dapl_rmr_validate_completion_flag(DAT_COMPLETION_SUPPRESS_FLAG, tduffy> - ep_ptr->param.ep_attr. tduffy> - request_completion_flags, tduffy> - cflags) || tduffy> - !dapl_rmr_validate_completion_flag(DAT_COMPLETION_UNSIGNALLED_FLAG, tduffy> - ep_ptr->param.ep_attr. tduffy> - request_completion_flags, tduffy> - cflags) || tduffy> - !dapl_rmr_validate_completion_flag( tduffy> - DAT_COMPLETION_BARRIER_FENCE_FLAG, tduffy> - ep_ptr->param.ep_attr.request_completion_flags, cflags)) { tduffy> - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); tduffy> - goto bail1; tduffy> - } tduffy> - tduffy> - status = dapl_rmr_cookie_alloc(&ep_ptr->req_buffer, rmr, user_cookie, tduffy> - &cookie); tduffy> - if (DAT_SUCCESS != status) tduffy> - goto bail1; tduffy> - tduffy> - is_signaled = (cflags & DAT_COMPLETION_UNSIGNALLED_FLAG) ? FALSE : TRUE; tduffy> - tduffy> - /* tduffy> - * Take reference before posting to avoid race conditions with tduffy> - * completions tduffy> - */ tduffy> - atomic_inc(&ep_ptr->req_count); tduffy> - tduffy> - status = dapl_ib_mw_unbind(rmr, ep_ptr, cookie, is_signaled); tduffy> - if (DAT_SUCCESS != status) { tduffy> - atomic_dec(&ep_ptr->req_count); tduffy> - dapl_cookie_dealloc(&ep_ptr->req_buffer, cookie); tduffy> - goto bail1; tduffy> - } tduffy> - tduffy> - /* if the RMR was previously bound */ tduffy> - if (NULL != rmr->lmr) tduffy> - atomic_dec(&rmr->lmr->lmr_ref_count); tduffy> - tduffy> - rmr->param.mem_priv = DAT_MEM_PRIV_NONE_FLAG; tduffy> - rmr->param.lmr_triplet.lmr_context = 0; tduffy> - rmr->param.lmr_triplet.virtual_address = 0; tduffy> - rmr->param.lmr_triplet.segment_length = 0; tduffy> - rmr->ep = ep_ptr; tduffy> - rmr->lmr = NULL; tduffy> - tduffy> -bail1: tduffy> - return status; tduffy> -} tduffy> - tduffy> -/* tduffy> - * dapl_rmr_bind tduffy> - * tduffy> - * DAPL Requirements Version xxx, 6.6.4.4 tduffy> - * tduffy> - * Bind the RMR to the specified memory region within the LMR and tduffy> - * provide a new rmr_context value. tduffy> - * tduffy> - * Input: tduffy> - * Output: tduffy> - */ tduffy> -u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_handle, tduffy> - const struct dat_lmr_triplet *lmr_triplet, tduffy> - enum dat_mem_priv_flags mem_priv, DAT_EP_HANDLE ep_handle, tduffy> - DAT_RMR_COOKIE user_cookie, tduffy> - enum dat_completion_flags completion_flags, tduffy> - DAT_RMR_CONTEXT *rmr_context) tduffy> -{ tduffy> - struct dapl_rmr *rmr; tduffy> - struct dapl_ep *ep_ptr; tduffy> - tduffy> - if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) tduffy> - return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); tduffy> - tduffy> - if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) tduffy> - return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); tduffy> - tduffy> - rmr = (struct dapl_rmr *)rmr_handle; tduffy> - ep_ptr = (struct dapl_ep *)ep_handle; tduffy> - tduffy> - /* if the rmr should be bound */ tduffy> - if (0 != lmr_triplet->segment_length) tduffy> - return dapl_rmr_bind_fuse(rmr, lmr_triplet, mem_priv, ep_ptr, tduffy> - user_cookie, completion_flags, tduffy> - rmr_context); tduffy> - else /* the rmr should be unbound */ tduffy> - return dapl_rmr_bind_unfuse(rmr, ep_ptr, user_cookie, tduffy> - completion_flags); tduffy> -} tduffy> Index: linux-kernel-rmr/dat-provider/dapl_openib_util.c tduffy> =================================================================== tduffy> --- linux-kernel-rmr/dat-provider/dapl_openib_util.c (revision 2483) tduffy> +++ linux-kernel-rmr/dat-provider/dapl_openib_util.c (working copy) tduffy> @@ -37,7 +37,6 @@ tduffy> #include "dapl_openib_util.h" tduffy> #include "dapl_evd_util.h" tduffy> #include "dapl_cr_util.h" tduffy> -#include "dapl_rmr_util.h" tduffy> #include "dapl_cookie.h" tduffy> #include "dapl_openib_cm.h" tduffy> tduffy> Index: linux-kernel-rmr/dat-provider/dapl_rmr_util.c tduffy> =================================================================== tduffy> --- linux-kernel-rmr/dat-provider/dapl_rmr_util.c (revision 2483) tduffy> +++ linux-kernel-rmr/dat-provider/dapl_rmr_util.c (working copy) tduffy> @@ -1,82 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -#include "dapl_rmr_util.h" tduffy> -#include "dapl_ia_util.h" tduffy> - tduffy> -struct dapl_rmr *dapl_rmr_alloc(struct dapl_pz *pz) tduffy> -{ tduffy> - struct dapl_rmr *rmr; tduffy> - tduffy> - /* Allocate LMR */ tduffy> - rmr = kmalloc(sizeof *rmr, GFP_ATOMIC); tduffy> - if (!rmr) tduffy> - return NULL; tduffy> - tduffy> - /* zero the structure */ tduffy> - memset(rmr, 0, sizeof *rmr); tduffy> - tduffy> - /* tduffy> - * initialize the header tduffy> - */ tduffy> - rmr->header.provider = pz->header.provider; tduffy> - rmr->header.magic = DAPL_MAGIC_RMR; tduffy> - rmr->header.handle_type = DAT_HANDLE_TYPE_RMR; tduffy> - rmr->header.owner_ia = pz->header.owner_ia; tduffy> - rmr->header.user_context.as_64 = 0; tduffy> - rmr->header.user_context.as_ptr = NULL; tduffy> - dapl_llist_init_entry(&rmr->header.ia_list_entry); tduffy> - dapl_ia_link_rmr(rmr->header.owner_ia, rmr); tduffy> - spin_lock_init(&rmr->header.lock); tduffy> - tduffy> - /* tduffy> - * initialize the body tduffy> - */ tduffy> - rmr->param.ia_handle = (DAT_IA_HANDLE) pz->header.owner_ia; tduffy> - rmr->param.pz_handle = (DAT_PZ_HANDLE) pz; tduffy> - rmr->param.lmr_triplet.lmr_context = 0; tduffy> - rmr->param.lmr_triplet.pad = 0; tduffy> - rmr->param.lmr_triplet.virtual_address = 0; tduffy> - rmr->param.lmr_triplet.segment_length = 0; tduffy> - tduffy> - rmr->param.mem_priv = 0; tduffy> - rmr->pz = pz; tduffy> - rmr->lmr = NULL; tduffy> - tduffy> - return rmr; tduffy> -} tduffy> - tduffy> -void dapl_rmr_dealloc(struct dapl_rmr *rmr) tduffy> -{ tduffy> - /* reset magic to prevent reuse */ tduffy> - rmr->header.magic = DAPL_MAGIC_INVALID; tduffy> - tduffy> - dapl_ia_unlink_rmr(rmr->header.owner_ia, rmr); tduffy> - /* no need to destroy rmr->header.lock */ tduffy> - tduffy> - kfree(rmr); tduffy> -} tduffy> Index: linux-kernel-rmr/dat-provider/dapl_rmr.c tduffy> =================================================================== tduffy> --- linux-kernel-rmr/dat-provider/dapl_rmr.c (revision 0) tduffy> +++ linux-kernel-rmr/dat-provider/dapl_rmr.c (revision 0) tduffy> @@ -0,0 +1,461 @@ tduffy> +/* tduffy> + * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * tduffy> + * This Software is licensed under one of the following licenses: tduffy> + * tduffy> + * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> + * available from the Open Source Initiative, see tduffy> + * http://www.opensource.org/licenses/cpl.php. tduffy> + * tduffy> + * 2) under the terms of the "The BSD License" a copy of which is tduffy> + * available from the Open Source Initiative, see tduffy> + * http://www.opensource.org/licenses/bsd-license.php. tduffy> + * tduffy> + * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> + * copy of which is available from the Open Source Initiative, see tduffy> + * http://www.opensource.org/licenses/gpl-license.php. tduffy> + * tduffy> + * Licensee has the right to choose one of the above licenses. tduffy> + * tduffy> + * Redistributions of source code must retain the above copyright tduffy> + * notice and one of the license notices. tduffy> + * tduffy> + * Redistributions in binary form must reproduce both the above copyright tduffy> + * notice, one of the license notices in the documentation tduffy> + * and/or other materials provided with the distribution. tduffy> + */ tduffy> + tduffy> +/* tduffy> + * $Id$ tduffy> + */ tduffy> + tduffy> +#include "dapl.h" tduffy> +#include "dapl_ep_util.h" tduffy> +#include "dapl_ia_util.h" tduffy> +#include "dapl_mr_util.h" tduffy> +#include "dapl_hash.h" tduffy> +#include "dapl_cookie.h" tduffy> +#include "dapl_openib_util.h" tduffy> + tduffy> +static inline boolean_t tduffy> +dapl_rmr_validate_completion_flag(enum dat_completion_flags mask, tduffy> + enum dat_completion_flags allow, tduffy> + enum dat_completion_flags request) tduffy> +{ tduffy> + if ((mask & request) && !(mask & allow)) tduffy> + return FALSE; tduffy> + else tduffy> + return TRUE; tduffy> +} tduffy> + tduffy> + tduffy> +static struct dapl_rmr *dapl_rmr_alloc(struct dapl_pz *pz) tduffy> +{ tduffy> + struct dapl_rmr *rmr; tduffy> + tduffy> + /* Allocate LMR */ tduffy> + rmr = kmalloc(sizeof *rmr, GFP_ATOMIC); tduffy> + if (!rmr) tduffy> + return NULL; tduffy> + tduffy> + /* zero the structure */ tduffy> + memset(rmr, 0, sizeof *rmr); tduffy> + tduffy> + /* tduffy> + * initialize the header tduffy> + */ tduffy> + rmr->header.provider = pz->header.provider; tduffy> + rmr->header.magic = DAPL_MAGIC_RMR; tduffy> + rmr->header.handle_type = DAT_HANDLE_TYPE_RMR; tduffy> + rmr->header.owner_ia = pz->header.owner_ia; tduffy> + rmr->header.user_context.as_64 = 0; tduffy> + rmr->header.user_context.as_ptr = NULL; tduffy> + dapl_llist_init_entry(&rmr->header.ia_list_entry); tduffy> + dapl_ia_link_rmr(rmr->header.owner_ia, rmr); tduffy> + spin_lock_init(&rmr->header.lock); tduffy> + tduffy> + /* tduffy> + * initialize the body tduffy> + */ tduffy> + rmr->param.ia_handle = (DAT_IA_HANDLE)pz->header.owner_ia; tduffy> + rmr->param.pz_handle = (DAT_PZ_HANDLE)pz; tduffy> + rmr->param.lmr_triplet.lmr_context = 0; tduffy> + rmr->param.lmr_triplet.pad = 0; tduffy> + rmr->param.lmr_triplet.virtual_address = 0; tduffy> + rmr->param.lmr_triplet.segment_length = 0; tduffy> + tduffy> + rmr->param.mem_priv = 0; tduffy> + rmr->pz = pz; tduffy> + rmr->lmr = NULL; tduffy> + tduffy> + return rmr; tduffy> +} tduffy> + tduffy> +static void dapl_rmr_dealloc(struct dapl_rmr *rmr) tduffy> +{ tduffy> + /* reset magic to prevent reuse */ tduffy> + rmr->header.magic = DAPL_MAGIC_INVALID; tduffy> + tduffy> + dapl_ia_unlink_rmr(rmr->header.owner_ia, rmr); tduffy> + /* no need to destroy rmr->header.lock */ tduffy> + tduffy> + kfree(rmr); tduffy> +} tduffy> + tduffy> +static inline u32 dapl_rmr_bind_fuse(struct dapl_rmr *rmr, tduffy> + const struct dat_lmr_triplet *lmr_triplet, tduffy> + enum dat_mem_priv_flags mem_priv, tduffy> + struct dapl_ep *ep_ptr, tduffy> + DAT_RMR_COOKIE user_cookie, tduffy> + enum dat_completion_flags completion_flags, tduffy> + DAT_RMR_CONTEXT *rmr_context) tduffy> +{ tduffy> + struct dapl_lmr *lmr; tduffy> + struct dapl_cookie *cookie; tduffy> + u32 status; tduffy> + boolean_t is_signaled; tduffy> + tduffy> + status = dapl_hash_search(rmr->header.owner_ia->hca_ptr->lmr_hash_table, tduffy> + lmr_triplet->lmr_context, tduffy> + (DAPL_HASH_DATA *) &lmr); tduffy> + if (DAT_SUCCESS != status) { tduffy> + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + /* tduffy> + * if the ep in unconnected return an error. IB requires that the tduffy> + * QP be connected to change a memory window binding since: tduffy> + * tduffy> + * - memory window bind operations are WQEs placed on a QP's tduffy> + * send queue tduffy> + * tduffy> + * - QP's only process WQEs on the send queue when the QP is in tduffy> + * the RTS state tduffy> + */ tduffy> + if (DAT_EP_STATE_CONNECTED != ep_ptr->param.ep_state) { tduffy> + status = tduffy> + DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep_ptr)); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + if (FALSE == tduffy> + dapl_mr_bounds_check(dapl_mr_get_address tduffy> + (lmr->param.region_desc, lmr->param.mem_type), tduffy> + lmr->param.length, tduffy> + lmr_triplet->virtual_address, tduffy> + lmr_triplet->segment_length)) { tduffy> + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + /* If the LMR, RMR, and EP are not in the same PZ, there is an error */ tduffy> + if ((ep_ptr->param.pz_handle != lmr->param.pz_handle) || tduffy> + (ep_ptr->param.pz_handle != rmr->param.pz_handle)) { tduffy> + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + if (!dapl_rmr_validate_completion_flag(DAT_COMPLETION_SUPPRESS_FLAG, tduffy> + ep_ptr->param.ep_attr. tduffy> + request_completion_flags, tduffy> + completion_flags) || tduffy> + !dapl_rmr_validate_completion_flag(DAT_COMPLETION_UNSIGNALLED_FLAG, tduffy> + ep_ptr->param.ep_attr. tduffy> + request_completion_flags, tduffy> + completion_flags) || tduffy> + !dapl_rmr_validate_completion_flag( tduffy> + DAT_COMPLETION_BARRIER_FENCE_FLAG, tduffy> + ep_ptr->param.ep_attr.request_completion_flags, tduffy> + completion_flags)) { tduffy> + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + status = dapl_rmr_cookie_alloc(&ep_ptr->req_buffer, rmr, user_cookie, tduffy> + &cookie); tduffy> + if (DAT_SUCCESS != status) tduffy> + goto bail; tduffy> + tduffy> + is_signaled = (completion_flags & DAT_COMPLETION_SUPPRESS_FLAG) ? tduffy> + FALSE : TRUE; tduffy> + tduffy> + /* tduffy> + * Take reference before posting to avoid race conditions with tduffy> + * completions tduffy> + */ tduffy> + atomic_inc(&ep_ptr->req_count); tduffy> + tduffy> + status = dapl_ib_mw_bind(rmr, lmr, ep_ptr, cookie, tduffy> + lmr_triplet->virtual_address, tduffy> + lmr_triplet->segment_length, tduffy> + mem_priv, is_signaled); tduffy> + if (DAT_SUCCESS != status) { tduffy> + atomic_dec(&ep_ptr->req_count); tduffy> + dapl_cookie_dealloc(&ep_ptr->req_buffer, cookie); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + atomic_inc(&lmr->lmr_ref_count); tduffy> + tduffy> + /* if the RMR was previously bound */ tduffy> + if (NULL != rmr->lmr) tduffy> + atomic_dec(&rmr->lmr->lmr_ref_count); tduffy> + tduffy> + rmr->param.mem_priv = mem_priv; tduffy> + rmr->param.lmr_triplet = *lmr_triplet; tduffy> + rmr->ep = ep_ptr; tduffy> + rmr->lmr = lmr; tduffy> + tduffy> + if (NULL != rmr_context) tduffy> + *rmr_context = rmr->param.rmr_context; tduffy> + tduffy> +bail: tduffy> + return status; tduffy> +} tduffy> + tduffy> +static inline u32 dapl_rmr_bind_unfuse(struct dapl_rmr *rmr, tduffy> + struct dapl_ep *ep_ptr, tduffy> + DAT_RMR_COOKIE user_cookie, tduffy> + enum dat_completion_flags cflags) tduffy> +{ tduffy> + struct dapl_cookie *cookie; tduffy> + u32 status = DAT_SUCCESS; tduffy> + boolean_t is_signaled; tduffy> + tduffy> + /* tduffy> + * if the ep in unconnected return an error. IB requires that the tduffy> + * QP be connected to change a memory window binding since: tduffy> + * tduffy> + * - memory window bind operations are WQEs placed on a QP's tduffy> + * send queue tduffy> + * tduffy> + * - QP's only process WQEs on the send queue when the QP is in tduffy> + * the RTS state tduffy> + */ tduffy> + if (DAT_EP_STATE_CONNECTED != ep_ptr->param.ep_state) { tduffy> + status = DAT_ERROR(DAT_INVALID_STATE, tduffy> + dapl_ep_state_subtype(ep_ptr)); tduffy> + goto bail1; tduffy> + } tduffy> + tduffy> + /* If the RMR and EP are not in the same PZ, there is an error */ tduffy> + if (ep_ptr->param.pz_handle != rmr->param.pz_handle) { tduffy> + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); tduffy> + goto bail1; tduffy> + } tduffy> + tduffy> + if (!dapl_rmr_validate_completion_flag(DAT_COMPLETION_SUPPRESS_FLAG, tduffy> + ep_ptr->param.ep_attr. tduffy> + request_completion_flags, tduffy> + cflags) || tduffy> + !dapl_rmr_validate_completion_flag(DAT_COMPLETION_UNSIGNALLED_FLAG, tduffy> + ep_ptr->param.ep_attr. tduffy> + request_completion_flags, tduffy> + cflags) || tduffy> + !dapl_rmr_validate_completion_flag( tduffy> + DAT_COMPLETION_BARRIER_FENCE_FLAG, tduffy> + ep_ptr->param.ep_attr.request_completion_flags, cflags)) { tduffy> + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); tduffy> + goto bail1; tduffy> + } tduffy> + tduffy> + status = dapl_rmr_cookie_alloc(&ep_ptr->req_buffer, rmr, user_cookie, tduffy> + &cookie); tduffy> + if (DAT_SUCCESS != status) tduffy> + goto bail1; tduffy> + tduffy> + is_signaled = (cflags & DAT_COMPLETION_UNSIGNALLED_FLAG) ? FALSE : TRUE; tduffy> + tduffy> + /* tduffy> + * Take reference before posting to avoid race conditions with tduffy> + * completions tduffy> + */ tduffy> + atomic_inc(&ep_ptr->req_count); tduffy> + tduffy> + status = dapl_ib_mw_unbind(rmr, ep_ptr, cookie, is_signaled); tduffy> + if (DAT_SUCCESS != status) { tduffy> + atomic_dec(&ep_ptr->req_count); tduffy> + dapl_cookie_dealloc(&ep_ptr->req_buffer, cookie); tduffy> + goto bail1; tduffy> + } tduffy> + tduffy> + /* if the RMR was previously bound */ tduffy> + if (NULL != rmr->lmr) tduffy> + atomic_dec(&rmr->lmr->lmr_ref_count); tduffy> + tduffy> + rmr->param.mem_priv = DAT_MEM_PRIV_NONE_FLAG; tduffy> + rmr->param.lmr_triplet.lmr_context = 0; tduffy> + rmr->param.lmr_triplet.virtual_address = 0; tduffy> + rmr->param.lmr_triplet.segment_length = 0; tduffy> + rmr->ep = ep_ptr; tduffy> + rmr->lmr = NULL; tduffy> + tduffy> +bail1: tduffy> + return status; tduffy> +} tduffy> + tduffy> +/* tduffy> + * dapl_rmr_bind tduffy> + * tduffy> + * DAPL Requirements Version xxx, 6.6.4.4 tduffy> + * tduffy> + * Bind the RMR to the specified memory region within the LMR and tduffy> + * provide a new rmr_context value. tduffy> + * tduffy> + * Input: tduffy> + * Output: tduffy> + */ tduffy> +u32 dapl_rmr_bind(DAT_RMR_HANDLE rmr_handle, tduffy> + const struct dat_lmr_triplet *lmr_triplet, tduffy> + enum dat_mem_priv_flags mem_priv, DAT_EP_HANDLE ep_handle, tduffy> + DAT_RMR_COOKIE user_cookie, tduffy> + enum dat_completion_flags completion_flags, tduffy> + DAT_RMR_CONTEXT *rmr_context) tduffy> +{ tduffy> + struct dapl_rmr *rmr; tduffy> + struct dapl_ep *ep_ptr; tduffy> + tduffy> + if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) tduffy> + return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); tduffy> + tduffy> + if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) tduffy> + return DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); tduffy> + tduffy> + rmr = (struct dapl_rmr *)rmr_handle; tduffy> + ep_ptr = (struct dapl_ep *)ep_handle; tduffy> + tduffy> + /* if the rmr should be bound */ tduffy> + if (0 != lmr_triplet->segment_length) tduffy> + return dapl_rmr_bind_fuse(rmr, lmr_triplet, mem_priv, ep_ptr, tduffy> + user_cookie, completion_flags, tduffy> + rmr_context); tduffy> + else /* the rmr should be unbound */ tduffy> + return dapl_rmr_bind_unfuse(rmr, ep_ptr, user_cookie, tduffy> + completion_flags); tduffy> +} tduffy> + tduffy> +/* tduffy> + * dapl_rmr_create tduffy> + * tduffy> + * Create a remote memory region for the specified protection zone tduffy> + * tduffy> + * Input: tduffy> + * pz_handle tduffy> + * tduffy> + * Output: tduffy> + * rmr_handle tduffy> + * tduffy> + * Returns: tduffy> + * DAT_SUCCESS tduffy> + * DAT_INSUFFICIENT_RESOURCES tduffy> + * DAT_INVALID_PARAMETER tduffy> + */ tduffy> +u32 dapl_rmr_create(DAT_PZ_HANDLE pz_handle, DAT_RMR_HANDLE *rmr_handle) tduffy> +{ tduffy> + struct dapl_pz *pz; tduffy> + struct dapl_rmr *rmr; tduffy> + u32 status = DAT_SUCCESS; tduffy> + tduffy> + if (DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { tduffy> + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + pz = (struct dapl_pz *)pz_handle; tduffy> + tduffy> + rmr = dapl_rmr_alloc(pz); tduffy> + tduffy> + if (rmr == NULL) { tduffy> + status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, tduffy> + DAT_RESOURCE_MEMORY); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + status = dapl_ib_mw_alloc(rmr); tduffy> + tduffy> + if (status != DAT_SUCCESS) { tduffy> + dapl_rmr_dealloc(rmr); tduffy> + status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, tduffy> + DAT_RESOURCE_MEMORY_REGION); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + atomic_inc(&pz->pz_ref_count); tduffy> + tduffy> + *rmr_handle = rmr; tduffy> + tduffy> +bail: tduffy> + return status; tduffy> +} tduffy> + tduffy> +/* tduffy> + * dapl_rmr_free tduffy> + * tduffy> + * Destroy an instance of the Remote Memory Region tduffy> + * tduffy> + * Input: tduffy> + * rmr_handle tduffy> + * tduffy> + * Output: tduffy> + * none tduffy> + * tduffy> + * Returns: tduffy> + * DAT_SUCCESS tduffy> + * DAT_INVALID_PARAMETER tduffy> + */ tduffy> +u32 dapl_rmr_free(DAT_RMR_HANDLE rmr_handle) tduffy> +{ tduffy> + struct dapl_rmr *rmr; tduffy> + u32 status = DAT_SUCCESS; tduffy> + tduffy> + if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) { tduffy> + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + rmr = (struct dapl_rmr *)rmr_handle; tduffy> + tduffy> + /* tduffy> + * If the user did not perform an unbind op, release tduffy> + * counts here. tduffy> + */ tduffy> + if (rmr->param.lmr_triplet.virtual_address != 0) { tduffy> + atomic_dec(&rmr->lmr->lmr_ref_count); tduffy> + rmr->param.lmr_triplet.virtual_address = 0; tduffy> + } tduffy> + tduffy> + status = dapl_ib_mw_free(rmr); tduffy> + tduffy> + if (status != DAT_SUCCESS) tduffy> + goto bail; tduffy> + tduffy> + atomic_dec(&rmr->pz->pz_ref_count); tduffy> + tduffy> + dapl_rmr_dealloc(rmr); tduffy> + tduffy> +bail: tduffy> + return status; tduffy> +} tduffy> + tduffy> +u32 dapl_rmr_query(DAT_RMR_HANDLE rmr_handle, struct dat_rmr_param *rmr_param) tduffy> +{ tduffy> + struct dapl_rmr *rmr; tduffy> + u32 status = DAT_SUCCESS; tduffy> + tduffy> + if (DAPL_BAD_HANDLE(rmr_handle, DAPL_MAGIC_RMR)) { tduffy> + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RMR); tduffy> + goto bail; tduffy> + } tduffy> + if (NULL == rmr_param) { tduffy> + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + rmr = (struct dapl_rmr *)rmr_handle; tduffy> + tduffy> + memcpy(rmr_param, &rmr->param, sizeof *rmr_param); tduffy> + tduffy> +bail: tduffy> + return status; tduffy> +} tduffy> Index: linux-kernel-rmr/dat-provider/dapl_rmr_util.h tduffy> =================================================================== tduffy> --- linux-kernel-rmr/dat-provider/dapl_rmr_util.h (revision 2483) tduffy> +++ linux-kernel-rmr/dat-provider/dapl_rmr_util.h (working copy) tduffy> @@ -1,52 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/* tduffy> - * $Id$ tduffy> - */ tduffy> - tduffy> -#ifndef DAPL_RMR_UTIL_H tduffy> -#define DAPL_RMR_UTIL_H tduffy> - tduffy> -#include "dapl_mr_util.h" tduffy> - tduffy> -extern struct dapl_rmr *dapl_rmr_alloc(struct dapl_pz *pz); tduffy> - tduffy> -extern void dapl_rmr_dealloc(struct dapl_rmr *rmr); tduffy> - tduffy> -static inline boolean_t tduffy> -dapl_rmr_validate_completion_flag(enum dat_completion_flags mask, tduffy> - enum dat_completion_flags allow, tduffy> - enum dat_completion_flags request) tduffy> -{ tduffy> - if ((mask & request) && !(mask & allow)) tduffy> - return FALSE; tduffy> - else tduffy> - return TRUE; tduffy> -} tduffy> - tduffy> -#endif /* DAPL_RMR_UTIL_H */ tduffy> Index: linux-kernel-rmr/patches/alt_dat_provider_makefile tduffy> =================================================================== tduffy> --- linux-kernel-rmr/patches/alt_dat_provider_makefile (revision 2483) tduffy> +++ linux-kernel-rmr/patches/alt_dat_provider_makefile (working copy) tduffy> @@ -16,7 +16,6 @@ PROVIDER_MODULES := \ tduffy> dapl_evd_kquery \ tduffy> dapl_evd_kcreate \ tduffy> dapl_evd_modify_upcall \ tduffy> - dapl_lmr_kcreate \ tduffy> dapl_cookie \ tduffy> dapl_cno_util \ tduffy> dapl_cr_accept \ tduffy> @@ -61,11 +60,7 @@ PROVIDER_MODULES := \ tduffy> dapl_ia_query \ tduffy> dapl_ia_util \ tduffy> dapl_llist \ tduffy> - dapl_lmr_free \ tduffy> - dapl_lmr_query \ tduffy> - dapl_lmr_sync_rdma_read \ tduffy> - dapl_lmr_sync_rdma_write \ tduffy> - dapl_lmr_util \ tduffy> + dapl_lmr \ tduffy> dapl_mr_util \ tduffy> dapl_provider \ tduffy> dapl_sp_util \ tduffy> @@ -73,16 +68,9 @@ PROVIDER_MODULES := \ tduffy> dapl_psp_create_any \ tduffy> dapl_psp_free \ tduffy> dapl_psp_query \ tduffy> - dapl_pz_create \ tduffy> - dapl_pz_free \ tduffy> - dapl_pz_query \ tduffy> - dapl_pz_util \ tduffy> + dapl_pz \ tduffy> dapl_ring_buffer_util \ tduffy> - dapl_rmr_bind \ tduffy> - dapl_rmr_create \ tduffy> - dapl_rmr_free \ tduffy> - dapl_rmr_query \ tduffy> - dapl_rmr_util \ tduffy> + dapl_rmr \ tduffy> dapl_rsp_create \ tduffy> dapl_rsp_free \ tduffy> dapl_rsp_query \ tduffy> @@ -103,7 +91,7 @@ PROVIDER_OBJS := $(foreach s, $(PROVIDER tduffy> KDAPL_DEBUG = 1 tduffy> ifeq (1,$(KDAPL_DEBUG)) tduffy> EXTRA_CFLAGS += -O0 -g tduffy> - EXTRA_CFLAGS += -DDAPL_DBG # -DDAPL_DBG_IO_TRC tduffy> + EXTRA_CFLAGS += -DDAPL_DBG tduffy> endif tduffy> tduffy> EXTRA_CFLAGS += -DDAPL_ATS -Idrivers/infiniband/include -I$(obj)/../dat -I$(obj)/../patches/ tduffy> @@ -121,4 +109,4 @@ default: tduffy> endif tduffy> tduffy> clean: tduffy> - rm -f *.o *.ko tduffy> + rm -f *.o *.ko .*.d .*.o.tmp *.mod.* .tmp_versions .*.cmd tduffy> Index: linux-kernel-rmr/patches/Makefile tduffy> =================================================================== tduffy> --- linux-kernel-rmr/patches/Makefile (revision 2483) tduffy> +++ linux-kernel-rmr/patches/Makefile (working copy) tduffy> @@ -20,4 +20,4 @@ default: tduffy> endif tduffy> tduffy> clean: tduffy> - rm -f *.o *.ko tduffy> + rm -rf *.o *.ko .*.d .*.o.tmp *.mod.* .tmp_versions .*.cmd tduffy> From jlentini at netapp.com Wed May 25 15:08:47 2005 From: jlentini at netapp.com (James Lentini) Date: Wed, 25 May 2005 18:08:47 -0400 (EDT) Subject: [openib-general] Re: [PATCH] remove redundant check in mthca_provider.c In-Reply-To: References: Message-ID: Did this patch get lost in the shuffle? Is the proposed change incorrect? james On Thu, 19 May 2005, James Lentini wrote: > > Remove redundant check of pd->uobject in mthca_provider.c > > Signed-off-by: James Lentini -------------- next part -------------- Index: infiniband/hw/mthca/mthca_provider.c =================================================================== --- infiniband/hw/mthca/mthca_provider.c (revision 2404) +++ infiniband/hw/mthca/mthca_provider.c (working copy) @@ -478,9 +478,7 @@ kfree(qp); return ERR_PTR(err); } - } - if (pd->uobject) { qp->mr.ibmr.lkey = ucmd.lkey; qp->sq.db_index = ucmd.sq_db_index; qp->rq.db_index = ucmd.rq_db_index; From roland at topspin.com Wed May 25 15:12:54 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 25 May 2005 15:12:54 -0700 Subject: [openib-general] Re: [PATCH] remove redundant check in mthca_provider.c In-Reply-To: (James Lentini's message of "Wed, 25 May 2005 18:08:47 -0400 (EDT)") References: Message-ID: <52vf57osyh.fsf@topspin.com> James> Did this patch get lost in the shuffle? Is the proposed James> change incorrect? Sorry, I just missed it the first time around... I just applied it. - R. From hozer at hozed.org Wed May 25 16:06:58 2005 From: hozer at hozed.org (Troy Benjegerdes) Date: Wed, 25 May 2005 18:06:58 -0500 Subject: [openib-general] umad abi 2 v 3 and multicast join failed Message-ID: <20050525230658.GY999@kalmia.hozed.org> I was running a crufty version of opensm (compiled from the roland-uverbs branch), and I started getting these kinds of errors for no apparent reason: ib0: multicast join failed for ff12:401b:ffff:0:0:0:ffff:ffff, status -22 ib0: multicast join failed for ff12:401b:ffff:0:0:0:ffff:ffff, status -22 I'm running 2.6.11 kernels, and 'stock' modules.. I just tried rebuilding opensm from the latest SVN, but it apparently needs a new umad driver.. warn: [24878] umad_init: wrong ABI version: /sys/class/infiniband_mad/abi_version is 2 but library ABI is 3 I suppose I need to rebuild the kernel ib_umad (and maybe everything else for good measure).. And if I do that, should I expect OpenSM to work better regarding the multicast issue? Also, what will happen if I run opensm on two different nodes? Will they fight, or will one of them figure out how to be a backup slave SM if the first goes down? -- -------------------------------------------------------------------------- Troy Benjegerdes 'da hozer' hozer at hozed.org Somone asked my why I work on this free (http://www.fsf.org/philosophy/) software stuff and not get a real job. Charles Shultz had the best answer: "Why do musicians compose symphonies and poets write poems? They do it because life wouldn't have any meaning for them if they didn't. That's why I draw cartoons. It's my life." -- Charles Shultz From robert.j.woodruff at intel.com Wed May 25 16:12:31 2005 From: robert.j.woodruff at intel.com (Bob Woodruff) Date: Wed, 25 May 2005 16:12:31 -0700 Subject: [openib-general] [HELP] Encounter Kernel Panic when Add MellanoxHCA Supporting on 2.6.9 Kernel In-Reply-To: <52d5rftj10.fsf@topspin.com> Message-ID: Lenber> Can I assume it is the problem of HCA card? Or the issue Lenber> is relative with the SMP platform? So strange.. Roland> It's possible it's the HCA but I'm not sure what could be wrong. With Roland> CONFIG_DEBUG_SPINLOCK can you get more of the traceback? The BUG() Roland> should be producing a full stack trace. Ok, I was able to reproduce this error on an IA32 system, running the redhat 2.6.9-5.EL (UP kernel) with the IB patches applied. It turned out to be a problem with the HCA card that had older firmware, 3.0.1. I updated the firmware to 3.3.2 and the system booted OK and everything seems to work fine. woody From roland at topspin.com Wed May 25 16:15:45 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 25 May 2005 16:15:45 -0700 Subject: [openib-general] [HELP] Encounter Kernel Panic when Add MellanoxHCA Supporting on 2.6.9 Kernel In-Reply-To: (Bob Woodruff's message of "Wed, 25 May 2005 16:12:31 -0700") References: Message-ID: <52acmiq4m6.fsf@topspin.com> Bob> Ok, I was able to reproduce this error on an IA32 system, Bob> running the redhat 2.6.9-5.EL (UP kernel) with the IB patches Bob> applied. It turned out to be a problem with the HCA card that Bob> had older firmware, 3.0.1. Did you get any kind of stack dump or traceback? We really shouldn't panic on downrev FW, so I'd like to get to the bottom of this. - R. From robert.j.woodruff at intel.com Wed May 25 16:25:38 2005 From: robert.j.woodruff at intel.com (Woodruff, Robert J) Date: Wed, 25 May 2005 16:25:38 -0700 Subject: [openib-general] [HELP] Encounter Kernel Panic when Add MellanoxHCA Supporting on 2.6.9 Kernel Message-ID: <1AC79F16F5C5284499BB9591B33D6F00047F91F4@orsmsx408> Bob> Ok, I was able to reproduce this error on an IA32 system, Bob> running the redhat 2.6.9-5.EL (UP kernel) with the IB patches Bob> applied. It turned out to be a problem with the HCA card that Bob> had older firmware, 3.0.1. Roland> Did you get any kind of stack dump or traceback? >We really shouldn't panic on downrev FW, so I'd like to get to the >bottom of this. > - R. Unfortunately not, I did not have CONFIG_DEBUG_SPINLOCK set and I did not save the old firmware before loading in the new firmware. Perhaps Lenber can get the traceback info before he updates his card. That would be helpful as I agree it is not desirable to panic on cards with old firmware. woody From roland at topspin.com Wed May 25 16:30:03 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 25 May 2005 16:30:03 -0700 Subject: [openib-general] [HELP] Encounter Kernel Panic when Add MellanoxHCA Supporting on 2.6.9 Kernel In-Reply-To: <1AC79F16F5C5284499BB9591B33D6F00047F91F4@orsmsx408> (Robert J. Woodruff's message of "Wed, 25 May 2005 16:25:38 -0700") References: <1AC79F16F5C5284499BB9591B33D6F00047F91F4@orsmsx408> Message-ID: <52vf56opdw.fsf@topspin.com> Robert> Unfortunately not, I did not have CONFIG_DEBUG_SPINLOCK Robert> set and I did not save the old firmware before loading in Robert> the new firmware. I'll try to build an old fw image and see if I can reproduce it here. - R. From caitlin.bestler at gmail.com Wed May 25 17:10:07 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Wed, 25 May 2005 17:10:07 -0700 Subject: [openib-general] [Fwd: RE: Use of dapl_ring_buffer in KDAPL !] In-Reply-To: <1117048212.12010.3.camel@duffman> References: <1117048212.12010.3.camel@duffman> Message-ID: <469958e0050525171015c2b9ae@mail.gmail.com> My recollection is that one of the original design factors for the ring_buffers was that they are invoked from "common" code (i.e., user or kernel mode). In user mode, the atomics can be more efficient than safely updating two linked lists in a multi-threaded environment. That benefit may have been largely negated by the decision to use dynamic memory allocation though. The third reason was to minimize OS dependencies, which is obviously no longer a factor when coding inside the kernel. In any event, if none of the code using these routines will ever be compiled in user mode then there are no remaining motivations for a specialized ring buffer. If any of the code will still be common between uDAPL and kDAPL then care should be taken to ensure that the ring buffer substitution is transparent to the caller and/or works in user mode. On 5/25/05, Tom Duffy wrote: > Moving this discussion on list since more people might know of a way to > do this with already existing linux primitives. > > -tduffy > > > > ---------- Forwarded message ---------- > From: James Lentini > To: Tom Duffy > Date: Wed, 25 May 2005 15:05:42 -0400 (EDT) > Subject: RE: Use of dapl_ring_buffer in KDAPL ! > > The original idea was that the ring buffer would perform better than > other data structures because it used atomic operations. I don't > believe that this theory was ever validated though. > > Two ring buffers were used because there are two different classes of > events stored: free events and pending events. > > If their is a native linux data structure that provides equivalent > functionality, dapl should use it. > > james > > On Tue, 24 May 2005, Tom Duffy wrote: > > > On Wed, 2005-05-25 at 00:09 +0300, Itamar Rabenstein wrote: > >> I just relised that we can implemet it with 2 lists empty_list and > >> events_list where the events_list will be pop from head and pust at > >> tail > >> > >> no need for dapl_ring_buffer > >> > >> What do you think ? > >> > > > > Since the llists in Linux are doubly linked and circular and they have > > the ability to act like a stack or a queue, I think they should suffice. > > > > -tduffy > > > > P.S. Any reason not to CC the list? > > > > > BodyID:12856038.5.n.logpart (stored separately) > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > > From libor at topspin.com Wed May 25 17:29:54 2005 From: libor at topspin.com (Libor Michalek) Date: Wed, 25 May 2005 17:29:54 -0700 Subject: [openib-general] Re: Re: [PATCH] (repost) sdp: replace mlock with get_user_pages In-Reply-To: <20050525202128.GA7463@mellanox.co.il>; from mst@mellanox.co.il on Wed, May 25, 2005 at 11:21:28PM +0300 References: <20050505110158.GH30653@mellanox.co.il> <20050506175415.I26823@topspin.com> <20050510143232.GF2474@mellanox.co.il> <20050512151051.C23734@topspin.com> <52fywst4z3.fsf@topspin.com> <20050513135145.GA320@mellanox.co.il> <20050513100149.B24123@topspin.com> <20050525202128.GA7463@mellanox.co.il> Message-ID: <20050525172954.A2979@topspin.com> On Wed, May 25, 2005 at 11:21:28PM +0300, Michael S. Tsirkin wrote: > Quoting r. Libor Michalek : > > Subject: Re: Re: [PATCH] (repost) sdp: replace mlock with get_user_pages > > > > On Fri, May 13, 2005 at 04:51:45PM +0300, Michael S. Tsirkin wrote: > > > Quoting r. Roland Dreier : > > > > > > > > Libor> Always call aio_complete() immediately when > > > > Libor> iocb_complete() is called, and only spawn the work thread > > > > Libor> to unlock the memory after the aio_complete() call. The > > > > Libor> patch is below. > > > > > > Libor, I dont think its a good idea - this will break other assumptions, > > > like the assumption that the task mm isnt destroyed before we unlock > > > the memory. > > > > That's a good point. > > > > > > Another alternative would be to always complete aios asynchronously, > > > > which should preserve the order. I guess this would hurt latency for > > > > small ios... > > > > > > To avoid hurting latency, lets count the number of outstanding > > > asynchronous AIOs, and if there are asynchronous AIOs complete > > > all of them asynchronously. > > > > > > Does this make sense? > > > > Yes, except that the current iocb code does not reference individual > > sockets anywhere, and do_iocb_complete would have to be the function > > which decremented the per connection counter of outstanding AIOs. Also, > > since we don't have to do get_user_pages a second time on the send > > path, this would only need to be done on the recv path. > > I thought about this some more: what if we set users to 1 before > releasing the irq spinlock, and call sdp_conn_unlock in thread context > after completing the aio iocb? > Any synchronous transfer would then wait till socket is unloacked. So use 'users' as a reference count, and basically increment the lock before spawning do_iocb_complete. do_iocb_complete would then unlock one reference and the function calling iocb_complete would unlock the other reference? This could work... Remeber that it's possible for many do_iocb_complete functions to be in flight for a given connection. FYI, The problem I saw occured when the iocb_complete was called during the sdp_conn_unlock CQ poll, which is done with IRQ diabled. -Libor From jagana at us.ibm.com Wed May 25 18:38:36 2005 From: jagana at us.ibm.com (Venkata Jagana) Date: Wed, 25 May 2005 18:38:36 -0700 Subject: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux Message-ID: I would like to start a discussion around the convergence of RDMA APIs and ULPs between OpenIB and OpenRDMA projects. As you all know, Infiniband and iWARP based RNICs support RDMA capabilities being exploited by both kernel and user based applications and which can take advantage of these RDMA capabilities through standards based RDMA APIs such as DAPL, IT-API (v1/v2). There exists a set of upper layer protocols, such as NFS, SRP/iSER, SDP, which are mostly kernel based and also exists user based middleware/applications such as DB2, Oracle, scientific applications which would like to use a common set of APIs supported by the underlying operating systems in order to work over different RDMA fabrics like IB and RNICs. >From Linux kernel perspective, it is undesirable to have a different set of APIs and ULPs supported for variety of reasons including but not limited to the duplication, testing effort etc. OpenIB and OpenRDMA projects are separate efforts and are actively working in its own paths to develop the corresponding RDMA support in Linux but we want to make sure we work together to avoid the duplication in providing the support. The proposal for both communities is to start thinking and discussing on how best we could accomplish this commonality between these two projects. BTW, To make this objective further clear - this proposal is not about merging these two projects since each project has its own objective of supporting its RDMA function and rather intended to steer both projects toward the goal of standardizing RDMA APIs and providing common ULPs as applicable. However, we also have a challenge to address in implementing these common ULPs and APIs since OpenIB is currently using verbs PI for Linux defined through an open source process and OpenRDMA is currently defining RNIC-PI (supporting RNIC and IB compatible verbs) for Linux based on the industry standard evolving through Opengroup/ICSC and open source community reviews. The ultimate challenge for us is to come up with a common PI acceptable in Linux while taking into account the standards, hardware vendors portability for device drivers, ULPs etc. Thanks, Venkat -------------- next part -------------- An HTML attachment was scrubbed... URL: From roland at topspin.com Wed May 25 19:35:52 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 25 May 2005 19:35:52 -0700 Subject: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux In-Reply-To: (Venkata Jagana's message of "Wed, 25 May 2005 18:38:36 -0700") References: Message-ID: <52is16ogs7.fsf@topspin.com> I believe the way forward is to evolve the existing drivers/infiniband code already in Linux into a drivers/rdma that supports both IB and RNICs. To be extremely blunt, I believe the RNIC-PI is irrelevant to the Linux kernel -- no IB vendors will support ripping out a working midlayer and starting from scratch, and it doesn't make sense to have two essentially equivalent midlayers in the same kernel. To put a really concrete proposal on the table, I would suggest to start by extending the current ib_client registration structure, which looks like struct ib_client { char *name; void (*add) (struct ib_device *); void (*remove)(struct ib_device *); struct list_head list; }; by extending the current enum ib_node_type to something like enum rdma_device_type { RDMA_DEVICE_IB_CA, RDMA_DEVICE_IB_SWITCH, RDMA_DEVICE_IB_ROUTER, RDMA_DEVICE_RNIC }; Then the various pieces of code layered on top of the RDMA midlayer can decide whether they want to deal with a particular device or not by looking at the node_type member. For example, the IB CM, IPoIB, etc. could ignore devices of type RDMA_DEVICE_RNIC, while SDP or iSER would use all devices and the RNIC CM would take only devices of type RDMA_DEVICE_RNIC. Then someone would have to start implementing a low-level driver for a specific RNIC, and find which modifications to the existing verbs are required. For example, I believe the QP attribute structure passed into the QP modify verb probably has to become a union containing the IB attributes and the RNIC attributes. However, most verbs should work fine with at most trivial modifications. The existing OpenIB SDP code will be a good example to study as we determine what abstractions need to be added to make it simple for consumers to deal with the differences between IB and RNIC. - R. From hozer at hozed.org Wed May 25 20:58:00 2005 From: hozer at hozed.org (Troy Benjegerdes) Date: Wed, 25 May 2005 22:58:00 -0500 Subject: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux In-Reply-To: <52is16ogs7.fsf@topspin.com> References: <52is16ogs7.fsf@topspin.com> Message-ID: <20050526035800.GZ999@kalmia.hozed.org> > Then someone would have to start implementing a low-level driver for a > specific RNIC, and find which modifications to the existing verbs are ^^^^^^^^^^^^^^^^ I'll believe that RNICs are actually going to work and it's worth talking about OpenRDMA when I can see code that runs. Initially, I'd say that extending OpenIB is going to be the best way forward, once there is a working RNIC driver. From halr at voltaire.com Wed May 25 20:56:38 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 25 May 2005 23:56:38 -0400 Subject: [openib-general] umad abi 2 v 3 and multicast join failed In-Reply-To: <20050525230658.GY999@kalmia.hozed.org> References: <20050525230658.GY999@kalmia.hozed.org> Message-ID: <1117079586.4479.432.camel@hal.voltaire.com> On Wed, 2005-05-25 at 19:06, Troy Benjegerdes wrote: > I was running a crufty version of opensm (compiled from the > roland-uverbs branch), roland-uverbs is an orphaned branch at this point. You should not be using this version. It is not supported and out of date. You should use the one on the trunk. See below. > and I started getting these kinds of errors for > no apparent reason: > > ib0: multicast join failed for ff12:401b:ffff:0:0:0:ffff:ffff, status > -22 > ib0: multicast join failed for ff12:401b:ffff:0:0:0:ffff:ffff, status > -22 > > I'm running 2.6.11 kernels, and 'stock' modules.. I just tried > rebuilding opensm from the latest SVN, but it apparently needs a new > umad driver.. > > warn: [24878] umad_init: wrong ABI version: > /sys/class/infiniband_mad/abi_version is 2 but library ABI is 3 Right. This is old OpenSM (actually old libibumad) with the latest from OpenIB svn (past where I put the changes to support send side RMPP in). Note that I did say the following: user_mad: Support RMPP on send side Note that this change will need a coordinated change to OpenSM and some userspace/management libraries which will be done as soon as possible once this patch is accepted. It was followed by a patch to userspace/management which includes OpenSM for this: userspace/management changes to support send side RMPP (needs change to linux-kernel/infiniband/core/user_mad.c) ABI_VERSION is now 3 RMPP is enabled in build SA GetTable is now supported properly (within current RMPP limitations) > I suppose I need to rebuild the kernel ib_umad (and maybe everything > else for good measure).. No. It's the other way around. You need to rebuild OpenSM. > And if I do that, should I expect OpenSM to > work better regarding the multicast issue? > > Also, what will happen if I run opensm on two different nodes? Will they > fight, or will one of them figure out how to be a backup slave SM if the > first goes down? SM mastership should work. You should be able to run any number of OpenSMs in a subnet and one of them will become master. [This is a separate issue from the ABI version change.] -- Hal From roland at topspin.com Wed May 25 21:07:25 2005 From: roland at topspin.com (Roland Dreier) Date: Wed, 25 May 2005 21:07:25 -0700 Subject: [openib-general] umad abi 2 v 3 and multicast join failed In-Reply-To: <1117079586.4479.432.camel@hal.voltaire.com> (Hal Rosenstock's message of "25 May 2005 23:56:38 -0400") References: <20050525230658.GY999@kalmia.hozed.org> <1117079586.4479.432.camel@hal.voltaire.com> Message-ID: <52br6yocjm.fsf@topspin.com> Troy> warn: [24878] umad_init: wrong ABI version: Troy> /sys/class/infiniband_mad/abi_version is 2 but library ABI Troy> is 3 Hal> Right. This is old OpenSM (actually old libibumad) with the Hal> latest from OpenIB svn (past where I put the changes to Hal> support send side RMPP in). Are you sure? The kernel seems to be reporting an older ABI version than the library is looking for. In general, given that kernels 2.6.11 and 2.6.12 are shipping with ABI version 2, does it make sense to avoid problems like this by keeping the old userspace code around and having the library decide at runtime which ABI to use? - R. From iod00d at hp.com Wed May 25 21:47:00 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 25 May 2005 21:47:00 -0700 Subject: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux In-Reply-To: <52is16ogs7.fsf@topspin.com> References: <52is16ogs7.fsf@topspin.com> Message-ID: <20050526044700.GK671@esmail.cup.hp.com> On Wed, May 25, 2005 at 07:35:52PM -0700, Roland Dreier wrote: > I believe the way forward is to evolve the existing drivers/infiniband > code already in Linux into a drivers/rdma that supports both IB and > RNICs. To be extremely blunt, I believe the RNIC-PI is irrelevant to > the Linux kernel -- no IB vendors will support ripping out a working > midlayer and starting from scratch, and it doesn't make sense to have > two essentially equivalent midlayers in the same kernel. Yes, I think that's an accurate assessment. ... > The existing OpenIB SDP code will be a good example to study as we > determine what abstractions need to be added to make it simple for > consumers to deal with the differences between IB and RNIC. Venkata, Interesting coincidence: I was talking with someone (at HP) today who knows substantially more than I do about RNICs. They indicated RNICs need to manage TCP state on the card from userspace. I suspect that's only possible through a private interface (e.g. ioctl() or /proc) or the non-existant (in kernel.org) TOE implementation. Is this correct? If it is correct, any ideas/proposals on how that functionality will get into kernel.org and where it might fit? Solving the RDMA part of the problem isn't useful if one can't configure/manage the TCP part of the RNIC. hth, grant From iod00d at hp.com Wed May 25 21:52:47 2005 From: iod00d at hp.com (Grant Grundler) Date: Wed, 25 May 2005 21:52:47 -0700 Subject: [openib-general] umad abi 2 v 3 and multicast join failed In-Reply-To: <52br6yocjm.fsf@topspin.com> References: <20050525230658.GY999@kalmia.hozed.org> <1117079586.4479.432.camel@hal.voltaire.com> <52br6yocjm.fsf@topspin.com> Message-ID: <20050526045247.GL671@esmail.cup.hp.com> On Wed, May 25, 2005 at 09:07:25PM -0700, Roland Dreier wrote: > In general, given that kernels 2.6.11 and 2.6.12 are shipping with ABI > version 2, does it make sense to avoid problems like this by keeping > the old userspace code around and having the library decide at runtime > which ABI to use? Only if you want to continue providing fixes for 2.6.11 and 2.6.12. It would be nice if someone put together a source snapshot that would work with those releases. But I'm not sure how to make that available to some random person who is using it. Personally, since no distro is providing support for ABI v2, I would not bundle both ABIs together in one library. grant From jagana at us.ibm.com Wed May 25 23:20:14 2005 From: jagana at us.ibm.com (Venkata Jagana) Date: Wed, 25 May 2005 23:20:14 -0700 Subject: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux In-Reply-To: <52is16ogs7.fsf@topspin.com> Message-ID: I am sure the developers in both of these communities have strong opinions in one way or the other about the use of common interface and whatever it is but ultimately we need to find a best possible way to move forward in order to support IB, RNICs and other future RDMA fabrics. :) Absolutely, we would like to see a common mid layer with RDMA-specific components (for example, IB-CM and RNIC-CM would mostly be different but with very minimal commonality between them) developed but also should work to provide a transport-neutral interface. Today, we do not have a RDMA transport-neutral support with IB verbs PI in the kernel but we would like to see such an interface so that we can extend it to support variety of RDMA fabrics. Just because we have some interface supported in the kernel today doesn't mean that we need to live with that interface forever. Within Linux kernel, we absolutely want to develop an interface that is extendable to support the future fabrics and we can evolve this through an open source process that is acceptable to linux kernel community. And BTW, I am not suggesting to rip of this existing code right away but rather continue to use and evolve it. I would like to really understand the technical reasons why you say RNIC-PI is irrelevant to Linux kernel. RNIC-PI is developed to support not only the RNICs but it is also IB compatible. This interface is something that is developed just like the open source process but with much broader community effort and also with review of linux OpenRDMA community and kernel community members. Thanks Venkat Roland Dreier wrote on 05/25/2005 07:35:52 PM: > I believe the way forward is to evolve the existing drivers/infiniband > code already in Linux into a drivers/rdma that supports both IB and > RNICs. To be extremely blunt, I believe the RNIC-PI is irrelevant to > the Linux kernel -- no IB vendors will support ripping out a working > midlayer and starting from scratch, and it doesn't make sense to have > two essentially equivalent midlayers in the same kernel. > > To put a really concrete proposal on the table, I would suggest to > start by extending the current ib_client registration structure, which > looks like > > struct ib_client { > char *name; > void (*add) (struct ib_device *); > void (*remove)(struct ib_device *); > > struct list_head list; > }; > > by extending the current enum ib_node_type to something like > > enum rdma_device_type { > RDMA_DEVICE_IB_CA, > RDMA_DEVICE_IB_SWITCH, > RDMA_DEVICE_IB_ROUTER, > RDMA_DEVICE_RNIC > }; > > Then the various pieces of code layered on top of the RDMA midlayer > can decide whether they want to deal with a particular device or not > by looking at the node_type member. For example, the IB CM, IPoIB, > etc. could ignore devices of type RDMA_DEVICE_RNIC, while SDP or iSER > would use all devices and the RNIC CM would take only devices of type > RDMA_DEVICE_RNIC. > > Then someone would have to start implementing a low-level driver for a > specific RNIC, and find which modifications to the existing verbs are > required. For example, I believe the QP attribute structure passed > into the QP modify verb probably has to become a union containing the > IB attributes and the RNIC attributes. However, most verbs should > work fine with at most trivial modifications. > > The existing OpenIB SDP code will be a good example to study as we > determine what abstractions need to be added to make it simple for > consumers to deal with the differences between IB and RNIC. > > - R. -------------- next part -------------- An HTML attachment was scrubbed... URL: From lenber.cong at intel.com Thu May 26 00:00:21 2005 From: lenber.cong at intel.com (Cong, Lenber) Date: Thu, 26 May 2005 15:00:21 +0800 Subject: [openib-general] [HELP] Encounter Kernel Panic when Add MellanoxHCA Supporting on 2.6.9 Kernel Message-ID: Dear Woody & Roland I downloaded the newest firmware and updated the system with it. Now it works. Kernel can be booted and module can be loaded successfully. Thanks for your great help. It is very useful to me. Thanks - Lenber -----Original Message----- From: Roland Dreier [mailto:roland at topspin.com] Sent: 2005年5月26日 7:30 To: Woodruff, Robert J Cc: Cong, Lenber; openib-general at openib.org Subject: Re: [openib-general] [HELP] Encounter Kernel Panic when Add MellanoxHCA Supporting on 2.6.9 Kernel Robert> Unfortunately not, I did not have CONFIG_DEBUG_SPINLOCK Robert> set and I did not save the old firmware before loading in Robert> the new firmware. I'll try to build an old fw image and see if I can reproduce it here. - R. From mst at mellanox.co.il Thu May 26 00:29:09 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Thu, 26 May 2005 10:29:09 +0300 Subject: [openib-general] Re: umad abi 2 v 3 and multicast join failed In-Reply-To: <20050526045247.GL671@esmail.cup.hp.com> References: <20050525230658.GY999@kalmia.hozed.org> <1117079586.4479.432.camel@hal.voltaire.com> <52br6yocjm.fsf@topspin.com> <20050526045247.GL671@esmail.cup.hp.com> Message-ID: <20050526072909.GT27428@mellanox.co.il> Quoting r. Grant Grundler : > Subject: Re: umad abi 2 v 3 and multicast join failed > > On Wed, May 25, 2005 at 09:07:25PM -0700, Roland Dreier wrote: > > In general, given that kernels 2.6.11 and 2.6.12 are shipping with ABI > > version 2, does it make sense to avoid problems like this by keeping > > the old userspace code around and having the library decide at runtime > > which ABI to use? > > Only if you want to continue providing fixes for 2.6.11 and 2.6.12. > It would be nice if someone put together a source snapshot that > would work with those releases. But I'm not sure how to > make that available to some random person who is using it. Making a subversion branch could be one way to start. Actually, it would be quite useful if whatever is submitted to mainline is kept on some subversion branch, too. Roland - would that disrupt your development process a lot? -- MST - Michael S. Tsirkin From knownature at 163.com Thu May 26 00:48:16 2005 From: knownature at 163.com (=?gb2312?B?bG9uZ2dlbiBwZW5n?=) Date: Thu, 26 May 2005 15:48:16 +0800 (CST) Subject: [openib-general] Question about APM in openib gen1 Message-ID: <42957F40.0000D5.01303@m117.163.com> Hi. I have tested APM in openib gen1 sources, but can not succssfully. I have two node, the node 1 has one cable link to switch and the node 2 has two cable to switch. When establishing connection using CM, the node 2 is the listen server, and the node 1 is the client. After connection established , the client call ib_cm_alternate_path_load() to load the alter path, it can return successfully , but CM report that the server reject to load a alter path, so the APM failed. The server node can get a TS_IB_CM_LAP_RECEIVED event , and do nothing when receive this event. Should the server do something to handle the LAP event? if true , how to do? Thank you! -------------- next part -------------- An HTML attachment was scrubbed... URL: From hch at lst.de Thu May 26 04:31:34 2005 From: hch at lst.de (Christoph Hellwig) Date: Thu, 26 May 2005 13:31:34 +0200 Subject: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux In-Reply-To: References: Message-ID: <20050526113134.GA31308@lst.de> On Wed, May 25, 2005 at 06:38:36PM -0700, Venkata Jagana wrote: > However, we also have a challenge to address in implementing these common > ULPs and APIs > since OpenIB is currently using verbs PI for Linux defined through an open > source process and > OpenRDMA is currently defining RNIC-PI (supporting RNIC and IB compatible > verbs) for Linux > based on the industry standard evolving through Opengroup/ICSC and open > source community reviews. The RNIC-PI APIs are a total piece of junk redoing all the mistakes DAPL, IT-API & friends did. And the comittee is absolutely not willing to listen to open source community feedback. I'd suggest your start looking at the current OpenIB APIs and how they can be extented to not only cover IB but iWARP aswell, adding layers or abstraction and generalization where needed, From hch at lst.de Thu May 26 04:33:46 2005 From: hch at lst.de (Christoph Hellwig) Date: Thu, 26 May 2005 13:33:46 +0200 Subject: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux In-Reply-To: References: <52is16ogs7.fsf@topspin.com> Message-ID: <20050526113346.GB31308@lst.de> On Wed, May 25, 2005 at 11:20:14PM -0700, Venkata Jagana wrote: > I would like to really understand the technical reasons why you say > RNIC-PI is irrelevant to Linux kernel. > RNIC-PI is developed to support not only the RNICs but it is also IB > compatible. This interface is > something that is developed just like the open source process but with > much broader community effort No, it's not. It's developed by an industry consortium that lacks any taste. From halr at voltaire.com Thu May 26 05:06:11 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 26 May 2005 08:06:11 -0400 Subject: [openib-general] umad abi 2 v 3 and multicast join failed In-Reply-To: <52br6yocjm.fsf@topspin.com> References: <20050525230658.GY999@kalmia.hozed.org> <1117079586.4479.432.camel@hal.voltaire.com> <52br6yocjm.fsf@topspin.com> Message-ID: <1117109170.4479.1008.camel@hal.voltaire.com> On Thu, 2005-05-26 at 00:07, Roland Dreier wrote: > Troy> warn: [24878] umad_init: wrong ABI version: > Troy> /sys/class/infiniband_mad/abi_version is 2 but library ABI > Troy> is 3 > > Hal> Right. This is old OpenSM (actually old libibumad) with the > Hal> latest from OpenIB svn (past where I put the changes to > Hal> support send side RMPP in). > > Are you sure? The kernel seems to be reporting an older ABI version > than the library is looking for. Then maybe I got it backwards. The email said that the OpenSM from roland-uverbs branch was being used which has to be the older ABI version. The fact that the library version is 3 indicates to me that the OpenSM being run is from the trunk. > In general, given that kernels 2.6.11 and 2.6.12 are shipping with ABI > version 2, also we know there is some 2.6.9 backport work. I'm not sure which ABI version this is using but if this is kept up to date, it will transition to the newer one. So the old ABI version is a current issue for 2.6.11/12 relative to the current trunk for userspace/management. > does it make sense to avoid problems like this by keeping > the old userspace code around and having the library decide at runtime > which ABI to use? It would be more than this to make it run time selectable as the API changes filter back into the userspace applications. Are there any examples of this elsewhere to look at ? Would a branch suffice for this (old ABI version) ? Would the older ABI version need to accomodate any OpenSM and diags development that might be applicable to it ? Any idea on how long the old ABI will need supporting ? What is the general rule on older ABI support ? Is there a limit on how many are supported and how long the older ABIs are supported ? Is this just based on the needs of the community ? These are questions I think need answering before deciding which way to go on this. -- Hal From Thomas.Talpey at netapp.com Thu May 26 05:27:40 2005 From: Thomas.Talpey at netapp.com (Talpey, Thomas) Date: Thu, 26 May 2005 08:27:40 -0400 Subject: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux In-Reply-To: References: <52is16ogs7.fsf@topspin.com> Message-ID: <6.2.1.2.2.20050526075735.04c9a480@exnane01.nane.netapp.com> At 02:20 AM 5/26/2005, Venkata Jagana wrote: >I would like to really understand the technical reasons why you say RNIC-PI is irrelevant to Linux kernel. >RNIC-PI is developed to support not only the RNICs but it is also IB compatible. I'm not Roland, but my belief is that until RNIC-PI exists as working code on a reference implementation, it is in fact not yet relevant to Linux. Perhaps you can outline the schedule to get there? I will point out that Roland's "science project" driver, which forms the base for the current OpenIB implementation, was released to the community on May 17, 2004 - one year ago. It is my assertion* that OpenRDMA must work with, and within, OpenIB to create a single RDMA implementation for Linux, in order to create the best and broadest support, as early as possible. Ok, so the names are a little upside-down. :-) Tom. * From hch at lst.de Thu May 26 05:45:18 2005 From: hch at lst.de (Christoph Hellwig) Date: Thu, 26 May 2005 14:45:18 +0200 Subject: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux In-Reply-To: <6.2.1.2.2.20050526075735.04c9a480@exnane01.nane.netapp.com> References: <52is16ogs7.fsf@topspin.com> <6.2.1.2.2.20050526075735.04c9a480@exnane01.nane.netapp.com> Message-ID: <20050526124518.GA32495@lst.de> On Thu, May 26, 2005 at 08:27:40AM -0400, Talpey, Thomas wrote: > * The sf.net mail archives are too fucked up to allow retrieving attachments. From blist at aon.at Thu May 26 06:28:15 2005 From: blist at aon.at (Bernhard Fischer) Date: Thu, 26 May 2005 15:28:15 +0200 Subject: [openib-general] [PATCH] mthca: cosmetic fix for handle_error_cqe() Message-ID: <20050526132815.GH15688@aon.at> Hi, - remove unneeded if-condition in mthca's cq error path Signed-off-by: Bernhard Fischer -------------- next part -------------- diff -X excl -rduNp gen2.2423.oorig/trunk/src/linux-kernel/infiniband/hw/mthca/mthca_cq.c gen2.2423/trunk/src/linux-kernel/infiniband/hw/mthca/mthca_cq.c --- gen2.2423.oorig/trunk/src/linux-kernel/infiniband/hw/mthca/mthca_cq.c 2005-04-08 10:22:08.000000000 +0200 +++ gen2.2423/trunk/src/linux-kernel/infiniband/hw/mthca/mthca_cq.c 2005-05-23 14:26:08.000000000 +0200 @@ -280,7 +280,7 @@ static int handle_error_cqe(struct mthca int dbd; u32 new_wqe; - if (1 && cqe->syndrome != SYNDROME_WR_FLUSH_ERR) { + if (cqe->syndrome != SYNDROME_WR_FLUSH_ERR) { int j; mthca_dbg(dev, "%x/%d: error CQE -> QPN %06x, WQE @ %08x\n", From Thomas.Talpey at netapp.com Thu May 26 06:48:02 2005 From: Thomas.Talpey at netapp.com (Talpey, Thomas) Date: Thu, 26 May 2005 09:48:02 -0400 Subject: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux In-Reply-To: <20050526124518.GA32495@lst.de> References: <52is16ogs7.fsf@topspin.com> <6.2.1.2.2.20050526075735.04c9a480@exnane01.nane.netapp.com> <20050526124518.GA32495@lst.de> Message-ID: <6.2.1.2.2.20050526094539.0614aeb0@exnane01.nane.netapp.com> At 08:45 AM 5/26/2005, Christoph Hellwig wrote: >>> >The sf.net mail archives are too f*d up to allow retrieving >attachments. Here it is again... My opinions/assertions only, presented at the April OpenRDMA f2f. Tom. -------------- next part -------------- A non-text attachment was scrubbed... Name: LinuxRDMA-Talpey20050526.pdf Type: application/pdf Size: 22753 bytes Desc: not available URL: From jagana at us.ibm.com Thu May 26 08:25:22 2005 From: jagana at us.ibm.com (Venkata Jagana) Date: Thu, 26 May 2005 08:25:22 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux In-Reply-To: <20050526113346.GB31308@lst.de> Message-ID: rdma-developers-admin at lists.sourceforge.net wrote on 05/26/2005 04:33:46 AM: > On Wed, May 25, 2005 at 11:20:14PM -0700, Venkata Jagana wrote: > > I would like to really understand the technical reasons why you say > > RNIC-PI is irrelevant to Linux kernel. > > RNIC-PI is developed to support not only the RNICs but it is also IB > > compatible. This interface is > > something that is developed just like the open source process but with > > much broader community effort > > No, it's not. It's developed by an industry consortium that lacks any > taste. OpenRDMA community is actively involved in the development of this specification. Of course, your critical review input is taken and responded to. As always from Linux implementation standpoint, the implementation of this PI will evolve through open source community effort just like any other PI. Thanks Venkat -------------- next part -------------- An HTML attachment was scrubbed... URL: From robert.j.woodruff at intel.com Thu May 26 08:49:59 2005 From: robert.j.woodruff at intel.com (Bob Woodruff) Date: Thu, 26 May 2005 08:49:59 -0700 Subject: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: Message-ID: Venkat> I would like to start a discussion around the convergence of RDMA APIs and ULPs Venkat> between OpenIB and OpenRDMA projects. There is already a RDMA device independent API being developed for the kernel by people on this list. It is starting with the kDAPL code base, which was designed to support both IBA and iWarp devices. People have already started to re-work this code and API so that meets the Linux coding standards, etc. It is being developed in an open source way, rather than by some committee. I would suggest that it is at this level that iWarp or other RDMA devices fit into the stack. ULPs that want to be able to run on any RDMA device (such as iSer, NFS-R) are already coding to this interface. I am not sure that having one mid-layer is really required, at least initially. Once the RNIC people have a mid-layer that interfaces with the RDMA API (kDAPL derivative), we can look at where there might be common code that could be shared between the InfiniBand mid-layer and the iWarp mid-layer. This would allow the RNIC vendors to develop code that could use the same ULPs without slowing down the InfiniBand development. Finally, until there is some consensus about allowing TCP offload in Linux, I see no need to start to hack up the InfiniBand stack to support iWarp. woody From jagana at us.ibm.com Thu May 26 09:13:37 2005 From: jagana at us.ibm.com (Venkata Jagana) Date: Thu, 26 May 2005 09:13:37 -0700 Subject: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: Message-ID: "Bob Woodruff" wrote on 05/26/2005 08:49:59 AM: > Venkat> I would like to start a discussion around the convergence of RDMA > APIs and ULPs > Venkat> between OpenIB and OpenRDMA projects. > > Once the RNIC people have a mid-layer that interfaces with the RDMA API > (kDAPL > derivative), we can look at where there might be common code that could be > shared between the InfiniBand mid-layer and the iWarp mid-layer. This would > allow the RNIC vendors to develop code that could use the same ULPs without > slowing down the InfiniBand development. Absolutely and this is exactly what I would like to see without impacting the IB development but supporting common ULPs is critical. Thanks Venkat -------------- next part -------------- An HTML attachment was scrubbed... URL: From blist at aon.at Thu May 26 09:14:55 2005 From: blist at aon.at (Bernhard Fischer) Date: Thu, 26 May 2005 18:14:55 +0200 Subject: [openib-general] [PATCH] mthca, ibverbs userspace readability Message-ID: <20050526161455.GL15688@aon.at> Hi, No object code changes. linux-kernel/infiniband/ulp/sdp/sdp_conn.c | 1 userspace/libibverbs/include/infiniband/driver.h | 34 +- userspace/libibverbs/include/infiniband/kern-abi.h | 2 userspace/libibverbs/include/infiniband/verbs.h | 253 ++++++++++----------- userspace/libmthca/src/cq.c | 9 userspace/libmthca/src/mthca.h | 87 +++---- 6 files changed, 198 insertions(+), 188 deletions(-) - improve readability of mthca and ibverbs headers installed to userspace. Signed-off-by Bernhard Fischer Those were hard/impossible to read for ts!=8 before. Please at least consider; I'm aware that those are purely cosmetic, granted. TIA for applying nonetheless. -------------- next part -------------- diff -X excl -rduNp gen2.2491.oorig/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_conn.c gen2.2491/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_conn.c --- gen2.2491.oorig/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_conn.c 2005-05-16 22:09:24.000000000 +0200 +++ gen2.2491/trunk/src/linux-kernel/infiniband/ulp/sdp/sdp_conn.c 2005-05-26 16:29:31.197039104 +0200 @@ -842,7 +842,6 @@ void sdp_conn_relock(struct sdp_opt *con sdp_dbg_warn(conn, "Error <%d> rearming send CQ", result); - rearm = 0; } else break; /* exit CQ handler routine */ diff -X excl -rduNp gen2.2491.oorig/trunk/src/userspace/libibverbs/include/infiniband/driver.h gen2.2491/trunk/src/userspace/libibverbs/include/infiniband/driver.h --- gen2.2491.oorig/trunk/src/userspace/libibverbs/include/infiniband/driver.h 2005-05-16 22:09:23.000000000 +0200 +++ gen2.2491/trunk/src/userspace/libibverbs/include/infiniband/driver.h 2005-05-26 16:29:31.200038648 +0200 @@ -60,35 +60,37 @@ * ops member filled in. If the driver does not support the device, * it should return NULL from openib_driver_init(). */ - + typedef struct ibv_device *(*ibv_driver_init_func)(struct sysfs_class_device *); extern int ibv_cmd_get_context(int num_comp, struct ibv_context *context, - struct ibv_get_context *cmd, size_t cmd_size); + struct ibv_get_context *cmd, size_t cmd_size); extern int ibv_cmd_query_port(struct ibv_context *context, uint8_t port_num, - struct ibv_port_attr *port_attr, - struct ibv_query_port *cmd, size_t cmd_size); + struct ibv_port_attr *port_attr, + struct ibv_query_port *cmd, size_t cmd_size); extern int ibv_cmd_alloc_pd(struct ibv_context *context, struct ibv_pd *pd, - struct ibv_alloc_pd *cmd, size_t cmd_size); + struct ibv_alloc_pd *cmd, size_t cmd_size); extern int ibv_cmd_dealloc_pd(struct ibv_pd *pd); extern int ibv_cmd_reg_mr(struct ibv_pd *pd, void *addr, size_t length, - uint64_t hca_va, enum ibv_access_flags access, - struct ibv_mr *mr, struct ibv_reg_mr *cmd, - size_t cmd_size); + uint64_t hca_va, enum ibv_access_flags access, + struct ibv_mr *mr, struct ibv_reg_mr *cmd, + size_t cmd_size); extern int ibv_cmd_dereg_mr(struct ibv_mr *mr); extern int ibv_cmd_create_cq(struct ibv_context *context, int cqe, - struct ibv_cq *cq, - struct ibv_create_cq *cmd, size_t cmd_size); + struct ibv_cq *cq, + struct ibv_create_cq *cmd, size_t cmd_size); extern int ibv_cmd_destroy_cq(struct ibv_cq *cq); extern int ibv_cmd_create_qp(struct ibv_pd *pd, - struct ibv_qp *qp, struct ibv_qp_init_attr *attr, - struct ibv_create_qp *cmd, size_t cmd_size); + struct ibv_qp *qp, struct ibv_qp_init_attr *attr, + struct ibv_create_qp *cmd, size_t cmd_size); extern int ibv_cmd_modify_qp(struct ibv_qp *qp, struct ibv_qp_attr *attr, - enum ibv_qp_attr_mask attr_mask, - struct ibv_modify_qp *cmd, size_t cmd_size); + enum ibv_qp_attr_mask attr_mask, + struct ibv_modify_qp *cmd, size_t cmd_size); extern int ibv_cmd_destroy_qp(struct ibv_qp *qp); -extern int ibv_cmd_attach_mcast(struct ibv_qp *qp, union ibv_gid *gid, uint16_t lid); -extern int ibv_cmd_detach_mcast(struct ibv_qp *qp, union ibv_gid *gid, uint16_t lid); +extern int ibv_cmd_attach_mcast(struct ibv_qp *qp, union ibv_gid *gid, + uint16_t lid); +extern int ibv_cmd_detach_mcast(struct ibv_qp *qp, union ibv_gid *gid, + uint16_t lid); #endif /* INFINIBAND_DRIVER_H */ diff -X excl -rduNp gen2.2491.oorig/trunk/src/userspace/libibverbs/include/infiniband/kern-abi.h gen2.2491/trunk/src/userspace/libibverbs/include/infiniband/kern-abi.h --- gen2.2491.oorig/trunk/src/userspace/libibverbs/include/infiniband/kern-abi.h 2005-04-26 20:58:02.000000000 +0200 +++ gen2.2491/trunk/src/userspace/libibverbs/include/infiniband/kern-abi.h 2005-05-26 17:05:42.530946176 +0200 @@ -165,7 +165,7 @@ struct ibv_query_port_resp { __u8 active_width; __u8 active_speed; __u8 phys_state; - __u8 reserved[3]; + __u8 reserved[3]; /* as always, pad to next __u64 */ }; struct ibv_query_gid { diff -X excl -rduNp gen2.2491.oorig/trunk/src/userspace/libibverbs/include/infiniband/verbs.h gen2.2491/trunk/src/userspace/libibverbs/include/infiniband/verbs.h --- gen2.2491.oorig/trunk/src/userspace/libibverbs/include/infiniband/verbs.h 2005-05-16 22:09:23.000000000 +0200 +++ gen2.2491/trunk/src/userspace/libibverbs/include/infiniband/verbs.h 2005-05-26 16:29:31.203038192 +0200 @@ -51,10 +51,10 @@ BEGIN_C_DECLS union ibv_gid { - uint8_t raw[16]; + uint8_t raw[16]; struct { - uint64_t subnet_prefix; - uint64_t interface_id; + uint64_t subnet_prefix; + uint64_t interface_id; } global; }; @@ -67,34 +67,34 @@ enum ibv_mtu { }; enum ibv_port_state { - IBV_PORT_NOP = 0, - IBV_PORT_DOWN = 1, - IBV_PORT_INIT = 2, - IBV_PORT_ARMED = 3, - IBV_PORT_ACTIVE = 4, - IBV_PORT_ACTIVE_DEFER = 5 + IBV_PORT_NOP = 0, + IBV_PORT_DOWN = 1, + IBV_PORT_INIT = 2, + IBV_PORT_ARMED = 3, + IBV_PORT_ACTIVE = 4, + IBV_PORT_ACTIVE_DEFER = 5 }; struct ibv_port_attr { - enum ibv_port_state state; - enum ibv_mtu max_mtu; - enum ibv_mtu active_mtu; - int gid_tbl_len; - uint32_t port_cap_flags; - uint32_t max_msg_sz; - uint32_t bad_pkey_cntr; - uint32_t qkey_viol_cntr; - uint16_t pkey_tbl_len; - uint16_t lid; - uint16_t sm_lid; - uint8_t lmc; - uint8_t max_vl_num; - uint8_t sm_sl; - uint8_t subnet_timeout; - uint8_t init_type_reply; - uint8_t active_width; - uint8_t active_speed; - uint8_t phys_state; + enum ibv_port_state state; + enum ibv_mtu max_mtu; + enum ibv_mtu active_mtu; + int gid_tbl_len; + uint32_t port_cap_flags; + uint32_t max_msg_sz; + uint32_t bad_pkey_cntr; + uint32_t qkey_viol_cntr; + uint16_t pkey_tbl_len; + uint16_t lid; + uint16_t sm_lid; + uint8_t lmc; + uint8_t max_vl_num; + uint8_t sm_sl; + uint8_t subnet_timeout; + uint8_t init_type_reply; + uint8_t active_width; + uint8_t active_speed; + uint8_t phys_state; }; enum ibv_event_type { @@ -159,68 +159,68 @@ enum ibv_wc_opcode { * Set value of IBV_WC_RECV so consumers can test if a completion is a * receive by testing (opcode & IBV_WC_RECV). */ - IBV_WC_RECV = 1 << 7, + IBV_WC_RECV = 1 << 7, IBV_WC_RECV_RDMA_WITH_IMM }; enum ibv_wc_flags { - IBV_WC_GRH = 1 << 0, - IBV_WC_WITH_IMM = 1 << 1 + IBV_WC_GRH = 1 << 0, + IBV_WC_WITH_IMM = 1 << 1 }; struct ibv_wc { - uint64_t wr_id; - enum ibv_wc_status status; - enum ibv_wc_opcode opcode; - uint32_t vendor_err; - uint32_t byte_len; - uint32_t imm_data; /* in network byte order */ - uint32_t qp_num; - uint32_t src_qp; - enum ibv_wc_flags wc_flags; - uint16_t pkey_index; - uint16_t slid; - uint8_t sl; - uint8_t dlid_path_bits; + uint64_t wr_id; + enum ibv_wc_status status; + enum ibv_wc_opcode opcode; + uint32_t vendor_err; + uint32_t byte_len; + uint32_t imm_data; /* in network byte order */ + uint32_t qp_num; + uint32_t src_qp; + enum ibv_wc_flags wc_flags; + uint16_t pkey_index; + uint16_t slid; + uint8_t sl; + uint8_t dlid_path_bits; }; enum ibv_access_flags { - IBV_ACCESS_LOCAL_WRITE = 1, - IBV_ACCESS_REMOTE_WRITE = (1<<1), - IBV_ACCESS_REMOTE_READ = (1<<2), - IBV_ACCESS_REMOTE_ATOMIC = (1<<3), - IBV_ACCESS_MW_BIND = (1<<4) + IBV_ACCESS_LOCAL_WRITE = 1, + IBV_ACCESS_REMOTE_WRITE = (1<<1), + IBV_ACCESS_REMOTE_READ = (1<<2), + IBV_ACCESS_REMOTE_ATOMIC = (1<<3), + IBV_ACCESS_MW_BIND = (1<<4) }; struct ibv_pd { struct ibv_context *context; - uint32_t handle; + uint32_t handle; }; struct ibv_mr { struct ibv_context *context; - struct ibv_pd *pd; - uint32_t handle; - uint32_t lkey; - uint32_t rkey; + struct ibv_pd *pd; + uint32_t handle; + uint32_t lkey; + uint32_t rkey; }; struct ibv_global_route { - union ibv_gid dgid; - uint32_t flow_label; - uint8_t sgid_index; - uint8_t hop_limit; - uint8_t traffic_class; + union ibv_gid dgid; + uint32_t flow_label; + uint8_t sgid_index; + uint8_t hop_limit; + uint8_t traffic_class; }; struct ibv_ah_attr { - struct ibv_global_route grh; - uint16_t dlid; - uint8_t sl; - uint8_t src_path_bits; - uint8_t static_rate; - uint8_t is_global; - uint8_t port_num; + struct ibv_global_route grh; + uint16_t dlid; + uint8_t sl; + uint8_t src_path_bits; + uint8_t static_rate; + uint8_t is_global; + uint8_t port_num; }; enum ibv_qp_type { @@ -230,45 +230,45 @@ enum ibv_qp_type { }; struct ibv_qp_cap { - uint32_t max_send_wr; - uint32_t max_recv_wr; - uint32_t max_send_sge; - uint32_t max_recv_sge; - uint32_t max_inline_data; + uint32_t max_send_wr; + uint32_t max_recv_wr; + uint32_t max_send_sge; + uint32_t max_recv_sge; + uint32_t max_inline_data; }; struct ibv_qp_init_attr { - void *qp_context; - struct ibv_cq *send_cq; - struct ibv_cq *recv_cq; - struct ibv_srq *srq; - struct ibv_qp_cap cap; - enum ibv_qp_type qp_type; - int sq_sig_all; + void *qp_context; + struct ibv_cq *send_cq; + struct ibv_cq *recv_cq; + struct ibv_srq *srq; + struct ibv_qp_cap cap; + enum ibv_qp_type qp_type; + int sq_sig_all; }; enum ibv_qp_attr_mask { - IBV_QP_STATE = 1 << 0, - IBV_QP_CUR_STATE = 1 << 1, - IBV_QP_EN_SQD_ASYNC_NOTIFY = 1 << 2, - IBV_QP_ACCESS_FLAGS = 1 << 3, - IBV_QP_PKEY_INDEX = 1 << 4, - IBV_QP_PORT = 1 << 5, - IBV_QP_QKEY = 1 << 6, - IBV_QP_AV = 1 << 7, - IBV_QP_PATH_MTU = 1 << 8, - IBV_QP_TIMEOUT = 1 << 9, - IBV_QP_RETRY_CNT = 1 << 10, - IBV_QP_RNR_RETRY = 1 << 11, - IBV_QP_RQ_PSN = 1 << 12, - IBV_QP_MAX_QP_RD_ATOMIC = 1 << 13, - IBV_QP_ALT_PATH = 1 << 14, - IBV_QP_MIN_RNR_TIMER = 1 << 15, - IBV_QP_SQ_PSN = 1 << 16, - IBV_QP_MAX_DEST_RD_ATOMIC = 1 << 17, - IBV_QP_PATH_MIG_STATE = 1 << 18, - IBV_QP_CAP = 1 << 19, - IBV_QP_DEST_QPN = 1 << 20 + IBV_QP_STATE = 1 << 0, + IBV_QP_CUR_STATE = 1 << 1, + IBV_QP_EN_SQD_ASYNC_NOTIFY = 1 << 2, + IBV_QP_ACCESS_FLAGS = 1 << 3, + IBV_QP_PKEY_INDEX = 1 << 4, + IBV_QP_PORT = 1 << 5, + IBV_QP_QKEY = 1 << 6, + IBV_QP_AV = 1 << 7, + IBV_QP_PATH_MTU = 1 << 8, + IBV_QP_TIMEOUT = 1 << 9, + IBV_QP_RETRY_CNT = 1 << 10, + IBV_QP_RNR_RETRY = 1 << 11, + IBV_QP_RQ_PSN = 1 << 12, + IBV_QP_MAX_QP_RD_ATOMIC = 1 << 13, + IBV_QP_ALT_PATH = 1 << 14, + IBV_QP_MIN_RNR_TIMER = 1 << 15, + IBV_QP_SQ_PSN = 1 << 16, + IBV_QP_MAX_DEST_RD_ATOMIC = 1 << 17, + IBV_QP_PATH_MIG_STATE = 1 << 18, + IBV_QP_CAP = 1 << 19, + IBV_QP_DEST_QPN = 1 << 20 }; enum ibv_qp_state { @@ -288,31 +288,31 @@ enum ibv_mig_state { }; struct ibv_qp_attr { - enum ibv_qp_state qp_state; - enum ibv_qp_state cur_qp_state; - enum ibv_mtu path_mtu; - enum ibv_mig_state path_mig_state; - uint32_t qkey; - uint32_t rq_psn; - uint32_t sq_psn; - uint32_t dest_qp_num; - int qp_access_flags; - struct ibv_qp_cap cap; - struct ibv_ah_attr ah_attr; - struct ibv_ah_attr alt_ah_attr; - uint16_t pkey_index; - uint16_t alt_pkey_index; - uint8_t en_sqd_async_notify; - uint8_t sq_draining; - uint8_t max_rd_atomic; - uint8_t max_dest_rd_atomic; - uint8_t min_rnr_timer; - uint8_t port_num; - uint8_t timeout; - uint8_t retry_cnt; - uint8_t rnr_retry; - uint8_t alt_port_num; - uint8_t alt_timeout; + enum ibv_qp_state qp_state; + enum ibv_qp_state cur_qp_state; + enum ibv_mtu path_mtu; + enum ibv_mig_state path_mig_state; + uint32_t qkey; + uint32_t rq_psn; + uint32_t sq_psn; + uint32_t dest_qp_num; + int qp_access_flags; + struct ibv_qp_cap cap; + struct ibv_ah_attr ah_attr; + struct ibv_ah_attr alt_ah_attr; + uint16_t pkey_index; + uint16_t alt_pkey_index; + uint8_t en_sqd_async_notify; + uint8_t sq_draining; + uint8_t max_rd_atomic; + uint8_t max_dest_rd_atomic; + uint8_t min_rnr_timer; + uint8_t port_num; + uint8_t timeout; + uint8_t retry_cnt; + uint8_t rnr_retry; + uint8_t alt_port_num; + uint8_t alt_timeout; }; enum ibv_wr_opcode { @@ -375,7 +375,7 @@ struct ibv_recv_wr { struct ibv_qp { struct ibv_context *context; void *qp_context; - struct ibv_pd *pd; + struct ibv_pd *pd; struct ibv_cq *send_cq; struct ibv_cq *recv_cq; uint32_t handle; @@ -442,7 +442,7 @@ struct ibv_context_ops { struct ibv_context { struct ibv_device *device; - struct ibv_context_ops ops; + struct ibv_context_ops ops; int cmd_fd; int async_fd; int num_comp; @@ -523,7 +523,6 @@ extern int ibv_destroy_cq(struct ibv_cq */ extern int ibv_get_cq_event(struct ibv_context *context, int comp_num, struct ibv_cq **cq, void **cq_context); - /** * ibv_poll_cq - Poll a CQ for work completions diff -X excl -rduNp gen2.2491.oorig/trunk/src/userspace/libmthca/src/cq.c gen2.2491/trunk/src/userspace/libmthca/src/cq.c --- gen2.2491.oorig/trunk/src/userspace/libmthca/src/cq.c 2005-04-26 20:58:01.000000000 +0200 +++ gen2.2491/trunk/src/userspace/libmthca/src/cq.c 2005-05-26 16:43:34.374856592 +0200 @@ -157,7 +157,8 @@ static inline void update_cons_index(str doorbell[0] = htonl(MTHCA_TAVOR_CQ_DB_INC_CI | cq->cqn); doorbell[1] = htonl(incr - 1); - mthca_write64(doorbell, to_mctx(cq->ibv_cq.context), MTHCA_CQ_DOORBELL); + mthca_write64(doorbell, to_mctx(cq->ibv_cq.context), + MTHCA_CQ_DOORBELL); } } @@ -302,7 +303,8 @@ static inline int mthca_poll_one(struct * because CQs will be locked while QPs are removed * from the table. */ - *cur_qp = mthca_find_qp(to_mctx(cq->ibv_cq.context), ntohl(cqe->my_qpn)); + *cur_qp = mthca_find_qp(to_mctx(cq->ibv_cq.context), + ntohl(cqe->my_qpn)); if (!*cur_qp) { err = CQ_POLL_ERR; goto out; @@ -313,7 +315,8 @@ static inline int mthca_poll_one(struct if (is_send) { wq = &(*cur_qp)->sq; - wqe_index = ((ntohl(cqe->wqe) - (*cur_qp)->send_wqe_offset) >> wq->wqe_shift); + wqe_index = ((ntohl(cqe->wqe) - + (*cur_qp)->send_wqe_offset) >> wq->wqe_shift); wc->wr_id = (*cur_qp)->wrid[wqe_index + (*cur_qp)->rq.max]; } else { wq = &(*cur_qp)->rq; diff -X excl -rduNp gen2.2491.oorig/trunk/src/userspace/libmthca/src/mthca.h gen2.2491/trunk/src/userspace/libmthca/src/mthca.h --- gen2.2491.oorig/trunk/src/userspace/libmthca/src/mthca.h 2005-05-26 11:47:53.961348000 +0200 +++ gen2.2491/trunk/src/userspace/libmthca/src/mthca.h 2005-05-26 16:35:22.191679808 +0200 @@ -104,13 +104,13 @@ struct mthca_context { struct mthca_db_table *db_tab; struct ibv_pd *pd; struct { - struct mthca_qp **table; - int refcnt; - } qp_table[MTHCA_QP_TABLE_SIZE]; + struct mthca_qp **table; + int refcnt; + } qp_table[MTHCA_QP_TABLE_SIZE]; pthread_mutex_t qp_table_mutex; int num_qps; - int qp_table_shift; - int qp_table_mask; + int qp_table_shift; + int qp_table_mask; /* * ibv_ctx must be last because we size it dynamically * depending on the number of CQ events available. @@ -126,12 +126,12 @@ struct mthca_pd { }; struct mthca_cq { - struct ibv_cq ibv_cq; - void *buf; - pthread_spinlock_t lock; - struct ibv_mr *mr; - uint32_t cqn; - uint32_t cons_index; + struct ibv_cq ibv_cq; + void *buf; + pthread_spinlock_t lock; + struct ibv_mr *mr; + uint32_t cqn; + uint32_t cons_index; /* Next fields are mem-free only */ int set_ci_db_index; @@ -143,14 +143,14 @@ struct mthca_cq { struct mthca_wq { pthread_spinlock_t lock; - int max; - unsigned next_ind; - unsigned last_comp; - unsigned head; - unsigned tail; - void *last; - int max_gs; - int wqe_shift; + int max; + unsigned next_ind; + unsigned last_comp; + unsigned head; + unsigned tail; + void *last; + int max_gs; + int wqe_shift; /* Next fields are mem-free only */ int db_index; @@ -247,25 +247,27 @@ static inline int mthca_is_memfree(struc return to_mdev(ibctx->device)->hca_type == MTHCA_ARBEL; } -extern int mthca_alloc_db(struct mthca_db_table *db_tab, enum mthca_db_type type, - uint32_t **db); +extern int mthca_alloc_db(struct mthca_db_table *db_tab, + enum mthca_db_type type, + uint32_t **db); extern void mthca_set_db_qn(uint32_t *db, enum mthca_db_type type, uint32_t qn); -extern void mthca_free_db(struct mthca_db_table *db_tab, enum mthca_db_type type, int db_index); +extern void mthca_free_db(struct mthca_db_table *db_tab, + enum mthca_db_type type, int db_index); extern struct mthca_db_table *mthca_alloc_db_tab(int uarc_size); extern void mthca_free_db_tab(struct mthca_db_table *db_tab); extern int mthca_query_port(struct ibv_context *context, uint8_t port, - struct ibv_port_attr *attr); + struct ibv_port_attr *attr); extern int mthca_query_gid(struct ibv_context *context, uint8_t port_num, - int index, union ibv_gid *gid); + int index, union ibv_gid *gid); extern int mthca_query_pkey(struct ibv_context *context, uint8_t port_num, - int index, uint16_t *pkey); + int index, uint16_t *pkey); extern struct ibv_pd *mthca_alloc_pd(struct ibv_context *context); extern int mthca_free_pd(struct ibv_pd *pd); extern struct ibv_mr *mthca_reg_mr(struct ibv_pd *pd, void *addr, - size_t length, enum ibv_access_flags access); + size_t length, enum ibv_access_flags access); extern int mthca_dereg_mr(struct ibv_mr *mr); struct ibv_cq *mthca_create_cq(struct ibv_context *context, int cqe); @@ -276,33 +278,38 @@ extern int mthca_arbel_arm_cq(struct ibv extern void mthca_arbel_cq_event(struct ibv_cq *cq); extern void mthca_init_cq_buf(struct mthca_cq *cq, int nent); -extern struct ibv_qp *mthca_create_qp(struct ibv_pd *pd, struct ibv_qp_init_attr *attr); +extern struct ibv_qp *mthca_create_qp(struct ibv_pd *pd, + struct ibv_qp_init_attr *attr); extern int mthca_modify_qp(struct ibv_qp *qp, struct ibv_qp_attr *attr, - enum ibv_qp_attr_mask attr_mask); + enum ibv_qp_attr_mask attr_mask); extern int mthca_destroy_qp(struct ibv_qp *qp); extern int mthca_tavor_post_send(struct ibv_qp *ibqp, struct ibv_send_wr *wr, - struct ibv_send_wr **bad_wr); + struct ibv_send_wr **bad_wr); extern int mthca_tavor_post_recv(struct ibv_qp *ibqp, struct ibv_recv_wr *wr, - struct ibv_recv_wr **bad_wr); + struct ibv_recv_wr **bad_wr); extern int mthca_arbel_post_send(struct ibv_qp *ibqp, struct ibv_send_wr *wr, - struct ibv_send_wr **bad_wr); + struct ibv_send_wr **bad_wr); extern int mthca_arbel_post_recv(struct ibv_qp *ibqp, struct ibv_recv_wr *wr, - struct ibv_recv_wr **bad_wr); + struct ibv_recv_wr **bad_wr); extern int mthca_alloc_qp_buf(struct ibv_pd *pd, struct ibv_qp_cap *cap, - struct mthca_qp *qp); + struct mthca_qp *qp); extern void mthca_return_cap(struct ibv_pd *pd, struct mthca_qp *qp, - struct ibv_qp_cap *cap); + struct ibv_qp_cap *cap); extern struct mthca_qp *mthca_find_qp(struct mthca_context *ctx, uint32_t qpn); -extern int mthca_store_qp(struct mthca_context *ctx, uint32_t qpn, struct mthca_qp *qp); +extern int mthca_store_qp(struct mthca_context *ctx, uint32_t qpn, + struct mthca_qp *qp); extern void mthca_clear_qp(struct mthca_context *ctx, uint32_t qpn); extern int mthca_free_err_wqe(struct mthca_qp *qp, int is_send, - int index, int *dbd, uint32_t *new_wqe); -extern struct ibv_ah *mthca_create_ah(struct ibv_pd *pd, struct ibv_ah_attr *attr); + int index, int *dbd, uint32_t *new_wqe); +extern struct ibv_ah *mthca_create_ah(struct ibv_pd *pd, + struct ibv_ah_attr *attr); extern int mthca_destroy_ah(struct ibv_ah *ah); extern int mthca_alloc_av(struct mthca_pd *pd, struct ibv_ah_attr *attr, - struct mthca_ah *ah); + struct mthca_ah *ah); extern void mthca_free_av(struct mthca_ah *ah); -extern int mthca_attach_mcast(struct ibv_qp *qp, union ibv_gid *gid, uint16_t lid); -extern int mthca_detach_mcast(struct ibv_qp *qp, union ibv_gid *gid, uint16_t lid); +extern int mthca_attach_mcast(struct ibv_qp *qp, union ibv_gid *gid, + uint16_t lid); +extern int mthca_detach_mcast(struct ibv_qp *qp, union ibv_gid *gid, + uint16_t lid); #endif /* MTHCA_H */ From hozer at hozed.org Thu May 26 09:21:31 2005 From: hozer at hozed.org (Troy Benjegerdes) Date: Thu, 26 May 2005 11:21:31 -0500 Subject: [openib-general] Re: umad abi 2 v 3 and multicast join failed In-Reply-To: <20050526072909.GT27428@mellanox.co.il> References: <20050525230658.GY999@kalmia.hozed.org> <1117079586.4479.432.camel@hal.voltaire.com> <52br6yocjm.fsf@topspin.com> <20050526045247.GL671@esmail.cup.hp.com> <20050526072909.GT27428@mellanox.co.il> Message-ID: <20050526162131.GA999@kalmia.hozed.org> On Thu, May 26, 2005 at 10:29:09AM +0300, Michael S. Tsirkin wrote: > Quoting r. Grant Grundler : > > Subject: Re: umad abi 2 v 3 and multicast join failed > > > > On Wed, May 25, 2005 at 09:07:25PM -0700, Roland Dreier wrote: > > > In general, given that kernels 2.6.11 and 2.6.12 are shipping with ABI > > > version 2, does it make sense to avoid problems like this by keeping > > > the old userspace code around and having the library decide at runtime > > > which ABI to use? > > > > Only if you want to continue providing fixes for 2.6.11 and 2.6.12. > > It would be nice if someone put together a source snapshot that > > would work with those releases. But I'm not sure how to > > make that available to some random person who is using it. > > Making a subversion branch could be one way to start. > > Actually, it would be quite useful if whatever is submitted to mainline > is kept on some subversion branch, too. > Roland - would that disrupt your development process a lot? What's the status on getting ABI ver 3 into 2.6.1X ? From robert.j.woodruff at intel.com Thu May 26 09:25:09 2005 From: robert.j.woodruff at intel.com (Bob Woodruff) Date: Thu, 26 May 2005 09:25:09 -0700 Subject: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: Message-ID: Venkat wrote, >Absolutely and this is exactly what I would like to see without impacting >the IB development but supporting common ULPs is critical. >Thanks >Venkat Great, the best way to influence the code is to send in patches to the list and then discuss them. James is currently coordinating all of the kDAPL work. Now is a good time to get involved since the Linuxization and morphing of kDAPL it into something acceptable to Linux is happening now. I would like to see the code for the RNIC hardware since without a hardware driver, mid-layer, kDAPL, or ULP support is irrelevant. woody From caitlin.bestler at gmail.com Thu May 26 09:25:27 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Thu, 26 May 2005 09:25:27 -0700 Subject: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: References: Message-ID: <469958e005052609253eba29da@mail.gmail.com> The current OpenIB structure is, not surprisingly, tremendously IB-centric. Any attempt to implement an iWARP RNIC through them will have to go through several layers of obfuscation. States do not match. Error codes do not match. Completion Statuses do not match. There is a lot of information that does not translate to iWARP, and there is information that is missing. RNIC-PI is at least an attempt at providing full control over both iWARP and IB while making as much common as possible. I believe it is possible to define a verb set that is largely compatible with current OpenIB IB verbs and have the same transport neutrality that RNIC-PI has. But this will not be just poking around and tweaking one or two elements in the OpenIB code. Is there a real willingness in OpenIB to make a transport neutral verb layer? Has anyone even *read* the RNIC-PI for substance rather than "pure Linux" debating points? From roland at topspin.com Thu May 26 09:27:35 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 26 May 2005 09:27:35 -0700 Subject: [openib-general] Re: umad abi 2 v 3 and multicast join failed In-Reply-To: <20050526072909.GT27428@mellanox.co.il> (Michael S. Tsirkin's message of "Thu, 26 May 2005 10:29:09 +0300") References: <20050525230658.GY999@kalmia.hozed.org> <1117079586.4479.432.camel@hal.voltaire.com> <52br6yocjm.fsf@topspin.com> <20050526045247.GL671@esmail.cup.hp.com> <20050526072909.GT27428@mellanox.co.il> Message-ID: <52oeaylzpk.fsf@topspin.com> Michael> Actually, it would be quite useful if whatever is Michael> submitted to mainline is kept on some subversion branch, Michael> too. Roland - would that disrupt your development Michael> process a lot? It's a bit of a pain to get started but perhaps it would be useful to have a branch where patches for merging can live. Unfortunately subversion is not really the best tool for that -- it would be better to have a tool that permits patches to be revised based on feedback when required. - R. From roland at topspin.com Thu May 26 09:28:21 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 26 May 2005 09:28:21 -0700 Subject: [openib-general] Re: umad abi 2 v 3 and multicast join failed In-Reply-To: <20050526162131.GA999@kalmia.hozed.org> (Troy Benjegerdes's message of "Thu, 26 May 2005 11:21:31 -0500") References: <20050525230658.GY999@kalmia.hozed.org> <1117079586.4479.432.camel@hal.voltaire.com> <52br6yocjm.fsf@topspin.com> <20050526045247.GL671@esmail.cup.hp.com> <20050526072909.GT27428@mellanox.co.il> <20050526162131.GA999@kalmia.hozed.org> Message-ID: <52k6lmlzoa.fsf@topspin.com> Troy> What's the status on getting ABI ver 3 into 2.6.1X ? Not submitted -- 2.6.12 is far too deep into the freeze phase to make a change like this. Once 2.6.13 opens up, we'll merge the pending MAD changes at least. - R. From roland at topspin.com Thu May 26 09:29:17 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 26 May 2005 09:29:17 -0700 Subject: [openib-general] [PATCH] mthca, ibverbs userspace readability In-Reply-To: <20050526161455.GL15688@aon.at> (Bernhard Fischer's message of "Thu, 26 May 2005 18:14:55 +0200") References: <20050526161455.GL15688@aon.at> Message-ID: <52ekbulzmq.fsf@topspin.com> Bernhard> - improve readability of mthca and ibverbs headers Bernhard> installed to userspace. This patch seems to consist mostly of replacing tabs with 8 spaces. Why is that an improvement for readability? Thanks, Roland From roland at topspin.com Thu May 26 09:30:53 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 26 May 2005 09:30:53 -0700 Subject: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <469958e005052609253eba29da@mail.gmail.com> (Caitlin Bestler's message of "Thu, 26 May 2005 09:25:27 -0700") References: <469958e005052609253eba29da@mail.gmail.com> Message-ID: <52acmilzk2.fsf@topspin.com> Caitlin> Is there a real willingness in OpenIB to make a transport Caitlin> neutral verb layer? Has anyone even *read* the RNIC-PI Caitlin> for substance rather than "pure Linux" debating points? There is definitely a willingness to consider thoroughly all changes proposed by the RNIC community (and of course anyone else as well). - R. From roland at topspin.com Thu May 26 09:32:31 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 26 May 2005 09:32:31 -0700 Subject: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: (Bob Woodruff's message of "Thu, 26 May 2005 08:49:59 -0700") References: Message-ID: <5264x6lzhc.fsf@topspin.com> Bob> There is already a RDMA device independent API being Bob> developed for the kernel by people on this list. It is Bob> starting with the kDAPL code base, which was designed to Bob> support both IBA and iWarp devices. I believe kDAPL-based layers are an OK short-term solution, but I don't think anything like this should be proposed for merging in the Linus kernel. If we need another abstraction layer on top of our existing abstraction layer, that just says to me that we should fix the current abstraction layer. - R. From Thomas.Talpey at netapp.com Thu May 26 09:29:23 2005 From: Thomas.Talpey at netapp.com (Talpey, Thomas) Date: Thu, 26 May 2005 12:29:23 -0400 Subject: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: References: Message-ID: <6.2.1.2.2.20050526120905.05485a80@exnane01.nane.netapp.com> At 11:49 AM 5/26/2005, Bob Woodruff wrote: >Finally, until there is some consensus about allowing TCP offload in Linux, >I see no need to start to hack up the InfiniBand stack to support iWarp. It is not a requirement that TCP offload be supported in order to support iWARP. Upper layers which don't perform an rdma-upgrade (e.g. NFSv3/RDMA) don't even care about "speaking sockets". And I believe a Connection Manager can and should address this for the ones that do. Tying iWARP to the TOE albatross furthers nothing. I completely agree with your other point that a lower-layer cut-in is preferable to another top-layer API. We already have APIs which provide transport-independent RDMA services to existing upper layers (such as NFS/RDMA and iSER, coded in fact to kDAPL). Do we have solid justification to add more? NFS/RDMA says "no". Tom. From blist at aon.at Thu May 26 09:42:03 2005 From: blist at aon.at (Bernhard Fischer) Date: Thu, 26 May 2005 18:42:03 +0200 Subject: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <469958e005052609253eba29da@mail.gmail.com> References: <469958e005052609253eba29da@mail.gmail.com> Message-ID: <20050526164203.GM15688@aon.at> On Thu, May 26, 2005 at 09:25:27AM -0700, Caitlin Bestler wrote: >The current OpenIB structure is, not surprisingly, tremendously >IB-centric. Any attempt to implement an iWARP RNIC through >them will have to go through several layers of obfuscation. > >States do not match. Error codes do not match. Completion >Statuses do not match. There is a lot of information that Statuses? stati >does not translate to iWARP, and there is information that >is missing. such as.. ? > >RNIC-PI is at least an attempt at providing full control over >both iWARP and IB while making as much common as >possible. Please point me to any infrastructure which allows any private peer to steek at RNIC. At least at peer-level with a working driver which includes a working ethtool -C infrastructure.. > >I believe it is possible to define a verb set that is largely >compatible with current OpenIB IB verbs and have the >same transport neutrality that RNIC-PI has. But this will >not be just poking around and tweaking one or two >elements in the OpenIB code. [What is RNIC anyway? dict rnic returns just NIL, so what? Don't mean to sound rude, but please provide at least a _little_ background for newcomers, will you?] > >Is there a real willingness in OpenIB to make a transport >neutral verb layer? Has anyone even *read* the RNIC-PI Not being involved (heared of \"community\" before?), still yes. Care to point (once again, as i'm buzzword agnostic) to the relevant IEEE/whatever RFCs? TIA. >for substance rather than "pure Linux" debating points? >_______________________________________________ >openib-general mailing list >openib-general at openib.org >http://openib.org/mailman/listinfo/openib-general > >To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general > From robert.j.woodruff at intel.com Thu May 26 09:45:10 2005 From: robert.j.woodruff at intel.com (Woodruff, Robert J) Date: Thu, 26 May 2005 09:45:10 -0700 Subject: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux Message-ID: <1AC79F16F5C5284499BB9591B33D6F0004831164@orsmsx408> Roland wrote, >I believe kDAPL-based layers are an OK short-term solution, but I >don't think anything like this should be proposed for merging in the >Linus kernel. If we need another abstraction layer on top of our >existing abstraction layer, that just says to me that we should fix >the current abstraction layer. > - R. The InfiniBand core mid-layer is infiniband specific. I don't think it was intended to be an RDMA device independent layer. kDAPL was intended to be that layer. Seems that at some level you need to have a RDMA device class specific layer. The IB core layer, CM, MAD services, are really pretty IB specific and don't apply to other RDMA types of devices. The core verbs are similar, but the other IB mid-layer components (SA query, mad service, CM) are very IB specific. woody From blist at aon.at Thu May 26 09:47:07 2005 From: blist at aon.at (Bernhard Fischer) Date: Thu, 26 May 2005 18:47:07 +0200 Subject: [openib-general] [PATCH] mthca, ibverbs userspace readability In-Reply-To: <52ekbulzmq.fsf@topspin.com> References: <20050526161455.GL15688@aon.at> <52ekbulzmq.fsf@topspin.com> Message-ID: <20050526164707.GN15688@aon.at> On Thu, May 26, 2005 at 09:29:17AM -0700, Roland Dreier wrote: > Bernhard> - improve readability of mthca and ibverbs headers > Bernhard> installed to userspace. > >This patch seems to consist mostly of replacing tabs with 8 spaces. >Why is that an improvement for readability? care to read one of those headers with ts=2 and code for userspace? without that patch, it simply is unreadable. Still, as noted before, just take it or leave it (fear you'll decide to go for the former :/ but i will continue to propagate the latter, fwiw.). > >Thanks, > Roland > From mshefty at ichips.intel.com Thu May 26 09:49:44 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Thu, 26 May 2005 09:49:44 -0700 Subject: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux In-Reply-To: <52is16ogs7.fsf@topspin.com> References: <52is16ogs7.fsf@topspin.com> Message-ID: <4295FE28.2000506@ichips.intel.com> Roland Dreier wrote: > I believe the way forward is to evolve the existing drivers/infiniband > code already in Linux into a drivers/rdma that supports both IB and > RNICs. To be extremely blunt, I believe the RNIC-PI is irrelevant to > the Linux kernel -- no IB vendors will support ripping out a working > midlayer and starting from scratch, and it doesn't make sense to have > two essentially equivalent midlayers in the same kernel. IMO, any APIs need to evolve out of the implementation. Trying to fit an implementation under an existing API tends to lead to either a poor implementation, or requires changes to the API anyway. I think a useful path is for someone to implement an RNIC driver and provide feedback on what changes would be required of the Infiniband/core layer to support it. > To put a really concrete proposal on the table, I would suggest to > start by extending the current ib_client registration structure Roland's proposal sounds like a reasonable and fair way to begin. Building an abstraction on top of the existing layers seems secondary to adding support for other RDMA devices. - Sean From blist at aon.at Thu May 26 09:58:19 2005 From: blist at aon.at (Bernhard Fischer) Date: Thu, 26 May 2005 18:58:19 +0200 Subject: [openib-general] [PATCH] mthca, ibverbs userspace readability In-Reply-To: <52ekbulzmq.fsf@topspin.com> References: <20050526161455.GL15688@aon.at> <52ekbulzmq.fsf@topspin.com> Message-ID: <20050526165819.GO15688@aon.at> On Thu, May 26, 2005 at 09:29:17AM -0700, Roland Dreier wrote: > Bernhard> - improve readability of mthca and ibverbs headers > Bernhard> installed to userspace. > >This patch seems to consist mostly of replacing tabs with 8 spaces. >Why is that an improvement for readability? It does indeed. But with the patch, it's easier to read and code for. Also, there are a couple of fixes for super-long lines, which are extremely disturbing if you happen to code userspace while referencing those headers :-/ From mshefty at ichips.intel.com Thu May 26 10:00:09 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Thu, 26 May 2005 10:00:09 -0700 Subject: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <5264x6lzhc.fsf@topspin.com> References: <5264x6lzhc.fsf@topspin.com> Message-ID: <42960099.3000503@ichips.intel.com> Roland Dreier wrote: > Bob> There is already a RDMA device independent API being > Bob> developed for the kernel by people on this list. It is > Bob> starting with the kDAPL code base, which was designed to > Bob> support both IBA and iWarp devices. > > I believe kDAPL-based layers are an OK short-term solution, but I > don't think anything like this should be proposed for merging in the > Linus kernel. If we need another abstraction layer on top of our > existing abstraction layer, that just says to me that we should fix > the current abstraction layer. My personal view is that any higher level abstractions could combine common functionality or simplify the interface, possibly with a loss of lower level control. For example, an interface could be created that allowed the user to operate with more abstract addressing, hiding SA query and CM interactions. Such an interface may not be needed yet, but if there were multiple RDMA transports, it could be valuable. However, I do not think this necessarily translates into the need to abstract concepts like PDs, QPs, and CQs. - Sean From roland at topspin.com Thu May 26 10:05:34 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 26 May 2005 10:05:34 -0700 Subject: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <1AC79F16F5C5284499BB9591B33D6F0004831164@orsmsx408> (Robert J. Woodruff's message of "Thu, 26 May 2005 09:45:10 -0700") References: <1AC79F16F5C5284499BB9591B33D6F0004831164@orsmsx408> Message-ID: <52mzqhlxy9.fsf@topspin.com> Robert> The InfiniBand core mid-layer is infiniband specific. I Robert> don't think it was intended to be an RDMA device Robert> independent layer. kDAPL was intended to be that layer. You're absolutely right that the IB core didn't start out that way. However, it seems better in the long run to fix it rather than just piling another layer on top to hide problems. - R. From roland at topspin.com Thu May 26 10:06:55 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 26 May 2005 10:06:55 -0700 Subject: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <42960099.3000503@ichips.intel.com> (Sean Hefty's message of "Thu, 26 May 2005 10:00:09 -0700") References: <5264x6lzhc.fsf@topspin.com> <42960099.3000503@ichips.intel.com> Message-ID: <52is15lxw0.fsf@topspin.com> Sean> My personal view is that any higher level abstractions could Sean> combine common functionality or simplify the interface, Sean> possibly with a loss of lower level control. For example, Sean> an interface could be created that allowed the user to Sean> operate with more abstract addressing, hiding SA query and Sean> CM interactions. Such an interface may not be needed yet, Sean> but if there were multiple RDMA transports, it could be Sean> valuable. However, I do not think this necessarily Sean> translates into the need to abstract concepts like PDs, QPs, Sean> and CQs. Yes, that is probably a better way of saying what I was trying to drive at. There's a significant portion of kDAPL that doesn't seem necessary or useful to me. For example, why do we need a new API for posting work requests? Why do we need the ability to combine completion events and CM events into a single queue? - R. From hch at lst.de Thu May 26 10:11:14 2005 From: hch at lst.de (Christoph Hellwig) Date: Thu, 26 May 2005 19:11:14 +0200 Subject: [openib-general] [PATCH] mthca, ibverbs userspace readability In-Reply-To: <20050526164707.GN15688@aon.at> References: <20050526161455.GL15688@aon.at> <52ekbulzmq.fsf@topspin.com> <20050526164707.GN15688@aon.at> Message-ID: <20050526171114.GA4055@lst.de> On Thu, May 26, 2005 at 06:47:07PM +0200, Bernhard Fischer wrote: > On Thu, May 26, 2005 at 09:29:17AM -0700, Roland Dreier wrote: > > Bernhard> - improve readability of mthca and ibverbs headers > > Bernhard> installed to userspace. > > > >This patch seems to consist mostly of replacing tabs with 8 spaces. > >Why is that an improvement for readability? > care to read one of those headers with ts=2 and code for userspace? then don't set a silly tabstop. linux code has tabstop=8 and that's pretty much set in stone. From hch at lst.de Thu May 26 10:11:48 2005 From: hch at lst.de (Christoph Hellwig) Date: Thu, 26 May 2005 19:11:48 +0200 Subject: [openib-general] [PATCH] mthca, ibverbs userspace readability In-Reply-To: <20050526165819.GO15688@aon.at> References: <20050526161455.GL15688@aon.at> <52ekbulzmq.fsf@topspin.com> <20050526165819.GO15688@aon.at> Message-ID: <20050526171148.GB4055@lst.de> On Thu, May 26, 2005 at 06:58:19PM +0200, Bernhard Fischer wrote: > On Thu, May 26, 2005 at 09:29:17AM -0700, Roland Dreier wrote: > > Bernhard> - improve readability of mthca and ibverbs headers > > Bernhard> installed to userspace. > > > >This patch seems to consist mostly of replacing tabs with 8 spaces. > >Why is that an improvement for readability? > > It does indeed. But with the patch, it's easier to read and code for. no, it's not. non-tab based indentation is a total pain in the ass. From robert.j.woodruff at intel.com Thu May 26 10:42:31 2005 From: robert.j.woodruff at intel.com (Woodruff, Robert J) Date: Thu, 26 May 2005 10:42:31 -0700 Subject: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux Message-ID: <1AC79F16F5C5284499BB9591B33D6F00048312AE@orsmsx408> >You're absolutely right that the IB core didn't start out that way. >However, it seems better in the long run to fix it rather than just >piling another layer on top to hide problems. >- R. I think that perhaps Sean's suggestion of having some RNIC vendor implement a verbs driver for their H/W and send in patches for the driver and the changes needed to plug into the core verbs mid-layer might be a way to move forward. Probably, enum rdma_device_type { RDMA_DEVICE_IB_CA, RDMA_DEVICE_IB_SWITCH, RDMA_DEVICE_IB_ROUTER, RDMA_DEVICE_RNIC }; is one of the first changes needed. woody From krause at cup.hp.com Thu May 26 10:30:59 2005 From: krause at cup.hp.com (Michael Krause) Date: Thu, 26 May 2005 10:30:59 -0700 Subject: [Rdma-developers] RE: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <6.2.1.2.2.20050526120905.05485a80@exnane01.nane.netapp.com > References: <6.2.1.2.2.20050526120905.05485a80@exnane01.nane.netapp.com> Message-ID: <6.2.0.14.2.20050526101615.02af8b48@esmail.cup.hp.com> At 09:29 AM 5/26/2005, Talpey, Thomas wrote: >At 11:49 AM 5/26/2005, Bob Woodruff wrote: > >Finally, until there is some consensus about allowing TCP offload in Linux, > >I see no need to start to hack up the InfiniBand stack to support iWarp. > >It is not a requirement that TCP offload be supported in order to >support iWARP. Upper layers which don't perform an rdma-upgrade >(e.g. NFSv3/RDMA) don't even care about "speaking sockets". TOE != Sockets. TOE is simply the TCP/IP protocol stack that is off-loaded for any TCP-based application / sub-system to use. >And I believe a Connection Manager can and should address this for the >ones that do. Tying iWARP to the TOE albatross furthers nothing. iWARP != TOE though one uses an integrated TOE in a RNIC if you want performance. Connection management is unique to IB as it is to TCP. It must exist as a separate interface for subsystems to use. Whether one can abstract a common API or just accept that IB and TCP are different and call accordingly once understanding the underlying hardware is debatable. >I completely agree with your other point that a lower-layer cut-in is >preferable to another top-layer API. We already have APIs which provide >transport-independent RDMA services to existing upper layers (such as >NFS/RDMA and iSER, coded in fact to kDAPL). kDAPL is being used by some sub-systems. One can argue that a verbs interface is a cleaner way to get directly to the hardware without requiring an intermediary API like DAPL (user or kernel). Caitlin will provide some good counters why this isn't necessarily perfect. Both sides have valid points in the debate. In the end, it is what people implement that matter and whether that gets accepted by customers in the end as good enough technology. Just because there are some point-in-time implementations for NFS/RDMA and iSER does not mean that kDAPL is the right or only answer. They are only what has been implemented to date. If someone builds a better implementation using another interface, then that could easily replace what is there today. The RNIC PI or IB verbs interfaces articulate the intent of those of us who created the verbs in the first place. We intentionally left it open as to whether further abstraction was required such as IT API or DAPL. We did intend for subsystems like MPI, SDP, iSER, etc. to be able to use the verbs directly if desired. I think many of us viewed IT API and DAPL as a potential point of abstraction to insert other value-add functionality that should be transparent to the applications but should not pollute the verbs, e.g. virtualization or specific types of load balancing or fail-over. >Do we have solid justification to add more? NFS/RDMA says "no". I don't believe any one person speaks for a given technology especially in an open source world. In fact, I suspect you'll find a large number of people who will disagree with this assertion as well. :) * Feel free to ignore the following * I also find it unprofessional some of the diatribes that occur at times on these reflectors. It would be more constructive if people focused on technical issues instead just painting broad negative statements. We all have opinions on how things should be developed. Keeping things objective is much more constructive. I always try to keep my responses focused on whether I would appreciate such a response or whether the response tone / language would be acceptable at my place of work. Colorful language, unnecessary inflammatory statements, etc. result in people loosing credibility and thus effectiveness in any environment. The worse thing for any engineer / professional is to be simply written off as a hot-head who is no longer being heard by any. Regards, Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From krause at cup.hp.com Thu May 26 10:34:38 2005 From: krause at cup.hp.com (Michael Krause) Date: Thu, 26 May 2005 10:34:38 -0700 Subject: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <20050526164203.GM15688@aon.at> References: <469958e005052609253eba29da@mail.gmail.com> <20050526164203.GM15688@aon.at> Message-ID: <6.2.0.14.2.20050526103237.02af88b8@esmail.cup.hp.com> At 09:42 AM 5/26/2005, Bernhard Fischer wrote: >On Thu, May 26, 2005 at 09:25:27AM -0700, Caitlin Bestler wrote: > >The current OpenIB structure is, not surprisingly, tremendously > >IB-centric. Any attempt to implement an iWARP RNIC through > >them will have to go through several layers of obfuscation. > > > >States do not match. Error codes do not match. Completion > >Statuses do not match. There is a lot of information that >Statuses? stati > >does not translate to iWARP, and there is information that > >is missing. >such as.. ? > > > >RNIC-PI is at least an attempt at providing full control over > >both iWARP and IB while making as much common as > >possible. >Please point me to any infrastructure which allows any private peer to >steek at RNIC. At least at peer-level with a working driver which >includes a working ethtool -C infrastructure.. > > > >I believe it is possible to define a verb set that is largely > >compatible with current OpenIB IB verbs and have the > >same transport neutrality that RNIC-PI has. But this will > >not be just poking around and tweaking one or two > >elements in the OpenIB code. >[What is RNIC anyway? dict rnic returns just NIL, so what? Don't mean to >sound rude, but please provide at least a _little_ background for >newcomers, will you?] A RNIC is a defined term in the RDMA Consortium specifications which were the basis for the IETF wire protocols. The specification and associated presentations explaining all of this is reasonable detail are available at the rdmaconsortium web site. There is also a good architectural block diagram at the OpenRDMA website that goes through the software architecture. Mike > > > >Is there a real willingness in OpenIB to make a transport > >neutral verb layer? Has anyone even *read* the RNIC-PI >Not being involved (heared of \"community\" before?), still yes. > >Care to point (once again, as i'm buzzword agnostic) to the >relevant IEEE/whatever RFCs? TIA. > >for substance rather than "pure Linux" debating points? -------------- next part -------------- An HTML attachment was scrubbed... URL: From krause at cup.hp.com Thu May 26 10:48:01 2005 From: krause at cup.hp.com (Michael Krause) Date: Thu, 26 May 2005 10:48:01 -0700 Subject: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux In-Reply-To: <4295FE28.2000506@ichips.intel.com> References: <52is16ogs7.fsf@topspin.com> <4295FE28.2000506@ichips.intel.com> Message-ID: <6.2.0.14.2.20050526103456.02af8dd8@esmail.cup.hp.com> At 09:49 AM 5/26/2005, Sean Hefty wrote: >Roland Dreier wrote: >>I believe the way forward is to evolve the existing drivers/infiniband >>code already in Linux into a drivers/rdma that supports both IB and >>RNICs. To be extremely blunt, I believe the RNIC-PI is irrelevant to >>the Linux kernel -- no IB vendors will support ripping out a working >>midlayer and starting from scratch, and it doesn't make sense to have >>two essentially equivalent midlayers in the same kernel. > >IMO, any APIs need to evolve out of the implementation. Trying to fit an >implementation under an existing API tends to lead to either a poor >implementation, or requires changes to the API anyway. > >I think a useful path is for someone to implement an RNIC driver and >provide feedback on what changes would be required of the Infiniband/core >layer to support it. There needs to be a balance between establishing a good, flexible architecture and how applications or subsystems interact with it. All of this needs to be grounded in implementation experience so there is a healthy does of reality. The problem with the iterative-TTM focused API definitions is they rarely produce something that is focused on the "end game" for a solid interface. They create instant legacy baggage that people are unwilling to shed as time goes forward. This legacy inertia is what has stifled quite a bit of innovation when it comes to software (some hazard that 90% of the software created today is focused primarily on legacy investment protection than really new innovation and when you think about it for a bit of time, you can see that much of this is re-inventing the wheel or to band-aid over a problem and packaging it as something innovative). So, the problem becomes one of finding balance. The people proposing the various API are people who have implemented various types of solutions so they are coming with both real experience as well as engineering judgement of what is required to get the right infrastructure in place for the "end game" needs. From a practical perspective, one needs to implement these API and see what is really of value and what should be changed. The key is to avoid creating the legacy inertia that ends up reducing fragmenting the interfaces and causes lost productivity, quality problems, etc. So before people write off the various API as unnecessary or as poor quality, there should be some implementations developed and some constructive debate of what features are really of value and what might be deprecated or eliminated. There is no requirement to ever implement all of an API as this is where an iterative approach has value. Implement what is needed to demonstrate the desired value and decide then if it is worthwhile. Just please don't discount it or toss it all out because it isn't all implemented today or you don't like some aspect. >>To put a really concrete proposal on the table, I would suggest to >>start by extending the current ib_client registration structure > >Roland's proposal sounds like a reasonable and fair way to >begin. Building an abstraction on top of the existing layers seems >secondary to adding support for other RDMA devices. Getting the right verbs interface is paramount. Whether one does IT API or DAPL on top of that for a given subsystem is secondary. Ideally, OpenIB and OpenRDMA should be focused on developing the RDMA infrastructure - RDMA verbs, connection management, IB-specific ULP, etc. They should not be focused on the upper subsystems or ULP. Those should be done as separate open source projects and they can decide where best to intersect with the Open* infrastructure. This one-size-fits-all approach for all subsystems to flow through a given interface is simply impractical to execute. It might happen over time but it cannot be force-fit. Let's focus the two efforts on getting as common of an infrastructure as possible. Some propose RNIC PI; other can propose something else if there is desire. RNIC PI has value in that it has focused on the common components between IB and iWARP and left the IB-specific and CM outside of its definition to allow an OS to decide how best to support or to allow IB to do what is needed for its particular usages. Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From halr at voltaire.com Thu May 26 10:44:53 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 26 May 2005 13:44:53 -0400 Subject: [openib-general] umad abi 2 v 3 and multicast join failed In-Reply-To: <20050525230658.GY999@kalmia.hozed.org> References: <20050525230658.GY999@kalmia.hozed.org> Message-ID: <1117129351.4479.204.camel@hal.voltaire.com> On Wed, 2005-05-25 at 19:06, Troy Benjegerdes wrote: > I was running a crufty version of opensm (compiled from the > roland-uverbs branch), and I started getting these kinds of errors for > no apparent reason: > > ib0: multicast join failed for ff12:401b:ffff:0:0:0:ffff:ffff, status > -22 > ib0: multicast join failed for ff12:401b:ffff:0:0:0:ffff:ffff, status > -22 > > I'm running 2.6.11 kernels, and 'stock' modules.. I just tried > rebuilding opensm from the latest SVN, but it apparently needs a new > umad driver.. > > warn: [24878] umad_init: wrong ABI version: > /sys/class/infiniband_mad/abi_version is 2 but library ABI is 3 > > > I suppose I need to rebuild the kernel ib_umad (and maybe everything > else for good measure).. And if I do that, should I expect OpenSM to > work better regarding the multicast issue? Yes, one quick workaround would be to pull the latest user_mad.c and user_mad.h files from OpenIB svn and rebuild the umad module. There might be some danger in this if RMPP is used as what has been pushed upstream does not include the changes for RMPP (MAD layer). Is this worth a try ? How quickly do you need a solution ? > Also, what will happen if I run opensm on two different nodes? Will they > fight, or will one of them figure out how to be a backup slave SM if the > first goes down? Is this relative to the umad ABI version ? That is just a local issue. There is no issue with OpenSMs on different nodes using different umad ABI versions as any communication between them is via standard MADs. -- Hal From mst at mellanox.co.il Thu May 26 11:31:14 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Thu, 26 May 2005 21:31:14 +0300 Subject: [openib-general] Re: umad abi 2 v 3 and multicast join failed In-Reply-To: <52oeaylzpk.fsf@topspin.com> References: <20050525230658.GY999@kalmia.hozed.org> <1117079586.4479.432.camel@hal.voltaire.com> <52br6yocjm.fsf@topspin.com> <20050526045247.GL671@esmail.cup.hp.com> <20050526072909.GT27428@mellanox.co.il> <52oeaylzpk.fsf@topspin.com> Message-ID: <20050526183114.GC17405@mellanox.co.il> Quoting r. Roland Dreier : > Subject: Re: umad abi 2 v 3 and multicast join failed > > Michael> Actually, it would be quite useful if whatever is > Michael> submitted to mainline is kept on some subversion branch, > Michael> too. Roland - would that disrupt your development > Michael> process a lot? > > It's a bit of a pain to get started but perhaps it would be useful to > have a branch where patches for merging can live. Unfortunately > subversion is not really the best tool for that -- it would be better > to have a tool that permits patches to be revised based on feedback > when required. > > - R. > No, I was proposing something less ambitious: basically just using svn branch as additional means of publishing patches. Thus each time you re-submit, you can just through a branch away and build a new one. -- MST - Michael S. Tsirkin From tduffy at sun.com Thu May 26 11:47:33 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 26 May 2005 11:47:33 -0700 Subject: [openib-general] [PATCH 1/3] kDAPL: consolidate dapl_srq*.c into one file Message-ID: <1117133253.17831.12.camel@duffman> Signed-off-by: Tom Duffy Index: linux-kernel-srq/dat-provider/dapl_srq.c =================================================================== --- linux-kernel-srq/dat-provider/dapl_srq.c (revision 0) +++ linux-kernel-srq/dat-provider/dapl_srq.c (revision 0) @@ -0,0 +1,500 @@ +/* + * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * + * This Software is licensed under one of the following licenses: + * + * 1) under the terms of the "Common Public License 1.0" a copy of which is + * available from the Open Source Initiative, see + * http://www.opensource.org/licenses/cpl.php. + * + * 2) under the terms of the "The BSD License" a copy of which is + * available from the Open Source Initiative, see + * http://www.opensource.org/licenses/bsd-license.php. + * + * 3) under the terms of the "GNU General Public License (GPL) Version 2" a + * copy of which is available from the Open Source Initiative, see + * http://www.opensource.org/licenses/gpl-license.php. + * + * Licensee has the right to choose one of the above licenses. + * + * Redistributions of source code must retain the above copyright + * notice and one of the license notices. + * + * Redistributions in binary form must reproduce both the above copyright + * notice, one of the license notices in the documentation + * and/or other materials provided with the distribution. + */ + +/* + * $Id$ + */ + +#include "dapl.h" +#include "dapl_ia_util.h" +#include "dapl_openib_util.h" +#include "dapl_cookie.h" + +/* + * dapl_srq_dealloc + * + * Free the passed in SRQ structure. + * + * Input: + * SRQ pointer + * + * Output: + * none + * + * Returns: + * none + * + */ +static void dapl_srq_dealloc(struct dapl_srq *srq_ptr) +{ + dapl_os_assert(srq_ptr->header.magic == DAPL_MAGIC_SRQ); + + /* reset magic to prevent reuse */ + srq_ptr->header.magic = DAPL_MAGIC_INVALID; + dapl_ia_unlink_srq(srq_ptr->header.owner_ia, srq_ptr); + dapl_cb_free(&srq_ptr->recv_buffer); + /* no need to destroy srq_ptr->header.lock */ + + kfree(srq_ptr); +} + +/* + * dapl_srq_alloc + * + * alloc and initialize an SRQ struct + * + * Input: + * IA INFO struct ptr + * SRQ ATTR ptr + * + * Output: + * none + * + * Returns: + * pointer to srq + * + */ +static struct dapl_srq *dapl_srq_alloc(struct dapl_ia *ia_ptr, + const struct dat_srq_attr *srq_attr) +{ + struct dapl_srq *srq_ptr; + + /* Allocate SRQ */ + srq_ptr = kmalloc(sizeof *srq_ptr, GFP_ATOMIC); + if (!srq_ptr) + goto bail; + + /* zero the structure */ + memset(srq_ptr, 0, sizeof *srq_ptr); + + /* + * initialize the header + */ + srq_ptr->header.provider = ia_ptr->header.provider; + srq_ptr->header.magic = DAPL_MAGIC_SRQ; + srq_ptr->header.handle_type = DAT_HANDLE_TYPE_SRQ; + srq_ptr->header.owner_ia = ia_ptr; + srq_ptr->header.user_context.as_64 = 0; + srq_ptr->header.user_context.as_ptr = NULL; + atomic_set(&srq_ptr->srq_ref_count, 0); + + dapl_llist_init_entry(&srq_ptr->header.ia_list_entry); + spin_lock_init(&srq_ptr->header.lock); + + /* + * Initialize the body. + * XXX Assume srq_attrs is required + */ + srq_ptr->param.max_recv_dtos = srq_attr->max_recv_dtos; + srq_ptr->param.max_recv_iov = srq_attr->max_recv_iov; + srq_ptr->param.low_watermark = srq_attr->low_watermark; + + /* Get a cookie buffer to track outstanding recvs */ + if (DAT_SUCCESS != dapl_cb_create(&srq_ptr->recv_buffer, + (struct dapl_ep *)srq_ptr, + srq_ptr->param.max_recv_dtos)) { + dapl_srq_dealloc(srq_ptr); + srq_ptr = NULL; + goto bail; + } + +bail: + return srq_ptr; +} + +/* + * dapl_srq_create + * + * Create an instance of a Shared Receive Queue that is provided to the + * consumer at srq_handle. + * + * Input: + * ia_handle + * pz_handle + * srq_attr + * + * Output: + * srq_handle + * + * Returns: + * DAT_SUCCESS + * DAT_INSUFFICIENT_RESOURCES + * DAT_INVALID_HANDLE + * DAT_INVALID_PARAMETER + * ?DAT_INVALID_ATTRIBUTE?? + * DAT_MODEL_NOT_SUPPORTED + */ +u32 dapl_srq_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, + struct dat_srq_attr *srq_attr, DAT_SRQ_HANDLE *srq_handle) +{ + struct dapl_ia *ia_ptr; + struct dapl_srq *srq_ptr; + u32 status = DAT_SUCCESS; + + dapl_dbg_log(DAPL_DBG_TYPE_API, + "dapl_srq_create (%p, %p, %p, %p)\n", + ia_handle, pz_handle, srq_attr, srq_handle); + + ia_ptr = (struct dapl_ia *)ia_handle; + + /* + * Verify parameters + */ + if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); + goto bail; + } + + /* + * Verify non-required parameters. + * N.B. Assumption: any parameter that can be + * modified by dat_ep_modify() is not strictly + * required when the EP is created + */ + if (pz_handle != NULL && DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); + goto bail; + } + + if (srq_handle == NULL) { + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); + goto bail; + } + if ((unsigned long)srq_attr & 3) { + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); + goto bail; + } + + /* Allocate SRQ */ + srq_ptr = dapl_srq_alloc(ia_ptr, srq_attr); + if (srq_ptr == NULL) { + status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, + DAT_RESOURCE_MEMORY); + goto bail; + } + + srq_ptr->param.ia_handle = (DAT_IA_HANDLE) ia_ptr; + srq_ptr->param.srq_state = DAT_SRQ_STATE_OPERATIONAL; + srq_ptr->param.pz_handle = pz_handle; + + /* + * XXX Allocate provider resource here!!! + */ + /* XXX */ status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); + /* XXX */ dapl_srq_dealloc(srq_ptr); + /* XXX */ goto bail; + + /* Link it onto the IA */ + dapl_ia_link_srq(ia_ptr, srq_ptr); + + *srq_handle = srq_ptr; + +bail: + return status; +} + +/* + * dapl_srq_free + * + * Destroy an instance of an SRQ + * + * Input: + * srq_handle + * + * Output: + * none + * + * Returns: + * DAT_SUCCESS + * DAT_INVALID_PARAMETER + * DAT_INVALID_STATE + */ +u32 dapl_srq_free(DAT_SRQ_HANDLE srq_handle) +{ + struct dapl_srq *srq_ptr; + struct dapl_ia *ia_ptr; + struct dat_srq_param *param; + u32 status = DAT_SUCCESS; + + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_free (%p)\n", srq_handle); + + srq_ptr = (struct dapl_srq *)srq_handle; + param = &srq_ptr->param; + + /* + * Verify parameter & state + */ + if (DAPL_BAD_HANDLE(srq_ptr, DAPL_MAGIC_SRQ)) { + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); + goto bail; + } + + if (atomic_read(&srq_ptr->srq_ref_count) != 0) { + /* + * The DAPL 1.2 spec says to return DAT_SRQ_IN_USE, which does + * not exist. Have filed the following as an eratta. + */ + status = DAT_ERROR(DAT_INVALID_STATE, + DAT_INVALID_STATE_SRQ_IN_USE); + goto bail; + } + + ia_ptr = srq_ptr->header.owner_ia; + + /* + * Do verification of parameters and the state change atomically. + */ + spin_lock_irqsave(&srq_ptr->header.lock, srq_ptr->header.flags); + + /* Remove the SRQ from the IA */ + dapl_ia_unlink_srq(ia_ptr, srq_ptr); + + spin_unlock_irqrestore(&srq_ptr->header.lock, srq_ptr->header.flags); + + /* + * Finish tearing everything down. + */ + + /* + * Take care of the transport resource + */ + + /* XXX Put provider code here!!! */ + + /* Free the resource */ + dapl_srq_dealloc(srq_ptr); + +bail: + return status; + +} + +/* + * dapl_srq_post_recv + * + * Post a receive buffer that can be used by any incoming + * message by any connected EP using the SRQ. Request to receive data + * over a connection of any ep handle into local_iov + * + * Input: + * srq_handle + * num_segments + * local_iov + * user_cookie + * + * Output: + * None. + * + * Returns: + * DAT_SUCCESS + * DAT_INSUFFICIENT_RESOURCES + * DAT_INVALID_PARAMETER + * DAT_INVALID_HANDLE + * DAT_INVALID_STATE + * DAT_PROTECTION_VIOLATION + * DAT_PROVILEGES_VIOLATION + */ +u32 dapl_srq_post_recv(DAT_SRQ_HANDLE srq_handle, int num_segments, + struct dat_lmr_triplet *local_iov, + DAT_DTO_COOKIE user_cookie) +{ + struct dapl_srq *srq_ptr; + struct dapl_cookie *cookie; + u32 status; + + dapl_dbg_log(DAPL_DBG_TYPE_API, + "dapl_srq_post_recv (%p, %d, %p, %P)\n", + srq_handle, num_segments, local_iov, user_cookie.as_64); + + if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); + goto bail; + } + + srq_ptr = (struct dapl_srq *)srq_handle; + + /* + * Synchronization ok since this buffer is only used for receive + * requests, which aren't allowed to race with each other. The + * app must syncronize access to the SRQ. + */ + status = dapl_dto_cookie_alloc(&srq_ptr->recv_buffer, + DAPL_DTO_TYPE_RECV, + user_cookie, &cookie); + if (DAT_SUCCESS != status) + goto bail; + + /* + * Take reference before posting to avoid race conditions with + * completions + */ + atomic_inc(&srq_ptr->recv_count); + + /* + * Invoke provider specific routine to post DTO + */ + /* XXX Put code here XXX */ + /* XXX */ status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); + + if (status != DAT_SUCCESS) { + atomic_dec(&srq_ptr->recv_count); + dapl_cookie_dealloc(&srq_ptr->recv_buffer, cookie); + } + +bail: + return status; +} + +u32 dapl_srq_query(DAT_SRQ_HANDLE srq_handle, struct dat_srq_param *srq_param) +{ + struct dapl_srq *srq_ptr; + u32 status = DAT_SUCCESS; + + dapl_dbg_log(DAPL_DBG_TYPE_API, + "dapl_srq_query (%p, %x, %p)\n", + srq_handle, srq_param); + + if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); + goto bail; + } + if (srq_param == NULL) { + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); + goto bail; + } + + srq_ptr = (struct dapl_srq *)srq_handle; + + /* + * XXX Need to calculate available_dto_count and outstanding_dto_count + */ + srq_ptr->param.available_dto_count = DAT_VALUE_UNKNOWN; + srq_ptr->param.outstanding_dto_count = DAT_VALUE_UNKNOWN; + + *srq_param = srq_ptr->param; + +bail: + return status; +} + +/* + * dapl_srq_resize + * + * Modify the size fo the event queue of a Shared Recieve Queue + * + * Input: + * srq_handle + * srq_max_recv_dto + * + * Output: + * none + * + * Returns: + * DAT_SUCCESS + * DAT_INVALID_HANDLE + * DAT_INVALID_PARAMETER + * DAT_INSUFFICIENT_RESOURCES + * DAT_INVALID_STATE + */ + +u32 dapl_srq_resize(DAT_SRQ_HANDLE srq_handle, int srq_max_recv_dto) +{ + struct dapl_ia *ia_ptr; + struct dapl_srq *srq_ptr; + u32 status = DAT_SUCCESS; + + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_resize (%p, %d)\n", + srq_handle, srq_max_recv_dto); + + if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); + goto bail; + } + + srq_ptr = (struct dapl_srq *)srq_handle; + ia_ptr = srq_ptr->header.owner_ia; + + /* + * Check for nonsense requests per the spec + */ + if (srq_max_recv_dto <= srq_ptr->param.low_watermark) { + status = DAT_ERROR(DAT_INVALID_STATE, DAT_NO_SUBTYPE); + goto bail; + } + + /* XXX Put implementation here XXX */ + + /* XXX */ status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); + +bail: + return status; +} + +/* + * dapl_srq_set_lw + * + * DAPL Requirements Version 1.2, 6.5.4 + * + * Set the low water mark for an SRQ and arm the SRQ to generate an + * event if it is reached. + * + * Input: + * srq_handle + * srq_max_recv_dto + * + * Output: + * none + * + * Returns: + * DAT_SUCCESS + * DAT_INVALID_HANDLE + * DAT_INVALID_PARAMETER + * DAT_MODEL_NOT_SUPPORTED + */ + +u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, int low_watermark) +{ + struct dapl_srq *srq_ptr; + u32 status = DAT_SUCCESS; + + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_set_lw (%p, %d)\n", + srq_handle, low_watermark); + + if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); + goto bail; + } + + srq_ptr = (struct dapl_srq *)srq_handle; + + /* XXX Put implementation here XXX */ + + /* XXX */ status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); + +bail: + return status; +} Index: linux-kernel-srq/dat-provider/dapl_srq_resize.c =================================================================== --- linux-kernel-srq/dat-provider/dapl_srq_resize.c (revision 2489) +++ linux-kernel-srq/dat-provider/dapl_srq_resize.c (working copy) @@ -1,88 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#include "dapl.h" -#include "dapl_srq_util.h" -#include "dapl_openib_util.h" - -/* - * dapl_srq_resize - * - * Modify the size fo the event queue of a Shared Recieve Queue - * - * Input: - * srq_handle - * srq_max_recv_dto - * - * Output: - * none - * - * Returns: - * DAT_SUCCESS - * DAT_INVALID_HANDLE - * DAT_INVALID_PARAMETER - * DAT_INSUFFICIENT_RESOURCES - * DAT_INVALID_STATE - */ - -u32 dapl_srq_resize(DAT_SRQ_HANDLE srq_handle, int srq_max_recv_dto) -{ - struct dapl_ia *ia_ptr; - struct dapl_srq *srq_ptr; - u32 dat_status = DAT_SUCCESS; - - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_resize (%p, %d)\n", - srq_handle, srq_max_recv_dto); - - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); - goto bail; - } - - srq_ptr = (struct dapl_srq *)srq_handle; - ia_ptr = srq_ptr->header.owner_ia; - - /* - * Check for nonsense requests per the spec - */ - if (srq_max_recv_dto <= srq_ptr->param.low_watermark) { - dat_status = DAT_ERROR(DAT_INVALID_STATE, DAT_NO_SUBTYPE); - goto bail; - } - - /* XXX Put implementation here XXX */ - - /* XXX */ dat_status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); - - bail: - return dat_status; -} Index: linux-kernel-srq/dat-provider/dapl_srq_create.c =================================================================== --- linux-kernel-srq/dat-provider/dapl_srq_create.c (revision 2489) +++ linux-kernel-srq/dat-provider/dapl_srq_create.c (working copy) @@ -1,128 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#include "dapl.h" -#include "dapl_ia_util.h" -#include "dapl_srq_util.h" -#include "dapl_openib_util.h" - -/* - * dapl_srq_create - * - * Create an instance of a Shared Receive Queue that is provided to the - * consumer at srq_handle. - * - * Input: - * ia_handle - * pz_handle - * srq_attr - * - * Output: - * srq_handle - * - * Returns: - * DAT_SUCCESS - * DAT_INSUFFICIENT_RESOURCES - * DAT_INVALID_HANDLE - * DAT_INVALID_PARAMETER - * ?DAT_INVALID_ATTRIBUTE?? - * DAT_MODEL_NOT_SUPPORTED - */ -u32 dapl_srq_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, - struct dat_srq_attr *srq_attr, DAT_SRQ_HANDLE *srq_handle) -{ - struct dapl_ia *ia_ptr; - struct dapl_srq *srq_ptr; - u32 dat_status = DAT_SUCCESS; - - dapl_dbg_log(DAPL_DBG_TYPE_API, - "dapl_srq_create (%p, %p, %p, %p)\n", - ia_handle, pz_handle, srq_attr, srq_handle); - - ia_ptr = (struct dapl_ia *)ia_handle; - - /* - * Verify parameters - */ - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); - goto bail; - } - - /* - * Verify non-required parameters. - * N.B. Assumption: any parameter that can be - * modified by dat_ep_modify() is not strictly - * required when the EP is created - */ - if (pz_handle != NULL && DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); - goto bail; - } - - if (srq_handle == NULL) { - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); - goto bail; - } - if ((unsigned long)srq_attr & 3) { - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); - goto bail; - } - - /* Allocate SRQ */ - srq_ptr = dapl_srq_alloc(ia_ptr, srq_attr); - if (srq_ptr == NULL) { - dat_status = - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); - goto bail; - } - - srq_ptr->param.ia_handle = (DAT_IA_HANDLE) ia_ptr; - srq_ptr->param.srq_state = DAT_SRQ_STATE_OPERATIONAL; - srq_ptr->param.pz_handle = pz_handle; - - /* - * XXX Allocate provider resource here!!! - */ - /* XXX */ dat_status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); - /* XXX */ dapl_srq_dealloc(srq_ptr); - /* XXX */ goto bail; - - /* Link it onto the IA */ - dapl_ia_link_srq(ia_ptr, srq_ptr); - - *srq_handle = srq_ptr; - - bail: - return dat_status; -} Index: linux-kernel-srq/dat-provider/dapl_srq_query.c =================================================================== --- linux-kernel-srq/dat-provider/dapl_srq_query.c (revision 2489) +++ linux-kernel-srq/dat-provider/dapl_srq_query.c (working copy) @@ -1,65 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#include "dapl.h" - -u32 dapl_srq_query(DAT_SRQ_HANDLE srq_handle, struct dat_srq_param *srq_param) -{ - struct dapl_srq *srq_ptr; - u32 dat_status = DAT_SUCCESS; - - dapl_dbg_log(DAPL_DBG_TYPE_API, - "dapl_srq_query (%p, %x, %p)\n", - srq_handle, srq_param); - - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); - goto bail; - } - if (srq_param == NULL) { - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); - goto bail; - } - - srq_ptr = (struct dapl_srq *)srq_handle; - - /* - * XXX Need to calculate available_dto_count and outstanding_dto_count - */ - srq_ptr->param.available_dto_count = DAT_VALUE_UNKNOWN; - srq_ptr->param.outstanding_dto_count = DAT_VALUE_UNKNOWN; - - *srq_param = srq_ptr->param; - - bail: - return dat_status; -} Index: linux-kernel-srq/dat-provider/Makefile =================================================================== --- linux-kernel-srq/dat-provider/Makefile (revision 2489) +++ linux-kernel-srq/dat-provider/Makefile (working copy) @@ -82,13 +82,7 @@ PROVIDER_MODULES := \ dapl_rsp_query \ dapl_set_consumer_context \ dapl_sp_util \ - dapl_srq_create \ - dapl_srq_free \ - dapl_srq_post_recv \ - dapl_srq_query \ - dapl_srq_resize \ - dapl_srq_set_lw \ - dapl_srq_util \ + dapl_srq \ dapl_timer_util \ dapl_util Index: linux-kernel-srq/dat-provider/dapl_srq_util.c =================================================================== --- linux-kernel-srq/dat-provider/dapl_srq_util.c (revision 2489) +++ linux-kernel-srq/dat-provider/dapl_srq_util.c (working copy) @@ -1,126 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#include "dapl_srq_util.h" -#include "dapl_ia_util.h" -#include "dapl_cookie.h" - -/* - * dapl_srq_alloc - * - * alloc and initialize an SRQ struct - * - * Input: - * IA INFO struct ptr - * SRQ ATTR ptr - * - * Output: - * none - * - * Returns: - * pointer to srq - * - */ -struct dapl_srq *dapl_srq_alloc(struct dapl_ia *ia_ptr, - const struct dat_srq_attr *srq_attr) -{ - struct dapl_srq *srq_ptr; - - /* Allocate SRQ */ - srq_ptr = kmalloc(sizeof *srq_ptr, GFP_ATOMIC); - if (!srq_ptr) - goto bail; - - /* zero the structure */ - memset(srq_ptr, 0, sizeof *srq_ptr); - - /* - * initialize the header - */ - srq_ptr->header.provider = ia_ptr->header.provider; - srq_ptr->header.magic = DAPL_MAGIC_SRQ; - srq_ptr->header.handle_type = DAT_HANDLE_TYPE_SRQ; - srq_ptr->header.owner_ia = ia_ptr; - srq_ptr->header.user_context.as_64 = 0; - srq_ptr->header.user_context.as_ptr = NULL; - atomic_set(&srq_ptr->srq_ref_count, 0); - - dapl_llist_init_entry(&srq_ptr->header.ia_list_entry); - spin_lock_init(&srq_ptr->header.lock); - - /* - * Initialize the body. - * XXX Assume srq_attrs is required - */ - srq_ptr->param.max_recv_dtos = srq_attr->max_recv_dtos; - srq_ptr->param.max_recv_iov = srq_attr->max_recv_iov; - srq_ptr->param.low_watermark = srq_attr->low_watermark; - - /* Get a cookie buffer to track outstanding recvs */ - if (DAT_SUCCESS != dapl_cb_create(&srq_ptr->recv_buffer, - (struct dapl_ep *)srq_ptr, - srq_ptr->param.max_recv_dtos)) { - dapl_srq_dealloc(srq_ptr); - srq_ptr = NULL; - goto bail; - } - -bail: - return srq_ptr; -} - -/* - * dapl_srq_dealloc - * - * Free the passed in SRQ structure. - * - * Input: - * SRQ pointer - * - * Output: - * none - * - * Returns: - * none - * - */ -void dapl_srq_dealloc(struct dapl_srq *srq_ptr) -{ - dapl_os_assert(srq_ptr->header.magic == DAPL_MAGIC_SRQ); - - /* reset magic to prevent reuse */ - srq_ptr->header.magic = DAPL_MAGIC_INVALID; - dapl_ia_unlink_srq(srq_ptr->header.owner_ia, srq_ptr); - dapl_cb_free(&srq_ptr->recv_buffer); - /* no need to destroy srq_ptr->header.lock */ - - kfree(srq_ptr); -} Index: linux-kernel-srq/dat-provider/dapl_srq_util.h =================================================================== --- linux-kernel-srq/dat-provider/dapl_srq_util.h (revision 2489) +++ linux-kernel-srq/dat-provider/dapl_srq_util.h (working copy) @@ -1,47 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/********************************************************************** - * - * HEADER: dapl_srq_util.h - * - * PURPOSE: Utility defs & routines for the SRQ data structure - * - * $Id$ - **********************************************************************/ - -#ifndef DAPL_SRQ_UTIL_H -#define DAPL_SRQ_UTIL_H - -#include "dapl.h" - -extern struct dapl_srq *dapl_srq_alloc(struct dapl_ia *ia, - const struct dat_srq_attr *srq_attr); - -extern void dapl_srq_dealloc(struct dapl_srq *srq_ptr); - -#endif /* DAPL_SRQ_UTIL_H */ Index: linux-kernel-srq/dat-provider/dapl_srq_set_lw.c =================================================================== --- linux-kernel-srq/dat-provider/dapl_srq_set_lw.c (revision 2489) +++ linux-kernel-srq/dat-provider/dapl_srq_set_lw.c (working copy) @@ -1,80 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#include "dapl.h" -#include "dapl_srq_util.h" -#include "dapl_openib_util.h" - -/* - * dapl_srq_set_lw - * - * DAPL Requirements Version 1.2, 6.5.4 - * - * Set the low water mark for an SRQ and arm the SRQ to generate an - * event if it is reached. - * - * Input: - * srq_handle - * srq_max_recv_dto - * - * Output: - * none - * - * Returns: - * DAT_SUCCESS - * DAT_INVALID_HANDLE - * DAT_INVALID_PARAMETER - * DAT_MODEL_NOT_SUPPORTED - */ - -u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, int low_watermark) -{ - struct dapl_srq *srq_ptr; - u32 dat_status = DAT_SUCCESS; - - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_set_lw (%p, %d)\n", - srq_handle, low_watermark); - - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); - goto bail; - } - - srq_ptr = (struct dapl_srq *)srq_handle; - - /* XXX Put implementation here XXX */ - - /* XXX */ dat_status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); - - bail: - return dat_status; -} Index: linux-kernel-srq/dat-provider/dapl_srq_post_recv.c =================================================================== --- linux-kernel-srq/dat-provider/dapl_srq_post_recv.c (revision 2489) +++ linux-kernel-srq/dat-provider/dapl_srq_post_recv.c (working copy) @@ -1,112 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#include "dapl.h" -#include "dapl_cookie.h" -#include "dapl_openib_util.h" - -/* - * dapl_srq_post_recv - * - * Post a receive buffer that can be used by any incoming - * message by any connected EP using the SRQ. Request to receive data - * over a connection of any ep handle into local_iov - * - * Input: - * srq_handle - * num_segments - * local_iov - * user_cookie - * - * Output: - * None. - * - * Returns: - * DAT_SUCCESS - * DAT_INSUFFICIENT_RESOURCES - * DAT_INVALID_PARAMETER - * DAT_INVALID_HANDLE - * DAT_INVALID_STATE - * DAT_PROTECTION_VIOLATION - * DAT_PROVILEGES_VIOLATION - */ -u32 dapl_srq_post_recv(DAT_SRQ_HANDLE srq_handle, int num_segments, - struct dat_lmr_triplet *local_iov, - DAT_DTO_COOKIE user_cookie) -{ - struct dapl_srq *srq_ptr; - struct dapl_cookie *cookie; - u32 dat_status; - - dapl_dbg_log(DAPL_DBG_TYPE_API, - "dapl_srq_post_recv (%p, %d, %p, %P)\n", - srq_handle, num_segments, local_iov, user_cookie.as_64); - - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); - goto bail; - } - - srq_ptr = (struct dapl_srq *)srq_handle; - - /* - * Synchronization ok since this buffer is only used for receive - * requests, which aren't allowed to race with each other. The - * app must syncronize access to the SRQ. - */ - dat_status = dapl_dto_cookie_alloc(&srq_ptr->recv_buffer, - DAPL_DTO_TYPE_RECV, - user_cookie, &cookie); - if (DAT_SUCCESS != dat_status) { - goto bail; - } - - /* - * Take reference before posting to avoid race conditions with - * completions - */ - atomic_inc(&srq_ptr->recv_count); - - /* - * Invoke provider specific routine to post DTO - */ - /* XXX Put code here XXX */ - /* XXX */ dat_status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); - - if (dat_status != DAT_SUCCESS) { - atomic_dec(&srq_ptr->recv_count); - dapl_cookie_dealloc(&srq_ptr->recv_buffer, cookie); - } - - bail: - return dat_status; -} Index: linux-kernel-srq/dat-provider/dapl_srq_free.c =================================================================== --- linux-kernel-srq/dat-provider/dapl_srq_free.c (revision 2489) +++ linux-kernel-srq/dat-provider/dapl_srq_free.c (working copy) @@ -1,112 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#include "dapl.h" -#include "dapl_ia_util.h" -#include "dapl_srq_util.h" -#include "dapl_openib_util.h" - -/* - * dapl_srq_free - * - * Destroy an instance of an SRQ - * - * Input: - * srq_handle - * - * Output: - * none - * - * Returns: - * DAT_SUCCESS - * DAT_INVALID_PARAMETER - * DAT_INVALID_STATE - */ -u32 dapl_srq_free(DAT_SRQ_HANDLE srq_handle) -{ - struct dapl_srq *srq_ptr; - struct dapl_ia *ia_ptr; - struct dat_srq_param *param; - u32 dat_status = DAT_SUCCESS; - - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_free (%p)\n", srq_handle); - - srq_ptr = (struct dapl_srq *)srq_handle; - param = &srq_ptr->param; - - /* - * Verify parameter & state - */ - if (DAPL_BAD_HANDLE(srq_ptr, DAPL_MAGIC_SRQ)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); - goto bail; - } - - if (atomic_read(&srq_ptr->srq_ref_count) != 0) { - /* - * The DAPL 1.2 spec says to return DAT_SRQ_IN_USE, which does - * not exist. Have filed the following as an eratta. - */ - dat_status = - DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_SRQ_IN_USE); - goto bail; - } - - ia_ptr = srq_ptr->header.owner_ia; - - /* - * Do verification of parameters and the state change atomically. - */ - spin_lock_irqsave(&srq_ptr->header.lock, srq_ptr->header.flags); - - /* Remove the SRQ from the IA */ - dapl_ia_unlink_srq(ia_ptr, srq_ptr); - - spin_unlock_irqrestore(&srq_ptr->header.lock, srq_ptr->header.flags); - - /* - * Finish tearing everything down. - */ - - /* - * Take care of the transport resource - */ - - /* XXX Put provider code here!!! */ - - /* Free the resource */ - dapl_srq_dealloc(srq_ptr); - - bail: - return dat_status; - -} Index: linux-kernel-srq/patches/alt_dat_provider_makefile =================================================================== --- linux-kernel-srq/patches/alt_dat_provider_makefile (revision 2489) +++ linux-kernel-srq/patches/alt_dat_provider_makefile (working copy) @@ -76,13 +76,7 @@ PROVIDER_MODULES := \ dapl_rsp_query \ dapl_set_consumer_context \ dapl_sp_util \ - dapl_srq_create \ - dapl_srq_free \ - dapl_srq_post_recv \ - dapl_srq_query \ - dapl_srq_resize \ - dapl_srq_set_lw \ - dapl_srq_util \ + dapl_srq \ dapl_timer_util \ dapl_util From tduffy at sun.com Thu May 26 11:49:57 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 26 May 2005 11:49:57 -0700 Subject: [openib-general] [PATCH 2/3] kDAPL: consolidate dapl_rsp*.c into one file In-Reply-To: <1117133253.17831.12.camel@duffman> References: <1117133253.17831.12.camel@duffman> Message-ID: <1117133397.17831.15.camel@duffman> Signed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel-srq/dat-provider/dapl_rsp.c linux-kernel-rsp/dat-provider/dapl_rsp.c --- linux-kernel-srq/dat-provider/dapl_rsp.c 1969-12-31 16:00:00.000000000 -0800 +++ linux-kernel-rsp/dat-provider/dapl_rsp.c 2005-05-25 14:58:10.928006000 -0700 @@ -0,0 +1,315 @@ +/* + * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * + * This Software is licensed under one of the following licenses: + * + * 1) under the terms of the "Common Public License 1.0" a copy of which is + * available from the Open Source Initiative, see + * http://www.opensource.org/licenses/cpl.php. + * + * 2) under the terms of the "The BSD License" a copy of which is + * available from the Open Source Initiative, see + * http://www.opensource.org/licenses/bsd-license.php. + * + * 3) under the terms of the "GNU General Public License (GPL) Version 2" a + * copy of which is available from the Open Source Initiative, see + * http://www.opensource.org/licenses/gpl-license.php. + * + * Licensee has the right to choose one of the above licenses. + * + * Redistributions of source code must retain the above copyright + * notice and one of the license notices. + * + * Redistributions in binary form must reproduce both the above copyright + * notice, one of the license notices in the documentation + * and/or other materials provided with the distribution. + */ + +/* + * $Id$ + */ + +#include "dapl.h" +#include "dapl_sp_util.h" +#include "dapl_ia_util.h" +#include "dapl_ep_util.h" +#include "dapl_openib_util.h" + +/* + * dapl_rsp_create + * + * Create a Resereved Service Point with the specified Endpoint + * that generates at most one Connection Request that is + * delivered to the specified Event Dispatcher in a notification + * event + * + * Input: + * ia_handle + * conn_qual + * ep_handle + * evd_handle + * + * Output: + * rsp_handle + * + * Returns: + * DAT_SUCCESS + * DAT_INSUFFICIENT_RESOURCES + * DAT_INVALID_PARAMETER + * DAT_INVALID_STATE + * DAT_CONN_QUAL_IN_USE + */ +u32 dapl_rsp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, + DAT_EP_HANDLE ep_handle, DAT_EVD_HANDLE evd_handle, + DAT_RSP_HANDLE *rsp_handle) +{ + struct dapl_ia *ia_ptr; + struct dapl_sp *sp_ptr; + struct dapl_evd *evd_ptr; + struct dapl_ep *ep_ptr; + boolean_t sp_found; + u32 status = DAT_SUCCESS; + + ia_ptr = (struct dapl_ia *)ia_handle; + + dapl_dbg_log(DAPL_DBG_TYPE_CM, + ">>> dapl_rsp_free conn_qual: %x EP: %p\n", + conn_qual, ep_handle); + + if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); + goto bail; + } + if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) { + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); + goto bail; + } + if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + status = DAT_ERROR(DAT_INVALID_HANDLE, + DAT_INVALID_HANDLE_EVD_CR); + goto bail; + } + + if (rsp_handle == NULL) { + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); + goto bail; + } + + ep_ptr = (struct dapl_ep *)ep_handle; + if (ep_ptr->param.ep_state != DAT_EP_STATE_UNCONNECTED) { + status = DAT_ERROR(DAT_INVALID_STATE, + dapl_ep_state_subtype(ep_ptr)); + goto bail; + } + + evd_ptr = (struct dapl_evd *)evd_handle; + if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { + status = DAT_ERROR(DAT_INVALID_HANDLE, + DAT_INVALID_HANDLE_EVD_CR); + goto bail; + } + + sp_ptr = dapl_ia_sp_search(ia_ptr, conn_qual, FALSE); + sp_found = TRUE; + if (sp_ptr == NULL) { + sp_found = FALSE; + + /* Allocate RSP */ + sp_ptr = dapl_sp_alloc(ia_ptr, FALSE); + if (sp_ptr == NULL) { + status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, + DAT_RESOURCE_MEMORY); + goto bail; + } + } + + /* + * Fill out the RSP args + */ + sp_ptr->ia_handle = ia_handle; + sp_ptr->conn_qual = conn_qual; + sp_ptr->evd_handle = evd_handle; + sp_ptr->psp_flags = 0; + sp_ptr->ep_handle = ep_handle; + + /* + * Take a reference on the EVD handle + */ + atomic_inc(&evd_ptr->evd_ref_count); + + /* + * Update the EP state indicating the provider now owns it + */ + ep_ptr->param.ep_state = DAT_EP_STATE_RESERVED; + + /* + * Set up a listener for a connection. Connections can arrive + * even before this call returns! + */ + sp_ptr->state = DAPL_SP_STATE_RSP_LISTENING; + sp_ptr->listening = TRUE; + + if (sp_found == FALSE) { + /* Link it onto the IA */ + dapl_ia_link_rsp(ia_ptr, sp_ptr); + + status = dapl_ib_setup_conn_listener(ia_ptr, sp_ptr); + + if (status != DAT_SUCCESS) { + /* + * Have a problem setting up the connection, something + * wrong! Decrements the EVD refcount & release it. Set + * the state to FREE, so we know the call failed. + */ + atomic_dec(&evd_ptr->evd_ref_count); + sp_ptr->evd_handle = NULL; + sp_ptr->state = DAPL_SP_STATE_FREE; + dapl_ia_unlink_sp(ia_ptr, sp_ptr); + dapl_sp_dealloc(sp_ptr); + + dapl_dbg_log(DAPL_DBG_TYPE_CM, + "--> dapl_rsp_create setup_conn_listener failed: %x\n", + status); + + goto bail; + } + } + + /* + * Return handle to the user + */ + *rsp_handle = (DAT_RSP_HANDLE)sp_ptr; + +bail: + return status; +} + +/* + * dapl_rsp_free + * + * Destroy a specific instance of a Reserved Service Point. + * + * Input: + * rsp_handle + * + * Output: + * none + * + * Returns: + * DAT_SUCCESS + * DAT_INVALID_HANDLE + */ +u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_handle) +{ + struct dapl_ia *ia_ptr; + struct dapl_sp *sp_ptr; + struct dapl_ep *ep_ptr; + u32 status = DAT_SUCCESS; + + sp_ptr = (struct dapl_sp *)rsp_handle; + /* + * Verify handle + */ + dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_rsp_free %p\n", rsp_handle); + if (DAPL_BAD_HANDLE(sp_ptr, DAPL_MAGIC_RSP)) { + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RSP); + goto bail; + } + + /* ia_ptr = (struct dapl_ia *)sp_ptr->header.owner_ia; */ + ia_ptr = sp_ptr->header.owner_ia; + + /* + * Remove the connection listener if there are no connections. If + * we defer removing the sp it becomes something of a zombie + * container until disconnection, after which it will be cleaned up. + */ + spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + + /* + * Make sure we don't leave a dangling EP. If the state is still + * RESERVED then the RSP still owns it. + */ + ep_ptr = (struct dapl_ep *)sp_ptr->ep_handle; + if (ep_ptr != NULL && ep_ptr->param.ep_state == DAT_EP_STATE_RESERVED) + ep_ptr->param.ep_state = DAT_EP_STATE_UNCONNECTED; + sp_ptr->ep_handle = NULL; + + /* + * Release reference on EVD. If an error was encountered in a previous + * free the evd_handle will be NULL + */ + if (sp_ptr->evd_handle) { + atomic_dec(&((struct dapl_evd *)sp_ptr->evd_handle)-> + evd_ref_count); + sp_ptr->evd_handle = NULL; + } + + /* + * Release the base resource if there are no outstanding connections; + * else the last disconnect on this RSP will free it up. The RSP + * is used to contain CR records for each connection, which + * contain information necessary to disconnect. + * sp_ptr->listening will be TRUE if there has never been a + * connection event, and FALSE if a connection attempt resulted + * in a reject. + */ + if (sp_ptr->cr_list_count == 0) { + /* This RSP has never been used. Clean it up */ + sp_ptr->listening = FALSE; + sp_ptr->state = DAPL_SP_STATE_FREE; + spin_unlock_irqrestore(&sp_ptr->header.lock, + sp_ptr->header.flags); + + status = dapl_ib_remove_conn_listener(ia_ptr, sp_ptr); + if (status != DAT_SUCCESS) { + sp_ptr->state = DAPL_SP_STATE_RSP_LISTENING; + goto bail; + } + dapl_ia_unlink_sp(ia_ptr, sp_ptr); + dapl_sp_dealloc(sp_ptr); + } else { + /* + * The RSP is now in the pending state, where it will sit until + * the connection terminates or the app uses the same + * ServiceID again, which will reactivate it. + */ + sp_ptr->state = DAPL_SP_STATE_RSP_PENDING; + spin_unlock_irqrestore(&sp_ptr->header.lock, + sp_ptr->header.flags); + } + +bail: + return status; +} + +u32 dapl_rsp_query(DAT_RSP_HANDLE rsp_handle, struct dat_rsp_param *rsp_param) +{ + struct dapl_sp *sp_ptr; + u32 status; + + if (DAPL_BAD_HANDLE(rsp_handle, DAPL_MAGIC_RSP)) { + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RSP); + goto bail; + } + + if (NULL == rsp_param) { + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); + goto bail; + } + + sp_ptr = (struct dapl_sp *)rsp_handle; + + /* + * Fill in the RSP params + */ + rsp_param->ia_handle = sp_ptr->ia_handle; + rsp_param->conn_qual = sp_ptr->conn_qual; + rsp_param->evd_handle = sp_ptr->evd_handle; + rsp_param->ep_handle = sp_ptr->ep_handle; + + status = DAT_SUCCESS; + +bail: + return status; +} diff -Nurp -X /home/tduffy/dontdiff linux-kernel-srq/dat-provider/dapl_rsp_create.c linux-kernel-rsp/dat-provider/dapl_rsp_create.c --- linux-kernel-srq/dat-provider/dapl_rsp_create.c 2005-05-23 22:21:46.240013000 -0700 +++ linux-kernel-rsp/dat-provider/dapl_rsp_create.c 1969-12-31 16:00:00.000000000 -0800 @@ -1,188 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id: dapl_rsp_create.c 2462 2005-05-24 02:28:24Z jlentini $ - */ - -#include "dapl.h" -#include "dapl_sp_util.h" -#include "dapl_ia_util.h" -#include "dapl_ep_util.h" -#include "dapl_openib_util.h" - -/* - * dapl_rsp_create - * - * Create a Resereved Service Point with the specified Endpoint - * that generates at most one Connection Request that is - * delivered to the specified Event Dispatcher in a notification - * event - * - * Input: - * ia_handle - * conn_qual - * ep_handle - * evd_handle - * - * Output: - * rsp_handle - * - * Returns: - * DAT_SUCCESS - * DAT_INSUFFICIENT_RESOURCES - * DAT_INVALID_PARAMETER - * DAT_INVALID_STATE - * DAT_CONN_QUAL_IN_USE - */ -u32 dapl_rsp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, - DAT_EP_HANDLE ep_handle, DAT_EVD_HANDLE evd_handle, - DAT_RSP_HANDLE *rsp_handle) -{ - struct dapl_ia *ia_ptr; - struct dapl_sp *sp_ptr; - struct dapl_evd *evd_ptr; - struct dapl_ep *ep_ptr; - boolean_t sp_found; - u32 dat_status = DAT_SUCCESS; - - ia_ptr = (struct dapl_ia *)ia_handle; - - dapl_dbg_log(DAPL_DBG_TYPE_CM, - ">>> dapl_rsp_free conn_qual: %x EP: %p\n", - conn_qual, ep_handle); - - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); - goto bail; - } - if (DAPL_BAD_HANDLE(ep_handle, DAPL_MAGIC_EP)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EP); - goto bail; - } - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); - goto bail; - } - - if (rsp_handle == NULL) { - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); - goto bail; - } - - ep_ptr = (struct dapl_ep *)ep_handle; - if (ep_ptr->param.ep_state != DAT_EP_STATE_UNCONNECTED) { - dat_status = - DAT_ERROR(DAT_INVALID_STATE, dapl_ep_state_subtype(ep_ptr)); - goto bail; - } - - evd_ptr = (struct dapl_evd *)evd_handle; - if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); - goto bail; - } - - sp_ptr = dapl_ia_sp_search(ia_ptr, conn_qual, FALSE); - sp_found = TRUE; - if (sp_ptr == NULL) { - sp_found = FALSE; - - /* Allocate RSP */ - sp_ptr = dapl_sp_alloc(ia_ptr, FALSE); - if (sp_ptr == NULL) { - dat_status = - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, - DAT_RESOURCE_MEMORY); - goto bail; - } - } - - /* - * Fill out the RSP args - */ - sp_ptr->ia_handle = ia_handle; - sp_ptr->conn_qual = conn_qual; - sp_ptr->evd_handle = evd_handle; - sp_ptr->psp_flags = 0; - sp_ptr->ep_handle = ep_handle; - - /* - * Take a reference on the EVD handle - */ - atomic_inc(&evd_ptr->evd_ref_count); - - /* - * Update the EP state indicating the provider now owns it - */ - ep_ptr->param.ep_state = DAT_EP_STATE_RESERVED; - - /* - * Set up a listener for a connection. Connections can arrive - * even before this call returns! - */ - sp_ptr->state = DAPL_SP_STATE_RSP_LISTENING; - sp_ptr->listening = TRUE; - - if (sp_found == FALSE) { - /* Link it onto the IA */ - dapl_ia_link_rsp(ia_ptr, sp_ptr); - - dat_status = dapl_ib_setup_conn_listener(ia_ptr, sp_ptr); - - if (dat_status != DAT_SUCCESS) { - /* - * Have a problem setting up the connection, something - * wrong! Decrements the EVD refcount & release it. Set - * the state to FREE, so we know the call failed. - */ - atomic_dec(&evd_ptr->evd_ref_count); - sp_ptr->evd_handle = NULL; - sp_ptr->state = DAPL_SP_STATE_FREE; - dapl_ia_unlink_sp(ia_ptr, sp_ptr); - dapl_sp_dealloc(sp_ptr); - - dapl_dbg_log(DAPL_DBG_TYPE_CM, - "--> dapl_rsp_create setup_conn_listener failed: %x\n", - dat_status); - - goto bail; - } - } - - /* - * Return handle to the user - */ - *rsp_handle = (DAT_RSP_HANDLE) sp_ptr; - - bail: - return dat_status; -} diff -Nurp -X /home/tduffy/dontdiff linux-kernel-srq/dat-provider/dapl_rsp_free.c linux-kernel-rsp/dat-provider/dapl_rsp_free.c --- linux-kernel-srq/dat-provider/dapl_rsp_free.c 2005-05-20 22:55:18.517008000 -0700 +++ linux-kernel-rsp/dat-provider/dapl_rsp_free.c 1969-12-31 16:00:00.000000000 -0800 @@ -1,133 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id: dapl_rsp_free.c 2433 2005-05-21 04:11:03Z jlentini $ - */ - -#include "dapl.h" -#include "dapl_sp_util.h" -#include "dapl_ia_util.h" -#include "dapl_openib_util.h" - -/* - * dapl_rsp_free - * - * Destroy a specific instance of a Reserved Service Point. - * - * Input: - * rsp_handle - * - * Output: - * none - * - * Returns: - * DAT_SUCCESS - * DAT_INVALID_HANDLE - */ -u32 dapl_rsp_free(DAT_RSP_HANDLE rsp_handle) -{ - struct dapl_ia *ia_ptr; - struct dapl_sp *sp_ptr; - struct dapl_ep *ep_ptr; - u32 dat_status = DAT_SUCCESS; - - sp_ptr = (struct dapl_sp *)rsp_handle; - /* - * Verify handle - */ - dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_rsp_free %p\n", rsp_handle); - if (DAPL_BAD_HANDLE(sp_ptr, DAPL_MAGIC_RSP)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RSP); - goto bail; - } - - /* ia_ptr = (struct dapl_ia *)sp_ptr->header.owner_ia; */ - ia_ptr = sp_ptr->header.owner_ia; - - /* - * Remove the connection listener if there are no connections. If - * we defer removing the sp it becomes something of a zombie - * container until disconnection, after which it will be cleaned up. - */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); - - /* - * Make sure we don't leave a dangling EP. If the state is still - * RESERVED then the RSP still owns it. - */ - ep_ptr = (struct dapl_ep *)sp_ptr->ep_handle; - if (ep_ptr != NULL && ep_ptr->param.ep_state == DAT_EP_STATE_RESERVED) - ep_ptr->param.ep_state = DAT_EP_STATE_UNCONNECTED; - sp_ptr->ep_handle = NULL; - - /* Release reference on EVD. If an error was encountered in a previous - * free the evd_handle will be NULL - */ - if (sp_ptr->evd_handle) { - atomic_dec(&((struct dapl_evd *)sp_ptr->evd_handle)-> - evd_ref_count); - sp_ptr->evd_handle = NULL; - } - - /* - * Release the base resource if there are no outstanding connections; - * else the last disconnect on this RSP will free it up. The RSP - * is used to contain CR records for each connection, which - * contain information necessary to disconnect. - * sp_ptr->listening will be TRUE if there has never been a - * connection event, and FALSE if a connection attempt resulted - * in a reject. - */ - if (sp_ptr->cr_list_count == 0) { - /* This RSP has never been used. Clean it up */ - sp_ptr->listening = FALSE; - sp_ptr->state = DAPL_SP_STATE_FREE; - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); - - dat_status = dapl_ib_remove_conn_listener(ia_ptr, sp_ptr); - if (dat_status != DAT_SUCCESS) { - sp_ptr->state = DAPL_SP_STATE_RSP_LISTENING; - goto bail; - } - dapl_ia_unlink_sp(ia_ptr, sp_ptr); - dapl_sp_dealloc(sp_ptr); - } else { - /* The RSP is now in the pending state, where it will sit until - * the connection terminates or the app uses the same - * ServiceID again, which will reactivate it. - */ - sp_ptr->state = DAPL_SP_STATE_RSP_PENDING; - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); - } - -bail: - return dat_status; -} diff -Nurp -X /home/tduffy/dontdiff linux-kernel-srq/dat-provider/dapl_rsp_query.c linux-kernel-rsp/dat-provider/dapl_rsp_query.c --- linux-kernel-srq/dat-provider/dapl_rsp_query.c 2005-05-20 22:55:18.990015000 -0700 +++ linux-kernel-rsp/dat-provider/dapl_rsp_query.c 1969-12-31 16:00:00.000000000 -0800 @@ -1,63 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id: dapl_rsp_query.c 2433 2005-05-21 04:11:03Z jlentini $ - */ - -#include "dapl.h" - -u32 dapl_rsp_query(DAT_RSP_HANDLE rsp_handle, struct dat_rsp_param *rsp_param) -{ - struct dapl_sp *sp_ptr; - u32 status; - - if (DAPL_BAD_HANDLE(rsp_handle, DAPL_MAGIC_RSP)) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_RSP); - goto bail; - } - - if (NULL == rsp_param) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); - goto bail; - } - - sp_ptr = (struct dapl_sp *)rsp_handle; - - /* - * Fill in the RSP params - */ - rsp_param->ia_handle = sp_ptr->ia_handle; - rsp_param->conn_qual = sp_ptr->conn_qual; - rsp_param->evd_handle = sp_ptr->evd_handle; - rsp_param->ep_handle = sp_ptr->ep_handle; - - status = DAT_SUCCESS; - -bail: - return status; -} diff -Nurp -X /home/tduffy/dontdiff linux-kernel-srq/dat-provider/Makefile linux-kernel-rsp/dat-provider/Makefile --- linux-kernel-srq/dat-provider/Makefile 2005-05-25 14:48:11.375001000 -0700 +++ linux-kernel-rsp/dat-provider/Makefile 2005-05-25 15:01:00.683014000 -0700 @@ -77,9 +77,7 @@ PROVIDER_MODULES := \ dapl_pz \ dapl_ring_buffer_util \ dapl_rmr \ - dapl_rsp_create \ - dapl_rsp_free \ - dapl_rsp_query \ + dapl_rsp \ dapl_set_consumer_context \ dapl_sp_util \ dapl_srq \ diff -Nurp -X /home/tduffy/dontdiff linux-kernel-srq/patches/alt_dat_provider_makefile linux-kernel-rsp/patches/alt_dat_provider_makefile --- linux-kernel-srq/patches/alt_dat_provider_makefile 2005-05-25 14:49:44.866002000 -0700 +++ linux-kernel-rsp/patches/alt_dat_provider_makefile 2005-05-25 14:53:43.453015000 -0700 @@ -71,9 +71,7 @@ PROVIDER_MODULES := \ dapl_pz \ dapl_ring_buffer_util \ dapl_rmr \ - dapl_rsp_create \ - dapl_rsp_free \ - dapl_rsp_query \ + dapl_rsp \ dapl_set_consumer_context \ dapl_sp_util \ dapl_srq \ From roland at topspin.com Thu May 26 11:50:44 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 26 May 2005 11:50:44 -0700 Subject: [openib-general] [HELP] Encounter Kernel Panic when Add MellanoxHCA Supporting on 2.6.9 Kernel In-Reply-To: <52vf56opdw.fsf@topspin.com> (Roland Dreier's message of "Wed, 25 May 2005 16:30:03 -0700") References: <1AC79F16F5C5284499BB9591B33D6F00047F91F4@orsmsx408> <52vf56opdw.fsf@topspin.com> Message-ID: <52wtplkeij.fsf@topspin.com> Hmm, I can't make it happen here unfortunately... I just get the following on 2.6.12-rc5 with CONFIG_DEBUG_SPINLOCK: [ 26.001979] ib_mthca: Mellanox InfiniBand HCA driver v0.06-pre (November 8, 2004) [ 26.026622] ib_mthca: Initializing Mellanox Technologies MT23108 InfiniHost (0000:04:00.0) [ 27.326318] ib_mthca 0000:04:00.0: HCA FW version 3.0.1 is old (3.3.2 is current). [ 27.351207] ib_mthca 0000:04:00.0: If you have problems, try updating your HCA FW. but it actually seems to work (well enough that the port goes to active). - R. From tduffy at sun.com Thu May 26 11:50:56 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 26 May 2005 11:50:56 -0700 Subject: [openib-general] [PATCH 3/3] kDAPL: consolidate dapl_psp*.c into one file In-Reply-To: <1117133397.17831.15.camel@duffman> References: <1117133253.17831.12.camel@duffman> <1117133397.17831.15.camel@duffman> Message-ID: <1117133456.17831.16.camel@duffman> Signed-off-by: Tom Duffy diff -Nurp -X /home/tduffy/dontdiff linux-kernel-rsp/dat-provider/dapl_psp.c linux-kernel-psp/dat-provider/dapl_psp.c --- linux-kernel-rsp/dat-provider/dapl_psp.c 1969-12-31 16:00:00.000000000 -0800 +++ linux-kernel-psp/dat-provider/dapl_psp.c 2005-05-25 15:11:42.274000000 -0700 @@ -0,0 +1,446 @@ +/* + * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. + * + * This Software is licensed under one of the following licenses: + * + * 1) under the terms of the "Common Public License 1.0" a copy of which is + * available from the Open Source Initiative, see + * http://www.opensource.org/licenses/cpl.php. + * + * 2) under the terms of the "The BSD License" a copy of which is + * available from the Open Source Initiative, see + * http://www.opensource.org/licenses/bsd-license.php. + * + * 3) under the terms of the "GNU General Public License (GPL) Version 2" a + * copy of which is available from the Open Source Initiative, see + * http://www.opensource.org/licenses/gpl-license.php. + * + * Licensee has the right to choose one of the above licenses. + * + * Redistributions of source code must retain the above copyright + * notice and one of the license notices. + * + * Redistributions in binary form must reproduce both the above copyright + * notice, one of the license notices in the documentation + * and/or other materials provided with the distribution. + */ + +/* + * $Id: dapl_psp_create_any.c 2462 2005-05-24 02:28:24Z jlentini $ + */ + +#include "dapl.h" +#include "dapl_sp_util.h" +#include "dapl_ia_util.h" +#include "dapl_openib_util.h" + +/* + * dapl_psp_create_any + * + * Create a persistent Public Service Point that can recieve multiple + * requests for connections and generate multiple connection request + * instances that wil be delivered to the specified Event Dispatcher + * in a notification event. Differs from dapl_psp_create() in that + * the conn_qual is selected by the implementation and returned to + * the user. + * + * Input: + * ia_handle + * evd_handle + * psp_flags + * + * Output: + * conn_qual + * psp_handle + * + * Returns: + * DAT_SUCCESS + * DAT_INSUFFICIENT_RESOURCES + * DAT_INVALID_HANDLE + * DAT_INVALID_PARAMETER + * DAT_CONN_QUAL_IN_USE + * DAT_MODEL_NOT_SUPPORTED + */ +u32 dapl_psp_create_any(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL *conn_qual, + DAT_EVD_HANDLE evd_handle, enum dat_psp_flags psp_flags, + DAT_PSP_HANDLE *psp_handle) +{ + static DAT_CONN_QUAL hint_conn_qual = 1000; /* seed value */ + struct dapl_ia *ia_ptr; + struct dapl_sp *sp_ptr; + struct dapl_evd *evd_ptr; + u32 status = DAT_SUCCESS; + int i; + + ia_ptr = (struct dapl_ia *)ia_handle; + + if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); + goto bail; + } + if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + status = + DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); + goto bail; + } + + if (psp_handle == NULL) { + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); + goto bail; + } + if (conn_qual == NULL) { + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); + goto bail; + } + + evd_ptr = (struct dapl_evd *)evd_handle; + if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { + status = DAT_ERROR(DAT_INVALID_HANDLE, + DAT_INVALID_HANDLE_EVD_CR); + goto bail; + } + + if (psp_flags != DAT_PSP_CONSUMER_FLAG && + psp_flags != DAT_PSP_PROVIDER_FLAG) { + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); + goto bail; + } + + /* Allocate PSP */ + sp_ptr = dapl_sp_alloc(ia_ptr, TRUE); + if (sp_ptr == NULL) { + status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, + DAT_RESOURCE_MEMORY); + goto bail; + } + + /* + * Fill out the args for a PSP + */ + sp_ptr->ia_handle = ia_handle; + sp_ptr->evd_handle = evd_handle; + sp_ptr->psp_flags = psp_flags; + sp_ptr->ep_handle = NULL; + + /* + * Take a reference on the EVD handle + */ + atomic_inc(&evd_ptr->evd_ref_count); + + /* Link it onto the IA */ + dapl_ia_link_psp(ia_ptr, sp_ptr); + + /* + * Set up a listener for a connection. Connections can arrive + * even before this call returns! + */ + sp_ptr->state = DAPL_SP_STATE_PSP_LISTENING; + sp_ptr->listening = TRUE; + + /* + * If we have a big number of tries and we still haven't + * found a service_ID we can use, bail out with an error, + * something is wrong! + */ + for (i = 0, sp_ptr->conn_qual = hint_conn_qual; i < 100000; i++) { + status = dapl_ib_setup_conn_listener(ia_ptr, sp_ptr); + if (DAT_SUCCESS == status) + break; + else + sp_ptr->conn_qual++; + } + hint_conn_qual = sp_ptr->conn_qual + 1; + + if (status != DAT_SUCCESS) { + atomic_dec(&evd_ptr->evd_ref_count); + sp_ptr->evd_handle = NULL; + dapl_ia_unlink_sp(ia_ptr, sp_ptr); + dapl_sp_dealloc(sp_ptr); + + dapl_dbg_log(DAPL_DBG_TYPE_ERR, + "dapl_psp_create cannot set up conn listener: %x\n", + status); + + goto bail; + } + + /* + * Return handle to the user + */ + *conn_qual = sp_ptr->conn_qual; + *psp_handle = (DAT_PSP_HANDLE) sp_ptr; + +bail: + return status; +} + +/* + * dapl_psp_create + * + * Create a persistent Public Service Point that can recieve multiple + * requests for connections and generate multiple connection request + * instances that wil be delivered to the specified Event Dispatcher + * in a notification event. + * + * Input: + * ia_handle + * conn_qual + * evd_handle + * psp_flags + * + * Output: + * psp_handle + * + * Returns: + * DAT_SUCCESS + * DAT_INSUFFICIENT_RESOURCES + * DAT_INVALID_PARAMETER + * DAT_CONN_QUAL_IN_USE + * DAT_MODEL_NOT_SUPPORTED + */ +u32 dapl_psp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, + DAT_EVD_HANDLE evd_handle, enum dat_psp_flags psp_flags, + DAT_PSP_HANDLE *psp_handle) +{ + struct dapl_ia *ia_ptr; + struct dapl_sp *sp_ptr; + struct dapl_evd *evd_ptr; + boolean_t sp_found; + u32 status = DAT_SUCCESS; + + ia_ptr = (struct dapl_ia *)ia_handle; + + if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); + goto bail; + } + if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { + status = DAT_ERROR(DAT_INVALID_HANDLE, + DAT_INVALID_HANDLE_EVD_CR); + goto bail; + } + + if (psp_handle == NULL) { + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); + goto bail; + } + + evd_ptr = (struct dapl_evd *)evd_handle; + if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { + status = DAT_ERROR(DAT_INVALID_HANDLE, + DAT_INVALID_HANDLE_EVD_CR); + goto bail; + } + + if (psp_flags != DAT_PSP_CONSUMER_FLAG && + psp_flags != DAT_PSP_PROVIDER_FLAG) { + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); + goto bail; + } + + /* + * See if we have a quiescent listener to use for this PSP, else + * create one and set it listening + */ + sp_ptr = dapl_ia_sp_search(ia_ptr, conn_qual, TRUE); + sp_found = TRUE; + if (sp_ptr == NULL) { + /* Allocate PSP */ + sp_found = FALSE; + sp_ptr = dapl_sp_alloc(ia_ptr, TRUE); + if (sp_ptr == NULL) { + status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, + DAT_RESOURCE_MEMORY); + goto bail; + } + } else if (sp_ptr->listening == TRUE) { + status = DAT_ERROR(DAT_CONN_QUAL_IN_USE, 0); + goto bail; + } + + /* + * Fill out the args for a PSP + */ + sp_ptr->ia_handle = ia_handle; + sp_ptr->conn_qual = conn_qual; + sp_ptr->evd_handle = evd_handle; + sp_ptr->psp_flags = psp_flags; + sp_ptr->ep_handle = NULL; + + /* + * Take a reference on the EVD handle + */ + atomic_inc(&evd_ptr->evd_ref_count); + + /* + * Set up a listener for a connection. Connections can arrive + * even before this call returns! + */ + sp_ptr->state = DAPL_SP_STATE_PSP_LISTENING; + sp_ptr->listening = TRUE; + + /* + * If this is a new sp we need to add it to the IA queue, and set up + * a conn_listener. + */ + if (sp_found == FALSE) { + /* + * Link it onto the IA before enabling it to receive conn + * requests + */ + dapl_ia_link_psp(ia_ptr, sp_ptr); + + status = dapl_ib_setup_conn_listener(ia_ptr, sp_ptr); + + if (status != DAT_SUCCESS) { + /* + * Have a problem setting up the connection, something + * wrong! Decrements the EVD refcount & release it. + */ + atomic_dec(&evd_ptr->evd_ref_count); + sp_ptr->evd_handle = NULL; + dapl_ia_unlink_sp(ia_ptr, sp_ptr); + dapl_sp_dealloc(sp_ptr); + + dapl_dbg_log(DAPL_DBG_TYPE_CM, "dapl_psp_create " + "setup_conn_listener failed: %x\n", + status); + + goto bail; + } + } + + /* + * Return handle to the user + */ + *psp_handle = (DAT_PSP_HANDLE) sp_ptr; + +bail: + return status; +} + +/* + * dapl_psp_free + * + * Destroy a specific instance of a Service Point. + * + * Input: + * psp_handle + * + * Output: + * none + * + * Returns: + * DAT_SUCCESS + * DAT_INVALID_PARAMETER + */ +u32 dapl_psp_free(DAT_PSP_HANDLE psp_handle) +{ + struct dapl_ia *ia_ptr; + struct dapl_sp *sp_ptr; + DAPL_SP_STATE save_state; + u32 status = DAT_SUCCESS; + + sp_ptr = (struct dapl_sp *)psp_handle; + /* + * Verify handle + */ + dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_psp_free %p\n", psp_handle); + + if (DAPL_BAD_HANDLE(sp_ptr, DAPL_MAGIC_PSP)) { + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PSP); + goto bail; + } + + /* ia_ptr = (struct dapl_ia *)sp_ptr->header.owner_ia; */ + ia_ptr = sp_ptr->header.owner_ia; + /* + * Remove the connection listener if it has been established + * and there are no current connections in progress. + * If we defer removing the sp it becomes something of a zombie + * container until the last connection is disconnected, after + * which it will be cleaned up. + */ + spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); + + sp_ptr->listening = FALSE; + + /* + * Release reference on EVD. If an error was encountered in a previous + * free the evd_handle will be NULL + */ + if (sp_ptr->evd_handle) { + atomic_dec(&((struct dapl_evd *)sp_ptr->evd_handle)-> + evd_ref_count); + sp_ptr->evd_handle = NULL; + } + + /* + * Release the base resource if there are no outstanding + * connections; else the last disconnect on this PSP will free it + * up. The PSP is used to contain CR records for each connection, + * which contain information necessary to disconnect. + */ + dapl_dbg_log(DAPL_DBG_TYPE_CM, + ">>> dapl_psp_free: state %d cr_list_count %d\n", + sp_ptr->state, sp_ptr->cr_list_count); + if ((sp_ptr->state == DAPL_SP_STATE_PSP_LISTENING || + sp_ptr->state == DAPL_SP_STATE_PSP_PENDING) && + sp_ptr->cr_list_count == 0) { + save_state = sp_ptr->state; + sp_ptr->state = DAPL_SP_STATE_FREE; + spin_unlock_irqrestore(&sp_ptr->header.lock, + sp_ptr->header.flags); + + status = dapl_ib_remove_conn_listener(ia_ptr, sp_ptr); + if (status != DAT_SUCCESS) { + /* revert to entry state on error */ + sp_ptr->state = save_state; + goto bail; + } + dapl_ia_unlink_sp(ia_ptr, sp_ptr); + dapl_sp_dealloc(sp_ptr); + } else { + /* + * The PSP is now in the pending state, where it will sit until + * the last connection terminates or the app uses the same + * ServiceID again, which will reactivate it. + */ + sp_ptr->state = DAPL_SP_STATE_PSP_PENDING; + spin_unlock_irqrestore(&sp_ptr->header.lock, sp_ptr->header.flags); + dapl_dbg_log(DAPL_DBG_TYPE_CM, + ">>> dapl_psp_free: PSP PENDING\n"); + } + +bail: + return status; +} + +u32 dapl_psp_query(DAT_PSP_HANDLE psp_handle, struct dat_psp_param *psp_param) +{ + struct dapl_sp *sp_ptr; + u32 status; + + if (DAPL_BAD_HANDLE(psp_handle, DAPL_MAGIC_PSP) || + ((struct dapl_sp *)psp_handle)->listening != TRUE) { + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PSP); + goto bail; + } + + if (NULL == psp_param) { + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); + goto bail; + } + + sp_ptr = (struct dapl_sp *)psp_handle; + + psp_param->ia_handle = sp_ptr->ia_handle; + psp_param->conn_qual = sp_ptr->conn_qual; + psp_param->evd_handle = sp_ptr->evd_handle; + psp_param->psp_flags = sp_ptr->psp_flags; + + status = DAT_SUCCESS; + +bail: + return status; +} diff -Nurp -X /home/tduffy/dontdiff linux-kernel-rsp/dat-provider/dapl_psp_create_any.c linux-kernel-psp/dat-provider/dapl_psp_create_any.c --- linux-kernel-rsp/dat-provider/dapl_psp_create_any.c 2005-05-23 22:21:46.111018000 -0700 +++ linux-kernel-psp/dat-provider/dapl_psp_create_any.c 1969-12-31 16:00:00.000000000 -0800 @@ -1,177 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id: dapl_psp_create_any.c 2462 2005-05-24 02:28:24Z jlentini $ - */ - -#include "dapl.h" -#include "dapl_sp_util.h" -#include "dapl_ia_util.h" -#include "dapl_openib_util.h" - -/* - * dapl_psp_create_any - * - * Create a persistent Public Service Point that can recieve multiple - * requests for connections and generate multiple connection request - * instances that wil be delivered to the specified Event Dispatcher - * in a notification event. Differs from dapl_psp_create() in that - * the conn_qual is selected by the implementation and returned to - * the user. - * - * Input: - * ia_handle - * evd_handle - * psp_flags - * - * Output: - * conn_qual - * psp_handle - * - * Returns: - * DAT_SUCCESS - * DAT_INSUFFICIENT_RESOURCES - * DAT_INVALID_HANDLE - * DAT_INVALID_PARAMETER - * DAT_CONN_QUAL_IN_USE - * DAT_MODEL_NOT_SUPPORTED - */ -u32 dapl_psp_create_any(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL *conn_qual, - DAT_EVD_HANDLE evd_handle, enum dat_psp_flags psp_flags, - DAT_PSP_HANDLE *psp_handle) -{ - static DAT_CONN_QUAL hint_conn_qual = 1000; /* seed value */ - struct dapl_ia *ia_ptr; - struct dapl_sp *sp_ptr; - struct dapl_evd *evd_ptr; - u32 status = DAT_SUCCESS; - int i; - - ia_ptr = (struct dapl_ia *)ia_handle; - - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); - goto bail; - } - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { - status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); - goto bail; - } - - if (psp_handle == NULL) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); - goto bail; - } - if (conn_qual == NULL) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); - goto bail; - } - - evd_ptr = (struct dapl_evd *)evd_handle; - if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { - status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); - goto bail; - } - - if (psp_flags != DAT_PSP_CONSUMER_FLAG && - psp_flags != DAT_PSP_PROVIDER_FLAG) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); - goto bail; - } - - /* Allocate PSP */ - sp_ptr = dapl_sp_alloc(ia_ptr, TRUE); - if (sp_ptr == NULL) { - status = - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); - goto bail; - } - - /* - * Fill out the args for a PSP - */ - sp_ptr->ia_handle = ia_handle; - sp_ptr->evd_handle = evd_handle; - sp_ptr->psp_flags = psp_flags; - sp_ptr->ep_handle = NULL; - - /* - * Take a reference on the EVD handle - */ - atomic_inc(&evd_ptr->evd_ref_count); - - /* Link it onto the IA */ - dapl_ia_link_psp(ia_ptr, sp_ptr); - - /* - * Set up a listener for a connection. Connections can arrive - * even before this call returns! - */ - sp_ptr->state = DAPL_SP_STATE_PSP_LISTENING; - sp_ptr->listening = TRUE; - - /* - * If we have a big number of tries and we still haven't - * found a service_ID we can use, bail out with an error, - * something is wrong! - */ - for (i = 0, sp_ptr->conn_qual = hint_conn_qual; i < 100000; i++) { - - status = dapl_ib_setup_conn_listener(ia_ptr, sp_ptr); - - if (DAT_SUCCESS == status) - break; - else - sp_ptr->conn_qual++; - } - hint_conn_qual = sp_ptr->conn_qual + 1; - - if (status != DAT_SUCCESS) { - atomic_dec(&evd_ptr->evd_ref_count); - sp_ptr->evd_handle = NULL; - dapl_ia_unlink_sp(ia_ptr, sp_ptr); - dapl_sp_dealloc(sp_ptr); - - dapl_dbg_log(DAPL_DBG_TYPE_ERR, - "dapl_psp_create cannot set up conn listener: %x\n", - status); - - goto bail; - } - - /* - * Return handle to the user - */ - *conn_qual = sp_ptr->conn_qual; - *psp_handle = (DAT_PSP_HANDLE) sp_ptr; - - bail: - return status; -} diff -Nurp -X /home/tduffy/dontdiff linux-kernel-rsp/dat-provider/dapl_psp_create.c linux-kernel-psp/dat-provider/dapl_psp_create.c --- linux-kernel-rsp/dat-provider/dapl_psp_create.c 2005-05-23 22:21:46.086003000 -0700 +++ linux-kernel-psp/dat-provider/dapl_psp_create.c 1969-12-31 16:00:00.000000000 -0800 @@ -1,181 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id: dapl_psp_create.c 2462 2005-05-24 02:28:24Z jlentini $ - */ - -#include "dapl.h" -#include "dapl_sp_util.h" -#include "dapl_ia_util.h" -#include "dapl_openib_util.h" - -/* - * dapl_psp_create - * - * Create a persistent Public Service Point that can recieve multiple - * requests for connections and generate multiple connection request - * instances that wil be delivered to the specified Event Dispatcher - * in a notification event. - * - * Input: - * ia_handle - * conn_qual - * evd_handle - * psp_flags - * - * Output: - * psp_handle - * - * Returns: - * DAT_SUCCESS - * DAT_INSUFFICIENT_RESOURCES - * DAT_INVALID_PARAMETER - * DAT_CONN_QUAL_IN_USE - * DAT_MODEL_NOT_SUPPORTED - */ -u32 dapl_psp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, - DAT_EVD_HANDLE evd_handle, enum dat_psp_flags psp_flags, - DAT_PSP_HANDLE *psp_handle) -{ - struct dapl_ia *ia_ptr; - struct dapl_sp *sp_ptr; - struct dapl_evd *evd_ptr; - boolean_t sp_found; - u32 dat_status = DAT_SUCCESS; - - ia_ptr = (struct dapl_ia *)ia_handle; - - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); - goto bail; - } - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); - goto bail; - } - - if (psp_handle == NULL) { - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); - goto bail; - } - - evd_ptr = (struct dapl_evd *)evd_handle; - if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); - goto bail; - } - - if (psp_flags != DAT_PSP_CONSUMER_FLAG && - psp_flags != DAT_PSP_PROVIDER_FLAG) { - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); - goto bail; - } - - /* - * See if we have a quiescent listener to use for this PSP, else - * create one and set it listening - */ - sp_ptr = dapl_ia_sp_search(ia_ptr, conn_qual, TRUE); - sp_found = TRUE; - if (sp_ptr == NULL) { - /* Allocate PSP */ - sp_found = FALSE; - sp_ptr = dapl_sp_alloc(ia_ptr, TRUE); - if (sp_ptr == NULL) { - dat_status = - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, - DAT_RESOURCE_MEMORY); - goto bail; - } - } else if (sp_ptr->listening == TRUE) { - dat_status = DAT_ERROR(DAT_CONN_QUAL_IN_USE, 0); - goto bail; - } - - /* - * Fill out the args for a PSP - */ - sp_ptr->ia_handle = ia_handle; - sp_ptr->conn_qual = conn_qual; - sp_ptr->evd_handle = evd_handle; - sp_ptr->psp_flags = psp_flags; - sp_ptr->ep_handle = NULL; - - /* - * Take a reference on the EVD handle - */ - atomic_inc(&evd_ptr->evd_ref_count); - - /* - * Set up a listener for a connection. Connections can arrive - * even before this call returns! - */ - sp_ptr->state = DAPL_SP_STATE_PSP_LISTENING; - sp_ptr->listening = TRUE; - - /* - * If this is a new sp we need to add it to the IA queue, and set up - * a conn_listener. - */ - if (sp_found == FALSE) { - /* Link it onto the IA before enabling it to receive conn - * requests - */ - dapl_ia_link_psp(ia_ptr, sp_ptr); - - dat_status = dapl_ib_setup_conn_listener(ia_ptr, sp_ptr); - - if (dat_status != DAT_SUCCESS) { - /* - * Have a problem setting up the connection, something - * wrong! Decrements the EVD refcount & release it. - */ - atomic_dec(&evd_ptr->evd_ref_count); - sp_ptr->evd_handle = NULL; - dapl_ia_unlink_sp(ia_ptr, sp_ptr); - dapl_sp_dealloc(sp_ptr); - - dapl_dbg_log(DAPL_DBG_TYPE_CM, "dapl_psp_create " - "setup_conn_listener failed: %x\n", - dat_status); - - goto bail; - } - } - - /* - * Return handle to the user - */ - *psp_handle = (DAT_PSP_HANDLE) sp_ptr; - - bail: - return dat_status; -} diff -Nurp -X /home/tduffy/dontdiff linux-kernel-rsp/dat-provider/dapl_psp_free.c linux-kernel-psp/dat-provider/dapl_psp_free.c --- linux-kernel-rsp/dat-provider/dapl_psp_free.c 2005-05-20 22:55:17.285001000 -0700 +++ linux-kernel-psp/dat-provider/dapl_psp_free.c 1969-12-31 16:00:00.000000000 -0800 @@ -1,131 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id: dapl_psp_free.c 2433 2005-05-21 04:11:03Z jlentini $ - */ - -#include "dapl.h" -#include "dapl_sp_util.h" -#include "dapl_ia_util.h" -#include "dapl_openib_util.h" - -/* - * dapl_psp_free - * - * Destroy a specific instance of a Service Point. - * - * Input: - * psp_handle - * - * Output: - * none - * - * Returns: - * DAT_SUCCESS - * DAT_INVALID_PARAMETER - */ -u32 dapl_psp_free(DAT_PSP_HANDLE psp_handle) -{ - struct dapl_ia *ia_ptr; - struct dapl_sp *sp_ptr; - DAPL_SP_STATE save_state; - u32 dat_status = DAT_SUCCESS; - - sp_ptr = (struct dapl_sp *)psp_handle; - /* - * Verify handle - */ - dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_psp_free %p\n", psp_handle); - - if (DAPL_BAD_HANDLE(sp_ptr, DAPL_MAGIC_PSP)) { - dat_status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PSP); - goto bail; - } - - /* ia_ptr = (struct dapl_ia *)sp_ptr->header.owner_ia; */ - ia_ptr = sp_ptr->header.owner_ia; - /* - * Remove the connection listener if it has been established - * and there are no current connections in progress. - * If we defer removing the sp it becomes something of a zombie - * container until the last connection is disconnected, after - * which it will be cleaned up. - */ - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); - - sp_ptr->listening = FALSE; - - /* Release reference on EVD. If an error was encountered in a previous - * free the evd_handle will be NULL - */ - if (sp_ptr->evd_handle) { - atomic_dec(&((struct dapl_evd *)sp_ptr->evd_handle)-> - evd_ref_count); - sp_ptr->evd_handle = NULL; - } - - /* - * Release the base resource if there are no outstanding - * connections; else the last disconnect on this PSP will free it - * up. The PSP is used to contain CR records for each connection, - * which contain information necessary to disconnect. - */ - dapl_dbg_log(DAPL_DBG_TYPE_CM, - ">>> dapl_psp_free: state %d cr_list_count %d\n", - sp_ptr->state, sp_ptr->cr_list_count); - if ((sp_ptr->state == DAPL_SP_STATE_PSP_LISTENING || - sp_ptr->state == DAPL_SP_STATE_PSP_PENDING) && - sp_ptr->cr_list_count == 0) { - save_state = sp_ptr->state; - sp_ptr->state = DAPL_SP_STATE_FREE; - spin_unlock_irqrestore(&sp_ptr->header.lock, - sp_ptr->header.flags); - - dat_status = dapl_ib_remove_conn_listener(ia_ptr, sp_ptr); - if (dat_status != DAT_SUCCESS) { - /* revert to entry state on error */ - sp_ptr->state = save_state; - goto bail; - } - dapl_ia_unlink_sp(ia_ptr, sp_ptr); - dapl_sp_dealloc(sp_ptr); - } else { - /* The PSP is now in the pending state, where it will sit until - * the last connection terminates or the app uses the same - * ServiceID again, which will reactivate it. - */ - sp_ptr->state = DAPL_SP_STATE_PSP_PENDING; - spin_unlock_irqrestore(&sp_ptr->header.lock, sp_ptr->header.flags); - dapl_dbg_log(DAPL_DBG_TYPE_CM, - ">>> dapl_psp_free: PSP PENDING\n"); - } - - bail: - return dat_status; -} diff -Nurp -X /home/tduffy/dontdiff linux-kernel-rsp/dat-provider/dapl_psp_query.c linux-kernel-psp/dat-provider/dapl_psp_query.c --- linux-kernel-rsp/dat-provider/dapl_psp_query.c 2005-05-20 22:55:18.853022000 -0700 +++ linux-kernel-psp/dat-provider/dapl_psp_query.c 1969-12-31 16:00:00.000000000 -0800 @@ -1,62 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id: dapl_psp_query.c 2433 2005-05-21 04:11:03Z jlentini $ - */ - -#include "dapl.h" - -u32 dapl_psp_query(DAT_PSP_HANDLE psp_handle, struct dat_psp_param *psp_param) -{ - struct dapl_sp *sp_ptr; - u32 status; - - if (DAPL_BAD_HANDLE(psp_handle, DAPL_MAGIC_PSP) || - ((struct dapl_sp *)psp_handle)->listening != TRUE) { - status = - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PSP); - goto bail; - } - - if (NULL == psp_param) { - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); - goto bail; - } - - sp_ptr = (struct dapl_sp *)psp_handle; - - psp_param->ia_handle = sp_ptr->ia_handle; - psp_param->conn_qual = sp_ptr->conn_qual; - psp_param->evd_handle = sp_ptr->evd_handle; - psp_param->psp_flags = sp_ptr->psp_flags; - - status = DAT_SUCCESS; - -bail: - return status; -} diff -Nurp -X /home/tduffy/dontdiff linux-kernel-rsp/dat-provider/Makefile linux-kernel-psp/dat-provider/Makefile --- linux-kernel-rsp/dat-provider/Makefile 2005-05-25 15:01:00.683014000 -0700 +++ linux-kernel-psp/dat-provider/Makefile 2005-05-25 15:12:34.054000000 -0700 @@ -70,10 +70,7 @@ PROVIDER_MODULES := \ dapl_mr_util \ dapl_provider \ dapl_sp_util \ - dapl_psp_create \ - dapl_psp_create_any \ - dapl_psp_free \ - dapl_psp_query \ + dapl_psp \ dapl_pz \ dapl_ring_buffer_util \ dapl_rmr \ diff -Nurp -X /home/tduffy/dontdiff linux-kernel-rsp/patches/alt_dat_provider_makefile linux-kernel-psp/patches/alt_dat_provider_makefile --- linux-kernel-rsp/patches/alt_dat_provider_makefile 2005-05-25 14:53:43.453015000 -0700 +++ linux-kernel-psp/patches/alt_dat_provider_makefile 2005-05-25 15:05:08.065003000 -0700 @@ -64,10 +64,7 @@ PROVIDER_MODULES := \ dapl_mr_util \ dapl_provider \ dapl_sp_util \ - dapl_psp_create \ - dapl_psp_create_any \ - dapl_psp_free \ - dapl_psp_query \ + dapl_psp \ dapl_pz \ dapl_ring_buffer_util \ dapl_rmr \ From roland at topspin.com Thu May 26 12:09:22 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 26 May 2005 12:09:22 -0700 Subject: [openib-general] Re: registering read-only memory In-Reply-To: <506C3D7B14CDD411A52C00025558DED605B1A87D@mtlex01.yok.mtl.com> (Eli Cohen's message of "Wed, 25 May 2005 09:54:00 +0300") References: <506C3D7B14CDD411A52C00025558DED605B1A87D@mtlex01.yok.mtl.com> Message-ID: <52hdgpkdnh.fsf@topspin.com> OK, I think I understand the problem. The compiler puts a string defined with "const char *foo" on a read-only page, but in uverbs_mem.c we always pass write=1 to get_user_pages(). I need to fix the code so that it only asks for writeable pages if we're registering with writable permissions. - R. From roland at topspin.com Thu May 26 12:23:05 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 26 May 2005 12:23:05 -0700 Subject: [openib-general] Re: registering read-only memory In-Reply-To: <52hdgpkdnh.fsf@topspin.com> (Roland Dreier's message of "Thu, 26 May 2005 12:09:22 -0700") References: <506C3D7B14CDD411A52C00025558DED605B1A87D@mtlex01.yok.mtl.com> <52hdgpkdnh.fsf@topspin.com> Message-ID: <52acmhkd0m.fsf@topspin.com> I just checked in a fix for this. I'm now able to register read-only memory (as long as the permissions on the registration are read-only). - R. From eli at mellanox.co.il Thu May 26 12:23:55 2005 From: eli at mellanox.co.il (Eli Cohen) Date: Thu, 26 May 2005 22:23:55 +0300 Subject: [openib-general] RE: registering read-only memory Message-ID: <506C3D7B14CDD411A52C00025558DED605B1A88A@mtlex01.yok.mtl.com> I'm afraid that won't solve the problem. From some reason the vma of such a region is flagged with VM_IO and that's why get_user_pages() will fail. We need to find another way to get the physical address of such memory. One thing I can think of is implementing something similar to follow_page(). Or better if the kernel will export follow_page(). -----Original Message----- From: Roland Dreier [mailto:roland at topspin.com] Sent: Thursday, May 26, 2005 10:09 PM To: Eli Cohen Cc: Michael S. Tsirkin; openib-general at openib.org Subject: Re: registering read-only memory OK, I think I understand the problem. The compiler puts a string defined with "const char *foo" on a read-only page, but in uverbs_mem.c we always pass write=1 to get_user_pages(). I need to fix the code so that it only asks for writeable pages if we're registering with writable permissions. - R. -------------- next part -------------- An HTML attachment was scrubbed... URL: From roland at topspin.com Thu May 26 12:28:40 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 26 May 2005 12:28:40 -0700 Subject: [openib-general] [PATCH] mthca, ibverbs userspace readability In-Reply-To: <20050526165819.GO15688@aon.at> (Bernhard Fischer's message of "Thu, 26 May 2005 18:58:19 +0200") References: <20050526161455.GL15688@aon.at> <52ekbulzmq.fsf@topspin.com> <20050526165819.GO15688@aon.at> Message-ID: <523bs9kcrb.fsf@topspin.com> Roland> This patch seems to consist mostly of replacing tabs with Roland> 8 spaces. Why is that an improvement for readability? Bernhard> It does indeed. But with the patch, it's easier to read Bernhard> and code for. Also, there are a couple of fixes for Bernhard> super-long lines, which are extremely disturbing if you Bernhard> happen to code userspace while referencing those headers Bernhard> :-/ I'm not going to pursue a tabs vs. spaces argument any further. For any code that I maintain, we're going to continue to indent using tabs. If you can break out the other changes, I'll take a look. However, I'm not that interested in breaking 85-character lines into two lines-- to me saving vertical space is more useful than obeying a strict 80-column limit. - R. From roland at topspin.com Thu May 26 14:02:02 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 26 May 2005 14:02:02 -0700 Subject: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <52is15lxw0.fsf@topspin.com> (Roland Dreier's message of "Thu, 26 May 2005 10:06:55 -0700") References: <5264x6lzhc.fsf@topspin.com> <42960099.3000503@ichips.intel.com> <52is15lxw0.fsf@topspin.com> Message-ID: <52wtplitv9.fsf@topspin.com> Roland> Yes, that is probably a better way of saying what I was Roland> trying to drive at. There's a significant portion of Roland> kDAPL that doesn't seem necessary or useful to me. For Roland> example, why do we need a new API for posting work Roland> requests? Why do we need the ability to combine Roland> completion events and CM events into a single queue? By the way, just to be clear, I definitely do support having library code that makes it easy to perform operations such as, for example, "connect this QP to such-and-such IP address." This would be useful for SDP, NFS/RDMA and iSER, which all should be able to have common code that runs on top of both InfiniBand and iWARP. - R. From roland at topspin.com Thu May 26 14:11:44 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 26 May 2005 14:11:44 -0700 Subject: [openib-general] [PATCH] mthca: cosmetic fix for handle_error_cqe() In-Reply-To: <20050526132815.GH15688@aon.at> (Bernhard Fischer's message of "Thu, 26 May 2005 15:28:15 +0200") References: <20050526132815.GH15688@aon.at> Message-ID: <52hdgpitf3.fsf@topspin.com> Thanks, I cleaned this up even more so we only dump error CQEs for local QP operation errors (we don't really need the see the CQE for other error completions). - R. From roland at topspin.com Thu May 26 14:12:24 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 26 May 2005 14:12:24 -0700 Subject: [openib-general] Re: registering read-only memory In-Reply-To: <506C3D7B14CDD411A52C00025558DED605B1A88A@mtlex01.yok.mtl.com> (Eli Cohen's message of "Thu, 26 May 2005 22:23:55 +0300") References: <506C3D7B14CDD411A52C00025558DED605B1A88A@mtlex01.yok.mtl.com> Message-ID: <52d5rditdz.fsf@topspin.com> Eli> I'm afraid that won't solve the problem. From some reason the Eli> vma of such a region is flagged with VM_IO and that's why Eli> get_user_pages() will fail. Are you sure? It seems to work for me if I just pass write=0 to get_user_pages(). - R. From jcarr at linuxmachines.com Thu May 26 14:17:07 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Thu, 26 May 2005 14:17:07 -0700 Subject: [openib-general] QP sq->tail always = 0 Message-ID: <42963CD3.7040901@linuxmachines.com> In my code (based on cmtest) I was using a single CQ for both send and recv. I broke them into seperate queues to try to figure out why I could only do N transfers for each CQ that I created. I added some debugging output to mthca_cq.c to dump out the SQ and RQ for each QP. When I run my test, the output for the server and client look similar. For the server, it first runs: ib_post_recv() for N times (N=5 here) ib_mthca 0000:09:00.0: SQ 050406 (1 head, 0 tail, 5 max) ib_mthca 0000:09:00.0: RQ 050406 (5 head, 0 tail, 5 max) ib_mthca 0000:09:00.0: SQ 050406 (2 head, 0 tail, 5 max) ib_mthca 0000:09:00.0: RQ 050406 (5 head, 0 tail, 5 max) ib_mthca 0000:09:00.0: SQ 050406 (3 head, 0 tail, 5 max) ib_mthca 0000:09:00.0: RQ 050406 (5 head, 0 tail, 5 max) ib_mthca 0000:09:00.0: SQ 050406 (4 head, 0 tail, 5 max) ib_mthca 0000:09:00.0: RQ 050406 (5 head, 0 tail, 5 max) ib_mthca 0000:09:00.0: SQ 050406 (5 head, 0 tail, 5 max) ib_mthca 0000:09:00.0: RQ 050406 (5 head, 0 tail, 5 max) Then, the client sends N=5 responses via ib_post_recv() that the server sees via ib_poll_cq(). ib_mthca 0000:09:00.0: SQ 050406 (5 head, 0 tail, 5 max) ib_mthca 0000:09:00.0: RQ 050406 (5 head, 0 tail, 5 max) ib_mthca 0000:09:00.0: SQ 050406 (5 head, 0 tail, 5 max) ib_mthca 0000:09:00.0: RQ 050406 (5 head, 1 tail, 5 max) ib_mthca 0000:09:00.0: SQ 050406 (5 head, 0 tail, 5 max) ib_mthca 0000:09:00.0: RQ 050406 (5 head, 2 tail, 5 max) ib_mthca 0000:09:00.0: SQ 050406 (5 head, 0 tail, 5 max) ib_mthca 0000:09:00.0: RQ 050406 (5 head, 3 tail, 5 max) ib_mthca 0000:09:00.0: SQ 050406 (5 head, 0 tail, 5 max) ib_mthca 0000:09:00.0: RQ 050406 (5 head, 4 tail, 5 max) ib_mthca 0000:09:00.0: RQ 050406 (5 head, 5 tail, 5 max ib_mthca 0000:09:00.0: SQ 050406 full (5 head, 0 tail, 5 max, 0 nreq) At this point, the RQ is empty: (head == tail) And at this point the SQ is full: ((head - tail) == max) Is this a correct understanding? In both the server & client cases, the SQ always remains full. How are the connections supposed to be tagged as completed or pulled off of the queue? Thanks for any help, Jeff PS: Some simple routines for debugging might be helpful for new users of infiniband. -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: mthca_qp.debug.diff URL: From jcarr at linuxmachines.com Thu May 26 14:27:23 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Thu, 26 May 2005 14:27:23 -0700 Subject: [openib-general] QP sq->tail always = 0 In-Reply-To: <42963CD3.7040901@linuxmachines.com> References: <42963CD3.7040901@linuxmachines.com> Message-ID: <42963F3B.7050008@linuxmachines.com> On 05/26/05 14:17, Jeff Carr wrote: > When I run my test, the output for the server and client look similar. > > For the server, it first runs: ib_post_recv() for N times (N=5 here) sorry s/ib_post_recv/ib_post_send/ > ib_mthca 0000:09:00.0: SQ 050406 (1 head, 0 tail, 5 max) > ib_mthca 0000:09:00.0: RQ 050406 (5 head, 0 tail, 5 max) > ib_mthca 0000:09:00.0: SQ 050406 (2 head, 0 tail, 5 max) > ib_mthca 0000:09:00.0: RQ 050406 (5 head, 0 tail, 5 max) > ib_mthca 0000:09:00.0: SQ 050406 (3 head, 0 tail, 5 max) > ib_mthca 0000:09:00.0: RQ 050406 (5 head, 0 tail, 5 max) > ib_mthca 0000:09:00.0: SQ 050406 (4 head, 0 tail, 5 max) > ib_mthca 0000:09:00.0: RQ 050406 (5 head, 0 tail, 5 max) > ib_mthca 0000:09:00.0: SQ 050406 (5 head, 0 tail, 5 max) > ib_mthca 0000:09:00.0: RQ 050406 (5 head, 0 tail, 5 max) > > Then, the client sends N=5 responses via ib_post_recv() that the server > sees via ib_poll_cq(). Then the server runs ib_poll_cq() to recieve the responses from the client that the client sends vi ib_post_send(). ib_post_recv runs on both the client and server N times before any transfers begain. > > ib_mthca 0000:09:00.0: SQ 050406 (5 head, 0 tail, 5 max) > ib_mthca 0000:09:00.0: RQ 050406 (5 head, 0 tail, 5 max) > ib_mthca 0000:09:00.0: SQ 050406 (5 head, 0 tail, 5 max) > ib_mthca 0000:09:00.0: RQ 050406 (5 head, 1 tail, 5 max) > ib_mthca 0000:09:00.0: SQ 050406 (5 head, 0 tail, 5 max) > ib_mthca 0000:09:00.0: RQ 050406 (5 head, 2 tail, 5 max) > ib_mthca 0000:09:00.0: SQ 050406 (5 head, 0 tail, 5 max) > ib_mthca 0000:09:00.0: RQ 050406 (5 head, 3 tail, 5 max) > ib_mthca 0000:09:00.0: SQ 050406 (5 head, 0 tail, 5 max) > ib_mthca 0000:09:00.0: RQ 050406 (5 head, 4 tail, 5 max) > > ib_mthca 0000:09:00.0: RQ 050406 (5 head, 5 tail, 5 max > ib_mthca 0000:09:00.0: SQ 050406 full (5 head, 0 tail, 5 max, 0 nreq) > > At this point, the RQ is empty: (head == tail) > And at this point the SQ is full: ((head - tail) == max) Regards, Jeff From roland at topspin.com Thu May 26 14:28:47 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 26 May 2005 14:28:47 -0700 Subject: [openib-general] QP sq->tail always = 0 In-Reply-To: <42963CD3.7040901@linuxmachines.com> (Jeff Carr's message of "Thu, 26 May 2005 14:17:07 -0700") References: <42963CD3.7040901@linuxmachines.com> Message-ID: <52zmuhhe28.fsf@topspin.com> Jeff> For the server, it first runs: ib_post_recv() for N times Jeff> (N=5 here) Jeff> ib_mthca 0000:09:00.0: SQ 050406 (1 head, 0 tail, 5 max) Jeff> ib_mthca 0000:09:00.0: RQ 050406 (5 head, 0 tail, 5 max) Jeff> ib_mthca 0000:09:00.0: SQ 050406 (2 head, 0 tail, 5 max) Jeff> ib_mthca 0000:09:00.0: RQ 050406 (5 head, 0 tail, 5 max) Jeff> ib_mthca 0000:09:00.0: SQ 050406 (3 head, 0 tail, 5 max) Jeff> ib_mthca 0000:09:00.0: RQ 050406 (5 head, 0 tail, 5 max) Jeff> ib_mthca 0000:09:00.0: SQ 050406 (4 head, 0 tail, 5 max) Jeff> ib_mthca 0000:09:00.0: RQ 050406 (5 head, 0 tail, 5 max) Jeff> ib_mthca 0000:09:00.0: SQ 050406 (5 head, 0 tail, 5 max) Jeff> ib_mthca 0000:09:00.0: RQ 050406 (5 head, 0 tail, 5 max) This looks like it has actually already posted 5 receives and is posting 5 sends one by one. Did you mean ib_post_send() above? Jeff> Then, the client sends N=5 responses via ib_post_recv() that Jeff> the server sees via ib_poll_cq(). Again, did you mean ib_post_send()? Jeff> ib_mthca 0000:09:00.0: SQ 050406 (5 head, 0 tail, 5 max) Jeff> ib_mthca 0000:09:00.0: RQ 050406 (5 head, 0 tail, 5 max) Jeff> ib_mthca 0000:09:00.0: SQ 050406 (5 head, 0 tail, 5 max) Jeff> ib_mthca 0000:09:00.0: RQ 050406 (5 head, 1 tail, 5 max) Jeff> ib_mthca 0000:09:00.0: SQ 050406 (5 head, 0 tail, 5 max) Jeff> ib_mthca 0000:09:00.0: RQ 050406 (5 head, 2 tail, 5 max) Jeff> ib_mthca 0000:09:00.0: SQ 050406 (5 head, 0 tail, 5 max) Jeff> ib_mthca 0000:09:00.0: RQ 050406 (5 head, 3 tail, 5 max) Jeff> ib_mthca 0000:09:00.0: SQ 050406 (5 head, 0 tail, 5 max) Jeff> ib_mthca 0000:09:00.0: RQ 050406 (5 head, 4 tail, 5 max) Jeff> ib_mthca 0000:09:00.0: RQ 050406 (5 head, 5 tail, 5 max Jeff> ib_mthca 0000:09:00.0: SQ 050406 full (5 head, 0 tail, 5 Jeff> max, 0 nreq) Jeff> At this point, the RQ is empty: (head == tail) And at this Jeff> point the SQ is full: ((head - tail) == max) Jeff> Is this a correct understanding? It looks that way -- you seem to have posted 5 sends and 5 receives, and you are getting 5 receive completions, which empties the receive queue. Are you seeing any send completions on the CQ attached to the send queue? One possibility is that you're not generating any send completions because you created your CQ with a sq_sig_type of IB_SIGNAL_REQ_WR and then posting send requests without the IB_SEND_SIGNALED flag. (The IB_SIGNAL_REQ_WR flag means that not all send requests posted to the send queue will generate a completion -- only those marked with the IB_SEND_SIGNALED flag. However, the driver can't free a send request from the send queue until it knows it has completed, and the only way for the driver to know that is to see a completion for the given request or a later request. Requests on a queue always complete in order, so if a later request completes and generates a completion, the driver can also free any earlier unsignaled requests) - R. From mshefty at ichips.intel.com Thu May 26 14:35:14 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Thu, 26 May 2005 14:35:14 -0700 Subject: [openib-general] QP sq->tail always = 0 In-Reply-To: <52zmuhhe28.fsf@topspin.com> References: <42963CD3.7040901@linuxmachines.com> <52zmuhhe28.fsf@topspin.com> Message-ID: <42964112.9040103@ichips.intel.com> Roland Dreier wrote: > Are you seeing any send completions on the CQ attached to the send > queue? One possibility is that you're not generating any send > completions because you created your CQ with a sq_sig_type of > IB_SIGNAL_REQ_WR and then posting send requests without the > IB_SEND_SIGNALED flag. > > (The IB_SIGNAL_REQ_WR flag means that not all send requests posted to > the send queue will generate a completion -- only those marked with > the IB_SEND_SIGNALED flag. However, the driver can't free a send > request from the send queue until it knows it has completed, and the > only way for the driver to know that is to see a completion for the > given request or a later request. Requests on a queue always complete > in order, so if a later request completes and generates a completion, > the driver can also free any earlier unsignaled requests) If the code is based on cmpost, this is likely the case. Cmpost creates the QP with IB_SIGNAL_REQ_WR, but does not set the IB_SEND_SIGNALED flag. Since people are refering to this as a sample, I will look at changing the code to avoid confusing people. - Sean From jcarr at linuxmachines.com Thu May 26 15:39:51 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Thu, 26 May 2005 15:39:51 -0700 Subject: [openib-general] QP sq->tail always = 0 In-Reply-To: <42964112.9040103@ichips.intel.com> References: <42963CD3.7040901@linuxmachines.com> <52zmuhhe28.fsf@topspin.com> <42964112.9040103@ichips.intel.com> Message-ID: <42965037.4010104@linuxmachines.com> On 05/26/05 14:35, Sean Hefty wrote: > Roland Dreier wrote: > >> Are you seeing any send completions on the CQ attached to the send >> queue? No. >> One possibility is that you're not generating any send >> completions because you created your CQ with a sq_sig_type of >> IB_SIGNAL_REQ_WR and then posting send requests without the >> IB_SEND_SIGNALED flag. Well, you seem to have hit the nail on the head. >> (The IB_SIGNAL_REQ_WR flag means that not all send requests posted to >> the send queue will generate a completion -- only those marked with >> the IB_SEND_SIGNALED flag. However, the driver can't free a send >> request from the send queue until it knows it has completed, and the >> only way for the driver to know that is to see a completion for the >> given request or a later request. Requests on a queue always complete >> in order, so if a later request completes and generates a completion, >> the driver can also free any earlier unsignaled requests) Thanks very much for this description. It's certainly not trivial to understand such relationships. The IB spec 11.2.4.1 has: · The Signaling Type must be specified for the Send Queue on this QP. The valid types are: · Non-selectable: All Work Requests submitted to the Send Queue always generate a completion entry. · Selectable: Consumer must specify on each Work Request submitted to the Send Queue whether to generate a completion entry for successful completions. So, this sounds like the same thing and will read more from there. Thanks for sharing your understanding of IB. > > If the code is based on cmpost, this is likely the case. Cmpost creates > the QP with IB_SIGNAL_REQ_WR, but does not set the IB_SEND_SIGNALED flag. > > Since people are refering to this as a sample, I will look at changing > the code to avoid confusing people. Thanks; I'll try it out also and will let you know how it goes. Jeff BTW, IB_SIGNAL_REQ_WR is only referenced 3 times in the kernel without any comments. Boy it would be nice if some of the people that spent so much time on the IB arch spec spent some time submitting comments as patches to the Linux kernel sources. That would be a dream. drivers/infiniband/ulp/sdp/sdp_conn.c: init_attr->sq_sig_type = IB_SIGNAL_REQ_WR; drivers/infiniband/include/ib_verbs.h: IB_SIGNAL_REQ_WR drivers/infiniband/core/uverbs_cmd.c: attr.sq_sig_type = cmd.sq_sig_all ? IB_SIGNAL_ALL_WR : IB_SIGNAL_REQ_WR; From chacker at fadmail.com Thu May 26 16:47:07 2005 From: chacker at fadmail.com (Ophelia Gabriel) Date: Thu, 26 May 2005 22:47:07 -0100 Subject: [openib-general] Instant low rates Message-ID: Hello, We tried contacting you awhile ago about your low interest morta(ge rate. You have qualified for the lowest rate in years... You could get over $380,000 for as little as $500 a month! Ba(d credit? Doesn't matter, low rates are fixed no matter what! To get a free, no obli,gation consultation click below: http://www.pr1ces.com/sign.asp Best Regards, Shauna Carpenter to be remov(ed: http://www.pr1ces.com/gone.asp this process takes one week, so please be patient. we do our best to take your email/s off but you have to fill out a rem/ove or else you will continue to recieve email/s. From jcarr at linuxmachines.com Thu May 26 15:56:03 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Thu, 26 May 2005 15:56:03 -0700 Subject: [openib-general] QP sq->tail always = 0 In-Reply-To: <42964112.9040103@ichips.intel.com> References: <42963CD3.7040901@linuxmachines.com> <52zmuhhe28.fsf@topspin.com> <42964112.9040103@ichips.intel.com> Message-ID: <42965403.7040400@linuxmachines.com> On 05/26/05 14:35, Sean Hefty wrote: > If the code is based on cmpost, this is likely the case. Cmpost creates > the QP with IB_SIGNAL_REQ_WR, but does not set the IB_SEND_SIGNALED flag. > > Since people are refering to this as a sample, I will look at changing > the code to avoid confusing people. Another thing I noticed that seemed very strange to me, is that when I went back to test this problem with cmpost, the debugging output was different for the SQ and RQ. cmpost.c has: qp_attr->send_cq = node->cq; qp_attr->recv_cq = node->cq; node->qp = ib_create_qp(test.pd, qp_attr); But, I would get: ib_poll_cq == 1 ib_mthca 0000:09:00.0: SQ 030406 (0 head, 0 tail, 5 max) ib_mthca 0000:09:00.0: RQ 030406 (5 head, 2 tail, 5 max SQ->head != RQ->head So this is kind of strange to me. I would think that the head & tail of the SQ and RQ should be the same. Maybe making the send and receive queue the same doesn't work correctly yet? Jeff From robert.j.woodruff at intel.com Thu May 26 15:56:46 2005 From: robert.j.woodruff at intel.com (Bob Woodruff) Date: Thu, 26 May 2005 15:56:46 -0700 Subject: [openib-general] [ANNOUNCE][PATCH] New Linux 2.6.9 backport patches and corresponding userspace tar ball available Message-ID: Hal asked if I could send out my latest backport patches for 2.6.9 kernels, so here they are. They are rather big, so I just put them into to SVN where people can get them. The latest ones I have are based on svn2425, but I will try to track the changes in SVN and provide newer updates periodically. Patches are located in the SVN tree under gen2/trunk/src/linux-kernel/patches/backport-to-2.6.9/ infiniband-backport-svn2425-to-2.6.9-kernel-fixups-01.diff infiniband-backport-svn2425-to-2.6.9-openib-drivers-02.diff infiniband-backport-svn2425-to-2.6.9-openib-fixups-03.diff infiniband-backport-svn2425-userspace.tar.gz woody From jcarr at linuxmachines.com Thu May 26 16:23:32 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Thu, 26 May 2005 16:23:32 -0700 Subject: [openib-general] OOPS: ib_mad crashery on bootup In-Reply-To: <4294B375.8070808@ichips.intel.com> References: <1116530995.980.9.camel@duffman> <4294B375.8070808@ichips.intel.com> Message-ID: <42965A74.6000309@linuxmachines.com> On 05/25/05 10:18, Sean Hefty wrote: > I've been able to hit an issue in the MAD layer that might be related. > I still do not know the root cause, however. Just to chim in with the same. I've been able to generate a potentially similar error on bootup. I put the openib svn on top of 2.6.12-rc4-git7. Jeff Configuring network interfaces...Unable to handle kernel NULL pointer dereference at virtual address 00000018 printing eip: f8a309f2 *pde = 00000000 Oops: 0000 [#1] PREEMPT SMP Modules linked in: ib_mthca i2c_i801 i2c_core uhci_hcd usbcore pci_hotplug parport_pc parport floppy evdev pcspkr ib_cm ib_ipoib ib_sa ib_mad ib_core dm_mod ide_cd cdrom rtc ide_disk ide_generic via82cxxx trm290 triflex slc90e66 sis5513 siimage serverworks sc1200 rz1000 pdc202xx_old pdc202xx_new opti621 ns87415 hpt366 hpt34x generic cy82c693 cs5530 cs5520 cmd64x atiixp amd74xx alim15x3 aec62xx unix xfs exportfs reiserfs ata_piix libata scsi_mod e1000 ext3 jbd mbcache piix ide_core CPU: 0 EIP: 0060:[] Not tainted VLI EFLAGS: 00010282 (2.6.12-rc4-git7-20050523) EIP is at ib_mad_send_done_handler+0x12/0x130 [ib_mad] eax: f7a73f08 ebx: f74b8680 ecx: 00000001 edx: 00000001 esi: 00000000 edi: df84d874 ebp: f74b8680 esp: f7a73ed0 ds: 007b es: 007b ss: 0068 Process ib_mad2 (pid: 2773, threadinfo=f7a72000 task=f7836540) Stack: df849800 81000001 00000000 00000001 00000001 df84d400 f7a73f08 df84d800 df84d874 df84d800 f8a30cec df84d800 f7a73f08 f7a73f08 f74b8680 00000000 00000000 00000000 0000000e c2010a14 dfd3a780 00000403 f7a73f8c 00000000 Call Trace: [] ib_mad_completion_handler+0x8c/0xb0 [ib_mad] [] worker_thread+0x1b9/0x260 [] ib_mad_completion_handler+0x0/0xb0 [ib_mad] [] default_wake_function+0x0/0x20 [] default_wake_function+0x0/0x20 [] worker_thread+0x0/0x260 [] kthread+0xba/0xc0 [] kthread+0x0/0xc0 [] kernel_thread_helper+0x5/0x10 Code: e8 44 34 00 00 89 44 24 08 48 0f 85 dc fe ff ff e9 28 ff ff ff 8d 74 26 00 55 57 56 53 83 ec 18 8b 44 24 30 8b 18 8b 73 08 89 dd <8b> 56 18 89 54 24 10 8d b4 26 00 00 00 00 89 f0 31 ff e8 b7 55 done. From mshefty at ichips.intel.com Thu May 26 16:28:10 2005 From: mshefty at ichips.intel.com (Sean Hefty) Date: Thu, 26 May 2005 16:28:10 -0700 Subject: [openib-general] OOPS: ib_mad crashery on bootup In-Reply-To: <42965A74.6000309@linuxmachines.com> References: <1116530995.980.9.camel@duffman> <4294B375.8070808@ichips.intel.com> <42965A74.6000309@linuxmachines.com> Message-ID: <42965B8A.90707@ichips.intel.com> Jeff Carr wrote: > Just to chim in with the same. I've been able to generate a potentially > similar error on bootup. I put the openib svn on top of 2.6.12-rc4-git7. Thanks for the info. I am looking into this. - Sean From ftillier at silverstorm.com Thu May 26 16:29:57 2005 From: ftillier at silverstorm.com (Fab Tillier) Date: Thu, 26 May 2005 16:29:57 -0700 Subject: [openib-general] QP sq->tail always = 0 In-Reply-To: <42965403.7040400@linuxmachines.com> Message-ID: <001d01c5624a$d4d46b40$9b5aa8c0@infiniconsys.com> > From: Jeff Carr [mailto:jcarr at linuxmachines.com] > Sent: Thursday, May 26, 2005 3:56 PM > > On 05/26/05 14:35, Sean Hefty wrote: > > If the code is based on cmpost, this is likely the case. Cmpost creates > > the QP with IB_SIGNAL_REQ_WR, but does not set the IB_SEND_SIGNALED flag. > > > > Since people are refering to this as a sample, I will look at changing > > the code to avoid confusing people. > > Another thing I noticed that seemed very strange to me, is that when I > went back to test this problem with cmpost, the debugging output was > different for the SQ and RQ. > > cmpost.c has: > qp_attr->send_cq = node->cq; > qp_attr->recv_cq = node->cq; > node->qp = ib_create_qp(test.pd, qp_attr); > > But, I would get: > > ib_poll_cq == 1 > ib_mthca 0000:09:00.0: SQ 030406 (0 head, 0 tail, 5 max) > ib_mthca 0000:09:00.0: RQ 030406 (5 head, 2 tail, 5 max > > SQ->head != RQ->head > > So this is kind of strange to me. I would think that the head & tail of > the SQ and RQ should be the same. Maybe making the send and receive > queue the same doesn't work correctly yet? You're confusing QP depth and CQ depth. The QP, being a pair, has both a send and receive queue. These have independent depths associated with them - that is, the RQ and SQ can have different maximums. Work requests are posted independently to these two queues. These queues are in effect sinks - you feed requests into them, and once posted you don't get any feedback from the QP related to their status. The CQ is the source of completion information - this is where you find out what completed. You can have a multitude of RQs and SQs associated with a single CQ. When a completion happens on a queue pair, the completion processing updates the indexes for the appropriate requests queue (SQ or RQ, depending on the completion). Because the RQ and SQ are independent, they have independent head and tails. The CQ is a single queue where all completions get reported in FIFO order, with no segregation based on the type of completion. That is, the CQ doesn't have a SQ and RQ. cmpost.c is just setting the same CQ for the SQ and RQ - this doesn't affect theses queue's operation - it just sets where completions get reported. I hope this makes sense. I'll be glad to expand if I've confused you. - Fab From robert.j.woodruff at intel.com Thu May 26 17:12:27 2005 From: robert.j.woodruff at intel.com (Woodruff, Robert J) Date: Thu, 26 May 2005 17:12:27 -0700 Subject: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux Message-ID: <1AC79F16F5C5284499BB9591B33D6F0004831A0F@orsmsx408> >RNIC-PI is at least an attempt at providing full control over >both iWARP and IB while making as much common as >possible. Where were you last year when the IB verbs header files/API were being discussed. Seems like that was the time to discuss new APIs. Now it is a bit late to propose something totally new like RNIC-PI. The best thing to do now is to try to influence the existing code base to meet your needs rather than recommend a totally new API. And the best way to do that is to make changes to the code that exists or write new code and send in patches to the list so that it can be discussed. woody From iod00d at hp.com Thu May 26 18:11:38 2005 From: iod00d at hp.com (Grant Grundler) Date: Thu, 26 May 2005 18:11:38 -0700 Subject: [openib-general] RE: registering read-only memory In-Reply-To: <506C3D7B14CDD411A52C00025558DED605B1A88A@mtlex01.yok.mtl.com> References: <506C3D7B14CDD411A52C00025558DED605B1A88A@mtlex01.yok.mtl.com> Message-ID: <20050527011138.GD4919@esmail.cup.hp.com> On Thu, May 26, 2005 at 10:23:55PM +0300, Eli Cohen wrote: > I'm afraid that won't solve the problem. From some reason the vma of such a > region is flagged with VM_IO and that's why get_user_pages() will fail. We > need to find another way to get the physical address of such memory. Sorry - why do you need the physical address of memory? Did you mean you need the DMA address? grant > One > thing I can think of is implementing something similar to follow_page(). Or > better if the kernel will export follow_page(). > > -----Original Message----- > From: Roland Dreier [mailto:roland at topspin.com] > Sent: Thursday, May 26, 2005 10:09 PM > To: Eli Cohen > Cc: Michael S. Tsirkin; openib-general at openib.org > Subject: Re: registering read-only memory > > > OK, I think I understand the problem. The compiler puts a string > defined with "const char *foo" on a read-only page, but in > uverbs_mem.c we always pass write=1 to get_user_pages(). I need to > fix the code so that it only asks for writeable pages if we're > registering with writable permissions. > > - R. > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From roland at topspin.com Thu May 26 19:14:57 2005 From: roland at topspin.com (Roland Dreier) Date: Thu, 26 May 2005 19:14:57 -0700 Subject: [openib-general] RE: registering read-only memory In-Reply-To: <20050527011138.GD4919@esmail.cup.hp.com> (Grant Grundler's message of "Thu, 26 May 2005 18:11:38 -0700") References: <506C3D7B14CDD411A52C00025558DED605B1A88A@mtlex01.yok.mtl.com> <20050527011138.GD4919@esmail.cup.hp.com> Message-ID: <52psvdh0ta.fsf@topspin.com> Grant> Sorry - why do you need the physical address of memory? Grant> Did you mean you need the DMA address? We don't really need the physical address. The userspace verbs do get_user_pages() on a userspace virtual address to get a bunch of struct page *s, and then pass those to dma_map_sg() to get DMA addresses. It's all totally kosher... - R. From tduffy at sun.com Thu May 26 19:41:08 2005 From: tduffy at sun.com (Tom Duffy) Date: Thu, 26 May 2005 19:41:08 -0700 Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: References: <1116889028.19768.6.camel@duffman> <4293775B.4080303@ichips.intel.com> Message-ID: <1117161668.19472.11.camel@duffman> On Tue, 2005-05-24 at 15:00 -0400, James Lentini wrote: > Tom, are you willing to rework the patch to use the CM timeout values? So, here is the strategy I am taking. Please let me know if it is wrong. When dapl_ep_connect() is called, I save off the timeout value into the dapl_ep struct. Then, when we get ready to call ib_send_cm_req(), I stuff the timeout value (after munging it into IB's strange format) into the conn params remote_cm_response_timeout. If the connection fails to complete within the timeout, dapl_cm_active_cb_handler() is called with IB_CM_REQ_ERROR which in turn calls dapl_evd_connection_callback() which does the same thing that dapl_ep_timeout() used to do -- tear down the connection. Here is a patch that implements this, *untested*, please take a look. Signed-off-by: Tom Duffy Index: linux-kernel-timer/test/dapltest/test/dapl_performance_util.c =================================================================== --- linux-kernel-timer/test/dapltest/test/dapl_performance_util.c (revision 2498) +++ linux-kernel-timer/test/dapltest/test/dapl_performance_util.c (working copy) @@ -538,7 +538,7 @@ DT_performance_wait ( pre_ts = DT_Mdep_GetTimeStamp (); ret = DT_Tdep_evd_wait ( evd_handle, - DAT_TIMEOUT_INFINITE, + MAX_SCHEDULE_TIMEOUT, &event); post_ts = DT_Mdep_GetTimeStamp (); Index: linux-kernel-timer/test/dapltest/test/dapl_performance_client.c =================================================================== --- linux-kernel-timer/test/dapltest/test/dapl_performance_client.c (revision 2498) +++ linux-kernel-timer/test/dapltest/test/dapl_performance_client.c (working copy) @@ -105,7 +105,7 @@ retry: ret = dat_ep_connect (test_ptr->ep_context.ep_handle, test_ptr->remote_ia_addr, test_ptr->ep_context.port, - DAT_TIMEOUT_INFINITE, + MAX_SCHEDULE_TIMEOUT, 0, (void *) 0, /* no private data */ test_ptr->cmd->qos, Index: linux-kernel-timer/test/dapltest/test/dapl_server.c =================================================================== --- linux-kernel-timer/test/dapltest/test/dapl_server.c (revision 2498) +++ linux-kernel-timer/test/dapltest/test/dapl_server.c (working copy) @@ -584,7 +584,7 @@ DT_cs_Server (Params_t * params_ptr) DT_Mdep_wait_object_wait ( &pt_ptr->synch_wait_object, - DAT_TIMEOUT_INFINITE); + MAX_SCHEDULE_TIMEOUT); /* Send the Server_Info */ DT_Tdep_PT_Debug (1,(phead,"%s: Send Server_Info\n", module)); Index: linux-kernel-timer/test/dapltest/test/dapl_transaction_test.c =================================================================== --- linux-kernel-timer/test/dapltest/test/dapl_transaction_test.c (revision 2498) +++ linux-kernel-timer/test/dapltest/test/dapl_transaction_test.c (working copy) @@ -714,7 +714,7 @@ retry: ret = dat_ep_connect (test_ptr->ep_context[i].ep_handle, test_ptr->remote_ia_addr, test_ptr->ep_context[i].ia_port, - DAT_TIMEOUT_INFINITE, + MAX_SCHEDULE_TIMEOUT, strlen (private_data_str), private_data_str, pt_ptr->Params.ReliabilityLevel, DAT_CONNECT_DEFAULT_FLAG); Index: linux-kernel-timer/test/dapltest/test/dapl_test_util.c =================================================================== --- linux-kernel-timer/test/dapltest/test/dapl_test_util.c (revision 2498) +++ linux-kernel-timer/test/dapltest/test/dapl_test_util.c (working copy) @@ -261,7 +261,7 @@ DT_cr_event_wait ( DT_Tdep_Print_Head *p u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); + ret = DT_Tdep_evd_wait (evd_handle, MAX_SCHEDULE_TIMEOUT, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (CR) failed: %s\n", @@ -316,7 +316,7 @@ DT_conn_event_wait (DT_Tdep_Print_Head * u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); + ret = DT_Tdep_evd_wait (evd_handle, MAX_SCHEDULE_TIMEOUT, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (CONN) failed: %s\n", @@ -369,7 +369,7 @@ DT_disco_event_wait ( DT_Tdep_Print_Head u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); + ret = DT_Tdep_evd_wait (evd_handle, MAX_SCHEDULE_TIMEOUT, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (DISCONN) failed: %s\n", @@ -492,7 +492,7 @@ DT_dto_event_wait (DT_Tdep_Print_Head *p u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); + ret = DT_Tdep_evd_wait (evd_handle, MAX_SCHEDULE_TIMEOUT, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (DTO) failed: %s\n", @@ -536,7 +536,7 @@ DT_rmr_event_wait (DT_Tdep_Print_Head *p u32 ret; struct dat_event event; - ret = DT_Tdep_evd_wait (evd_handle, DAT_TIMEOUT_INFINITE, &event); + ret = DT_Tdep_evd_wait (evd_handle, MAX_SCHEDULE_TIMEOUT, &event); if (ret != DAT_SUCCESS) { DT_Tdep_PT_Printf (phead, "Test Error: dapl_event_wait (RMR) failed: %s\n", Index: linux-kernel-timer/test/dapltest/test/dapl_client.c =================================================================== --- linux-kernel-timer/test/dapltest/test/dapl_client.c (revision 2498) +++ linux-kernel-timer/test/dapltest/test/dapl_client.c (working copy) @@ -256,7 +256,7 @@ retry: ret = dat_ep_connect (ep_handle, server_netaddr, SERVER_PORT_NUMBER, - DAT_TIMEOUT_INFINITE, + MAX_SCHEDULE_TIMEOUT, 0, (void *) 0, /* no private data */ params_ptr->ReliabilityLevel, DAT_CONNECT_DEFAULT_FLAG); Index: linux-kernel-timer/test/dapltest/include/dapl_tdep.h =================================================================== --- linux-kernel-timer/test/dapltest/include/dapl_tdep.h (revision 2498) +++ linux-kernel-timer/test/dapltest/include/dapl_tdep.h (working copy) @@ -56,7 +56,7 @@ DT_Tdep_evd_free (DAT_EVD_HANDLE evd_ha u32 DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_handle, - DAT_TIMEOUT timeout, + unsigned long timeout, struct dat_event *event); u32 DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_handle, Index: linux-kernel-timer/test/dapltest/include/dapl_proto.h =================================================================== --- linux-kernel-timer/test/dapltest/include/dapl_proto.h (revision 2498) +++ linux-kernel-timer/test/dapltest/include/dapl_proto.h (working copy) @@ -189,7 +189,7 @@ void DT_Mdep_Thread_EXIT ( void * thr int DT_Mdep_wait_object_init ( DT_WAIT_OBJECT *wait_obj); int DT_Mdep_wait_object_wait ( DT_WAIT_OBJECT *wait_obj, - int timeout_val); + unsigned long timeout_val); int DT_Mdep_wait_object_wakeup ( DT_WAIT_OBJECT *wait_obj); int DT_Mdep_wait_object_destroy ( DT_WAIT_OBJECT *wait_obj); Index: linux-kernel-timer/test/dapltest/include/dapl_transaction_test.h =================================================================== --- linux-kernel-timer/test/dapltest/include/dapl_transaction_test.h (revision 2498) +++ linux-kernel-timer/test/dapltest/include/dapl_transaction_test.h (working copy) @@ -79,7 +79,7 @@ typedef struct Transaction_Cmd_t *cmd; struct sockaddr * remote_ia_addr; DAT_CONN_QUAL base_port; - DAT_TIMEOUT time_out; + unsigned long time_out; int evd_length; Thread *thread; Index: linux-kernel-timer/test/dapltest/mdep/linux/dapl_mdep_user.c =================================================================== --- linux-kernel-timer/test/dapltest/mdep/linux/dapl_mdep_user.c (revision 2498) +++ linux-kernel-timer/test/dapltest/mdep/linux/dapl_mdep_user.c (working copy) @@ -459,7 +459,7 @@ DT_Mdep_wait_object_init ( int DT_Mdep_wait_object_wait ( DT_WAIT_OBJECT *wait_obj, - int timeout_val) + unsigned long timeout_val) { int dat_status; int pthread_status; Index: linux-kernel-timer/test/dapltest/mdep/linux/dapl_mdep_kernel.c =================================================================== --- linux-kernel-timer/test/dapltest/mdep/linux/dapl_mdep_kernel.c (revision 2498) +++ linux-kernel-timer/test/dapltest/mdep/linux/dapl_mdep_kernel.c (working copy) @@ -313,13 +313,13 @@ DT_Mdep_wait_object_init ( int DT_Mdep_wait_object_wait ( DT_WAIT_OBJECT *wait_obj, - int timeout_val) + unsigned long timeout_val) { int expire; int dat_status; dat_status = DAT_SUCCESS; - if ( DAT_TIMEOUT_INFINITE == timeout_val) + if ( MAX_SCHEDULE_TIMEOUT == timeout_val) { interruptible_sleep_on (wait_obj); } Index: linux-kernel-timer/test/dapltest/kdapl/kdapl_tdep_evd.c =================================================================== --- linux-kernel-timer/test/dapltest/kdapl/kdapl_tdep_evd.c (revision 2498) +++ linux-kernel-timer/test/dapltest/kdapl/kdapl_tdep_evd.c (working copy) @@ -214,7 +214,7 @@ DT_Tdep_evd_dequeue (DAT_EVD_HANDLE evd_ u32 DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_handle, - DAT_TIMEOUT timeout, + unsigned long timeout, struct dat_event *dat_event) { u32 dat_status; @@ -240,7 +240,7 @@ DT_Tdep_evd_wait (DAT_EVD_HANDLE evd_han return dat_status; } /* wait for an event */ - if (timeout == DAT_TIMEOUT_INFINITE) + if (timeout == MAX_SCHEDULE_TIMEOUT) { rc = wait_event_interruptible (evd_ptr->wait_object, evd_ptr->event_next != NULL); Index: linux-kernel-timer/dat-provider/dapl_ep_connect.c =================================================================== --- linux-kernel-timer/dat-provider/dapl_ep_connect.c (revision 2498) +++ linux-kernel-timer/dat-provider/dapl_ep_connect.c (working copy) @@ -35,7 +35,6 @@ #include "dapl_ep_util.h" #include "dapl_openib_util.h" #include "dapl_evd_util.h" -#include "dapl_timer_util.h" /* * Request a connection be established between the local Endpoint @@ -44,7 +43,7 @@ */ u32 dapl_ep_connect(DAT_EP_HANDLE ep_handle, struct sockaddr *remote_ia_address, - DAT_CONN_QUAL remote_conn_qual, DAT_TIMEOUT timeout, + DAT_CONN_QUAL remote_conn_qual, unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos qos, enum dat_connect_flags connect_flags) { @@ -277,23 +276,13 @@ u32 dapl_ep_connect(DAT_EP_HANDLE ep_han spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); if (ep_ptr->param.ep_state == DAT_EP_STATE_ACTIVE_CONNECTION_PENDING - && timeout != DAT_TIMEOUT_INFINITE) { - ep_ptr->cxn_timer = kmalloc(sizeof *ep_ptr->cxn_timer, - GFP_ATOMIC); - if (!ep_ptr->cxn_timer) { - dat_status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, - DAT_RESOURCE_MEMORY); - goto bail; - } - - dapl_timer_set(ep_ptr->cxn_timer, - dapl_ep_timeout, ep_ptr, timeout); - } + && timeout != MAX_SCHEDULE_TIMEOUT) + ep_ptr->timeout = timeout; spin_unlock_irqrestore(&ep_ptr->header.lock, ep_ptr->header.flags); } - bail: +bail: dapl_dbg_log(DAPL_DBG_TYPE_RTN | DAPL_DBG_TYPE_CM, "dapl_ep_connect () returns 0x%x\n", dat_status); Index: linux-kernel-timer/dat-provider/dapl_module.c =================================================================== --- linux-kernel-timer/dat-provider/dapl_module.c (revision 2498) +++ linux-kernel-timer/dat-provider/dapl_module.c (working copy) @@ -36,7 +36,6 @@ #include "dapl_mr_util.h" #include "dapl_util.h" #include "dapl_openib_util.h" -#include "dapl_timer_util.h" MODULE_LICENSE("Dual BSD/GPL"); MODULE_DESCRIPTION("DAT Provider for InfiniBand"); @@ -57,8 +56,6 @@ static int __init dapl_init(void) dapl_dbg_log(DAPL_DBG_TYPE_UTIL, "provider started\n"); - dapl_timer_init(); - dat_status = dapl_provider_list_create(); if (DAT_SUCCESS != dat_status) { dapl_dbg_log(DAPL_DBG_TYPE_ERR, Index: linux-kernel-timer/dat-provider/dapl_evd_connection_callb.c =================================================================== --- linux-kernel-timer/dat-provider/dapl_evd_connection_callb.c (revision 2498) +++ linux-kernel-timer/dat-provider/dapl_evd_connection_callb.c (working copy) @@ -32,7 +32,6 @@ #include "dapl.h" #include "dapl_evd_util.h" #include "dapl_ep_util.h" -#include "dapl_timer_util.h" /* * dapl_evd_connection_callback @@ -87,15 +86,6 @@ void dapl_evd_connection_callback(struct */ spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); - /* - * If a connection timer has been set up on this EP, cancel it now - */ - if (ep_ptr->cxn_timer != NULL) { - dapl_timer_cancel(ep_ptr->cxn_timer); - kfree(ep_ptr->cxn_timer); - ep_ptr->cxn_timer = NULL; - } - switch (event) { case DAT_CONNECTION_EVENT_ESTABLISHED: { Index: linux-kernel-timer/dat-provider/dapl_ep_free.c =================================================================== --- linux-kernel-timer/dat-provider/dapl_ep_free.c (revision 2498) +++ linux-kernel-timer/dat-provider/dapl_ep_free.c (working copy) @@ -34,7 +34,6 @@ #include "dapl_ep_util.h" #include "dapl_openib_util.h" #include "dapl_ring_buffer_util.h" -#include "dapl_timer_util.h" /* * dapl_ep_free @@ -107,12 +106,6 @@ u32 dapl_ep_free(DAT_EP_HANDLE ep_handle */ spin_lock_irqsave(&ep_ptr->header.lock, ep_ptr->header.flags); - if (ep_ptr->cxn_timer != NULL) { - dapl_timer_cancel(ep_ptr->cxn_timer); - kfree(ep_ptr->cxn_timer); - ep_ptr->cxn_timer = NULL; - } - /* Remove the EP from the IA */ dapl_ia_unlink_ep(ia_ptr, ep_ptr); Index: linux-kernel-timer/dat-provider/dapl_openib_cm.c =================================================================== --- linux-kernel-timer/dat-provider/dapl_openib_cm.c (revision 2498) +++ linux-kernel-timer/dat-provider/dapl_openib_cm.c (working copy) @@ -288,6 +288,26 @@ static int dapl_cm_passive_cb_handler(st return ret; } +/* + * approximately transforms miliseconds to 4.096us*2^x + * 63(+8) is max return + */ +static inline u8 dapl_convert_ms_to_kookyib(unsigned long ms) { + unsigned long converged = 2; + u8 i; + + if (2 > ms) + return 8; + + for (i = 1; i < 64; i++) { + if (converged >= ms) + break; + converged = 2*converged; + } + + return i+8; +} + static void dapl_path_comp_handler(u64 req_id, void *context, int rec_num) { struct dapl_cm_id *conn = context; @@ -321,6 +341,10 @@ static void dapl_path_comp_handler(u64 r if (conn->dapl_path.mtu > IB_MTU_1024) conn->dapl_path.mtu = IB_MTU_1024; + if (conn->ep && conn->ep->timeout != 0) + conn->param.remote_cm_response_timeout = + dapl_convert_ms_to_kookyib(conn->ep->timeout); + conn->param.primary_path = &conn->dapl_path; conn->param.alternate_path = NULL; Index: linux-kernel-timer/dat-provider/dapl_timer_util.c =================================================================== --- linux-kernel-timer/dat-provider/dapl_timer_util.c (revision 2498) +++ linux-kernel-timer/dat-provider/dapl_timer_util.c (working copy) @@ -1,313 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * Routines to add and cancel timer records. A timer record - * is put on the global timer queue. If the timer thread is - * not running, start it. The timer thread will sleep - * until a timer event or until a process wakes it up - * to notice a new timer is available; we use a DAPL_WAIT_OBJ - * for synchronization. - * - * If a timer is cancelled, it is simlpy removed from the - * queue. The timer may wakeup and notice there is no timer - * record to awaken at this time, so it will reset for the - * next entry. When there are no timer records to manage, - * the timer thread just sleeps until awakened. - * - * This file also contains the timer handler thread, - * embodied in dapl_timer_thread(). - * - * $Id$ - */ - -#include "dapl.h" -#include "dapl_timer_util.h" - -struct dapl_timer_head { - struct dapl_llist_entry *timer_list_head; - spinlock_t lock; - unsigned long flags; - DAPL_OS_WAIT_OBJECT wait_object; - int thread_id; -}; - -static struct dapl_timer_head g_dapl_timer_head; - -static int dapl_timer_thread(void *arg); - -void dapl_timer_init(void) -{ - /* - * Set up the timer thread elements. The timer thread isn't - * started until it is actually needed - */ - g_dapl_timer_head.timer_list_head = NULL; - spin_lock_init(&g_dapl_timer_head.lock); - dapl_os_wait_object_init(&g_dapl_timer_head.wait_object); - g_dapl_timer_head.thread_id = -1; -} - -/* - * dapl_timer_set - * - * Set a timer. The timer will invoke the specified function - * after a number of useconds expires. - * - * Input: - * timer User provided timer structure - * func Function to invoke when timer expires - * data Argument passed to func() - * expires microseconds until timer fires - * - * Returns: - * no return value - * - */ -u32 dapl_timer_set(struct dapl_timer_entry *timer, void (*func) (void *), - void *data, DAPL_OS_TIMEVAL expires) -{ - struct dapl_timer_entry *list_ptr; - DAPL_OS_TIMEVAL cur_time; - boolean_t wakeup_tmo_thread; - - /* - * Start the timer thread the first time we need a timer - */ - if ( -1 == g_dapl_timer_head.thread_id ) { - g_dapl_timer_head.thread_id = - kernel_thread(dapl_timer_thread, &g_dapl_timer_head, 0); - } - - dapl_llist_init_entry(&timer->list_entry); - wakeup_tmo_thread = FALSE; - dapl_os_get_time(&cur_time); - timer->expires = cur_time + expires; /* calculate future time */ - timer->function = func; - timer->data = data; - - /* - * Put the element on the queue: sorted by wakeup time, eariliest - * first. - */ - spin_lock_irqsave(&g_dapl_timer_head.lock, g_dapl_timer_head.flags); - /* - * Deal with 3 cases due to our list structure: - * 1) list is empty: become the list head - * 2) New timer is sooner than list head: become the list head - * 3) otherwise, sort the timer into the list, no need to wake - * the timer thread up - */ - if (dapl_llist_is_empty(&g_dapl_timer_head.timer_list_head)) { - /* Case 1: add entry to head of list */ - dapl_llist_add_head(&g_dapl_timer_head.timer_list_head, - (struct dapl_llist_entry *) - &timer->list_entry, - timer); - wakeup_tmo_thread = TRUE; - } else { - list_ptr = (struct dapl_timer_entry *) - dapl_llist_peek_head(&g_dapl_timer_head.timer_list_head); - - if (timer->expires < list_ptr->expires) { - /* Case 2: add entry to head of list */ - dapl_llist_add_head(&g_dapl_timer_head.timer_list_head, - (struct dapl_llist_entry *) - &timer->list_entry, timer); - wakeup_tmo_thread = TRUE; - } else { - /* Case 3: figure out where entry goes in sorted list */ - list_ptr = - dapl_llist_next_entry(&g_dapl_timer_head. - timer_list_head, - (struct dapl_llist_entry *) - &list_ptr->list_entry); - - while (list_ptr != NULL) { - if (timer->expires < list_ptr->expires) { - dapl_llist_add_entry(&g_dapl_timer_head. - timer_list_head, - (struct dapl_llist_entry *) - &list_ptr->list_entry, - (struct dapl_llist_entry *) - &timer->list_entry, - timer); - break; - - } - list_ptr = - dapl_llist_next_entry(&g_dapl_timer_head. - timer_list_head, - (struct dapl_llist_entry *) - &list_ptr->list_entry); - } - if (list_ptr == NULL) { - /* entry goes to the end of the list */ - dapl_llist_add_tail(&g_dapl_timer_head. - timer_list_head, - (struct dapl_llist_entry *) - &timer->list_entry, timer); - } - } - - } - spin_unlock_irqrestore(&g_dapl_timer_head.lock, - g_dapl_timer_head.flags); - - if (wakeup_tmo_thread == TRUE) { - dapl_os_wait_object_wakeup(&g_dapl_timer_head.wait_object); - } - - return DAT_SUCCESS; -} - -/* - * dapl_timer_cancel - * - * Cancel a timer. Simply deletes the timer with no function invocations - * - * Input: - * timer User provided timer structure - * - * Returns: - * no return value - */ -void dapl_timer_cancel(struct dapl_timer_entry *timer) -{ - spin_lock_irqsave(&g_dapl_timer_head.lock, g_dapl_timer_head.flags); - /* - * make sure the entry has not been removed by another thread - */ - if (!dapl_llist_is_empty(&g_dapl_timer_head.timer_list_head) && - timer->list_entry.list_head == &g_dapl_timer_head.timer_list_head) { - dapl_llist_remove_entry(&g_dapl_timer_head.timer_list_head, - (struct dapl_llist_entry *) - &timer->list_entry); - } - /* - * If this was the first entry on the queue we could awaken the - * thread and have it reset the list; but it will just wake up - * and find that the timer entry has been removed, then go back - * to sleep, so don't bother. - */ - spin_unlock_irqrestore(&g_dapl_timer_head.lock, - g_dapl_timer_head.flags); -} - -/* - * dapl_timer_thread - * - * Core worker thread dealing with all timers. Basic algorithm: - * - Sleep until work shows up - * - Take first element of sorted timer list and wake - * invoke the callback if expired - * - Sleep for the timeout period if not expired - * - * Input: - * timer_head Timer head structure to manage timer lists - * - * Returns: - * int - */ -int dapl_timer_thread(void *arg) -{ - struct dapl_timer_entry *list_ptr; - DAPL_OS_TIMEVAL cur_time; - struct dapl_timer_head *timer_head; - u32 dat_status; - - timer_head = arg; - - for (;;) { - if (dapl_llist_is_empty(&timer_head->timer_list_head)) { - dat_status = - dapl_os_wait_object_wait(&timer_head->wait_object, - DAT_TIMEOUT_INFINITE); - } - - /* - * Lock policy: - * While this thread is accessing the timer list, it holds the - * lock. Otherwise, it doesn't. - */ - spin_lock_irqsave(&timer_head->lock, timer_head->flags); - while (!dapl_llist_is_empty(&timer_head->timer_list_head)) { - list_ptr = (struct dapl_timer_entry *) - dapl_llist_peek_head(&g_dapl_timer_head. - timer_list_head); - dapl_os_get_time(&cur_time); - - if (list_ptr->expires <= cur_time) { - /* - * Remove the entry from the list. Sort out how much - * time we need to sleep for the next one - */ - list_ptr = - dapl_llist_remove_head(&timer_head-> - timer_list_head); - spin_unlock_irqrestore(&timer_head->lock, - timer_head->flags); - - /* - * Invoke the user callback - */ - list_ptr->function(list_ptr->data); - /* timer structure was allocated by caller, we don't - * free it here. - */ - - /* reacquire the lock */ - spin_lock_irqsave(&timer_head->lock, - timer_head->flags); - } else { - spin_unlock_irqrestore(&timer_head->lock, - timer_head->flags); - dat_status = - dapl_os_wait_object_wait(&timer_head-> - wait_object, - (DAT_TIMEOUT) - (list_ptr-> - expires - - cur_time)); - spin_lock_irqsave(&timer_head->lock, - timer_head->flags); - } - } - /* - * release the lock before going back to the top to sleep - */ - spin_unlock_irqrestore(&timer_head->lock, timer_head->flags); - - if (dat_status == DAT_INTERNAL_ERROR) { - /* - * XXX What do we do here? - */ - } - } /* for (;;) */ - - return 0; -} Index: linux-kernel-timer/dat-provider/dapl_timer_util.h =================================================================== --- linux-kernel-timer/dat-provider/dapl_timer_util.h (revision 2498) +++ linux-kernel-timer/dat-provider/dapl_timer_util.h (working copy) @@ -1,42 +0,0 @@ -/* - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. - * - * This Software is licensed under one of the following licenses: - * - * 1) under the terms of the "Common Public License 1.0" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/cpl.php. - * - * 2) under the terms of the "The BSD License" a copy of which is - * available from the Open Source Initiative, see - * http://www.opensource.org/licenses/bsd-license.php. - * - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a - * copy of which is available from the Open Source Initiative, see - * http://www.opensource.org/licenses/gpl-license.php. - * - * Licensee has the right to choose one of the above licenses. - * - * Redistributions of source code must retain the above copyright - * notice and one of the license notices. - * - * Redistributions in binary form must reproduce both the above copyright - * notice, one of the license notices in the documentation - * and/or other materials provided with the distribution. - */ - -/* - * $Id$ - */ - -#ifndef DAPL_TIMER_UTIL_H -#define DAPL_TIMER_UTIL_H - -void dapl_timer_init(void); - -u32 dapl_timer_set(struct dapl_timer_entry *timer, void (*func) (void *), - void *data, DAPL_OS_TIMEVAL expires); - -void dapl_timer_cancel(struct dapl_timer_entry *timer); - -#endif /* DAPL_TIMER_UTIL_H */ Index: linux-kernel-timer/dat-provider/dapl_ep_util.c =================================================================== --- linux-kernel-timer/dat-provider/dapl_ep_util.c (revision 2498) +++ linux-kernel-timer/dat-provider/dapl_ep_util.c (working copy) @@ -168,9 +168,6 @@ void dapl_ep_dealloc(struct dapl_ep *ep_ if (ep_ptr->send_iov) kfree(ep_ptr->send_iov); - if (ep_ptr->cxn_timer) - kfree(ep_ptr->cxn_timer); - kfree(ep_ptr); } @@ -242,32 +239,6 @@ bail: } /* - * dapl_ep_timeout - * - * If this routine is invoked before a connection occurs, generate an - * event - */ -void dapl_ep_timeout(void *arg) -{ - struct dapl_ep *ep_ptr; - - dapl_dbg_log(DAPL_DBG_TYPE_CM, "--> dapl_ep_timeout! ep %lx\n", arg); - - ep_ptr = (struct dapl_ep *)arg; - - /* reset the EP state */ - ep_ptr->param.ep_state = DAT_EP_STATE_DISCONNECTED; - - /* Clean up the EP and put the underlying QP into the ERROR state. */ - dapl_ib_disconnect_clean(ep_ptr, TRUE); - - (void)dapl_evd_post_connection_event((struct dapl_evd *)ep_ptr->param. - connect_evd_handle, - DAT_CONNECTION_EVENT_TIMED_OUT, - (DAT_HANDLE) ep_ptr, 0, NULL); -} - -/* * dapl_ep_state_subtype * * Return the INVALID_STATE connection subtype associated with an Index: linux-kernel-timer/dat-provider/dapl_ep_util.h =================================================================== --- linux-kernel-timer/dat-provider/dapl_ep_util.h (revision 2498) +++ linux-kernel-timer/dat-provider/dapl_ep_util.h (working copy) @@ -52,8 +52,6 @@ extern u32 dapl_ep_post_send_req(DAT_EP_ DAPL_DTO_TYPE dto_type, enum ib_wr_opcode op_type); -void dapl_ep_timeout(void *arg); - enum dat_return_subtype dapl_ep_state_subtype(struct dapl_ep *ep_ptr); #endif /* DAPL_EP_UTIL_H */ Index: linux-kernel-timer/dat-provider/dapl_util.c =================================================================== --- linux-kernel-timer/dat-provider/dapl_util.c (revision 2498) +++ linux-kernel-timer/dat-provider/dapl_util.c (working copy) @@ -33,104 +33,6 @@ #include "dapl_module.h" #include "dapl_util.h" -/* - * dapl_os_get_time - * - * Return 64 bit value of current time in microseconds. - * - * Input: - * loc User location to place current time - * - * Returns: - * DAT_SUCCESS - */ - -u32 dapl_os_get_time(DAPL_OS_TIMEVAL *loc) -{ - struct timeval tv; - - do_gettimeofday(&tv); - *loc = ((u64) (tv.tv_sec) * 1000000L) + (u64) tv.tv_usec; - - return DAT_SUCCESS; -} - -/* - * Wait object routines - */ - -/* - * dapl_os_wait_object_init - * - * Initialize a wait object - * - * Input: - * wait_obj - * - * Returns: - * DAT_SUCCESS - * DAT_INTERNAL_ERROR - */ -u32 dapl_os_wait_object_init(DAPL_OS_WAIT_OBJECT *wait_obj) -{ - init_waitqueue_head(&wait_obj->wait_queue); - - return DAT_SUCCESS; -} - -/* Wait on the supplied wait object, up to the specified time_out. - * A timeout of DAT_TIMEOUT_INFINITE will wait indefinitely. - * Timeout should be specified in micro seconds. - * - * Functional returns: - * DAT_SUCCESS -- another thread invoked dapl_os_wait object_wakeup - * DAT_INVALID_STATE -- someone else is already waiting in this wait - * object. - * only one waiter is allowed at a time. - * DAT_ABORT -- another thread invoked dapl_os_wait_object_destroy - * DAT_TIMEOUT -- the specified time limit was reached. - */ - -u32 dapl_os_wait_object_wait(DAPL_OS_WAIT_OBJECT *wait_obj, - DAT_TIMEOUT timeout_val) -{ - u32 dat_status = DAT_SUCCESS; - - if (DAT_TIMEOUT_INFINITE == timeout_val) { - interruptible_sleep_on(&wait_obj->wait_queue); - } else { - int expire; - - expire = timeout_val * HZ / 1000000; - - while (expire) { - current->state = TASK_INTERRUPTIBLE; - expire = schedule_timeout(expire); - } - dat_status = DAT_ERROR(DAT_TIMEOUT_EXPIRED, 0); - } - return dat_status; -} - -/* - * dapl_os_wait_object_wakeup - * - * Wakeup a thread waiting on a wait object - * - * Input: - * wait_obj - * - * Returns: - * DAT_SUCCESS - * DAT_INTERNAL_ERROR - */ -u32 dapl_os_wait_object_wakeup(DAPL_OS_WAIT_OBJECT *wait_obj) -{ - wake_up_interruptible(&wait_obj->wait_queue); - - return DAT_SUCCESS; -} - #ifdef DAPL_DBG void dapl_dbg_log(DAPL_DBG_TYPE type, const char *fmt, ...) Index: linux-kernel-timer/dat-provider/dapl_util.h =================================================================== --- linux-kernel-timer/dat-provider/dapl_util.h (revision 2498) +++ linux-kernel-timer/dat-provider/dapl_util.h (working copy) @@ -107,24 +107,6 @@ static inline int dapl_os_atomic_assign( */ /* - * The wait object invariant: Presuming a call to dapl_os_wait_object_wait - * occurs at some point, there will be at least one wakeup after each call - * to dapl_os_wait_object_signal. I.e. Signals are not ignored, though - * they may be coallesced. - */ -typedef struct dapl_os_wait_object { - wait_queue_head_t wait_queue; -} DAPL_OS_WAIT_OBJECT; - -/* function prototypes */ -u32 dapl_os_wait_object_init(DAPL_OS_WAIT_OBJECT *wait_obj); - -u32 dapl_os_wait_object_wait(DAPL_OS_WAIT_OBJECT *wait_obj, - DAT_TIMEOUT timeout_val); - -u32 dapl_os_wait_object_wakeup(DAPL_OS_WAIT_OBJECT *wait_obj); - -/* * Memory Functions */ @@ -154,15 +136,6 @@ static inline char *dapl_os_strdup(const } /* - * Timer Functions - */ - -typedef u64 DAPL_OS_TIMEVAL; - -/* timer function prototype */ -u32 dapl_os_get_time(DAPL_OS_TIMEVAL *); - -/* * *printf format helper. We use the C string constant concatenation * ability to define 64 bit formats, which unfortunatly are non standard * in the C compiler world. Index: linux-kernel-timer/dat-provider/dapl_ep_dup_connect.c =================================================================== --- linux-kernel-timer/dat-provider/dapl_ep_dup_connect.c (revision 2498) +++ linux-kernel-timer/dat-provider/dapl_ep_dup_connect.c (working copy) @@ -32,7 +32,6 @@ #include "dapl.h" #include "dapl_ep_util.h" #include "dapl_openib_util.h" -#include "dapl_timer_util.h" /* * dapl_ep_dup_connect @@ -63,7 +62,7 @@ * DAT_MODEL_NOT_SUPPORTED */ u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, DAT_EP_HANDLE ep_dup_handle, - DAT_TIMEOUT timeout, int private_data_size, + unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos qos) { struct dapl_ep *ep_dup_ptr; @@ -114,6 +113,6 @@ u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep private_data_size, private_data, qos, DAT_CONNECT_DEFAULT_FLAG); - bail: +bail: return dat_status; } Index: linux-kernel-timer/dat-provider/dapl.h =================================================================== --- linux-kernel-timer/dat-provider/dapl.h (revision 2498) +++ linux-kernel-timer/dat-provider/dapl.h (working copy) @@ -236,14 +236,6 @@ struct dapl_evd { struct dapl_cno *cno_ptr; }; -/* DAPL timer entry, used to queue timeouts */ -struct dapl_timer_entry { - struct dapl_llist_entry list_entry; /* link entry on ia struct */ - DAPL_OS_TIMEVAL expires; - void (*function) (void *); - void *data; -}; - /* struct dapl_ep maps to DAT_EP_HANDLE */ struct dapl_ep { struct dapl_header header; @@ -264,8 +256,7 @@ struct dapl_ep { /* For passive connections we maintain a back pointer to the CR */ void *cr_ptr; - /* pointer to connection timer, if set */ - struct dapl_timer_entry *cxn_timer; + unsigned long timeout; /* private data container */ char private_data[DAPL_MAX_PRIVATE_DATA_SIZE]; @@ -498,14 +489,15 @@ extern u32 dapl_ep_modify(DAT_EP_HANDLE extern u32 dapl_ep_connect(DAT_EP_HANDLE ep_handle, struct sockaddr *remote_ia_address, DAT_CONN_QUAL remote_conn_qual, - DAT_TIMEOUT timeout, + unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos quality_of_service, enum dat_connect_flags connect_flags); extern u32 dapl_ep_dup_connect(DAT_EP_HANDLE ep_handle, - DAT_EP_HANDLE ep_dup_handle, DAT_TIMEOUT timeout, + DAT_EP_HANDLE ep_dup_handle, + unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos quality_of_service); Index: linux-kernel-timer/patches/alt_dat_provider_makefile =================================================================== --- linux-kernel-timer/patches/alt_dat_provider_makefile (revision 2498) +++ linux-kernel-timer/patches/alt_dat_provider_makefile (working copy) @@ -83,7 +83,6 @@ PROVIDER_MODULES := \ dapl_srq_resize \ dapl_srq_set_lw \ dapl_srq_util \ - dapl_timer_util \ dapl_util PROVIDER_OBJS := $(foreach s, $(PROVIDER_MODULES), $(s).o) @@ -109,4 +108,4 @@ default: endif clean: - rm -f *.o *.ko + rm -rf *.o *.ko .*.cmd *.mod.* .tmp_versions .*.d .*.o.tmp cscope.out Index: linux-kernel-timer/dat/dat.h =================================================================== --- linux-kernel-timer/dat/dat.h (revision 2498) +++ linux-kernel-timer/dat/dat.h (working copy) @@ -383,11 +383,6 @@ enum dat_completion_flags { DAT_COMPLETION_BARRIER_FENCE_FLAG = 0x08, }; -typedef u32 DAT_TIMEOUT; /* microseconds */ - -/* timeout = infinity */ -#define DAT_TIMEOUT_INFINITE ((DAT_TIMEOUT) ~0) - /* dat handles */ typedef void *DAT_HANDLE; typedef DAT_HANDLE DAT_CR_HANDLE; @@ -1139,12 +1134,12 @@ typedef u32 (*DAT_EP_MODIFY_FUNC)(DAT_EP const struct dat_ep_param *); typedef u32 (*DAT_EP_CONNECT_FUNC)(DAT_EP_HANDLE, struct sockaddr *, - DAT_CONN_QUAL, DAT_TIMEOUT, int, + DAT_CONN_QUAL, unsigned long timeout, int, const void *, enum dat_qos, enum dat_connect_flags); typedef u32 (*DAT_EP_DUP_CONNECT_FUNC)(DAT_EP_HANDLE, DAT_EP_HANDLE, - DAT_TIMEOUT, int, const void *, + unsigned long timeout, int, const void *, enum dat_qos); typedef u32 (*DAT_EP_DISCONNECT_FUNC)(DAT_EP_HANDLE, enum dat_close_flags); @@ -1493,7 +1488,7 @@ static inline u32 dat_evd_resize(DAT_EVD } static inline u32 dat_ep_connect(DAT_EP_HANDLE ep, struct sockaddr *ia_addr, - DAT_CONN_QUAL conn_qual, DAT_TIMEOUT timeout, + DAT_CONN_QUAL conn_qual, unsigned long timeout, int private_data_size, const void *private_data, enum dat_qos qos, enum dat_connect_flags flags) @@ -1533,7 +1528,8 @@ static inline u32 dat_ep_disconnect(DAT_ } static inline u32 dat_ep_dup_connect(DAT_EP_HANDLE ep, DAT_EP_HANDLE dup_ep, - DAT_TIMEOUT timeout, int private_data_size, + unsigned long timeout, + int private_data_size, const void *private_data, enum dat_qos qos) { return DAT_CALL_PROVIDER_FUNC(ep_dup_connect_func, ep, dup_ep, timeout, From sean.hefty at intel.com Thu May 26 22:25:10 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Thu, 26 May 2005 22:25:10 -0700 Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: <1117161668.19472.11.camel@duffman> Message-ID: >So, here is the strategy I am taking. Please let me know if it is >wrong. > >When dapl_ep_connect() is called, I save off the timeout value into the >dapl_ep struct. Then, when we get ready to call ib_send_cm_req(), I >stuff the timeout value (after munging it into IB's strange format) into >the conn params remote_cm_response_timeout. >From a CM perspective, this sounds fine. Note that the CM timeout will not occur until the number of retries has been met. So I don't know if the timeout passed to dapl_ep_connect() should convert directly into the remote_cm_response_timeout, or needs to be divided by the number of retries. >If the connection fails to complete within the timeout, >dapl_cm_active_cb_handler() is called with IB_CM_REQ_ERROR which in turn >calls dapl_evd_connection_callback() which does the same thing that >dapl_ep_timeout() used to do -- tear down the connection. I haven't looked at your changes, but note that calling ib_destroy_cm_id from within the CM callback thread will hang. The callback holds a reference on the cm_id. The good news is that there should be code in kDAPL to catch this. >Here is a patch that implements this, *untested*, please take a look. I'll look over the patch tomorrow and let you know if anything stands out, but I'm not overly familiar with the kDAPL code... - Sean From eli at mellanox.co.il Thu May 26 22:26:26 2005 From: eli at mellanox.co.il (Eli Cohen) Date: Fri, 27 May 2005 08:26:26 +0300 Subject: [openib-general] RE: registering read-only memory Message-ID: <506C3D7B14CDD411A52C00025558DED605B1A88B@mtlex01.yok.mtl.com> R> Are you sure? It seems to work for me if I just pass write=0 to R> get_user_pages(). I saw this this on Suse 9.3 and the memory was like: static const char *my_string = "Hello world"; defined locally in a function. This produced a VMA with VM_IO set. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jagana at us.ibm.com Thu May 26 22:47:05 2005 From: jagana at us.ibm.com (Venkata Jagana) Date: Thu, 26 May 2005 22:47:05 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux In-Reply-To: <20050526044700.GK671@esmail.cup.hp.com> Message-ID: rdma-developers-admin at lists.sourceforge.net wrote on 05/25/2005 09:47:00 PM: > Venkata, > Interesting coincidence: I was talking with someone (at HP) today > who knows substantially more than I do about RNICs. > They indicated RNICs need to manage TCP state on the card from userspace. > I suspect that's only possible through a private interface > (e.g. ioctl() or /proc) or the non-existant (in kernel.org) > TOE implementation. Is this correct? > Not correct. Since RNICs are offloaded adapters with RDMA protocols layered on top of TCP stack, they do maintain the TCP state internally but it does not expose to the host. RNIC expose only RNIC Verbs interface to the host bot not TOE interface. Thanks Venkat > > hth, > grant > > > ------------------------------------------------------- > SF.Net email is sponsored by: GoToMeeting - the easiest way to collaborate > online with coworkers and clients while avoiding the high cost of travel and > communications. There is no equipment to buy and you can meet as often as > you want. Try it free.http://ads.osdn.com/?ad_id=7402&alloc_id=16135&op=click > _______________________________________________ > Rdma-developers mailing list > Rdma-developers at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/rdma-developers -------------- next part -------------- An HTML attachment was scrubbed... URL: From jagana at us.ibm.com Thu May 26 23:16:52 2005 From: jagana at us.ibm.com (Venkata Jagana) Date: Thu, 26 May 2005 23:16:52 -0700 Subject: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <1AC79F16F5C5284499BB9591B33D6F0004831A0F@orsmsx408> Message-ID: "Woodruff, Robert J" wrote on 05/26/2005 05:12:27 PM: > >RNIC-PI is at least an attempt at providing full control over > >both iWARP and IB while making as much common as > >possible. > > Where were you last year when the IB verbs header files/API were being > discussed. > Seems like that was the time to discuss new APIs. > Now it is a bit late to propose something totally new like RNIC-PI. > The best thing to do now is to try to influence the existing code > base to meet your needs rather than recommend a totally new API. > And the best way to do that is to make changes to the code that exists > or write new code and send in patches to the list so that it can be > discussed. > > woody > It is just unfortunate to make such an IB-centric statement. I will ask the same question back to you - why didn't you consider RNICs when the IB APIs were designed a year ago? If that was considered a year ago, We wouldn't have this problem today, right? Nothing can be designed if we don't know what exactly need to be done for RNICs a year ago. The real answer is because it was too early to adopt RNIC-verbs at that time when the specs were still evolving in standard bodies (RDMAC/IETF). Again because we have developed something a year ago doesn't mean that we will have to live with that interface forever. Of course, the question is when and why should this change? As it was discussed before, we would like to see the common PI developed without impacting the IB development. Thanks Venkat -------------- next part -------------- An HTML attachment was scrubbed... URL: From sganguly at yahoo.com Fri May 27 06:40:43 2005 From: sganguly at yahoo.com (Sukanta ganguly) Date: Fri, 27 May 2005 06:40:43 -0700 (PDT) Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux In-Reply-To: 6667 Message-ID: <20050527134043.92998.qmail@web51901.mail.yahoo.com> Venkata, How will that work? If the RNIC offloads RDMA and TCP completely from the Operating System and does not share any state information then the application running on the host will never be in the position to utilize the socket interface to use the communication logic to send and receive data between the remote node and itself. Some information needs to be shared. How much of it and what exactly needs to be shared is the question. Thanks SG --- Venkata Jagana wrote: > > > > > > > rdma-developers-admin at lists.sourceforge.net wrote on > 05/25/2005 09:47:00 > PM: > > > Venkata, > > Interesting coincidence: I was talking with > someone (at HP) today > > who knows substantially more than I do about > RNICs. > > They indicated RNICs need to manage TCP state on > the card from userspace. > > I suspect that's only possible through a private > interface > > (e.g. ioctl() or /proc) or the non-existant (in > kernel.org) > > TOE implementation. Is this correct? > > > > Not correct. > > Since RNICs are offloaded adapters with RDMA > protocols layered on > top of TCP stack, they do maintain the TCP state > internally but > it does not expose to the host. RNIC expose only > RNIC Verbs interface > to the host bot not TOE interface. > > Thanks > Venkat > > > > > hth, > > grant > > > > > > > ------------------------------------------------------- > > SF.Net email is sponsored by: GoToMeeting - the > easiest way to > collaborate > > online with coworkers and clients while avoiding > the high cost of travel > and > > communications. There is no equipment to buy and > you can meet as often as > > you want. Try it > free.http://ads.osdn.com/?ad_id=7402&alloc_id=16135&op=click > > _______________________________________________ > > Rdma-developers mailing list > > Rdma-developers at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/rdma-developers __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From sganguly at yahoo.com Fri May 27 06:46:39 2005 From: sganguly at yahoo.com (Sukanta ganguly) Date: Fri, 27 May 2005 06:46:39 -0700 (PDT) Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux In-Reply-To: <6.2.0.14.2.20050526103456.02af8dd8@esmail.cup.hp.com> Message-ID: <20050527134639.38189.qmail@web51905.mail.yahoo.com> I completely agree with Michael's viewpoint here. Doing a clean RDMA focussed implementation would bring out the exact requirements that need to be changed. Retrofitting IB/Infiniband based API into the RDMA based subsystem just to reuse an existing base is not a good start. Thanks SG --- Michael Krause wrote: > At 09:49 AM 5/26/2005, Sean Hefty wrote: > >Roland Dreier wrote: > >>I believe the way forward is to evolve the > existing drivers/infiniband > >>code already in Linux into a drivers/rdma that > supports both IB and > >>RNICs. To be extremely blunt, I believe the > RNIC-PI is irrelevant to > >>the Linux kernel -- no IB vendors will support > ripping out a working > >>midlayer and starting from scratch, and it doesn't > make sense to have > >>two essentially equivalent midlayers in the same > kernel. > > > >IMO, any APIs need to evolve out of the > implementation. Trying to fit an > >implementation under an existing API tends to lead > to either a poor > >implementation, or requires changes to the API > anyway. > > > >I think a useful path is for someone to implement > an RNIC driver and > >provide feedback on what changes would be required > of the Infiniband/core > >layer to support it. > > There needs to be a balance between establishing a > good, flexible > architecture and how applications or subsystems > interact with it. All of > this needs to be grounded in implementation > experience so there is a > healthy does of reality. The problem with the > iterative-TTM focused API > definitions is they rarely produce something that is > focused on the "end > game" for a solid interface. They create instant > legacy baggage that > people are unwilling to shed as time goes forward. > This legacy inertia is > what has stifled quite a bit of innovation when it > comes to software (some > hazard that 90% of the software created today is > focused primarily on > legacy investment protection than really new > innovation and when you think > about it for a bit of time, you can see that much of > this is re-inventing > the wheel or to band-aid over a problem and > packaging it as something > innovative). > > So, the problem becomes one of finding balance. The > people proposing the > various API are people who have implemented various > types of solutions so > they are coming with both real experience as well as > engineering judgement > of what is required to get the right infrastructure > in place for the "end > game" needs. From a practical perspective, one > needs to implement these > API and see what is really of value and what should > be changed. The key is > to avoid creating the legacy inertia that ends up > reducing fragmenting the > interfaces and causes lost productivity, quality > problems, etc. So before > people write off the various API as unnecessary or > as poor quality, there > should be some implementations developed and some > constructive debate of > what features are really of value and what might be > deprecated or > eliminated. There is no requirement to ever > implement all of an API as > this is where an iterative approach has value. > Implement what is needed to > demonstrate the desired value and decide then if it > is worthwhile. Just > please don't discount it or toss it all out because > it isn't all > implemented today or you don't like some aspect. > > > >>To put a really concrete proposal on the table, I > would suggest to > >>start by extending the current ib_client > registration structure > > > >Roland's proposal sounds like a reasonable and fair > way to > >begin. Building an abstraction on top of the > existing layers seems > >secondary to adding support for other RDMA devices. > > Getting the right verbs interface is paramount. > Whether one does IT API or > DAPL on top of that for a given subsystem is > secondary. > > Ideally, OpenIB and OpenRDMA should be focused on > developing the RDMA > infrastructure - RDMA verbs, connection management, > IB-specific ULP, > etc. They should not be focused on the upper > subsystems or ULP. Those > should be done as separate open source projects and > they can decide where > best to intersect with the Open* infrastructure. > This one-size-fits-all > approach for all subsystems to flow through a given > interface is simply > impractical to execute. It might happen over time > but it cannot be > force-fit. Let's focus the two efforts on getting > as common of an > infrastructure as possible. Some propose RNIC PI; > other can propose > something else if there is desire. RNIC PI has > value in that it has > focused on the common components between IB and > iWARP and left the > IB-specific and CM outside of its definition to > allow an OS to decide how > best to support or to allow IB to do what is needed > for its particular usages. > > > Mike __________________________________ Do you Yahoo!? Yahoo! Small Business - Try our new Resources site http://smallbusiness.yahoo.com/resources/ From krause at cup.hp.com Fri May 27 07:24:44 2005 From: krause at cup.hp.com (Michael Krause) Date: Fri, 27 May 2005 07:24:44 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux In-Reply-To: <20050527134043.92998.qmail@web51901.mail.yahoo.com> References: <20050527134043.92998.qmail@web51901.mail.yahoo.com> Message-ID: <6.2.0.14.2.20050527071821.02071338@esmail.cup.hp.com> At 06:40 AM 5/27/2005, Sukanta ganguly wrote: >Venkata, > How will that work? If the RNIC offloads RDMA and >TCP completely from the Operating System and does not >share any state information then the application >running on the host will never be in the position to >utilize the socket interface to use the communication >logic to send and receive data between the remote node >and itself. Some information needs to be shared. How >much of it and what exactly needs to be shared is the >question. Ok. It all depends upon what level of integration / interaction a TOE and thus a RNIC will have with the host network stack. For example, if a customer wants to have TCP and IP stats kept for the off-loaded stack even if it is just being using for RDMA, then there needs to be a method defined to consolidate these stats back into the host network stack tool chain. Similarly, if one wants to maintain a single routing table to manage, etc. on the host, then the RNIC needs to access / update that information accordingly. One can progress through other aspects of integration, e.g. connection management, security interactions (e.g. DOS protection), and so forth. What is exposed again depends upon the level of integration and how customers want to manage their services. This problem also exists for IB but most people have not thought about this from a customer perspective and how to integrate the IB semantics into the way customers manage their infrastructures, do billing, etc. For some environments, they simply do not care but if IB is to be used in the enterprise space, then some thought will be required here since most IT don't see anything as being "free" or self-managed. Again, Sockets is an application API and not how one communicates to a TOE or RDMA component. The RNIC PI has been proposed as an interface to the RDMA functionality. The PI supports all of the iWARP and IB v 1.2 verbs. Mike >Thanks >SG > >--- Venkata Jagana wrote: > > > > > > > > > > > > > > rdma-developers-admin at lists.sourceforge.net wrote on > > 05/25/2005 09:47:00 > > PM: > > > > > Venkata, > > > Interesting coincidence: I was talking with > > someone (at HP) today > > > who knows substantially more than I do about > > RNICs. > > > They indicated RNICs need to manage TCP state on > > the card from userspace. > > > I suspect that's only possible through a private > > interface > > > (e.g. ioctl() or /proc) or the non-existant (in > > kernel.org) > > > TOE implementation. Is this correct? > > > > > > > Not correct. > > > > Since RNICs are offloaded adapters with RDMA > > protocols layered on > > top of TCP stack, they do maintain the TCP state > > internally but > > it does not expose to the host. RNIC expose only > > RNIC Verbs interface > > to the host bot not TOE interface. > > > > Thanks > > Venkat > > > > > > > > hth, > > > grant > > > > > > > > > > > >------------------------------------------------------- > > > SF.Net email is sponsored by: GoToMeeting - the > > easiest way to > > collaborate > > > online with coworkers and clients while avoiding > > the high cost of travel > > and > > > communications. There is no equipment to buy and > > you can meet as often as > > > you want. Try it > > >free.http://ads.osdn.com/?ad_id=7402&alloc_id=16135&op=click > > > _______________________________________________ > > > Rdma-developers mailing list > > > Rdma-developers at lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/rdma-developers > >__________________________________________________ >Do You Yahoo!? >Tired of spam? Yahoo! Mail has the best spam protection around >http://mail.yahoo.com > > >------------------------------------------------------- >This SF.Net email is sponsored by Yahoo. >Introducing Yahoo! Search Developer Network - Create apps using Yahoo! >Search APIs Find out how you can build Yahoo! directly into your own >Applications - visit http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005 >_______________________________________________ >Rdma-developers mailing list >Rdma-developers at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/rdma-developers -------------- next part -------------- An HTML attachment was scrubbed... URL: From BMT at zurich.ibm.com Fri May 27 08:05:01 2005 From: BMT at zurich.ibm.com (Bernard Metzler) Date: Fri, 27 May 2005 17:05:01 +0200 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux In-Reply-To: <20050527134043.92998.qmail@web51901.mail.yahoo.com> Message-ID: Sukanta, without touching any TOE issues (this question is about RDMA, right?), after transforming a TCP connection into RDMA mode and using an RDMA API, the socket file descriptor is not longer to be used for communication. In fact, on some implementations the stream socket resources will even get released. That is, if the socket was the direct consumer of the TCP stream, then now it is RDMAP/DDP/MPA. RDMA APIs such as IT-API defining a specific call to convert a socket based connection into RDMA mode (e.g., it_socket_convert()). Other consumers may directly start via an RDMA API, never opening a consumer controlled socket. So, in RDMA mode, communication will happen via the RDMA API. At this stage, the kernel still have to keep completely in its hands the synchronisation of state information related to that offloaded connection(s) with the host stack (it would have to protect the local port used by the offloaded connection for example, others are routing, ARP, SNMP...), but it is not involved at the data path. With respect to the kernel based TCP stack, what is not needed is a hack into the stack and scatter/gather state information of the live TCP connection between kernel and RNIC, but to find one clean interface to transfer state information out of that stack and to the RNIC. With limited benefit, one could of course also implement native sockets over RDMA, where an in-kernel midlayer on top of kernel RDMA Verbs is doing the translation between send(), receive() to post_send, post_receive. But usage of 'true RDMA' operations like RDMA READ or WRITE might be limited, and I don't see much value for the user here. One variety of this approach with less limited access to RMDA benefits might be sockets with extended RDMA semantics. Bernard. rdma-developers-admin at lists.sourceforge.net wrote on 27.05.2005 15:40:43: > Venkata, > How will that work? If the RNIC offloads RDMA and > TCP completely from the Operating System and does not > share any state information then the application > running on the host will never be in the position to > utilize the socket interface to use the communication > logic to send and receive data between the remote node > and itself. Some information needs to be shared. How > much of it and what exactly needs to be shared is the > question. > > Thanks > SG > > --- Venkata Jagana wrote: > > > > > > > > > > > > > > rdma-developers-admin at lists.sourceforge.net wrote on > > 05/25/2005 09:47:00 > > PM: > > > > > Venkata, > > > Interesting coincidence: I was talking with > > someone (at HP) today > > > who knows substantially more than I do about > > RNICs. > > > They indicated RNICs need to manage TCP state on > > the card from userspace. > > > I suspect that's only possible through a private > > interface > > > (e.g. ioctl() or /proc) or the non-existant (in > > kernel.org) > > > TOE implementation. Is this correct? > > > > > > > Not correct. > > > > Since RNICs are offloaded adapters with RDMA > > protocols layered on > > top of TCP stack, they do maintain the TCP state > > internally but > > it does not expose to the host. RNIC expose only > > RNIC Verbs interface > > to the host bot not TOE interface. > > > > Thanks > > Venkat > > > > > > > > hth, > > > grant > > > > > > > > > > > > ------------------------------------------------------- > > > SF.Net email is sponsored by: GoToMeeting - the > > easiest way to > > collaborate > > > online with coworkers and clients while avoiding > > the high cost of travel > > and > > > communications. There is no equipment to buy and > > you can meet as often as > > > you want. Try it > > > free.http://ads.osdn.com/?ad_id=7402&alloc_id=16135&op=click > > > _______________________________________________ > > > Rdma-developers mailing list > > > Rdma-developers at lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/rdma-developers > > __________________________________________________ > Do You Yahoo!? > Tired of spam? Yahoo! Mail has the best spam protection around > http://mail.yahoo.com > > > ------------------------------------------------------- > This SF.Net email is sponsored by Yahoo. > Introducing Yahoo! Search Developer Network - Create apps using Yahoo! > Search APIs Find out how you can build Yahoo! directly into your own > Applications - visit http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005 > _______________________________________________ > Rdma-developers mailing list > Rdma-developers at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/rdma-developers -------------- next part -------------- An HTML attachment was scrubbed... URL: From jlentini at netapp.com Fri May 27 08:12:02 2005 From: jlentini at netapp.com (James Lentini) Date: Fri, 27 May 2005 11:12:02 -0400 (EDT) Subject: [openib-general] Re: [PATCH 1/3] kDAPL: consolidate dapl_srq*.c into one file In-Reply-To: <1117133253.17831.12.camel@duffman> References: <1117133253.17831.12.camel@duffman> Message-ID: Committed in revision 2500 On Thu, 26 May 2005, Tom Duffy wrote: tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> Index: linux-kernel-srq/dat-provider/dapl_srq.c tduffy> =================================================================== tduffy> --- linux-kernel-srq/dat-provider/dapl_srq.c (revision 0) tduffy> +++ linux-kernel-srq/dat-provider/dapl_srq.c (revision 0) tduffy> @@ -0,0 +1,500 @@ tduffy> +/* tduffy> + * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * tduffy> + * This Software is licensed under one of the following licenses: tduffy> + * tduffy> + * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> + * available from the Open Source Initiative, see tduffy> + * http://www.opensource.org/licenses/cpl.php. tduffy> + * tduffy> + * 2) under the terms of the "The BSD License" a copy of which is tduffy> + * available from the Open Source Initiative, see tduffy> + * http://www.opensource.org/licenses/bsd-license.php. tduffy> + * tduffy> + * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> + * copy of which is available from the Open Source Initiative, see tduffy> + * http://www.opensource.org/licenses/gpl-license.php. tduffy> + * tduffy> + * Licensee has the right to choose one of the above licenses. tduffy> + * tduffy> + * Redistributions of source code must retain the above copyright tduffy> + * notice and one of the license notices. tduffy> + * tduffy> + * Redistributions in binary form must reproduce both the above copyright tduffy> + * notice, one of the license notices in the documentation tduffy> + * and/or other materials provided with the distribution. tduffy> + */ tduffy> + tduffy> +/* tduffy> + * $Id$ tduffy> + */ tduffy> + tduffy> +#include "dapl.h" tduffy> +#include "dapl_ia_util.h" tduffy> +#include "dapl_openib_util.h" tduffy> +#include "dapl_cookie.h" tduffy> + tduffy> +/* tduffy> + * dapl_srq_dealloc tduffy> + * tduffy> + * Free the passed in SRQ structure. tduffy> + * tduffy> + * Input: tduffy> + * SRQ pointer tduffy> + * tduffy> + * Output: tduffy> + * none tduffy> + * tduffy> + * Returns: tduffy> + * none tduffy> + * tduffy> + */ tduffy> +static void dapl_srq_dealloc(struct dapl_srq *srq_ptr) tduffy> +{ tduffy> + dapl_os_assert(srq_ptr->header.magic == DAPL_MAGIC_SRQ); tduffy> + tduffy> + /* reset magic to prevent reuse */ tduffy> + srq_ptr->header.magic = DAPL_MAGIC_INVALID; tduffy> + dapl_ia_unlink_srq(srq_ptr->header.owner_ia, srq_ptr); tduffy> + dapl_cb_free(&srq_ptr->recv_buffer); tduffy> + /* no need to destroy srq_ptr->header.lock */ tduffy> + tduffy> + kfree(srq_ptr); tduffy> +} tduffy> + tduffy> +/* tduffy> + * dapl_srq_alloc tduffy> + * tduffy> + * alloc and initialize an SRQ struct tduffy> + * tduffy> + * Input: tduffy> + * IA INFO struct ptr tduffy> + * SRQ ATTR ptr tduffy> + * tduffy> + * Output: tduffy> + * none tduffy> + * tduffy> + * Returns: tduffy> + * pointer to srq tduffy> + * tduffy> + */ tduffy> +static struct dapl_srq *dapl_srq_alloc(struct dapl_ia *ia_ptr, tduffy> + const struct dat_srq_attr *srq_attr) tduffy> +{ tduffy> + struct dapl_srq *srq_ptr; tduffy> + tduffy> + /* Allocate SRQ */ tduffy> + srq_ptr = kmalloc(sizeof *srq_ptr, GFP_ATOMIC); tduffy> + if (!srq_ptr) tduffy> + goto bail; tduffy> + tduffy> + /* zero the structure */ tduffy> + memset(srq_ptr, 0, sizeof *srq_ptr); tduffy> + tduffy> + /* tduffy> + * initialize the header tduffy> + */ tduffy> + srq_ptr->header.provider = ia_ptr->header.provider; tduffy> + srq_ptr->header.magic = DAPL_MAGIC_SRQ; tduffy> + srq_ptr->header.handle_type = DAT_HANDLE_TYPE_SRQ; tduffy> + srq_ptr->header.owner_ia = ia_ptr; tduffy> + srq_ptr->header.user_context.as_64 = 0; tduffy> + srq_ptr->header.user_context.as_ptr = NULL; tduffy> + atomic_set(&srq_ptr->srq_ref_count, 0); tduffy> + tduffy> + dapl_llist_init_entry(&srq_ptr->header.ia_list_entry); tduffy> + spin_lock_init(&srq_ptr->header.lock); tduffy> + tduffy> + /* tduffy> + * Initialize the body. tduffy> + * XXX Assume srq_attrs is required tduffy> + */ tduffy> + srq_ptr->param.max_recv_dtos = srq_attr->max_recv_dtos; tduffy> + srq_ptr->param.max_recv_iov = srq_attr->max_recv_iov; tduffy> + srq_ptr->param.low_watermark = srq_attr->low_watermark; tduffy> + tduffy> + /* Get a cookie buffer to track outstanding recvs */ tduffy> + if (DAT_SUCCESS != dapl_cb_create(&srq_ptr->recv_buffer, tduffy> + (struct dapl_ep *)srq_ptr, tduffy> + srq_ptr->param.max_recv_dtos)) { tduffy> + dapl_srq_dealloc(srq_ptr); tduffy> + srq_ptr = NULL; tduffy> + goto bail; tduffy> + } tduffy> + tduffy> +bail: tduffy> + return srq_ptr; tduffy> +} tduffy> + tduffy> +/* tduffy> + * dapl_srq_create tduffy> + * tduffy> + * Create an instance of a Shared Receive Queue that is provided to the tduffy> + * consumer at srq_handle. tduffy> + * tduffy> + * Input: tduffy> + * ia_handle tduffy> + * pz_handle tduffy> + * srq_attr tduffy> + * tduffy> + * Output: tduffy> + * srq_handle tduffy> + * tduffy> + * Returns: tduffy> + * DAT_SUCCESS tduffy> + * DAT_INSUFFICIENT_RESOURCES tduffy> + * DAT_INVALID_HANDLE tduffy> + * DAT_INVALID_PARAMETER tduffy> + * ?DAT_INVALID_ATTRIBUTE?? tduffy> + * DAT_MODEL_NOT_SUPPORTED tduffy> + */ tduffy> +u32 dapl_srq_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, tduffy> + struct dat_srq_attr *srq_attr, DAT_SRQ_HANDLE *srq_handle) tduffy> +{ tduffy> + struct dapl_ia *ia_ptr; tduffy> + struct dapl_srq *srq_ptr; tduffy> + u32 status = DAT_SUCCESS; tduffy> + tduffy> + dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> + "dapl_srq_create (%p, %p, %p, %p)\n", tduffy> + ia_handle, pz_handle, srq_attr, srq_handle); tduffy> + tduffy> + ia_ptr = (struct dapl_ia *)ia_handle; tduffy> + tduffy> + /* tduffy> + * Verify parameters tduffy> + */ tduffy> + if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { tduffy> + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + /* tduffy> + * Verify non-required parameters. tduffy> + * N.B. Assumption: any parameter that can be tduffy> + * modified by dat_ep_modify() is not strictly tduffy> + * required when the EP is created tduffy> + */ tduffy> + if (pz_handle != NULL && DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { tduffy> + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + if (srq_handle == NULL) { tduffy> + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); tduffy> + goto bail; tduffy> + } tduffy> + if ((unsigned long)srq_attr & 3) { tduffy> + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + /* Allocate SRQ */ tduffy> + srq_ptr = dapl_srq_alloc(ia_ptr, srq_attr); tduffy> + if (srq_ptr == NULL) { tduffy> + status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, tduffy> + DAT_RESOURCE_MEMORY); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + srq_ptr->param.ia_handle = (DAT_IA_HANDLE) ia_ptr; tduffy> + srq_ptr->param.srq_state = DAT_SRQ_STATE_OPERATIONAL; tduffy> + srq_ptr->param.pz_handle = pz_handle; tduffy> + tduffy> + /* tduffy> + * XXX Allocate provider resource here!!! tduffy> + */ tduffy> + /* XXX */ status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); tduffy> + /* XXX */ dapl_srq_dealloc(srq_ptr); tduffy> + /* XXX */ goto bail; tduffy> + tduffy> + /* Link it onto the IA */ tduffy> + dapl_ia_link_srq(ia_ptr, srq_ptr); tduffy> + tduffy> + *srq_handle = srq_ptr; tduffy> + tduffy> +bail: tduffy> + return status; tduffy> +} tduffy> + tduffy> +/* tduffy> + * dapl_srq_free tduffy> + * tduffy> + * Destroy an instance of an SRQ tduffy> + * tduffy> + * Input: tduffy> + * srq_handle tduffy> + * tduffy> + * Output: tduffy> + * none tduffy> + * tduffy> + * Returns: tduffy> + * DAT_SUCCESS tduffy> + * DAT_INVALID_PARAMETER tduffy> + * DAT_INVALID_STATE tduffy> + */ tduffy> +u32 dapl_srq_free(DAT_SRQ_HANDLE srq_handle) tduffy> +{ tduffy> + struct dapl_srq *srq_ptr; tduffy> + struct dapl_ia *ia_ptr; tduffy> + struct dat_srq_param *param; tduffy> + u32 status = DAT_SUCCESS; tduffy> + tduffy> + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_free (%p)\n", srq_handle); tduffy> + tduffy> + srq_ptr = (struct dapl_srq *)srq_handle; tduffy> + param = &srq_ptr->param; tduffy> + tduffy> + /* tduffy> + * Verify parameter & state tduffy> + */ tduffy> + if (DAPL_BAD_HANDLE(srq_ptr, DAPL_MAGIC_SRQ)) { tduffy> + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + if (atomic_read(&srq_ptr->srq_ref_count) != 0) { tduffy> + /* tduffy> + * The DAPL 1.2 spec says to return DAT_SRQ_IN_USE, which does tduffy> + * not exist. Have filed the following as an eratta. tduffy> + */ tduffy> + status = DAT_ERROR(DAT_INVALID_STATE, tduffy> + DAT_INVALID_STATE_SRQ_IN_USE); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + ia_ptr = srq_ptr->header.owner_ia; tduffy> + tduffy> + /* tduffy> + * Do verification of parameters and the state change atomically. tduffy> + */ tduffy> + spin_lock_irqsave(&srq_ptr->header.lock, srq_ptr->header.flags); tduffy> + tduffy> + /* Remove the SRQ from the IA */ tduffy> + dapl_ia_unlink_srq(ia_ptr, srq_ptr); tduffy> + tduffy> + spin_unlock_irqrestore(&srq_ptr->header.lock, srq_ptr->header.flags); tduffy> + tduffy> + /* tduffy> + * Finish tearing everything down. tduffy> + */ tduffy> + tduffy> + /* tduffy> + * Take care of the transport resource tduffy> + */ tduffy> + tduffy> + /* XXX Put provider code here!!! */ tduffy> + tduffy> + /* Free the resource */ tduffy> + dapl_srq_dealloc(srq_ptr); tduffy> + tduffy> +bail: tduffy> + return status; tduffy> + tduffy> +} tduffy> + tduffy> +/* tduffy> + * dapl_srq_post_recv tduffy> + * tduffy> + * Post a receive buffer that can be used by any incoming tduffy> + * message by any connected EP using the SRQ. Request to receive data tduffy> + * over a connection of any ep handle into local_iov tduffy> + * tduffy> + * Input: tduffy> + * srq_handle tduffy> + * num_segments tduffy> + * local_iov tduffy> + * user_cookie tduffy> + * tduffy> + * Output: tduffy> + * None. tduffy> + * tduffy> + * Returns: tduffy> + * DAT_SUCCESS tduffy> + * DAT_INSUFFICIENT_RESOURCES tduffy> + * DAT_INVALID_PARAMETER tduffy> + * DAT_INVALID_HANDLE tduffy> + * DAT_INVALID_STATE tduffy> + * DAT_PROTECTION_VIOLATION tduffy> + * DAT_PROVILEGES_VIOLATION tduffy> + */ tduffy> +u32 dapl_srq_post_recv(DAT_SRQ_HANDLE srq_handle, int num_segments, tduffy> + struct dat_lmr_triplet *local_iov, tduffy> + DAT_DTO_COOKIE user_cookie) tduffy> +{ tduffy> + struct dapl_srq *srq_ptr; tduffy> + struct dapl_cookie *cookie; tduffy> + u32 status; tduffy> + tduffy> + dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> + "dapl_srq_post_recv (%p, %d, %p, %P)\n", tduffy> + srq_handle, num_segments, local_iov, user_cookie.as_64); tduffy> + tduffy> + if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { tduffy> + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + srq_ptr = (struct dapl_srq *)srq_handle; tduffy> + tduffy> + /* tduffy> + * Synchronization ok since this buffer is only used for receive tduffy> + * requests, which aren't allowed to race with each other. The tduffy> + * app must syncronize access to the SRQ. tduffy> + */ tduffy> + status = dapl_dto_cookie_alloc(&srq_ptr->recv_buffer, tduffy> + DAPL_DTO_TYPE_RECV, tduffy> + user_cookie, &cookie); tduffy> + if (DAT_SUCCESS != status) tduffy> + goto bail; tduffy> + tduffy> + /* tduffy> + * Take reference before posting to avoid race conditions with tduffy> + * completions tduffy> + */ tduffy> + atomic_inc(&srq_ptr->recv_count); tduffy> + tduffy> + /* tduffy> + * Invoke provider specific routine to post DTO tduffy> + */ tduffy> + /* XXX Put code here XXX */ tduffy> + /* XXX */ status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); tduffy> + tduffy> + if (status != DAT_SUCCESS) { tduffy> + atomic_dec(&srq_ptr->recv_count); tduffy> + dapl_cookie_dealloc(&srq_ptr->recv_buffer, cookie); tduffy> + } tduffy> + tduffy> +bail: tduffy> + return status; tduffy> +} tduffy> + tduffy> +u32 dapl_srq_query(DAT_SRQ_HANDLE srq_handle, struct dat_srq_param *srq_param) tduffy> +{ tduffy> + struct dapl_srq *srq_ptr; tduffy> + u32 status = DAT_SUCCESS; tduffy> + tduffy> + dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> + "dapl_srq_query (%p, %x, %p)\n", tduffy> + srq_handle, srq_param); tduffy> + tduffy> + if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { tduffy> + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); tduffy> + goto bail; tduffy> + } tduffy> + if (srq_param == NULL) { tduffy> + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + srq_ptr = (struct dapl_srq *)srq_handle; tduffy> + tduffy> + /* tduffy> + * XXX Need to calculate available_dto_count and outstanding_dto_count tduffy> + */ tduffy> + srq_ptr->param.available_dto_count = DAT_VALUE_UNKNOWN; tduffy> + srq_ptr->param.outstanding_dto_count = DAT_VALUE_UNKNOWN; tduffy> + tduffy> + *srq_param = srq_ptr->param; tduffy> + tduffy> +bail: tduffy> + return status; tduffy> +} tduffy> + tduffy> +/* tduffy> + * dapl_srq_resize tduffy> + * tduffy> + * Modify the size fo the event queue of a Shared Recieve Queue tduffy> + * tduffy> + * Input: tduffy> + * srq_handle tduffy> + * srq_max_recv_dto tduffy> + * tduffy> + * Output: tduffy> + * none tduffy> + * tduffy> + * Returns: tduffy> + * DAT_SUCCESS tduffy> + * DAT_INVALID_HANDLE tduffy> + * DAT_INVALID_PARAMETER tduffy> + * DAT_INSUFFICIENT_RESOURCES tduffy> + * DAT_INVALID_STATE tduffy> + */ tduffy> + tduffy> +u32 dapl_srq_resize(DAT_SRQ_HANDLE srq_handle, int srq_max_recv_dto) tduffy> +{ tduffy> + struct dapl_ia *ia_ptr; tduffy> + struct dapl_srq *srq_ptr; tduffy> + u32 status = DAT_SUCCESS; tduffy> + tduffy> + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_resize (%p, %d)\n", tduffy> + srq_handle, srq_max_recv_dto); tduffy> + tduffy> + if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { tduffy> + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + srq_ptr = (struct dapl_srq *)srq_handle; tduffy> + ia_ptr = srq_ptr->header.owner_ia; tduffy> + tduffy> + /* tduffy> + * Check for nonsense requests per the spec tduffy> + */ tduffy> + if (srq_max_recv_dto <= srq_ptr->param.low_watermark) { tduffy> + status = DAT_ERROR(DAT_INVALID_STATE, DAT_NO_SUBTYPE); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + /* XXX Put implementation here XXX */ tduffy> + tduffy> + /* XXX */ status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); tduffy> + tduffy> +bail: tduffy> + return status; tduffy> +} tduffy> + tduffy> +/* tduffy> + * dapl_srq_set_lw tduffy> + * tduffy> + * DAPL Requirements Version 1.2, 6.5.4 tduffy> + * tduffy> + * Set the low water mark for an SRQ and arm the SRQ to generate an tduffy> + * event if it is reached. tduffy> + * tduffy> + * Input: tduffy> + * srq_handle tduffy> + * srq_max_recv_dto tduffy> + * tduffy> + * Output: tduffy> + * none tduffy> + * tduffy> + * Returns: tduffy> + * DAT_SUCCESS tduffy> + * DAT_INVALID_HANDLE tduffy> + * DAT_INVALID_PARAMETER tduffy> + * DAT_MODEL_NOT_SUPPORTED tduffy> + */ tduffy> + tduffy> +u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, int low_watermark) tduffy> +{ tduffy> + struct dapl_srq *srq_ptr; tduffy> + u32 status = DAT_SUCCESS; tduffy> + tduffy> + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_set_lw (%p, %d)\n", tduffy> + srq_handle, low_watermark); tduffy> + tduffy> + if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { tduffy> + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + srq_ptr = (struct dapl_srq *)srq_handle; tduffy> + tduffy> + /* XXX Put implementation here XXX */ tduffy> + tduffy> + /* XXX */ status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); tduffy> + tduffy> +bail: tduffy> + return status; tduffy> +} tduffy> Index: linux-kernel-srq/dat-provider/dapl_srq_resize.c tduffy> =================================================================== tduffy> --- linux-kernel-srq/dat-provider/dapl_srq_resize.c (revision 2489) tduffy> +++ linux-kernel-srq/dat-provider/dapl_srq_resize.c (working copy) tduffy> @@ -1,88 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/* tduffy> - * $Id$ tduffy> - */ tduffy> - tduffy> -#include "dapl.h" tduffy> -#include "dapl_srq_util.h" tduffy> -#include "dapl_openib_util.h" tduffy> - tduffy> -/* tduffy> - * dapl_srq_resize tduffy> - * tduffy> - * Modify the size fo the event queue of a Shared Recieve Queue tduffy> - * tduffy> - * Input: tduffy> - * srq_handle tduffy> - * srq_max_recv_dto tduffy> - * tduffy> - * Output: tduffy> - * none tduffy> - * tduffy> - * Returns: tduffy> - * DAT_SUCCESS tduffy> - * DAT_INVALID_HANDLE tduffy> - * DAT_INVALID_PARAMETER tduffy> - * DAT_INSUFFICIENT_RESOURCES tduffy> - * DAT_INVALID_STATE tduffy> - */ tduffy> - tduffy> -u32 dapl_srq_resize(DAT_SRQ_HANDLE srq_handle, int srq_max_recv_dto) tduffy> -{ tduffy> - struct dapl_ia *ia_ptr; tduffy> - struct dapl_srq *srq_ptr; tduffy> - u32 dat_status = DAT_SUCCESS; tduffy> - tduffy> - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_resize (%p, %d)\n", tduffy> - srq_handle, srq_max_recv_dto); tduffy> - tduffy> - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - srq_ptr = (struct dapl_srq *)srq_handle; tduffy> - ia_ptr = srq_ptr->header.owner_ia; tduffy> - tduffy> - /* tduffy> - * Check for nonsense requests per the spec tduffy> - */ tduffy> - if (srq_max_recv_dto <= srq_ptr->param.low_watermark) { tduffy> - dat_status = DAT_ERROR(DAT_INVALID_STATE, DAT_NO_SUBTYPE); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - /* XXX Put implementation here XXX */ tduffy> - tduffy> - /* XXX */ dat_status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); tduffy> - tduffy> - bail: tduffy> - return dat_status; tduffy> -} tduffy> Index: linux-kernel-srq/dat-provider/dapl_srq_create.c tduffy> =================================================================== tduffy> --- linux-kernel-srq/dat-provider/dapl_srq_create.c (revision 2489) tduffy> +++ linux-kernel-srq/dat-provider/dapl_srq_create.c (working copy) tduffy> @@ -1,128 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/* tduffy> - * $Id$ tduffy> - */ tduffy> - tduffy> -#include "dapl.h" tduffy> -#include "dapl_ia_util.h" tduffy> -#include "dapl_srq_util.h" tduffy> -#include "dapl_openib_util.h" tduffy> - tduffy> -/* tduffy> - * dapl_srq_create tduffy> - * tduffy> - * Create an instance of a Shared Receive Queue that is provided to the tduffy> - * consumer at srq_handle. tduffy> - * tduffy> - * Input: tduffy> - * ia_handle tduffy> - * pz_handle tduffy> - * srq_attr tduffy> - * tduffy> - * Output: tduffy> - * srq_handle tduffy> - * tduffy> - * Returns: tduffy> - * DAT_SUCCESS tduffy> - * DAT_INSUFFICIENT_RESOURCES tduffy> - * DAT_INVALID_HANDLE tduffy> - * DAT_INVALID_PARAMETER tduffy> - * ?DAT_INVALID_ATTRIBUTE?? tduffy> - * DAT_MODEL_NOT_SUPPORTED tduffy> - */ tduffy> -u32 dapl_srq_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, tduffy> - struct dat_srq_attr *srq_attr, DAT_SRQ_HANDLE *srq_handle) tduffy> -{ tduffy> - struct dapl_ia *ia_ptr; tduffy> - struct dapl_srq *srq_ptr; tduffy> - u32 dat_status = DAT_SUCCESS; tduffy> - tduffy> - dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> - "dapl_srq_create (%p, %p, %p, %p)\n", tduffy> - ia_handle, pz_handle, srq_attr, srq_handle); tduffy> - tduffy> - ia_ptr = (struct dapl_ia *)ia_handle; tduffy> - tduffy> - /* tduffy> - * Verify parameters tduffy> - */ tduffy> - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - /* tduffy> - * Verify non-required parameters. tduffy> - * N.B. Assumption: any parameter that can be tduffy> - * modified by dat_ep_modify() is not strictly tduffy> - * required when the EP is created tduffy> - */ tduffy> - if (pz_handle != NULL && DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - if (srq_handle == NULL) { tduffy> - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); tduffy> - goto bail; tduffy> - } tduffy> - if ((unsigned long)srq_attr & 3) { tduffy> - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - /* Allocate SRQ */ tduffy> - srq_ptr = dapl_srq_alloc(ia_ptr, srq_attr); tduffy> - if (srq_ptr == NULL) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - srq_ptr->param.ia_handle = (DAT_IA_HANDLE) ia_ptr; tduffy> - srq_ptr->param.srq_state = DAT_SRQ_STATE_OPERATIONAL; tduffy> - srq_ptr->param.pz_handle = pz_handle; tduffy> - tduffy> - /* tduffy> - * XXX Allocate provider resource here!!! tduffy> - */ tduffy> - /* XXX */ dat_status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); tduffy> - /* XXX */ dapl_srq_dealloc(srq_ptr); tduffy> - /* XXX */ goto bail; tduffy> - tduffy> - /* Link it onto the IA */ tduffy> - dapl_ia_link_srq(ia_ptr, srq_ptr); tduffy> - tduffy> - *srq_handle = srq_ptr; tduffy> - tduffy> - bail: tduffy> - return dat_status; tduffy> -} tduffy> Index: linux-kernel-srq/dat-provider/dapl_srq_query.c tduffy> =================================================================== tduffy> --- linux-kernel-srq/dat-provider/dapl_srq_query.c (revision 2489) tduffy> +++ linux-kernel-srq/dat-provider/dapl_srq_query.c (working copy) tduffy> @@ -1,65 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/* tduffy> - * $Id$ tduffy> - */ tduffy> - tduffy> -#include "dapl.h" tduffy> - tduffy> -u32 dapl_srq_query(DAT_SRQ_HANDLE srq_handle, struct dat_srq_param *srq_param) tduffy> -{ tduffy> - struct dapl_srq *srq_ptr; tduffy> - u32 dat_status = DAT_SUCCESS; tduffy> - tduffy> - dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> - "dapl_srq_query (%p, %x, %p)\n", tduffy> - srq_handle, srq_param); tduffy> - tduffy> - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); tduffy> - goto bail; tduffy> - } tduffy> - if (srq_param == NULL) { tduffy> - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - srq_ptr = (struct dapl_srq *)srq_handle; tduffy> - tduffy> - /* tduffy> - * XXX Need to calculate available_dto_count and outstanding_dto_count tduffy> - */ tduffy> - srq_ptr->param.available_dto_count = DAT_VALUE_UNKNOWN; tduffy> - srq_ptr->param.outstanding_dto_count = DAT_VALUE_UNKNOWN; tduffy> - tduffy> - *srq_param = srq_ptr->param; tduffy> - tduffy> - bail: tduffy> - return dat_status; tduffy> -} tduffy> Index: linux-kernel-srq/dat-provider/Makefile tduffy> =================================================================== tduffy> --- linux-kernel-srq/dat-provider/Makefile (revision 2489) tduffy> +++ linux-kernel-srq/dat-provider/Makefile (working copy) tduffy> @@ -82,13 +82,7 @@ PROVIDER_MODULES := \ tduffy> dapl_rsp_query \ tduffy> dapl_set_consumer_context \ tduffy> dapl_sp_util \ tduffy> - dapl_srq_create \ tduffy> - dapl_srq_free \ tduffy> - dapl_srq_post_recv \ tduffy> - dapl_srq_query \ tduffy> - dapl_srq_resize \ tduffy> - dapl_srq_set_lw \ tduffy> - dapl_srq_util \ tduffy> + dapl_srq \ tduffy> dapl_timer_util \ tduffy> dapl_util tduffy> tduffy> Index: linux-kernel-srq/dat-provider/dapl_srq_util.c tduffy> =================================================================== tduffy> --- linux-kernel-srq/dat-provider/dapl_srq_util.c (revision 2489) tduffy> +++ linux-kernel-srq/dat-provider/dapl_srq_util.c (working copy) tduffy> @@ -1,126 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/* tduffy> - * $Id$ tduffy> - */ tduffy> - tduffy> -#include "dapl_srq_util.h" tduffy> -#include "dapl_ia_util.h" tduffy> -#include "dapl_cookie.h" tduffy> - tduffy> -/* tduffy> - * dapl_srq_alloc tduffy> - * tduffy> - * alloc and initialize an SRQ struct tduffy> - * tduffy> - * Input: tduffy> - * IA INFO struct ptr tduffy> - * SRQ ATTR ptr tduffy> - * tduffy> - * Output: tduffy> - * none tduffy> - * tduffy> - * Returns: tduffy> - * pointer to srq tduffy> - * tduffy> - */ tduffy> -struct dapl_srq *dapl_srq_alloc(struct dapl_ia *ia_ptr, tduffy> - const struct dat_srq_attr *srq_attr) tduffy> -{ tduffy> - struct dapl_srq *srq_ptr; tduffy> - tduffy> - /* Allocate SRQ */ tduffy> - srq_ptr = kmalloc(sizeof *srq_ptr, GFP_ATOMIC); tduffy> - if (!srq_ptr) tduffy> - goto bail; tduffy> - tduffy> - /* zero the structure */ tduffy> - memset(srq_ptr, 0, sizeof *srq_ptr); tduffy> - tduffy> - /* tduffy> - * initialize the header tduffy> - */ tduffy> - srq_ptr->header.provider = ia_ptr->header.provider; tduffy> - srq_ptr->header.magic = DAPL_MAGIC_SRQ; tduffy> - srq_ptr->header.handle_type = DAT_HANDLE_TYPE_SRQ; tduffy> - srq_ptr->header.owner_ia = ia_ptr; tduffy> - srq_ptr->header.user_context.as_64 = 0; tduffy> - srq_ptr->header.user_context.as_ptr = NULL; tduffy> - atomic_set(&srq_ptr->srq_ref_count, 0); tduffy> - tduffy> - dapl_llist_init_entry(&srq_ptr->header.ia_list_entry); tduffy> - spin_lock_init(&srq_ptr->header.lock); tduffy> - tduffy> - /* tduffy> - * Initialize the body. tduffy> - * XXX Assume srq_attrs is required tduffy> - */ tduffy> - srq_ptr->param.max_recv_dtos = srq_attr->max_recv_dtos; tduffy> - srq_ptr->param.max_recv_iov = srq_attr->max_recv_iov; tduffy> - srq_ptr->param.low_watermark = srq_attr->low_watermark; tduffy> - tduffy> - /* Get a cookie buffer to track outstanding recvs */ tduffy> - if (DAT_SUCCESS != dapl_cb_create(&srq_ptr->recv_buffer, tduffy> - (struct dapl_ep *)srq_ptr, tduffy> - srq_ptr->param.max_recv_dtos)) { tduffy> - dapl_srq_dealloc(srq_ptr); tduffy> - srq_ptr = NULL; tduffy> - goto bail; tduffy> - } tduffy> - tduffy> -bail: tduffy> - return srq_ptr; tduffy> -} tduffy> - tduffy> -/* tduffy> - * dapl_srq_dealloc tduffy> - * tduffy> - * Free the passed in SRQ structure. tduffy> - * tduffy> - * Input: tduffy> - * SRQ pointer tduffy> - * tduffy> - * Output: tduffy> - * none tduffy> - * tduffy> - * Returns: tduffy> - * none tduffy> - * tduffy> - */ tduffy> -void dapl_srq_dealloc(struct dapl_srq *srq_ptr) tduffy> -{ tduffy> - dapl_os_assert(srq_ptr->header.magic == DAPL_MAGIC_SRQ); tduffy> - tduffy> - /* reset magic to prevent reuse */ tduffy> - srq_ptr->header.magic = DAPL_MAGIC_INVALID; tduffy> - dapl_ia_unlink_srq(srq_ptr->header.owner_ia, srq_ptr); tduffy> - dapl_cb_free(&srq_ptr->recv_buffer); tduffy> - /* no need to destroy srq_ptr->header.lock */ tduffy> - tduffy> - kfree(srq_ptr); tduffy> -} tduffy> Index: linux-kernel-srq/dat-provider/dapl_srq_util.h tduffy> =================================================================== tduffy> --- linux-kernel-srq/dat-provider/dapl_srq_util.h (revision 2489) tduffy> +++ linux-kernel-srq/dat-provider/dapl_srq_util.h (working copy) tduffy> @@ -1,47 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/********************************************************************** tduffy> - * tduffy> - * HEADER: dapl_srq_util.h tduffy> - * tduffy> - * PURPOSE: Utility defs & routines for the SRQ data structure tduffy> - * tduffy> - * $Id$ tduffy> - **********************************************************************/ tduffy> - tduffy> -#ifndef DAPL_SRQ_UTIL_H tduffy> -#define DAPL_SRQ_UTIL_H tduffy> - tduffy> -#include "dapl.h" tduffy> - tduffy> -extern struct dapl_srq *dapl_srq_alloc(struct dapl_ia *ia, tduffy> - const struct dat_srq_attr *srq_attr); tduffy> - tduffy> -extern void dapl_srq_dealloc(struct dapl_srq *srq_ptr); tduffy> - tduffy> -#endif /* DAPL_SRQ_UTIL_H */ tduffy> Index: linux-kernel-srq/dat-provider/dapl_srq_set_lw.c tduffy> =================================================================== tduffy> --- linux-kernel-srq/dat-provider/dapl_srq_set_lw.c (revision 2489) tduffy> +++ linux-kernel-srq/dat-provider/dapl_srq_set_lw.c (working copy) tduffy> @@ -1,80 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/* tduffy> - * $Id$ tduffy> - */ tduffy> - tduffy> -#include "dapl.h" tduffy> -#include "dapl_srq_util.h" tduffy> -#include "dapl_openib_util.h" tduffy> - tduffy> -/* tduffy> - * dapl_srq_set_lw tduffy> - * tduffy> - * DAPL Requirements Version 1.2, 6.5.4 tduffy> - * tduffy> - * Set the low water mark for an SRQ and arm the SRQ to generate an tduffy> - * event if it is reached. tduffy> - * tduffy> - * Input: tduffy> - * srq_handle tduffy> - * srq_max_recv_dto tduffy> - * tduffy> - * Output: tduffy> - * none tduffy> - * tduffy> - * Returns: tduffy> - * DAT_SUCCESS tduffy> - * DAT_INVALID_HANDLE tduffy> - * DAT_INVALID_PARAMETER tduffy> - * DAT_MODEL_NOT_SUPPORTED tduffy> - */ tduffy> - tduffy> -u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, int low_watermark) tduffy> -{ tduffy> - struct dapl_srq *srq_ptr; tduffy> - u32 dat_status = DAT_SUCCESS; tduffy> - tduffy> - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_set_lw (%p, %d)\n", tduffy> - srq_handle, low_watermark); tduffy> - tduffy> - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - srq_ptr = (struct dapl_srq *)srq_handle; tduffy> - tduffy> - /* XXX Put implementation here XXX */ tduffy> - tduffy> - /* XXX */ dat_status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); tduffy> - tduffy> - bail: tduffy> - return dat_status; tduffy> -} tduffy> Index: linux-kernel-srq/dat-provider/dapl_srq_post_recv.c tduffy> =================================================================== tduffy> --- linux-kernel-srq/dat-provider/dapl_srq_post_recv.c (revision 2489) tduffy> +++ linux-kernel-srq/dat-provider/dapl_srq_post_recv.c (working copy) tduffy> @@ -1,112 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/* tduffy> - * $Id$ tduffy> - */ tduffy> - tduffy> -#include "dapl.h" tduffy> -#include "dapl_cookie.h" tduffy> -#include "dapl_openib_util.h" tduffy> - tduffy> -/* tduffy> - * dapl_srq_post_recv tduffy> - * tduffy> - * Post a receive buffer that can be used by any incoming tduffy> - * message by any connected EP using the SRQ. Request to receive data tduffy> - * over a connection of any ep handle into local_iov tduffy> - * tduffy> - * Input: tduffy> - * srq_handle tduffy> - * num_segments tduffy> - * local_iov tduffy> - * user_cookie tduffy> - * tduffy> - * Output: tduffy> - * None. tduffy> - * tduffy> - * Returns: tduffy> - * DAT_SUCCESS tduffy> - * DAT_INSUFFICIENT_RESOURCES tduffy> - * DAT_INVALID_PARAMETER tduffy> - * DAT_INVALID_HANDLE tduffy> - * DAT_INVALID_STATE tduffy> - * DAT_PROTECTION_VIOLATION tduffy> - * DAT_PROVILEGES_VIOLATION tduffy> - */ tduffy> -u32 dapl_srq_post_recv(DAT_SRQ_HANDLE srq_handle, int num_segments, tduffy> - struct dat_lmr_triplet *local_iov, tduffy> - DAT_DTO_COOKIE user_cookie) tduffy> -{ tduffy> - struct dapl_srq *srq_ptr; tduffy> - struct dapl_cookie *cookie; tduffy> - u32 dat_status; tduffy> - tduffy> - dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> - "dapl_srq_post_recv (%p, %d, %p, %P)\n", tduffy> - srq_handle, num_segments, local_iov, user_cookie.as_64); tduffy> - tduffy> - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - srq_ptr = (struct dapl_srq *)srq_handle; tduffy> - tduffy> - /* tduffy> - * Synchronization ok since this buffer is only used for receive tduffy> - * requests, which aren't allowed to race with each other. The tduffy> - * app must syncronize access to the SRQ. tduffy> - */ tduffy> - dat_status = dapl_dto_cookie_alloc(&srq_ptr->recv_buffer, tduffy> - DAPL_DTO_TYPE_RECV, tduffy> - user_cookie, &cookie); tduffy> - if (DAT_SUCCESS != dat_status) { tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - /* tduffy> - * Take reference before posting to avoid race conditions with tduffy> - * completions tduffy> - */ tduffy> - atomic_inc(&srq_ptr->recv_count); tduffy> - tduffy> - /* tduffy> - * Invoke provider specific routine to post DTO tduffy> - */ tduffy> - /* XXX Put code here XXX */ tduffy> - /* XXX */ dat_status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); tduffy> - tduffy> - if (dat_status != DAT_SUCCESS) { tduffy> - atomic_dec(&srq_ptr->recv_count); tduffy> - dapl_cookie_dealloc(&srq_ptr->recv_buffer, cookie); tduffy> - } tduffy> - tduffy> - bail: tduffy> - return dat_status; tduffy> -} tduffy> Index: linux-kernel-srq/dat-provider/dapl_srq_free.c tduffy> =================================================================== tduffy> --- linux-kernel-srq/dat-provider/dapl_srq_free.c (revision 2489) tduffy> +++ linux-kernel-srq/dat-provider/dapl_srq_free.c (working copy) tduffy> @@ -1,112 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/* tduffy> - * $Id$ tduffy> - */ tduffy> - tduffy> -#include "dapl.h" tduffy> -#include "dapl_ia_util.h" tduffy> -#include "dapl_srq_util.h" tduffy> -#include "dapl_openib_util.h" tduffy> - tduffy> -/* tduffy> - * dapl_srq_free tduffy> - * tduffy> - * Destroy an instance of an SRQ tduffy> - * tduffy> - * Input: tduffy> - * srq_handle tduffy> - * tduffy> - * Output: tduffy> - * none tduffy> - * tduffy> - * Returns: tduffy> - * DAT_SUCCESS tduffy> - * DAT_INVALID_PARAMETER tduffy> - * DAT_INVALID_STATE tduffy> - */ tduffy> -u32 dapl_srq_free(DAT_SRQ_HANDLE srq_handle) tduffy> -{ tduffy> - struct dapl_srq *srq_ptr; tduffy> - struct dapl_ia *ia_ptr; tduffy> - struct dat_srq_param *param; tduffy> - u32 dat_status = DAT_SUCCESS; tduffy> - tduffy> - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_free (%p)\n", srq_handle); tduffy> - tduffy> - srq_ptr = (struct dapl_srq *)srq_handle; tduffy> - param = &srq_ptr->param; tduffy> - tduffy> - /* tduffy> - * Verify parameter & state tduffy> - */ tduffy> - if (DAPL_BAD_HANDLE(srq_ptr, DAPL_MAGIC_SRQ)) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - if (atomic_read(&srq_ptr->srq_ref_count) != 0) { tduffy> - /* tduffy> - * The DAPL 1.2 spec says to return DAT_SRQ_IN_USE, which does tduffy> - * not exist. Have filed the following as an eratta. tduffy> - */ tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_SRQ_IN_USE); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - ia_ptr = srq_ptr->header.owner_ia; tduffy> - tduffy> - /* tduffy> - * Do verification of parameters and the state change atomically. tduffy> - */ tduffy> - spin_lock_irqsave(&srq_ptr->header.lock, srq_ptr->header.flags); tduffy> - tduffy> - /* Remove the SRQ from the IA */ tduffy> - dapl_ia_unlink_srq(ia_ptr, srq_ptr); tduffy> - tduffy> - spin_unlock_irqrestore(&srq_ptr->header.lock, srq_ptr->header.flags); tduffy> - tduffy> - /* tduffy> - * Finish tearing everything down. tduffy> - */ tduffy> - tduffy> - /* tduffy> - * Take care of the transport resource tduffy> - */ tduffy> - tduffy> - /* XXX Put provider code here!!! */ tduffy> - tduffy> - /* Free the resource */ tduffy> - dapl_srq_dealloc(srq_ptr); tduffy> - tduffy> - bail: tduffy> - return dat_status; tduffy> - tduffy> -} tduffy> Index: linux-kernel-srq/patches/alt_dat_provider_makefile tduffy> =================================================================== tduffy> --- linux-kernel-srq/patches/alt_dat_provider_makefile (revision 2489) tduffy> +++ linux-kernel-srq/patches/alt_dat_provider_makefile (working copy) tduffy> @@ -76,13 +76,7 @@ PROVIDER_MODULES := \ tduffy> dapl_rsp_query \ tduffy> dapl_set_consumer_context \ tduffy> dapl_sp_util \ tduffy> - dapl_srq_create \ tduffy> - dapl_srq_free \ tduffy> - dapl_srq_post_recv \ tduffy> - dapl_srq_query \ tduffy> - dapl_srq_resize \ tduffy> - dapl_srq_set_lw \ tduffy> - dapl_srq_util \ tduffy> + dapl_srq \ tduffy> dapl_timer_util \ tduffy> dapl_util tduffy> tduffy> From jlentini at netapp.com Fri May 27 08:16:02 2005 From: jlentini at netapp.com (James Lentini) Date: Fri, 27 May 2005 11:16:02 -0400 (EDT) Subject: [openib-general] Re: [PATCH 1/3] kDAPL: consolidate dapl_srq*.c into one file In-Reply-To: <1117133253.17831.12.camel@duffman> References: <1117133253.17831.12.camel@duffman> Message-ID: Committed in revision 2501. On Thu, 26 May 2005, Tom Duffy wrote: tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> Index: linux-kernel-srq/dat-provider/dapl_srq.c tduffy> =================================================================== tduffy> --- linux-kernel-srq/dat-provider/dapl_srq.c (revision 0) tduffy> +++ linux-kernel-srq/dat-provider/dapl_srq.c (revision 0) tduffy> @@ -0,0 +1,500 @@ tduffy> +/* tduffy> + * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * tduffy> + * This Software is licensed under one of the following licenses: tduffy> + * tduffy> + * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> + * available from the Open Source Initiative, see tduffy> + * http://www.opensource.org/licenses/cpl.php. tduffy> + * tduffy> + * 2) under the terms of the "The BSD License" a copy of which is tduffy> + * available from the Open Source Initiative, see tduffy> + * http://www.opensource.org/licenses/bsd-license.php. tduffy> + * tduffy> + * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> + * copy of which is available from the Open Source Initiative, see tduffy> + * http://www.opensource.org/licenses/gpl-license.php. tduffy> + * tduffy> + * Licensee has the right to choose one of the above licenses. tduffy> + * tduffy> + * Redistributions of source code must retain the above copyright tduffy> + * notice and one of the license notices. tduffy> + * tduffy> + * Redistributions in binary form must reproduce both the above copyright tduffy> + * notice, one of the license notices in the documentation tduffy> + * and/or other materials provided with the distribution. tduffy> + */ tduffy> + tduffy> +/* tduffy> + * $Id$ tduffy> + */ tduffy> + tduffy> +#include "dapl.h" tduffy> +#include "dapl_ia_util.h" tduffy> +#include "dapl_openib_util.h" tduffy> +#include "dapl_cookie.h" tduffy> + tduffy> +/* tduffy> + * dapl_srq_dealloc tduffy> + * tduffy> + * Free the passed in SRQ structure. tduffy> + * tduffy> + * Input: tduffy> + * SRQ pointer tduffy> + * tduffy> + * Output: tduffy> + * none tduffy> + * tduffy> + * Returns: tduffy> + * none tduffy> + * tduffy> + */ tduffy> +static void dapl_srq_dealloc(struct dapl_srq *srq_ptr) tduffy> +{ tduffy> + dapl_os_assert(srq_ptr->header.magic == DAPL_MAGIC_SRQ); tduffy> + tduffy> + /* reset magic to prevent reuse */ tduffy> + srq_ptr->header.magic = DAPL_MAGIC_INVALID; tduffy> + dapl_ia_unlink_srq(srq_ptr->header.owner_ia, srq_ptr); tduffy> + dapl_cb_free(&srq_ptr->recv_buffer); tduffy> + /* no need to destroy srq_ptr->header.lock */ tduffy> + tduffy> + kfree(srq_ptr); tduffy> +} tduffy> + tduffy> +/* tduffy> + * dapl_srq_alloc tduffy> + * tduffy> + * alloc and initialize an SRQ struct tduffy> + * tduffy> + * Input: tduffy> + * IA INFO struct ptr tduffy> + * SRQ ATTR ptr tduffy> + * tduffy> + * Output: tduffy> + * none tduffy> + * tduffy> + * Returns: tduffy> + * pointer to srq tduffy> + * tduffy> + */ tduffy> +static struct dapl_srq *dapl_srq_alloc(struct dapl_ia *ia_ptr, tduffy> + const struct dat_srq_attr *srq_attr) tduffy> +{ tduffy> + struct dapl_srq *srq_ptr; tduffy> + tduffy> + /* Allocate SRQ */ tduffy> + srq_ptr = kmalloc(sizeof *srq_ptr, GFP_ATOMIC); tduffy> + if (!srq_ptr) tduffy> + goto bail; tduffy> + tduffy> + /* zero the structure */ tduffy> + memset(srq_ptr, 0, sizeof *srq_ptr); tduffy> + tduffy> + /* tduffy> + * initialize the header tduffy> + */ tduffy> + srq_ptr->header.provider = ia_ptr->header.provider; tduffy> + srq_ptr->header.magic = DAPL_MAGIC_SRQ; tduffy> + srq_ptr->header.handle_type = DAT_HANDLE_TYPE_SRQ; tduffy> + srq_ptr->header.owner_ia = ia_ptr; tduffy> + srq_ptr->header.user_context.as_64 = 0; tduffy> + srq_ptr->header.user_context.as_ptr = NULL; tduffy> + atomic_set(&srq_ptr->srq_ref_count, 0); tduffy> + tduffy> + dapl_llist_init_entry(&srq_ptr->header.ia_list_entry); tduffy> + spin_lock_init(&srq_ptr->header.lock); tduffy> + tduffy> + /* tduffy> + * Initialize the body. tduffy> + * XXX Assume srq_attrs is required tduffy> + */ tduffy> + srq_ptr->param.max_recv_dtos = srq_attr->max_recv_dtos; tduffy> + srq_ptr->param.max_recv_iov = srq_attr->max_recv_iov; tduffy> + srq_ptr->param.low_watermark = srq_attr->low_watermark; tduffy> + tduffy> + /* Get a cookie buffer to track outstanding recvs */ tduffy> + if (DAT_SUCCESS != dapl_cb_create(&srq_ptr->recv_buffer, tduffy> + (struct dapl_ep *)srq_ptr, tduffy> + srq_ptr->param.max_recv_dtos)) { tduffy> + dapl_srq_dealloc(srq_ptr); tduffy> + srq_ptr = NULL; tduffy> + goto bail; tduffy> + } tduffy> + tduffy> +bail: tduffy> + return srq_ptr; tduffy> +} tduffy> + tduffy> +/* tduffy> + * dapl_srq_create tduffy> + * tduffy> + * Create an instance of a Shared Receive Queue that is provided to the tduffy> + * consumer at srq_handle. tduffy> + * tduffy> + * Input: tduffy> + * ia_handle tduffy> + * pz_handle tduffy> + * srq_attr tduffy> + * tduffy> + * Output: tduffy> + * srq_handle tduffy> + * tduffy> + * Returns: tduffy> + * DAT_SUCCESS tduffy> + * DAT_INSUFFICIENT_RESOURCES tduffy> + * DAT_INVALID_HANDLE tduffy> + * DAT_INVALID_PARAMETER tduffy> + * ?DAT_INVALID_ATTRIBUTE?? tduffy> + * DAT_MODEL_NOT_SUPPORTED tduffy> + */ tduffy> +u32 dapl_srq_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, tduffy> + struct dat_srq_attr *srq_attr, DAT_SRQ_HANDLE *srq_handle) tduffy> +{ tduffy> + struct dapl_ia *ia_ptr; tduffy> + struct dapl_srq *srq_ptr; tduffy> + u32 status = DAT_SUCCESS; tduffy> + tduffy> + dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> + "dapl_srq_create (%p, %p, %p, %p)\n", tduffy> + ia_handle, pz_handle, srq_attr, srq_handle); tduffy> + tduffy> + ia_ptr = (struct dapl_ia *)ia_handle; tduffy> + tduffy> + /* tduffy> + * Verify parameters tduffy> + */ tduffy> + if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { tduffy> + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + /* tduffy> + * Verify non-required parameters. tduffy> + * N.B. Assumption: any parameter that can be tduffy> + * modified by dat_ep_modify() is not strictly tduffy> + * required when the EP is created tduffy> + */ tduffy> + if (pz_handle != NULL && DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { tduffy> + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + if (srq_handle == NULL) { tduffy> + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); tduffy> + goto bail; tduffy> + } tduffy> + if ((unsigned long)srq_attr & 3) { tduffy> + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + /* Allocate SRQ */ tduffy> + srq_ptr = dapl_srq_alloc(ia_ptr, srq_attr); tduffy> + if (srq_ptr == NULL) { tduffy> + status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, tduffy> + DAT_RESOURCE_MEMORY); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + srq_ptr->param.ia_handle = (DAT_IA_HANDLE) ia_ptr; tduffy> + srq_ptr->param.srq_state = DAT_SRQ_STATE_OPERATIONAL; tduffy> + srq_ptr->param.pz_handle = pz_handle; tduffy> + tduffy> + /* tduffy> + * XXX Allocate provider resource here!!! tduffy> + */ tduffy> + /* XXX */ status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); tduffy> + /* XXX */ dapl_srq_dealloc(srq_ptr); tduffy> + /* XXX */ goto bail; tduffy> + tduffy> + /* Link it onto the IA */ tduffy> + dapl_ia_link_srq(ia_ptr, srq_ptr); tduffy> + tduffy> + *srq_handle = srq_ptr; tduffy> + tduffy> +bail: tduffy> + return status; tduffy> +} tduffy> + tduffy> +/* tduffy> + * dapl_srq_free tduffy> + * tduffy> + * Destroy an instance of an SRQ tduffy> + * tduffy> + * Input: tduffy> + * srq_handle tduffy> + * tduffy> + * Output: tduffy> + * none tduffy> + * tduffy> + * Returns: tduffy> + * DAT_SUCCESS tduffy> + * DAT_INVALID_PARAMETER tduffy> + * DAT_INVALID_STATE tduffy> + */ tduffy> +u32 dapl_srq_free(DAT_SRQ_HANDLE srq_handle) tduffy> +{ tduffy> + struct dapl_srq *srq_ptr; tduffy> + struct dapl_ia *ia_ptr; tduffy> + struct dat_srq_param *param; tduffy> + u32 status = DAT_SUCCESS; tduffy> + tduffy> + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_free (%p)\n", srq_handle); tduffy> + tduffy> + srq_ptr = (struct dapl_srq *)srq_handle; tduffy> + param = &srq_ptr->param; tduffy> + tduffy> + /* tduffy> + * Verify parameter & state tduffy> + */ tduffy> + if (DAPL_BAD_HANDLE(srq_ptr, DAPL_MAGIC_SRQ)) { tduffy> + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + if (atomic_read(&srq_ptr->srq_ref_count) != 0) { tduffy> + /* tduffy> + * The DAPL 1.2 spec says to return DAT_SRQ_IN_USE, which does tduffy> + * not exist. Have filed the following as an eratta. tduffy> + */ tduffy> + status = DAT_ERROR(DAT_INVALID_STATE, tduffy> + DAT_INVALID_STATE_SRQ_IN_USE); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + ia_ptr = srq_ptr->header.owner_ia; tduffy> + tduffy> + /* tduffy> + * Do verification of parameters and the state change atomically. tduffy> + */ tduffy> + spin_lock_irqsave(&srq_ptr->header.lock, srq_ptr->header.flags); tduffy> + tduffy> + /* Remove the SRQ from the IA */ tduffy> + dapl_ia_unlink_srq(ia_ptr, srq_ptr); tduffy> + tduffy> + spin_unlock_irqrestore(&srq_ptr->header.lock, srq_ptr->header.flags); tduffy> + tduffy> + /* tduffy> + * Finish tearing everything down. tduffy> + */ tduffy> + tduffy> + /* tduffy> + * Take care of the transport resource tduffy> + */ tduffy> + tduffy> + /* XXX Put provider code here!!! */ tduffy> + tduffy> + /* Free the resource */ tduffy> + dapl_srq_dealloc(srq_ptr); tduffy> + tduffy> +bail: tduffy> + return status; tduffy> + tduffy> +} tduffy> + tduffy> +/* tduffy> + * dapl_srq_post_recv tduffy> + * tduffy> + * Post a receive buffer that can be used by any incoming tduffy> + * message by any connected EP using the SRQ. Request to receive data tduffy> + * over a connection of any ep handle into local_iov tduffy> + * tduffy> + * Input: tduffy> + * srq_handle tduffy> + * num_segments tduffy> + * local_iov tduffy> + * user_cookie tduffy> + * tduffy> + * Output: tduffy> + * None. tduffy> + * tduffy> + * Returns: tduffy> + * DAT_SUCCESS tduffy> + * DAT_INSUFFICIENT_RESOURCES tduffy> + * DAT_INVALID_PARAMETER tduffy> + * DAT_INVALID_HANDLE tduffy> + * DAT_INVALID_STATE tduffy> + * DAT_PROTECTION_VIOLATION tduffy> + * DAT_PROVILEGES_VIOLATION tduffy> + */ tduffy> +u32 dapl_srq_post_recv(DAT_SRQ_HANDLE srq_handle, int num_segments, tduffy> + struct dat_lmr_triplet *local_iov, tduffy> + DAT_DTO_COOKIE user_cookie) tduffy> +{ tduffy> + struct dapl_srq *srq_ptr; tduffy> + struct dapl_cookie *cookie; tduffy> + u32 status; tduffy> + tduffy> + dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> + "dapl_srq_post_recv (%p, %d, %p, %P)\n", tduffy> + srq_handle, num_segments, local_iov, user_cookie.as_64); tduffy> + tduffy> + if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { tduffy> + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + srq_ptr = (struct dapl_srq *)srq_handle; tduffy> + tduffy> + /* tduffy> + * Synchronization ok since this buffer is only used for receive tduffy> + * requests, which aren't allowed to race with each other. The tduffy> + * app must syncronize access to the SRQ. tduffy> + */ tduffy> + status = dapl_dto_cookie_alloc(&srq_ptr->recv_buffer, tduffy> + DAPL_DTO_TYPE_RECV, tduffy> + user_cookie, &cookie); tduffy> + if (DAT_SUCCESS != status) tduffy> + goto bail; tduffy> + tduffy> + /* tduffy> + * Take reference before posting to avoid race conditions with tduffy> + * completions tduffy> + */ tduffy> + atomic_inc(&srq_ptr->recv_count); tduffy> + tduffy> + /* tduffy> + * Invoke provider specific routine to post DTO tduffy> + */ tduffy> + /* XXX Put code here XXX */ tduffy> + /* XXX */ status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); tduffy> + tduffy> + if (status != DAT_SUCCESS) { tduffy> + atomic_dec(&srq_ptr->recv_count); tduffy> + dapl_cookie_dealloc(&srq_ptr->recv_buffer, cookie); tduffy> + } tduffy> + tduffy> +bail: tduffy> + return status; tduffy> +} tduffy> + tduffy> +u32 dapl_srq_query(DAT_SRQ_HANDLE srq_handle, struct dat_srq_param *srq_param) tduffy> +{ tduffy> + struct dapl_srq *srq_ptr; tduffy> + u32 status = DAT_SUCCESS; tduffy> + tduffy> + dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> + "dapl_srq_query (%p, %x, %p)\n", tduffy> + srq_handle, srq_param); tduffy> + tduffy> + if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { tduffy> + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); tduffy> + goto bail; tduffy> + } tduffy> + if (srq_param == NULL) { tduffy> + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + srq_ptr = (struct dapl_srq *)srq_handle; tduffy> + tduffy> + /* tduffy> + * XXX Need to calculate available_dto_count and outstanding_dto_count tduffy> + */ tduffy> + srq_ptr->param.available_dto_count = DAT_VALUE_UNKNOWN; tduffy> + srq_ptr->param.outstanding_dto_count = DAT_VALUE_UNKNOWN; tduffy> + tduffy> + *srq_param = srq_ptr->param; tduffy> + tduffy> +bail: tduffy> + return status; tduffy> +} tduffy> + tduffy> +/* tduffy> + * dapl_srq_resize tduffy> + * tduffy> + * Modify the size fo the event queue of a Shared Recieve Queue tduffy> + * tduffy> + * Input: tduffy> + * srq_handle tduffy> + * srq_max_recv_dto tduffy> + * tduffy> + * Output: tduffy> + * none tduffy> + * tduffy> + * Returns: tduffy> + * DAT_SUCCESS tduffy> + * DAT_INVALID_HANDLE tduffy> + * DAT_INVALID_PARAMETER tduffy> + * DAT_INSUFFICIENT_RESOURCES tduffy> + * DAT_INVALID_STATE tduffy> + */ tduffy> + tduffy> +u32 dapl_srq_resize(DAT_SRQ_HANDLE srq_handle, int srq_max_recv_dto) tduffy> +{ tduffy> + struct dapl_ia *ia_ptr; tduffy> + struct dapl_srq *srq_ptr; tduffy> + u32 status = DAT_SUCCESS; tduffy> + tduffy> + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_resize (%p, %d)\n", tduffy> + srq_handle, srq_max_recv_dto); tduffy> + tduffy> + if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { tduffy> + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + srq_ptr = (struct dapl_srq *)srq_handle; tduffy> + ia_ptr = srq_ptr->header.owner_ia; tduffy> + tduffy> + /* tduffy> + * Check for nonsense requests per the spec tduffy> + */ tduffy> + if (srq_max_recv_dto <= srq_ptr->param.low_watermark) { tduffy> + status = DAT_ERROR(DAT_INVALID_STATE, DAT_NO_SUBTYPE); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + /* XXX Put implementation here XXX */ tduffy> + tduffy> + /* XXX */ status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); tduffy> + tduffy> +bail: tduffy> + return status; tduffy> +} tduffy> + tduffy> +/* tduffy> + * dapl_srq_set_lw tduffy> + * tduffy> + * DAPL Requirements Version 1.2, 6.5.4 tduffy> + * tduffy> + * Set the low water mark for an SRQ and arm the SRQ to generate an tduffy> + * event if it is reached. tduffy> + * tduffy> + * Input: tduffy> + * srq_handle tduffy> + * srq_max_recv_dto tduffy> + * tduffy> + * Output: tduffy> + * none tduffy> + * tduffy> + * Returns: tduffy> + * DAT_SUCCESS tduffy> + * DAT_INVALID_HANDLE tduffy> + * DAT_INVALID_PARAMETER tduffy> + * DAT_MODEL_NOT_SUPPORTED tduffy> + */ tduffy> + tduffy> +u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, int low_watermark) tduffy> +{ tduffy> + struct dapl_srq *srq_ptr; tduffy> + u32 status = DAT_SUCCESS; tduffy> + tduffy> + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_set_lw (%p, %d)\n", tduffy> + srq_handle, low_watermark); tduffy> + tduffy> + if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { tduffy> + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + srq_ptr = (struct dapl_srq *)srq_handle; tduffy> + tduffy> + /* XXX Put implementation here XXX */ tduffy> + tduffy> + /* XXX */ status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); tduffy> + tduffy> +bail: tduffy> + return status; tduffy> +} tduffy> Index: linux-kernel-srq/dat-provider/dapl_srq_resize.c tduffy> =================================================================== tduffy> --- linux-kernel-srq/dat-provider/dapl_srq_resize.c (revision 2489) tduffy> +++ linux-kernel-srq/dat-provider/dapl_srq_resize.c (working copy) tduffy> @@ -1,88 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/* tduffy> - * $Id$ tduffy> - */ tduffy> - tduffy> -#include "dapl.h" tduffy> -#include "dapl_srq_util.h" tduffy> -#include "dapl_openib_util.h" tduffy> - tduffy> -/* tduffy> - * dapl_srq_resize tduffy> - * tduffy> - * Modify the size fo the event queue of a Shared Recieve Queue tduffy> - * tduffy> - * Input: tduffy> - * srq_handle tduffy> - * srq_max_recv_dto tduffy> - * tduffy> - * Output: tduffy> - * none tduffy> - * tduffy> - * Returns: tduffy> - * DAT_SUCCESS tduffy> - * DAT_INVALID_HANDLE tduffy> - * DAT_INVALID_PARAMETER tduffy> - * DAT_INSUFFICIENT_RESOURCES tduffy> - * DAT_INVALID_STATE tduffy> - */ tduffy> - tduffy> -u32 dapl_srq_resize(DAT_SRQ_HANDLE srq_handle, int srq_max_recv_dto) tduffy> -{ tduffy> - struct dapl_ia *ia_ptr; tduffy> - struct dapl_srq *srq_ptr; tduffy> - u32 dat_status = DAT_SUCCESS; tduffy> - tduffy> - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_resize (%p, %d)\n", tduffy> - srq_handle, srq_max_recv_dto); tduffy> - tduffy> - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - srq_ptr = (struct dapl_srq *)srq_handle; tduffy> - ia_ptr = srq_ptr->header.owner_ia; tduffy> - tduffy> - /* tduffy> - * Check for nonsense requests per the spec tduffy> - */ tduffy> - if (srq_max_recv_dto <= srq_ptr->param.low_watermark) { tduffy> - dat_status = DAT_ERROR(DAT_INVALID_STATE, DAT_NO_SUBTYPE); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - /* XXX Put implementation here XXX */ tduffy> - tduffy> - /* XXX */ dat_status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); tduffy> - tduffy> - bail: tduffy> - return dat_status; tduffy> -} tduffy> Index: linux-kernel-srq/dat-provider/dapl_srq_create.c tduffy> =================================================================== tduffy> --- linux-kernel-srq/dat-provider/dapl_srq_create.c (revision 2489) tduffy> +++ linux-kernel-srq/dat-provider/dapl_srq_create.c (working copy) tduffy> @@ -1,128 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/* tduffy> - * $Id$ tduffy> - */ tduffy> - tduffy> -#include "dapl.h" tduffy> -#include "dapl_ia_util.h" tduffy> -#include "dapl_srq_util.h" tduffy> -#include "dapl_openib_util.h" tduffy> - tduffy> -/* tduffy> - * dapl_srq_create tduffy> - * tduffy> - * Create an instance of a Shared Receive Queue that is provided to the tduffy> - * consumer at srq_handle. tduffy> - * tduffy> - * Input: tduffy> - * ia_handle tduffy> - * pz_handle tduffy> - * srq_attr tduffy> - * tduffy> - * Output: tduffy> - * srq_handle tduffy> - * tduffy> - * Returns: tduffy> - * DAT_SUCCESS tduffy> - * DAT_INSUFFICIENT_RESOURCES tduffy> - * DAT_INVALID_HANDLE tduffy> - * DAT_INVALID_PARAMETER tduffy> - * ?DAT_INVALID_ATTRIBUTE?? tduffy> - * DAT_MODEL_NOT_SUPPORTED tduffy> - */ tduffy> -u32 dapl_srq_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, tduffy> - struct dat_srq_attr *srq_attr, DAT_SRQ_HANDLE *srq_handle) tduffy> -{ tduffy> - struct dapl_ia *ia_ptr; tduffy> - struct dapl_srq *srq_ptr; tduffy> - u32 dat_status = DAT_SUCCESS; tduffy> - tduffy> - dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> - "dapl_srq_create (%p, %p, %p, %p)\n", tduffy> - ia_handle, pz_handle, srq_attr, srq_handle); tduffy> - tduffy> - ia_ptr = (struct dapl_ia *)ia_handle; tduffy> - tduffy> - /* tduffy> - * Verify parameters tduffy> - */ tduffy> - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - /* tduffy> - * Verify non-required parameters. tduffy> - * N.B. Assumption: any parameter that can be tduffy> - * modified by dat_ep_modify() is not strictly tduffy> - * required when the EP is created tduffy> - */ tduffy> - if (pz_handle != NULL && DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - if (srq_handle == NULL) { tduffy> - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); tduffy> - goto bail; tduffy> - } tduffy> - if ((unsigned long)srq_attr & 3) { tduffy> - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - /* Allocate SRQ */ tduffy> - srq_ptr = dapl_srq_alloc(ia_ptr, srq_attr); tduffy> - if (srq_ptr == NULL) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - srq_ptr->param.ia_handle = (DAT_IA_HANDLE) ia_ptr; tduffy> - srq_ptr->param.srq_state = DAT_SRQ_STATE_OPERATIONAL; tduffy> - srq_ptr->param.pz_handle = pz_handle; tduffy> - tduffy> - /* tduffy> - * XXX Allocate provider resource here!!! tduffy> - */ tduffy> - /* XXX */ dat_status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); tduffy> - /* XXX */ dapl_srq_dealloc(srq_ptr); tduffy> - /* XXX */ goto bail; tduffy> - tduffy> - /* Link it onto the IA */ tduffy> - dapl_ia_link_srq(ia_ptr, srq_ptr); tduffy> - tduffy> - *srq_handle = srq_ptr; tduffy> - tduffy> - bail: tduffy> - return dat_status; tduffy> -} tduffy> Index: linux-kernel-srq/dat-provider/dapl_srq_query.c tduffy> =================================================================== tduffy> --- linux-kernel-srq/dat-provider/dapl_srq_query.c (revision 2489) tduffy> +++ linux-kernel-srq/dat-provider/dapl_srq_query.c (working copy) tduffy> @@ -1,65 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/* tduffy> - * $Id$ tduffy> - */ tduffy> - tduffy> -#include "dapl.h" tduffy> - tduffy> -u32 dapl_srq_query(DAT_SRQ_HANDLE srq_handle, struct dat_srq_param *srq_param) tduffy> -{ tduffy> - struct dapl_srq *srq_ptr; tduffy> - u32 dat_status = DAT_SUCCESS; tduffy> - tduffy> - dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> - "dapl_srq_query (%p, %x, %p)\n", tduffy> - srq_handle, srq_param); tduffy> - tduffy> - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); tduffy> - goto bail; tduffy> - } tduffy> - if (srq_param == NULL) { tduffy> - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - srq_ptr = (struct dapl_srq *)srq_handle; tduffy> - tduffy> - /* tduffy> - * XXX Need to calculate available_dto_count and outstanding_dto_count tduffy> - */ tduffy> - srq_ptr->param.available_dto_count = DAT_VALUE_UNKNOWN; tduffy> - srq_ptr->param.outstanding_dto_count = DAT_VALUE_UNKNOWN; tduffy> - tduffy> - *srq_param = srq_ptr->param; tduffy> - tduffy> - bail: tduffy> - return dat_status; tduffy> -} tduffy> Index: linux-kernel-srq/dat-provider/Makefile tduffy> =================================================================== tduffy> --- linux-kernel-srq/dat-provider/Makefile (revision 2489) tduffy> +++ linux-kernel-srq/dat-provider/Makefile (working copy) tduffy> @@ -82,13 +82,7 @@ PROVIDER_MODULES := \ tduffy> dapl_rsp_query \ tduffy> dapl_set_consumer_context \ tduffy> dapl_sp_util \ tduffy> - dapl_srq_create \ tduffy> - dapl_srq_free \ tduffy> - dapl_srq_post_recv \ tduffy> - dapl_srq_query \ tduffy> - dapl_srq_resize \ tduffy> - dapl_srq_set_lw \ tduffy> - dapl_srq_util \ tduffy> + dapl_srq \ tduffy> dapl_timer_util \ tduffy> dapl_util tduffy> tduffy> Index: linux-kernel-srq/dat-provider/dapl_srq_util.c tduffy> =================================================================== tduffy> --- linux-kernel-srq/dat-provider/dapl_srq_util.c (revision 2489) tduffy> +++ linux-kernel-srq/dat-provider/dapl_srq_util.c (working copy) tduffy> @@ -1,126 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/* tduffy> - * $Id$ tduffy> - */ tduffy> - tduffy> -#include "dapl_srq_util.h" tduffy> -#include "dapl_ia_util.h" tduffy> -#include "dapl_cookie.h" tduffy> - tduffy> -/* tduffy> - * dapl_srq_alloc tduffy> - * tduffy> - * alloc and initialize an SRQ struct tduffy> - * tduffy> - * Input: tduffy> - * IA INFO struct ptr tduffy> - * SRQ ATTR ptr tduffy> - * tduffy> - * Output: tduffy> - * none tduffy> - * tduffy> - * Returns: tduffy> - * pointer to srq tduffy> - * tduffy> - */ tduffy> -struct dapl_srq *dapl_srq_alloc(struct dapl_ia *ia_ptr, tduffy> - const struct dat_srq_attr *srq_attr) tduffy> -{ tduffy> - struct dapl_srq *srq_ptr; tduffy> - tduffy> - /* Allocate SRQ */ tduffy> - srq_ptr = kmalloc(sizeof *srq_ptr, GFP_ATOMIC); tduffy> - if (!srq_ptr) tduffy> - goto bail; tduffy> - tduffy> - /* zero the structure */ tduffy> - memset(srq_ptr, 0, sizeof *srq_ptr); tduffy> - tduffy> - /* tduffy> - * initialize the header tduffy> - */ tduffy> - srq_ptr->header.provider = ia_ptr->header.provider; tduffy> - srq_ptr->header.magic = DAPL_MAGIC_SRQ; tduffy> - srq_ptr->header.handle_type = DAT_HANDLE_TYPE_SRQ; tduffy> - srq_ptr->header.owner_ia = ia_ptr; tduffy> - srq_ptr->header.user_context.as_64 = 0; tduffy> - srq_ptr->header.user_context.as_ptr = NULL; tduffy> - atomic_set(&srq_ptr->srq_ref_count, 0); tduffy> - tduffy> - dapl_llist_init_entry(&srq_ptr->header.ia_list_entry); tduffy> - spin_lock_init(&srq_ptr->header.lock); tduffy> - tduffy> - /* tduffy> - * Initialize the body. tduffy> - * XXX Assume srq_attrs is required tduffy> - */ tduffy> - srq_ptr->param.max_recv_dtos = srq_attr->max_recv_dtos; tduffy> - srq_ptr->param.max_recv_iov = srq_attr->max_recv_iov; tduffy> - srq_ptr->param.low_watermark = srq_attr->low_watermark; tduffy> - tduffy> - /* Get a cookie buffer to track outstanding recvs */ tduffy> - if (DAT_SUCCESS != dapl_cb_create(&srq_ptr->recv_buffer, tduffy> - (struct dapl_ep *)srq_ptr, tduffy> - srq_ptr->param.max_recv_dtos)) { tduffy> - dapl_srq_dealloc(srq_ptr); tduffy> - srq_ptr = NULL; tduffy> - goto bail; tduffy> - } tduffy> - tduffy> -bail: tduffy> - return srq_ptr; tduffy> -} tduffy> - tduffy> -/* tduffy> - * dapl_srq_dealloc tduffy> - * tduffy> - * Free the passed in SRQ structure. tduffy> - * tduffy> - * Input: tduffy> - * SRQ pointer tduffy> - * tduffy> - * Output: tduffy> - * none tduffy> - * tduffy> - * Returns: tduffy> - * none tduffy> - * tduffy> - */ tduffy> -void dapl_srq_dealloc(struct dapl_srq *srq_ptr) tduffy> -{ tduffy> - dapl_os_assert(srq_ptr->header.magic == DAPL_MAGIC_SRQ); tduffy> - tduffy> - /* reset magic to prevent reuse */ tduffy> - srq_ptr->header.magic = DAPL_MAGIC_INVALID; tduffy> - dapl_ia_unlink_srq(srq_ptr->header.owner_ia, srq_ptr); tduffy> - dapl_cb_free(&srq_ptr->recv_buffer); tduffy> - /* no need to destroy srq_ptr->header.lock */ tduffy> - tduffy> - kfree(srq_ptr); tduffy> -} tduffy> Index: linux-kernel-srq/dat-provider/dapl_srq_util.h tduffy> =================================================================== tduffy> --- linux-kernel-srq/dat-provider/dapl_srq_util.h (revision 2489) tduffy> +++ linux-kernel-srq/dat-provider/dapl_srq_util.h (working copy) tduffy> @@ -1,47 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/********************************************************************** tduffy> - * tduffy> - * HEADER: dapl_srq_util.h tduffy> - * tduffy> - * PURPOSE: Utility defs & routines for the SRQ data structure tduffy> - * tduffy> - * $Id$ tduffy> - **********************************************************************/ tduffy> - tduffy> -#ifndef DAPL_SRQ_UTIL_H tduffy> -#define DAPL_SRQ_UTIL_H tduffy> - tduffy> -#include "dapl.h" tduffy> - tduffy> -extern struct dapl_srq *dapl_srq_alloc(struct dapl_ia *ia, tduffy> - const struct dat_srq_attr *srq_attr); tduffy> - tduffy> -extern void dapl_srq_dealloc(struct dapl_srq *srq_ptr); tduffy> - tduffy> -#endif /* DAPL_SRQ_UTIL_H */ tduffy> Index: linux-kernel-srq/dat-provider/dapl_srq_set_lw.c tduffy> =================================================================== tduffy> --- linux-kernel-srq/dat-provider/dapl_srq_set_lw.c (revision 2489) tduffy> +++ linux-kernel-srq/dat-provider/dapl_srq_set_lw.c (working copy) tduffy> @@ -1,80 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/* tduffy> - * $Id$ tduffy> - */ tduffy> - tduffy> -#include "dapl.h" tduffy> -#include "dapl_srq_util.h" tduffy> -#include "dapl_openib_util.h" tduffy> - tduffy> -/* tduffy> - * dapl_srq_set_lw tduffy> - * tduffy> - * DAPL Requirements Version 1.2, 6.5.4 tduffy> - * tduffy> - * Set the low water mark for an SRQ and arm the SRQ to generate an tduffy> - * event if it is reached. tduffy> - * tduffy> - * Input: tduffy> - * srq_handle tduffy> - * srq_max_recv_dto tduffy> - * tduffy> - * Output: tduffy> - * none tduffy> - * tduffy> - * Returns: tduffy> - * DAT_SUCCESS tduffy> - * DAT_INVALID_HANDLE tduffy> - * DAT_INVALID_PARAMETER tduffy> - * DAT_MODEL_NOT_SUPPORTED tduffy> - */ tduffy> - tduffy> -u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, int low_watermark) tduffy> -{ tduffy> - struct dapl_srq *srq_ptr; tduffy> - u32 dat_status = DAT_SUCCESS; tduffy> - tduffy> - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_set_lw (%p, %d)\n", tduffy> - srq_handle, low_watermark); tduffy> - tduffy> - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - srq_ptr = (struct dapl_srq *)srq_handle; tduffy> - tduffy> - /* XXX Put implementation here XXX */ tduffy> - tduffy> - /* XXX */ dat_status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); tduffy> - tduffy> - bail: tduffy> - return dat_status; tduffy> -} tduffy> Index: linux-kernel-srq/dat-provider/dapl_srq_post_recv.c tduffy> =================================================================== tduffy> --- linux-kernel-srq/dat-provider/dapl_srq_post_recv.c (revision 2489) tduffy> +++ linux-kernel-srq/dat-provider/dapl_srq_post_recv.c (working copy) tduffy> @@ -1,112 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/* tduffy> - * $Id$ tduffy> - */ tduffy> - tduffy> -#include "dapl.h" tduffy> -#include "dapl_cookie.h" tduffy> -#include "dapl_openib_util.h" tduffy> - tduffy> -/* tduffy> - * dapl_srq_post_recv tduffy> - * tduffy> - * Post a receive buffer that can be used by any incoming tduffy> - * message by any connected EP using the SRQ. Request to receive data tduffy> - * over a connection of any ep handle into local_iov tduffy> - * tduffy> - * Input: tduffy> - * srq_handle tduffy> - * num_segments tduffy> - * local_iov tduffy> - * user_cookie tduffy> - * tduffy> - * Output: tduffy> - * None. tduffy> - * tduffy> - * Returns: tduffy> - * DAT_SUCCESS tduffy> - * DAT_INSUFFICIENT_RESOURCES tduffy> - * DAT_INVALID_PARAMETER tduffy> - * DAT_INVALID_HANDLE tduffy> - * DAT_INVALID_STATE tduffy> - * DAT_PROTECTION_VIOLATION tduffy> - * DAT_PROVILEGES_VIOLATION tduffy> - */ tduffy> -u32 dapl_srq_post_recv(DAT_SRQ_HANDLE srq_handle, int num_segments, tduffy> - struct dat_lmr_triplet *local_iov, tduffy> - DAT_DTO_COOKIE user_cookie) tduffy> -{ tduffy> - struct dapl_srq *srq_ptr; tduffy> - struct dapl_cookie *cookie; tduffy> - u32 dat_status; tduffy> - tduffy> - dapl_dbg_log(DAPL_DBG_TYPE_API, tduffy> - "dapl_srq_post_recv (%p, %d, %p, %P)\n", tduffy> - srq_handle, num_segments, local_iov, user_cookie.as_64); tduffy> - tduffy> - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - srq_ptr = (struct dapl_srq *)srq_handle; tduffy> - tduffy> - /* tduffy> - * Synchronization ok since this buffer is only used for receive tduffy> - * requests, which aren't allowed to race with each other. The tduffy> - * app must syncronize access to the SRQ. tduffy> - */ tduffy> - dat_status = dapl_dto_cookie_alloc(&srq_ptr->recv_buffer, tduffy> - DAPL_DTO_TYPE_RECV, tduffy> - user_cookie, &cookie); tduffy> - if (DAT_SUCCESS != dat_status) { tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - /* tduffy> - * Take reference before posting to avoid race conditions with tduffy> - * completions tduffy> - */ tduffy> - atomic_inc(&srq_ptr->recv_count); tduffy> - tduffy> - /* tduffy> - * Invoke provider specific routine to post DTO tduffy> - */ tduffy> - /* XXX Put code here XXX */ tduffy> - /* XXX */ dat_status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); tduffy> - tduffy> - if (dat_status != DAT_SUCCESS) { tduffy> - atomic_dec(&srq_ptr->recv_count); tduffy> - dapl_cookie_dealloc(&srq_ptr->recv_buffer, cookie); tduffy> - } tduffy> - tduffy> - bail: tduffy> - return dat_status; tduffy> -} tduffy> Index: linux-kernel-srq/dat-provider/dapl_srq_free.c tduffy> =================================================================== tduffy> --- linux-kernel-srq/dat-provider/dapl_srq_free.c (revision 2489) tduffy> +++ linux-kernel-srq/dat-provider/dapl_srq_free.c (working copy) tduffy> @@ -1,112 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/* tduffy> - * $Id$ tduffy> - */ tduffy> - tduffy> -#include "dapl.h" tduffy> -#include "dapl_ia_util.h" tduffy> -#include "dapl_srq_util.h" tduffy> -#include "dapl_openib_util.h" tduffy> - tduffy> -/* tduffy> - * dapl_srq_free tduffy> - * tduffy> - * Destroy an instance of an SRQ tduffy> - * tduffy> - * Input: tduffy> - * srq_handle tduffy> - * tduffy> - * Output: tduffy> - * none tduffy> - * tduffy> - * Returns: tduffy> - * DAT_SUCCESS tduffy> - * DAT_INVALID_PARAMETER tduffy> - * DAT_INVALID_STATE tduffy> - */ tduffy> -u32 dapl_srq_free(DAT_SRQ_HANDLE srq_handle) tduffy> -{ tduffy> - struct dapl_srq *srq_ptr; tduffy> - struct dapl_ia *ia_ptr; tduffy> - struct dat_srq_param *param; tduffy> - u32 dat_status = DAT_SUCCESS; tduffy> - tduffy> - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_free (%p)\n", srq_handle); tduffy> - tduffy> - srq_ptr = (struct dapl_srq *)srq_handle; tduffy> - param = &srq_ptr->param; tduffy> - tduffy> - /* tduffy> - * Verify parameter & state tduffy> - */ tduffy> - if (DAPL_BAD_HANDLE(srq_ptr, DAPL_MAGIC_SRQ)) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - if (atomic_read(&srq_ptr->srq_ref_count) != 0) { tduffy> - /* tduffy> - * The DAPL 1.2 spec says to return DAT_SRQ_IN_USE, which does tduffy> - * not exist. Have filed the following as an eratta. tduffy> - */ tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_SRQ_IN_USE); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - ia_ptr = srq_ptr->header.owner_ia; tduffy> - tduffy> - /* tduffy> - * Do verification of parameters and the state change atomically. tduffy> - */ tduffy> - spin_lock_irqsave(&srq_ptr->header.lock, srq_ptr->header.flags); tduffy> - tduffy> - /* Remove the SRQ from the IA */ tduffy> - dapl_ia_unlink_srq(ia_ptr, srq_ptr); tduffy> - tduffy> - spin_unlock_irqrestore(&srq_ptr->header.lock, srq_ptr->header.flags); tduffy> - tduffy> - /* tduffy> - * Finish tearing everything down. tduffy> - */ tduffy> - tduffy> - /* tduffy> - * Take care of the transport resource tduffy> - */ tduffy> - tduffy> - /* XXX Put provider code here!!! */ tduffy> - tduffy> - /* Free the resource */ tduffy> - dapl_srq_dealloc(srq_ptr); tduffy> - tduffy> - bail: tduffy> - return dat_status; tduffy> - tduffy> -} tduffy> Index: linux-kernel-srq/patches/alt_dat_provider_makefile tduffy> =================================================================== tduffy> --- linux-kernel-srq/patches/alt_dat_provider_makefile (revision 2489) tduffy> +++ linux-kernel-srq/patches/alt_dat_provider_makefile (working copy) tduffy> @@ -76,13 +76,7 @@ PROVIDER_MODULES := \ tduffy> dapl_rsp_query \ tduffy> dapl_set_consumer_context \ tduffy> dapl_sp_util \ tduffy> - dapl_srq_create \ tduffy> - dapl_srq_free \ tduffy> - dapl_srq_post_recv \ tduffy> - dapl_srq_query \ tduffy> - dapl_srq_resize \ tduffy> - dapl_srq_set_lw \ tduffy> - dapl_srq_util \ tduffy> + dapl_srq \ tduffy> dapl_timer_util \ tduffy> dapl_util tduffy> tduffy> From jlentini at netapp.com Fri May 27 08:17:27 2005 From: jlentini at netapp.com (James Lentini) Date: Fri, 27 May 2005 11:17:27 -0400 (EDT) Subject: [openib-general] Re: [PATCH 1/3] kDAPL: consolidate dapl_srq*.c into one file In-Reply-To: References: <1117133253.17831.12.camel@duffman> Message-ID: Actually, "[PATCH 2/3] kDAPL: consolidate dapl_rsp*.c into one file" was committed in revision 2501. On Fri, 27 May 2005, James Lentini wrote: > > Committed in revision 2501. > > On Thu, 26 May 2005, Tom Duffy wrote: > > tduffy> Signed-off-by: Tom Duffy > tduffy> > tduffy> Index: linux-kernel-srq/dat-provider/dapl_srq.c > tduffy> =================================================================== > tduffy> --- linux-kernel-srq/dat-provider/dapl_srq.c (revision 0) > tduffy> +++ linux-kernel-srq/dat-provider/dapl_srq.c (revision 0) > tduffy> @@ -0,0 +1,500 @@ > tduffy> +/* > tduffy> + * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. > tduffy> + * > tduffy> + * This Software is licensed under one of the following licenses: > tduffy> + * > tduffy> + * 1) under the terms of the "Common Public License 1.0" a copy of which is > tduffy> + * available from the Open Source Initiative, see > tduffy> + * http://www.opensource.org/licenses/cpl.php. > tduffy> + * > tduffy> + * 2) under the terms of the "The BSD License" a copy of which is > tduffy> + * available from the Open Source Initiative, see > tduffy> + * http://www.opensource.org/licenses/bsd-license.php. > tduffy> + * > tduffy> + * 3) under the terms of the "GNU General Public License (GPL) Version 2" a > tduffy> + * copy of which is available from the Open Source Initiative, see > tduffy> + * http://www.opensource.org/licenses/gpl-license.php. > tduffy> + * > tduffy> + * Licensee has the right to choose one of the above licenses. > tduffy> + * > tduffy> + * Redistributions of source code must retain the above copyright > tduffy> + * notice and one of the license notices. > tduffy> + * > tduffy> + * Redistributions in binary form must reproduce both the above copyright > tduffy> + * notice, one of the license notices in the documentation > tduffy> + * and/or other materials provided with the distribution. > tduffy> + */ > tduffy> + > tduffy> +/* > tduffy> + * $Id$ > tduffy> + */ > tduffy> + > tduffy> +#include "dapl.h" > tduffy> +#include "dapl_ia_util.h" > tduffy> +#include "dapl_openib_util.h" > tduffy> +#include "dapl_cookie.h" > tduffy> + > tduffy> +/* > tduffy> + * dapl_srq_dealloc > tduffy> + * > tduffy> + * Free the passed in SRQ structure. > tduffy> + * > tduffy> + * Input: > tduffy> + * SRQ pointer > tduffy> + * > tduffy> + * Output: > tduffy> + * none > tduffy> + * > tduffy> + * Returns: > tduffy> + * none > tduffy> + * > tduffy> + */ > tduffy> +static void dapl_srq_dealloc(struct dapl_srq *srq_ptr) > tduffy> +{ > tduffy> + dapl_os_assert(srq_ptr->header.magic == DAPL_MAGIC_SRQ); > tduffy> + > tduffy> + /* reset magic to prevent reuse */ > tduffy> + srq_ptr->header.magic = DAPL_MAGIC_INVALID; > tduffy> + dapl_ia_unlink_srq(srq_ptr->header.owner_ia, srq_ptr); > tduffy> + dapl_cb_free(&srq_ptr->recv_buffer); > tduffy> + /* no need to destroy srq_ptr->header.lock */ > tduffy> + > tduffy> + kfree(srq_ptr); > tduffy> +} > tduffy> + > tduffy> +/* > tduffy> + * dapl_srq_alloc > tduffy> + * > tduffy> + * alloc and initialize an SRQ struct > tduffy> + * > tduffy> + * Input: > tduffy> + * IA INFO struct ptr > tduffy> + * SRQ ATTR ptr > tduffy> + * > tduffy> + * Output: > tduffy> + * none > tduffy> + * > tduffy> + * Returns: > tduffy> + * pointer to srq > tduffy> + * > tduffy> + */ > tduffy> +static struct dapl_srq *dapl_srq_alloc(struct dapl_ia *ia_ptr, > tduffy> + const struct dat_srq_attr *srq_attr) > tduffy> +{ > tduffy> + struct dapl_srq *srq_ptr; > tduffy> + > tduffy> + /* Allocate SRQ */ > tduffy> + srq_ptr = kmalloc(sizeof *srq_ptr, GFP_ATOMIC); > tduffy> + if (!srq_ptr) > tduffy> + goto bail; > tduffy> + > tduffy> + /* zero the structure */ > tduffy> + memset(srq_ptr, 0, sizeof *srq_ptr); > tduffy> + > tduffy> + /* > tduffy> + * initialize the header > tduffy> + */ > tduffy> + srq_ptr->header.provider = ia_ptr->header.provider; > tduffy> + srq_ptr->header.magic = DAPL_MAGIC_SRQ; > tduffy> + srq_ptr->header.handle_type = DAT_HANDLE_TYPE_SRQ; > tduffy> + srq_ptr->header.owner_ia = ia_ptr; > tduffy> + srq_ptr->header.user_context.as_64 = 0; > tduffy> + srq_ptr->header.user_context.as_ptr = NULL; > tduffy> + atomic_set(&srq_ptr->srq_ref_count, 0); > tduffy> + > tduffy> + dapl_llist_init_entry(&srq_ptr->header.ia_list_entry); > tduffy> + spin_lock_init(&srq_ptr->header.lock); > tduffy> + > tduffy> + /* > tduffy> + * Initialize the body. > tduffy> + * XXX Assume srq_attrs is required > tduffy> + */ > tduffy> + srq_ptr->param.max_recv_dtos = srq_attr->max_recv_dtos; > tduffy> + srq_ptr->param.max_recv_iov = srq_attr->max_recv_iov; > tduffy> + srq_ptr->param.low_watermark = srq_attr->low_watermark; > tduffy> + > tduffy> + /* Get a cookie buffer to track outstanding recvs */ > tduffy> + if (DAT_SUCCESS != dapl_cb_create(&srq_ptr->recv_buffer, > tduffy> + (struct dapl_ep *)srq_ptr, > tduffy> + srq_ptr->param.max_recv_dtos)) { > tduffy> + dapl_srq_dealloc(srq_ptr); > tduffy> + srq_ptr = NULL; > tduffy> + goto bail; > tduffy> + } > tduffy> + > tduffy> +bail: > tduffy> + return srq_ptr; > tduffy> +} > tduffy> + > tduffy> +/* > tduffy> + * dapl_srq_create > tduffy> + * > tduffy> + * Create an instance of a Shared Receive Queue that is provided to the > tduffy> + * consumer at srq_handle. > tduffy> + * > tduffy> + * Input: > tduffy> + * ia_handle > tduffy> + * pz_handle > tduffy> + * srq_attr > tduffy> + * > tduffy> + * Output: > tduffy> + * srq_handle > tduffy> + * > tduffy> + * Returns: > tduffy> + * DAT_SUCCESS > tduffy> + * DAT_INSUFFICIENT_RESOURCES > tduffy> + * DAT_INVALID_HANDLE > tduffy> + * DAT_INVALID_PARAMETER > tduffy> + * ?DAT_INVALID_ATTRIBUTE?? > tduffy> + * DAT_MODEL_NOT_SUPPORTED > tduffy> + */ > tduffy> +u32 dapl_srq_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, > tduffy> + struct dat_srq_attr *srq_attr, DAT_SRQ_HANDLE *srq_handle) > tduffy> +{ > tduffy> + struct dapl_ia *ia_ptr; > tduffy> + struct dapl_srq *srq_ptr; > tduffy> + u32 status = DAT_SUCCESS; > tduffy> + > tduffy> + dapl_dbg_log(DAPL_DBG_TYPE_API, > tduffy> + "dapl_srq_create (%p, %p, %p, %p)\n", > tduffy> + ia_handle, pz_handle, srq_attr, srq_handle); > tduffy> + > tduffy> + ia_ptr = (struct dapl_ia *)ia_handle; > tduffy> + > tduffy> + /* > tduffy> + * Verify parameters > tduffy> + */ > tduffy> + if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { > tduffy> + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); > tduffy> + goto bail; > tduffy> + } > tduffy> + > tduffy> + /* > tduffy> + * Verify non-required parameters. > tduffy> + * N.B. Assumption: any parameter that can be > tduffy> + * modified by dat_ep_modify() is not strictly > tduffy> + * required when the EP is created > tduffy> + */ > tduffy> + if (pz_handle != NULL && DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { > tduffy> + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); > tduffy> + goto bail; > tduffy> + } > tduffy> + > tduffy> + if (srq_handle == NULL) { > tduffy> + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); > tduffy> + goto bail; > tduffy> + } > tduffy> + if ((unsigned long)srq_attr & 3) { > tduffy> + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); > tduffy> + goto bail; > tduffy> + } > tduffy> + > tduffy> + /* Allocate SRQ */ > tduffy> + srq_ptr = dapl_srq_alloc(ia_ptr, srq_attr); > tduffy> + if (srq_ptr == NULL) { > tduffy> + status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, > tduffy> + DAT_RESOURCE_MEMORY); > tduffy> + goto bail; > tduffy> + } > tduffy> + > tduffy> + srq_ptr->param.ia_handle = (DAT_IA_HANDLE) ia_ptr; > tduffy> + srq_ptr->param.srq_state = DAT_SRQ_STATE_OPERATIONAL; > tduffy> + srq_ptr->param.pz_handle = pz_handle; > tduffy> + > tduffy> + /* > tduffy> + * XXX Allocate provider resource here!!! > tduffy> + */ > tduffy> + /* XXX */ status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); > tduffy> + /* XXX */ dapl_srq_dealloc(srq_ptr); > tduffy> + /* XXX */ goto bail; > tduffy> + > tduffy> + /* Link it onto the IA */ > tduffy> + dapl_ia_link_srq(ia_ptr, srq_ptr); > tduffy> + > tduffy> + *srq_handle = srq_ptr; > tduffy> + > tduffy> +bail: > tduffy> + return status; > tduffy> +} > tduffy> + > tduffy> +/* > tduffy> + * dapl_srq_free > tduffy> + * > tduffy> + * Destroy an instance of an SRQ > tduffy> + * > tduffy> + * Input: > tduffy> + * srq_handle > tduffy> + * > tduffy> + * Output: > tduffy> + * none > tduffy> + * > tduffy> + * Returns: > tduffy> + * DAT_SUCCESS > tduffy> + * DAT_INVALID_PARAMETER > tduffy> + * DAT_INVALID_STATE > tduffy> + */ > tduffy> +u32 dapl_srq_free(DAT_SRQ_HANDLE srq_handle) > tduffy> +{ > tduffy> + struct dapl_srq *srq_ptr; > tduffy> + struct dapl_ia *ia_ptr; > tduffy> + struct dat_srq_param *param; > tduffy> + u32 status = DAT_SUCCESS; > tduffy> + > tduffy> + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_free (%p)\n", srq_handle); > tduffy> + > tduffy> + srq_ptr = (struct dapl_srq *)srq_handle; > tduffy> + param = &srq_ptr->param; > tduffy> + > tduffy> + /* > tduffy> + * Verify parameter & state > tduffy> + */ > tduffy> + if (DAPL_BAD_HANDLE(srq_ptr, DAPL_MAGIC_SRQ)) { > tduffy> + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); > tduffy> + goto bail; > tduffy> + } > tduffy> + > tduffy> + if (atomic_read(&srq_ptr->srq_ref_count) != 0) { > tduffy> + /* > tduffy> + * The DAPL 1.2 spec says to return DAT_SRQ_IN_USE, which does > tduffy> + * not exist. Have filed the following as an eratta. > tduffy> + */ > tduffy> + status = DAT_ERROR(DAT_INVALID_STATE, > tduffy> + DAT_INVALID_STATE_SRQ_IN_USE); > tduffy> + goto bail; > tduffy> + } > tduffy> + > tduffy> + ia_ptr = srq_ptr->header.owner_ia; > tduffy> + > tduffy> + /* > tduffy> + * Do verification of parameters and the state change atomically. > tduffy> + */ > tduffy> + spin_lock_irqsave(&srq_ptr->header.lock, srq_ptr->header.flags); > tduffy> + > tduffy> + /* Remove the SRQ from the IA */ > tduffy> + dapl_ia_unlink_srq(ia_ptr, srq_ptr); > tduffy> + > tduffy> + spin_unlock_irqrestore(&srq_ptr->header.lock, srq_ptr->header.flags); > tduffy> + > tduffy> + /* > tduffy> + * Finish tearing everything down. > tduffy> + */ > tduffy> + > tduffy> + /* > tduffy> + * Take care of the transport resource > tduffy> + */ > tduffy> + > tduffy> + /* XXX Put provider code here!!! */ > tduffy> + > tduffy> + /* Free the resource */ > tduffy> + dapl_srq_dealloc(srq_ptr); > tduffy> + > tduffy> +bail: > tduffy> + return status; > tduffy> + > tduffy> +} > tduffy> + > tduffy> +/* > tduffy> + * dapl_srq_post_recv > tduffy> + * > tduffy> + * Post a receive buffer that can be used by any incoming > tduffy> + * message by any connected EP using the SRQ. Request to receive data > tduffy> + * over a connection of any ep handle into local_iov > tduffy> + * > tduffy> + * Input: > tduffy> + * srq_handle > tduffy> + * num_segments > tduffy> + * local_iov > tduffy> + * user_cookie > tduffy> + * > tduffy> + * Output: > tduffy> + * None. > tduffy> + * > tduffy> + * Returns: > tduffy> + * DAT_SUCCESS > tduffy> + * DAT_INSUFFICIENT_RESOURCES > tduffy> + * DAT_INVALID_PARAMETER > tduffy> + * DAT_INVALID_HANDLE > tduffy> + * DAT_INVALID_STATE > tduffy> + * DAT_PROTECTION_VIOLATION > tduffy> + * DAT_PROVILEGES_VIOLATION > tduffy> + */ > tduffy> +u32 dapl_srq_post_recv(DAT_SRQ_HANDLE srq_handle, int num_segments, > tduffy> + struct dat_lmr_triplet *local_iov, > tduffy> + DAT_DTO_COOKIE user_cookie) > tduffy> +{ > tduffy> + struct dapl_srq *srq_ptr; > tduffy> + struct dapl_cookie *cookie; > tduffy> + u32 status; > tduffy> + > tduffy> + dapl_dbg_log(DAPL_DBG_TYPE_API, > tduffy> + "dapl_srq_post_recv (%p, %d, %p, %P)\n", > tduffy> + srq_handle, num_segments, local_iov, user_cookie.as_64); > tduffy> + > tduffy> + if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { > tduffy> + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); > tduffy> + goto bail; > tduffy> + } > tduffy> + > tduffy> + srq_ptr = (struct dapl_srq *)srq_handle; > tduffy> + > tduffy> + /* > tduffy> + * Synchronization ok since this buffer is only used for receive > tduffy> + * requests, which aren't allowed to race with each other. The > tduffy> + * app must syncronize access to the SRQ. > tduffy> + */ > tduffy> + status = dapl_dto_cookie_alloc(&srq_ptr->recv_buffer, > tduffy> + DAPL_DTO_TYPE_RECV, > tduffy> + user_cookie, &cookie); > tduffy> + if (DAT_SUCCESS != status) > tduffy> + goto bail; > tduffy> + > tduffy> + /* > tduffy> + * Take reference before posting to avoid race conditions with > tduffy> + * completions > tduffy> + */ > tduffy> + atomic_inc(&srq_ptr->recv_count); > tduffy> + > tduffy> + /* > tduffy> + * Invoke provider specific routine to post DTO > tduffy> + */ > tduffy> + /* XXX Put code here XXX */ > tduffy> + /* XXX */ status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); > tduffy> + > tduffy> + if (status != DAT_SUCCESS) { > tduffy> + atomic_dec(&srq_ptr->recv_count); > tduffy> + dapl_cookie_dealloc(&srq_ptr->recv_buffer, cookie); > tduffy> + } > tduffy> + > tduffy> +bail: > tduffy> + return status; > tduffy> +} > tduffy> + > tduffy> +u32 dapl_srq_query(DAT_SRQ_HANDLE srq_handle, struct dat_srq_param *srq_param) > tduffy> +{ > tduffy> + struct dapl_srq *srq_ptr; > tduffy> + u32 status = DAT_SUCCESS; > tduffy> + > tduffy> + dapl_dbg_log(DAPL_DBG_TYPE_API, > tduffy> + "dapl_srq_query (%p, %x, %p)\n", > tduffy> + srq_handle, srq_param); > tduffy> + > tduffy> + if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { > tduffy> + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); > tduffy> + goto bail; > tduffy> + } > tduffy> + if (srq_param == NULL) { > tduffy> + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); > tduffy> + goto bail; > tduffy> + } > tduffy> + > tduffy> + srq_ptr = (struct dapl_srq *)srq_handle; > tduffy> + > tduffy> + /* > tduffy> + * XXX Need to calculate available_dto_count and outstanding_dto_count > tduffy> + */ > tduffy> + srq_ptr->param.available_dto_count = DAT_VALUE_UNKNOWN; > tduffy> + srq_ptr->param.outstanding_dto_count = DAT_VALUE_UNKNOWN; > tduffy> + > tduffy> + *srq_param = srq_ptr->param; > tduffy> + > tduffy> +bail: > tduffy> + return status; > tduffy> +} > tduffy> + > tduffy> +/* > tduffy> + * dapl_srq_resize > tduffy> + * > tduffy> + * Modify the size fo the event queue of a Shared Recieve Queue > tduffy> + * > tduffy> + * Input: > tduffy> + * srq_handle > tduffy> + * srq_max_recv_dto > tduffy> + * > tduffy> + * Output: > tduffy> + * none > tduffy> + * > tduffy> + * Returns: > tduffy> + * DAT_SUCCESS > tduffy> + * DAT_INVALID_HANDLE > tduffy> + * DAT_INVALID_PARAMETER > tduffy> + * DAT_INSUFFICIENT_RESOURCES > tduffy> + * DAT_INVALID_STATE > tduffy> + */ > tduffy> + > tduffy> +u32 dapl_srq_resize(DAT_SRQ_HANDLE srq_handle, int srq_max_recv_dto) > tduffy> +{ > tduffy> + struct dapl_ia *ia_ptr; > tduffy> + struct dapl_srq *srq_ptr; > tduffy> + u32 status = DAT_SUCCESS; > tduffy> + > tduffy> + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_resize (%p, %d)\n", > tduffy> + srq_handle, srq_max_recv_dto); > tduffy> + > tduffy> + if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { > tduffy> + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); > tduffy> + goto bail; > tduffy> + } > tduffy> + > tduffy> + srq_ptr = (struct dapl_srq *)srq_handle; > tduffy> + ia_ptr = srq_ptr->header.owner_ia; > tduffy> + > tduffy> + /* > tduffy> + * Check for nonsense requests per the spec > tduffy> + */ > tduffy> + if (srq_max_recv_dto <= srq_ptr->param.low_watermark) { > tduffy> + status = DAT_ERROR(DAT_INVALID_STATE, DAT_NO_SUBTYPE); > tduffy> + goto bail; > tduffy> + } > tduffy> + > tduffy> + /* XXX Put implementation here XXX */ > tduffy> + > tduffy> + /* XXX */ status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); > tduffy> + > tduffy> +bail: > tduffy> + return status; > tduffy> +} > tduffy> + > tduffy> +/* > tduffy> + * dapl_srq_set_lw > tduffy> + * > tduffy> + * DAPL Requirements Version 1.2, 6.5.4 > tduffy> + * > tduffy> + * Set the low water mark for an SRQ and arm the SRQ to generate an > tduffy> + * event if it is reached. > tduffy> + * > tduffy> + * Input: > tduffy> + * srq_handle > tduffy> + * srq_max_recv_dto > tduffy> + * > tduffy> + * Output: > tduffy> + * none > tduffy> + * > tduffy> + * Returns: > tduffy> + * DAT_SUCCESS > tduffy> + * DAT_INVALID_HANDLE > tduffy> + * DAT_INVALID_PARAMETER > tduffy> + * DAT_MODEL_NOT_SUPPORTED > tduffy> + */ > tduffy> + > tduffy> +u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, int low_watermark) > tduffy> +{ > tduffy> + struct dapl_srq *srq_ptr; > tduffy> + u32 status = DAT_SUCCESS; > tduffy> + > tduffy> + dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_set_lw (%p, %d)\n", > tduffy> + srq_handle, low_watermark); > tduffy> + > tduffy> + if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { > tduffy> + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); > tduffy> + goto bail; > tduffy> + } > tduffy> + > tduffy> + srq_ptr = (struct dapl_srq *)srq_handle; > tduffy> + > tduffy> + /* XXX Put implementation here XXX */ > tduffy> + > tduffy> + /* XXX */ status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); > tduffy> + > tduffy> +bail: > tduffy> + return status; > tduffy> +} > tduffy> Index: linux-kernel-srq/dat-provider/dapl_srq_resize.c > tduffy> =================================================================== > tduffy> --- linux-kernel-srq/dat-provider/dapl_srq_resize.c (revision 2489) > tduffy> +++ linux-kernel-srq/dat-provider/dapl_srq_resize.c (working copy) > tduffy> @@ -1,88 +0,0 @@ > tduffy> -/* > tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. > tduffy> - * > tduffy> - * This Software is licensed under one of the following licenses: > tduffy> - * > tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is > tduffy> - * available from the Open Source Initiative, see > tduffy> - * http://www.opensource.org/licenses/cpl.php. > tduffy> - * > tduffy> - * 2) under the terms of the "The BSD License" a copy of which is > tduffy> - * available from the Open Source Initiative, see > tduffy> - * http://www.opensource.org/licenses/bsd-license.php. > tduffy> - * > tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a > tduffy> - * copy of which is available from the Open Source Initiative, see > tduffy> - * http://www.opensource.org/licenses/gpl-license.php. > tduffy> - * > tduffy> - * Licensee has the right to choose one of the above licenses. > tduffy> - * > tduffy> - * Redistributions of source code must retain the above copyright > tduffy> - * notice and one of the license notices. > tduffy> - * > tduffy> - * Redistributions in binary form must reproduce both the above copyright > tduffy> - * notice, one of the license notices in the documentation > tduffy> - * and/or other materials provided with the distribution. > tduffy> - */ > tduffy> - > tduffy> -/* > tduffy> - * $Id$ > tduffy> - */ > tduffy> - > tduffy> -#include "dapl.h" > tduffy> -#include "dapl_srq_util.h" > tduffy> -#include "dapl_openib_util.h" > tduffy> - > tduffy> -/* > tduffy> - * dapl_srq_resize > tduffy> - * > tduffy> - * Modify the size fo the event queue of a Shared Recieve Queue > tduffy> - * > tduffy> - * Input: > tduffy> - * srq_handle > tduffy> - * srq_max_recv_dto > tduffy> - * > tduffy> - * Output: > tduffy> - * none > tduffy> - * > tduffy> - * Returns: > tduffy> - * DAT_SUCCESS > tduffy> - * DAT_INVALID_HANDLE > tduffy> - * DAT_INVALID_PARAMETER > tduffy> - * DAT_INSUFFICIENT_RESOURCES > tduffy> - * DAT_INVALID_STATE > tduffy> - */ > tduffy> - > tduffy> -u32 dapl_srq_resize(DAT_SRQ_HANDLE srq_handle, int srq_max_recv_dto) > tduffy> -{ > tduffy> - struct dapl_ia *ia_ptr; > tduffy> - struct dapl_srq *srq_ptr; > tduffy> - u32 dat_status = DAT_SUCCESS; > tduffy> - > tduffy> - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_resize (%p, %d)\n", > tduffy> - srq_handle, srq_max_recv_dto); > tduffy> - > tduffy> - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { > tduffy> - dat_status = > tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); > tduffy> - goto bail; > tduffy> - } > tduffy> - > tduffy> - srq_ptr = (struct dapl_srq *)srq_handle; > tduffy> - ia_ptr = srq_ptr->header.owner_ia; > tduffy> - > tduffy> - /* > tduffy> - * Check for nonsense requests per the spec > tduffy> - */ > tduffy> - if (srq_max_recv_dto <= srq_ptr->param.low_watermark) { > tduffy> - dat_status = DAT_ERROR(DAT_INVALID_STATE, DAT_NO_SUBTYPE); > tduffy> - goto bail; > tduffy> - } > tduffy> - > tduffy> - /* XXX Put implementation here XXX */ > tduffy> - > tduffy> - /* XXX */ dat_status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); > tduffy> - > tduffy> - bail: > tduffy> - return dat_status; > tduffy> -} > tduffy> Index: linux-kernel-srq/dat-provider/dapl_srq_create.c > tduffy> =================================================================== > tduffy> --- linux-kernel-srq/dat-provider/dapl_srq_create.c (revision 2489) > tduffy> +++ linux-kernel-srq/dat-provider/dapl_srq_create.c (working copy) > tduffy> @@ -1,128 +0,0 @@ > tduffy> -/* > tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. > tduffy> - * > tduffy> - * This Software is licensed under one of the following licenses: > tduffy> - * > tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is > tduffy> - * available from the Open Source Initiative, see > tduffy> - * http://www.opensource.org/licenses/cpl.php. > tduffy> - * > tduffy> - * 2) under the terms of the "The BSD License" a copy of which is > tduffy> - * available from the Open Source Initiative, see > tduffy> - * http://www.opensource.org/licenses/bsd-license.php. > tduffy> - * > tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a > tduffy> - * copy of which is available from the Open Source Initiative, see > tduffy> - * http://www.opensource.org/licenses/gpl-license.php. > tduffy> - * > tduffy> - * Licensee has the right to choose one of the above licenses. > tduffy> - * > tduffy> - * Redistributions of source code must retain the above copyright > tduffy> - * notice and one of the license notices. > tduffy> - * > tduffy> - * Redistributions in binary form must reproduce both the above copyright > tduffy> - * notice, one of the license notices in the documentation > tduffy> - * and/or other materials provided with the distribution. > tduffy> - */ > tduffy> - > tduffy> -/* > tduffy> - * $Id$ > tduffy> - */ > tduffy> - > tduffy> -#include "dapl.h" > tduffy> -#include "dapl_ia_util.h" > tduffy> -#include "dapl_srq_util.h" > tduffy> -#include "dapl_openib_util.h" > tduffy> - > tduffy> -/* > tduffy> - * dapl_srq_create > tduffy> - * > tduffy> - * Create an instance of a Shared Receive Queue that is provided to the > tduffy> - * consumer at srq_handle. > tduffy> - * > tduffy> - * Input: > tduffy> - * ia_handle > tduffy> - * pz_handle > tduffy> - * srq_attr > tduffy> - * > tduffy> - * Output: > tduffy> - * srq_handle > tduffy> - * > tduffy> - * Returns: > tduffy> - * DAT_SUCCESS > tduffy> - * DAT_INSUFFICIENT_RESOURCES > tduffy> - * DAT_INVALID_HANDLE > tduffy> - * DAT_INVALID_PARAMETER > tduffy> - * ?DAT_INVALID_ATTRIBUTE?? > tduffy> - * DAT_MODEL_NOT_SUPPORTED > tduffy> - */ > tduffy> -u32 dapl_srq_create(DAT_IA_HANDLE ia_handle, DAT_PZ_HANDLE pz_handle, > tduffy> - struct dat_srq_attr *srq_attr, DAT_SRQ_HANDLE *srq_handle) > tduffy> -{ > tduffy> - struct dapl_ia *ia_ptr; > tduffy> - struct dapl_srq *srq_ptr; > tduffy> - u32 dat_status = DAT_SUCCESS; > tduffy> - > tduffy> - dapl_dbg_log(DAPL_DBG_TYPE_API, > tduffy> - "dapl_srq_create (%p, %p, %p, %p)\n", > tduffy> - ia_handle, pz_handle, srq_attr, srq_handle); > tduffy> - > tduffy> - ia_ptr = (struct dapl_ia *)ia_handle; > tduffy> - > tduffy> - /* > tduffy> - * Verify parameters > tduffy> - */ > tduffy> - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { > tduffy> - dat_status = > tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); > tduffy> - goto bail; > tduffy> - } > tduffy> - > tduffy> - /* > tduffy> - * Verify non-required parameters. > tduffy> - * N.B. Assumption: any parameter that can be > tduffy> - * modified by dat_ep_modify() is not strictly > tduffy> - * required when the EP is created > tduffy> - */ > tduffy> - if (pz_handle != NULL && DAPL_BAD_HANDLE(pz_handle, DAPL_MAGIC_PZ)) { > tduffy> - dat_status = > tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PZ); > tduffy> - goto bail; > tduffy> - } > tduffy> - > tduffy> - if (srq_handle == NULL) { > tduffy> - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); > tduffy> - goto bail; > tduffy> - } > tduffy> - if ((unsigned long)srq_attr & 3) { > tduffy> - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); > tduffy> - goto bail; > tduffy> - } > tduffy> - > tduffy> - /* Allocate SRQ */ > tduffy> - srq_ptr = dapl_srq_alloc(ia_ptr, srq_attr); > tduffy> - if (srq_ptr == NULL) { > tduffy> - dat_status = > tduffy> - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); > tduffy> - goto bail; > tduffy> - } > tduffy> - > tduffy> - srq_ptr->param.ia_handle = (DAT_IA_HANDLE) ia_ptr; > tduffy> - srq_ptr->param.srq_state = DAT_SRQ_STATE_OPERATIONAL; > tduffy> - srq_ptr->param.pz_handle = pz_handle; > tduffy> - > tduffy> - /* > tduffy> - * XXX Allocate provider resource here!!! > tduffy> - */ > tduffy> - /* XXX */ dat_status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); > tduffy> - /* XXX */ dapl_srq_dealloc(srq_ptr); > tduffy> - /* XXX */ goto bail; > tduffy> - > tduffy> - /* Link it onto the IA */ > tduffy> - dapl_ia_link_srq(ia_ptr, srq_ptr); > tduffy> - > tduffy> - *srq_handle = srq_ptr; > tduffy> - > tduffy> - bail: > tduffy> - return dat_status; > tduffy> -} > tduffy> Index: linux-kernel-srq/dat-provider/dapl_srq_query.c > tduffy> =================================================================== > tduffy> --- linux-kernel-srq/dat-provider/dapl_srq_query.c (revision 2489) > tduffy> +++ linux-kernel-srq/dat-provider/dapl_srq_query.c (working copy) > tduffy> @@ -1,65 +0,0 @@ > tduffy> -/* > tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. > tduffy> - * > tduffy> - * This Software is licensed under one of the following licenses: > tduffy> - * > tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is > tduffy> - * available from the Open Source Initiative, see > tduffy> - * http://www.opensource.org/licenses/cpl.php. > tduffy> - * > tduffy> - * 2) under the terms of the "The BSD License" a copy of which is > tduffy> - * available from the Open Source Initiative, see > tduffy> - * http://www.opensource.org/licenses/bsd-license.php. > tduffy> - * > tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a > tduffy> - * copy of which is available from the Open Source Initiative, see > tduffy> - * http://www.opensource.org/licenses/gpl-license.php. > tduffy> - * > tduffy> - * Licensee has the right to choose one of the above licenses. > tduffy> - * > tduffy> - * Redistributions of source code must retain the above copyright > tduffy> - * notice and one of the license notices. > tduffy> - * > tduffy> - * Redistributions in binary form must reproduce both the above copyright > tduffy> - * notice, one of the license notices in the documentation > tduffy> - * and/or other materials provided with the distribution. > tduffy> - */ > tduffy> - > tduffy> -/* > tduffy> - * $Id$ > tduffy> - */ > tduffy> - > tduffy> -#include "dapl.h" > tduffy> - > tduffy> -u32 dapl_srq_query(DAT_SRQ_HANDLE srq_handle, struct dat_srq_param *srq_param) > tduffy> -{ > tduffy> - struct dapl_srq *srq_ptr; > tduffy> - u32 dat_status = DAT_SUCCESS; > tduffy> - > tduffy> - dapl_dbg_log(DAPL_DBG_TYPE_API, > tduffy> - "dapl_srq_query (%p, %x, %p)\n", > tduffy> - srq_handle, srq_param); > tduffy> - > tduffy> - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { > tduffy> - dat_status = > tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); > tduffy> - goto bail; > tduffy> - } > tduffy> - if (srq_param == NULL) { > tduffy> - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); > tduffy> - goto bail; > tduffy> - } > tduffy> - > tduffy> - srq_ptr = (struct dapl_srq *)srq_handle; > tduffy> - > tduffy> - /* > tduffy> - * XXX Need to calculate available_dto_count and outstanding_dto_count > tduffy> - */ > tduffy> - srq_ptr->param.available_dto_count = DAT_VALUE_UNKNOWN; > tduffy> - srq_ptr->param.outstanding_dto_count = DAT_VALUE_UNKNOWN; > tduffy> - > tduffy> - *srq_param = srq_ptr->param; > tduffy> - > tduffy> - bail: > tduffy> - return dat_status; > tduffy> -} > tduffy> Index: linux-kernel-srq/dat-provider/Makefile > tduffy> =================================================================== > tduffy> --- linux-kernel-srq/dat-provider/Makefile (revision 2489) > tduffy> +++ linux-kernel-srq/dat-provider/Makefile (working copy) > tduffy> @@ -82,13 +82,7 @@ PROVIDER_MODULES := \ > tduffy> dapl_rsp_query \ > tduffy> dapl_set_consumer_context \ > tduffy> dapl_sp_util \ > tduffy> - dapl_srq_create \ > tduffy> - dapl_srq_free \ > tduffy> - dapl_srq_post_recv \ > tduffy> - dapl_srq_query \ > tduffy> - dapl_srq_resize \ > tduffy> - dapl_srq_set_lw \ > tduffy> - dapl_srq_util \ > tduffy> + dapl_srq \ > tduffy> dapl_timer_util \ > tduffy> dapl_util > tduffy> > tduffy> Index: linux-kernel-srq/dat-provider/dapl_srq_util.c > tduffy> =================================================================== > tduffy> --- linux-kernel-srq/dat-provider/dapl_srq_util.c (revision 2489) > tduffy> +++ linux-kernel-srq/dat-provider/dapl_srq_util.c (working copy) > tduffy> @@ -1,126 +0,0 @@ > tduffy> -/* > tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. > tduffy> - * > tduffy> - * This Software is licensed under one of the following licenses: > tduffy> - * > tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is > tduffy> - * available from the Open Source Initiative, see > tduffy> - * http://www.opensource.org/licenses/cpl.php. > tduffy> - * > tduffy> - * 2) under the terms of the "The BSD License" a copy of which is > tduffy> - * available from the Open Source Initiative, see > tduffy> - * http://www.opensource.org/licenses/bsd-license.php. > tduffy> - * > tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a > tduffy> - * copy of which is available from the Open Source Initiative, see > tduffy> - * http://www.opensource.org/licenses/gpl-license.php. > tduffy> - * > tduffy> - * Licensee has the right to choose one of the above licenses. > tduffy> - * > tduffy> - * Redistributions of source code must retain the above copyright > tduffy> - * notice and one of the license notices. > tduffy> - * > tduffy> - * Redistributions in binary form must reproduce both the above copyright > tduffy> - * notice, one of the license notices in the documentation > tduffy> - * and/or other materials provided with the distribution. > tduffy> - */ > tduffy> - > tduffy> -/* > tduffy> - * $Id$ > tduffy> - */ > tduffy> - > tduffy> -#include "dapl_srq_util.h" > tduffy> -#include "dapl_ia_util.h" > tduffy> -#include "dapl_cookie.h" > tduffy> - > tduffy> -/* > tduffy> - * dapl_srq_alloc > tduffy> - * > tduffy> - * alloc and initialize an SRQ struct > tduffy> - * > tduffy> - * Input: > tduffy> - * IA INFO struct ptr > tduffy> - * SRQ ATTR ptr > tduffy> - * > tduffy> - * Output: > tduffy> - * none > tduffy> - * > tduffy> - * Returns: > tduffy> - * pointer to srq > tduffy> - * > tduffy> - */ > tduffy> -struct dapl_srq *dapl_srq_alloc(struct dapl_ia *ia_ptr, > tduffy> - const struct dat_srq_attr *srq_attr) > tduffy> -{ > tduffy> - struct dapl_srq *srq_ptr; > tduffy> - > tduffy> - /* Allocate SRQ */ > tduffy> - srq_ptr = kmalloc(sizeof *srq_ptr, GFP_ATOMIC); > tduffy> - if (!srq_ptr) > tduffy> - goto bail; > tduffy> - > tduffy> - /* zero the structure */ > tduffy> - memset(srq_ptr, 0, sizeof *srq_ptr); > tduffy> - > tduffy> - /* > tduffy> - * initialize the header > tduffy> - */ > tduffy> - srq_ptr->header.provider = ia_ptr->header.provider; > tduffy> - srq_ptr->header.magic = DAPL_MAGIC_SRQ; > tduffy> - srq_ptr->header.handle_type = DAT_HANDLE_TYPE_SRQ; > tduffy> - srq_ptr->header.owner_ia = ia_ptr; > tduffy> - srq_ptr->header.user_context.as_64 = 0; > tduffy> - srq_ptr->header.user_context.as_ptr = NULL; > tduffy> - atomic_set(&srq_ptr->srq_ref_count, 0); > tduffy> - > tduffy> - dapl_llist_init_entry(&srq_ptr->header.ia_list_entry); > tduffy> - spin_lock_init(&srq_ptr->header.lock); > tduffy> - > tduffy> - /* > tduffy> - * Initialize the body. > tduffy> - * XXX Assume srq_attrs is required > tduffy> - */ > tduffy> - srq_ptr->param.max_recv_dtos = srq_attr->max_recv_dtos; > tduffy> - srq_ptr->param.max_recv_iov = srq_attr->max_recv_iov; > tduffy> - srq_ptr->param.low_watermark = srq_attr->low_watermark; > tduffy> - > tduffy> - /* Get a cookie buffer to track outstanding recvs */ > tduffy> - if (DAT_SUCCESS != dapl_cb_create(&srq_ptr->recv_buffer, > tduffy> - (struct dapl_ep *)srq_ptr, > tduffy> - srq_ptr->param.max_recv_dtos)) { > tduffy> - dapl_srq_dealloc(srq_ptr); > tduffy> - srq_ptr = NULL; > tduffy> - goto bail; > tduffy> - } > tduffy> - > tduffy> -bail: > tduffy> - return srq_ptr; > tduffy> -} > tduffy> - > tduffy> -/* > tduffy> - * dapl_srq_dealloc > tduffy> - * > tduffy> - * Free the passed in SRQ structure. > tduffy> - * > tduffy> - * Input: > tduffy> - * SRQ pointer > tduffy> - * > tduffy> - * Output: > tduffy> - * none > tduffy> - * > tduffy> - * Returns: > tduffy> - * none > tduffy> - * > tduffy> - */ > tduffy> -void dapl_srq_dealloc(struct dapl_srq *srq_ptr) > tduffy> -{ > tduffy> - dapl_os_assert(srq_ptr->header.magic == DAPL_MAGIC_SRQ); > tduffy> - > tduffy> - /* reset magic to prevent reuse */ > tduffy> - srq_ptr->header.magic = DAPL_MAGIC_INVALID; > tduffy> - dapl_ia_unlink_srq(srq_ptr->header.owner_ia, srq_ptr); > tduffy> - dapl_cb_free(&srq_ptr->recv_buffer); > tduffy> - /* no need to destroy srq_ptr->header.lock */ > tduffy> - > tduffy> - kfree(srq_ptr); > tduffy> -} > tduffy> Index: linux-kernel-srq/dat-provider/dapl_srq_util.h > tduffy> =================================================================== > tduffy> --- linux-kernel-srq/dat-provider/dapl_srq_util.h (revision 2489) > tduffy> +++ linux-kernel-srq/dat-provider/dapl_srq_util.h (working copy) > tduffy> @@ -1,47 +0,0 @@ > tduffy> -/* > tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. > tduffy> - * > tduffy> - * This Software is licensed under one of the following licenses: > tduffy> - * > tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is > tduffy> - * available from the Open Source Initiative, see > tduffy> - * http://www.opensource.org/licenses/cpl.php. > tduffy> - * > tduffy> - * 2) under the terms of the "The BSD License" a copy of which is > tduffy> - * available from the Open Source Initiative, see > tduffy> - * http://www.opensource.org/licenses/bsd-license.php. > tduffy> - * > tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a > tduffy> - * copy of which is available from the Open Source Initiative, see > tduffy> - * http://www.opensource.org/licenses/gpl-license.php. > tduffy> - * > tduffy> - * Licensee has the right to choose one of the above licenses. > tduffy> - * > tduffy> - * Redistributions of source code must retain the above copyright > tduffy> - * notice and one of the license notices. > tduffy> - * > tduffy> - * Redistributions in binary form must reproduce both the above copyright > tduffy> - * notice, one of the license notices in the documentation > tduffy> - * and/or other materials provided with the distribution. > tduffy> - */ > tduffy> - > tduffy> -/********************************************************************** > tduffy> - * > tduffy> - * HEADER: dapl_srq_util.h > tduffy> - * > tduffy> - * PURPOSE: Utility defs & routines for the SRQ data structure > tduffy> - * > tduffy> - * $Id$ > tduffy> - **********************************************************************/ > tduffy> - > tduffy> -#ifndef DAPL_SRQ_UTIL_H > tduffy> -#define DAPL_SRQ_UTIL_H > tduffy> - > tduffy> -#include "dapl.h" > tduffy> - > tduffy> -extern struct dapl_srq *dapl_srq_alloc(struct dapl_ia *ia, > tduffy> - const struct dat_srq_attr *srq_attr); > tduffy> - > tduffy> -extern void dapl_srq_dealloc(struct dapl_srq *srq_ptr); > tduffy> - > tduffy> -#endif /* DAPL_SRQ_UTIL_H */ > tduffy> Index: linux-kernel-srq/dat-provider/dapl_srq_set_lw.c > tduffy> =================================================================== > tduffy> --- linux-kernel-srq/dat-provider/dapl_srq_set_lw.c (revision 2489) > tduffy> +++ linux-kernel-srq/dat-provider/dapl_srq_set_lw.c (working copy) > tduffy> @@ -1,80 +0,0 @@ > tduffy> -/* > tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. > tduffy> - * > tduffy> - * This Software is licensed under one of the following licenses: > tduffy> - * > tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is > tduffy> - * available from the Open Source Initiative, see > tduffy> - * http://www.opensource.org/licenses/cpl.php. > tduffy> - * > tduffy> - * 2) under the terms of the "The BSD License" a copy of which is > tduffy> - * available from the Open Source Initiative, see > tduffy> - * http://www.opensource.org/licenses/bsd-license.php. > tduffy> - * > tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a > tduffy> - * copy of which is available from the Open Source Initiative, see > tduffy> - * http://www.opensource.org/licenses/gpl-license.php. > tduffy> - * > tduffy> - * Licensee has the right to choose one of the above licenses. > tduffy> - * > tduffy> - * Redistributions of source code must retain the above copyright > tduffy> - * notice and one of the license notices. > tduffy> - * > tduffy> - * Redistributions in binary form must reproduce both the above copyright > tduffy> - * notice, one of the license notices in the documentation > tduffy> - * and/or other materials provided with the distribution. > tduffy> - */ > tduffy> - > tduffy> -/* > tduffy> - * $Id$ > tduffy> - */ > tduffy> - > tduffy> -#include "dapl.h" > tduffy> -#include "dapl_srq_util.h" > tduffy> -#include "dapl_openib_util.h" > tduffy> - > tduffy> -/* > tduffy> - * dapl_srq_set_lw > tduffy> - * > tduffy> - * DAPL Requirements Version 1.2, 6.5.4 > tduffy> - * > tduffy> - * Set the low water mark for an SRQ and arm the SRQ to generate an > tduffy> - * event if it is reached. > tduffy> - * > tduffy> - * Input: > tduffy> - * srq_handle > tduffy> - * srq_max_recv_dto > tduffy> - * > tduffy> - * Output: > tduffy> - * none > tduffy> - * > tduffy> - * Returns: > tduffy> - * DAT_SUCCESS > tduffy> - * DAT_INVALID_HANDLE > tduffy> - * DAT_INVALID_PARAMETER > tduffy> - * DAT_MODEL_NOT_SUPPORTED > tduffy> - */ > tduffy> - > tduffy> -u32 dapl_srq_set_lw(DAT_SRQ_HANDLE srq_handle, int low_watermark) > tduffy> -{ > tduffy> - struct dapl_srq *srq_ptr; > tduffy> - u32 dat_status = DAT_SUCCESS; > tduffy> - > tduffy> - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_set_lw (%p, %d)\n", > tduffy> - srq_handle, low_watermark); > tduffy> - > tduffy> - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { > tduffy> - dat_status = > tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); > tduffy> - goto bail; > tduffy> - } > tduffy> - > tduffy> - srq_ptr = (struct dapl_srq *)srq_handle; > tduffy> - > tduffy> - /* XXX Put implementation here XXX */ > tduffy> - > tduffy> - /* XXX */ dat_status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); > tduffy> - > tduffy> - bail: > tduffy> - return dat_status; > tduffy> -} > tduffy> Index: linux-kernel-srq/dat-provider/dapl_srq_post_recv.c > tduffy> =================================================================== > tduffy> --- linux-kernel-srq/dat-provider/dapl_srq_post_recv.c (revision 2489) > tduffy> +++ linux-kernel-srq/dat-provider/dapl_srq_post_recv.c (working copy) > tduffy> @@ -1,112 +0,0 @@ > tduffy> -/* > tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. > tduffy> - * > tduffy> - * This Software is licensed under one of the following licenses: > tduffy> - * > tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is > tduffy> - * available from the Open Source Initiative, see > tduffy> - * http://www.opensource.org/licenses/cpl.php. > tduffy> - * > tduffy> - * 2) under the terms of the "The BSD License" a copy of which is > tduffy> - * available from the Open Source Initiative, see > tduffy> - * http://www.opensource.org/licenses/bsd-license.php. > tduffy> - * > tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a > tduffy> - * copy of which is available from the Open Source Initiative, see > tduffy> - * http://www.opensource.org/licenses/gpl-license.php. > tduffy> - * > tduffy> - * Licensee has the right to choose one of the above licenses. > tduffy> - * > tduffy> - * Redistributions of source code must retain the above copyright > tduffy> - * notice and one of the license notices. > tduffy> - * > tduffy> - * Redistributions in binary form must reproduce both the above copyright > tduffy> - * notice, one of the license notices in the documentation > tduffy> - * and/or other materials provided with the distribution. > tduffy> - */ > tduffy> - > tduffy> -/* > tduffy> - * $Id$ > tduffy> - */ > tduffy> - > tduffy> -#include "dapl.h" > tduffy> -#include "dapl_cookie.h" > tduffy> -#include "dapl_openib_util.h" > tduffy> - > tduffy> -/* > tduffy> - * dapl_srq_post_recv > tduffy> - * > tduffy> - * Post a receive buffer that can be used by any incoming > tduffy> - * message by any connected EP using the SRQ. Request to receive data > tduffy> - * over a connection of any ep handle into local_iov > tduffy> - * > tduffy> - * Input: > tduffy> - * srq_handle > tduffy> - * num_segments > tduffy> - * local_iov > tduffy> - * user_cookie > tduffy> - * > tduffy> - * Output: > tduffy> - * None. > tduffy> - * > tduffy> - * Returns: > tduffy> - * DAT_SUCCESS > tduffy> - * DAT_INSUFFICIENT_RESOURCES > tduffy> - * DAT_INVALID_PARAMETER > tduffy> - * DAT_INVALID_HANDLE > tduffy> - * DAT_INVALID_STATE > tduffy> - * DAT_PROTECTION_VIOLATION > tduffy> - * DAT_PROVILEGES_VIOLATION > tduffy> - */ > tduffy> -u32 dapl_srq_post_recv(DAT_SRQ_HANDLE srq_handle, int num_segments, > tduffy> - struct dat_lmr_triplet *local_iov, > tduffy> - DAT_DTO_COOKIE user_cookie) > tduffy> -{ > tduffy> - struct dapl_srq *srq_ptr; > tduffy> - struct dapl_cookie *cookie; > tduffy> - u32 dat_status; > tduffy> - > tduffy> - dapl_dbg_log(DAPL_DBG_TYPE_API, > tduffy> - "dapl_srq_post_recv (%p, %d, %p, %P)\n", > tduffy> - srq_handle, num_segments, local_iov, user_cookie.as_64); > tduffy> - > tduffy> - if (DAPL_BAD_HANDLE(srq_handle, DAPL_MAGIC_SRQ)) { > tduffy> - dat_status = > tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); > tduffy> - goto bail; > tduffy> - } > tduffy> - > tduffy> - srq_ptr = (struct dapl_srq *)srq_handle; > tduffy> - > tduffy> - /* > tduffy> - * Synchronization ok since this buffer is only used for receive > tduffy> - * requests, which aren't allowed to race with each other. The > tduffy> - * app must syncronize access to the SRQ. > tduffy> - */ > tduffy> - dat_status = dapl_dto_cookie_alloc(&srq_ptr->recv_buffer, > tduffy> - DAPL_DTO_TYPE_RECV, > tduffy> - user_cookie, &cookie); > tduffy> - if (DAT_SUCCESS != dat_status) { > tduffy> - goto bail; > tduffy> - } > tduffy> - > tduffy> - /* > tduffy> - * Take reference before posting to avoid race conditions with > tduffy> - * completions > tduffy> - */ > tduffy> - atomic_inc(&srq_ptr->recv_count); > tduffy> - > tduffy> - /* > tduffy> - * Invoke provider specific routine to post DTO > tduffy> - */ > tduffy> - /* XXX Put code here XXX */ > tduffy> - /* XXX */ dat_status = DAT_ERROR(DAT_NOT_IMPLEMENTED, DAT_NO_SUBTYPE); > tduffy> - > tduffy> - if (dat_status != DAT_SUCCESS) { > tduffy> - atomic_dec(&srq_ptr->recv_count); > tduffy> - dapl_cookie_dealloc(&srq_ptr->recv_buffer, cookie); > tduffy> - } > tduffy> - > tduffy> - bail: > tduffy> - return dat_status; > tduffy> -} > tduffy> Index: linux-kernel-srq/dat-provider/dapl_srq_free.c > tduffy> =================================================================== > tduffy> --- linux-kernel-srq/dat-provider/dapl_srq_free.c (revision 2489) > tduffy> +++ linux-kernel-srq/dat-provider/dapl_srq_free.c (working copy) > tduffy> @@ -1,112 +0,0 @@ > tduffy> -/* > tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. > tduffy> - * > tduffy> - * This Software is licensed under one of the following licenses: > tduffy> - * > tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is > tduffy> - * available from the Open Source Initiative, see > tduffy> - * http://www.opensource.org/licenses/cpl.php. > tduffy> - * > tduffy> - * 2) under the terms of the "The BSD License" a copy of which is > tduffy> - * available from the Open Source Initiative, see > tduffy> - * http://www.opensource.org/licenses/bsd-license.php. > tduffy> - * > tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a > tduffy> - * copy of which is available from the Open Source Initiative, see > tduffy> - * http://www.opensource.org/licenses/gpl-license.php. > tduffy> - * > tduffy> - * Licensee has the right to choose one of the above licenses. > tduffy> - * > tduffy> - * Redistributions of source code must retain the above copyright > tduffy> - * notice and one of the license notices. > tduffy> - * > tduffy> - * Redistributions in binary form must reproduce both the above copyright > tduffy> - * notice, one of the license notices in the documentation > tduffy> - * and/or other materials provided with the distribution. > tduffy> - */ > tduffy> - > tduffy> -/* > tduffy> - * $Id$ > tduffy> - */ > tduffy> - > tduffy> -#include "dapl.h" > tduffy> -#include "dapl_ia_util.h" > tduffy> -#include "dapl_srq_util.h" > tduffy> -#include "dapl_openib_util.h" > tduffy> - > tduffy> -/* > tduffy> - * dapl_srq_free > tduffy> - * > tduffy> - * Destroy an instance of an SRQ > tduffy> - * > tduffy> - * Input: > tduffy> - * srq_handle > tduffy> - * > tduffy> - * Output: > tduffy> - * none > tduffy> - * > tduffy> - * Returns: > tduffy> - * DAT_SUCCESS > tduffy> - * DAT_INVALID_PARAMETER > tduffy> - * DAT_INVALID_STATE > tduffy> - */ > tduffy> -u32 dapl_srq_free(DAT_SRQ_HANDLE srq_handle) > tduffy> -{ > tduffy> - struct dapl_srq *srq_ptr; > tduffy> - struct dapl_ia *ia_ptr; > tduffy> - struct dat_srq_param *param; > tduffy> - u32 dat_status = DAT_SUCCESS; > tduffy> - > tduffy> - dapl_dbg_log(DAPL_DBG_TYPE_API, "dapl_srq_free (%p)\n", srq_handle); > tduffy> - > tduffy> - srq_ptr = (struct dapl_srq *)srq_handle; > tduffy> - param = &srq_ptr->param; > tduffy> - > tduffy> - /* > tduffy> - * Verify parameter & state > tduffy> - */ > tduffy> - if (DAPL_BAD_HANDLE(srq_ptr, DAPL_MAGIC_SRQ)) { > tduffy> - dat_status = > tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_SRQ); > tduffy> - goto bail; > tduffy> - } > tduffy> - > tduffy> - if (atomic_read(&srq_ptr->srq_ref_count) != 0) { > tduffy> - /* > tduffy> - * The DAPL 1.2 spec says to return DAT_SRQ_IN_USE, which does > tduffy> - * not exist. Have filed the following as an eratta. > tduffy> - */ > tduffy> - dat_status = > tduffy> - DAT_ERROR(DAT_INVALID_STATE, DAT_INVALID_STATE_SRQ_IN_USE); > tduffy> - goto bail; > tduffy> - } > tduffy> - > tduffy> - ia_ptr = srq_ptr->header.owner_ia; > tduffy> - > tduffy> - /* > tduffy> - * Do verification of parameters and the state change atomically. > tduffy> - */ > tduffy> - spin_lock_irqsave(&srq_ptr->header.lock, srq_ptr->header.flags); > tduffy> - > tduffy> - /* Remove the SRQ from the IA */ > tduffy> - dapl_ia_unlink_srq(ia_ptr, srq_ptr); > tduffy> - > tduffy> - spin_unlock_irqrestore(&srq_ptr->header.lock, srq_ptr->header.flags); > tduffy> - > tduffy> - /* > tduffy> - * Finish tearing everything down. > tduffy> - */ > tduffy> - > tduffy> - /* > tduffy> - * Take care of the transport resource > tduffy> - */ > tduffy> - > tduffy> - /* XXX Put provider code here!!! */ > tduffy> - > tduffy> - /* Free the resource */ > tduffy> - dapl_srq_dealloc(srq_ptr); > tduffy> - > tduffy> - bail: > tduffy> - return dat_status; > tduffy> - > tduffy> -} > tduffy> Index: linux-kernel-srq/patches/alt_dat_provider_makefile > tduffy> =================================================================== > tduffy> --- linux-kernel-srq/patches/alt_dat_provider_makefile (revision 2489) > tduffy> +++ linux-kernel-srq/patches/alt_dat_provider_makefile (working copy) > tduffy> @@ -76,13 +76,7 @@ PROVIDER_MODULES := \ > tduffy> dapl_rsp_query \ > tduffy> dapl_set_consumer_context \ > tduffy> dapl_sp_util \ > tduffy> - dapl_srq_create \ > tduffy> - dapl_srq_free \ > tduffy> - dapl_srq_post_recv \ > tduffy> - dapl_srq_query \ > tduffy> - dapl_srq_resize \ > tduffy> - dapl_srq_set_lw \ > tduffy> - dapl_srq_util \ > tduffy> + dapl_srq \ > tduffy> dapl_timer_util \ > tduffy> dapl_util > tduffy> > tduffy> > From jlentini at netapp.com Fri May 27 08:21:34 2005 From: jlentini at netapp.com (James Lentini) Date: Fri, 27 May 2005 11:21:34 -0400 (EDT) Subject: [openib-general] Re: [PATCH 3/3] kDAPL: consolidate dapl_psp*.c into one file In-Reply-To: <1117133456.17831.16.camel@duffman> References: <1117133253.17831.12.camel@duffman> <1117133397.17831.15.camel@duffman> <1117133456.17831.16.camel@duffman> Message-ID: Committed in revision 2502. On Thu, 26 May 2005, Tom Duffy wrote: tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel-rsp/dat-provider/dapl_psp.c linux-kernel-psp/dat-provider/dapl_psp.c tduffy> --- linux-kernel-rsp/dat-provider/dapl_psp.c 1969-12-31 16:00:00.000000000 -0800 tduffy> +++ linux-kernel-psp/dat-provider/dapl_psp.c 2005-05-25 15:11:42.274000000 -0700 tduffy> @@ -0,0 +1,446 @@ tduffy> +/* tduffy> + * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> + * tduffy> + * This Software is licensed under one of the following licenses: tduffy> + * tduffy> + * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> + * available from the Open Source Initiative, see tduffy> + * http://www.opensource.org/licenses/cpl.php. tduffy> + * tduffy> + * 2) under the terms of the "The BSD License" a copy of which is tduffy> + * available from the Open Source Initiative, see tduffy> + * http://www.opensource.org/licenses/bsd-license.php. tduffy> + * tduffy> + * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> + * copy of which is available from the Open Source Initiative, see tduffy> + * http://www.opensource.org/licenses/gpl-license.php. tduffy> + * tduffy> + * Licensee has the right to choose one of the above licenses. tduffy> + * tduffy> + * Redistributions of source code must retain the above copyright tduffy> + * notice and one of the license notices. tduffy> + * tduffy> + * Redistributions in binary form must reproduce both the above copyright tduffy> + * notice, one of the license notices in the documentation tduffy> + * and/or other materials provided with the distribution. tduffy> + */ tduffy> + tduffy> +/* tduffy> + * $Id: dapl_psp_create_any.c 2462 2005-05-24 02:28:24Z jlentini $ tduffy> + */ tduffy> + tduffy> +#include "dapl.h" tduffy> +#include "dapl_sp_util.h" tduffy> +#include "dapl_ia_util.h" tduffy> +#include "dapl_openib_util.h" tduffy> + tduffy> +/* tduffy> + * dapl_psp_create_any tduffy> + * tduffy> + * Create a persistent Public Service Point that can recieve multiple tduffy> + * requests for connections and generate multiple connection request tduffy> + * instances that wil be delivered to the specified Event Dispatcher tduffy> + * in a notification event. Differs from dapl_psp_create() in that tduffy> + * the conn_qual is selected by the implementation and returned to tduffy> + * the user. tduffy> + * tduffy> + * Input: tduffy> + * ia_handle tduffy> + * evd_handle tduffy> + * psp_flags tduffy> + * tduffy> + * Output: tduffy> + * conn_qual tduffy> + * psp_handle tduffy> + * tduffy> + * Returns: tduffy> + * DAT_SUCCESS tduffy> + * DAT_INSUFFICIENT_RESOURCES tduffy> + * DAT_INVALID_HANDLE tduffy> + * DAT_INVALID_PARAMETER tduffy> + * DAT_CONN_QUAL_IN_USE tduffy> + * DAT_MODEL_NOT_SUPPORTED tduffy> + */ tduffy> +u32 dapl_psp_create_any(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL *conn_qual, tduffy> + DAT_EVD_HANDLE evd_handle, enum dat_psp_flags psp_flags, tduffy> + DAT_PSP_HANDLE *psp_handle) tduffy> +{ tduffy> + static DAT_CONN_QUAL hint_conn_qual = 1000; /* seed value */ tduffy> + struct dapl_ia *ia_ptr; tduffy> + struct dapl_sp *sp_ptr; tduffy> + struct dapl_evd *evd_ptr; tduffy> + u32 status = DAT_SUCCESS; tduffy> + int i; tduffy> + tduffy> + ia_ptr = (struct dapl_ia *)ia_handle; tduffy> + tduffy> + if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { tduffy> + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); tduffy> + goto bail; tduffy> + } tduffy> + if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { tduffy> + status = tduffy> + DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + if (psp_handle == NULL) { tduffy> + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); tduffy> + goto bail; tduffy> + } tduffy> + if (conn_qual == NULL) { tduffy> + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + evd_ptr = (struct dapl_evd *)evd_handle; tduffy> + if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { tduffy> + status = DAT_ERROR(DAT_INVALID_HANDLE, tduffy> + DAT_INVALID_HANDLE_EVD_CR); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + if (psp_flags != DAT_PSP_CONSUMER_FLAG && tduffy> + psp_flags != DAT_PSP_PROVIDER_FLAG) { tduffy> + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + /* Allocate PSP */ tduffy> + sp_ptr = dapl_sp_alloc(ia_ptr, TRUE); tduffy> + if (sp_ptr == NULL) { tduffy> + status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, tduffy> + DAT_RESOURCE_MEMORY); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + /* tduffy> + * Fill out the args for a PSP tduffy> + */ tduffy> + sp_ptr->ia_handle = ia_handle; tduffy> + sp_ptr->evd_handle = evd_handle; tduffy> + sp_ptr->psp_flags = psp_flags; tduffy> + sp_ptr->ep_handle = NULL; tduffy> + tduffy> + /* tduffy> + * Take a reference on the EVD handle tduffy> + */ tduffy> + atomic_inc(&evd_ptr->evd_ref_count); tduffy> + tduffy> + /* Link it onto the IA */ tduffy> + dapl_ia_link_psp(ia_ptr, sp_ptr); tduffy> + tduffy> + /* tduffy> + * Set up a listener for a connection. Connections can arrive tduffy> + * even before this call returns! tduffy> + */ tduffy> + sp_ptr->state = DAPL_SP_STATE_PSP_LISTENING; tduffy> + sp_ptr->listening = TRUE; tduffy> + tduffy> + /* tduffy> + * If we have a big number of tries and we still haven't tduffy> + * found a service_ID we can use, bail out with an error, tduffy> + * something is wrong! tduffy> + */ tduffy> + for (i = 0, sp_ptr->conn_qual = hint_conn_qual; i < 100000; i++) { tduffy> + status = dapl_ib_setup_conn_listener(ia_ptr, sp_ptr); tduffy> + if (DAT_SUCCESS == status) tduffy> + break; tduffy> + else tduffy> + sp_ptr->conn_qual++; tduffy> + } tduffy> + hint_conn_qual = sp_ptr->conn_qual + 1; tduffy> + tduffy> + if (status != DAT_SUCCESS) { tduffy> + atomic_dec(&evd_ptr->evd_ref_count); tduffy> + sp_ptr->evd_handle = NULL; tduffy> + dapl_ia_unlink_sp(ia_ptr, sp_ptr); tduffy> + dapl_sp_dealloc(sp_ptr); tduffy> + tduffy> + dapl_dbg_log(DAPL_DBG_TYPE_ERR, tduffy> + "dapl_psp_create cannot set up conn listener: %x\n", tduffy> + status); tduffy> + tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + /* tduffy> + * Return handle to the user tduffy> + */ tduffy> + *conn_qual = sp_ptr->conn_qual; tduffy> + *psp_handle = (DAT_PSP_HANDLE) sp_ptr; tduffy> + tduffy> +bail: tduffy> + return status; tduffy> +} tduffy> + tduffy> +/* tduffy> + * dapl_psp_create tduffy> + * tduffy> + * Create a persistent Public Service Point that can recieve multiple tduffy> + * requests for connections and generate multiple connection request tduffy> + * instances that wil be delivered to the specified Event Dispatcher tduffy> + * in a notification event. tduffy> + * tduffy> + * Input: tduffy> + * ia_handle tduffy> + * conn_qual tduffy> + * evd_handle tduffy> + * psp_flags tduffy> + * tduffy> + * Output: tduffy> + * psp_handle tduffy> + * tduffy> + * Returns: tduffy> + * DAT_SUCCESS tduffy> + * DAT_INSUFFICIENT_RESOURCES tduffy> + * DAT_INVALID_PARAMETER tduffy> + * DAT_CONN_QUAL_IN_USE tduffy> + * DAT_MODEL_NOT_SUPPORTED tduffy> + */ tduffy> +u32 dapl_psp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, tduffy> + DAT_EVD_HANDLE evd_handle, enum dat_psp_flags psp_flags, tduffy> + DAT_PSP_HANDLE *psp_handle) tduffy> +{ tduffy> + struct dapl_ia *ia_ptr; tduffy> + struct dapl_sp *sp_ptr; tduffy> + struct dapl_evd *evd_ptr; tduffy> + boolean_t sp_found; tduffy> + u32 status = DAT_SUCCESS; tduffy> + tduffy> + ia_ptr = (struct dapl_ia *)ia_handle; tduffy> + tduffy> + if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { tduffy> + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); tduffy> + goto bail; tduffy> + } tduffy> + if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { tduffy> + status = DAT_ERROR(DAT_INVALID_HANDLE, tduffy> + DAT_INVALID_HANDLE_EVD_CR); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + if (psp_handle == NULL) { tduffy> + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + evd_ptr = (struct dapl_evd *)evd_handle; tduffy> + if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { tduffy> + status = DAT_ERROR(DAT_INVALID_HANDLE, tduffy> + DAT_INVALID_HANDLE_EVD_CR); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + if (psp_flags != DAT_PSP_CONSUMER_FLAG && tduffy> + psp_flags != DAT_PSP_PROVIDER_FLAG) { tduffy> + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + /* tduffy> + * See if we have a quiescent listener to use for this PSP, else tduffy> + * create one and set it listening tduffy> + */ tduffy> + sp_ptr = dapl_ia_sp_search(ia_ptr, conn_qual, TRUE); tduffy> + sp_found = TRUE; tduffy> + if (sp_ptr == NULL) { tduffy> + /* Allocate PSP */ tduffy> + sp_found = FALSE; tduffy> + sp_ptr = dapl_sp_alloc(ia_ptr, TRUE); tduffy> + if (sp_ptr == NULL) { tduffy> + status = DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, tduffy> + DAT_RESOURCE_MEMORY); tduffy> + goto bail; tduffy> + } tduffy> + } else if (sp_ptr->listening == TRUE) { tduffy> + status = DAT_ERROR(DAT_CONN_QUAL_IN_USE, 0); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + /* tduffy> + * Fill out the args for a PSP tduffy> + */ tduffy> + sp_ptr->ia_handle = ia_handle; tduffy> + sp_ptr->conn_qual = conn_qual; tduffy> + sp_ptr->evd_handle = evd_handle; tduffy> + sp_ptr->psp_flags = psp_flags; tduffy> + sp_ptr->ep_handle = NULL; tduffy> + tduffy> + /* tduffy> + * Take a reference on the EVD handle tduffy> + */ tduffy> + atomic_inc(&evd_ptr->evd_ref_count); tduffy> + tduffy> + /* tduffy> + * Set up a listener for a connection. Connections can arrive tduffy> + * even before this call returns! tduffy> + */ tduffy> + sp_ptr->state = DAPL_SP_STATE_PSP_LISTENING; tduffy> + sp_ptr->listening = TRUE; tduffy> + tduffy> + /* tduffy> + * If this is a new sp we need to add it to the IA queue, and set up tduffy> + * a conn_listener. tduffy> + */ tduffy> + if (sp_found == FALSE) { tduffy> + /* tduffy> + * Link it onto the IA before enabling it to receive conn tduffy> + * requests tduffy> + */ tduffy> + dapl_ia_link_psp(ia_ptr, sp_ptr); tduffy> + tduffy> + status = dapl_ib_setup_conn_listener(ia_ptr, sp_ptr); tduffy> + tduffy> + if (status != DAT_SUCCESS) { tduffy> + /* tduffy> + * Have a problem setting up the connection, something tduffy> + * wrong! Decrements the EVD refcount & release it. tduffy> + */ tduffy> + atomic_dec(&evd_ptr->evd_ref_count); tduffy> + sp_ptr->evd_handle = NULL; tduffy> + dapl_ia_unlink_sp(ia_ptr, sp_ptr); tduffy> + dapl_sp_dealloc(sp_ptr); tduffy> + tduffy> + dapl_dbg_log(DAPL_DBG_TYPE_CM, "dapl_psp_create " tduffy> + "setup_conn_listener failed: %x\n", tduffy> + status); tduffy> + tduffy> + goto bail; tduffy> + } tduffy> + } tduffy> + tduffy> + /* tduffy> + * Return handle to the user tduffy> + */ tduffy> + *psp_handle = (DAT_PSP_HANDLE) sp_ptr; tduffy> + tduffy> +bail: tduffy> + return status; tduffy> +} tduffy> + tduffy> +/* tduffy> + * dapl_psp_free tduffy> + * tduffy> + * Destroy a specific instance of a Service Point. tduffy> + * tduffy> + * Input: tduffy> + * psp_handle tduffy> + * tduffy> + * Output: tduffy> + * none tduffy> + * tduffy> + * Returns: tduffy> + * DAT_SUCCESS tduffy> + * DAT_INVALID_PARAMETER tduffy> + */ tduffy> +u32 dapl_psp_free(DAT_PSP_HANDLE psp_handle) tduffy> +{ tduffy> + struct dapl_ia *ia_ptr; tduffy> + struct dapl_sp *sp_ptr; tduffy> + DAPL_SP_STATE save_state; tduffy> + u32 status = DAT_SUCCESS; tduffy> + tduffy> + sp_ptr = (struct dapl_sp *)psp_handle; tduffy> + /* tduffy> + * Verify handle tduffy> + */ tduffy> + dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_psp_free %p\n", psp_handle); tduffy> + tduffy> + if (DAPL_BAD_HANDLE(sp_ptr, DAPL_MAGIC_PSP)) { tduffy> + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PSP); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + /* ia_ptr = (struct dapl_ia *)sp_ptr->header.owner_ia; */ tduffy> + ia_ptr = sp_ptr->header.owner_ia; tduffy> + /* tduffy> + * Remove the connection listener if it has been established tduffy> + * and there are no current connections in progress. tduffy> + * If we defer removing the sp it becomes something of a zombie tduffy> + * container until the last connection is disconnected, after tduffy> + * which it will be cleaned up. tduffy> + */ tduffy> + spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); tduffy> + tduffy> + sp_ptr->listening = FALSE; tduffy> + tduffy> + /* tduffy> + * Release reference on EVD. If an error was encountered in a previous tduffy> + * free the evd_handle will be NULL tduffy> + */ tduffy> + if (sp_ptr->evd_handle) { tduffy> + atomic_dec(&((struct dapl_evd *)sp_ptr->evd_handle)-> tduffy> + evd_ref_count); tduffy> + sp_ptr->evd_handle = NULL; tduffy> + } tduffy> + tduffy> + /* tduffy> + * Release the base resource if there are no outstanding tduffy> + * connections; else the last disconnect on this PSP will free it tduffy> + * up. The PSP is used to contain CR records for each connection, tduffy> + * which contain information necessary to disconnect. tduffy> + */ tduffy> + dapl_dbg_log(DAPL_DBG_TYPE_CM, tduffy> + ">>> dapl_psp_free: state %d cr_list_count %d\n", tduffy> + sp_ptr->state, sp_ptr->cr_list_count); tduffy> + if ((sp_ptr->state == DAPL_SP_STATE_PSP_LISTENING || tduffy> + sp_ptr->state == DAPL_SP_STATE_PSP_PENDING) && tduffy> + sp_ptr->cr_list_count == 0) { tduffy> + save_state = sp_ptr->state; tduffy> + sp_ptr->state = DAPL_SP_STATE_FREE; tduffy> + spin_unlock_irqrestore(&sp_ptr->header.lock, tduffy> + sp_ptr->header.flags); tduffy> + tduffy> + status = dapl_ib_remove_conn_listener(ia_ptr, sp_ptr); tduffy> + if (status != DAT_SUCCESS) { tduffy> + /* revert to entry state on error */ tduffy> + sp_ptr->state = save_state; tduffy> + goto bail; tduffy> + } tduffy> + dapl_ia_unlink_sp(ia_ptr, sp_ptr); tduffy> + dapl_sp_dealloc(sp_ptr); tduffy> + } else { tduffy> + /* tduffy> + * The PSP is now in the pending state, where it will sit until tduffy> + * the last connection terminates or the app uses the same tduffy> + * ServiceID again, which will reactivate it. tduffy> + */ tduffy> + sp_ptr->state = DAPL_SP_STATE_PSP_PENDING; tduffy> + spin_unlock_irqrestore(&sp_ptr->header.lock, sp_ptr->header.flags); tduffy> + dapl_dbg_log(DAPL_DBG_TYPE_CM, tduffy> + ">>> dapl_psp_free: PSP PENDING\n"); tduffy> + } tduffy> + tduffy> +bail: tduffy> + return status; tduffy> +} tduffy> + tduffy> +u32 dapl_psp_query(DAT_PSP_HANDLE psp_handle, struct dat_psp_param *psp_param) tduffy> +{ tduffy> + struct dapl_sp *sp_ptr; tduffy> + u32 status; tduffy> + tduffy> + if (DAPL_BAD_HANDLE(psp_handle, DAPL_MAGIC_PSP) || tduffy> + ((struct dapl_sp *)psp_handle)->listening != TRUE) { tduffy> + status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PSP); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + if (NULL == psp_param) { tduffy> + status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); tduffy> + goto bail; tduffy> + } tduffy> + tduffy> + sp_ptr = (struct dapl_sp *)psp_handle; tduffy> + tduffy> + psp_param->ia_handle = sp_ptr->ia_handle; tduffy> + psp_param->conn_qual = sp_ptr->conn_qual; tduffy> + psp_param->evd_handle = sp_ptr->evd_handle; tduffy> + psp_param->psp_flags = sp_ptr->psp_flags; tduffy> + tduffy> + status = DAT_SUCCESS; tduffy> + tduffy> +bail: tduffy> + return status; tduffy> +} tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel-rsp/dat-provider/dapl_psp_create_any.c linux-kernel-psp/dat-provider/dapl_psp_create_any.c tduffy> --- linux-kernel-rsp/dat-provider/dapl_psp_create_any.c 2005-05-23 22:21:46.111018000 -0700 tduffy> +++ linux-kernel-psp/dat-provider/dapl_psp_create_any.c 1969-12-31 16:00:00.000000000 -0800 tduffy> @@ -1,177 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/* tduffy> - * $Id: dapl_psp_create_any.c 2462 2005-05-24 02:28:24Z jlentini $ tduffy> - */ tduffy> - tduffy> -#include "dapl.h" tduffy> -#include "dapl_sp_util.h" tduffy> -#include "dapl_ia_util.h" tduffy> -#include "dapl_openib_util.h" tduffy> - tduffy> -/* tduffy> - * dapl_psp_create_any tduffy> - * tduffy> - * Create a persistent Public Service Point that can recieve multiple tduffy> - * requests for connections and generate multiple connection request tduffy> - * instances that wil be delivered to the specified Event Dispatcher tduffy> - * in a notification event. Differs from dapl_psp_create() in that tduffy> - * the conn_qual is selected by the implementation and returned to tduffy> - * the user. tduffy> - * tduffy> - * Input: tduffy> - * ia_handle tduffy> - * evd_handle tduffy> - * psp_flags tduffy> - * tduffy> - * Output: tduffy> - * conn_qual tduffy> - * psp_handle tduffy> - * tduffy> - * Returns: tduffy> - * DAT_SUCCESS tduffy> - * DAT_INSUFFICIENT_RESOURCES tduffy> - * DAT_INVALID_HANDLE tduffy> - * DAT_INVALID_PARAMETER tduffy> - * DAT_CONN_QUAL_IN_USE tduffy> - * DAT_MODEL_NOT_SUPPORTED tduffy> - */ tduffy> -u32 dapl_psp_create_any(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL *conn_qual, tduffy> - DAT_EVD_HANDLE evd_handle, enum dat_psp_flags psp_flags, tduffy> - DAT_PSP_HANDLE *psp_handle) tduffy> -{ tduffy> - static DAT_CONN_QUAL hint_conn_qual = 1000; /* seed value */ tduffy> - struct dapl_ia *ia_ptr; tduffy> - struct dapl_sp *sp_ptr; tduffy> - struct dapl_evd *evd_ptr; tduffy> - u32 status = DAT_SUCCESS; tduffy> - int i; tduffy> - tduffy> - ia_ptr = (struct dapl_ia *)ia_handle; tduffy> - tduffy> - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { tduffy> - status = DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); tduffy> - goto bail; tduffy> - } tduffy> - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { tduffy> - status = tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - if (psp_handle == NULL) { tduffy> - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); tduffy> - goto bail; tduffy> - } tduffy> - if (conn_qual == NULL) { tduffy> - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG2); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - evd_ptr = (struct dapl_evd *)evd_handle; tduffy> - if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { tduffy> - status = tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - if (psp_flags != DAT_PSP_CONSUMER_FLAG && tduffy> - psp_flags != DAT_PSP_PROVIDER_FLAG) { tduffy> - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - /* Allocate PSP */ tduffy> - sp_ptr = dapl_sp_alloc(ia_ptr, TRUE); tduffy> - if (sp_ptr == NULL) { tduffy> - status = tduffy> - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, DAT_RESOURCE_MEMORY); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - /* tduffy> - * Fill out the args for a PSP tduffy> - */ tduffy> - sp_ptr->ia_handle = ia_handle; tduffy> - sp_ptr->evd_handle = evd_handle; tduffy> - sp_ptr->psp_flags = psp_flags; tduffy> - sp_ptr->ep_handle = NULL; tduffy> - tduffy> - /* tduffy> - * Take a reference on the EVD handle tduffy> - */ tduffy> - atomic_inc(&evd_ptr->evd_ref_count); tduffy> - tduffy> - /* Link it onto the IA */ tduffy> - dapl_ia_link_psp(ia_ptr, sp_ptr); tduffy> - tduffy> - /* tduffy> - * Set up a listener for a connection. Connections can arrive tduffy> - * even before this call returns! tduffy> - */ tduffy> - sp_ptr->state = DAPL_SP_STATE_PSP_LISTENING; tduffy> - sp_ptr->listening = TRUE; tduffy> - tduffy> - /* tduffy> - * If we have a big number of tries and we still haven't tduffy> - * found a service_ID we can use, bail out with an error, tduffy> - * something is wrong! tduffy> - */ tduffy> - for (i = 0, sp_ptr->conn_qual = hint_conn_qual; i < 100000; i++) { tduffy> - tduffy> - status = dapl_ib_setup_conn_listener(ia_ptr, sp_ptr); tduffy> - tduffy> - if (DAT_SUCCESS == status) tduffy> - break; tduffy> - else tduffy> - sp_ptr->conn_qual++; tduffy> - } tduffy> - hint_conn_qual = sp_ptr->conn_qual + 1; tduffy> - tduffy> - if (status != DAT_SUCCESS) { tduffy> - atomic_dec(&evd_ptr->evd_ref_count); tduffy> - sp_ptr->evd_handle = NULL; tduffy> - dapl_ia_unlink_sp(ia_ptr, sp_ptr); tduffy> - dapl_sp_dealloc(sp_ptr); tduffy> - tduffy> - dapl_dbg_log(DAPL_DBG_TYPE_ERR, tduffy> - "dapl_psp_create cannot set up conn listener: %x\n", tduffy> - status); tduffy> - tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - /* tduffy> - * Return handle to the user tduffy> - */ tduffy> - *conn_qual = sp_ptr->conn_qual; tduffy> - *psp_handle = (DAT_PSP_HANDLE) sp_ptr; tduffy> - tduffy> - bail: tduffy> - return status; tduffy> -} tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel-rsp/dat-provider/dapl_psp_create.c linux-kernel-psp/dat-provider/dapl_psp_create.c tduffy> --- linux-kernel-rsp/dat-provider/dapl_psp_create.c 2005-05-23 22:21:46.086003000 -0700 tduffy> +++ linux-kernel-psp/dat-provider/dapl_psp_create.c 1969-12-31 16:00:00.000000000 -0800 tduffy> @@ -1,181 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/* tduffy> - * $Id: dapl_psp_create.c 2462 2005-05-24 02:28:24Z jlentini $ tduffy> - */ tduffy> - tduffy> -#include "dapl.h" tduffy> -#include "dapl_sp_util.h" tduffy> -#include "dapl_ia_util.h" tduffy> -#include "dapl_openib_util.h" tduffy> - tduffy> -/* tduffy> - * dapl_psp_create tduffy> - * tduffy> - * Create a persistent Public Service Point that can recieve multiple tduffy> - * requests for connections and generate multiple connection request tduffy> - * instances that wil be delivered to the specified Event Dispatcher tduffy> - * in a notification event. tduffy> - * tduffy> - * Input: tduffy> - * ia_handle tduffy> - * conn_qual tduffy> - * evd_handle tduffy> - * psp_flags tduffy> - * tduffy> - * Output: tduffy> - * psp_handle tduffy> - * tduffy> - * Returns: tduffy> - * DAT_SUCCESS tduffy> - * DAT_INSUFFICIENT_RESOURCES tduffy> - * DAT_INVALID_PARAMETER tduffy> - * DAT_CONN_QUAL_IN_USE tduffy> - * DAT_MODEL_NOT_SUPPORTED tduffy> - */ tduffy> -u32 dapl_psp_create(DAT_IA_HANDLE ia_handle, DAT_CONN_QUAL conn_qual, tduffy> - DAT_EVD_HANDLE evd_handle, enum dat_psp_flags psp_flags, tduffy> - DAT_PSP_HANDLE *psp_handle) tduffy> -{ tduffy> - struct dapl_ia *ia_ptr; tduffy> - struct dapl_sp *sp_ptr; tduffy> - struct dapl_evd *evd_ptr; tduffy> - boolean_t sp_found; tduffy> - u32 dat_status = DAT_SUCCESS; tduffy> - tduffy> - ia_ptr = (struct dapl_ia *)ia_handle; tduffy> - tduffy> - if (DAPL_BAD_HANDLE(ia_ptr, DAPL_MAGIC_IA)) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_IA); tduffy> - goto bail; tduffy> - } tduffy> - if (DAPL_BAD_HANDLE(evd_handle, DAPL_MAGIC_EVD)) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - if (psp_handle == NULL) { tduffy> - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG5); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - evd_ptr = (struct dapl_evd *)evd_handle; tduffy> - if (!(evd_ptr->evd_flags & DAT_EVD_CR_FLAG)) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_EVD_CR); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - if (psp_flags != DAT_PSP_CONSUMER_FLAG && tduffy> - psp_flags != DAT_PSP_PROVIDER_FLAG) { tduffy> - dat_status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG4); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - /* tduffy> - * See if we have a quiescent listener to use for this PSP, else tduffy> - * create one and set it listening tduffy> - */ tduffy> - sp_ptr = dapl_ia_sp_search(ia_ptr, conn_qual, TRUE); tduffy> - sp_found = TRUE; tduffy> - if (sp_ptr == NULL) { tduffy> - /* Allocate PSP */ tduffy> - sp_found = FALSE; tduffy> - sp_ptr = dapl_sp_alloc(ia_ptr, TRUE); tduffy> - if (sp_ptr == NULL) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INSUFFICIENT_RESOURCES, tduffy> - DAT_RESOURCE_MEMORY); tduffy> - goto bail; tduffy> - } tduffy> - } else if (sp_ptr->listening == TRUE) { tduffy> - dat_status = DAT_ERROR(DAT_CONN_QUAL_IN_USE, 0); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - /* tduffy> - * Fill out the args for a PSP tduffy> - */ tduffy> - sp_ptr->ia_handle = ia_handle; tduffy> - sp_ptr->conn_qual = conn_qual; tduffy> - sp_ptr->evd_handle = evd_handle; tduffy> - sp_ptr->psp_flags = psp_flags; tduffy> - sp_ptr->ep_handle = NULL; tduffy> - tduffy> - /* tduffy> - * Take a reference on the EVD handle tduffy> - */ tduffy> - atomic_inc(&evd_ptr->evd_ref_count); tduffy> - tduffy> - /* tduffy> - * Set up a listener for a connection. Connections can arrive tduffy> - * even before this call returns! tduffy> - */ tduffy> - sp_ptr->state = DAPL_SP_STATE_PSP_LISTENING; tduffy> - sp_ptr->listening = TRUE; tduffy> - tduffy> - /* tduffy> - * If this is a new sp we need to add it to the IA queue, and set up tduffy> - * a conn_listener. tduffy> - */ tduffy> - if (sp_found == FALSE) { tduffy> - /* Link it onto the IA before enabling it to receive conn tduffy> - * requests tduffy> - */ tduffy> - dapl_ia_link_psp(ia_ptr, sp_ptr); tduffy> - tduffy> - dat_status = dapl_ib_setup_conn_listener(ia_ptr, sp_ptr); tduffy> - tduffy> - if (dat_status != DAT_SUCCESS) { tduffy> - /* tduffy> - * Have a problem setting up the connection, something tduffy> - * wrong! Decrements the EVD refcount & release it. tduffy> - */ tduffy> - atomic_dec(&evd_ptr->evd_ref_count); tduffy> - sp_ptr->evd_handle = NULL; tduffy> - dapl_ia_unlink_sp(ia_ptr, sp_ptr); tduffy> - dapl_sp_dealloc(sp_ptr); tduffy> - tduffy> - dapl_dbg_log(DAPL_DBG_TYPE_CM, "dapl_psp_create " tduffy> - "setup_conn_listener failed: %x\n", tduffy> - dat_status); tduffy> - tduffy> - goto bail; tduffy> - } tduffy> - } tduffy> - tduffy> - /* tduffy> - * Return handle to the user tduffy> - */ tduffy> - *psp_handle = (DAT_PSP_HANDLE) sp_ptr; tduffy> - tduffy> - bail: tduffy> - return dat_status; tduffy> -} tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel-rsp/dat-provider/dapl_psp_free.c linux-kernel-psp/dat-provider/dapl_psp_free.c tduffy> --- linux-kernel-rsp/dat-provider/dapl_psp_free.c 2005-05-20 22:55:17.285001000 -0700 tduffy> +++ linux-kernel-psp/dat-provider/dapl_psp_free.c 1969-12-31 16:00:00.000000000 -0800 tduffy> @@ -1,131 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/* tduffy> - * $Id: dapl_psp_free.c 2433 2005-05-21 04:11:03Z jlentini $ tduffy> - */ tduffy> - tduffy> -#include "dapl.h" tduffy> -#include "dapl_sp_util.h" tduffy> -#include "dapl_ia_util.h" tduffy> -#include "dapl_openib_util.h" tduffy> - tduffy> -/* tduffy> - * dapl_psp_free tduffy> - * tduffy> - * Destroy a specific instance of a Service Point. tduffy> - * tduffy> - * Input: tduffy> - * psp_handle tduffy> - * tduffy> - * Output: tduffy> - * none tduffy> - * tduffy> - * Returns: tduffy> - * DAT_SUCCESS tduffy> - * DAT_INVALID_PARAMETER tduffy> - */ tduffy> -u32 dapl_psp_free(DAT_PSP_HANDLE psp_handle) tduffy> -{ tduffy> - struct dapl_ia *ia_ptr; tduffy> - struct dapl_sp *sp_ptr; tduffy> - DAPL_SP_STATE save_state; tduffy> - u32 dat_status = DAT_SUCCESS; tduffy> - tduffy> - sp_ptr = (struct dapl_sp *)psp_handle; tduffy> - /* tduffy> - * Verify handle tduffy> - */ tduffy> - dapl_dbg_log(DAPL_DBG_TYPE_CM, ">>> dapl_psp_free %p\n", psp_handle); tduffy> - tduffy> - if (DAPL_BAD_HANDLE(sp_ptr, DAPL_MAGIC_PSP)) { tduffy> - dat_status = tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PSP); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - /* ia_ptr = (struct dapl_ia *)sp_ptr->header.owner_ia; */ tduffy> - ia_ptr = sp_ptr->header.owner_ia; tduffy> - /* tduffy> - * Remove the connection listener if it has been established tduffy> - * and there are no current connections in progress. tduffy> - * If we defer removing the sp it becomes something of a zombie tduffy> - * container until the last connection is disconnected, after tduffy> - * which it will be cleaned up. tduffy> - */ tduffy> - spin_lock_irqsave(&sp_ptr->header.lock, sp_ptr->header.flags); tduffy> - tduffy> - sp_ptr->listening = FALSE; tduffy> - tduffy> - /* Release reference on EVD. If an error was encountered in a previous tduffy> - * free the evd_handle will be NULL tduffy> - */ tduffy> - if (sp_ptr->evd_handle) { tduffy> - atomic_dec(&((struct dapl_evd *)sp_ptr->evd_handle)-> tduffy> - evd_ref_count); tduffy> - sp_ptr->evd_handle = NULL; tduffy> - } tduffy> - tduffy> - /* tduffy> - * Release the base resource if there are no outstanding tduffy> - * connections; else the last disconnect on this PSP will free it tduffy> - * up. The PSP is used to contain CR records for each connection, tduffy> - * which contain information necessary to disconnect. tduffy> - */ tduffy> - dapl_dbg_log(DAPL_DBG_TYPE_CM, tduffy> - ">>> dapl_psp_free: state %d cr_list_count %d\n", tduffy> - sp_ptr->state, sp_ptr->cr_list_count); tduffy> - if ((sp_ptr->state == DAPL_SP_STATE_PSP_LISTENING || tduffy> - sp_ptr->state == DAPL_SP_STATE_PSP_PENDING) && tduffy> - sp_ptr->cr_list_count == 0) { tduffy> - save_state = sp_ptr->state; tduffy> - sp_ptr->state = DAPL_SP_STATE_FREE; tduffy> - spin_unlock_irqrestore(&sp_ptr->header.lock, tduffy> - sp_ptr->header.flags); tduffy> - tduffy> - dat_status = dapl_ib_remove_conn_listener(ia_ptr, sp_ptr); tduffy> - if (dat_status != DAT_SUCCESS) { tduffy> - /* revert to entry state on error */ tduffy> - sp_ptr->state = save_state; tduffy> - goto bail; tduffy> - } tduffy> - dapl_ia_unlink_sp(ia_ptr, sp_ptr); tduffy> - dapl_sp_dealloc(sp_ptr); tduffy> - } else { tduffy> - /* The PSP is now in the pending state, where it will sit until tduffy> - * the last connection terminates or the app uses the same tduffy> - * ServiceID again, which will reactivate it. tduffy> - */ tduffy> - sp_ptr->state = DAPL_SP_STATE_PSP_PENDING; tduffy> - spin_unlock_irqrestore(&sp_ptr->header.lock, sp_ptr->header.flags); tduffy> - dapl_dbg_log(DAPL_DBG_TYPE_CM, tduffy> - ">>> dapl_psp_free: PSP PENDING\n"); tduffy> - } tduffy> - tduffy> - bail: tduffy> - return dat_status; tduffy> -} tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel-rsp/dat-provider/dapl_psp_query.c linux-kernel-psp/dat-provider/dapl_psp_query.c tduffy> --- linux-kernel-rsp/dat-provider/dapl_psp_query.c 2005-05-20 22:55:18.853022000 -0700 tduffy> +++ linux-kernel-psp/dat-provider/dapl_psp_query.c 1969-12-31 16:00:00.000000000 -0800 tduffy> @@ -1,62 +0,0 @@ tduffy> -/* tduffy> - * Copyright (c) 2002-2005, Network Appliance, Inc. All rights reserved. tduffy> - * tduffy> - * This Software is licensed under one of the following licenses: tduffy> - * tduffy> - * 1) under the terms of the "Common Public License 1.0" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/cpl.php. tduffy> - * tduffy> - * 2) under the terms of the "The BSD License" a copy of which is tduffy> - * available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/bsd-license.php. tduffy> - * tduffy> - * 3) under the terms of the "GNU General Public License (GPL) Version 2" a tduffy> - * copy of which is available from the Open Source Initiative, see tduffy> - * http://www.opensource.org/licenses/gpl-license.php. tduffy> - * tduffy> - * Licensee has the right to choose one of the above licenses. tduffy> - * tduffy> - * Redistributions of source code must retain the above copyright tduffy> - * notice and one of the license notices. tduffy> - * tduffy> - * Redistributions in binary form must reproduce both the above copyright tduffy> - * notice, one of the license notices in the documentation tduffy> - * and/or other materials provided with the distribution. tduffy> - */ tduffy> - tduffy> -/* tduffy> - * $Id: dapl_psp_query.c 2433 2005-05-21 04:11:03Z jlentini $ tduffy> - */ tduffy> - tduffy> -#include "dapl.h" tduffy> - tduffy> -u32 dapl_psp_query(DAT_PSP_HANDLE psp_handle, struct dat_psp_param *psp_param) tduffy> -{ tduffy> - struct dapl_sp *sp_ptr; tduffy> - u32 status; tduffy> - tduffy> - if (DAPL_BAD_HANDLE(psp_handle, DAPL_MAGIC_PSP) || tduffy> - ((struct dapl_sp *)psp_handle)->listening != TRUE) { tduffy> - status = tduffy> - DAT_ERROR(DAT_INVALID_HANDLE, DAT_INVALID_HANDLE_PSP); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - if (NULL == psp_param) { tduffy> - status = DAT_ERROR(DAT_INVALID_PARAMETER, DAT_INVALID_ARG3); tduffy> - goto bail; tduffy> - } tduffy> - tduffy> - sp_ptr = (struct dapl_sp *)psp_handle; tduffy> - tduffy> - psp_param->ia_handle = sp_ptr->ia_handle; tduffy> - psp_param->conn_qual = sp_ptr->conn_qual; tduffy> - psp_param->evd_handle = sp_ptr->evd_handle; tduffy> - psp_param->psp_flags = sp_ptr->psp_flags; tduffy> - tduffy> - status = DAT_SUCCESS; tduffy> - tduffy> -bail: tduffy> - return status; tduffy> -} tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel-rsp/dat-provider/Makefile linux-kernel-psp/dat-provider/Makefile tduffy> --- linux-kernel-rsp/dat-provider/Makefile 2005-05-25 15:01:00.683014000 -0700 tduffy> +++ linux-kernel-psp/dat-provider/Makefile 2005-05-25 15:12:34.054000000 -0700 tduffy> @@ -70,10 +70,7 @@ PROVIDER_MODULES := \ tduffy> dapl_mr_util \ tduffy> dapl_provider \ tduffy> dapl_sp_util \ tduffy> - dapl_psp_create \ tduffy> - dapl_psp_create_any \ tduffy> - dapl_psp_free \ tduffy> - dapl_psp_query \ tduffy> + dapl_psp \ tduffy> dapl_pz \ tduffy> dapl_ring_buffer_util \ tduffy> dapl_rmr \ tduffy> diff -Nurp -X /home/tduffy/dontdiff linux-kernel-rsp/patches/alt_dat_provider_makefile linux-kernel-psp/patches/alt_dat_provider_makefile tduffy> --- linux-kernel-rsp/patches/alt_dat_provider_makefile 2005-05-25 14:53:43.453015000 -0700 tduffy> +++ linux-kernel-psp/patches/alt_dat_provider_makefile 2005-05-25 15:05:08.065003000 -0700 tduffy> @@ -64,10 +64,7 @@ PROVIDER_MODULES := \ tduffy> dapl_mr_util \ tduffy> dapl_provider \ tduffy> dapl_sp_util \ tduffy> - dapl_psp_create \ tduffy> - dapl_psp_create_any \ tduffy> - dapl_psp_free \ tduffy> - dapl_psp_query \ tduffy> + dapl_psp \ tduffy> dapl_pz \ tduffy> dapl_ring_buffer_util \ tduffy> dapl_rmr \ tduffy> From sean.hefty at intel.com Fri May 27 08:44:26 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Fri, 27 May 2005 08:44:26 -0700 Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: <1117161668.19472.11.camel@duffman> Message-ID: >+/* >+ * approximately transforms miliseconds to 4.096us*2^x >+ * 63(+8) is max return I think that the max return is 64+8. >+ */ >+static inline u8 dapl_convert_ms_to_kookyib(unsigned long ms) { I like the function name. :) >+ unsigned long converged = 2; >+ u8 i; >+ >+ if (2 > ms) >+ return 8; >+ >+ for (i = 1; i < 64; i++) { >+ if (converged >= ms) >+ break; >+ converged = 2*converged; >+ } >+ >+ return i+8; >+} I didn't notice any other issues looking over the changes, but see my other email regarding setting the timeout based on the number of retries. - Sean From robert.j.woodruff at intel.com Fri May 27 09:04:57 2005 From: robert.j.woodruff at intel.com (Woodruff, Robert J) Date: Fri, 27 May 2005 09:04:57 -0700 Subject: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux Message-ID: <1AC79F16F5C5284499BB9591B33D6F0004831DCF@orsmsx408> Venkat wrote > >It is just unfortunate to make such an IB-centric statement. >I will ask the same question back to you - why didn't you consider >RNICs when the IB APIs were designed a year ago? I did. I expected that RNICs could plug into the DAPL layer, as this is what it was designed for. I did not see the need for an RDMA independent verbs layer, as this is what DAPL is. DAPL has already been proven to support multiple RDMA capable devices, InfiniBand, RNICS (Ammasso), Myrinet, and some others. I still think that having a verbs layer for IB and a separate one for RNICs is a viable option. At some level, they are different. We are not talking about a huge amount of code here, only a couple of thousand lines of code for the core verbs. The rest of the IB mid-layer, the CM, SA query, MAD services, are so IB specific, they do not apply to iWarp and iWarp would need to have their own version of say a CM. Thus, I think that the ib_core.ko, ib_cm.ko, ib_sa.ko, ib_mad.ko layer is the device class specific layer for InfiniBand. An RDMA device independent layer needs to be developed above that. It is also clear that from a kernel standpoint, kDAPL was not acceptable since it did not meet the Linux coding standards. Thus, a major overhaul is already underway to convert it into the RDMA device independent layer acceptable to Linux. >Again because we have developed something a year ago doesn't mean that >we will have to live with that interface forever. Of course, >the question is when and why should this change? As it was discussed >before, we would like to see the common PI developed without >impacting the IB development. This is true and it has already been stated on this list that we are open to changes (and the best way to effect change is to develop code) that allows RNICs to take advantage of common ULPs. I personally would suggest working with the kDAPL people to evolve that code into something that will work for you, since it is under major overhaul already. Roland has also stated that he is open to modification to the ib_core.ko verbs to support RNICS, so if that is the approach you want to take and it does not slow down the InfiniBand development, then, by all means, submit patches to modify ib_core. Sean expressed some concern that changing the ib_core to also support RNICs would increase complexity and double the testing effort as for every change, it would need to be re-tested on both IB and RNICs. woody From jlentini at netapp.com Fri May 27 09:22:12 2005 From: jlentini at netapp.com (James Lentini) Date: Fri, 27 May 2005 12:22:12 -0400 (EDT) Subject: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <1AC79F16F5C5284499BB9591B33D6F0004831DCF@orsmsx408> References: <1AC79F16F5C5284499BB9591B33D6F0004831DCF@orsmsx408> Message-ID: On Fri, 27 May 2005, Woodruff, Robert J wrote: > I personally would suggest working with the kDAPL people > to evolve that code into something that will work for you, since it > is under major overhaul already. I and the other kDAPL developers would welcome any input you have. From sean.hefty at intel.com Fri May 27 09:22:01 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Fri, 27 May 2005 09:22:01 -0700 Subject: [openib-general] OpenIB and OpenRDMA: Convergence on commonRDMA APIs and ULPs for Linux In-Reply-To: <52is16ogs7.fsf@topspin.com> Message-ID: >To put a really concrete proposal on the table, I would suggest to >start by extending the current ib_client registration structure, which >looks like > > struct ib_client { > char *name; > void (*add) (struct ib_device *); > void (*remove)(struct ib_device *); > > struct list_head list; > }; > >by extending the current enum ib_node_type to something like > > enum rdma_device_type { > RDMA_DEVICE_IB_CA, > RDMA_DEVICE_IB_SWITCH, > RDMA_DEVICE_IB_ROUTER, > RDMA_DEVICE_RNIC > }; > >Then the various pieces of code layered on top of the RDMA midlayer >can decide whether they want to deal with a particular device or not >by looking at the node_type member. For example, the IB CM, IPoIB, >etc. could ignore devices of type RDMA_DEVICE_RNIC, while SDP or iSER >would use all devices and the RNIC CM would take only devices of type >RDMA_DEVICE_RNIC. > >Then someone would have to start implementing a low-level driver for a >specific RNIC, and find which modifications to the existing verbs are >required. For example, I believe the QP attribute structure passed >into the QP modify verb probably has to become a union containing the >IB attributes and the RNIC attributes. However, most verbs should >work fine with at most trivial modifications. Is there some technical reason why this proposal cannot be used? I didn't notice any other proposal. - Sean From iod00d at hp.com Fri May 27 09:29:06 2005 From: iod00d at hp.com (Grant Grundler) Date: Fri, 27 May 2005 09:29:06 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux In-Reply-To: <6.2.0.14.2.20050527071821.02071338@esmail.cup.hp.com> References: <20050527134043.92998.qmail@web51901.mail.yahoo.com> <6.2.0.14.2.20050527071821.02071338@esmail.cup.hp.com> Message-ID: <20050527162906.GA7444@esmail.cup.hp.com> On Fri, May 27, 2005 at 07:24:44AM -0700, Michael Krause wrote: ... > Again, Sockets is an application API and not how one communicates to a TOE > or RDMA component. Mike, What address family is used to open a socket over iWARP? AF_INET? Or something else? I understand most of what you wrote but am still missing one bit: How is the RNIC told what the peer IP is it should communicate with? > The RNIC PI has been proposed as an interface to the > RDMA functionality. The PI supports all of the iWARP and IB v 1.2 verbs. That's good. Folks from RDMA consortium will have to look at openib implementations and see whats missing/wrong. Then submit proposals to fill in the gaps. I'm obviously not the first one to say this. I expect most of the principals involved with openib.org do NOT have time to browse through RNIC PI at this point. They are struggling to get openib.org filled in sufficiently so it can go into a commercial distro (RH/SuSE primarily). Revenue for them comes from selling IB equipment. Having openib.org code in kernel.org is a key enabler for getting into commercial distros. I expect the same is true for RNIC vendors as well. RNIC Vendors (and related switch Vendors) will have to decide which path is the right one for them to get the support into kernel.org. Several openib.org people have suggested one (like I have). RNIC folks need to listen and decide if the advice is good or not. If RNIC folks think they know better, then please take another look at where openib.org is today and where rdmaconsortium is. I'm certain openib.org would be dead now if policies and direction changes had not made last year as demanded by several key linux developers and "users" (Gov Labs). grant From caitlinb at siliquent.com Fri May 27 09:38:21 2005 From: caitlinb at siliquent.com (Caitlin Bestler) Date: Fri, 27 May 2005 09:38:21 -0700 Subject: [openib-general] iWARP IP Stack Dependency (Was Convergence on common RDMA APIs and ULPs for Linux) Message-ID: <8508251A6FC08A489844A94261D3693A05758C@fiona.siliquent.com> > -----Original Message----- > From: rdma-developers-admin at lists.sourceforge.net > [mailto:rdma-developers-admin at lists.sourceforge.net] On > Behalf Of Sukanta ganguly > Sent: Friday, May 27, 2005 6:41 AM > To: Venkata Jagana; Grant Grundler > Cc: openib-general at openib.org; rdma-developers at lists.sourceforge.net > Subject: Re: [Rdma-developers] Re: [openib-general] OpenIB > and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux > > Venkata, > How will that work? If the RNIC offloads RDMA and TCP > completely from the Operating System and does not share any > state information then the application running on the host > will never be in the position to utilize the socket interface > to use the communication logic to send and receive data > between the remote node and itself. Some information needs to > be shared. How much of it and what exactly needs to be shared > is the question. > > Thanks > SG iWARP/TCP requires that when a QP is modified to the RTR or RTS state that it be supplied an existing TCP connection. At that time the TCP Connection will be modified so that it is processed in MPA Mode, and so that it's payload is delivered to and generated by the QP rather than by its current IP stack. The degree that the Connection is "totally" handed over varies by implementation. But in no case can the consumer rely upon continued use of the TCP connection's socket to send or receive data. In fact generally there is nothing that can be done with the old socket after that. One thing that is *not* changed is the routing of the TCP connection. Therefore an RDMA device can only "convert" a TCP connection that is already routed through it. That does not mean that the RDMA Device (conventionally RNIC, almost entirely equivalent of an IB HCA) has to have a companion TOE service. It may have a companion NIC service instead, or also, that acts as a plain Ethernet device for TCP connections that have not been converted for RDMA processing. So the existing TCP connection must be transferred from its current IP stack to what is essentially an RDMA stack. RNIC-PI has specified tha the RDMA stack itself MUST NOT establish connections. That must be done by the Host stack, or by a TOE stack that is already integrated with and accepted by the Host stack. The RDMA Device MUST NOT circumvent the host stack's ability to control acceptance of TCP connections. That is one area where commonality between IB and iWARP stacks would be valuable. The Host IP stack wouldn't want the IPoIB stack creating unapproved connections behind its back either. In nuts-and-bolts terms, this means that modify-qp-to-rts accepts a socket handle. That socket handle must be for a TCP connection that is already established and is already routed through a physial port that the RNIC controls. It might be a handle for a TCP connection fully under the control of the host stack, or it may be a handle for a TCP connection that is managed by a TOE. But the domain of the handle is the host OS IP stack. You cannot reference a TCP connection that is unkown to the IP stack; it is a handle that is being passed in not a complete TCP Connection Control Block. This is one of many examples where defining a low-level API that is truly transport neutral is challenging. There are many places where things are truly common (Protection Domain IDs, the fact that a QP references send and receive CQs, setting send queue and receive queue sizes, SRQs, CQs, etc.). But then there is a substantial slice where QP States, QP Attributes, RNIC Attributes, etc. do not align and you simply have to have a union to split between the transport specific fields (and label certain enum values as being transport specific). > > --- Venkata Jagana wrote: > > > > > > > > > > > > > > rdma-developers-admin at lists.sourceforge.net wrote on > > 05/25/2005 09:47:00 > > PM: > > > > > Venkata, > > > Interesting coincidence: I was talking with > > someone (at HP) today > > > who knows substantially more than I do about > > RNICs. > > > They indicated RNICs need to manage TCP state on > > the card from userspace. > > > I suspect that's only possible through a private > > interface > > > (e.g. ioctl() or /proc) or the non-existant (in > > kernel.org) > > > TOE implementation. Is this correct? > > > > > > > Not correct. > > > > Since RNICs are offloaded adapters with RDMA protocols > layered on top > > of TCP stack, they do maintain the TCP state internally but it does > > not expose to the host. RNIC expose only RNIC Verbs > interface to the > > host bot not TOE interface. > > > > Thanks > > Venkat > > > > > > > > hth, > > > grant > > > > > > > > > > > > ------------------------------------------------------- > > > SF.Net email is sponsored by: GoToMeeting - the > > easiest way to > > collaborate > > > online with coworkers and clients while avoiding > > the high cost of travel > > and > > > communications. There is no equipment to buy and > > you can meet as often as > > > you want. Try it > > > free.http://ads.osdn.com/?ad_id=7402&alloc_id=16135&op=click > > > _______________________________________________ > > > Rdma-developers mailing list > > > Rdma-developers at lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/rdma-developers > > __________________________________________________ > Do You Yahoo!? > Tired of spam? Yahoo! Mail has the best spam protection > around http://mail.yahoo.com > > > ------------------------------------------------------- > This SF.Net email is sponsored by Yahoo. > Introducing Yahoo! Search Developer Network - Create apps using Yahoo! > Search APIs Find out how you can build Yahoo! directly into your own > Applications - visit > http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005 > _______________________________________________ > Rdma-developers mailing list > Rdma-developers at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/rdma-developers > From caitlinb at siliquent.com Fri May 27 09:45:06 2005 From: caitlinb at siliquent.com (Caitlin Bestler) Date: Fri, 27 May 2005 09:45:06 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux Message-ID: <8508251A6FC08A489844A94261D3693A05758D@fiona.siliquent.com> ________________________________ From: rdma-developers-admin at lists.sourceforge.net [mailto:rdma-developers-admin at lists.sourceforge.net] On Behalf Of Michael Krause Sent: Friday, May 27, 2005 7:25 AM To: Sukanta ganguly Cc: openib-general at openib.org; rdma-developers at lists.sourceforge.net Subject: Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux At 06:40 AM 5/27/2005, Sukanta ganguly wrote: Venkata, How will that work? If the RNIC offloads RDMA and TCP completely from the Operating System and does not share any state information then the application running on the host will never be in the position to utilize the socket interface to use the communication logic to send and receive data between the remote node and itself. Some information needs to be shared. How much of it and what exactly needs to be shared is the question. Ok. It all depends upon what level of integration / interaction a TOE and thus a RNIC will have with the host network stack. For example, if a customer wants to have TCP and IP stats kept for the off-loaded stack even if it is just being using for RDMA, then there needs to be a method defined to consolidate these stats back into the host network stack tool chain. Similarly, if one wants to maintain a single routing table to manage, etc. on the host, then the RNIC needs to access / update that information accordingly. One can progress through other aspects of integration, e.g. connection management, security interactions (e.g. DOS protection), and so forth. What is exposed again depends upon the level of integration and how customers want to manage their services. This problem also exists for IB but most people have not thought about this from a customer perspective and how to integrate the IB semantics into the way customers manage their infrastructures, do billing, etc. For some environments, they simply do not care but if IB is to be used in the enterprise space, then some thought will be required here since most IT don't see anything as being "free" or self-managed. Again, Sockets is an application API and not how one communicates to a TOE or RDMA component. The RNIC PI has been proposed as an interface to the RDMA functionality. The PI supports all of the iWARP and IB v 1.2 verbs. Mike I'd like to add that RNIC-PI is planning on explicitly defining some of these "obvious" dependencies between the RDMA stack and the primary IP stack. For example, the RDMA stack cannot maintain any connection in a state that contradicts current IP stack routing. It has to adapt or break the connection. We can't have an RNIC that has its own ARP table that is not in sync with the host's ARP table. An iWarp RDMA stack gains the benefit of many pre-existing network services (such as DNS, ARP and routing). But that also carries with it the need to not contradict those exisiting services. So it is both a benefit and a restriction -- and a major divergence from an IB RDMA stack. -------------- next part -------------- An HTML attachment was scrubbed... URL: From roland at topspin.com Fri May 27 09:48:37 2005 From: roland at topspin.com (Roland Dreier) Date: Fri, 27 May 2005 09:48:37 -0700 Subject: [openib-general] Re: registering read-only memory In-Reply-To: <506C3D7B14CDD411A52C00025558DED605B1A88B@mtlex01.yok.mtl.com> (Eli Cohen's message of "Fri, 27 May 2005 08:26:26 +0300") References: <506C3D7B14CDD411A52C00025558DED605B1A88B@mtlex01.yok.mtl.com> Message-ID: <52r7fsfwd6.fsf@topspin.com> Eli> I saw this this on Suse 9.3 and the memory was like: static Eli> const char *my_string = "Hello world"; defined locally in a Eli> function. This produced a VMA with VM_IO set. What kernel version/architecture? I'm having a hard time seeing how it's possible to get a VMA with VM_IO set except by using remap_pfn_range(), and I can't see how ordinary const data from userspace could cause that. - R. From caitlinb at siliquent.com Fri May 27 09:49:06 2005 From: caitlinb at siliquent.com (Caitlin Bestler) Date: Fri, 27 May 2005 09:49:06 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux Message-ID: <8508251A6FC08A489844A94261D3693A05758E@fiona.siliquent.com> With respect to the kernel based TCP stack, what is not needed is a hack into the stack and scatter/gather state information of the live TCP connection between kernel and RNIC, but to find one clean interface to transfer state information out of that stack and to the RNIC. Agreed that is an OS-specific service that should not be implemented in a device dependent way by each hardware vendor. But the information extracted from the kernel IP stack cannot be encoded as though it came from an InfiniBand CM. -------------- next part -------------- An HTML attachment was scrubbed... URL: From krause at cup.hp.com Fri May 27 09:40:41 2005 From: krause at cup.hp.com (Michael Krause) Date: Fri, 27 May 2005 09:40:41 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux In-Reply-To: <20050527162906.GA7444@esmail.cup.hp.com> References: <20050527134043.92998.qmail@web51901.mail.yahoo.com> <6.2.0.14.2.20050527071821.02071338@esmail.cup.hp.com> <20050527162906.GA7444@esmail.cup.hp.com> Message-ID: <6.2.0.14.2.20050527093019.02830450@esmail.cup.hp.com> At 09:29 AM 5/27/2005, Grant Grundler wrote: >On Fri, May 27, 2005 at 07:24:44AM -0700, Michael Krause wrote: >... > > Again, Sockets is an application API and not how one communicates to a TOE > > or RDMA component. > >Mike, >What address family is used to open a socket over iWARP? AF_INET? >Or something else? TCP = AF_INET. Address family != Sockets. Sockets is an API that can operate over multiple address families. An application can be coded to Sockets, IT API, DAPL, or a verbs interface like the RNIC PI. It is a matter of choice as well as what is trying to be accomplished. The RNIC PI is an acceptable interface for any RDMA-focused ULP. There are pros / cons to using such a verbs interface directly but I do not believe any one can deny that a general-purpose verbs API is a good thing at the end of the day as it works for the volume verbs definition. Whether one applies further hardware semantics abstraction such as IT API / DAPL should be a choice for the individual subsystem as there is no single right answer across all subsystems. Attempting to force fit isn't practical. >I understand most of what you wrote but am still missing one bit: >How is the RNIC told what the peer IP is it should communicate with? The destination address (IB GID or IP) is derived from the CM services. This is where the two interconnects differ in what is required to physical inject a packet on the wire. This is why I call it out as separate from the verbs interface and something that could be abstracted to some extent but at the end of the day, really requires the subsystem to understand the underlying fabric type to make some intelligent choices. Given this effort is still nascent, most of the issues beyond basic bootstrap have not really been discussed as yet. > > The RNIC PI has been proposed as an interface to the > > RDMA functionality. The PI supports all of the iWARP and IB v 1.2 verbs. > >That's good. Folks from RDMA consortium will have to look at openib >implementations and see whats missing/wrong. >Then submit proposals to fill in the gaps. I'm obviously not the first one >to say this. There are two open source efforts. The question is whether to move to a single effort (I tried to get this to occur before OpenIB was formally launched but it seem to fall on deaf ears for TTM marketing purposes) or whether to just coordinate on some of the basics. My preference remains that the efforts remained strictly focused on the RDMA infrastructure and interconnect-specific components and leave the ULP / services as separate efforts who will make their own decisions on how best to interface with the RDMA infrastructure. >I expect most of the principals involved with openib.org do NOT have time >to browse through RNIC PI at this point. They are struggling to get >openib.org filled in sufficiently so it can go into a commercial distro >(RH/SuSE primarily). Hence, why OpenRDMA needs to get source being developed to enable the RNIC community. If people find value in the work, then people can look at finding the right solution for both IB and iWARP when it makes sense. >Revenue for them comes from selling IB equipment. Having openib.org code >in kernel.org is a key enabler for getting >into commercial distros. I expect the same is true for RNIC vendors as well. > >RNIC Vendors (and related switch Vendors) will have to decide which path >is the right one for them to get the support into kernel.org. Several >openib.org people have suggested one (like I have). RNIC folks need to >listen and decide if the advice is good or not. If RNIC folks think they >know better, then please take another look at where openib.org is today >and where rdmaconsortium is. > >I'm certain openib.org would be dead now if policies and direction changes >had not made last year as demanded by several key linux developers and >"users" (Gov Labs). Understood. Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From krause at cup.hp.com Fri May 27 09:50:33 2005 From: krause at cup.hp.com (Michael Krause) Date: Fri, 27 May 2005 09:50:33 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux In-Reply-To: References: <20050527134043.92998.qmail@web51901.mail.yahoo.com> Message-ID: <6.2.0.14.2.20050527094045.028301c0@esmail.cup.hp.com> At 08:05 AM 5/27/2005, Bernard Metzler wrote: >Sukanta, > >without touching any TOE issues (this question is about RDMA, right?), after >transforming a TCP connection into RDMA mode and using an RDMA API, >the socket file descriptor is not longer to be used for communication. Whether one uses a Socket FD or not is an application issue. A QP has a handle identifier which may be mapped to a FD or may be used directly by an application. There is no requirement that anything flow through the Sockets API - that is a choice for the application to make. >In fact, on some implementations the stream socket resources will even get >released. That is, if the socket was the direct consumer of the TCP >stream, then now it is RDMAP/DDP/MPA. RDMA APIs such as IT-API defining a >specific call to convert a socket based connection into RDMA mode (e.g., >it_socket_convert()). Other consumers may directly start via an RDMA API, >never >opening a consumer controlled socket. Correct. >So, in RDMA mode, communication will happen via the RDMA API. At this >stage, the kernel still have to keep completely in its hands the >synchronisation of state information related to that offloaded >connection(s) with the host stack (it would have to protect the local port >used by the offloaded connection for example, others are routing, ARP, >SNMP...), but it is not involved at the data path. This is true only when the host and the RNIC share a common IP address. If they are separate subnets, then there is no need to coordinate sans some of the reasons I noted in an earlier e-mail. I think there is great value and customer need to have an infrastructure that can coordinate information and enable customers to use the existing tool chains to understand what is going on in a given device or endnode. >With respect to the kernel based TCP stack, what is not needed is a hack >into the stack and scatter/gather state information of the live TCP >connection between kernel and RNIC, but to find one clean interface to >transfer state information out of that stack and to the RNIC. At a minimum, one needs: - A host network stack get state call to extract connection state and quiesce the port from the host's perspective. - A host network stack set state call to populate the host structures with the associated state and enable the port - A RNIC get state call to extract all transport and RMDA context - A RNIC set state call to populate all transport and RDMA context These calls should be "standardized" so that both sides of the infrastructure will be able to utilize without having to hack anything. >With limited benefit, one could of course also implement native sockets >over RDMA, where an in-kernel midlayer on top of kernel >RDMA Verbs is doing the translation between send(), receive() to >post_send, post_receive. But usage of 'true RDMA' operations >like RDMA READ or WRITE might be limited, and I don't see much value for >the user here. One variety of this approach with less limited access to >RMDA benefits might be sockets with extended RDMA semantics. Sockets with RDMA extensions was proposed a couple of years back by some of us. There would be quite a bit of benefit as most developers know how to code to Sockets and the RDMA extensions are relatively simple when you think about it. It would also eliminate the need to use SDP by placing the RDMA knowledge requirement on the application. SDP has benefits for enabling existing Sockets applications to operate transparently over RDMA but the explicit use of RDMA extensions I think would provide greater benefit in the end. While it is important to provide legacy investment protection, real innovation will not occur until people start developing technology that enables a smarter customer. Simplification and performance also come with enabling a smarter customer. Mike >Bernard. > >rdma-developers-admin at lists.sourceforge.net wrote on 27.05.2005 15:40:43: > > > Venkata, > > How will that work? If the RNIC offloads RDMA and > > TCP completely from the Operating System and does not > > share any state information then the application > > running on the host will never be in the position to > > utilize the socket interface to use the communication > > logic to send and receive data between the remote node > > and itself. Some information needs to be shared. How > > much of it and what exactly needs to be shared is the > > question. > > > > Thanks > > SG > > > > --- Venkata Jagana wrote: > > > > > > > > > > > > > > > > > > > > > rdma-developers-admin at lists.sourceforge.net wrote on > > > 05/25/2005 09:47:00 > > > PM: > > > > > > > Venkata, > > > > Interesting coincidence: I was talking with > > > someone (at HP) today > > > > who knows substantially more than I do about > > > RNICs. > > > > They indicated RNICs need to manage TCP state on > > > the card from userspace. > > > > I suspect that's only possible through a private > > > interface > > > > (e.g. ioctl() or /proc) or the non-existant (in > > > kernel.org) > > > > TOE implementation. Is this correct? > > > > > > > > > > Not correct. > > > > > > Since RNICs are offloaded adapters with RDMA > > > protocols layered on > > > top of TCP stack, they do maintain the TCP state > > > internally but > > > it does not expose to the host. RNIC expose only > > > RNIC Verbs interface > > > to the host bot not TOE interface. > > > > > > Thanks > > > Venkat > > > > > > > > > > > hth, > > > > grant > > > > > > > > > > > > > > > > > ------------------------------------------------------- > > > > SF.Net email is sponsored by: GoToMeeting - the > > > easiest way to > > > collaborate > > > > online with coworkers and clients while avoiding > > > the high cost of travel > > > and > > > > communications. There is no equipment to buy and > > > you can meet as often as > > > > you want. Try it > > > > > free.http://ads.osdn.com/?ad_id=7402&alloc_id=16135&op=click > > > > _______________________________________________ > > > > Rdma-developers mailing list > > > > Rdma-developers at lists.sourceforge.net > > > > https://lists.sourceforge.net/lists/listinfo/rdma-developers > > > > __________________________________________________ > > Do You Yahoo!? > > Tired of spam? Yahoo! Mail has the best spam protection around > > http://mail.yahoo.com > > > > > > ------------------------------------------------------- > > This SF.Net email is sponsored by Yahoo. > > Introducing Yahoo! Search Developer Network - Create apps using Yahoo! > > Search APIs Find out how you can build Yahoo! directly into your own > > Applications - visit http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005 > > _______________________________________________ > > Rdma-developers mailing list > > Rdma-developers at lists.sourceforge.net > > https://lists.sourceforge.net/lists/listinfo/rdma-developers >_______________________________________________ >openib-general mailing list >openib-general at openib.org >http://openib.org/mailman/listinfo/openib-general > >To unsubscribe, please visit >http://openib.org/mailman/listinfo/openib-general -------------- next part -------------- An HTML attachment was scrubbed... URL: From caitlinb at siliquent.com Fri May 27 09:55:05 2005 From: caitlinb at siliquent.com (Caitlin Bestler) Date: Fri, 27 May 2005 09:55:05 -0700 Subject: [Rdma-developers] RE: [openib-general] OpenIB and OpenRDMA: Convergence on commonRDMA APIs and ULPs for Linux Message-ID: <8508251A6FC08A489844A94261D3693A05758F@fiona.siliquent.com> > -----Original Message----- > From: rdma-developers-admin at lists.sourceforge.net > [mailto:rdma-developers-admin at lists.sourceforge.net] On > Behalf Of Sean Hefty > Sent: Friday, May 27, 2005 9:22 AM > To: 'Roland Dreier'; Venkata Jagana > Cc: rdma-developers at lists.sourceforge.net; openib-general at openib.org > Subject: [Rdma-developers] RE: [openib-general] OpenIB and > OpenRDMA: Convergence on commonRDMA APIs and ULPs for Linux > > >To put a really concrete proposal on the table, I would suggest to > >start by extending the current ib_client registration > structure, which > >looks like > > > > struct ib_client { > > char *name; > > void (*add) (struct ib_device *); > > void (*remove)(struct ib_device *); > > > > struct list_head list; > > }; > > > >by extending the current enum ib_node_type to something like > > > > enum rdma_device_type { > > RDMA_DEVICE_IB_CA, > > RDMA_DEVICE_IB_SWITCH, > > RDMA_DEVICE_IB_ROUTER, > > RDMA_DEVICE_RNIC > > }; > > > >Then the various pieces of code layered on top of the RDMA > midlayer can > >decide whether they want to deal with a particular device or not by > >looking at the node_type member. For example, the IB CM, > IPoIB, etc. > >could ignore devices of type RDMA_DEVICE_RNIC, while SDP or > iSER would > >use all devices and the RNIC CM would take only devices of type > >RDMA_DEVICE_RNIC. > > > >Then someone would have to start implementing a low-level > driver for a > >specific RNIC, and find which modifications to the existing > verbs are > >required. For example, I believe the QP attribute structure passed > >into the QP modify verb probably has to become a union > containing the > >IB attributes and the RNIC attributes. However, most verbs > should work > >fine with at most trivial modifications. > > Is there some technical reason why this proposal cannot be > used? I didn't notice any other proposal. > > - Sean > There isn't enough there to go farther. What methods is the RDMA_DEVICE_RNIC required to support? a) Totally iWARP specific methods. That works, but all you've done is taken two stacks and put a switch on top of them. kDAPL already does that quite well. b) The same methods, but with different struct and enum definitions? (i.e., a struct iwarp_qp versus a struct ib_qp). IMHO, that doesn't differ much from a) c) The same methods but with struct/enums that have common and transport specific portions? That is doable, in fact that is what RNIC-PI is today. Repeating that work with the gen2 verbs will be time consuming. I don't want to have to wait 4 months to debate the details of this before I can start working on my next generation of verbs. d) Use the methods and structs 'as is' and just somehow figure out how they map to iWARP? That is no more "transport neutral" than Tom DeLay is "bipartisan". From caitlinb at siliquent.com Fri May 27 09:58:44 2005 From: caitlinb at siliquent.com (Caitlin Bestler) Date: Fri, 27 May 2005 09:58:44 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux Message-ID: <8508251A6FC08A489844A94261D3693A057590@fiona.siliquent.com> > -----Original Message----- > From: rdma-developers-admin at lists.sourceforge.net > [mailto:rdma-developers-admin at lists.sourceforge.net] On > Behalf Of Grant Grundler > Sent: Friday, May 27, 2005 9:29 AM > To: Michael Krause > Cc: Sukanta ganguly; rdma-developers at lists.sourceforge.net; > openib-general at openib.org > Subject: Re: [Rdma-developers] Re: [openib-general] OpenIB > and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux > > On Fri, May 27, 2005 at 07:24:44AM -0700, Michael Krause wrote: > ... > > Again, Sockets is an application API and not how one > communicates to a > > TOE or RDMA component. > > Mike, > What address family is used to open a socket over iWARP? AF_INET? > Or something else? > > I understand most of what you wrote but am still missing one bit: > How is the RNIC told what the peer IP is it should communicate with? > The TCP Connection is already established before the RNIC takes control of the connection. Therefore the entire TCP state is already established (source/dest IP Address/Port, negotiated options, EMSS, local options, etc.). From iod00d at hp.com Fri May 27 10:05:57 2005 From: iod00d at hp.com (Grant Grundler) Date: Fri, 27 May 2005 10:05:57 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux In-Reply-To: <8508251A6FC08A489844A94261D3693A05758D@fiona.siliquent.com> References: <8508251A6FC08A489844A94261D3693A05758D@fiona.siliquent.com> Message-ID: <20050527170557.GB7444@esmail.cup.hp.com> On Fri, May 27, 2005 at 09:45:06AM -0700, Caitlin Bestler wrote: > I'd like to add that RNIC-PI is planning on explicitly defining some of > these "obvious" dependencies > between the RDMA stack and the primary IP stack. For example, the RDMA > stack cannot maintain > any connection in a state that contradicts current IP stack routing. It > has to adapt or break the connection. That's what I was just thinking. Could RNIC driver support both existing linux NIC interfaces (e.g. use ifconfig/ethtool) *and* openib RDMA interface? Essentially that's what openib.org does today but uses ib_ipoib driver to support TCP/IP communication. Ie use an AF_INET socket to setup an RDMA connection like the rdma_lat.c does: https://openib.org/svn/gen2/trunk/src/userspace/perftest/ Of course, I'm "blissfully ignorant" of how ugly that might be in real world implementation of RNIC driver. Seems simple in concept at least. > We can't have an RNIC that has its own ARP table that is not in sync > with the host's ARP table. Yes, the RDMA and NIC parts of the driver would have to be aware of each other. grant > An iWarp RDMA stack gains the benefit of many pre-existing network > services (such as DNS, ARP > and routing). But that also carries with it the need to not contradict > those exisiting services. So it is > both a benefit and a restriction -- and a major divergence from an IB > RDMA stack. From caitlinb at siliquent.com Fri May 27 10:07:16 2005 From: caitlinb at siliquent.com (Caitlin Bestler) Date: Fri, 27 May 2005 10:07:16 -0700 Subject: [Rdma-developers] RE: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux Message-ID: <8508251A6FC08A489844A94261D3693A057591@fiona.siliquent.com> > -----Original Message----- > From: rdma-developers-admin at lists.sourceforge.net > [mailto:rdma-developers-admin at lists.sourceforge.net] On > Behalf Of Woodruff, Robert J > Sent: Thursday, May 26, 2005 5:12 PM > To: Caitlin Bestler > Cc: Venkata Jagana; rdma-developers at lists.sourceforge.net; > openib-general at openib.org > Subject: [Rdma-developers] RE: [openib-general] OpenIB and > OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux > > >RNIC-PI is at least an attempt at providing full control over both > >iWARP and IB while making as much common as possible. > > Where were you last year when the IB verbs header files/API > were being discussed. > Seems like that was the time to discuss new APIs. > Now it is a bit late to propose something totally new like RNIC-PI. > The best thing to do now is to try to influence the existing > code base to meet your needs rather than recommend a totally new API. > And the best way to do that is to make changes to the code > that exists or write new code and send in patches to the list > so that it can be discussed. > > woody > OpenIB's announced charter was IB specific. There was no announcement that the intent was to define *all* RDMA services. Where were you when RNIC-PI was discussing these issues? It's announced charter *was* to be transport neutral and OS neutral. But that's water under the bridge. The fact is that both IB and iWARP vendors have existing development efforts and any realistic plan will recognize that. The question is to what degree code can be made common without requiring anyone to start over from scratch. That may involve merely defining critical interfaces with the rest of the OS, and then splitting into two transport dependent sub-systems. Or it may be possible to come up with a transport-neutral low level API. The latter goal is indeed desirable, but I am skeptical that it is compatible with the needs of either IB or iWARP vendors to be making forward progress on actual code today. It may make more sense to agree on the fundamental interfaces to the rest of the kernel, and on approximately what transport neutral verbs might look like someday, and then to develop in parallel until there is two branches of code that both work before attempting to merge them. From iod00d at hp.com Fri May 27 10:10:28 2005 From: iod00d at hp.com (Grant Grundler) Date: Fri, 27 May 2005 10:10:28 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux In-Reply-To: <8508251A6FC08A489844A94261D3693A057590@fiona.siliquent.com> References: <8508251A6FC08A489844A94261D3693A057590@fiona.siliquent.com> Message-ID: <20050527171028.GC7444@esmail.cup.hp.com> On Fri, May 27, 2005 at 09:58:44AM -0700, Caitlin Bestler wrote: > > I understand most of what you wrote but am still missing one bit: > > How is the RNIC told what the peer IP is it should communicate with? > > The TCP Connection is already established before the RNIC takes > control of the connection. Therefore the entire TCP state is already > established (source/dest IP Address/Port, negotiated options, EMSS, > local options, etc.). Would "shares the connection" be a better description? I think I get the concepts at least now...this follows what I was suggesting earlier about having an RNIC support both NAPI (New API for NICs) (code under drivers/net) and openib.org Gen2 stack (code in drivers/infiniband). thanks, grant From caitlinb at siliquent.com Fri May 27 10:13:23 2005 From: caitlinb at siliquent.com (Caitlin Bestler) Date: Fri, 27 May 2005 10:13:23 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux Message-ID: <8508251A6FC08A489844A94261D3693A057592@fiona.siliquent.com> Both uDAPL and kDAPL were designed for *application* use. Even kDAPL is more intended for use by a kernel daemon that is loaded separately from the kernel than for use within the kernel itself. An ideal API for use within the kernel would abstract as much as possible (without requiring emulation), and then have transport specific unions or enum values. It would hide no control options, merely provide common controls for common capabilities. It would also not bother to unify the event streams, but rahter present the raw event streams. Applications find that unification useful and enabling, but that is why application APIs like uDAPL, kDAPL and IT-API exist. Having the lowest level API possible within the kernel is certainly a good long term goal, but as outlined elsewhere I am skeptical that is achievable in the short term. kDAPL would certainly provide an acceptable fill-in until such an API could be ironed out. > -----Original Message----- > From: rdma-developers-admin at lists.sourceforge.net > [mailto:rdma-developers-admin at lists.sourceforge.net] On > Behalf Of Roland Dreier > Sent: Thursday, May 26, 2005 9:33 AM > To: Bob Woodruff > Cc: 'Venkata Jagana'; rdma-developers at lists.sourceforge.net; > openib-general at openib.org > Subject: [Rdma-developers] Re: [openib-general] OpenIB and > OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux > > Bob> There is already a RDMA device independent API being > Bob> developed for the kernel by people on this list. It is > Bob> starting with the kDAPL code base, which was designed to > Bob> support both IBA and iWarp devices. > > I believe kDAPL-based layers are an OK short-term solution, > but I don't think anything like this should be proposed for > merging in the Linus kernel. If we need another abstraction > layer on top of our existing abstraction layer, that just > says to me that we should fix the current abstraction layer. > > - R. > > > ------------------------------------------------------- > This SF.Net email is sponsored by Yahoo. > Introducing Yahoo! Search Developer Network - Create apps using Yahoo! > Search APIs Find out how you can build Yahoo! directly into > your own Applications - visit > http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005 > _______________________________________________ > Rdma-developers mailing list > Rdma-developers at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/rdma-developers > From caitlinb at siliquent.com Fri May 27 10:16:26 2005 From: caitlinb at siliquent.com (Caitlin Bestler) Date: Fri, 27 May 2005 10:16:26 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux Message-ID: <8508251A6FC08A489844A94261D3693A057593@fiona.siliquent.com> > -----Original Message----- > From: rdma-developers-admin at lists.sourceforge.net > [mailto:rdma-developers-admin at lists.sourceforge.net] On > Behalf Of Talpey, Thomas > Sent: Thursday, May 26, 2005 5:28 AM > To: Venkata Jagana > Cc: rdma-developers at lists.sourceforge.net; openib-general at openib.org > Subject: [Rdma-developers] Re: [openib-general] OpenIB and > OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux > > At 02:20 AM 5/26/2005, Venkata Jagana wrote: > >I would like to really understand the technical reasons why > you say RNIC-PI is irrelevant to Linux kernel. > >RNIC-PI is developed to support not only the RNICs but it is > also IB compatible. > > I'm not Roland, but my belief is that until RNIC-PI exists as > working code on a reference implementation, it is in fact not > yet relevant to Linux. Perhaps you can outline the schedule > to get there? > That is a slightly circular argument. The primary reason that work to implement DAPL and IT-API over RNIC-PI is not already in progress is the pressure to come up with a "one stack" solution. From sganguly at yahoo.com Fri May 27 10:25:36 2005 From: sganguly at yahoo.com (Sukanta ganguly) Date: Fri, 27 May 2005 10:25:36 -0700 (PDT) Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux In-Reply-To: <6.2.0.14.2.20050527071821.02071338@esmail.cup.hp.com> Message-ID: <20050527172536.83795.qmail@web51912.mail.yahoo.com> Mike, I am not sure I do understand what your were trying to communicate. Let me try and decode this. My basic point was to respond to Venkata's response about complete offload without any interaction with the host system. I disagree with that in its totality as I think there are dependencies and that needs to be specified in a formal manner, i.e. in the specs, so that we do not have multiple proprietary interafaces which change and application users have to change this consumption based on individual implementations. And you just brought up the reasons why what I was saying seemed to be justfied. Did I read that correctly? Thanks SG --- Michael Krause wrote: > At 06:40 AM 5/27/2005, Sukanta ganguly wrote: > >Venkata, > > How will that work? If the RNIC offloads RDMA > and > >TCP completely from the Operating System and does > not > >share any state information then the application > >running on the host will never be in the position > to > >utilize the socket interface to use the > communication > >logic to send and receive data between the remote > node > >and itself. Some information needs to be shared. > How > >much of it and what exactly needs to be shared is > the > >question. > > Ok. It all depends upon what level of integration / > interaction a TOE and > thus a RNIC will have with the host network stack. > For example, if a > customer wants to have TCP and IP stats kept for the > off-loaded stack even > if it is just being using for RDMA, then there needs > to be a method defined > to consolidate these stats back into the host > network stack tool > chain. Similarly, if one wants to maintain a single > routing table to > manage, etc. on the host, then the RNIC needs to > access / update that > information accordingly. One can progress through > other aspects of > integration, e.g. connection management, security > interactions (e.g. DOS > protection), and so forth. What is exposed again > depends upon the level of > integration and how customers want to manage their > services. This problem > also exists for IB but most people have not thought > about this from a > customer perspective and how to integrate the IB > semantics into the way > customers manage their infrastructures, do billing, > etc. For some > environments, they simply do not care but if IB is > to be used in the > enterprise space, then some thought will be required > here since most IT > don't see anything as being "free" or self-managed. > > Again, Sockets is an application API and not how one > communicates to a TOE > or RDMA component. The RNIC PI has been proposed as > an interface to the > RDMA functionality. The PI supports all of the > iWARP and IB v 1.2 verbs. > > Mike > > > >Thanks > >SG > > > >--- Venkata Jagana wrote: > > > > > > > > > > > > > > > > > > > > > rdma-developers-admin at lists.sourceforge.net > wrote on > > > 05/25/2005 09:47:00 > > > PM: > > > > > > > Venkata, > > > > Interesting coincidence: I was talking with > > > someone (at HP) today > > > > who knows substantially more than I do about > > > RNICs. > > > > They indicated RNICs need to manage TCP state > on > > > the card from userspace. > > > > I suspect that's only possible through a > private > > > interface > > > > (e.g. ioctl() or /proc) or the non-existant > (in > > > kernel.org) > > > > TOE implementation. Is this correct? > > > > > > > > > > Not correct. > > > > > > Since RNICs are offloaded adapters with RDMA > > > protocols layered on > > > top of TCP stack, they do maintain the TCP state > > > internally but > > > it does not expose to the host. RNIC expose only > > > RNIC Verbs interface > > > to the host bot not TOE interface. > > > > > > Thanks > > > Venkat > > > > > > > > > > > hth, > > > > grant > > > > > > > > > > > > > > > > >------------------------------------------------------- > > > > SF.Net email is sponsored by: GoToMeeting - > the > > > easiest way to > > > collaborate > > > > online with coworkers and clients while > avoiding > > > the high cost of travel > > > and > > > > communications. There is no equipment to buy > and > > > you can meet as often as > > > > you want. Try it > > > > >free.http://ads.osdn.com/?ad_id=7402&alloc_id=16135&op=click > > > > > _______________________________________________ > > > > Rdma-developers mailing list > > > > Rdma-developers at lists.sourceforge.net > > > > > https://lists.sourceforge.net/lists/listinfo/rdma-developers > > > >__________________________________________________ > >Do You Yahoo!? > >Tired of spam? Yahoo! Mail has the best spam > protection around > >http://mail.yahoo.com > > > > > >------------------------------------------------------- > >This SF.Net email is sponsored by Yahoo. > >Introducing Yahoo! Search Developer Network - > Create apps using Yahoo! > >Search APIs Find out how you can build Yahoo! > directly into your own > >Applications - visit > http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005 > >_______________________________________________ > >Rdma-developers mailing list > >Rdma-developers at lists.sourceforge.net > >https://lists.sourceforge.net/lists/listinfo/rdma-developers > __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From sganguly at yahoo.com Fri May 27 10:28:43 2005 From: sganguly at yahoo.com (Sukanta ganguly) Date: Fri, 27 May 2005 10:28:43 -0700 (PDT) Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux In-Reply-To: Message-ID: <20050527172843.6976.qmail@web51905.mail.yahoo.com> I agree. SG --- Bernard Metzler wrote: > Sukanta, > > without touching any TOE issues (this question is > about RDMA, right?), > after > transforming a TCP connection into RDMA mode and > using an RDMA API, > the socket file descriptor is not longer to be used > for communication. > In fact, on some implementations the stream socket > resources will > even get released. That is, if the socket was the > direct consumer > of the TCP stream, then now it is RDMAP/DDP/MPA. > RDMA APIs such as IT-API defining a specific call to > convert a socket > based > connection into RDMA mode (e.g., > it_socket_convert()). > Other consumers may directly start via an RDMA API, > never > opening a consumer controlled socket. > > So, in RDMA mode, communication will happen via the > RDMA API. At this > stage, the kernel still have to keep completely in > its hands the > synchronisation of state information related to that > offloaded > connection(s) with the host stack (it would have to > protect the > local port used by the offloaded connection for > example, others > are routing, ARP, SNMP...), but it is not involved > at the data path. > > With respect to the kernel based TCP stack, what is > not needed is > a hack into the stack and scatter/gather state > information of the live > TCP connection between kernel and RNIC, but to find > one clean interface > to transfer state information out of that stack and > to the RNIC. > > > With limited benefit, one could of course also > implement native > sockets over RDMA, where an in-kernel midlayer on > top of kernel > RDMA Verbs is doing the translation between send(), > receive() to > post_send, post_receive. But usage of 'true RDMA' > operations > like RDMA READ or WRITE might be limited, and I > don't see much value > for the user here. One variety of this approach with > less limited > access to RMDA benefits might be sockets with > extended RDMA > semantics. > > Bernard. > > rdma-developers-admin at lists.sourceforge.net wrote on > 27.05.2005 15:40:43: > > > Venkata, > > How will that work? If the RNIC offloads RDMA > and > > TCP completely from the Operating System and does > not > > share any state information then the application > > running on the host will never be in the position > to > > utilize the socket interface to use the > communication > > logic to send and receive data between the remote > node > > and itself. Some information needs to be shared. > How > > much of it and what exactly needs to be shared is > the > > question. > > > > Thanks > > SG > > > > --- Venkata Jagana wrote: > > > > > > > > > > > > > > > > > > > > > rdma-developers-admin at lists.sourceforge.net > wrote on > > > 05/25/2005 09:47:00 > > > PM: > > > > > > > Venkata, > > > > Interesting coincidence: I was talking with > > > someone (at HP) today > > > > who knows substantially more than I do about > > > RNICs. > > > > They indicated RNICs need to manage TCP state > on > > > the card from userspace. > > > > I suspect that's only possible through a > private > > > interface > > > > (e.g. ioctl() or /proc) or the non-existant > (in > > > kernel.org) > > > > TOE implementation. Is this correct? > > > > > > > > > > Not correct. > > > > > > Since RNICs are offloaded adapters with RDMA > > > protocols layered on > > > top of TCP stack, they do maintain the TCP state > > > internally but > > > it does not expose to the host. RNIC expose only > > > RNIC Verbs interface > > > to the host bot not TOE interface. > > > > > > Thanks > > > Venkat > > > > > > > > > > > hth, > > > > grant > > > > > > > > > > > > > > > > > > ------------------------------------------------------- > > > > SF.Net email is sponsored by: GoToMeeting - > the > > > easiest way to > > > collaborate > > > > online with coworkers and clients while > avoiding > > > the high cost of travel > > > and > > > > communications. There is no equipment to buy > and > > > you can meet as often as > > > > you want. Try it > > > > > > free.http://ads.osdn.com/?ad_id=7402&alloc_id=16135&op=click > > > > > _______________________________________________ > > > > Rdma-developers mailing list > > > > Rdma-developers at lists.sourceforge.net > > > > > https://lists.sourceforge.net/lists/listinfo/rdma-developers > > > > __________________________________________________ > > Do You Yahoo!? > > Tired of spam? Yahoo! Mail has the best spam > protection around > > http://mail.yahoo.com > > > > > > > ------------------------------------------------------- > > This SF.Net email is sponsored by Yahoo. > > Introducing Yahoo! Search Developer Network - > Create apps using Yahoo! > > Search APIs Find out how you can build Yahoo! > directly into your own > > Applications - visit > http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005 > > _______________________________________________ > > Rdma-developers mailing list > > Rdma-developers at lists.sourceforge.net > > > https://lists.sourceforge.net/lists/listinfo/rdma-developers > __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From caitlinb at siliquent.com Fri May 27 10:30:13 2005 From: caitlinb at siliquent.com (Caitlin Bestler) Date: Fri, 27 May 2005 10:30:13 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux Message-ID: <8508251A6FC08A489844A94261D3693A057594@fiona.siliquent.com> ________________________________ From: rdma-developers-admin at lists.sourceforge.net [mailto:rdma-developers-admin at lists.sourceforge.net] On Behalf Of Michael Krause Sent: Friday, May 27, 2005 9:51 AM To: Bernard Metzler; Sukanta ganguly Cc: rdma-developers at lists.sourceforge.net; openib-general at openib.org; rdma-developers-admin at lists.sourceforge.net Subject: Re: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux At 08:05 AM 5/27/2005, Bernard Metzler wrote: Sukanta, without touching any TOE issues (this question is about RDMA, right?), after transforming a TCP connection into RDMA mode and using an RDMA API, the socket file descriptor is not longer to be used for communication. Whether one uses a Socket FD or not is an application issue. A QP has a handle identifier which may be mapped to a FD or may be used directly by an application. There is no requirement that anything flow through the Sockets API - that is a choice for the application to make. True. The current RNIC-PI spec is deliberately vague about what an "LLP Handle" is, other than that it must include a Socket FD. It may be possible for it to include some sort of "connection request handle" for a pending connection on the offload device, but you would have to ensure that this "connection requestion handle" was reviewed by the Host IP stack so that it could forbid connections that contradict the host's IP firewall policies. In fact, on some implementations the stream socket resources will even get released. That is, if the socket was the direct consumer of the TCP stream, then now it is RDMAP/DDP/MPA. RDMA APIs such as IT-API defining a specific call to convert a socket based connection into RDMA mode (e.g., it_socket_convert()). Other consumers may directly start via an RDMA API, never opening a consumer controlled socket. Correct. There may never be a *consumer* controlled socket, but there will be a *host* controlled socket used by the Connection Manager and subject to normal host firewall controls. It is in fact highly desirable from both a security and performance basis for the streaming mode TCP connection to never be exposed to the application. Only specialized applications, such as iSER, require this capability. So, in RDMA mode, communication will happen via the RDMA API. At this stage, the kernel still have to keep completely in its hands the synchronisation of state information related to that offloaded connection(s) with the host stack (it would have to protect the local port used by the offloaded connection for example, others are routing, ARP, SNMP...), but it is not involved at the data path. This is true only when the host and the RNIC share a common IP address. If they are separate subnets, then there is no need to coordinate sans some of the reasons I noted in an earlier e-mail. I think there is great value and customer need to have an infrastructure that can coordinate information and enable customers to use the existing tool chains to understand what is going on in a given device or endnode. Good point. The key is that the RNIC needs to do co-ordinate all of these with the IP stack that owns the IP address. If it is that owner, it must co-ordinate with itself. (Actually that's a clause that needs to be made more explicit in the RNIC-PI spec, which was focused on the shared IP scenario). But when the RNIC is "sub-leasing" the IP it needs to co-ordinate its action with the owner of the IP address. With respect to the kernel based TCP stack, what is not needed is a hack into the stack and scatter/gather state information of the live TCP connection between kernel and RNIC, but to find one clean interface to transfer state information out of that stack and to the RNIC. At a minimum, one needs: - A host network stack get state call to extract connection state and quiesce the port from the host's perspective. - A host network stack set state call to populate the host structures with the associated state and enable the port - A RNIC get state call to extract all transport and RMDA context - A RNIC set state call to populate all transport and RDMA context An RNIC "get state" is nice, but not essential. It enables transfer of an RDMA state to another RNIC for failover purposes. But in terms of minimal control, the ability to kill an RDMA controlled connection is sufficient. Additionally, any transfer of RDMA Context will be limited to transfer between like models for some time, and even then it is not likely to be a standard feature. -------------- next part -------------- An HTML attachment was scrubbed... URL: From caitlinb at siliquent.com Fri May 27 10:35:17 2005 From: caitlinb at siliquent.com (Caitlin Bestler) Date: Fri, 27 May 2005 10:35:17 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux Message-ID: <8508251A6FC08A489844A94261D3693A057595@fiona.siliquent.com> > -----Original Message----- > From: Grant Grundler [mailto:iod00d at hp.com] > Sent: Friday, May 27, 2005 10:06 AM > To: Caitlin Bestler > Cc: Michael Krause; Sukanta ganguly; > rdma-developers at lists.sourceforge.net; openib-general at openib.org > Subject: Re: [Rdma-developers] Re: [openib-general] OpenIB > and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux > > On Fri, May 27, 2005 at 09:45:06AM -0700, Caitlin Bestler wrote: > > I'd like to add that RNIC-PI is planning on explicitly > defining some > > of these "obvious" dependencies between the RDMA stack and > the primary > > IP stack. For example, the RDMA stack cannot maintain any > connection > > in a state that contradicts current IP stack routing. It > has to adapt > > or break the connection. > > That's what I was just thinking. Could RNIC driver support > both existing linux NIC interfaces (e.g. use > ifconfig/ethtool) *and* openib RDMA interface? > > Essentially that's what openib.org does today but uses > ib_ipoib driver to support TCP/IP communication. > > Ie use an AF_INET socket to setup an RDMA connection like the > rdma_lat.c does: > https://openib.org/svn/gen2/trunk/src/userspace/perftest/ > > Of course, I'm "blissfully ignorant" of how ugly that might > be in real world implementation of RNIC driver. Seems simple > in concept at least. > For time-to-market reasons, and to avoid having to go hacking deeply into the kernel from a driver, many or most existing implementations require or at least prefer using TOE sockets. It is much easier to extract the TCP state from a socket that you already control than from the host stack. But that is exactly the sort of limitation that integrating with the kernel should eliminate. Given a stable interface to "get TCP state" from a host controlled connection I can't imagine why an RNIC vendor would not eagerly embrace it. From sganguly at yahoo.com Fri May 27 10:38:24 2005 From: sganguly at yahoo.com (Sukanta ganguly) Date: Fri, 27 May 2005 10:38:24 -0700 (PDT) Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux In-Reply-To: 6667 Message-ID: <20050527173825.3103.qmail@web51911.mail.yahoo.com> --- Caitlin Bestler wrote: > > > > ________________________________ > > From: rdma-developers-admin at lists.sourceforge.net > [mailto:rdma-developers-admin at lists.sourceforge.net] > On Behalf Of > Michael Krause > Sent: Friday, May 27, 2005 7:25 AM > To: Sukanta ganguly > Cc: openib-general at openib.org; > rdma-developers at lists.sourceforge.net > Subject: Re: [Rdma-developers] Re: [openib-general] > OpenIB and > OpenRDMA: Convergence on common RDMA APIs and ULPs > for Linux > > > At 06:40 AM 5/27/2005, Sukanta ganguly wrote: > > > Venkata, > How will that work? If the RNIC offloads RDMA > and > TCP completely from the Operating System and does > not > share any state information then the application > running on the host will never be in the position > to > utilize the socket interface to use the > communication > logic to send and receive data between the remote > node > and itself. Some information needs to be shared. > How > much of it and what exactly needs to be shared is > the > question. > > > Ok. It all depends upon what level of integration > / interaction > a TOE and thus a RNIC will have with the host > network stack. For > example, if a customer wants to have TCP and IP > stats kept for the > off-loaded stack even if it is just being using for > RDMA, then there > needs to be a method defined to consolidate these > stats back into the > host network stack tool chain. Similarly, if one > wants to maintain a > single routing table to manage, etc. on the host, > then the RNIC needs to > access / update that information accordingly. One > can progress through > other aspects of integration, e.g. connection > management, security > interactions (e.g. DOS protection), and so forth. > What is exposed again > depends upon the level of integration and how > customers want to manage > their services. This problem also exists for IB but > most people have > not thought about this from a customer perspective > and how to integrate > the IB semantics into the way customers manage their > infrastructures, do > billing, etc. For some environments, they simply do > not care but if IB > is to be used in the enterprise space, then some > thought will be > required here since most IT don't see anything as > being "free" or > self-managed. > > Again, Sockets is an application API and not how > one > communicates to a TOE or RDMA component. The RNIC > PI has been proposed > as an interface to the RDMA functionality. The PI > supports all of the > iWARP and IB v 1.2 verbs. > > Mike > > > > I'd like to add that RNIC-PI is planning on > explicitly defining some of > these "obvious" dependencies > between the RDMA stack and the primary IP stack. For > example, the RDMA > stack cannot maintain > any connection in a state that contradicts current > IP stack routing. It > has to adapt or break the connection. > We can't have an RNIC that has its own ARP table > that is not in sync > with the host's ARP table. > > An iWarp RDMA stack gains the benefit of many > pre-existing network > services (such as DNS, ARP > and routing). But that also carries with it the need > to not contradict > those exisiting services. So it is > both a benefit and a restriction -- and a major > divergence from an IB > RDMA stack. > > > This is exactly what I was getting at and seems like the RNIC-PI is already on its way to do that. Thanks SG __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From caitlinb at siliquent.com Fri May 27 10:40:20 2005 From: caitlinb at siliquent.com (Caitlin Bestler) Date: Fri, 27 May 2005 10:40:20 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux Message-ID: <8508251A6FC08A489844A94261D3693A057597@fiona.siliquent.com> > -----Original Message----- > From: Grant Grundler [mailto:iod00d at hp.com] > Sent: Friday, May 27, 2005 10:10 AM > To: Caitlin Bestler > Cc: Sukanta ganguly; rdma-developers at lists.sourceforge.net; > openib-general at openib.org > Subject: Re: [Rdma-developers] Re: [openib-general] OpenIB > and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux > > On Fri, May 27, 2005 at 09:58:44AM -0700, Caitlin Bestler wrote: > > > I understand most of what you wrote but am still missing one bit: > > > How is the RNIC told what the peer IP is it should > communicate with? > > > > The TCP Connection is already established before the RNIC takes > > control of the connection. Therefore the entire TCP state > is already > > established (source/dest IP Address/Port, negotiated options, EMSS, > > local options, etc.). > > Would "shares the connection" be a better description? > No, it does not "share the connection". After conversion the RDMA stack is in full control of the TCP layer. The host cannot modify anything about the TCP layer, or use its services. For example the host cannot enable Nagle on the TCP connection after it has been turned over to the RNIC. It definetly cannot send or receive streaming data. Because of the nature of the MPA protocol, a TCP connection that has been enabled for MPA can *never* revert to non-MPA mode. It can only be used in MPA mode or terminated. It's the L2 and L3 layer data that is shared (or synchronized) between the host and the RNIC. > I think I get the concepts at least now...this follows what I > was suggesting earlier about having an RNIC support both NAPI > (New API for NICs) (code under drivers/net) and openib.org > Gen2 stack (code in drivers/infiniband). > > thanks, > grant > From jagana at us.ibm.com Fri May 27 10:43:33 2005 From: jagana at us.ibm.com (Venkata Jagana) Date: Fri, 27 May 2005 10:43:33 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux In-Reply-To: <20050527172536.83795.qmail@web51912.mail.yahoo.com> Message-ID: rdma-developers-admin at lists.sourceforge.net wrote on 05/27/2005 10:25:36 AM: > Mike, > I am not sure I do understand what your were trying > to communicate. Let me try and decode this. My basic > point was to respond to Venkata's response about > complete offload without any interaction with the host > system. I disagree with that in its totality as I > think there are dependencies and that needs to be > specified in a formal manner, i.e. in the specs, so > that we do not have multiple proprietary interafaces > which change and application users have to change this > consumption based on individual implementations. > And you just brought up the reasons why what I was > saying seemed to be justfied. Did I read that > correctly? > > Thanks > SG > Hello Sukanta, Yes, there will be interactions with the host system for an RNIC but through an interface something similar to RNIC-PI. The clarification that I was trying to make is that from an RNIC perspective, it doesn't have to expose any ToE interface other than the fact that adapters might be providing multiple services including ToE, RDMA, iSCSI but that's outside the scope of an RNIC interface. Thanks Venkat -------------- next part -------------- An HTML attachment was scrubbed... URL: From sganguly at yahoo.com Fri May 27 10:46:10 2005 From: sganguly at yahoo.com (Sukanta ganguly) Date: Fri, 27 May 2005 10:46:10 -0700 (PDT) Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux In-Reply-To: 6667 Message-ID: <20050527174610.12213.qmail@web51905.mail.yahoo.com> Venkata, I understand what you were saying now. Thanks SG --- Venkata Jagana wrote: > > > > > > rdma-developers-admin at lists.sourceforge.net wrote on > 05/27/2005 10:25:36 > AM: > > > Mike, > > I am not sure I do understand what your were > trying > > to communicate. Let me try and decode this. My > basic > > point was to respond to Venkata's response about > > complete offload without any interaction with the > host > > system. I disagree with that in its totality as I > > think there are dependencies and that needs to be > > specified in a formal manner, i.e. in the specs, > so > > that we do not have multiple proprietary > interafaces > > which change and application users have to change > this > > consumption based on individual implementations. > > And you just brought up the reasons why what I > was > > saying seemed to be justfied. Did I read that > > correctly? > > > > Thanks > > SG > > > > Hello Sukanta, > > Yes, there will be interactions with the host system > for an RNIC > but through an interface something similar to > RNIC-PI. > > The clarification that I was trying to make is that > from an RNIC > perspective, it > doesn't have to expose any ToE interface other than > the fact that adapters > might be providing > multiple services including ToE, RDMA, iSCSI but > that's outside the scope > of an > RNIC interface. > > Thanks > Venkat __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From jagana at us.ibm.com Fri May 27 10:49:36 2005 From: jagana at us.ibm.com (Venkata Jagana) Date: Fri, 27 May 2005 10:49:36 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux In-Reply-To: <8508251A6FC08A489844A94261D3693A057597@fiona.siliquent.com> Message-ID: rdma-developers-admin at lists.sourceforge.net wrote on 05/27/2005 10:40:20 AM: > > On Fri, May 27, 2005 at 09:58:44AM -0700, Caitlin Bestler wrote: > > > > I understand most of what you wrote but am still missing one bit: > > > > How is the RNIC told what the peer IP is it should > > communicate with? > > > > > > The TCP Connection is already established before the RNIC takes > > > control of the connection. Therefore the entire TCP state > > is already > > > established (source/dest IP Address/Port, negotiated options, EMSS, > > > local options, etc.). > > > > Would "shares the connection" be a better description? > > > > No, it does not "share the connection". After conversion the > RDMA stack is in full control of the TCP layer. The host cannot > modify anything about the TCP layer, or use its services. > For example the host cannot enable Nagle on the TCP connection > after it has been turned over to the RNIC. It definetly cannot > send or receive streaming data. Because of the nature of the > MPA protocol, a TCP connection that has been enabled for MPA > can *never* revert to non-MPA mode. It can only be used in MPA > mode or terminated. > Agree that it is never being shared but once the connection has been turned over to the RNIC, would it ever be possible to hand it back to the host or host taking control from the RNIC adapter? Might be helpful for failover situations. Thanks Venkat > It's the L2 and L3 layer data that is shared (or synchronized) > between the host and the RNIC. > > > I think I get the concepts at least now...this follows what I > > was suggesting earlier about having an RNIC support both NAPI > > (New API for NICs) (code under drivers/net) and openib.org > > Gen2 stack (code in drivers/infiniband). > > > > thanks, > > grant > > > > > ------------------------------------------------------- > This SF.Net email is sponsored by Yahoo. > Introducing Yahoo! Search Developer Network - Create apps using Yahoo! > Search APIs Find out how you can build Yahoo! directly into your own > Applications - visit http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005 > _______________________________________________ > Rdma-developers mailing list > Rdma-developers at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/rdma-developers -------------- next part -------------- An HTML attachment was scrubbed... URL: From roland at topspin.com Fri May 27 11:09:46 2005 From: roland at topspin.com (Roland Dreier) Date: Fri, 27 May 2005 11:09:46 -0700 Subject: [Rdma-developers] RE: [openib-general] OpenIB and OpenRDMA: Convergence on commonRDMA APIs and ULPs for Linux In-Reply-To: <8508251A6FC08A489844A94261D3693A05758F@fiona.siliquent.com> (Caitlin Bestler's message of "Fri, 27 May 2005 09:55:05 -0700") References: <8508251A6FC08A489844A94261D3693A05758F@fiona.siliquent.com> Message-ID: <52d5rcfslx.fsf@topspin.com> Caitlin> There isn't enough there to go farther. I think there is. In fact I outlined exactly what I would do if I were working at an iWARP company: Roland> Then someone would have to start implementing a low-level Roland> driver for a specific RNIC, and find which modifications Roland> to the existing verbs are required. For example, I Roland> believe the QP attribute structure passed into the QP Roland> modify verb probably has to become a union containing the Roland> IB attributes and the RNIC attributes. However, most Roland> verbs should work fine with at most trivial modifications. As for which option I was suggesting, it was c: Caitlin> c) The same methods but with struct/enums that have Caitlin> common and transport specific portions? That is doable, Caitlin> in fact that is what RNIC-PI is today. Repeating that Caitlin> work with the gen2 verbs will be time consuming. I don't Caitlin> want to have to wait 4 months to debate the details of Caitlin> this before I can start working on my next generation of Caitlin> verbs. OpenIB chose to focus on getting working code released quickly. OpenRDMA chose to focus on writing architecture documents and API specifications. The results were completely unsurprising. I have outlined what I believe are good and valid reasons why there should not be two verbs layers in the Linux kernel. If you think I'm wrong and that you will be able to have RNIC-PI merged into the Linux kernel alongside the existing IB midlayer, then implementing iWARP support through RNIC-PI is a reasonable way forward. If you believe that the single verbs layer in the kernel should be RNIC-PI, then you should extend RNIC-PI to support the IB features currently supported (eg datagrams), and port the existing IB code to the new API. Once that's done I would certainly be happy to review the changes for merging. However, I don't think any current OpenIB contributors will be willing to do the work to port to RNIC-PI. I see no reason to spend a lot of effort to end up with a result that will be at best equivalent to what we have today and will likely be worse in some real ways (for example, how does RNIC-PI handle adapter hotplug?) If neither of these options appeals to you, then the only alternative left seems to be to work with the OpenIB community to evolve the existing IB midlayer into an RDMA midlayer than can support both IB and iWARP. Without seeing some real patches from the iWARP side, it's hard for me to see any value in continuing to participate in this debate. - R. From tduffy at sun.com Fri May 27 11:17:02 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 27 May 2005 11:17:02 -0700 Subject: [Rdma-developers] RE: [openib-general] OpenIB and OpenRDMA: Convergence on commonRDMA APIs and ULPs for Linux In-Reply-To: <8508251A6FC08A489844A94261D3693A05758F@fiona.siliquent.com> References: <8508251A6FC08A489844A94261D3693A05758F@fiona.siliquent.com> Message-ID: <1117217822.32354.4.camel@duffman> On Fri, 2005-05-27 at 09:55 -0700, Caitlin Bestler wrote: > There isn't enough there to go farther. > > What methods is the RDMA_DEVICE_RNIC required to support? > > a) Totally iWARP specific methods. That works, but all you've done > is taken two stacks and put a switch on top of them. kDAPL already > does that quite well. > b) The same methods, but with different struct and enum definitions? > (i.e., a struct iwarp_qp versus a struct ib_qp). IMHO, that doesn't > differ much from a) > c) The same methods but with struct/enums that have common and transport > specific portions? That is doable, in fact that is what RNIC-PI is > today. Repeating that work with the gen2 verbs will be time > consuming. > I don't want to have to wait 4 months to debate the details of this > before I can start working on my next generation of verbs. > d) Use the methods and structs 'as is' and just somehow figure out how > they map to iWARP? That is no more "transport neutral" than Tom DeLay > is "bipartisan". I think a combination of c) and d) will be the fastest time to kernel. Oh, and please stop waiting and debating. Code speaks much louder than words. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From tduffy at sun.com Fri May 27 11:19:57 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 27 May 2005 11:19:57 -0700 Subject: [Rdma-developers] RE: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <8508251A6FC08A489844A94261D3693A057591@fiona.siliquent.com> References: <8508251A6FC08A489844A94261D3693A057591@fiona.siliquent.com> Message-ID: <1117217997.32354.6.camel@duffman> On Fri, 2005-05-27 at 10:07 -0700, Caitlin Bestler wrote: > OpenIB's announced charter was IB specific. There was no > announcement that the intent was to define *all* RDMA > services. The OpenIB board will be voting, maybe even at the next meeting, whether to expand OpenIB's charter to include *all* RDMA services. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From Thomas.Talpey at netapp.com Fri May 27 11:24:05 2005 From: Thomas.Talpey at netapp.com (Talpey, Thomas) Date: Fri, 27 May 2005 14:24:05 -0400 Subject: [Rdma-developers] RE: [openib-general] OpenIB and OpenRDMA: Convergence on commonRDMA APIs and ULPs for Linux In-Reply-To: <52d5rcfslx.fsf@topspin.com> References: <8508251A6FC08A489844A94261D3693A05758F@fiona.siliquent.com> <52d5rcfslx.fsf@topspin.com> Message-ID: <6.2.1.2.2.20050527142051.041151b0@exnane01.nane.netapp.com> I usually try to avoid sending messages like the following, but... Well said, Roland. TTKO - Time To Kernel dot Org. I for one want iWARP there along with IB. Before 2007. Before 2006! Tom. At 02:09 PM 5/27/2005, Roland Dreier wrote: > Caitlin> There isn't enough there to go farther. > >I think there is. In fact I outlined exactly what I would do if I >were working at an iWARP company: > > Roland> Then someone would have to start implementing a low-level > Roland> driver for a specific RNIC, and find which modifications > Roland> to the existing verbs are required. For example, I > Roland> believe the QP attribute structure passed into the QP > Roland> modify verb probably has to become a union containing the > Roland> IB attributes and the RNIC attributes. However, most > Roland> verbs should work fine with at most trivial modifications. > >As for which option I was suggesting, it was c: > > Caitlin> c) The same methods but with struct/enums that have > Caitlin> common and transport specific portions? That is doable, > Caitlin> in fact that is what RNIC-PI is today. Repeating that > Caitlin> work with the gen2 verbs will be time consuming. I don't > Caitlin> want to have to wait 4 months to debate the details of > Caitlin> this before I can start working on my next generation of > Caitlin> verbs. > >OpenIB chose to focus on getting working code released quickly. >OpenRDMA chose to focus on writing architecture documents and API >specifications. The results were completely unsurprising. > >I have outlined what I believe are good and valid reasons why there >should not be two verbs layers in the Linux kernel. If you think I'm >wrong and that you will be able to have RNIC-PI merged into the Linux >kernel alongside the existing IB midlayer, then implementing iWARP >support through RNIC-PI is a reasonable way forward. > >If you believe that the single verbs layer in the kernel should be >RNIC-PI, then you should extend RNIC-PI to support the IB features >currently supported (eg datagrams), and port the existing IB code to >the new API. Once that's done I would certainly be happy to review >the changes for merging. However, I don't think any current OpenIB >contributors will be willing to do the work to port to RNIC-PI. I see >no reason to spend a lot of effort to end up with a result that will >be at best equivalent to what we have today and will likely be worse >in some real ways (for example, how does RNIC-PI handle adapter hotplug?) > >If neither of these options appeals to you, then the only alternative >left seems to be to work with the OpenIB community to evolve the >existing IB midlayer into an RDMA midlayer than can support both IB >and iWARP. > >Without seeing some real patches from the iWARP side, it's hard for me >to see any value in continuing to participate in this debate. > > - R. >_______________________________________________ >openib-general mailing list >openib-general at openib.org >http://openib.org/mailman/listinfo/openib-general > >To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From tduffy at sun.com Fri May 27 11:27:31 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 27 May 2005 11:27:31 -0700 Subject: [openib-general] OpenSM crash Message-ID: <1117218451.32354.10.camel@duffman> I just noticed that my opensm had segv'ed and dumped core. Here is the gdb backtrace. #0 stack_dump () at src/stack.c:72 72 if (!__builtin_frame_address(2)) (gdb) bt #0 stack_dump () at src/stack.c:72 #1 0x00002aaaaabb71a6 in handler (x=11) at src/stack.c:151 #2 #3 osm_dump_dr_smp (p_log=0x552498, p_smp=0x0, log_level=32 ' ') at osm_helper.c:1446 #4 0x000000000042eed1 in __osm_vl15_poller (p_ptr=0x552498) at osm_madw.h:575 #5 0x00002aaaaadc911e in __cl_thread_wrapper (arg=0x0) at cl_thread.c:61 #6 0x00000036d28060aa in start_thread () from /lib64/tls/libpthread.so.0 #7 0x00000036d19c53d3 in clone () from /lib64/tls/libc.so.6 #8 0x0000000000000000 in ?? () -tduffy -- I wish we lived in the America of yesteryear that only exists in the minds of us Republicans. -- Ned Flanders -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From tduffy at sun.com Fri May 27 11:31:32 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 27 May 2005 11:31:32 -0700 Subject: [openib-general] OpenSM crash In-Reply-To: <1117218451.32354.10.camel@duffman> References: <1117218451.32354.10.camel@duffman> Message-ID: <1117218692.32354.13.camel@duffman> On Fri, 2005-05-27 at 11:27 -0700, Tom Duffy wrote: > I just noticed that my opensm had segv'ed and dumped core. BTW, here was the tail of the osm.log: May 27 01:44:09 [43005960] -> osm_vendor_get: [ May 27 01:44:09 [43806960] -> __osm_vl15_poller: Servicing p_madw = 0x5678f0 (mad 0x5f33f0 req 1) May 27 01:44:09 [43005960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x567908, size = 256. May 27 01:44:09 [43005960] -> osm_vendor_get: Acquired UMAD 0x5f3640, size = 256. May 27 01:44:09 [43005960] -> osm_vendor_get: ] May 27 01:44:09 [43005960] -> osm_mad_pool_get: Acquired p_madw = 0x5678f0, p_mad = 0x5f3670, size = 256. May 27 01:44:09 [43005960] -> osm_mad_pool_get: ] May 27 01:44:09 [43005960] -> osm_req_get: Getting P_KeyTable (0x16), modifier = 0x10001, TID = 0x1c149. May 27 01:44:09 [43005960] -> osm_vl15_post: [ May 27 01:44:09 [43005960] -> osm_vl15_post: Servicing p_madw = 0x5678f0 (mad 0x5f3670 req 1) May 27 01:44:09 [43005960] -> osm_vl15_post: 4294967295 MADs on wire, 2 MADs outstanding. May 27 01:44:09 [43005960] -> osm_vl15_poll: [ May 27 01:44:09 [43005960] -> osm_vl15_poll: Signalling poller thread. May 27 01:44:09 [43005960] -> osm_vl15_poll: ] May 27 01:44:09 [43005960] -> osm_vl15_post: ] May 27 01:44:09 [43005960] -> osm_req_get: ] May 27 01:44:09 [43005960] -> osm_physp_has_pkey: ] May 27 01:44:09 [43005960] -> __osm_pi_rcv_get_pkey_slvl_vla_tables: ] May 27 01:44:09 [43005960] -> osm_pi_rcv_process: ] May 27 01:44:09 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [ -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From caitlinb at siliquent.com Fri May 27 11:46:30 2005 From: caitlinb at siliquent.com (Caitlin Bestler) Date: Fri, 27 May 2005 11:46:30 -0700 Subject: [Rdma-developers] RE: [openib-general] OpenIB and OpenRDMA:Convergence on common RDMAAPIs and ULPs for Linux Message-ID: <8508251A6FC08A489844A94261D3693A057598@fiona.siliquent.com> That's great. I hope it is approved. But lacking a time machine that vote won't motivate iWARP vendors to have started working with OpenIB a year ago -- which was the immediate context of my comment. > -----Original Message----- > From: Tom Duffy [mailto:tduffy at sun.com] > Sent: Friday, May 27, 2005 11:20 AM > To: Caitlin Bestler > Cc: rdma-developers at lists.sourceforge.net; openib-general at openib.org > Subject: RE: [Rdma-developers] RE: [openib-general] OpenIB > and OpenRDMA:Convergence on common RDMAAPIs and ULPs for Linux > > On Fri, 2005-05-27 at 10:07 -0700, Caitlin Bestler wrote: > > OpenIB's announced charter was IB specific. There was no > announcement > > that the intent was to define *all* RDMA services. > > The OpenIB board will be voting, maybe even at the next > meeting, whether to expand OpenIB's charter to include *all* > RDMA services. > > -tduffy > From halr at voltaire.com Fri May 27 11:54:19 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 27 May 2005 14:54:19 -0400 Subject: [openib-general] OpenSM crash In-Reply-To: <1117218451.32354.10.camel@duffman> References: <1117218451.32354.10.camel@duffman> Message-ID: <1117220005.4477.21.camel@erez-s.us.voltaire.com> On Fri, 2005-05-27 at 14:27, Tom Duffy wrote: > I just noticed that my opensm had segv'ed and dumped core. Here is the > gdb backtrace. > > #0 stack_dump () at src/stack.c:72 > 72 if (!__builtin_frame_address(2)) > (gdb) bt > #0 stack_dump () at src/stack.c:72 > #1 0x00002aaaaabb71a6 in handler (x=11) at src/stack.c:151 > #2 Looks like osm_dump_dr_smp was called with a NULL p_smp so: osm_madw_get_smp_ptr(p_madw) returned NULL for some unknown reason and that is an unexpected (should not occur) condition. > #3 osm_dump_dr_smp (p_log=0x552498, p_smp=0x0, log_level=32 ' ') > at osm_helper.c:1446 > #4 0x000000000042eed1 in __osm_vl15_poller (p_ptr=0x552498) at osm_madw.h:575 > #5 0x00002aaaaadc911e in __cl_thread_wrapper (arg=0x0) at cl_thread.c:61 > #6 0x00000036d28060aa in start_thread () from /lib64/tls/libpthread.so.0 > #7 0x00000036d19c53d3 in clone () from /lib64/tls/libc.so.6 > #8 0x0000000000000000 in ?? () Anything "special" about your configuration/what was going on ? Can you reproduce this ? Also, what version of OpenSM are you using ? -- Hal From halr at voltaire.com Fri May 27 12:07:59 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 27 May 2005 15:07:59 -0400 Subject: [openib-general] mthca_cq compile warning with MTHCA_DEBUG off Message-ID: <1117220879.4477.25.camel@erez-s.us.voltaire.com> drivers/infiniband/hw/mthca/mthca_cq.c: In function `dump_cqe': drivers/infiniband/hw/mthca/mthca_cq.c:176: warning: unused variable `cqe' From tduffy at sun.com Fri May 27 12:26:42 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 27 May 2005 12:26:42 -0700 Subject: [openib-general] OpenSM crash In-Reply-To: <1117220005.4477.21.camel@erez-s.us.voltaire.com> References: <1117218451.32354.10.camel@duffman> <1117220005.4477.21.camel@erez-s.us.voltaire.com> Message-ID: <1117222002.24117.7.camel@duffman> On Fri, 2005-05-27 at 14:54 -0400, Hal Rosenstock wrote: > Anything "special" about your configuration/what was going on ? This was in the middle of the night. I wasn't doing anything to the systems at the time. > Can you reproduce this ? nope. > Also, what version of OpenSM are you using ? It was pretty close to the head of the tree, although a couple of files were updated when I did a svn update after the crash. -tduffy -- I wish we lived in the America of yesteryear that only exists in the minds of us Republicans. -- Ned Flanders -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From roland at topspin.com Fri May 27 12:27:39 2005 From: roland at topspin.com (Roland Dreier) Date: Fri, 27 May 2005 12:27:39 -0700 Subject: [openib-general] Re: mthca_cq compile warning with MTHCA_DEBUG off In-Reply-To: <1117220879.4477.25.camel@erez-s.us.voltaire.com> (Hal Rosenstock's message of "27 May 2005 15:07:59 -0400") References: <1117220879.4477.25.camel@erez-s.us.voltaire.com> Message-ID: <52y8a0eafo.fsf@topspin.com> Thanks, fixed now. - R. From krause at cup.hp.com Fri May 27 12:48:19 2005 From: krause at cup.hp.com (Michael Krause) Date: Fri, 27 May 2005 12:48:19 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMA APIs and ULPs for Linux In-Reply-To: <20050527172536.83795.qmail@web51912.mail.yahoo.com> References: <6.2.0.14.2.20050527071821.02071338@esmail.cup.hp.com> <20050527172536.83795.qmail@web51912.mail.yahoo.com> Message-ID: <6.2.0.14.2.20050527124647.02825be0@esmail.cup.hp.com> At 10:25 AM 5/27/2005, Sukanta ganguly wrote: >Mike, > I am not sure I do understand what your were trying >to communicate. Let me try and decode this. My basic >point was to respond to Venkata's response about >complete offload without any interaction with the host >system. I disagree with that in its totality as I >think there are dependencies and that needs to be >specified in a formal manner, i.e. in the specs, so >that we do not have multiple proprietary interafaces >which change and application users have to change this >consumption based on individual implementations. > And you just brought up the reasons why what I was >saying seemed to be justfied. Did I read that >correctly? You understand correctly. We need defined interface points where other subsystems can choose to interact with the RDMA infrastructure. Further, there is a set of issues that have yet to be addressed, e.g. that go to how a customer uses / manages the attached fabrics and how their tool chain is impacted or not. Mike >Thanks >SG > >--- Michael Krause wrote: > > > At 06:40 AM 5/27/2005, Sukanta ganguly wrote: > > >Venkata, > > > How will that work? If the RNIC offloads RDMA > > and > > >TCP completely from the Operating System and does > > not > > >share any state information then the application > > >running on the host will never be in the position > > to > > >utilize the socket interface to use the > > communication > > >logic to send and receive data between the remote > > node > > >and itself. Some information needs to be shared. > > How > > >much of it and what exactly needs to be shared is > > the > > >question. > > > > Ok. It all depends upon what level of integration / > > interaction a TOE and > > thus a RNIC will have with the host network stack. > > For example, if a > > customer wants to have TCP and IP stats kept for the > > off-loaded stack even > > if it is just being using for RDMA, then there needs > > to be a method defined > > to consolidate these stats back into the host > > network stack tool > > chain. Similarly, if one wants to maintain a single > > routing table to > > manage, etc. on the host, then the RNIC needs to > > access / update that > > information accordingly. One can progress through > > other aspects of > > integration, e.g. connection management, security > > interactions (e.g. DOS > > protection), and so forth. What is exposed again > > depends upon the level of > > integration and how customers want to manage their > > services. This problem > > also exists for IB but most people have not thought > > about this from a > > customer perspective and how to integrate the IB > > semantics into the way > > customers manage their infrastructures, do billing, > > etc. For some > > environments, they simply do not care but if IB is > > to be used in the > > enterprise space, then some thought will be required > > here since most IT > > don't see anything as being "free" or self-managed. > > > > Again, Sockets is an application API and not how one > > communicates to a TOE > > or RDMA component. The RNIC PI has been proposed as > > an interface to the > > RDMA functionality. The PI supports all of the > > iWARP and IB v 1.2 verbs. > > > > Mike > > > > > > >Thanks > > >SG > > > > > >--- Venkata Jagana wrote: > > > > > > > > > > > > > > > > > > > > > > > > > > > > rdma-developers-admin at lists.sourceforge.net > > wrote on > > > > 05/25/2005 09:47:00 > > > > PM: > > > > > > > > > Venkata, > > > > > Interesting coincidence: I was talking with > > > > someone (at HP) today > > > > > who knows substantially more than I do about > > > > RNICs. > > > > > They indicated RNICs need to manage TCP state > > on > > > > the card from userspace. > > > > > I suspect that's only possible through a > > private > > > > interface > > > > > (e.g. ioctl() or /proc) or the non-existant > > (in > > > > kernel.org) > > > > > TOE implementation. Is this correct? > > > > > > > > > > > > > Not correct. > > > > > > > > Since RNICs are offloaded adapters with RDMA > > > > protocols layered on > > > > top of TCP stack, they do maintain the TCP state > > > > internally but > > > > it does not expose to the host. RNIC expose only > > > > RNIC Verbs interface > > > > to the host bot not TOE interface. > > > > > > > > Thanks > > > > Venkat > > > > > > > > > > > > > > hth, > > > > > grant > > > > > > > > > > > > > > > > > > > > > > >------------------------------------------------------- > > > > > SF.Net email is sponsored by: GoToMeeting - > > the > > > > easiest way to > > > > collaborate > > > > > online with coworkers and clients while > > avoiding > > > > the high cost of travel > > > > and > > > > > communications. There is no equipment to buy > > and > > > > you can meet as often as > > > > > you want. Try it > > > > > > > >free.http://ads.osdn.com/?ad_id=7402&alloc_id=16135&op=click > > > > > > > _______________________________________________ > > > > > Rdma-developers mailing list > > > > > Rdma-developers at lists.sourceforge.net > > > > > > > >https://lists.sourceforge.net/lists/listinfo/rdma-developers > > > > > >__________________________________________________ > > >Do You Yahoo!? > > >Tired of spam? Yahoo! Mail has the best spam > > protection around > > >http://mail.yahoo.com > > > > > > > > > >------------------------------------------------------- > > >This SF.Net email is sponsored by Yahoo. > > >Introducing Yahoo! Search Developer Network - > > Create apps using Yahoo! > > >Search APIs Find out how you can build Yahoo! > > directly into your own > > >Applications - visit > > >http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005 > > >_______________________________________________ > > >Rdma-developers mailing list > > >Rdma-developers at lists.sourceforge.net > > > >https://lists.sourceforge.net/lists/listinfo/rdma-developers > > > > >__________________________________________________ >Do You Yahoo!? >Tired of spam? Yahoo! Mail has the best spam protection around >http://mail.yahoo.com > > >------------------------------------------------------- >This SF.Net email is sponsored by Yahoo. >Introducing Yahoo! Search Developer Network - Create apps using Yahoo! >Search APIs Find out how you can build Yahoo! directly into your own >Applications - visit http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005 >_______________________________________________ >Rdma-developers mailing list >Rdma-developers at lists.sourceforge.net >https://lists.sourceforge.net/lists/listinfo/rdma-developers -------------- next part -------------- An HTML attachment was scrubbed... URL: From halr at voltaire.com Fri May 27 13:25:28 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 27 May 2005 16:25:28 -0400 Subject: [openib-general] OpenSM crash In-Reply-To: <1117222002.24117.7.camel@duffman> References: <1117218451.32354.10.camel@duffman> <1117220005.4477.21.camel@erez-s.us.voltaire.com> <1117222002.24117.7.camel@duffman> Message-ID: <1117225528.4479.14.camel@hal.voltaire.com> On Fri, 2005-05-27 at 15:26, Tom Duffy wrote: > > Also, what version of OpenSM are you using ? > > It was pretty close to the head of the tree, although a couple of files > were updated when I did a svn update after the crash. When was your last update of OpenSM ? Was it after Tues AM ? Also, did you pick up the user_mad.c fix on Tuesday AM ? If it was, any other changes are either not related or trivial. After you picked up these changes, did you regenerate the various OpenSM makefiles (a define for RMPP changed in them) or just rebuild ? [This would not explain the crash, but is different from how my OpenSM is built.] -- Hal From tduffy at sun.com Fri May 27 13:31:11 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 27 May 2005 13:31:11 -0700 Subject: [openib-general] [PATCH] kDAPL: remove typedef DAT_CONTEXT Message-ID: <1117225871.24117.11.camel@duffman> Get rid of the typedef DAT_CONTEXT. Signed-off-by: Tom Duffy Index: linux-kernel/test/dapltest/include/dapl_common.h =================================================================== --- linux-kernel/test/dapltest/include/dapl_common.h (revision 2506) +++ linux-kernel/test/dapltest/include/dapl_common.h (working copy) @@ -42,7 +42,7 @@ typedef enum typedef struct { DAT_RMR_CONTEXT rmr_context; - DAT_CONTEXT mem_address; + union dat_context mem_address; } RemoteMemoryInfo; #pragma pack() Index: linux-kernel/dat-provider/dapl_get_consumer_context.c =================================================================== --- linux-kernel/dat-provider/dapl_get_consumer_context.c (revision 2506) +++ linux-kernel/dat-provider/dapl_get_consumer_context.c (working copy) @@ -48,7 +48,7 @@ * DAT_SUCCESS * DAT_INVALID_PARAMETER */ -u32 dapl_get_consumer_context(DAT_HANDLE dat_handle, DAT_CONTEXT *context) +u32 dapl_get_consumer_context(DAT_HANDLE dat_handle, union dat_context *context) { u32 dat_status = DAT_SUCCESS; struct dapl_header *header; Index: linux-kernel/dat-provider/dapl_set_consumer_context.c =================================================================== --- linux-kernel/dat-provider/dapl_set_consumer_context.c (revision 2506) +++ linux-kernel/dat-provider/dapl_set_consumer_context.c (working copy) @@ -47,7 +47,7 @@ * DAT_SUCCESS * DAT_INVALID_HANDLE */ -u32 dapl_set_consumer_context(DAT_HANDLE dat_handle, DAT_CONTEXT context) +u32 dapl_set_consumer_context(DAT_HANDLE dat_handle, union dat_context context) { u32 dat_status = DAT_SUCCESS; struct dapl_header *header; Index: linux-kernel/dat-provider/dapl.h =================================================================== --- linux-kernel/dat-provider/dapl.h (revision 2506) +++ linux-kernel/dat-provider/dapl.h (working copy) @@ -177,7 +177,7 @@ struct dapl_header { enum dat_handle_type handle_type; struct dapl_ia *owner_ia; struct dapl_llist_entry ia_list_entry; - DAT_CONTEXT user_context; /* user context - opaque to DAPL */ + union dat_context user_context; /* user context - opaque to DAPL */ spinlock_t lock; unsigned long flags; /* saved lock flag values */ }; @@ -423,9 +423,11 @@ extern u32 dapl_ia_query(DAT_IA_HANDLE, /* helper functions */ -extern u32 dapl_set_consumer_context(DAT_HANDLE handle, DAT_CONTEXT context); +extern u32 dapl_set_consumer_context(DAT_HANDLE handle, + union dat_context context); -extern u32 dapl_get_consumer_context(DAT_HANDLE handle, DAT_CONTEXT *context); +extern u32 dapl_get_consumer_context(DAT_HANDLE handle, + union dat_context *context); extern u32 dapl_get_handle_type(DAT_HANDLE handle, enum dat_handle_type *type); Index: linux-kernel/dat/dat.h =================================================================== --- linux-kernel/dat/dat.h (revision 2506) +++ linux-kernel/dat/dat.h (working copy) @@ -361,14 +361,14 @@ typedef enum { TRUE = 1 } boolean_t; -typedef union dat_context { +union dat_context { void *as_ptr; u64 as_64; unsigned long long as_index; -} DAT_CONTEXT; +}; -typedef DAT_CONTEXT DAT_DTO_COOKIE; -typedef DAT_CONTEXT DAT_RMR_COOKIE; +typedef union dat_context DAT_DTO_COOKIE; +typedef union dat_context DAT_RMR_COOKIE; enum dat_completion_flags { /* Completes with notification */ @@ -920,13 +920,6 @@ struct dat_upcall_object { DAT_UPCALL_FUNC upcall_func; }; -/* Define NULL upcall */ - -#define DAT_UPCALL_NULL \ - ((struct dat_upcall_object) { (void *) NULL, (DAT_UPCALL_FUNC) NULL }) - -#define DAT_UPCALL_SAME ((struct dat_upcall_object *) NULL) - /* Provider/registration info */ struct dat_provider_info { @@ -1098,9 +1091,9 @@ typedef u32 (*DAT_IA_QUERY_FUNC)(DAT_IA_ struct dat_ia_attr *, struct dat_provider_attr *); -typedef u32 (*DAT_SET_CONSUMER_CONTEXT_FUNC)(DAT_HANDLE, DAT_CONTEXT); +typedef u32 (*DAT_SET_CONSUMER_CONTEXT_FUNC)(DAT_HANDLE, union dat_context); -typedef u32 (*DAT_GET_CONSUMER_CONTEXT_FUNC)(DAT_HANDLE, DAT_CONTEXT *); +typedef u32 (*DAT_GET_CONSUMER_CONTEXT_FUNC)(DAT_HANDLE, union dat_context *); typedef u32 (*DAT_GET_HANDLE_TYPE_FUNC)(DAT_HANDLE, enum dat_handle_type *); @@ -1387,14 +1380,14 @@ extern u32 dat_registry_list_providers(i DAT_HANDLE_TO_PROVIDER(handle)->func(handle, ##__VA_ARGS__) static inline u32 dat_set_consumer_context(DAT_HANDLE handle, - DAT_CONTEXT context) + union dat_context context) { return DAT_CALL_PROVIDER_FUNC(set_consumer_context_func, handle, context); } static inline u32 dat_get_consumer_context(DAT_HANDLE handle, - DAT_CONTEXT *context) + union dat_context *context) { return DAT_CALL_PROVIDER_FUNC(get_consumer_context_func, handle, context); From halr at voltaire.com Fri May 27 14:15:40 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 27 May 2005 17:15:40 -0400 Subject: [openib-general] OpenSM crash In-Reply-To: <1117218692.32354.13.camel@duffman> References: <1117218451.32354.10.camel@duffman> <1117218692.32354.13.camel@duffman> Message-ID: <1117228540.4479.140.camel@hal.voltaire.com> On Fri, 2005-05-27 at 14:31, Tom Duffy wrote: > On Fri, 2005-05-27 at 11:27 -0700, Tom Duffy wrote: > > I just noticed that my opensm had segv'ed and dumped core. > > BTW, here was the tail of the osm.log: > > May 27 01:44:09 [43005960] -> osm_vendor_get: [ > May 27 01:44:09 [43806960] -> __osm_vl15_poller: Servicing p_madw = 0x5678f0 (mad 0x5f33f0 req 1) > May 27 01:44:09 [43005960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x567908, size = 256. > May 27 01:44:09 [43005960] -> osm_vendor_get: Acquired UMAD 0x5f3640, size = 256. > May 27 01:44:09 [43005960] -> osm_vendor_get: ] > May 27 01:44:09 [43005960] -> osm_mad_pool_get: Acquired p_madw = 0x5678f0, p_mad = 0x5f3670, size = 256. > May 27 01:44:09 [43005960] -> osm_mad_pool_get: ] > May 27 01:44:09 [43005960] -> osm_req_get: Getting P_KeyTable (0x16), modifier = 0x10001, TID = 0x1c149. > May 27 01:44:09 [43005960] -> osm_vl15_post: [ > May 27 01:44:09 [43005960] -> osm_vl15_post: Servicing p_madw = 0x5678f0 (mad 0x5f3670 req 1) > May 27 01:44:09 [43005960] -> osm_vl15_post: 4294967295 MADs on wire, 2 MADs outstanding. ^^^^^^^^^^ This looks weird. > May 27 01:44:09 [43005960] -> osm_vl15_poll: [ > May 27 01:44:09 [43005960] -> osm_vl15_poll: Signalling poller thread. > May 27 01:44:09 [43005960] -> osm_vl15_poll: ] > May 27 01:44:09 [43005960] -> osm_vl15_post: ] > May 27 01:44:09 [43005960] -> osm_req_get: ] > May 27 01:44:09 [43005960] -> osm_physp_has_pkey: ] > May 27 01:44:09 [43005960] -> __osm_pi_rcv_get_pkey_slvl_vla_tables: ] > May 27 01:44:09 [43005960] -> osm_pi_rcv_process: ] > May 27 01:44:09 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [ Wonder why __osm_sm_mad_ctrl_disp_done_callback wasn't on the stack shown in the previous email as this makes it look like it should be. Could you go back a little further in the log ? I'd like to see what is before the start of __osm_pi_rcv_get_pkey_slvl_vla_tables and osm_pi_rcv_process. It's also seems weird to me that there is no other log message between these two. >From the stack trace: #3 osm_dump_dr_smp (p_log=0x552498, p_smp=0x0, log_level=32 ' ') at osm_helper.c:1446 #4 0x000000000042eed1 in __osm_vl15_poller (p_ptr=0x552498) at osm_madw.h:575 It looks like OpenSM was in osm_vl15intf.c::__osm_vl15_poller if( p_madw != (osm_madw_t*)cl_qlist_end( p_fifo ) ) { if( osm_log_is_active( p_vl->p_log, OSM_LOG_DEBUG ) ) { osm_log( p_vl->p_log, OSM_LOG_DEBUG, "__osm_vl15_poller: " "Servicing p_madw = %p (mad %p req %d)\n", p_madw, p_madw->p_mad, p_madw->resp_expected); } if( osm_log_is_active( p_vl->p_log, OSM_LOG_FRAMES ) ) { osm_dump_dr_smp( p_vl->p_log, osm_madw_get_smp_ptr( p_madw ), OSM_LOG_FRAMES ); <=== here } when it died but I didn't see the previous log message in the code "osm_vl15_poller: Servicing p_madw" which I also would have expected. [This would have been telling as p_madw->p_mad would have been logged]. I also didn't see the __osm_vl15_poller entry message either. -- Hal From halr at voltaire.com Fri May 27 14:24:41 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 27 May 2005 17:24:41 -0400 Subject: [openib-general] OpenSM crash In-Reply-To: <1117228540.4479.140.camel@hal.voltaire.com> References: <1117218451.32354.10.camel@duffman> <1117218692.32354.13.camel@duffman> <1117228540.4479.140.camel@hal.voltaire.com> Message-ID: <1117229081.4479.161.camel@hal.voltaire.com> On Fri, 2005-05-27 at 17:15, Hal Rosenstock wrote: > > May 27 01:44:09 [43005960] -> osm_vl15_post: 4294967295 MADs on wire, 2 MADs outstanding. > ^^^^^^^^^^ > This looks weird. I take that back. That's just a lot of MADs have been sent (on the IB wire). OpenSM was probably up and running for a while... -- Hal From tduffy at sun.com Fri May 27 14:30:05 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 27 May 2005 14:30:05 -0700 Subject: [openib-general] OpenSM crash In-Reply-To: <1117225528.4479.14.camel@hal.voltaire.com> References: <1117218451.32354.10.camel@duffman> <1117220005.4477.21.camel@erez-s.us.voltaire.com> <1117222002.24117.7.camel@duffman> <1117225528.4479.14.camel@hal.voltaire.com> Message-ID: <1117229405.16429.2.camel@duffman> On Fri, 2005-05-27 at 16:25 -0400, Hal Rosenstock wrote: > On Fri, 2005-05-27 at 15:26, Tom Duffy wrote: > > > Also, what version of OpenSM are you using ? > > > > It was pretty close to the head of the tree, although a couple of files > > were updated when I did a svn update after the crash. > > When was your last update of OpenSM ? Was it after Tues AM ? To be honest, I can't remember. > Also, did > you pick up the user_mad.c fix on Tuesday AM ? If it was, any other > changes are either not related or trivial. > > After you picked up these changes, did you regenerate the various OpenSM > makefiles (a define for RMPP changed in them) or just rebuild ? [This > would not explain the crash, but is different from how my OpenSM is > built.] I just reran make from the toplevel (management) after updating. I would think it would rebuild them if something changed, no? -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From tduffy at sun.com Fri May 27 14:31:48 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 27 May 2005 14:31:48 -0700 Subject: [openib-general] OpenSM crash In-Reply-To: <1117228540.4479.140.camel@hal.voltaire.com> References: <1117218451.32354.10.camel@duffman> <1117218692.32354.13.camel@duffman> <1117228540.4479.140.camel@hal.voltaire.com> Message-ID: <1117229508.16429.5.camel@duffman> On Fri, 2005-05-27 at 17:15 -0400, Hal Rosenstock wrote: > On Fri, 2005-05-27 at 14:31, Tom Duffy wrote: > > On Fri, 2005-05-27 at 11:27 -0700, Tom Duffy wrote: > > > I just noticed that my opensm had segv'ed and dumped core. > > > > BTW, here was the tail of the osm.log: > > > > May 27 01:44:09 [43005960] -> osm_vendor_get: [ > > May 27 01:44:09 [43806960] -> __osm_vl15_poller: Servicing p_madw = 0x5678f0 (mad 0x5f33f0 req 1) > > May 27 01:44:09 [43005960] -> osm_vendor_get: Acquiring UMAD for p_madw = 0x567908, size = 256. > > May 27 01:44:09 [43005960] -> osm_vendor_get: Acquired UMAD 0x5f3640, size = 256. > > May 27 01:44:09 [43005960] -> osm_vendor_get: ] > > May 27 01:44:09 [43005960] -> osm_mad_pool_get: Acquired p_madw = 0x5678f0, p_mad = 0x5f3670, size = 256. > > May 27 01:44:09 [43005960] -> osm_mad_pool_get: ] > > May 27 01:44:09 [43005960] -> osm_req_get: Getting P_KeyTable (0x16), modifier = 0x10001, TID = 0x1c149. > > May 27 01:44:09 [43005960] -> osm_vl15_post: [ > > May 27 01:44:09 [43005960] -> osm_vl15_post: Servicing p_madw = 0x5678f0 (mad 0x5f3670 req 1) > > May 27 01:44:09 [43005960] -> osm_vl15_post: 4294967295 MADs on wire, 2 MADs outstanding. > ^^^^^^^^^^ > This looks weird. > > > May 27 01:44:09 [43005960] -> osm_vl15_poll: [ > > May 27 01:44:09 [43005960] -> osm_vl15_poll: Signalling poller thread. > > May 27 01:44:09 [43005960] -> osm_vl15_poll: ] > > May 27 01:44:09 [43005960] -> osm_vl15_post: ] > > May 27 01:44:09 [43005960] -> osm_req_get: ] > > May 27 01:44:09 [43005960] -> osm_physp_has_pkey: ] > > May 27 01:44:09 [43005960] -> __osm_pi_rcv_get_pkey_slvl_vla_tables: ] > > May 27 01:44:09 [43005960] -> osm_pi_rcv_process: ] > > May 27 01:44:09 [43005960] -> __osm_sm_mad_ctrl_disp_done_callback: [ > > Wonder why __osm_sm_mad_ctrl_disp_done_callback wasn't on the stack > shown in the previous email as this makes it look like it should be. > > Could you go back a little further in the log ? I'd like to see what is > before the start of __osm_pi_rcv_get_pkey_slvl_vla_tables and > osm_pi_rcv_process. The log had grown to almost 1G, so I actually deleted it. Shit, sorry. > It's also seems weird to me that there is no other > log message between these two. > > >From the stack trace: > #3 osm_dump_dr_smp (p_log=0x552498, p_smp=0x0, log_level=32 ' ') > at osm_helper.c:1446 > #4 0x000000000042eed1 in __osm_vl15_poller (p_ptr=0x552498) at > osm_madw.h:575 > > It looks like OpenSM was in osm_vl15intf.c::__osm_vl15_poller > > if( p_madw != (osm_madw_t*)cl_qlist_end( p_fifo ) ) > { > if( osm_log_is_active( p_vl->p_log, OSM_LOG_DEBUG ) ) > { > osm_log( p_vl->p_log, OSM_LOG_DEBUG, > "__osm_vl15_poller: " > "Servicing p_madw = %p (mad %p req %d)\n", > p_madw, p_madw->p_mad, p_madw->resp_expected); > } > > if( osm_log_is_active( p_vl->p_log, OSM_LOG_FRAMES ) ) > { > osm_dump_dr_smp( p_vl->p_log, > osm_madw_get_smp_ptr( p_madw ), OSM_LOG_FRAMES ); <=== here > } > > when it died but I didn't see the previous log message in the code > "osm_vl15_poller: Servicing p_madw" which I also would have expected. > [This would have been telling as p_madw->p_mad would have been logged]. > I also didn't see the __osm_vl15_poller entry message either. well, if it segv'ed maybe it never finished writing out to the file... -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From roland at topspin.com Fri May 27 14:33:13 2005 From: roland at topspin.com (Roland Dreier) Date: Fri, 27 May 2005 14:33:13 -0700 Subject: [openib-general] OpenSM crash In-Reply-To: <1117229081.4479.161.camel@hal.voltaire.com> (Hal Rosenstock's message of "27 May 2005 17:24:41 -0400") References: <1117218451.32354.10.camel@duffman> <1117218692.32354.13.camel@duffman> <1117228540.4479.140.camel@hal.voltaire.com> <1117229081.4479.161.camel@hal.voltaire.com> Message-ID: <52k6lke4me.fsf@topspin.com> > May 27 01:44:09 [43005960] -> osm_vl15_post: 4294967295 MADs on wire, 2 MADs outstanding. Hal> I take that back. That's just a lot of MADs have been sent Hal> (on the IB wire). OpenSM was probably up and running for a Hal> while... I find it hard to believe that OpenSM has sent 4 billion MADs -- that's more than 1000 MADs a second for a solid month. It also looks very suspicious that the value is equal to ((unsigned int) -1). - R. From halr at voltaire.com Fri May 27 14:37:10 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 27 May 2005 17:37:10 -0400 Subject: [openib-general] OpenSM crash In-Reply-To: <52k6lke4me.fsf@topspin.com> References: <1117218451.32354.10.camel@duffman> <1117218692.32354.13.camel@duffman> <1117228540.4479.140.camel@hal.voltaire.com> <1117229081.4479.161.camel@hal.voltaire.com> <52k6lke4me.fsf@topspin.com> Message-ID: <1117229753.4479.190.camel@hal.voltaire.com> On Fri, 2005-05-27 at 17:33, Roland Dreier wrote: > > May 27 01:44:09 [43005960] -> osm_vl15_post: 4294967295 MADs on wire, 2 MADs outstanding. > > Hal> I take that back. That's just a lot of MADs have been sent > Hal> (on the IB wire). OpenSM was probably up and running for a > Hal> while... > > I find it hard to believe that OpenSM has sent 4 billion MADs -- > that's more than 1000 MADs a second for a solid month. It also looks > very suspicious that the value is equal to ((unsigned int) -1). ^^^^^^^^^^^^^^^^^^ on a 32 bit machine. Good point. The fact that it gets to -1 is significant as I think that is used as a magic value for some computations. -- Hal From tduffy at sun.com Fri May 27 14:52:26 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 27 May 2005 14:52:26 -0700 Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: References: Message-ID: <1117230746.16429.8.camel@duffman> On Fri, 2005-05-27 at 08:44 -0700, Sean Hefty wrote: > >+/* > >+ * approximately transforms miliseconds to 4.096us*2^x > >+ * 63(+8) is max return > > I think that the max return is 64+8. I guess that is technically true, but converged overflows when i is 64. So, maybe I should stop one before that. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From tduffy at sun.com Fri May 27 14:57:36 2005 From: tduffy at sun.com (Tom Duffy) Date: Fri, 27 May 2005 14:57:36 -0700 Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: References: Message-ID: <1117231056.16429.15.camel@duffman> On Thu, 2005-05-26 at 22:25 -0700, Sean Hefty wrote: > >So, here is the strategy I am taking. Please let me know if it is > >wrong. > > > >When dapl_ep_connect() is called, I save off the timeout value into the > >dapl_ep struct. Then, when we get ready to call ib_send_cm_req(), I > >stuff the timeout value (after munging it into IB's strange format) into > >the conn params remote_cm_response_timeout. > > From a CM perspective, this sounds fine. Note that the CM timeout will not > occur until the number of retries has been met. So I don't know if the > timeout passed to dapl_ep_connect() should convert directly into the > remote_cm_response_timeout, or needs to be divided by the number of retries. So, are you saying that if you have a timeout of 4 seconds (you pass in 20) and you have retries set to 2, that it will fail after 8 seconds? James, what is the timeout value passed into dapl_ep_connect mean, the total timeout time? Or how much for each retry? Also, did you notice that dapl_ib_connect always sets the timeout to 20 (4 seconds) no matter what? Should this be the case? > >If the connection fails to complete within the timeout, > >dapl_cm_active_cb_handler() is called with IB_CM_REQ_ERROR which in turn > >calls dapl_evd_connection_callback() which does the same thing that > >dapl_ep_timeout() used to do -- tear down the connection. > > I haven't looked at your changes, but note that calling ib_destroy_cm_id > from within the CM callback thread will hang. The callback holds a > reference on the cm_id. The good news is that there should be code in kDAPL > to catch this. I will take a look and see if this could happen. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From roland at topspin.com Fri May 27 14:59:35 2005 From: roland at topspin.com (Roland Dreier) Date: Fri, 27 May 2005 14:59:35 -0700 Subject: [openib-general] OpenSM crash In-Reply-To: <1117229753.4479.190.camel@hal.voltaire.com> (Hal Rosenstock's message of "27 May 2005 17:37:10 -0400") References: <1117218451.32354.10.camel@duffman> <1117218692.32354.13.camel@duffman> <1117228540.4479.140.camel@hal.voltaire.com> <1117229081.4479.161.camel@hal.voltaire.com> <52k6lke4me.fsf@topspin.com> <1117229753.4479.190.camel@hal.voltaire.com> Message-ID: <52br6we3eg.fsf@topspin.com> Roland> I find it hard to believe that OpenSM has sent 4 billion Roland> MADs -- that's more than 1000 MADs a second for a solid Roland> month. It also looks very suspicious that the value is Roland> equal to ((unsigned int) -1). Hal> ^^^^^^^^^^^^^^^^^^ on a 32 bit machine. This is really a very minor point but the following program #include int main(int argc, char *argv[]) { printf("%u\n", ((unsigned int) -1)); return 0; } prints 4294967295 on any 64-bit Linux machine I have access to... - R. From sean.hefty at intel.com Fri May 27 15:31:53 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Fri, 27 May 2005 15:31:53 -0700 Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers insteadof own In-Reply-To: <1117231056.16429.15.camel@duffman> Message-ID: >> From a CM perspective, this sounds fine. Note that the CM timeout will >not >> occur until the number of retries has been met. So I don't know if the >> timeout passed to dapl_ep_connect() should convert directly into the >> remote_cm_response_timeout, or needs to be divided by the number of >retries. > >So, are you saying that if you have a timeout of 4 seconds (you pass in >20) and you have retries set to 2, that it will fail after 8 seconds? > >James, what is the timeout value passed into dapl_ep_connect mean, the >total timeout time? Or how much for each retry? If you pass in a timeout of 4 seconds with retries to 2, the call will timeout in 12 seconds. The request will be sent 3 times (2 retries). I should also note that the CM timeout includes the packet lifetime (round trip time) in its timeout calculation, but this should be small. - Sean From robert.j.woodruff at intel.com Fri May 27 15:56:58 2005 From: robert.j.woodruff at intel.com (Bob Woodruff) Date: Fri, 27 May 2005 15:56:58 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <8508251A6FC08A489844A94261D3693A057592@fiona.siliquent.com> Message-ID: Caitlin wrote, >Both uDAPL and kDAPL were designed for *application* use. >Even kDAPL is more intended for use by a kernel daemon that >is loaded separately from the kernel than for use within >the kernel itself. kDAPL is intended as a kernel-level API for RDMA enabled fabrics. As it was initially written, it does not meet the Linux coding style and that is why it is being totally reworked as we speak to meet that goal. >An ideal API for use within the kernel would abstract as >much as possible (without requiring emulation), and then >have transport specific unions or enum values. It would >hide no control options, merely provide common controls >for common capabilities. So for every new RDMA device type that comes along, you need to add a new enum, and unions for device class specific stuff, etc. Seems rather static and not easily extended. Not to mention that testing nightmare when the thing has to support 20 different types of RDMA enabled devices. I think code like that could get pretty ugly pretty fast. I'd rather see a registration mechanism like what we already have with DAPL that does not require any code changes to add a new RDMA device/provider. We have already proven that this works in DAPL as I know if at least 3 providers, IB, Myrinet, and RNIC (Ammasso) that were developed separately and were able to co-exist without any changes (enums and device class unions) in the DAT mid-layer. I assume that this can also be done with kDAPL in the kernel, but I defer to the DAPL experts to answer that one. James/Arkady Comments ? woody From caitlinb at siliquent.com Fri May 27 16:39:23 2005 From: caitlinb at siliquent.com (Caitlin Bestler) Date: Fri, 27 May 2005 16:39:23 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux Message-ID: <8508251A6FC08A489844A94261D3693A0575A5@fiona.siliquent.com> > -----Original Message----- > From: Bob Woodruff [mailto:robert.j.woodruff at intel.com] > Sent: Friday, May 27, 2005 3:57 PM > To: Caitlin Bestler; Roland Dreier; 'James Lentini'; > Arkady.Kanevsky at netapp.com > Cc: Venkata Jagana; rdma-developers at lists.sourceforge.net; > openib-general at openib.org > Subject: RE: [Rdma-developers] Re: [openib-general] OpenIB > and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux > > Caitlin wrote, > >Both uDAPL and kDAPL were designed for *application* use. > >Even kDAPL is more intended for use by a kernel daemon that > is loaded > >separately from the kernel than for use within the kernel itself. > > kDAPL is intended as a kernel-level API > for RDMA enabled fabrics. As it was initially written, it > does not meet the Linux coding style and that is why it is > being totally reworked as we speak to meet that goal. > One of the primary features of both uDAPL and kDAPL is a unified event reporting mechanism. This feature was adopted by IT-API as well. The intent of this feature, which I am mildly familiar with, was to simplify the writing of applications -- whether those applications were written to run in user or kernel space. This unification is nice, and in my opinion vital, for applications. Roland had properly pointed out that some kDAPL features go beyond what is stricly necessary to achieve transport neutrality. I agree. Event unification is extremely valuable *for applications*, but it is not *necessary* to provide a transport neutral definition of RDMA services *within* the kernel. In an ideal world the in-kernel API would provide transport and vendor neutral definitions of reliable RDMA services, and as little else as possible. That said, if I were developing a kernel daemon today I would definitely use kDAPL. It is defined today, and its "extras" do not cost all that much, and largely duplicate things that my application would have to do anyway. > >An ideal API for use within the kernel would abstract as much as > >possible (without requiring emulation), and then have transport > >specific unions or enum values. It would hide no control options, > >merely provide common controls for common capabilities. > > So for every new RDMA device type that comes along, you need > to add a new enum, and unions for device class specific stuff, etc. > Seems rather static and not easily extended. Not to mention > that testing nightmare when the thing has to support 20 > different types of RDMA enabled devices. > I think code like that could get pretty ugly pretty fast. > > I'd rather see a registration mechanism like what we already > have with DAPL that does not require any code changes to add > a new RDMA device/provider. We have already proven that this > works in DAPL as I know if at least 3 providers, IB, Myrinet, > and RNIC (Ammasso) that were developed separately and were > able to co-exist without any changes (enums and device class > unions) in the DAT mid-layer. > I assume that this can also be done with kDAPL in the kernel, > but I defer to the DAPL experts to answer that one. > We are discussing a low level interface to be used by the most privileged code within a system. Controls should not be hidden from it. Each transport should be expected to standardize their own transport specific controls, and should greatly minimize them and use controls defined in RDMA terms as much as possible. But there are InfiniBand-specific control options required to fully control an InfiniBand HCA, and there are iWARP-specific controls required to fully control an iWARP RNIC. If you don't like it, go get the protocols respecified. The DAPL code actually required *extensive* changes to support iWARP (I did the first round of them myself). And the glue layer required for an iWARP vendor is extensive precisely because it must pretend to be an InfiniBand device to Sourceforge "common" code. Implementing iWARP under an API designed for InfiniBand is a mistake I do not wish to repeat. The sourceforge DAPL requires extensive parallel data between the DAPL layer and the verbs layer, that has a measurable impact on system performance. RNIC-PI not only avoids requiring either IB HCAs to pretend to be iWARP RNICs, or iWARP RNICs to pretend to be IB HCAs, it also provides features such as kernel mode completions and 'os_data' markers that eliminate the need for parallel DAPl/verbs data structures. Once this lowest-possible-RDMA-API is defined it will make it possible for *most* applications to work with only transport neutral fields and enums, and virtually all applications to do so for their non-error paths. But such an API is not a few minor tweaks away from the Gen2 verbs. Trying to sweep the differences under the rug in a "low level API" is what produces truly ugly code. But once there is an error, one of the requirements of a "as low as possible" API is that it not conceal data from the kernel. That means there will be transport dependencies hidden in unions. It's inescapable. From robert.j.woodruff at intel.com Fri May 27 17:35:42 2005 From: robert.j.woodruff at intel.com (Bob Woodruff) Date: Fri, 27 May 2005 17:35:42 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <8508251A6FC08A489844A94261D3693A0575A5@fiona.siliquent.com> Message-ID: Catlin wrote, >The sourceforge DAPL requires extensive parallel data between >the DAPL layer and the verbs layer, that has a measurable impact >on system performance. RNIC-PI not only avoids requiring either >IB HCAs to pretend to be iWARP RNICs, or iWARP RNICs to pretend >to be IB HCAs, it also provides features such as kernel mode >completions and 'os_data' markers that eliminate the need for >parallel DAPl/verbs data structures. >Once this lowest-possible-RDMA-API is defined it will make it >possible for *most* applications to work with only transport >neutral fields and enums, and virtually all applications to >do so for their non-error paths. But such an API is not a few >minor tweaks away from the Gen2 verbs. Trying to sweep the >differences under the rug in a "low level API" is what >produces truly ugly code. The IB companies are not going to throw away a working implementation and move to RNIC-PI, but people have already said that they are willing to work with the iWarp community to add support and do what is right for Linux. However, as Roland said earlier, >Without seeing some real patches from the iWARP side, it's hard for me >to see any value in continuing to participate in this debate. > - R. woody From caitlinb at siliquent.com Fri May 27 18:24:46 2005 From: caitlinb at siliquent.com (Caitlin Bestler) Date: Fri, 27 May 2005 18:24:46 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux Message-ID: <8508251A6FC08A489844A94261D3693A0575A8@fiona.siliquent.com> > -----Original Message----- > From: Bob Woodruff [mailto:robert.j.woodruff at intel.com] > Sent: Friday, May 27, 2005 5:36 PM > To: Caitlin Bestler; Roland Dreier; James Lentini; > Arkady.Kanevsky at netapp.com > Cc: Venkata Jagana; rdma-developers at lists.sourceforge.net; > openib-general at openib.org > Subject: RE: [Rdma-developers] Re: [openib-general] OpenIB > and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux > > However, as Roland said earlier, > > >Without seeing some real patches from the iWARP side, it's > hard for me > >to see any value in continuing to participate in this debate. > > > - R. > Oh that's a great development plan for me to propose to my business people. Let's go write lots of code, and *after* we write it the IB people (our competitors) have agreed to *look* at it (but they're all busy, don't forget) and *then* we just might get merged in. If nobody is willing to discuss what the characteristics of this merged API will look like *before* we have to code it then there is no motivation to work inside OpenIB at all. From iod00d at hp.com Fri May 27 19:15:04 2005 From: iod00d at hp.com (Grant Grundler) Date: Fri, 27 May 2005 19:15:04 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <8508251A6FC08A489844A94261D3693A0575A8@fiona.siliquent.com> References: <8508251A6FC08A489844A94261D3693A0575A8@fiona.siliquent.com> Message-ID: <20050528021504.GR7444@esmail.cup.hp.com> On Fri, May 27, 2005 at 06:24:46PM -0700, Caitlin Bestler wrote: > Oh that's a great development plan for me to propose > to my business people. Let's go write lots of code, > and *after* we write it the IB people (our competitors) > have agreed to *look* at it (but they're all busy, don't > forget) and *then* we just might get merged in. You will never get a guarantee your code will go into kernel.org regardless of which path you take. I've seen three of the lead developers (a) promise to review any proposed code changes and (b) indicate they are willing to merge in support for RNICs. That's more than most people get. > If nobody is willing to discuss what the characteristics > of this merged API will look like *before* we have to > code it then there is no motivation to work inside OpenIB > at all. The openib.org APIs today only exists as code. Bob Woodruff is writing a paper documenting some of the interfaces but it's not done yet. Code is the primary currency to exchange ideas in open source forums. It doesn't have to be working code - psuedo code is probably sufficient to start with in this case. Look at the relevant openib.org header files (e.g. ib_verbs.h, ib_user_verbs.h) and make a list of changes needed for it to be useful to an RNIC. Just pointing at RNIC-PI isn't interesting. hth, grant From caitlinb at siliquent.com Fri May 27 20:04:26 2005 From: caitlinb at siliquent.com (Caitlin Bestler) Date: Fri, 27 May 2005 20:04:26 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux Message-ID: <8508251A6FC08A489844A94261D3693A0575A9@fiona.siliquent.com> > -----Original Message----- > From: Grant Grundler [mailto:iod00d at hp.com] > Sent: Friday, May 27, 2005 7:15 PM > To: Caitlin Bestler > Cc: Bob Woodruff; Roland Dreier; James Lentini; > Arkady.Kanevsky at netapp.com; Venkata Jagana; > rdma-developers at lists.sourceforge.net; openib-general at openib.org > Subject: Re: [Rdma-developers] Re: [openib-general] OpenIB > and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux > > On Fri, May 27, 2005 at 06:24:46PM -0700, Caitlin Bestler wrote: > > Oh that's a great development plan for me to propose to my business > > people. Let's go write lots of code, and *after* we write it the IB > > people (our competitors) have agreed to *look* at it (but > they're all > > busy, don't > > forget) and *then* we just might get merged in. > > You will never get a guarantee your code will go into > kernel.org regardless of which path you take. > I've seen three of the lead developers (a) promise to review > any proposed code changes and (b) indicate they are willing > to merge in support for RNICs. > A commitment that after merger that the API will be transport neutral is essential. That is, IB HCAs do not have to emulate iWARP. iWARP HCAs do not have to emulate IB. I fail to see why an advance agreement about the goal cannot be reached. The iWARP vendors are willing to concede that current Gen2 verb modules still have to be supported in a merged solution. What is your definiton of a merged solution? Is there a commitment that "RDMA Services" does not mean "InfiniBand, and the iWARP vendors have to pretend to be InfiniBand but we'll add one or two fields for problems they *can't* solve." > > Look at the relevant openib.org header files (e.g. ib_verbs.h, > ib_user_verbs.h) and make a list of changes needed for it to > be useful to an RNIC. Just pointing at RNIC-PI isn't interesting. > Well since we are talking about what a final transport neutral API would look like, as a guide when writing new code that will be compatible with the final merger, then looking at an example of an API that attempts to be transport neutral strikes me as highly relevant. Is there anything in the general structure of RNIC-PI that you do not like? The IB-specific portions are different than Gen2, and that certainly would not be true of any final merged verbs. So far the only comments received are that RNIC-PI was not written to be inside the Linux Kernel. That's correct, it was designed to be an OS neutral API that could be supported on a wide variety of Operating systems. Changes *have* been made, such as always having struct/enum/union tags, precisely to guarantee that a Linux version of rnicpi.h can be written that will be interoperable with code written to the man pages. As for gen2, here's one minor example: struct ib_qp_attr { enum ib_qp_state qp_state; // States defined are IB specific enum ib_qp_state cur_qp_state; enum ib_mtu path_mtu; // The path MTU is not a QP attribute in iWARP, // The LLP maintains that. The maximum DDP Segment // size is exported to the DDP layer, but not to // the consumer. enum ib_mig_state path_mig_state; // Path migration is handled at L2 or L3, it is // not visisble to an iWARP QP. u32 qkey; // There is no qkey u32 rq_psn; // There is no psn. There is an MSN for send/recvs. // It is not normally exported to the consumer. u32 sq_psn; // There is no psn. u32 dest_qp_num; // The destination QP is not known. int qp_access_flags; // presumably mostly compatible, but its not documented // in the .h file struct ib_qp_cap cap; // different capacities are controllable between // iWARP and IB. RNIC-PI defined the union of the // two, and left it to the providers to simply replicate // counts that are redundant under their transport. // Example: RDMAC allows different sge capacities // depending on the type of work request (send vs. write). struct ib_ah_attr ah_attr; // entirely IB specific information. TCP equivalents are // not part of QP state. struct ib_ah_attr alt_ah_attr; // see ah_attr u16 pkey_index; // VLANs are the equivalent, and are managed below the // RDMA layer u16 alt_pkey_index; // ditto u8 en_sqd_async_notify; // There is no sqd state u8 sq_draining; u8 max_rd_atomic; // there are no atomics u8 max_dest_rd_atomic; u8 min_rnr_timer; // Retries and pauses are handled below the RDMA layer u8 port_num; // Egress ports are managed at the IP layer, not as // part of the RDMA layer. Egress ports can be // reassigned without changing the QP's state // at the RDMA layer u8 timeout; // managed by the LLP (TCP or SCTP) u8 retry_cnt; // managed by the LLP (TCP or SCTP) u8 rnr_retry; // managed by the LLP (TCP or SCTP) u8 alt_port_num; // see port num u8 alt_timeout; // see timeout }; In a merged transport neutral API virtually the entire current struct would become part of an ib union. Just adding comments that this field is "IB only" doesn't achieve neutrality, it makes iWARP developers wade through structs to determine which fields can be effected, and makes it all too likely that an application developer will set a field not realizing that an iWARP RNIC will not look at that field. If a transport neutral API based on unioning everything that is not transport neutral is not acceptable as the goal of eventual merger is not acceptable then the emphasis should be on kDAPL as the transport neutral API. From iod00d at hp.com Fri May 27 22:09:16 2005 From: iod00d at hp.com (Grant Grundler) Date: Fri, 27 May 2005 22:09:16 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <8508251A6FC08A489844A94261D3693A0575A9@fiona.siliquent.com> References: <8508251A6FC08A489844A94261D3693A0575A9@fiona.siliquent.com> Message-ID: <20050528050916.GB9941@esmail.cup.hp.com> On Fri, May 27, 2005 at 08:04:26PM -0700, Caitlin Bestler wrote: > > You will never get a guarantee your code will go into > > kernel.org regardless of which path you take. > > I've seen three of the lead developers (a) promise to review > > any proposed code changes and (b) indicate they are willing > > to merge in support for RNICs. > > A commitment that after merger that the API will be transport > neutral is essential. That is, IB HCAs do not have to emulate > iWARP. iWARP HCAs do not have to emulate IB. "commitment" is a heavy word. One that should only be used to describe individual actions in an open source forum. I think you've already got agreement (not commitment) that this is a reasonable goal - it totally depends on what the proposed patches look like and if the person doing the work is trustworthy and competent. The folks running the openib source tree are a very accomodating and helpful crowd. Don't get discourage by initial negative feedback. Once you "get it" (process wise) things should move pretty quickly. Sounds like you've never worked on an opensource project before. Have you read Documentation/SubmittingPatches in your favorite source tree? Do you want other pointers on "how to collaborate on open source projects"? > I fail to see why an advance agreement about the goal cannot > be reached. The iWARP vendors are willing to concede that > current Gen2 verb modules still have to be supported in a > merged solution. That's a good first step. I think you've already gotten as much advance agreement as anyone is willing to give. Trust me, it doesn't get much better than what you and Venkata (IBM) have been promised already. > What is your definiton of a merged solution? One that starts with a code change proposal. Start with any existing file in openib.org tree and suggest changes that would make it fit your needs better. The more specific, the better. One piece at time. > Is there a commitment > that "RDMA Services" does not mean "InfiniBand, and the iWARP > vendors have to pretend to be InfiniBand but we'll add one or > two fields for problems they *can't* solve." No. This is open source. We depend on maintainers to use good judgement and listen to other people. Projects without such leaders don't survive. Linus has a cute, short essay on that in Documentation/ManagementStyle. Please read it. It's quite amusing and insightful at the same time. > > Look at the relevant openib.org header files (e.g. ib_verbs.h, > > ib_user_verbs.h) and make a list of changes needed for it to > > be useful to an RNIC. Just pointing at RNIC-PI isn't interesting. > > Well since we are talking about what a final transport neutral > API would look like, as a guide when writing new code that will > be compatible with the final merger, then looking at an example > of an API that attempts to be transport neutral strikes me as > highly relevant. Uhm, sort of. Yes, in that the design probably has relevance. The implementation less so. This discussion is about starting with openib gen2, right? If it's not, I'm wasting my time. > Is there anything in the general structure of RNIC-PI that you > do not like? The IB-specific portions are different than Gen2, > and that certainly would not be true of any final merged verbs. > So far the only comments received are that RNIC-PI was not written > to be inside the Linux Kernel. That's correct, it was designed to > be an OS neutral API that could be supported on a wide variety > of Operating systems. Changes *have* been made, such as always > having struct/enum/union tags, precisely to guarantee that a > Linux version of rnicpi.h can be written that will be interoperable > with code written to the man pages. I barely have time to keep up with code changes going on in openib.org tree and don't understand details of 1/2 of those. I'm the wrong person to ask even about specifics in openib.org. My expertise is not in IB transport (it's PCI chipsets and platform IRQ/DMA support.) I've worked full time on open source projects for the past 5 years and have a clue how they do (and don't) work. Some definitely work better than others. It always revolves around the people leading the project. openib.org is doing fairly well right now IMHO. > As for gen2, here's one minor example: Good - that's a start. ISTR Someone already pointed out some attr fields need to change. > struct ib_qp_attr { > enum ib_qp_state qp_state; // States > defined are IB specific Please don't use C++ style comments. I talk about how this field (and others) *might* be organized below. > enum ib_qp_state cur_qp_state; > enum ib_mtu path_mtu; // The path MTU > is not a QP attribute in iWARP, It's ok if *some* fields are unused I think. (not that this needs to be here) > // The LLP > maintains that. The maximum DDP Segment > // size is > exported to the DDP layer, but not to > // the consumer. Please try to keep lines shorter than 80 columns. > enum ib_mig_state path_mig_state; // Path migration is > handled at L2 or L3, it is > // not visisble > to an iWARP QP. > u32 qkey; // There is no > qkey > u32 rq_psn; // There is no psn. > There is an MSN for send/recvs. > // It is not > normally exported to the consumer. Is this a kernel or user space header file? > u32 sq_psn; // There is no psn. > u32 dest_qp_num; // The destination QP is > not known. > int qp_access_flags; // presumably mostly > compatible, but its not documented > // in the .h > file Right - you (or someone from rdmaconsortium) needs to track down usage and verify. > struct ib_qp_cap cap; // different > capacities are controllable between > // iWARP and IB. > RNIC-PI defined the union of the > // two, and left > it to the providers to simply replicate > // counts that > are redundant under their transport. > // Example: > RDMAC allows different sge capacities > // depending on > the type of work request (send vs. write). > struct ib_ah_attr ah_attr; // entirely IB > specific information. TCP equivalents are > // not part of > QP state. > struct ib_ah_attr alt_ah_attr; // see ah_attr > u16 pkey_index; // VLANs are the > equivalent, and are managed below the > // RDMA layer > u16 alt_pkey_index; // ditto > u8 en_sqd_async_notify; // There is no > sqd state > u8 sq_draining; > u8 max_rd_atomic; // there are no > atomics > u8 max_dest_rd_atomic; > u8 min_rnr_timer; // Retries and > pauses are handled below the RDMA layer > u8 port_num; // > Egress ports are managed at the IP layer, not as > // part > of the RDMA layer. Egress ports can be > // > reassigned without changing the QP's state > // at > the RDMA layer > u8 timeout; // > managed by the LLP (TCP or SCTP) > u8 retry_cnt; // > managed by the LLP (TCP or SCTP) > u8 rnr_retry; // > managed by the LLP (TCP or SCTP) > u8 alt_port_num; // see port num > u8 alt_timeout; // see timeout > }; Sounds like this struct is a good place start seperating generic (subsystem) and transport specific parts. There are lots of examples of this in the linux kernel already. Shouldn't be that big of a deal since "struct ib_qp_attr" is only referenced in 5 files under drivers/infiniband in my source tree. I also expect some of those consumers could move out of the generic code into some IB specific part of the source tree (if possible). > In a merged transport neutral API virtually the entire current struct > would become part of an ib union. Just adding comments that this field is "IB > only" > doesn't achieve neutrality, it makes iWARP developers wade through > structs > to determine which fields can be effected, and makes it all too likely > that > an application developer will set a field not realizing that an iWARP > RNIC > will not look at that field. My preferred way to deal with this is to allow the transport to allocate extra space behind the generic part that is only for use by the transport specific code. Transport specific header files would have something like this in them: struct ib_qp_attr { struct rdma_qp_attr; /* generic stuff */ ...IB specific qp attr... }; The transport driver (maybe rdma interface driver like mthca for IB) could allocate it's control structures using this. Make sense? > If a transport neutral API based on unioning everything that is not > transport > neutral is not acceptable as the goal of eventual merger is not > acceptable > then the emphasis should be on kDAPL as the transport neutral API. I don't think I parsed this sentence correctly. But I'm really not a fan of unions. Just my $0.02. They should only be needed when knowledge of different layers *must* be shared across modules. I'm hoping they aren't necessary here because; o it means test and a branch for most accesses in generic code o we didn't split the generic/transport specific code right But I'm not the maintainer and other people will know if they are needed or not. hth, grant From hch at lst.de Sat May 28 00:13:44 2005 From: hch at lst.de (Christoph Hellwig) Date: Sat, 28 May 2005 09:13:44 +0200 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: References: <8508251A6FC08A489844A94261D3693A057592@fiona.siliquent.com> Message-ID: <20050528071344.GA1638@lst.de> On Fri, May 27, 2005 at 03:56:58PM -0700, Bob Woodruff wrote: > Caitlin wrote, > >Both uDAPL and kDAPL were designed for *application* use. > >Even kDAPL is more intended for use by a kernel daemon that > >is loaded separately from the kernel than for use within > >the kernel itself. > > kDAPL is intended as a kernel-level API > for RDMA enabled fabrics. As it was initially written, > it does not meet the Linux coding style and that is why > it is being totally reworked as we speak to meet that goal. The codingstyle alone isn't the problem. The whole design philosophy is rather odd. > >An ideal API for use within the kernel would abstract as > >much as possible (without requiring emulation), and then > >have transport specific unions or enum values. It would > >hide no control options, merely provide common controls > >for common capabilities. > > So for every new RDMA device type that comes along, you need to add a new > enum, and unions for device class specific stuff, etc. > Seems rather static and not easily extended. Not > to mention that testing nightmare when the thing has to support > 20 different types of RDMA enabled devices. > I think code like that could get pretty ugly pretty fast. read the _as much as possible_ above. There's a reason you'll need new ABIs for the socket interface aswell when adding new address families. From hch at lst.de Sat May 28 00:15:44 2005 From: hch at lst.de (Christoph Hellwig) Date: Sat, 28 May 2005 09:15:44 +0200 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <8508251A6FC08A489844A94261D3693A0575A8@fiona.siliquent.com> References: <8508251A6FC08A489844A94261D3693A0575A8@fiona.siliquent.com> Message-ID: <20050528071544.GB1638@lst.de> On Fri, May 27, 2005 at 06:24:46PM -0700, Caitlin Bestler wrote: > Oh that's a great development plan for me to propose > to my business people. Let's go write lots of code, > and *after* we write it the IB people (our competitors) > have agreed to *look* at it (but they're all busy, don't > forget) and *then* we just might get merged in. fortunately we don't care about your business people. either you do something sane, or your hardware won't we supported by linux at all. tough life. From hch at lst.de Sat May 28 00:21:52 2005 From: hch at lst.de (Christoph Hellwig) Date: Sat, 28 May 2005 09:21:52 +0200 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <8508251A6FC08A489844A94261D3693A0575A9@fiona.siliquent.com> References: <8508251A6FC08A489844A94261D3693A0575A9@fiona.siliquent.com> Message-ID: <20050528072152.GC1638@lst.de> On Fri, May 27, 2005 at 08:04:26PM -0700, Caitlin Bestler wrote: > A commitment that after merger that the API will be transport > neutral is essential. That is, IB HCAs do not have to emulate > iWARP. iWARP HCAs do not have to emulate IB. You won't get any commitments from us at all. > Is there anything in the general structure of RNIC-PI that you the API shows the usual signs of design by committee is is totally focused on tacking up separate propritary modules instead of focussing integrated stacks. It makes the usual mistake of mixing up user and kernel APIs, and the APIs are much too complicated. From halr at voltaire.com Sat May 28 04:49:58 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 28 May 2005 07:49:58 -0400 Subject: [openib-general] OpenSM crash In-Reply-To: <1117229753.4479.190.camel@hal.voltaire.com> References: <1117218451.32354.10.camel@duffman> <1117218692.32354.13.camel@duffman> <1117228540.4479.140.camel@hal.voltaire.com> <1117229081.4479.161.camel@hal.voltaire.com> <52k6lke4me.fsf@topspin.com> <1117229753.4479.190.camel@hal.voltaire.com> Message-ID: <1117280998.4479.3565.camel@hal.voltaire.com> On Fri, 2005-05-27 at 17:37, Hal Rosenstock wrote: > On Fri, 2005-05-27 at 17:33, Roland Dreier wrote: > > > May 27 01:44:09 [43005960] -> osm_vl15_post: 4294967295 MADs on wire, 2 MADs outstanding. > > > > Hal> I take that back. That's just a lot of MADs have been sent > > Hal> (on the IB wire). OpenSM was probably up and running for a > > Hal> while... > > > > I find it hard to believe that OpenSM has sent 4 billion MADs -- > > that's more than 1000 MADs a second for a solid month. It also looks > > very suspicious that the value is equal to ((unsigned int) -1). > ^^^^^^^^^^^^^^^^^^ > on a 32 bit machine. > > Good point. The fact that it gets to -1 is significant as I think that > is used as a magic value for some computations. I'm pretty sure that I see a way this could have gone negative in the vendor layer. I'm working on a patch for this. -- Hal From sganguly at yahoo.com Sat May 28 05:17:54 2005 From: sganguly at yahoo.com (Sukanta ganguly) Date: Sat, 28 May 2005 05:17:54 -0700 (PDT) Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: 6667 Message-ID: <20050528121755.44489.qmail@web51907.mail.yahoo.com> --- Christoph Hellwig wrote: > On Fri, May 27, 2005 at 06:24:46PM -0700, Caitlin > Bestler wrote: > > Oh that's a great development plan for me to > propose > > to my business people. Let's go write lots of > code, > > and *after* we write it the IB people (our > competitors) > > have agreed to *look* at it (but they're all busy, > don't > > forget) and *then* we just might get merged in. > > fortunately we don't care about your business > people. either > you do something sane, or your hardware won't we > supported by > linux at all. tough life. > > > > That's a pretty bold statement. Linux grew up to be popular via mass acceptance. Seems like that charter has changed and a few have control over Linux and its future. The "My way or the highway" philosophy has gotten embedded in the Linux way of life. Life is getting tough. ------------------------------------------------------- > This SF.Net email is sponsored by Yahoo. > Introducing Yahoo! Search Developer Network - Create > apps using Yahoo! > Search APIs Find out how you can build Yahoo! > directly into your own > Applications - visit > http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005 > _______________________________________________ > Rdma-developers mailing list > Rdma-developers at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/rdma-developers > __________________________________________________ Do You Yahoo!? Tired of spam? Yahoo! Mail has the best spam protection around http://mail.yahoo.com From hch at lst.de Sat May 28 06:47:46 2005 From: hch at lst.de (Christoph Hellwig) Date: Sat, 28 May 2005 15:47:46 +0200 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <20050528121755.44489.qmail@web51907.mail.yahoo.com> References: <20050528121755.44489.qmail@web51907.mail.yahoo.com> Message-ID: <20050528134746.GA6277@lst.de> On Sat, May 28, 2005 at 05:17:54AM -0700, Sukanta ganguly wrote: > That's a pretty bold statement. Linux grew up to be > popular via mass acceptance. Seems like that charter > has changed and a few have control over Linux and its > future. The "My way or the highway" philosophy has > gotten embedded in the Linux way of life. > Life is getting tough. You're totally missing the point. Linux is successfull exactly because it's lookinf for the right solution, not something the business people need short-term. From caitlin.bestler at gmail.com Sat May 28 16:26:43 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Sat, 28 May 2005 16:26:43 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <20050528134746.GA6277@lst.de> References: <20050528121755.44489.qmail@web51907.mail.yahoo.com> <20050528134746.GA6277@lst.de> Message-ID: <469958e005052816262c8ff2b4@mail.gmail.com> On 5/28/05, Christoph Hellwig wrote: > On Sat, May 28, 2005 at 05:17:54AM -0700, Sukanta ganguly wrote: > > That's a pretty bold statement. Linux grew up to be > > popular via mass acceptance. Seems like that charter > > has changed and a few have control over Linux and its > > future. The "My way or the highway" philosophy has > > gotten embedded in the Linux way of life. > > Life is getting tough. > > You're totally missing the point. Linux is successfull exactly > because it's lookinf for the right solution, not something the > business people need short-term. Such myopic cheerleading gets annoying and accomplishes nothing. The topic under discussion is whether a low level API for RDMA is necessary (as opposed to a higher level API such as kDAPL) and if so what the best strategy for achieving it is (try to plan an IB/iWARP merge immediately or wait until there is an iWARP code base). Claiming that an InfiniBand-specific interface is somehow thinking "long term" is just plain ludicrous. Now it may be that the short term interest of the InfiniBand vendors is such that they cannot commit resources to helping build a transport neutral API. That is always a legitimate tradeoff, but it is "short term corporate thinking". Last time I looked most of the commits being made to OpenIB (or sourceforge DAPL) were from being drawing paychecks from those "evil corporations". From caitlin.bestler at gmail.com Sat May 28 17:18:39 2005 From: caitlin.bestler at gmail.com (Caitlin Bestler) Date: Sat, 28 May 2005 17:18:39 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: References: <8508251A6FC08A489844A94261D3693A057592@fiona.siliquent.com> Message-ID: <469958e00505281718100915c2@mail.gmail.com> On 5/27/05, Bob Woodruff wrote: > > So for every new RDMA device type that comes along, you need to add a new > enum, and unions for device class specific stuff, etc. > Seems rather static and not easily extended. Not > to mention that testing nightmare when the thing has to support > 20 different types of RDMA enabled devices. > I think code like that could get pretty ugly pretty fast. > It would not be every new device, it would be every new RDMA verbs layer API. Gen2 already covers the needs of InfiniBand. RNIC-PI covers the needs of InfiniBand, iWARP/TCP and iWARP/SCTP. I would not anticipate a high rate of additional RDMA verbs appearing, particularly if the two existing verb APIs are unified. Creating new verb APIs is something vendors do to enable new markets, not because they have excess underutilized staff that needs to be kept busy. Well, who knows what vendors with excess underutilized staff do, there haven't been sightings of such creatures in memory. But in any event, that can be dealt with in one of two ways: struct rdma_xyz_ib { }; struct rdma_xyz_iwarp { }; union rdma_xyz_xport { struct rdma_xyz_ib ib; struct rdma_xyz_iwarp iwarp; }; struct rdma_xyz { /* common fields */ union rdma_xyz_xport xport; }; Verus.. struct rdma_xyz { /* common fields */ }; struct rdma_xyz_ib { struct rdma_xyz common; /* ib fields */ }; struct rdma_xyz_iwarp { struct rdma_xyz common; /* iwarp fields */ }; The latter style is extensible, but makes it difficult to properly allocate a buffer that works for all variants. The union style is also already in use in both IT-API and RNIC-PI. I personally prefer sub-classing to unions, but I have found myself in the minority on *most* projects where the issue has been discussed. One reason is that sub-classing provides very little type-safety. struct sockaddr is an example of this. It takes manual inspection to ensure that the variants are properly differentiated and it is still common for developers to pass in a plain struct sockaddr without realizing that it is not large enough for a struct sockaddr_in6. > I'd rather see a registration mechanism like what we already have > with DAPL that does not require any code changes to add a new RDMA > device/provider. We have already proven that this works in DAPL > as I know if at least 3 providers, IB, Myrinet, and RNIC (Ammasso) > that were developed separately and were able to co-exist without > any changes (enums and device class unions) in the DAT mid-layer. > I assume that this can also be done with kDAPL in the kernel, but > I defer to the DAPL experts to answer that one. > There would be a registration method. And each device would load a dynamic method table. What is in question is how many signatures are supported. From jcarr at linuxmachines.com Sat May 28 19:48:44 2005 From: jcarr at linuxmachines.com (Jeff Carr) Date: Sat, 28 May 2005 19:48:44 -0700 Subject: [openib-general] cmpost: failure sending REQ: -22 Message-ID: <42992D8C.9090206@linuxmachines.com> Has anyone seen ib_send_cm_req() return -22? I seem to be stuck with this error today. I've seen it in the past; and it seems to have gone away without my being able to discern a conclusive reason. Usually rebooting the systems will cause it to work again. This go around, It's really refusing this time; I reverted to kernel 2.6.11.7 and infiniband sources from around the end of april (even binaries I built then that were working don't work now.) I rebooted the SM also, but now I'm going to actually powerdown the systems. Although it seems like it shouldn't be possible, this behavior really makes it look like the HCA can get in a broken state that persists after a reboot. Enjoy, Jeff ib1: no IPv6 routers present cmpost: starting client cmpost: connecting cmpost: failure sending REQ: -22 cmpost: test complete From itamar at mellanox.co.il Sun May 29 03:18:33 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Sun, 29 May 2005 13:18:33 +0300 Subject: [openib-general] oops in ib_mad Message-ID: <91DB792C7985D411BEC300B40080D29CC35BDF@mtvex01.mtv.mtl.com> I hit this oops when trying to run kdapltest x86 + Suse 9.1 + k2.6.11 + svn rev 2507 May 29 12:51:45 mtvs43 kernel: Unable to handle kernel paging request at virtual address 00100104 May 29 12:51:45 mtvs43 kernel: printing eip: May 29 12:51:45 mtvs43 kernel: e09cb56c May 29 12:51:45 mtvs43 kernel: *pde = 00000000 May 29 12:51:45 mtvs43 kernel: Oops: 0002 [#1] May 29 12:51:45 mtvs43 kernel: SMP May 29 12:51:45 mtvs43 kernel: Modules linked in: kdapltest ib_dat_provider ib_cm ib_at dat ib_umad ib_i poib ib_sa edd joydev st sd_mod sr_mod scsi_mod ide_cd cdrom nvram usbserial parport_pc lp parport autof s ipv6 thermal processor fan button battery ac e100 mii ib_mthca ib_mad ib_core i2c_i801 i2c_core hw_ran dom uhci_hcd usbcore evdev dm_mod ext3 jbd May 29 12:51:45 mtvs43 kernel: CPU: 0 May 29 12:51:45 mtvs43 kernel: EIP: 0060:[pg0+542578028/1069171712] Not tainted VLI May 29 12:51:45 mtvs43 kernel: EIP: 0060:[] Not tainted VLI May 29 12:51:45 mtvs43 kernel: EFLAGS: 00010046 (2.6.11-gen2) May 29 12:51:45 mtvs43 kernel: EIP is at ib_mad_complete_send_wr+0x9c/0x160 [ib_mad] May 29 12:51:45 mtvs43 kernel: eax: 00100100 ebx: dbdafd80 ecx: dbdafd8c edx: 00200200 May 29 12:51:45 mtvs43 kernel: esi: c167d180 edi: c167d1b4 ebp: ddbb5f24 esp: ddbb5ee8 May 29 12:51:45 mtvs43 kernel: ds: 007b es: 007b ss: 0068 May 29 12:51:45 mtvs43 kernel: Process ib_mad1 (pid: 2953, threadinfo=ddbb4000 task=c16db550) May 29 12:51:45 mtvs43 kernel: Stack: 00000003 00000246 dbdafd80 dd501910 00000000 dbdafd80 e09cb65b dd501908 May 29 12:51:45 mtvs43 kernel: ddbb5f24 de39d400 deb6e800 dd501800 ddbb5f24 dd501800 e09cb8d9 dc4d6300 May 29 12:51:45 mtvs43 kernel: 00000000 00000000 00000000 ddbb5f58 c011b749 00000000 00000402 00000006 May 29 12:51:45 mtvs43 kernel: Call Trace: May 29 12:51:45 mtvs43 kernel: Call Trace: May 29 12:51:45 mtvs43 kernel: [pg0+542578267/1069171712] ib_mad_send_done_handler+0x2b/0x120 [ib_mad] May 29 12:51:45 mtvs43 kernel: [] ib_mad_send_done_handler+0x2b/0x120 [ib_mad] May 29 12:51:45 mtvs43 kernel: [pg0+542578905/1069171712] ib_mad_completion_handler+0x89/0xa0 [ib_mad] May 29 12:51:45 mtvs43 kernel: [] ib_mad_completion_handler+0x89/0xa0 [ib_mad] May 29 12:51:45 mtvs43 kernel: [__wake_up+41/64] __wake_up+0x29/0x40 May 29 12:51:45 mtvs43 kernel: [] __wake_up+0x29/0x40 May 29 12:51:45 mtvs43 kernel: [worker_thread+424/560] worker_thread+0x1a8/0x230 May 29 12:51:45 mtvs43 kernel: [] worker_thread+0x1a8/0x230 May 29 12:51:45 mtvs43 kernel: [pg0+542578768/1069171712] ib_mad_completion_handler+0x0/0xa0 [ib_mad] May 29 12:51:45 mtvs43 kernel: [] ib_mad_completion_handler+0x0/0xa0 [ib_mad] May 29 12:51:45 mtvs43 kernel: [default_wake_function+0/16] default_wake_function+0x0/0x10 May 29 12:51:45 mtvs43 kernel: [] default_wake_function+0x0/0x10 May 29 12:51:45 mtvs43 kernel: [default_wake_function+0/16] default_wake_function+0x0/0x10 May 29 12:51:45 mtvs43 kernel: [] default_wake_function+0x0/0x10 May 29 12:51:45 mtvs43 kernel: [worker_thread+0/560] worker_thread+0x0/0x230 May 29 12:51:45 mtvs43 kernel: [] worker_thread+0x0/0x230 May 29 12:51:45 mtvs43 kernel: [kthread+136/176] kthread+0x88/0xb0 May 29 12:51:45 mtvs43 kernel: [] kthread+0x88/0xb0 May 29 12:51:45 mtvs43 kernel: [kthread+0/176] kthread+0x0/0xb0 May 29 12:51:45 mtvs43 kernel: [] kthread+0x0/0xb0 May 29 12:51:45 mtvs43 kernel: [kernel_thread_helper+5/16] kernel_thread_helper+0x5/0x10 May 29 12:51:45 mtvs43 kernel: [] kernel_thread_helper+0x5/0x10 May 29 12:51:45 mtvs43 kernel: Code: 24 04 89 f8 59 5b 5b 5e 5f 5d e9 c0 38 94 df 8b 93 90 00 00 00 8d 4 2 ff 89 83 90 00 00 00 85 c0 7f d5 8d 4b 0c 8b 43 0c 8b 51 04 <89> 50 04 c7 41 04 00 02 20 00 89 02 89 f 0 c7 43 0c 00 01 10 00 Itamar Rabenstein Mellanox Technologies Ltd mailto : itamar at mellanox.co.il phone: 972-3-6259506 -------------- next part -------------- An HTML attachment was scrubbed... URL: From itamar at mellanox.co.il Sun May 29 03:23:16 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Sun, 29 May 2005 13:23:16 +0300 Subject: [openib-general] oops in ib_at when running with CONFIG_DEBUG_PAGEALLOC Message-ID: <91DB792C7985D411BEC300B40080D29CC35BE1@mtvex01.mtv.mtl.com> I am running kdapltest -T Q with CONFIG_DEBUG_PAGEALLOC on and i am geting this oops x86 + Suse9.1 + k2.6.11 + svn rev 2507 + kdapl and ib_at from users/jlentini when running without CONFIG_DEBUG_PAGEALLOC i dont see the oops May 29 11:38:24 mtvs42 kernel: Kdapltest Module Loaded - major 254 May 29 11:38:51 mtvs42 kernel: ib_mthca 0000:04:00.0: Registering memory at 2c5a8000 (iova 2c5a8000) in PD 5; shift 12, npages 6. May 29 11:38:52 mtvs42 kernel: Unable to handle kernel paging request at virtual address eca6af74 May 29 11:38:52 mtvs42 kernel: printing eip: May 29 11:38:52 mtvs42 kernel: f8ad33bc May 29 11:38:52 mtvs42 kernel: *pde = 0047e067 May 29 11:38:52 mtvs42 kernel: Oops: 0002 [#1] May 29 11:38:52 mtvs42 kernel: SMP DEBUG_PAGEALLOC May 29 11:38:52 mtvs42 kernel: Modules linked in: kdapltest ib_dat_provider ib_cm ib_at dat i b_umad ib_ipoib ib_sa edd joydev st sd_mod sr_mod scsi_mod ide_cd cdrom nvram usbserial parpo rt_pc lp parport autofs ipv6 thermal processor fan button battery ac e100 mii ib_mthca ib_mad ib_core i2c_i801 i2c_core hw_random uhci_hcd usbcore evdev dm_mod ext3 jbd May 29 11:38:52 mtvs42 kernel: CPU: 0 May 29 11:38:52 mtvs42 kernel: EIP: 0060:[pg0+946295740/1069155328] Not tainted VLI May 29 11:38:52 mtvs42 kernel: EIP: 0060:[] Not tainted VLI May 29 11:38:52 mtvs42 kernel: EFLAGS: 00010297 (2.6.11-gen2) May 29 11:38:52 mtvs42 kernel: EIP is at req_free+0x4c/0x70 [ib_at] May 29 11:38:52 mtvs42 kernel: eax: 00000001 ebx: eca6af6c ecx: c1594d40 edx: 00000640 May 29 11:38:52 mtvs42 kernel: esi: ec6ae7f8 edi: eca6af90 ebp: eca6af6c esp: f0409f5c May 29 11:38:52 mtvs42 kernel: ds: 007b es: 007b ss: 0068 May 29 11:38:52 mtvs42 kernel: Process ib_at_wq/0 (pid: 9619, threadinfo=f0408000 task=ec8f7ad0) May 29 11:38:52 mtvs42 kernel: Stack: eca6af8c c012eea8 ec6ae820 f8ad3470 00000286 f0408000 ec6ae808 ffffffff May 29 11:38:52 mtvs42 kernel: ffffffff 00000001 00000000 c011bbd0 00010000 00000000 00000000 ef579f38 May 29 11:38:52 mtvs42 kernel: ece8ed8c 00000000 ec8f7ad0 c011bbd0 00100100 00200200 88497fc0 000f428d May 29 11:38:52 mtvs42 kernel: Call Trace: May 29 11:38:52 mtvs42 kernel: [worker_thread+424/560] worker_thread+0x1a8/0x230 May 29 11:38:52 mtvs42 kernel: [] worker_thread+0x1a8/0x230 May 29 11:38:52 mtvs42 kernel: [pg0+946295920/1069155328] req_comp_work+0x0/0x20 [ib_at] May 29 11:38:52 mtvs42 kernel: [] req_comp_work+0x0/0x20 [ib_at] May 29 11:38:52 mtvs42 kernel: [default_wake_function+0/16] default_wake_function+0x0/0x10 May 29 11:38:52 mtvs42 kernel: [] default_wake_function+0x0/0x10 May 29 11:38:52 mtvs42 kernel: [default_wake_function+0/16] default_wake_function+0x0/0x10 May 29 11:38:52 mtvs42 kernel: [] default_wake_function+0x0/0x10 May 29 11:38:52 mtvs42 kernel: [worker_thread+0/560] worker_thread+0x0/0x230 May 29 11:38:52 mtvs42 kernel: [] worker_thread+0x0/0x230 May 29 11:38:52 mtvs42 kernel: [kthread+136/176] kthread+0x88/0xb0 May 29 11:38:52 mtvs42 kernel: [] kthread+0x88/0xb0 May 29 11:38:52 mtvs42 kernel: [kthread+0/176] kthread+0x0/0xb0 May 29 11:38:52 mtvs42 kernel: [] kthread+0x0/0xb0 May 29 11:38:52 mtvs42 kernel: [kernel_thread_helper+5/16] kernel_thread_helper+0x5/0x10 May 29 11:38:52 mtvs42 kernel: [] kernel_thread_helper+0x5/0x10 May 29 11:38:52 mtvs42 kernel: Code: ad f8 e8 28 ca 64 c7 83 c4 0c c7 43 08 00 00 00 00 c7 43 0c 00 00 00 00 c7 83 84 00 00 00 00 00 00 00 5b c3 89 d8 e8 34 03 00 00 43 08 00 00 00 00 c7 43 0c 00 00 00 00 c7 83 84 00 00 00 00 Itamar Rabenstein Mellanox Technologies Ltd mailto : itamar at mellanox.co.il phone: 972-3-6259506 -------------- next part -------------- An HTML attachment was scrubbed... URL: From itamar at mellanox.co.il Sun May 29 04:43:38 2005 From: itamar at mellanox.co.il (Itamar) Date: Sun, 29 May 2005 14:43:38 +0300 Subject: [openib-general] [PATCH][kdapl] fix fatal bug in triger the evd upcall Message-ID: <20050529114338.GA16560@mellanox.co.il> Hi James, This patch fix a fatal bug that exist in current lastet bits in kdapl (svn rev 2507) As you can see we need to triger the upcall when dapl_evd_dequeue return with good status and quit the method when dapl_evd_dequeue return with non zero status which mean queue is empty. In the current bits no kdapltest can run even the simple quit test. Please in the future before you commit changes to the svn run a simple regression. Any way with this patch the code is working again. fix fatal bug in triger the evd upcall Signed-off-by: Itamar Rabenstein Index: dapl_cno_util.c =================================================================== --- dapl_cno_util.c (revision 2509) +++ dapl_cno_util.c (working copy) @@ -115,12 +115,8 @@ for (;;) { status = dapl_evd_dequeue((DAT_EVD_HANDLE)evd, &event); - if (DAT_SUCCESS == status) { - dapl_dbg_log(DAPL_DBG_TYPE_ERR, - "dapl_evd_dequeue failed: %x\n", status); + if (DAT_SUCCESS != status) return; - } - cno->cno_upcall.upcall_func(cno->cno_upcall.instance_data, &event, FALSE); } -- Itamar From itamar at mellanox.co.il Sun May 29 05:10:29 2005 From: itamar at mellanox.co.il (Itamar) Date: Sun, 29 May 2005 15:10:29 +0300 Subject: [openib-general] [PATCH][kdapl] replace spin_lock with spin_lock_irqsave in kdapltest Message-ID: <20050529121029.GA16745@mellanox.co.il> With this patch i can run kdapltest -T T ... -t 4 -w 8 ... I still see problems but in general this patch helps the stability a lot. replace spin_lock with spin_lock_irqsave in kdapltest Signed-off-by: Itamar Rabenstein Index: test/dapl_transaction_stats.c =================================================================== --- test/dapl_transaction_stats.c (revision 2509) +++ test/dapl_transaction_stats.c (working copy) @@ -45,12 +45,13 @@ DT_transaction_stats_set_ready (DT_Tdep_Print_Head *phead, Transaction_Stats_t * transaction_stats) { - DT_Mdep_Lock (&transaction_stats->lock); + unsigned long flags; + spin_lock_irqsave (&transaction_stats->lock,flags); transaction_stats->wait_count--; DT_Tdep_PT_Debug (1,(phead,"Received Sync Message from server (%d left)\n", transaction_stats->wait_count)); - DT_Mdep_Unlock (&transaction_stats->lock); + spin_unlock_irqrestore (&transaction_stats->lock,flags); } boolean_t @@ -86,7 +87,8 @@ unsigned int bytes_rdma_read, unsigned int bytes_rdma_write) { - DT_Mdep_Lock (&transaction_stats->lock); + unsigned long flags; + spin_lock_irqsave (&transaction_stats->lock,flags); /* look for the longest time... */ if (time_ms > transaction_stats->time_ms) @@ -99,5 +101,5 @@ transaction_stats->bytes_recv += bytes_recv; transaction_stats->bytes_rdma_read += bytes_rdma_read; transaction_stats->bytes_rdma_write += bytes_rdma_write; - DT_Mdep_Unlock (&transaction_stats->lock); + spin_unlock_irqrestore (&transaction_stats->lock,flags); } Index: test/dapl_server.c =================================================================== --- test/dapl_server.c (revision 2509) +++ test/dapl_server.c (working copy) @@ -49,7 +49,7 @@ unsigned char *buffp = NULL; unsigned char *module = "DT_cs_Server"; int status = 0; - + unsigned long flags; DAT_DTO_COOKIE dto_cookie; struct dat_dto_completion_event_data dto_stat; u32 ret; @@ -616,9 +616,9 @@ /* Count this new client and get ready for the next */ - DT_Mdep_Lock (&ps_ptr->num_clients_lock); + spin_lock_irqsave (&ps_ptr->num_clients_lock,flags); ps_ptr->num_clients++; - DT_Mdep_Unlock (&ps_ptr->num_clients_lock); + spin_unlock_irqrestore (&ps_ptr->num_clients_lock,flags); /* we passed the pt_ptr to the thread and must now 'forget' it */ pt_ptr = NULL; Index: test/dapl_thread.c =================================================================== --- test/dapl_thread.c (revision 2509) +++ test/dapl_thread.c (working copy) @@ -83,6 +83,7 @@ unsigned int stacksize) { Thread *thread_ptr; + unsigned long flags; thread_ptr = (Thread *) DT_MemListAlloc (pt_ptr, "thread.c", THREAD, sizeof (Thread)); if (thread_ptr == NULL) { @@ -93,9 +94,9 @@ thread_ptr->thread_handle = 0; thread_ptr->stacksize = stacksize; - DT_Mdep_Lock (&pt_ptr->Thread_counter_lock); + spin_lock_irqsave (&pt_ptr->Thread_counter_lock,flags); pt_ptr->Thread_counter++; - DT_Mdep_Unlock (&pt_ptr->Thread_counter_lock); + spin_unlock_irqrestore (&pt_ptr->Thread_counter_lock,flags); DT_Mdep_Thread_Init_Attributes (thread_ptr); @@ -108,11 +109,12 @@ void DT_Thread_Destroy (Thread * thread_ptr, Per_Test_Data_t * pt_ptr) { + unsigned long flags; if (thread_ptr) { - DT_Mdep_Lock (&pt_ptr->Thread_counter_lock); + spin_lock_irqsave (&pt_ptr->Thread_counter_lock,flags); pt_ptr->Thread_counter--; - DT_Mdep_Unlock (&pt_ptr->Thread_counter_lock); + spin_unlock_irqrestore (&pt_ptr->Thread_counter_lock,flags); DT_Mdep_Thread_Destroy_Attributes (thread_ptr); DT_MemListFree (pt_ptr, thread_ptr); Index: test/dapl_test_data.c =================================================================== --- test/dapl_test_data.c (revision 2509) +++ test/dapl_test_data.c (working copy) @@ -27,7 +27,7 @@ #include "dapl_proto.h" -DT_Mdep_LockType g_PerfTestLock; +/*DT_Mdep_LockType g_PerfTestLock;*/ /* * check memory leaking int alloc_count; DT_Mdep_LockType * Alloc_Count_Lock; Index: test/dapl_transaction_test.c =================================================================== --- test/dapl_transaction_test.c (revision 2509) +++ test/dapl_transaction_test.c (working copy) @@ -99,7 +99,7 @@ unsigned int i; DT_Tdep_Print_Head *phead; int status = 0; - + unsigned long flags; phead = pt_ptr->Params.phead; pt_ptr->Countdown_Counter = cmd->num_threads; @@ -129,9 +129,9 @@ } DT_Thread_Destroy (pt_ptr->thread, pt_ptr); /* destroy Master thread */ - DT_Mdep_Lock (&pt_ptr->ps_ptr->num_clients_lock); + spin_lock_irqsave (&pt_ptr->ps_ptr->num_clients_lock,flags); pt_ptr->ps_ptr->num_clients--; - DT_Mdep_Unlock (&pt_ptr->ps_ptr->num_clients_lock); + spin_unlock_irqrestore (&pt_ptr->ps_ptr->num_clients_lock,flags); /* NB: Server has no pt_ptr->remote_netaddr */ DT_PrintMemList (pt_ptr); /* check if we return all space allocated */ @@ -239,7 +239,7 @@ enum dat_event_number event_num; DT_Tdep_Print_Head *phead; int status = 0; - + unsigned long flags; pt_ptr = test_ptr->pt_ptr; thread = test_ptr->thread; phead = pt_ptr->Params.phead; @@ -502,7 +502,7 @@ */ if (pt_ptr->local_is_server) { - DT_Mdep_Lock (&pt_ptr->Thread_counter_lock); + spin_lock_irqsave (&pt_ptr->Thread_counter_lock,flags); pt_ptr->Countdown_Counter--; /* Deliberate pre-decrement. Post decrement won't * work here, so don't do it. @@ -512,7 +512,7 @@ DT_Mdep_wait_object_wakeup (&pt_ptr->synch_wait_object); } - DT_Mdep_Unlock (&pt_ptr->Thread_counter_lock); + spin_unlock_irqrestore (&pt_ptr->Thread_counter_lock,flags); } for (i = 0; i < test_ptr->cmd->eps_per_thread; i++) Index: test/dapl_performance_server.c =================================================================== --- test/dapl_performance_server.c (revision 2509) +++ test/dapl_performance_server.c (working copy) @@ -36,7 +36,7 @@ Performance_Test_t *test_ptr = NULL; DT_Tdep_Print_Head *phead; int status = 0; - + unsigned long flags; phead = pt_ptr->Params.phead; DT_Tdep_PT_Debug (1,(phead,"Server: Starting performance test\n")); @@ -75,9 +75,9 @@ DT_Mdep_Thread_Detach (DT_Mdep_Thread_SELF ()); /* AMM */ DT_Thread_Destroy (pt_ptr->thread, pt_ptr); /* destroy Master thread */ - DT_Mdep_Lock (&pt_ptr->ps_ptr->num_clients_lock); + spin_lock_irqsave (&pt_ptr->ps_ptr->num_clients_lock,flags); pt_ptr->ps_ptr->num_clients--; - DT_Mdep_Unlock (&pt_ptr->ps_ptr->num_clients_lock); + spin_unlock_irqrestore (&pt_ptr->ps_ptr->num_clients_lock,flags); DT_PrintMemList (pt_ptr); /* check if we return all space allocated */ DT_Mdep_LockDestroy (&pt_ptr->Thread_counter_lock); Index: test/dapl_memlist.c =================================================================== --- test/dapl_memlist.c (revision 2509) +++ test/dapl_memlist.c (working copy) @@ -42,6 +42,7 @@ { void *buffptr; MemListEntry_t *entry_ptr; + unsigned long flags; buffptr = NULL; entry_ptr = NULL; @@ -64,10 +65,10 @@ entry_ptr->MemType = t; entry_ptr->mem_ptr = buffptr; - DT_Mdep_Lock (&pt_ptr->MemListLock); + spin_lock_irqsave (&pt_ptr->MemListLock,flags); entry_ptr->next = pt_ptr->MemListHead; pt_ptr->MemListHead = entry_ptr; - DT_Mdep_Unlock (&pt_ptr->MemListLock); + spin_unlock_irqrestore (&pt_ptr->MemListLock,flags); return buffptr; } @@ -76,12 +77,13 @@ DT_MemListFree (Per_Test_Data_t * pt_ptr, void *ptr) { MemListEntry_t *pre, *cur; + unsigned long flags; if (pt_ptr == 0) /* not use mem_list */ { DT_Mdep_Free (ptr); return; } - DT_Mdep_Lock (&pt_ptr->MemListLock); + spin_lock_irqsave (&pt_ptr->MemListLock,flags); pre = NULL; cur = pt_ptr->MemListHead; while (cur) @@ -106,7 +108,7 @@ cur = cur->next; } unlock_and_return: - DT_Mdep_Unlock (&pt_ptr->MemListLock); + spin_unlock_irqrestore (&pt_ptr->MemListLock,flags); } void @@ -119,9 +121,9 @@ }; DT_Tdep_Print_Head *phead; MemListEntry_t *cur; - + unsigned long flags; phead = pt_ptr->Params.phead; - DT_Mdep_Lock (&pt_ptr->MemListLock); + spin_lock_irqsave (&pt_ptr->MemListLock,flags); cur = pt_ptr->MemListHead; if (cur != 0) { @@ -133,5 +135,5 @@ cur->filename, type[cur->MemType]); cur = cur->next; } - DT_Mdep_Unlock (&pt_ptr->MemListLock); + spin_unlock_irqrestore (&pt_ptr->MemListLock,flags); } Index: include/dapl_test_data.h =================================================================== --- include/dapl_test_data.h (revision 2509) +++ include/dapl_test_data.h (working copy) @@ -42,7 +42,7 @@ * problems on the server side occasionally * the server will reject connections. */ -extern DT_Mdep_LockType g_PerfTestLock; +/*extern DT_Mdep_LockType g_PerfTestLock; */ /* * check memory leaking extern int alloc_count ; extern Index: kdapl/kdapl_module.c =================================================================== --- kdapl/kdapl_module.c (revision 2509) +++ kdapl/kdapl_module.c (working copy) @@ -95,10 +95,10 @@ break; } - DT_Mdep_Lock(&kdapltest_lock); + spin_lock_irq(&kdapltest_lock); kdapltest_num++; instance = kdapltest_num; - DT_Mdep_Unlock(&kdapltest_lock); + spin_unlock_irq(&kdapltest_lock); local_params->phead = KDT_Print_Alloc(instance); if (!local_params->phead) { @@ -201,7 +201,7 @@ static __init int kdapltest_init(void) { - DT_Mdep_LockInit(&g_PerfTestLock); /* For uDAPL, this is done in Tdep_init() */ + /* DT_Mdep_LockInit(&g_PerfTestLock); For uDAPL, this is done in Tdep_init() */ kdapltest_major = register_chrdev(0, "kdapltest", &kdapltest_fops); DT_Mdep_LockInit(&kdapltest_lock); kdapltest_num = 0; @@ -218,7 +218,7 @@ KDT_Print_Destroy(); KDT_Evd_Destroy(); DT_Mdep_LockDestroy(&kdapltest_lock); - DT_Mdep_LockDestroy(&g_PerfTestLock); /* For uDAPL, this is done in Tdep_end() */ + /* DT_Mdep_LockDestroy(&g_PerfTestLock); For uDAPL, this is done in Tdep_end() */ } module_init(kdapltest_init); Index: kdapl/kdapl_tdep_print.c =================================================================== --- kdapl/kdapl_tdep_print.c (revision 2509) +++ kdapl/kdapl_tdep_print.c (working copy) @@ -106,7 +106,7 @@ va_list args; int len; Tdep_Print_Entry *entry; - + unsigned long flags; entry = DT_Mdep_Malloc (sizeof (Tdep_Print_Entry)); va_start (args, fmt); @@ -120,7 +120,7 @@ } entry->next = NULL; - DT_Mdep_Lock (&phead->lock); + spin_lock_irqsave (&phead->lock,flags); if (phead->head == NULL) { @@ -133,7 +133,7 @@ phead->tail = entry; } DT_Mdep_wait_object_wakeup (&phead->wait_object); - DT_Mdep_Unlock (&phead->lock); + spin_unlock_irqrestore (&phead->lock,flags); } int @@ -143,10 +143,10 @@ DT_Tdep_Print_Head *phead; Tdep_Print_Entry *entry; int rc; - + unsigned long flags; buffer[0] = '\0'; - DT_Mdep_Lock (&DT_Print_List_Lock); + spin_lock_irqsave (&DT_Print_List_Lock,flags); phead = DT_Print_List; while (phead) { @@ -156,7 +156,7 @@ } phead = phead->next; } - DT_Mdep_Unlock (&DT_Print_List_Lock); + spin_unlock_irqrestore (&DT_Print_List_Lock,flags); if (!phead) { @@ -172,7 +172,7 @@ return 0; } - DT_Mdep_Lock (&phead->lock); + spin_lock_irqsave (&phead->lock,flags); entry = phead->head; if (entry) { @@ -184,7 +184,7 @@ strncpy (buffer, entry->buffer, PRINT_MAX); DT_Mdep_Free (entry); } - DT_Mdep_Unlock (&phead->lock); + spin_unlock_irqrestore (&phead->lock,flags); } len = strlen (buffer); Index: kdapl/kdapl_tdep_evd.c =================================================================== --- kdapl/kdapl_tdep_evd.c (revision 2509) +++ kdapl/kdapl_tdep_evd.c (working copy) @@ -105,7 +105,7 @@ u32 dat_status; struct dat_upcall_object upcall; Tdep_Evd *evd_ptr; - + unsigned long flags; evd_ptr = NULL; dat_status = DAT_SUCCESS; @@ -139,10 +139,10 @@ DT_Mdep_wait_object_init (&evd_ptr->wait_object); /* add evd_ptr to front of evd list */ - DT_Mdep_Lock (&DT_Evd_Lock); + spin_lock_irqsave (&DT_Evd_Lock, flags); evd_ptr->evd_next = DT_Evd_List; DT_Evd_List = evd_ptr; - DT_Mdep_Unlock (&DT_Evd_Lock); + spin_unlock_irqrestore (&DT_Evd_Lock, flags); error: if (dat_status != DAT_SUCCESS) @@ -161,7 +161,7 @@ u32 dat_status; Tdep_Evd *evd_ptr; Tdep_Event *event; - + unsigned long flags; dat_status = DAT_SUCCESS; /* find the evd_ptr associated with evd_handle */ @@ -181,7 +181,7 @@ DAT_INTERNAL_ERROR; } /* Get event */ - DT_Mdep_Lock (&DT_Evd_Lock); + spin_lock_irqsave (&DT_Evd_Lock, flags); event = evd_ptr->event_next; /* event present in evd_ptr corresponding to the given evd_handle */ if (event) { @@ -207,7 +207,7 @@ { dat_status = DAT_QUEUE_EMPTY; } - DT_Mdep_Unlock (&DT_Evd_Lock); + spin_unlock_irqrestore (&DT_Evd_Lock, flags); return dat_status; } @@ -261,7 +261,7 @@ } /* Got event */ - DT_Mdep_Lock (&DT_Evd_Lock); + spin_lock_irq (&DT_Evd_Lock); event = evd_ptr->event_next; if (event) { @@ -286,7 +286,7 @@ } } - DT_Mdep_Unlock (&DT_Evd_Lock); + spin_unlock_irq (&DT_Evd_Lock); return dat_status; } @@ -297,7 +297,7 @@ Tdep_Evd *last; last = NULL; - DT_Mdep_Lock (&DT_Evd_Lock); + spin_lock_irq (&DT_Evd_Lock); next = DT_Evd_List; if (next->evd_handle == evd_handle) { @@ -315,7 +315,7 @@ last->evd_next = next->evd_next; } } - DT_Mdep_Unlock (&DT_Evd_Lock); + spin_unlock_irq (&DT_Evd_Lock); DT_Mdep_Free (next); return dat_evd_free(evd_handle); @@ -331,7 +331,7 @@ evd_ptr = (Tdep_Evd *) instance_data; - DT_Mdep_Lock (&DT_Evd_Lock); + spin_lock_irq (&DT_Evd_Lock); if (DT_Event_Free_List) { event = DT_Event_Free_List; @@ -356,6 +356,6 @@ evd_ptr->event_last = event; } } - DT_Mdep_Unlock (&DT_Evd_Lock); + spin_unlock_irq (&DT_Evd_Lock); DT_Mdep_wait_object_wakeup (&evd_ptr->wait_object); } -- Itamar From halr at voltaire.com Sun May 29 08:11:15 2005 From: halr at voltaire.com (Hal Rosenstock) Date: Sun, 29 May 2005 18:11:15 +0300 Subject: [openib-general] oops in ib_mad Message-ID: <5CE025EE7D88BA4599A2C8FEFCF226F5175B61@taurus.voltaire.com> Hi Itamar, This looks similar to the ib_mad oops reported by Tom Duffy. Were these messages are cut and pasted into the email ? Can your email client be set not line wrap these messages ? -- Hal -----Original Message----- From: openib-general-bounces at openib.org on behalf of Itamar Rabenstein Sent: Sun 5/29/2005 6:18 AM To: openib-general at openib.org Subject: [openib-general] oops in ib_mad I hit this oops when trying to run kdapltest x86 + Suse 9.1 + k2.6.11 + svn rev 2507 May 29 12:51:45 mtvs43 kernel: Unable to handle kernel paging request at virtual address 00100104 May 29 12:51:45 mtvs43 kernel: printing eip: May 29 12:51:45 mtvs43 kernel: e09cb56c May 29 12:51:45 mtvs43 kernel: *pde = 00000000 May 29 12:51:45 mtvs43 kernel: Oops: 0002 [#1] May 29 12:51:45 mtvs43 kernel: SMP May 29 12:51:45 mtvs43 kernel: Modules linked in: kdapltest ib_dat_provider ib_cm ib_at dat ib_umad ib_i poib ib_sa edd joydev st sd_mod sr_mod scsi_mod ide_cd cdrom nvram usbserial parport_pc lp parport autof s ipv6 thermal processor fan button battery ac e100 mii ib_mthca ib_mad ib_core i2c_i801 i2c_core hw_ran dom uhci_hcd usbcore evdev dm_mod ext3 jbd May 29 12:51:45 mtvs43 kernel: CPU: 0 May 29 12:51:45 mtvs43 kernel: EIP: 0060:[pg0+542578028/1069171712] Not tainted VLI May 29 12:51:45 mtvs43 kernel: EIP: 0060:[] Not tainted VLI May 29 12:51:45 mtvs43 kernel: EFLAGS: 00010046 (2.6.11-gen2) May 29 12:51:45 mtvs43 kernel: EIP is at ib_mad_complete_send_wr+0x9c/0x160 [ib_mad] May 29 12:51:45 mtvs43 kernel: eax: 00100100 ebx: dbdafd80 ecx: dbdafd8c edx: 00200200 May 29 12:51:45 mtvs43 kernel: esi: c167d180 edi: c167d1b4 ebp: ddbb5f24 esp: ddbb5ee8 May 29 12:51:45 mtvs43 kernel: ds: 007b es: 007b ss: 0068 May 29 12:51:45 mtvs43 kernel: Process ib_mad1 (pid: 2953, threadinfo=ddbb4000 task=c16db550) May 29 12:51:45 mtvs43 kernel: Stack: 00000003 00000246 dbdafd80 dd501910 00000000 dbdafd80 e09cb65b dd501908 May 29 12:51:45 mtvs43 kernel: ddbb5f24 de39d400 deb6e800 dd501800 ddbb5f24 dd501800 e09cb8d9 dc4d6300 May 29 12:51:45 mtvs43 kernel: 00000000 00000000 00000000 ddbb5f58 c011b749 00000000 00000402 00000006 May 29 12:51:45 mtvs43 kernel: Call Trace: May 29 12:51:45 mtvs43 kernel: Call Trace: May 29 12:51:45 mtvs43 kernel: [pg0+542578267/1069171712] ib_mad_send_done_handler+0x2b/0x120 [ib_mad] May 29 12:51:45 mtvs43 kernel: [] ib_mad_send_done_handler+0x2b/0x120 [ib_mad] May 29 12:51:45 mtvs43 kernel: [pg0+542578905/1069171712] ib_mad_completion_handler+0x89/0xa0 [ib_mad] May 29 12:51:45 mtvs43 kernel: [] ib_mad_completion_handler+0x89/0xa0 [ib_mad] May 29 12:51:45 mtvs43 kernel: [__wake_up+41/64] __wake_up+0x29/0x40 May 29 12:51:45 mtvs43 kernel: [] __wake_up+0x29/0x40 May 29 12:51:45 mtvs43 kernel: [worker_thread+424/560] worker_thread+0x1a8/0x230 May 29 12:51:45 mtvs43 kernel: [] worker_thread+0x1a8/0x230 May 29 12:51:45 mtvs43 kernel: [pg0+542578768/1069171712] ib_mad_completion_handler+0x0/0xa0 [ib_mad] May 29 12:51:45 mtvs43 kernel: [] ib_mad_completion_handler+0x0/0xa0 [ib_mad] May 29 12:51:45 mtvs43 kernel: [default_wake_function+0/16] default_wake_function+0x0/0x10 May 29 12:51:45 mtvs43 kernel: [] default_wake_function+0x0/0x10 May 29 12:51:45 mtvs43 kernel: [default_wake_function+0/16] default_wake_function+0x0/0x10 May 29 12:51:45 mtvs43 kernel: [] default_wake_function+0x0/0x10 May 29 12:51:45 mtvs43 kernel: [worker_thread+0/560] worker_thread+0x0/0x230 May 29 12:51:45 mtvs43 kernel: [] worker_thread+0x0/0x230 May 29 12:51:45 mtvs43 kernel: [kthread+136/176] kthread+0x88/0xb0 May 29 12:51:45 mtvs43 kernel: [] kthread+0x88/0xb0 May 29 12:51:45 mtvs43 kernel: [kthread+0/176] kthread+0x0/0xb0 May 29 12:51:45 mtvs43 kernel: [] kthread+0x0/0xb0 May 29 12:51:45 mtvs43 kernel: [kernel_thread_helper+5/16] kernel_thread_helper+0x5/0x10 May 29 12:51:45 mtvs43 kernel: [] kernel_thread_helper+0x5/0x10 May 29 12:51:45 mtvs43 kernel: Code: 24 04 89 f8 59 5b 5b 5e 5f 5d e9 c0 38 94 df 8b 93 90 00 00 00 8d 4 2 ff 89 83 90 00 00 00 85 c0 7f d5 8d 4b 0c 8b 43 0c 8b 51 04 <89> 50 04 c7 41 04 00 02 20 00 89 02 89 f 0 c7 43 0c 00 01 10 00 Itamar Rabenstein Mellanox Technologies Ltd mailto : itamar at mellanox.co.il phone: 972-3-6259506 From mst at mellanox.co.il Sun May 29 10:29:23 2005 From: mst at mellanox.co.il (Michael S. Tsirkin) Date: Sun, 29 May 2005 20:29:23 +0300 Subject: [openib-general] Re: [ANNOUNCE][PATCH] New Linux 2.6.9 backport patches and corresponding userspace tar ball available In-Reply-To: References: Message-ID: <20050529172923.GA3526@mellanox.co.il> Quoting r. Bob Woodruff : > Subject: [ANNOUNCE][PATCH] New Linux 2.6.9 backport patches and corresponding userspace tar ball available > > Hal asked if I could send out my latest backport > patches for 2.6.9 kernels, so here they are. They > are rather big, so I just put them into to SVN where people > can get them. > > The latest ones I have are based on svn2425, but I will > try to track the changes in SVN and provide newer updates > periodically. I'd like to suggest that svn switch or svn copy commands might make it easier to manage the backport and keep it updated. These are described in the subversion book in the chapter about branches. > > Patches are located in the SVN tree under > gen2/trunk/src/linux-kernel/patches/backport-to-2.6.9/ > > infiniband-backport-svn2425-to-2.6.9-kernel-fixups-01.diff > infiniband-backport-svn2425-to-2.6.9-openib-drivers-02.diff > infiniband-backport-svn2425-to-2.6.9-openib-fixups-03.diff > infiniband-backport-svn2425-userspace.tar.gz > > woody > Woody, could you please move these patches to gen2/branches? -- MST - Michael S. Tsirkin From info at yama-no-ibuki.net Sun May 29 23:57:08 2005 From: info at yama-no-ibuki.net (info at yama-no-ibuki.net) Date: 30 May 2005 15:57:08 +0900 Subject: [openib-general] $B!&!&!&$H8@$&$3$H$G$9!#(B Message-ID: <20050530065708.30647.qmail@mail.yama-no-ibuki.net> $B!!A4$F=w at -%9%?%C%U$G1?1D$7$F$$$kEv%5%$%H$O$rF@$F$$$^$9!#CO0hJL$NC4Ev$K1h$C$?CK at -$r>R2p$r(B $B9T$C$F$$$^$7$?$,!"(BGW$BO"5Y$N1F6A$G at .N)?t$,5^A}$7$?$?$a!"NW;~$K(B $BDI2CJg=8$r$7$?$$$H;W$$%a!<%k$r:9$7>e$2$?]$K$J$C$?5.J}MM$OF~2q6b!">R2pNA$O%b%A%m%s$J$/!"(B $B-d3NG'o$KM-Mx$H$J$C$F$*$j$^$9!#(B $B!!62$i$/CK at -2q0w?t$NA}2Cl9g$bM-$j(B $B$^$9$,!"$3$N%a!<%k$NZ$H$J$j(B $B$^$9$N$G!"$40B?42<$5$$!#(B $B!!(B $B=w at -2q0w$NMWK>$K$h$j$4EPO?D>8e!J(B3$BJ,0JFb!K6a=jM%@h=g$G>R2p$r(B $B9T$$$^$9$N$G=w at -$N>pJs(B($B OK thanks. I had another problem in MVAPI. -----Original Message----- From: Roland Dreier [mailto:roland at topspin.com] Sent: Friday, May 27, 2005 7:49 PM To: Eli Cohen Cc: Michael S. Tsirkin; openib-general at openib.org Subject: Re: registering read-only memory Eli> I saw this this on Suse 9.3 and the memory was like: static Eli> const char *my_string = "Hello world"; defined locally in a Eli> function. This produced a VMA with VM_IO set. What kernel version/architecture? I'm having a hard time seeing how it's possible to get a VMA with VM_IO set except by using remap_pfn_range(), and I can't see how ordinary const data from userspace could cause that. - R. -------------- next part -------------- An HTML attachment was scrubbed... URL: From halr at voltaire.com Mon May 30 10:21:38 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 30 May 2005 13:21:38 -0400 Subject: [openib-general] oops in ib_at when running with CONFIG_DEBUG_PAGEALLOC In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35BE1@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35BE1@mtvex01.mtv.mtl.com> Message-ID: <1117473697.4476.6.camel@hal.voltaire.com> On Sun, 2005-05-29 at 06:23, Itamar Rabenstein wrote: > I am running kdapltest -T Q with CONFIG_DEBUG_PAGEALLOC on > and i am geting this oops > > x86 + Suse9.1 + k2.6.11 + svn rev 2507 + kdapl and ib_at from > users/jlentini > > when running without CONFIG_DEBUG_PAGEALLOC i dont see the oops > > > May 29 11:38:24 mtvs42 kernel: Kdapltest Module Loaded - major 254 > May 29 11:38:51 mtvs42 kernel: ib_mthca 0000:04:00.0: Registering > memory at 2c5a8000 (iova 2c5a8000) in PD 5; shift 12, npages 6. > May 29 11:38:52 mtvs42 kernel: Unable to handle kernel paging request > at virtual address eca6af74 > May 29 11:38:52 mtvs42 kernel: printing eip: > May 29 11:38:52 mtvs42 kernel: f8ad33bc > May 29 11:38:52 mtvs42 kernel: *pde = 0047e067 > May 29 11:38:52 mtvs42 kernel: Oops: 0002 [#1] > May 29 11:38:52 mtvs42 kernel: SMP DEBUG_PAGEALLOC > May 29 11:38:52 mtvs42 kernel: Modules linked in: kdapltest > ib_dat_provider ib_cm ib_at dat i > b_umad ib_ipoib ib_sa edd joydev st sd_mod sr_mod scsi_mod ide_cd > cdrom nvram usbserial parpo > rt_pc lp parport autofs ipv6 thermal processor fan button battery ac > e100 mii ib_mthca ib_mad > ib_core i2c_i801 i2c_core hw_random uhci_hcd usbcore evdev dm_mod > ext3 jbd > May 29 11:38:52 mtvs42 kernel: CPU: 0 > May 29 11:38:52 mtvs42 kernel: EIP: > 0060:[pg0+946295740/1069155328] Not tainted VLI > May 29 11:38:52 mtvs42 kernel: EIP: 0060:[] Not > tainted VLI > May 29 11:38:52 mtvs42 kernel: EFLAGS: 00010297 (2.6.11-gen2) > May 29 11:38:52 mtvs42 kernel: EIP is at req_free+0x4c/0x70 [ib_at] > May 29 11:38:52 mtvs42 kernel: eax: 00000001 ebx: eca6af6c ecx: > c1594d40 edx: 00000640 > May 29 11:38:52 mtvs42 kernel: esi: ec6ae7f8 edi: eca6af90 ebp: > eca6af6c esp: f0409f5c > May 29 11:38:52 mtvs42 kernel: ds: 007b es: 007b ss: 0068 > May 29 11:38:52 mtvs42 kernel: Process ib_at_wq/0 (pid: 9619, > threadinfo=f0408000 task=ec8f7ad0) > May 29 11:38:52 mtvs42 kernel: Stack: eca6af8c c012eea8 ec6ae820 > f8ad3470 00000286 f0408000 ec6ae808 ffffffff > May 29 11:38:52 mtvs42 kernel: ffffffff 00000001 00000000 > c011bbd0 00010000 00000000 00000000 ef579f38 > May 29 11:38:52 mtvs42 kernel: ece8ed8c 00000000 ec8f7ad0 > c011bbd0 00100100 00200200 88497fc0 000f428d > May 29 11:38:52 mtvs42 kernel: Call Trace: > May 29 11:38:52 mtvs42 kernel: [worker_thread+424/560] > worker_thread+0x1a8/0x230 > May 29 11:38:52 mtvs42 kernel: [] worker_thread+0x1a8/0x230 > May 29 11:38:52 mtvs42 kernel: [pg0+946295920/1069155328] > req_comp_work+0x0/0x20 [ib_at] > May 29 11:38:52 mtvs42 kernel: [] req_comp_work+0x0/0x20 > [ib_at] > May 29 11:38:52 mtvs42 kernel: [default_wake_function+0/16] > default_wake_function+0x0/0x10 > May 29 11:38:52 mtvs42 kernel: [] > default_wake_function+0x0/0x10 > May 29 11:38:52 mtvs42 kernel: [default_wake_function+0/16] > default_wake_function+0x0/0x10 > May 29 11:38:52 mtvs42 kernel: [] > default_wake_function+0x0/0x10 > May 29 11:38:52 mtvs42 kernel: [worker_thread+0/560] > worker_thread+0x0/0x230 > May 29 11:38:52 mtvs42 kernel: [] worker_thread+0x0/0x230 > May 29 11:38:52 mtvs42 kernel: [kthread+136/176] kthread+0x88/0xb0 > May 29 11:38:52 mtvs42 kernel: [] kthread+0x88/0xb0 > May 29 11:38:52 mtvs42 kernel: [kthread+0/176] kthread+0x0/0xb0 > May 29 11:38:52 mtvs42 kernel: [] kthread+0x0/0xb0 > May 29 11:38:52 mtvs42 kernel: [kernel_thread_helper+5/16] > kernel_thread_helper+0x5/0x10 > May 29 11:38:52 mtvs42 kernel: [] > kernel_thread_helper+0x5/0x10 > May 29 11:38:52 mtvs42 kernel: Code: ad f8 e8 28 ca 64 c7 83 c4 0c c7 > 43 08 00 00 00 00 c7 43 > 0c 00 00 00 00 c7 83 84 00 00 00 00 00 00 00 5b c3 89 d8 e8 34 03 00 > 00 43 08 00 00 00 > 00 c7 43 0c 00 00 00 00 c7 83 84 00 00 00 00 I think this may be an earlier form of the other oops reported by Tom. I think I now see what is causing this. I should have a patch for this in a short while. -- Hal From halr at voltaire.com Mon May 30 10:37:26 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 30 May 2005 13:37:26 -0400 Subject: [openib-general] [PATCH] [ib_at]: Update async structure prior to returning requests to appropriate cache Message-ID: <1117474645.4476.23.camel@hal.voltaire.com> [ib_at]: Update async structure prior to returning requests to appropriate cache. This change affacts req_free, free_route_req, and free_path_req. Also, some other minor changes to eliminate unneeded parameter passed to path_req_output and changes to some DEBUG messages. Signed-off-by: Hal Rosenstock Index: at.c =================================================================== --- at.c (revision 2507) +++ at.c (working copy) @@ -155,7 +155,8 @@ static void free_route_req(void *async); static void free_path_req(void *async); -static void path_req_complete(int stat, struct ib_sa_path_rec *ret, void *ctx); +static void path_req_complete(int status, struct ib_sa_path_rec *resp, + void *context); static int resolve_path(struct path_req *req); static int resolve_ip(struct ib_at_src *src, u32 dst_ip, u32 src_ip, @@ -274,7 +275,6 @@ } memset(dgid, 0, sizeof *dgid); - return 0; } @@ -319,11 +319,10 @@ break; default: WARN("bad async req type %d", pend->type); + pend->status = IB_AT_STATUS_INVALID; + pend->type = IBAT_REQ_NONE; + pend->sa_query = NULL; } - - pend->status = IB_AT_STATUS_INVALID; - pend->type = IBAT_REQ_NONE; - pend->sa_query = NULL; } static int req_start(struct async *q, struct async *pend, @@ -464,6 +463,11 @@ struct route_req *req = container_of(async, struct route_req, pend); DEBUG("free async %p req %p", async, req); + + req->pend.status = IB_AT_STATUS_INVALID; + req->pend.type = IBAT_REQ_NONE; + req->pend.sa_query = NULL; + kmem_cache_free(route_req_cache, req); } @@ -472,6 +476,11 @@ struct path_req *req = container_of(async, struct path_req, pend); DEBUG("free async %p req %p", async, req); + + req->pend.status = IB_AT_STATUS_INVALID; + req->pend.type = IBAT_REQ_NONE; + req->pend.sa_query = NULL; + kmem_cache_free(path_req_cache, req); } @@ -537,15 +546,14 @@ return 1; /* one entry is filled */ } -static int path_req_output(struct path_req *req, struct ib_sa_path_rec *resp, - int npath, struct ib_sa_path_rec *out, int nelem) +static int path_req_output(struct ib_sa_path_rec *resp, int npath, + struct ib_sa_path_rec *out, int nelem) { int n = min(npath, nelem); - DEBUG("parent %p output %d records", req, n); + DEBUG("fill ib_sa_path_rec %p output %d records", out, n); memcpy(out, resp, n * sizeof (struct ib_sa_path_rec)); - return n; } @@ -579,7 +587,7 @@ unsigned long flags; struct async *pend; - DEBUG("req %p", req); + DEBUG("req %p status %d", req, status); if (req->pend.parent) { WARN("for child req %p???", req); @@ -598,12 +606,12 @@ return; } - req->pend.nelem = path_req_output(req, resp, 1, + req->pend.nelem = path_req_output(resp, 1, req->pend.data, req->pend.nelem); spin_lock_irqsave(&pending_reqs.lock, flags); for (pend = req->pend.waiting; pend; pend = pend->waiting) - pend->nelem = path_req_output(req, resp, 1, + pend->nelem = path_req_output(resp, 1, pend->data, pend->nelem); req_end(&req->pend, req->pend.nelem, NULL); @@ -876,7 +884,7 @@ if (in_cache) { DEBUG("!in_cache free req %p", preq); kmem_cache_free(path_req_cache, preq); - return path_req_output(preq, cached_arr, n, path_arr, npath); + return path_req_output(cached_arr, n, path_arr, npath); } */ @@ -969,7 +977,6 @@ EXPORT_SYMBOL(ib_at_status); - /* * Internal init/cleanup functions: */ From halr at voltaire.com Mon May 30 12:09:44 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 30 May 2005 15:09:44 -0400 Subject: [openib-general] Send Side RMPP and OpenSM GetTableResp Message-ID: <1117480184.4476.110.camel@hal.voltaire.com> Hi Sean, When running a Solaris 10 client against OpenSM with RMPP supported turned on, I see the following sequence: SA Client SA SA GetTable ----------------------------> TID 6 SA GetTable ----------------------------> TID 7 SA GetTable ----------------------------> TID 8 <---------------------------- SA GetTableResp RMPP flags 0x07 (Data, First, Last) SegmentNumber 1 PayloadLength 0x8C TID 6 SA GetTable ----------------------------> RMPP flags 0x02 (ACK) SegmentNumber 1 NewWindowLast 6 TID 6 <---------------------------- SA GetTableResp RMPP flags 0x07 (Data, First, Last) SegmentNumber 1 PayloadLength 0x8C TID 7 SA GetTable ----------------------------> RMPP flags 0x02 (ACK) SegmentNumber 1 NewWindowLast 6 TID 7 <---------------------------- SA GetTableResp RMPP flags 0x03 (Data, First) SegmentNumber 1 PayloadLength 0x370 TID 8 SA GetTable ----------------------------> RMPP flags 0x02 (ACK) SegmentNumber 1 NewWindowLast 6 TID 8 <---------------------------- SA GetTableResp RMPP flags 0x01 (Data) SegmentNumber 2 PayloadLength 0x370 TID 8 <---------------------------- SA GetTableResp RMPP flags 0x01 (Data) SegmentNumber 3 PayloadLength 0x370 TID 8 <---------------------------- SA GetTableResp RMPP flags 0x05 (Data, Last) SegmentNumber 4 PayloadLength 0x34 TID 8 SA GetTable ----------------------------> RMPP flags 0x02 (ACK) SegmentNumber 1 NewWindowLast 6 TID 8 <---------------------------- SA GetTableResp RMPP flags 0x01 (Data) SegmentNumber 5 PayloadLength 0x34 TID 8 I presume the reACKing is used as a keep alive so a response timeout (Resp) does not occur. The SA client is using RRespTime of 0xE. The OpenIB side sets this field to 0 (not sure if this affects the SA client side). Some questions on the RMPP sender side (SA): 1. I wouldn't think that reACKing the same segment (1) by the receiver (SA client end) would cause the sender side to send segment 5. 2. In the resend, the header (everything up to SA data) appears to be good but the data appears to be garbage. Have you tested RMPP retransmission ? Have you tested simultaneous transactions in progress ? Thanks. -- Hal From sean.hefty at intel.com Tue May 31 00:51:49 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Tue, 31 May 2005 00:51:49 -0700 Subject: [openib-general] cmpost: failure sending REQ: -22 In-Reply-To: <42992D8C.9090206@linuxmachines.com> Message-ID: >Has anyone seen ib_send_cm_req() return -22? I believe that this is a timeout error, possibly indicating that the server side of the connection wasn't running. You may also want to verify the slid and dlid are correct for your configuration. - Sean From sean.hefty at intel.com Tue May 31 01:27:57 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Tue, 31 May 2005 01:27:57 -0700 Subject: [openib-general] Send Side RMPP and OpenSM GetTableResp In-Reply-To: <1117480184.4476.110.camel@hal.voltaire.com> Message-ID: > <---------------------------- SA GetTableResp > > RMPP flags 0x05 (Data, Last) > SegmentNumber 4 > PayloadLength 0x34 > TID 8 >SA GetTable ----------------------------> >RMPP flags 0x02 (ACK) >SegmentNumber 1 >NewWindowLast 6 >TID 8 This segment number is off - not sure why. It could indicate that segment 2 was lost, or that its processing came after that of a later segment. The RMPP code always updates its SegmentNumber in an ACK based on the last received packet that arrived in order. This ACK should be dropped by the SA side as a duplicate. The SA would then rely on a timeout to resend. Did you ever see an ACK for segment 4? Regardless what went wrong on the SA side, the client needs to be able to deal with it. > <---------------------------- SA GetTableResp > RMPP flags 0x01 (Data) > SegmentNumber 5 > PayloadLength 0x34 > TID 8 This should not occur. The maximum segment number sent should have stayed at 4. I guess one area to check is to make sure that the PayloadLength in the original MAD is set correctly. I do not know what would happen if it were set incorrectly. There could also be an error in how RMPP calculates the number of segments that will be sent. This segment should have been dropped by the client as an invalid segment number. >I presume the reACKing is used as a keep alive so a response timeout >(Resp) does not occur. The SA client is using RRespTime of 0xE. The >OpenIB side sets this field to 0 (not sure if this affects the SA client >side). RMPP is using hard-coded timeouts at this point. It would require path record information to calculate one that's more accurate. >Some questions on the RMPP sender side (SA): > >1. I wouldn't think that reACKing the same segment (1) by the receiver >(SA client end) would cause the sender side to send segment 5. Correct - nothing should cause the SA to send segment 5... >2. In the resend, the header (everything up to SA data) appears to be >good but the data appears to be garbage. This leads me to think that it is an issue with the setting or calculations based on the PayloadLength or sge size. I need to see if the code that calculates the total number of segments to send matches up with the code that determines if a segment should have the last bit set. >Have you tested RMPP retransmission ? Have you tested simultaneous >transactions in progress ? Both of these have been tested. I've forced failures at multiple points in the retransmissions and tested a few thousands simultaneous transactions, and I've never noticed a problem like what you're seeing. But that could very well could be because of limitations to my test program. I will spend a little time tomorrow afternoon looking at this... - Sean From halr at voltaire.com Tue May 31 04:30:47 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 31 May 2005 07:30:47 -0400 Subject: [openib-general] cmpost: failure sending REQ: -22 In-Reply-To: References: Message-ID: <1117539047.4476.199.camel@hal.voltaire.com> On Tue, 2005-05-31 at 03:51, Sean Hefty wrote: > >Has anyone seen ib_send_cm_req() return -22? > > I believe that this is a timeout error, possibly indicating that the server > side of the connection wasn't running. You may also want to verify the slid > and dlid are correct for your configuration. Don't you get a REJ now when there is no one listening on a service ID requested ? -22 is EINVAL. In terms of ib_send_cm_req, it is returned for a number of cases: 1. peer to peer connection is requested 2. No primary path is supplied 3. QP is not RC or UC 4. private data is supplied and length > 92 5. alternate path supplied and PKEY or MTU does not match primary path 6. connection state is not IDLE 7. Primary or alternate path SGID or PKey does not match those of port -- Hal From halr at voltaire.com Tue May 31 04:56:02 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 31 May 2005 07:56:02 -0400 Subject: [openib-general] Send Side RMPP and OpenSM GetTableResp In-Reply-To: References: Message-ID: <1117540561.4476.226.camel@hal.voltaire.com> On Tue, 2005-05-31 at 04:27, Sean Hefty wrote: > > <-- SA GetTableResp > > > > RMPP flags 0x05 (Data, Last) > > SegmentNumber 4 > > PayloadLength 0x34 > > TID 8 > >SA GetTable --> > >RMPP flags 0x02 (ACK) > >SegmentNumber 1 > >NewWindowLast 6 > >TID 8 > > This segment number is off - not sure why. It is off in that the 3 segments just sent are not acknowledged but it is legal to acknowledge what you have already received. This does not violate anything. > It could indicate that segment 2 was lost, That's one possibility but I doubt it is getting lost. > or that its processing came after that of a later segment. This seems more likely. The timing was each packet was slightly less than 20 usec apart until the SA appeared to resend (the gap there was almost almost 770 usec). > The RMPP code always updates its SegmentNumber in an ACK based on the last > received packet that arrived in order. This ACK should be dropped by the SA > side as a duplicate. The SA would then rely on a timeout to resend. The gap was 769.912 usec. Not sure whether this corresponds to any IBTA timeout. Is this the hardcoded timeout that is used ? > Did you ever see an ACK for segment 4? No. > Regardless what went wrong on the SA > side, the client needs to be able to deal with it. This applies in both directions but in this case I think you mean the other direction (whatever went wrong on the SA client side the SA needs to be able to deal with it). > > <-- SA GetTableResp > > RMPP flags 0x01 (Data) > > SegmentNumber 5 > > PayloadLength 0x34 > > TID 8 > > This should not occur. The maximum segment number sent should have stayed > at 4. I guess one area to check is to make sure that the PayloadLength in > the original MAD is set correctly. I do not know what would happen if it > were set incorrectly. There could also be an error in how RMPP calculates > the number of segments that will be sent. It does look like it is trying to resend the last (at least based on the PayloadLength) ? I will find where to instrument this in the code. > This segment should have been dropped by the client as an invalid segment > number. It's not invalid, is it ? Just a repeat. Should it reset one of the RMPP timers too ? > >I presume the reACKing is used as a keep alive so a response timeout > >(Resp) does not occur. The SA client is using RRespTime of 0xE. The > >OpenIB side sets this field to 0 (not sure if this affects the SA client > >side). > > RMPP is using hard-coded timeouts at this point. It would require path > record information to calculate one that's more accurate. It also fills in 0 in RRespTime. Should it fill in something to correspond to the hard coded time it uses ? Or perhaps 32 (0x1F) ? > >Some questions on the RMPP sender side (SA): > > > >1. I wouldn't think that reACKing the same segment (1) by the receiver > >(SA client end) would cause the sender side to send segment 5. > > Correct - nothing should cause the SA to send segment 5... > > >2. In the resend, the header (everything up to SA data) appears to be > >good but the data appears to be garbage. > > This leads me to think that it is an issue with the setting or calculations > based on the PayloadLength or sge size. I need to see if the code that > calculates the total number of segments to send matches up with the code > that determines if a segment should have the last bit set. > > >Have you tested RMPP retransmission ? Have you tested simultaneous > >transactions in progress ? > > Both of these have been tested. I've forced failures at multiple points in > the retransmissions and tested a few thousands simultaneous transactions, > and I've never noticed a problem like what you're seeing. But that could > very well could be because of limitations to my test program. I will spend > a little time tomorrow afternoon looking at this... I will try to get back to gathering more info on this. Thanks. -- Hal From itamar at mellanox.co.il Tue May 31 06:34:21 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Tue, 31 May 2005 16:34:21 +0300 Subject: [openib-general] [PATCH] [ib_at]: Update async structure prio r to returning requests to appropriate cache Message-ID: <91DB792C7985D411BEC300B40080D29CC35BEE@mtvex01.mtv.mtl.com> Thanks Hal, this patch fixed the problem (oops in ib_at.c) Itamar > -----Original Message----- > From: Hal Rosenstock [mailto:halr at voltaire.com] > Sent: Monday, May 30, 2005 8:37 PM > To: James Lentini > Cc: openib-general at openib.org > Subject: [openib-general] [PATCH] [ib_at]: Update async > structure prior > to returning requests to appropriate cache > > > [ib_at]: Update async structure prior to returning requests to > appropriate cache. This change affacts req_free, free_route_req, and > free_path_req. > > Also, some other minor changes to eliminate unneeded > parameter passed to > path_req_output and changes to some DEBUG messages. > > Signed-off-by: Hal Rosenstock > > Index: at.c > =================================================================== > --- at.c (revision 2507) > +++ at.c (working copy) > @@ -155,7 +155,8 @@ > > static void free_route_req(void *async); > static void free_path_req(void *async); > -static void path_req_complete(int stat, struct > ib_sa_path_rec *ret, void *ctx); > +static void path_req_complete(int status, struct > ib_sa_path_rec *resp, > + void *context); > static int resolve_path(struct path_req *req); > > static int resolve_ip(struct ib_at_src *src, u32 dst_ip, u32 src_ip, > @@ -274,7 +275,6 @@ > } > > memset(dgid, 0, sizeof *dgid); > - > return 0; > } > > @@ -319,11 +319,10 @@ > break; > default: > WARN("bad async req type %d", pend->type); > + pend->status = IB_AT_STATUS_INVALID; > + pend->type = IBAT_REQ_NONE; > + pend->sa_query = NULL; > } > - > - pend->status = IB_AT_STATUS_INVALID; > - pend->type = IBAT_REQ_NONE; > - pend->sa_query = NULL; > } > > static int req_start(struct async *q, struct async *pend, > @@ -464,6 +463,11 @@ > struct route_req *req = container_of(async, struct > route_req, pend); > > DEBUG("free async %p req %p", async, req); > + > + req->pend.status = IB_AT_STATUS_INVALID; > + req->pend.type = IBAT_REQ_NONE; > + req->pend.sa_query = NULL; > + > kmem_cache_free(route_req_cache, req); > } > > @@ -472,6 +476,11 @@ > struct path_req *req = container_of(async, struct > path_req, pend); > > DEBUG("free async %p req %p", async, req); > + > + req->pend.status = IB_AT_STATUS_INVALID; > + req->pend.type = IBAT_REQ_NONE; > + req->pend.sa_query = NULL; > + > kmem_cache_free(path_req_cache, req); > } > > @@ -537,15 +546,14 @@ > return 1; /* one entry is filled */ > } > > -static int path_req_output(struct path_req *req, struct > ib_sa_path_rec *resp, > - int npath, struct ib_sa_path_rec > *out, int nelem) > +static int path_req_output(struct ib_sa_path_rec *resp, int npath, > + struct ib_sa_path_rec *out, int nelem) > { > int n = min(npath, nelem); > > - DEBUG("parent %p output %d records", req, n); > + DEBUG("fill ib_sa_path_rec %p output %d records", out, n); > > memcpy(out, resp, n * sizeof (struct ib_sa_path_rec)); > - > return n; > } > > @@ -579,7 +587,7 @@ > unsigned long flags; > struct async *pend; > > - DEBUG("req %p", req); > + DEBUG("req %p status %d", req, status); > > if (req->pend.parent) { > WARN("for child req %p???", req); > @@ -598,12 +606,12 @@ > return; > } > > - req->pend.nelem = path_req_output(req, resp, 1, > + req->pend.nelem = path_req_output(resp, 1, > req->pend.data, > req->pend.nelem); > > spin_lock_irqsave(&pending_reqs.lock, flags); > for (pend = req->pend.waiting; pend; pend = pend->waiting) > - pend->nelem = path_req_output(req, resp, 1, > + pend->nelem = path_req_output(resp, 1, > pend->data, pend->nelem); > > req_end(&req->pend, req->pend.nelem, NULL); > @@ -876,7 +884,7 @@ > if (in_cache) { > DEBUG("!in_cache free req %p", preq); > kmem_cache_free(path_req_cache, preq); > - return path_req_output(preq, cached_arr, n, > path_arr, npath); > + return path_req_output(cached_arr, n, path_arr, npath); > } > */ > > @@ -969,7 +977,6 @@ > EXPORT_SYMBOL(ib_at_status); > > > - > /* > * Internal init/cleanup functions: > */ > > > > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From jlentini at netapp.com Tue May 31 07:46:16 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 31 May 2005 10:46:16 -0400 (EDT) Subject: [openib-general] Re: [PATCH][kdapl] replace spin_lock with spin_lock_irqsave in kdapltest In-Reply-To: <20050529121029.GA16745@mellanox.co.il> References: <20050529121029.GA16745@mellanox.co.il> Message-ID: Itamar, Why does this patch comment out uses of the g_PerfTestLock? james On Sun, 29 May 2005, Itamar wrote: itamar> With this patch i can run kdapltest -T T ... -t 4 -w 8 ... itamar> I still see problems but in general this patch helps the stability a lot. itamar> itamar> replace spin_lock with spin_lock_irqsave in kdapltest itamar> Signed-off-by: Itamar Rabenstein itamar> itamar> Index: test/dapl_transaction_stats.c itamar> =================================================================== itamar> --- test/dapl_transaction_stats.c (revision 2509) itamar> +++ test/dapl_transaction_stats.c (working copy) itamar> @@ -45,12 +45,13 @@ itamar> DT_transaction_stats_set_ready (DT_Tdep_Print_Head *phead, itamar> Transaction_Stats_t * transaction_stats) itamar> { itamar> - DT_Mdep_Lock (&transaction_stats->lock); itamar> + unsigned long flags; itamar> + spin_lock_irqsave (&transaction_stats->lock,flags); itamar> transaction_stats->wait_count--; itamar> itamar> DT_Tdep_PT_Debug (1,(phead,"Received Sync Message from server (%d left)\n", itamar> transaction_stats->wait_count)); itamar> - DT_Mdep_Unlock (&transaction_stats->lock); itamar> + spin_unlock_irqrestore (&transaction_stats->lock,flags); itamar> } itamar> itamar> boolean_t itamar> @@ -86,7 +87,8 @@ itamar> unsigned int bytes_rdma_read, itamar> unsigned int bytes_rdma_write) itamar> { itamar> - DT_Mdep_Lock (&transaction_stats->lock); itamar> + unsigned long flags; itamar> + spin_lock_irqsave (&transaction_stats->lock,flags); itamar> itamar> /* look for the longest time... */ itamar> if (time_ms > transaction_stats->time_ms) itamar> @@ -99,5 +101,5 @@ itamar> transaction_stats->bytes_recv += bytes_recv; itamar> transaction_stats->bytes_rdma_read += bytes_rdma_read; itamar> transaction_stats->bytes_rdma_write += bytes_rdma_write; itamar> - DT_Mdep_Unlock (&transaction_stats->lock); itamar> + spin_unlock_irqrestore (&transaction_stats->lock,flags); itamar> } itamar> Index: test/dapl_server.c itamar> =================================================================== itamar> --- test/dapl_server.c (revision 2509) itamar> +++ test/dapl_server.c (working copy) itamar> @@ -49,7 +49,7 @@ itamar> unsigned char *buffp = NULL; itamar> unsigned char *module = "DT_cs_Server"; itamar> int status = 0; itamar> - itamar> + unsigned long flags; itamar> DAT_DTO_COOKIE dto_cookie; itamar> struct dat_dto_completion_event_data dto_stat; itamar> u32 ret; itamar> @@ -616,9 +616,9 @@ itamar> itamar> itamar> /* Count this new client and get ready for the next */ itamar> - DT_Mdep_Lock (&ps_ptr->num_clients_lock); itamar> + spin_lock_irqsave (&ps_ptr->num_clients_lock,flags); itamar> ps_ptr->num_clients++; itamar> - DT_Mdep_Unlock (&ps_ptr->num_clients_lock); itamar> + spin_unlock_irqrestore (&ps_ptr->num_clients_lock,flags); itamar> itamar> /* we passed the pt_ptr to the thread and must now 'forget' it */ itamar> pt_ptr = NULL; itamar> Index: test/dapl_thread.c itamar> =================================================================== itamar> --- test/dapl_thread.c (revision 2509) itamar> +++ test/dapl_thread.c (working copy) itamar> @@ -83,6 +83,7 @@ itamar> unsigned int stacksize) itamar> { itamar> Thread *thread_ptr; itamar> + unsigned long flags; itamar> thread_ptr = (Thread *) DT_MemListAlloc (pt_ptr, "thread.c", THREAD, sizeof (Thread)); itamar> if (thread_ptr == NULL) itamar> { itamar> @@ -93,9 +94,9 @@ itamar> thread_ptr->thread_handle = 0; itamar> thread_ptr->stacksize = stacksize; itamar> itamar> - DT_Mdep_Lock (&pt_ptr->Thread_counter_lock); itamar> + spin_lock_irqsave (&pt_ptr->Thread_counter_lock,flags); itamar> pt_ptr->Thread_counter++; itamar> - DT_Mdep_Unlock (&pt_ptr->Thread_counter_lock); itamar> + spin_unlock_irqrestore (&pt_ptr->Thread_counter_lock,flags); itamar> itamar> DT_Mdep_Thread_Init_Attributes (thread_ptr); itamar> itamar> @@ -108,11 +109,12 @@ itamar> void itamar> DT_Thread_Destroy (Thread * thread_ptr, Per_Test_Data_t * pt_ptr) itamar> { itamar> + unsigned long flags; itamar> if (thread_ptr) itamar> { itamar> - DT_Mdep_Lock (&pt_ptr->Thread_counter_lock); itamar> + spin_lock_irqsave (&pt_ptr->Thread_counter_lock,flags); itamar> pt_ptr->Thread_counter--; itamar> - DT_Mdep_Unlock (&pt_ptr->Thread_counter_lock); itamar> + spin_unlock_irqrestore (&pt_ptr->Thread_counter_lock,flags); itamar> itamar> DT_Mdep_Thread_Destroy_Attributes (thread_ptr); itamar> DT_MemListFree (pt_ptr, thread_ptr); itamar> Index: test/dapl_test_data.c itamar> =================================================================== itamar> --- test/dapl_test_data.c (revision 2509) itamar> +++ test/dapl_test_data.c (working copy) itamar> @@ -27,7 +27,7 @@ itamar> itamar> #include "dapl_proto.h" itamar> itamar> -DT_Mdep_LockType g_PerfTestLock; itamar> +/*DT_Mdep_LockType g_PerfTestLock;*/ itamar> /* itamar> * check memory leaking int alloc_count; DT_Mdep_LockType itamar> * Alloc_Count_Lock; itamar> Index: test/dapl_transaction_test.c itamar> =================================================================== itamar> --- test/dapl_transaction_test.c (revision 2509) itamar> +++ test/dapl_transaction_test.c (working copy) itamar> @@ -99,7 +99,7 @@ itamar> unsigned int i; itamar> DT_Tdep_Print_Head *phead; itamar> int status = 0; itamar> - itamar> + unsigned long flags; itamar> phead = pt_ptr->Params.phead; itamar> itamar> pt_ptr->Countdown_Counter = cmd->num_threads; itamar> @@ -129,9 +129,9 @@ itamar> } itamar> DT_Thread_Destroy (pt_ptr->thread, pt_ptr); /* destroy Master thread */ itamar> itamar> - DT_Mdep_Lock (&pt_ptr->ps_ptr->num_clients_lock); itamar> + spin_lock_irqsave (&pt_ptr->ps_ptr->num_clients_lock,flags); itamar> pt_ptr->ps_ptr->num_clients--; itamar> - DT_Mdep_Unlock (&pt_ptr->ps_ptr->num_clients_lock); itamar> + spin_unlock_irqrestore (&pt_ptr->ps_ptr->num_clients_lock,flags); itamar> itamar> /* NB: Server has no pt_ptr->remote_netaddr */ itamar> DT_PrintMemList (pt_ptr); /* check if we return all space allocated */ itamar> @@ -239,7 +239,7 @@ itamar> enum dat_event_number event_num; itamar> DT_Tdep_Print_Head *phead; itamar> int status = 0; itamar> - itamar> + unsigned long flags; itamar> pt_ptr = test_ptr->pt_ptr; itamar> thread = test_ptr->thread; itamar> phead = pt_ptr->Params.phead; itamar> @@ -502,7 +502,7 @@ itamar> */ itamar> if (pt_ptr->local_is_server) itamar> { itamar> - DT_Mdep_Lock (&pt_ptr->Thread_counter_lock); itamar> + spin_lock_irqsave (&pt_ptr->Thread_counter_lock,flags); itamar> pt_ptr->Countdown_Counter--; itamar> /* Deliberate pre-decrement. Post decrement won't itamar> * work here, so don't do it. itamar> @@ -512,7 +512,7 @@ itamar> DT_Mdep_wait_object_wakeup (&pt_ptr->synch_wait_object); itamar> } itamar> itamar> - DT_Mdep_Unlock (&pt_ptr->Thread_counter_lock); itamar> + spin_unlock_irqrestore (&pt_ptr->Thread_counter_lock,flags); itamar> } itamar> itamar> for (i = 0; i < test_ptr->cmd->eps_per_thread; i++) itamar> Index: test/dapl_performance_server.c itamar> =================================================================== itamar> --- test/dapl_performance_server.c (revision 2509) itamar> +++ test/dapl_performance_server.c (working copy) itamar> @@ -36,7 +36,7 @@ itamar> Performance_Test_t *test_ptr = NULL; itamar> DT_Tdep_Print_Head *phead; itamar> int status = 0; itamar> - itamar> + unsigned long flags; itamar> phead = pt_ptr->Params.phead; itamar> DT_Tdep_PT_Debug (1,(phead,"Server: Starting performance test\n")); itamar> itamar> @@ -75,9 +75,9 @@ itamar> DT_Mdep_Thread_Detach (DT_Mdep_Thread_SELF ()); /* AMM */ itamar> DT_Thread_Destroy (pt_ptr->thread, pt_ptr); /* destroy Master thread */ itamar> itamar> - DT_Mdep_Lock (&pt_ptr->ps_ptr->num_clients_lock); itamar> + spin_lock_irqsave (&pt_ptr->ps_ptr->num_clients_lock,flags); itamar> pt_ptr->ps_ptr->num_clients--; itamar> - DT_Mdep_Unlock (&pt_ptr->ps_ptr->num_clients_lock); itamar> + spin_unlock_irqrestore (&pt_ptr->ps_ptr->num_clients_lock,flags); itamar> itamar> DT_PrintMemList (pt_ptr); /* check if we return all space allocated */ itamar> DT_Mdep_LockDestroy (&pt_ptr->Thread_counter_lock); itamar> Index: test/dapl_memlist.c itamar> =================================================================== itamar> --- test/dapl_memlist.c (revision 2509) itamar> +++ test/dapl_memlist.c (working copy) itamar> @@ -42,6 +42,7 @@ itamar> { itamar> void *buffptr; itamar> MemListEntry_t *entry_ptr; itamar> + unsigned long flags; itamar> buffptr = NULL; itamar> entry_ptr = NULL; itamar> itamar> @@ -64,10 +65,10 @@ itamar> entry_ptr->MemType = t; itamar> entry_ptr->mem_ptr = buffptr; itamar> itamar> - DT_Mdep_Lock (&pt_ptr->MemListLock); itamar> + spin_lock_irqsave (&pt_ptr->MemListLock,flags); itamar> entry_ptr->next = pt_ptr->MemListHead; itamar> pt_ptr->MemListHead = entry_ptr; itamar> - DT_Mdep_Unlock (&pt_ptr->MemListLock); itamar> + spin_unlock_irqrestore (&pt_ptr->MemListLock,flags); itamar> itamar> return buffptr; itamar> } itamar> @@ -76,12 +77,13 @@ itamar> DT_MemListFree (Per_Test_Data_t * pt_ptr, void *ptr) itamar> { itamar> MemListEntry_t *pre, *cur; itamar> + unsigned long flags; itamar> if (pt_ptr == 0) /* not use mem_list */ itamar> { itamar> DT_Mdep_Free (ptr); itamar> return; itamar> } itamar> - DT_Mdep_Lock (&pt_ptr->MemListLock); itamar> + spin_lock_irqsave (&pt_ptr->MemListLock,flags); itamar> pre = NULL; itamar> cur = pt_ptr->MemListHead; itamar> while (cur) itamar> @@ -106,7 +108,7 @@ itamar> cur = cur->next; itamar> } itamar> unlock_and_return: itamar> - DT_Mdep_Unlock (&pt_ptr->MemListLock); itamar> + spin_unlock_irqrestore (&pt_ptr->MemListLock,flags); itamar> } itamar> itamar> void itamar> @@ -119,9 +121,9 @@ itamar> }; itamar> DT_Tdep_Print_Head *phead; itamar> MemListEntry_t *cur; itamar> - itamar> + unsigned long flags; itamar> phead = pt_ptr->Params.phead; itamar> - DT_Mdep_Lock (&pt_ptr->MemListLock); itamar> + spin_lock_irqsave (&pt_ptr->MemListLock,flags); itamar> cur = pt_ptr->MemListHead; itamar> if (cur != 0) itamar> { itamar> @@ -133,5 +135,5 @@ itamar> cur->filename, type[cur->MemType]); itamar> cur = cur->next; itamar> } itamar> - DT_Mdep_Unlock (&pt_ptr->MemListLock); itamar> + spin_unlock_irqrestore (&pt_ptr->MemListLock,flags); itamar> } itamar> Index: include/dapl_test_data.h itamar> =================================================================== itamar> --- include/dapl_test_data.h (revision 2509) itamar> +++ include/dapl_test_data.h (working copy) itamar> @@ -42,7 +42,7 @@ itamar> * problems on the server side occasionally itamar> * the server will reject connections. itamar> */ itamar> -extern DT_Mdep_LockType g_PerfTestLock; itamar> +/*extern DT_Mdep_LockType g_PerfTestLock; */ itamar> itamar> /* itamar> * check memory leaking extern int alloc_count ; extern itamar> Index: kdapl/kdapl_module.c itamar> =================================================================== itamar> --- kdapl/kdapl_module.c (revision 2509) itamar> +++ kdapl/kdapl_module.c (working copy) itamar> @@ -95,10 +95,10 @@ itamar> break; itamar> } itamar> itamar> - DT_Mdep_Lock(&kdapltest_lock); itamar> + spin_lock_irq(&kdapltest_lock); itamar> kdapltest_num++; itamar> instance = kdapltest_num; itamar> - DT_Mdep_Unlock(&kdapltest_lock); itamar> + spin_unlock_irq(&kdapltest_lock); itamar> itamar> local_params->phead = KDT_Print_Alloc(instance); itamar> if (!local_params->phead) { itamar> @@ -201,7 +201,7 @@ itamar> static __init int kdapltest_init(void) itamar> { itamar> itamar> - DT_Mdep_LockInit(&g_PerfTestLock); /* For uDAPL, this is done in Tdep_init() */ itamar> + /* DT_Mdep_LockInit(&g_PerfTestLock); For uDAPL, this is done in Tdep_init() */ itamar> kdapltest_major = register_chrdev(0, "kdapltest", &kdapltest_fops); itamar> DT_Mdep_LockInit(&kdapltest_lock); itamar> kdapltest_num = 0; itamar> @@ -218,7 +218,7 @@ itamar> KDT_Print_Destroy(); itamar> KDT_Evd_Destroy(); itamar> DT_Mdep_LockDestroy(&kdapltest_lock); itamar> - DT_Mdep_LockDestroy(&g_PerfTestLock); /* For uDAPL, this is done in Tdep_end() */ itamar> + /* DT_Mdep_LockDestroy(&g_PerfTestLock); For uDAPL, this is done in Tdep_end() */ itamar> } itamar> itamar> module_init(kdapltest_init); itamar> Index: kdapl/kdapl_tdep_print.c itamar> =================================================================== itamar> --- kdapl/kdapl_tdep_print.c (revision 2509) itamar> +++ kdapl/kdapl_tdep_print.c (working copy) itamar> @@ -106,7 +106,7 @@ itamar> va_list args; itamar> int len; itamar> Tdep_Print_Entry *entry; itamar> - itamar> + unsigned long flags; itamar> entry = DT_Mdep_Malloc (sizeof (Tdep_Print_Entry)); itamar> itamar> va_start (args, fmt); itamar> @@ -120,7 +120,7 @@ itamar> } itamar> entry->next = NULL; itamar> itamar> - DT_Mdep_Lock (&phead->lock); itamar> + spin_lock_irqsave (&phead->lock,flags); itamar> itamar> if (phead->head == NULL) itamar> { itamar> @@ -133,7 +133,7 @@ itamar> phead->tail = entry; itamar> } itamar> DT_Mdep_wait_object_wakeup (&phead->wait_object); itamar> - DT_Mdep_Unlock (&phead->lock); itamar> + spin_unlock_irqrestore (&phead->lock,flags); itamar> } itamar> itamar> int itamar> @@ -143,10 +143,10 @@ itamar> DT_Tdep_Print_Head *phead; itamar> Tdep_Print_Entry *entry; itamar> int rc; itamar> - itamar> + unsigned long flags; itamar> buffer[0] = '\0'; itamar> itamar> - DT_Mdep_Lock (&DT_Print_List_Lock); itamar> + spin_lock_irqsave (&DT_Print_List_Lock,flags); itamar> phead = DT_Print_List; itamar> while (phead) itamar> { itamar> @@ -156,7 +156,7 @@ itamar> } itamar> phead = phead->next; itamar> } itamar> - DT_Mdep_Unlock (&DT_Print_List_Lock); itamar> + spin_unlock_irqrestore (&DT_Print_List_Lock,flags); itamar> itamar> if (!phead) itamar> { itamar> @@ -172,7 +172,7 @@ itamar> return 0; itamar> } itamar> itamar> - DT_Mdep_Lock (&phead->lock); itamar> + spin_lock_irqsave (&phead->lock,flags); itamar> entry = phead->head; itamar> if (entry) itamar> { itamar> @@ -184,7 +184,7 @@ itamar> strncpy (buffer, entry->buffer, PRINT_MAX); itamar> DT_Mdep_Free (entry); itamar> } itamar> - DT_Mdep_Unlock (&phead->lock); itamar> + spin_unlock_irqrestore (&phead->lock,flags); itamar> } itamar> itamar> len = strlen (buffer); itamar> Index: kdapl/kdapl_tdep_evd.c itamar> =================================================================== itamar> --- kdapl/kdapl_tdep_evd.c (revision 2509) itamar> +++ kdapl/kdapl_tdep_evd.c (working copy) itamar> @@ -105,7 +105,7 @@ itamar> u32 dat_status; itamar> struct dat_upcall_object upcall; itamar> Tdep_Evd *evd_ptr; itamar> - itamar> + unsigned long flags; itamar> evd_ptr = NULL; itamar> dat_status = DAT_SUCCESS; itamar> itamar> @@ -139,10 +139,10 @@ itamar> DT_Mdep_wait_object_init (&evd_ptr->wait_object); itamar> itamar> /* add evd_ptr to front of evd list */ itamar> - DT_Mdep_Lock (&DT_Evd_Lock); itamar> + spin_lock_irqsave (&DT_Evd_Lock, flags); itamar> evd_ptr->evd_next = DT_Evd_List; itamar> DT_Evd_List = evd_ptr; itamar> - DT_Mdep_Unlock (&DT_Evd_Lock); itamar> + spin_unlock_irqrestore (&DT_Evd_Lock, flags); itamar> itamar> error: itamar> if (dat_status != DAT_SUCCESS) itamar> @@ -161,7 +161,7 @@ itamar> u32 dat_status; itamar> Tdep_Evd *evd_ptr; itamar> Tdep_Event *event; itamar> - itamar> + unsigned long flags; itamar> dat_status = DAT_SUCCESS; itamar> itamar> /* find the evd_ptr associated with evd_handle */ itamar> @@ -181,7 +181,7 @@ itamar> DAT_INTERNAL_ERROR; itamar> } itamar> /* Get event */ itamar> - DT_Mdep_Lock (&DT_Evd_Lock); itamar> + spin_lock_irqsave (&DT_Evd_Lock, flags); itamar> event = evd_ptr->event_next; /* event present in evd_ptr corresponding to the given evd_handle */ itamar> if (event) itamar> { itamar> @@ -207,7 +207,7 @@ itamar> { itamar> dat_status = DAT_QUEUE_EMPTY; itamar> } itamar> - DT_Mdep_Unlock (&DT_Evd_Lock); itamar> + spin_unlock_irqrestore (&DT_Evd_Lock, flags); itamar> itamar> return dat_status; itamar> } itamar> @@ -261,7 +261,7 @@ itamar> } itamar> itamar> /* Got event */ itamar> - DT_Mdep_Lock (&DT_Evd_Lock); itamar> + spin_lock_irq (&DT_Evd_Lock); itamar> event = evd_ptr->event_next; itamar> if (event) itamar> { itamar> @@ -286,7 +286,7 @@ itamar> } itamar> } itamar> itamar> - DT_Mdep_Unlock (&DT_Evd_Lock); itamar> + spin_unlock_irq (&DT_Evd_Lock); itamar> return dat_status; itamar> } itamar> itamar> @@ -297,7 +297,7 @@ itamar> Tdep_Evd *last; itamar> itamar> last = NULL; itamar> - DT_Mdep_Lock (&DT_Evd_Lock); itamar> + spin_lock_irq (&DT_Evd_Lock); itamar> next = DT_Evd_List; itamar> if (next->evd_handle == evd_handle) itamar> { itamar> @@ -315,7 +315,7 @@ itamar> last->evd_next = next->evd_next; itamar> } itamar> } itamar> - DT_Mdep_Unlock (&DT_Evd_Lock); itamar> + spin_unlock_irq (&DT_Evd_Lock); itamar> itamar> DT_Mdep_Free (next); itamar> return dat_evd_free(evd_handle); itamar> @@ -331,7 +331,7 @@ itamar> itamar> evd_ptr = (Tdep_Evd *) instance_data; itamar> itamar> - DT_Mdep_Lock (&DT_Evd_Lock); itamar> + spin_lock_irq (&DT_Evd_Lock); itamar> if (DT_Event_Free_List) itamar> { itamar> event = DT_Event_Free_List; itamar> @@ -356,6 +356,6 @@ itamar> evd_ptr->event_last = event; itamar> } itamar> } itamar> - DT_Mdep_Unlock (&DT_Evd_Lock); itamar> + spin_unlock_irq (&DT_Evd_Lock); itamar> DT_Mdep_wait_object_wakeup (&evd_ptr->wait_object); itamar> } itamar> -- itamar> Itamar itamar> From glebn at voltaire.com Tue May 31 07:59:17 2005 From: glebn at voltaire.com (Gleb Natapov) Date: Tue, 31 May 2005 17:59:17 +0300 Subject: [openib-general] Problem compiling userspace driver. Message-ID: <20050531145917.GB15647@minantech.com> Hello, I am trying to compile libmthca but I get following error: src/mthca.c:101: error: unknown field `query_gid' specified in initializer src/mthca.c:101: warning: initialization from incompatible pointer type src/mthca.c:102: error: unknown field `query_pkey' specified in initializer src/mthca.c:102: warning: initialization from incompatible pointer type Those fields indeed are missing in verbs.h. If I remove those two lines driver compiles but when I run ibv_devices I get: libibverbs: Warning: no userspace device-specific driver found for uverbs0 driver search path: /home/glebn/OpenIB/install/lib/infiniband $ ls /home/glebn/OpenIB/install/lib/infiniband mthca.a mthca.la mthca.so Any help? -- Gleb. From halr at voltaire.com Tue May 31 08:09:58 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 31 May 2005 11:09:58 -0400 Subject: [openib-general] Problem compiling userspace driver. In-Reply-To: <20050531145917.GB15647@minantech.com> References: <20050531145917.GB15647@minantech.com> Message-ID: <1117552198.4477.4.camel@hal.voltaire.com> On Tue, 2005-05-31 at 10:59, Gleb Natapov wrote: > Hello, > > I am trying to compile libmthca but I get following error: > src/mthca.c:101: error: unknown field `query_gid' specified in initializer > src/mthca.c:101: warning: initialization from incompatible pointer type > src/mthca.c:102: error: unknown field `query_pkey' specified in initializer > src/mthca.c:102: warning: initialization from incompatible pointer type Also: src/mthca.c:115: unknown field `attach_mcast' specified in initializer src/mthca.c:115: warning: excess elements in struct initializer src/mthca.c:115: warning: (near initialization for `mthca_ctx_ops') src/mthca.c:116: unknown field `detach_mcast' specified in initializer src/mthca.c:117: warning: excess elements in struct initializer > Those fields indeed are missing in verbs.h. > > If I remove those two lines driver compiles but when I run ibv_devices I > get: > libibverbs: Warning: no userspace device-specific driver found for > uverbs0 > driver search path: /home/glebn/OpenIB/install/lib/infiniband > > $ ls /home/glebn/OpenIB/install/lib/infiniband > mthca.a mthca.la mthca.so Did you modprobe ib_uverbs ? -- Hal > Any help? > > -- > Gleb. > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general From glebn at voltaire.com Tue May 31 08:18:32 2005 From: glebn at voltaire.com (Gleb Natapov) Date: Tue, 31 May 2005 18:18:32 +0300 Subject: ***SPAM:***Re: [openib-general] Problem compiling userspace driver. In-Reply-To: <1117552198.4477.4.camel@hal.voltaire.com> References: <20050531145917.GB15647@minantech.com> <1117552198.4477.4.camel@hal.voltaire.com> Message-ID: <20050531151832.GC15647@minantech.com> On Tue, May 31, 2005 at 11:09:58AM -0400, Hal Rosenstock wrote: > On Tue, 2005-05-31 at 10:59, Gleb Natapov wrote: > > Hello, > > > > I am trying to compile libmthca but I get following error: > > src/mthca.c:101: error: unknown field `query_gid' specified in initializer > > src/mthca.c:101: warning: initialization from incompatible pointer type > > src/mthca.c:102: error: unknown field `query_pkey' specified in initializer > > src/mthca.c:102: warning: initialization from incompatible pointer type > > Also: > src/mthca.c:115: unknown field `attach_mcast' specified in initializer > src/mthca.c:115: warning: excess elements in struct initializer > src/mthca.c:115: warning: (near initialization for `mthca_ctx_ops') > src/mthca.c:116: unknown field `detach_mcast' specified in initializer > src/mthca.c:117: warning: excess elements in struct initializer > Right, but those only warnings. > > Those fields indeed are missing in verbs.h. > > > > If I remove those two lines driver compiles but when I run ibv_devices I > > get: > > libibverbs: Warning: no userspace device-specific driver found for > > uverbs0 > > driver search path: /home/glebn/OpenIB/install/lib/infiniband > > > > $ ls /home/glebn/OpenIB/install/lib/infiniband > > mthca.a mthca.la mthca.so > > Did you modprobe ib_uverbs ? Yes. I had another error before I did this. -- Gleb. From jlentini at netapp.com Tue May 31 08:30:35 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 31 May 2005 11:30:35 -0400 (EDT) Subject: [openib-general] Re: [PATCH] [ib_at]: Update async structure prior to returning requests to appropriate cache In-Reply-To: <1117474645.4476.23.camel@hal.voltaire.com> References: <1117474645.4476.23.camel@hal.voltaire.com> Message-ID: Committed in revision 2513. On Mon, 30 May 2005, Hal Rosenstock wrote: halr> [ib_at]: Update async structure prior to returning requests to halr> appropriate cache. This change affacts req_free, free_route_req, and halr> free_path_req. halr> halr> Also, some other minor changes to eliminate unneeded parameter passed to halr> path_req_output and changes to some DEBUG messages. halr> halr> Signed-off-by: Hal Rosenstock halr> halr> Index: at.c halr> =================================================================== halr> --- at.c (revision 2507) halr> +++ at.c (working copy) halr> @@ -155,7 +155,8 @@ halr> halr> static void free_route_req(void *async); halr> static void free_path_req(void *async); halr> -static void path_req_complete(int stat, struct ib_sa_path_rec *ret, void *ctx); halr> +static void path_req_complete(int status, struct ib_sa_path_rec *resp, halr> + void *context); halr> static int resolve_path(struct path_req *req); halr> halr> static int resolve_ip(struct ib_at_src *src, u32 dst_ip, u32 src_ip, halr> @@ -274,7 +275,6 @@ halr> } halr> halr> memset(dgid, 0, sizeof *dgid); halr> - halr> return 0; halr> } halr> halr> @@ -319,11 +319,10 @@ halr> break; halr> default: halr> WARN("bad async req type %d", pend->type); halr> + pend->status = IB_AT_STATUS_INVALID; halr> + pend->type = IBAT_REQ_NONE; halr> + pend->sa_query = NULL; halr> } halr> - halr> - pend->status = IB_AT_STATUS_INVALID; halr> - pend->type = IBAT_REQ_NONE; halr> - pend->sa_query = NULL; halr> } halr> halr> static int req_start(struct async *q, struct async *pend, halr> @@ -464,6 +463,11 @@ halr> struct route_req *req = container_of(async, struct route_req, pend); halr> halr> DEBUG("free async %p req %p", async, req); halr> + halr> + req->pend.status = IB_AT_STATUS_INVALID; halr> + req->pend.type = IBAT_REQ_NONE; halr> + req->pend.sa_query = NULL; halr> + halr> kmem_cache_free(route_req_cache, req); halr> } halr> halr> @@ -472,6 +476,11 @@ halr> struct path_req *req = container_of(async, struct path_req, pend); halr> halr> DEBUG("free async %p req %p", async, req); halr> + halr> + req->pend.status = IB_AT_STATUS_INVALID; halr> + req->pend.type = IBAT_REQ_NONE; halr> + req->pend.sa_query = NULL; halr> + halr> kmem_cache_free(path_req_cache, req); halr> } halr> halr> @@ -537,15 +546,14 @@ halr> return 1; /* one entry is filled */ halr> } halr> halr> -static int path_req_output(struct path_req *req, struct ib_sa_path_rec *resp, halr> - int npath, struct ib_sa_path_rec *out, int nelem) halr> +static int path_req_output(struct ib_sa_path_rec *resp, int npath, halr> + struct ib_sa_path_rec *out, int nelem) halr> { halr> int n = min(npath, nelem); halr> halr> - DEBUG("parent %p output %d records", req, n); halr> + DEBUG("fill ib_sa_path_rec %p output %d records", out, n); halr> halr> memcpy(out, resp, n * sizeof (struct ib_sa_path_rec)); halr> - halr> return n; halr> } halr> halr> @@ -579,7 +587,7 @@ halr> unsigned long flags; halr> struct async *pend; halr> halr> - DEBUG("req %p", req); halr> + DEBUG("req %p status %d", req, status); halr> halr> if (req->pend.parent) { halr> WARN("for child req %p???", req); halr> @@ -598,12 +606,12 @@ halr> return; halr> } halr> halr> - req->pend.nelem = path_req_output(req, resp, 1, halr> + req->pend.nelem = path_req_output(resp, 1, halr> req->pend.data, req->pend.nelem); halr> halr> spin_lock_irqsave(&pending_reqs.lock, flags); halr> for (pend = req->pend.waiting; pend; pend = pend->waiting) halr> - pend->nelem = path_req_output(req, resp, 1, halr> + pend->nelem = path_req_output(resp, 1, halr> pend->data, pend->nelem); halr> halr> req_end(&req->pend, req->pend.nelem, NULL); halr> @@ -876,7 +884,7 @@ halr> if (in_cache) { halr> DEBUG("!in_cache free req %p", preq); halr> kmem_cache_free(path_req_cache, preq); halr> - return path_req_output(preq, cached_arr, n, path_arr, npath); halr> + return path_req_output(cached_arr, n, path_arr, npath); halr> } halr> */ halr> halr> @@ -969,7 +977,6 @@ halr> EXPORT_SYMBOL(ib_at_status); halr> halr> halr> - halr> /* halr> * Internal init/cleanup functions: halr> */ halr> halr> halr> From jlentini at netapp.com Tue May 31 08:32:20 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 31 May 2005 11:32:20 -0400 (EDT) Subject: [openib-general] Re: [PATCH][kdapl] fix fatal bug in triger the evd upcall In-Reply-To: <20050529114338.GA16560@mellanox.co.il> References: <20050529114338.GA16560@mellanox.co.il> Message-ID: Committed in revision 2514. On Sun, 29 May 2005, Itamar wrote: itamar> Hi James, itamar> itamar> This patch fix a fatal bug that exist in current lastet bits in kdapl (svn rev 2507) itamar> As you can see we need to triger the upcall when dapl_evd_dequeue return with good status itamar> and quit the method when dapl_evd_dequeue return with non zero status which mean queue is empty. itamar> In the current bits no kdapltest can run even the simple quit test. itamar> itamar> Please in the future before you commit changes to the svn run a simple regression. itamar> Any way with this patch the code is working again. itamar> itamar> fix fatal bug in triger the evd upcall itamar> Signed-off-by: Itamar Rabenstein itamar> itamar> Index: dapl_cno_util.c itamar> =================================================================== itamar> --- dapl_cno_util.c (revision 2509) itamar> +++ dapl_cno_util.c (working copy) itamar> @@ -115,12 +115,8 @@ itamar> itamar> for (;;) { itamar> status = dapl_evd_dequeue((DAT_EVD_HANDLE)evd, &event); itamar> - if (DAT_SUCCESS == status) { itamar> - dapl_dbg_log(DAPL_DBG_TYPE_ERR, itamar> - "dapl_evd_dequeue failed: %x\n", status); itamar> + if (DAT_SUCCESS != status) itamar> return; itamar> - } itamar> - itamar> cno->cno_upcall.upcall_func(cno->cno_upcall.instance_data, itamar> &event, FALSE); itamar> } itamar> -- itamar> Itamar itamar> From itamar at mellanox.co.il Tue May 31 08:54:47 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Tue, 31 May 2005 18:54:47 +0300 Subject: [openib-general] RE: [PATCH][kdapl] replace spin_lock with spin_lock_irqsave in kd apltest Message-ID: <91DB792C7985D411BEC300B40080D29CC35BEF@mtvex01.mtv.mtl.com> it is only declared not in use so we dont need it (;-) Itamar > -----Original Message----- > From: James Lentini [mailto:jlentini at netapp.com] > Sent: Tuesday, May 31, 2005 5:46 PM > To: Itamar > Cc: openib-general > Subject: Re: [PATCH][kdapl] replace spin_lock with > spin_lock_irqsave in > kdapltest > > > > Itamar, > > Why does this patch comment out uses of the g_PerfTestLock? > > james > > On Sun, 29 May 2005, Itamar wrote: > > itamar> With this patch i can run kdapltest -T T ... -t 4 -w 8 ... > itamar> I still see problems but in general this patch helps > the stability a lot. > itamar> > itamar> replace spin_lock with spin_lock_irqsave in kdapltest > itamar> Signed-off-by: Itamar Rabenstein > itamar> > itamar> Index: test/dapl_transaction_stats.c > itamar> > =================================================================== > itamar> --- test/dapl_transaction_stats.c (revision 2509) > itamar> +++ test/dapl_transaction_stats.c (working copy) > itamar> @@ -45,12 +45,13 @@ > itamar> DT_transaction_stats_set_ready (DT_Tdep_Print_Head *phead, > itamar> Transaction_Stats_t * > transaction_stats) > itamar> { > itamar> - DT_Mdep_Lock (&transaction_stats->lock); > itamar> + unsigned long flags; > itamar> + spin_lock_irqsave (&transaction_stats->lock,flags); > itamar> transaction_stats->wait_count--; > itamar> > itamar> DT_Tdep_PT_Debug (1,(phead,"Received Sync > Message from server (%d left)\n", > itamar> transaction_stats->wait_count)); > itamar> - DT_Mdep_Unlock (&transaction_stats->lock); > itamar> + spin_unlock_irqrestore (&transaction_stats->lock,flags); > itamar> } > itamar> > itamar> boolean_t > itamar> @@ -86,7 +87,8 @@ > itamar> unsigned int bytes_rdma_read, > itamar> unsigned int bytes_rdma_write) > itamar> { > itamar> - DT_Mdep_Lock (&transaction_stats->lock); > itamar> + unsigned long flags; > itamar> + spin_lock_irqsave (&transaction_stats->lock,flags); > itamar> > itamar> /* look for the longest time... */ > itamar> if (time_ms > transaction_stats->time_ms) > itamar> @@ -99,5 +101,5 @@ > itamar> transaction_stats->bytes_recv += bytes_recv; > itamar> transaction_stats->bytes_rdma_read += bytes_rdma_read; > itamar> transaction_stats->bytes_rdma_write += bytes_rdma_write; > itamar> - DT_Mdep_Unlock (&transaction_stats->lock); > itamar> + spin_unlock_irqrestore (&transaction_stats->lock,flags); > itamar> } > itamar> Index: test/dapl_server.c > itamar> > =================================================================== > itamar> --- test/dapl_server.c (revision 2509) > itamar> +++ test/dapl_server.c (working copy) > itamar> @@ -49,7 +49,7 @@ > itamar> unsigned char *buffp = NULL; > itamar> unsigned char *module = "DT_cs_Server"; > itamar> int status = 0; > itamar> - > itamar> + unsigned long flags; > itamar> DAT_DTO_COOKIE dto_cookie; > itamar> struct dat_dto_completion_event_data dto_stat; > itamar> u32 ret; > itamar> @@ -616,9 +616,9 @@ > itamar> > itamar> > itamar> /* Count this new client and get ready for the next */ > itamar> - DT_Mdep_Lock (&ps_ptr->num_clients_lock); > itamar> + spin_lock_irqsave (&ps_ptr->num_clients_lock,flags); > itamar> ps_ptr->num_clients++; > itamar> - DT_Mdep_Unlock (&ps_ptr->num_clients_lock); > itamar> + spin_unlock_irqrestore > (&ps_ptr->num_clients_lock,flags); > itamar> > itamar> /* we passed the pt_ptr to the thread and must > now 'forget' it */ > itamar> pt_ptr = NULL; > itamar> Index: test/dapl_thread.c > itamar> > =================================================================== > itamar> --- test/dapl_thread.c (revision 2509) > itamar> +++ test/dapl_thread.c (working copy) > itamar> @@ -83,6 +83,7 @@ > itamar> unsigned int stacksize) > itamar> { > itamar> Thread *thread_ptr; > itamar> + unsigned long flags; > itamar> thread_ptr = (Thread *) DT_MemListAlloc (pt_ptr, > "thread.c", THREAD, sizeof (Thread)); > itamar> if (thread_ptr == NULL) > itamar> { > itamar> @@ -93,9 +94,9 @@ > itamar> thread_ptr->thread_handle = 0; > itamar> thread_ptr->stacksize = stacksize; > itamar> > itamar> - DT_Mdep_Lock (&pt_ptr->Thread_counter_lock); > itamar> + spin_lock_irqsave (&pt_ptr->Thread_counter_lock,flags); > itamar> pt_ptr->Thread_counter++; > itamar> - DT_Mdep_Unlock (&pt_ptr->Thread_counter_lock); > itamar> + spin_unlock_irqrestore > (&pt_ptr->Thread_counter_lock,flags); > itamar> > itamar> DT_Mdep_Thread_Init_Attributes (thread_ptr); > itamar> > itamar> @@ -108,11 +109,12 @@ > itamar> void > itamar> DT_Thread_Destroy (Thread * thread_ptr, > Per_Test_Data_t * pt_ptr) > itamar> { > itamar> + unsigned long flags; > itamar> if (thread_ptr) > itamar> { > itamar> - DT_Mdep_Lock (&pt_ptr->Thread_counter_lock); > itamar> + spin_lock_irqsave (&pt_ptr->Thread_counter_lock,flags); > itamar> pt_ptr->Thread_counter--; > itamar> - DT_Mdep_Unlock (&pt_ptr->Thread_counter_lock); > itamar> + spin_unlock_irqrestore > (&pt_ptr->Thread_counter_lock,flags); > itamar> > itamar> DT_Mdep_Thread_Destroy_Attributes (thread_ptr); > itamar> DT_MemListFree (pt_ptr, thread_ptr); > itamar> Index: test/dapl_test_data.c > itamar> > =================================================================== > itamar> --- test/dapl_test_data.c (revision 2509) > itamar> +++ test/dapl_test_data.c (working copy) > itamar> @@ -27,7 +27,7 @@ > itamar> > itamar> #include "dapl_proto.h" > itamar> > itamar> -DT_Mdep_LockType g_PerfTestLock; > itamar> +/*DT_Mdep_LockType g_PerfTestLock;*/ > itamar> /* > itamar> * check memory leaking int alloc_count; > DT_Mdep_LockType > itamar> * Alloc_Count_Lock; > itamar> Index: test/dapl_transaction_test.c > itamar> > =================================================================== > itamar> --- test/dapl_transaction_test.c (revision 2509) > itamar> +++ test/dapl_transaction_test.c (working copy) > itamar> @@ -99,7 +99,7 @@ > itamar> unsigned int i; > itamar> DT_Tdep_Print_Head *phead; > itamar> int status = 0; > itamar> - > itamar> + unsigned long flags; > itamar> phead = pt_ptr->Params.phead; > itamar> > itamar> pt_ptr->Countdown_Counter = cmd->num_threads; > itamar> @@ -129,9 +129,9 @@ > itamar> } > itamar> DT_Thread_Destroy (pt_ptr->thread, pt_ptr); > /* destroy Master thread */ > itamar> > itamar> - DT_Mdep_Lock (&pt_ptr->ps_ptr->num_clients_lock); > itamar> + spin_lock_irqsave > (&pt_ptr->ps_ptr->num_clients_lock,flags); > itamar> pt_ptr->ps_ptr->num_clients--; > itamar> - DT_Mdep_Unlock (&pt_ptr->ps_ptr->num_clients_lock); > itamar> + spin_unlock_irqrestore > (&pt_ptr->ps_ptr->num_clients_lock,flags); > itamar> > itamar> /* NB: Server has no pt_ptr->remote_netaddr */ > itamar> DT_PrintMemList (pt_ptr); /* check if > we return all space allocated */ > itamar> @@ -239,7 +239,7 @@ > itamar> enum dat_event_number event_num; > itamar> DT_Tdep_Print_Head *phead; > itamar> int status = 0; > itamar> - > itamar> + unsigned long flags; > itamar> pt_ptr = test_ptr->pt_ptr; > itamar> thread = test_ptr->thread; > itamar> phead = pt_ptr->Params.phead; > itamar> @@ -502,7 +502,7 @@ > itamar> */ > itamar> if (pt_ptr->local_is_server) > itamar> { > itamar> - DT_Mdep_Lock (&pt_ptr->Thread_counter_lock); > itamar> + spin_lock_irqsave (&pt_ptr->Thread_counter_lock,flags); > itamar> pt_ptr->Countdown_Counter--; > itamar> /* Deliberate pre-decrement. Post decrement won't > itamar> * work here, so don't do it. > itamar> @@ -512,7 +512,7 @@ > itamar> DT_Mdep_wait_object_wakeup > (&pt_ptr->synch_wait_object); > itamar> } > itamar> > itamar> - DT_Mdep_Unlock (&pt_ptr->Thread_counter_lock); > itamar> + spin_unlock_irqrestore > (&pt_ptr->Thread_counter_lock,flags); > itamar> } > itamar> > itamar> for (i = 0; i < test_ptr->cmd->eps_per_thread; i++) > itamar> Index: test/dapl_performance_server.c > itamar> > =================================================================== > itamar> --- test/dapl_performance_server.c (revision 2509) > itamar> +++ test/dapl_performance_server.c (working copy) > itamar> @@ -36,7 +36,7 @@ > itamar> Performance_Test_t *test_ptr = NULL; > itamar> DT_Tdep_Print_Head *phead; > itamar> int status = 0; > itamar> - > itamar> + unsigned long flags; > itamar> phead = pt_ptr->Params.phead; > itamar> DT_Tdep_PT_Debug (1,(phead,"Server: Starting > performance test\n")); > itamar> > itamar> @@ -75,9 +75,9 @@ > itamar> DT_Mdep_Thread_Detach (DT_Mdep_Thread_SELF ()); > /* AMM */ > itamar> DT_Thread_Destroy (pt_ptr->thread, pt_ptr); > /* destroy Master thread */ > itamar> > itamar> - DT_Mdep_Lock (&pt_ptr->ps_ptr->num_clients_lock); > itamar> + spin_lock_irqsave > (&pt_ptr->ps_ptr->num_clients_lock,flags); > itamar> pt_ptr->ps_ptr->num_clients--; > itamar> - DT_Mdep_Unlock (&pt_ptr->ps_ptr->num_clients_lock); > itamar> + spin_unlock_irqrestore > (&pt_ptr->ps_ptr->num_clients_lock,flags); > itamar> > itamar> DT_PrintMemList (pt_ptr); /* check if > we return all space allocated */ > itamar> DT_Mdep_LockDestroy (&pt_ptr->Thread_counter_lock); > itamar> Index: test/dapl_memlist.c > itamar> > =================================================================== > itamar> --- test/dapl_memlist.c (revision 2509) > itamar> +++ test/dapl_memlist.c (working copy) > itamar> @@ -42,6 +42,7 @@ > itamar> { > itamar> void *buffptr; > itamar> MemListEntry_t *entry_ptr; > itamar> + unsigned long flags; > itamar> buffptr = NULL; > itamar> entry_ptr = NULL; > itamar> > itamar> @@ -64,10 +65,10 @@ > itamar> entry_ptr->MemType = t; > itamar> entry_ptr->mem_ptr = buffptr; > itamar> > itamar> - DT_Mdep_Lock (&pt_ptr->MemListLock); > itamar> + spin_lock_irqsave (&pt_ptr->MemListLock,flags); > itamar> entry_ptr->next = pt_ptr->MemListHead; > itamar> pt_ptr->MemListHead = entry_ptr; > itamar> - DT_Mdep_Unlock (&pt_ptr->MemListLock); > itamar> + spin_unlock_irqrestore (&pt_ptr->MemListLock,flags); > itamar> > itamar> return buffptr; > itamar> } > itamar> @@ -76,12 +77,13 @@ > itamar> DT_MemListFree (Per_Test_Data_t * pt_ptr, void *ptr) > itamar> { > itamar> MemListEntry_t *pre, *cur; > itamar> + unsigned long flags; > itamar> if (pt_ptr == 0) /* not use mem_list */ > itamar> { > itamar> DT_Mdep_Free (ptr); > itamar> return; > itamar> } > itamar> - DT_Mdep_Lock (&pt_ptr->MemListLock); > itamar> + spin_lock_irqsave (&pt_ptr->MemListLock,flags); > itamar> pre = NULL; > itamar> cur = pt_ptr->MemListHead; > itamar> while (cur) > itamar> @@ -106,7 +108,7 @@ > itamar> cur = cur->next; > itamar> } > itamar> unlock_and_return: > itamar> - DT_Mdep_Unlock (&pt_ptr->MemListLock); > itamar> + spin_unlock_irqrestore (&pt_ptr->MemListLock,flags); > itamar> } > itamar> > itamar> void > itamar> @@ -119,9 +121,9 @@ > itamar> }; > itamar> DT_Tdep_Print_Head *phead; > itamar> MemListEntry_t *cur; > itamar> - > itamar> + unsigned long flags; > itamar> phead = pt_ptr->Params.phead; > itamar> - DT_Mdep_Lock (&pt_ptr->MemListLock); > itamar> + spin_lock_irqsave (&pt_ptr->MemListLock,flags); > itamar> cur = pt_ptr->MemListHead; > itamar> if (cur != 0) > itamar> { > itamar> @@ -133,5 +135,5 @@ > itamar> cur->filename, type[cur->MemType]); > itamar> cur = cur->next; > itamar> } > itamar> - DT_Mdep_Unlock (&pt_ptr->MemListLock); > itamar> + spin_unlock_irqrestore (&pt_ptr->MemListLock,flags); > itamar> } > itamar> Index: include/dapl_test_data.h > itamar> > =================================================================== > itamar> --- include/dapl_test_data.h (revision 2509) > itamar> +++ include/dapl_test_data.h (working copy) > itamar> @@ -42,7 +42,7 @@ > itamar> * problems on the server side occasionally > itamar> * the server will reject connections. > itamar> */ > itamar> -extern DT_Mdep_LockType g_PerfTestLock; > itamar> +/*extern DT_Mdep_LockType g_PerfTestLock; */ > itamar> > itamar> /* > itamar> * check memory leaking extern int > alloc_count ; extern > itamar> Index: kdapl/kdapl_module.c > itamar> > =================================================================== > itamar> --- kdapl/kdapl_module.c (revision 2509) > itamar> +++ kdapl/kdapl_module.c (working copy) > itamar> @@ -95,10 +95,10 @@ > itamar> break; > itamar> } > itamar> > itamar> - DT_Mdep_Lock(&kdapltest_lock); > itamar> + spin_lock_irq(&kdapltest_lock); > itamar> kdapltest_num++; > itamar> instance = kdapltest_num; > itamar> - DT_Mdep_Unlock(&kdapltest_lock); > itamar> + spin_unlock_irq(&kdapltest_lock); > itamar> > itamar> local_params->phead = > KDT_Print_Alloc(instance); > itamar> if (!local_params->phead) { > itamar> @@ -201,7 +201,7 @@ > itamar> static __init int kdapltest_init(void) > itamar> { > itamar> > itamar> - DT_Mdep_LockInit(&g_PerfTestLock); /* For > uDAPL, this is done in Tdep_init() */ > itamar> + /* DT_Mdep_LockInit(&g_PerfTestLock); For > uDAPL, this is done in Tdep_init() */ > itamar> kdapltest_major = register_chrdev(0, > "kdapltest", &kdapltest_fops); > itamar> DT_Mdep_LockInit(&kdapltest_lock); > itamar> kdapltest_num = 0; > itamar> @@ -218,7 +218,7 @@ > itamar> KDT_Print_Destroy(); > itamar> KDT_Evd_Destroy(); > itamar> DT_Mdep_LockDestroy(&kdapltest_lock); > itamar> - DT_Mdep_LockDestroy(&g_PerfTestLock); /* For > uDAPL, this is done in Tdep_end() */ > itamar> + /* DT_Mdep_LockDestroy(&g_PerfTestLock); For > uDAPL, this is done in Tdep_end() */ > itamar> } > itamar> > itamar> module_init(kdapltest_init); > itamar> Index: kdapl/kdapl_tdep_print.c > itamar> > =================================================================== > itamar> --- kdapl/kdapl_tdep_print.c (revision 2509) > itamar> +++ kdapl/kdapl_tdep_print.c (working copy) > itamar> @@ -106,7 +106,7 @@ > itamar> va_list args; > itamar> int len; > itamar> Tdep_Print_Entry *entry; > itamar> - > itamar> + unsigned long flags; > itamar> entry = DT_Mdep_Malloc (sizeof (Tdep_Print_Entry)); > itamar> > itamar> va_start (args, fmt); > itamar> @@ -120,7 +120,7 @@ > itamar> } > itamar> entry->next = NULL; > itamar> > itamar> - DT_Mdep_Lock (&phead->lock); > itamar> + spin_lock_irqsave (&phead->lock,flags); > itamar> > itamar> if (phead->head == NULL) > itamar> { > itamar> @@ -133,7 +133,7 @@ > itamar> phead->tail = entry; > itamar> } > itamar> DT_Mdep_wait_object_wakeup (&phead->wait_object); > itamar> - DT_Mdep_Unlock (&phead->lock); > itamar> + spin_unlock_irqrestore (&phead->lock,flags); > itamar> } > itamar> > itamar> int > itamar> @@ -143,10 +143,10 @@ > itamar> DT_Tdep_Print_Head *phead; > itamar> Tdep_Print_Entry *entry; > itamar> int rc; > itamar> - > itamar> + unsigned long flags; > itamar> buffer[0] = '\0'; > itamar> > itamar> - DT_Mdep_Lock (&DT_Print_List_Lock); > itamar> + spin_lock_irqsave (&DT_Print_List_Lock,flags); > itamar> phead = DT_Print_List; > itamar> while (phead) > itamar> { > itamar> @@ -156,7 +156,7 @@ > itamar> } > itamar> phead = phead->next; > itamar> } > itamar> - DT_Mdep_Unlock (&DT_Print_List_Lock); > itamar> + spin_unlock_irqrestore (&DT_Print_List_Lock,flags); > itamar> > itamar> if (!phead) > itamar> { > itamar> @@ -172,7 +172,7 @@ > itamar> return 0; > itamar> } > itamar> > itamar> - DT_Mdep_Lock (&phead->lock); > itamar> + spin_lock_irqsave (&phead->lock,flags); > itamar> entry = phead->head; > itamar> if (entry) > itamar> { > itamar> @@ -184,7 +184,7 @@ > itamar> strncpy (buffer, entry->buffer, PRINT_MAX); > itamar> DT_Mdep_Free (entry); > itamar> } > itamar> - DT_Mdep_Unlock (&phead->lock); > itamar> + spin_unlock_irqrestore (&phead->lock,flags); > itamar> } > itamar> > itamar> len = strlen (buffer); > itamar> Index: kdapl/kdapl_tdep_evd.c > itamar> > =================================================================== > itamar> --- kdapl/kdapl_tdep_evd.c (revision 2509) > itamar> +++ kdapl/kdapl_tdep_evd.c (working copy) > itamar> @@ -105,7 +105,7 @@ > itamar> u32 dat_status; > itamar> struct dat_upcall_object upcall; > itamar> Tdep_Evd *evd_ptr; > itamar> - > itamar> + unsigned long flags; > itamar> evd_ptr = NULL; > itamar> dat_status = DAT_SUCCESS; > itamar> > itamar> @@ -139,10 +139,10 @@ > itamar> DT_Mdep_wait_object_init (&evd_ptr->wait_object); > itamar> > itamar> /* add evd_ptr to front of evd list */ > itamar> - DT_Mdep_Lock (&DT_Evd_Lock); > itamar> + spin_lock_irqsave (&DT_Evd_Lock, flags); > itamar> evd_ptr->evd_next = DT_Evd_List; > itamar> DT_Evd_List = evd_ptr; > itamar> - DT_Mdep_Unlock (&DT_Evd_Lock); > itamar> + spin_unlock_irqrestore (&DT_Evd_Lock, flags); > itamar> > itamar> error: > itamar> if (dat_status != DAT_SUCCESS) > itamar> @@ -161,7 +161,7 @@ > itamar> u32 dat_status; > itamar> Tdep_Evd *evd_ptr; > itamar> Tdep_Event *event; > itamar> - > itamar> + unsigned long flags; > itamar> dat_status = DAT_SUCCESS; > itamar> > itamar> /* find the evd_ptr associated with evd_handle */ > itamar> @@ -181,7 +181,7 @@ > itamar> DAT_INTERNAL_ERROR; > itamar> } > itamar> /* Get event */ > itamar> - DT_Mdep_Lock (&DT_Evd_Lock); > itamar> + spin_lock_irqsave (&DT_Evd_Lock, flags); > itamar> event = evd_ptr->event_next; /* event present in > evd_ptr corresponding to the given evd_handle */ > itamar> if (event) > itamar> { > itamar> @@ -207,7 +207,7 @@ > itamar> { > itamar> dat_status = DAT_QUEUE_EMPTY; > itamar> } > itamar> - DT_Mdep_Unlock (&DT_Evd_Lock); > itamar> + spin_unlock_irqrestore (&DT_Evd_Lock, flags); > itamar> > itamar> return dat_status; > itamar> } > itamar> @@ -261,7 +261,7 @@ > itamar> } > itamar> > itamar> /* Got event */ > itamar> - DT_Mdep_Lock (&DT_Evd_Lock); > itamar> + spin_lock_irq (&DT_Evd_Lock); > itamar> event = evd_ptr->event_next; > itamar> if (event) > itamar> { > itamar> @@ -286,7 +286,7 @@ > itamar> } > itamar> } > itamar> > itamar> - DT_Mdep_Unlock (&DT_Evd_Lock); > itamar> + spin_unlock_irq (&DT_Evd_Lock); > itamar> return dat_status; > itamar> } > itamar> > itamar> @@ -297,7 +297,7 @@ > itamar> Tdep_Evd *last; > itamar> > itamar> last = NULL; > itamar> - DT_Mdep_Lock (&DT_Evd_Lock); > itamar> + spin_lock_irq (&DT_Evd_Lock); > itamar> next = DT_Evd_List; > itamar> if (next->evd_handle == evd_handle) > itamar> { > itamar> @@ -315,7 +315,7 @@ > itamar> last->evd_next = next->evd_next; > itamar> } > itamar> } > itamar> - DT_Mdep_Unlock (&DT_Evd_Lock); > itamar> + spin_unlock_irq (&DT_Evd_Lock); > itamar> > itamar> DT_Mdep_Free (next); > itamar> return dat_evd_free(evd_handle); > itamar> @@ -331,7 +331,7 @@ > itamar> > itamar> evd_ptr = (Tdep_Evd *) instance_data; > itamar> > itamar> - DT_Mdep_Lock (&DT_Evd_Lock); > itamar> + spin_lock_irq (&DT_Evd_Lock); > itamar> if (DT_Event_Free_List) > itamar> { > itamar> event = DT_Event_Free_List; > itamar> @@ -356,6 +356,6 @@ > itamar> evd_ptr->event_last = event; > itamar> } > itamar> } > itamar> - DT_Mdep_Unlock (&DT_Evd_Lock); > itamar> + spin_unlock_irq (&DT_Evd_Lock); > itamar> DT_Mdep_wait_object_wakeup (&evd_ptr->wait_object); > itamar> } > itamar> -- > itamar> Itamar > itamar> > From jlentini at netapp.com Tue May 31 08:39:16 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 31 May 2005 11:39:16 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: remove typedef DAT_CONTEXT In-Reply-To: <1117225871.24117.11.camel@duffman> References: <1117225871.24117.11.camel@duffman> Message-ID: Mostly committed in revision 2515. I didn't remove DAT_UPCALL_NULL and DAT_UPCALL_SAME. DAT_UPCALL_NULL is provided as a convenience to the consumer. I think it is useful, but I'm willing to hear other opinions. The provider's implementation of dat_evd_modify_upcall() should check for the DAT_UPCALL_SAME value. The fact that it doesn't is a bug. james On Fri, 27 May 2005, Tom Duffy wrote: tduffy> Get rid of the typedef DAT_CONTEXT. tduffy> tduffy> Signed-off-by: Tom Duffy tduffy> tduffy> Index: linux-kernel/test/dapltest/include/dapl_common.h tduffy> =================================================================== tduffy> --- linux-kernel/test/dapltest/include/dapl_common.h (revision 2506) tduffy> +++ linux-kernel/test/dapltest/include/dapl_common.h (working copy) tduffy> @@ -42,7 +42,7 @@ typedef enum tduffy> typedef struct tduffy> { tduffy> DAT_RMR_CONTEXT rmr_context; tduffy> - DAT_CONTEXT mem_address; tduffy> + union dat_context mem_address; tduffy> } RemoteMemoryInfo; tduffy> #pragma pack() tduffy> tduffy> Index: linux-kernel/dat-provider/dapl_get_consumer_context.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_get_consumer_context.c (revision 2506) tduffy> +++ linux-kernel/dat-provider/dapl_get_consumer_context.c (working copy) tduffy> @@ -48,7 +48,7 @@ tduffy> * DAT_SUCCESS tduffy> * DAT_INVALID_PARAMETER tduffy> */ tduffy> -u32 dapl_get_consumer_context(DAT_HANDLE dat_handle, DAT_CONTEXT *context) tduffy> +u32 dapl_get_consumer_context(DAT_HANDLE dat_handle, union dat_context *context) tduffy> { tduffy> u32 dat_status = DAT_SUCCESS; tduffy> struct dapl_header *header; tduffy> Index: linux-kernel/dat-provider/dapl_set_consumer_context.c tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl_set_consumer_context.c (revision 2506) tduffy> +++ linux-kernel/dat-provider/dapl_set_consumer_context.c (working copy) tduffy> @@ -47,7 +47,7 @@ tduffy> * DAT_SUCCESS tduffy> * DAT_INVALID_HANDLE tduffy> */ tduffy> -u32 dapl_set_consumer_context(DAT_HANDLE dat_handle, DAT_CONTEXT context) tduffy> +u32 dapl_set_consumer_context(DAT_HANDLE dat_handle, union dat_context context) tduffy> { tduffy> u32 dat_status = DAT_SUCCESS; tduffy> struct dapl_header *header; tduffy> Index: linux-kernel/dat-provider/dapl.h tduffy> =================================================================== tduffy> --- linux-kernel/dat-provider/dapl.h (revision 2506) tduffy> +++ linux-kernel/dat-provider/dapl.h (working copy) tduffy> @@ -177,7 +177,7 @@ struct dapl_header { tduffy> enum dat_handle_type handle_type; tduffy> struct dapl_ia *owner_ia; tduffy> struct dapl_llist_entry ia_list_entry; tduffy> - DAT_CONTEXT user_context; /* user context - opaque to DAPL */ tduffy> + union dat_context user_context; /* user context - opaque to DAPL */ tduffy> spinlock_t lock; tduffy> unsigned long flags; /* saved lock flag values */ tduffy> }; tduffy> @@ -423,9 +423,11 @@ extern u32 dapl_ia_query(DAT_IA_HANDLE, tduffy> tduffy> /* helper functions */ tduffy> tduffy> -extern u32 dapl_set_consumer_context(DAT_HANDLE handle, DAT_CONTEXT context); tduffy> +extern u32 dapl_set_consumer_context(DAT_HANDLE handle, tduffy> + union dat_context context); tduffy> tduffy> -extern u32 dapl_get_consumer_context(DAT_HANDLE handle, DAT_CONTEXT *context); tduffy> +extern u32 dapl_get_consumer_context(DAT_HANDLE handle, tduffy> + union dat_context *context); tduffy> tduffy> extern u32 dapl_get_handle_type(DAT_HANDLE handle, tduffy> enum dat_handle_type *type); tduffy> Index: linux-kernel/dat/dat.h tduffy> =================================================================== tduffy> --- linux-kernel/dat/dat.h (revision 2506) tduffy> +++ linux-kernel/dat/dat.h (working copy) tduffy> @@ -361,14 +361,14 @@ typedef enum { tduffy> TRUE = 1 tduffy> } boolean_t; tduffy> tduffy> -typedef union dat_context { tduffy> +union dat_context { tduffy> void *as_ptr; tduffy> u64 as_64; tduffy> unsigned long long as_index; tduffy> -} DAT_CONTEXT; tduffy> +}; tduffy> tduffy> -typedef DAT_CONTEXT DAT_DTO_COOKIE; tduffy> -typedef DAT_CONTEXT DAT_RMR_COOKIE; tduffy> +typedef union dat_context DAT_DTO_COOKIE; tduffy> +typedef union dat_context DAT_RMR_COOKIE; tduffy> tduffy> enum dat_completion_flags { tduffy> /* Completes with notification */ tduffy> @@ -920,13 +920,6 @@ struct dat_upcall_object { tduffy> DAT_UPCALL_FUNC upcall_func; tduffy> }; tduffy> tduffy> -/* Define NULL upcall */ tduffy> - tduffy> -#define DAT_UPCALL_NULL \ tduffy> - ((struct dat_upcall_object) { (void *) NULL, (DAT_UPCALL_FUNC) NULL }) tduffy> - tduffy> -#define DAT_UPCALL_SAME ((struct dat_upcall_object *) NULL) tduffy> - tduffy> /* Provider/registration info */ tduffy> tduffy> struct dat_provider_info { tduffy> @@ -1098,9 +1091,9 @@ typedef u32 (*DAT_IA_QUERY_FUNC)(DAT_IA_ tduffy> struct dat_ia_attr *, tduffy> struct dat_provider_attr *); tduffy> tduffy> -typedef u32 (*DAT_SET_CONSUMER_CONTEXT_FUNC)(DAT_HANDLE, DAT_CONTEXT); tduffy> +typedef u32 (*DAT_SET_CONSUMER_CONTEXT_FUNC)(DAT_HANDLE, union dat_context); tduffy> tduffy> -typedef u32 (*DAT_GET_CONSUMER_CONTEXT_FUNC)(DAT_HANDLE, DAT_CONTEXT *); tduffy> +typedef u32 (*DAT_GET_CONSUMER_CONTEXT_FUNC)(DAT_HANDLE, union dat_context *); tduffy> tduffy> typedef u32 (*DAT_GET_HANDLE_TYPE_FUNC)(DAT_HANDLE, enum dat_handle_type *); tduffy> tduffy> @@ -1387,14 +1380,14 @@ extern u32 dat_registry_list_providers(i tduffy> DAT_HANDLE_TO_PROVIDER(handle)->func(handle, ##__VA_ARGS__) tduffy> tduffy> static inline u32 dat_set_consumer_context(DAT_HANDLE handle, tduffy> - DAT_CONTEXT context) tduffy> + union dat_context context) tduffy> { tduffy> return DAT_CALL_PROVIDER_FUNC(set_consumer_context_func, handle, tduffy> context); tduffy> } tduffy> tduffy> static inline u32 dat_get_consumer_context(DAT_HANDLE handle, tduffy> - DAT_CONTEXT *context) tduffy> + union dat_context *context) tduffy> { tduffy> return DAT_CALL_PROVIDER_FUNC(get_consumer_context_func, handle, tduffy> context); tduffy> From jlentini at netapp.com Tue May 31 08:40:49 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 31 May 2005 11:40:49 -0400 (EDT) Subject: [openib-general] RE: [PATCH][kdapl] replace spin_lock with spin_lock_irqsave in kd apltest In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35BEF@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35BEF@mtvex01.mtv.mtl.com> Message-ID: Ok, then it should be removed completely not commented out. I'll do that and commit. On Tue, 31 May 2005, Itamar Rabenstein wrote: itamar> it is only declared not in use so we dont need it (;-) itamar> itamar> Itamar itamar> itamar> > -----Original Message----- itamar> > From: James Lentini [mailto:jlentini at netapp.com] itamar> > Sent: Tuesday, May 31, 2005 5:46 PM itamar> > To: Itamar itamar> > Cc: openib-general itamar> > Subject: Re: [PATCH][kdapl] replace spin_lock with itamar> > spin_lock_irqsave in itamar> > kdapltest itamar> > itamar> > itamar> > itamar> > Itamar, itamar> > itamar> > Why does this patch comment out uses of the g_PerfTestLock? itamar> > itamar> > james itamar> > itamar> > On Sun, 29 May 2005, Itamar wrote: itamar> > itamar> > itamar> With this patch i can run kdapltest -T T ... -t 4 -w 8 ... itamar> > itamar> I still see problems but in general this patch helps itamar> > the stability a lot. itamar> > itamar> itamar> > itamar> replace spin_lock with spin_lock_irqsave in kdapltest itamar> > itamar> Signed-off-by: Itamar Rabenstein itamar> > itamar> itamar> > itamar> Index: test/dapl_transaction_stats.c itamar> > itamar> itamar> > =================================================================== itamar> > itamar> --- test/dapl_transaction_stats.c (revision 2509) itamar> > itamar> +++ test/dapl_transaction_stats.c (working copy) itamar> > itamar> @@ -45,12 +45,13 @@ itamar> > itamar> DT_transaction_stats_set_ready (DT_Tdep_Print_Head *phead, itamar> > itamar> Transaction_Stats_t * itamar> > transaction_stats) itamar> > itamar> { itamar> > itamar> - DT_Mdep_Lock (&transaction_stats->lock); itamar> > itamar> + unsigned long flags; itamar> > itamar> + spin_lock_irqsave (&transaction_stats->lock,flags); itamar> > itamar> transaction_stats->wait_count--; itamar> > itamar> itamar> > itamar> DT_Tdep_PT_Debug (1,(phead,"Received Sync itamar> > Message from server (%d left)\n", itamar> > itamar> transaction_stats->wait_count)); itamar> > itamar> - DT_Mdep_Unlock (&transaction_stats->lock); itamar> > itamar> + spin_unlock_irqrestore (&transaction_stats->lock,flags); itamar> > itamar> } itamar> > itamar> itamar> > itamar> boolean_t itamar> > itamar> @@ -86,7 +87,8 @@ itamar> > itamar> unsigned int bytes_rdma_read, itamar> > itamar> unsigned int bytes_rdma_write) itamar> > itamar> { itamar> > itamar> - DT_Mdep_Lock (&transaction_stats->lock); itamar> > itamar> + unsigned long flags; itamar> > itamar> + spin_lock_irqsave (&transaction_stats->lock,flags); itamar> > itamar> itamar> > itamar> /* look for the longest time... */ itamar> > itamar> if (time_ms > transaction_stats->time_ms) itamar> > itamar> @@ -99,5 +101,5 @@ itamar> > itamar> transaction_stats->bytes_recv += bytes_recv; itamar> > itamar> transaction_stats->bytes_rdma_read += bytes_rdma_read; itamar> > itamar> transaction_stats->bytes_rdma_write += bytes_rdma_write; itamar> > itamar> - DT_Mdep_Unlock (&transaction_stats->lock); itamar> > itamar> + spin_unlock_irqrestore (&transaction_stats->lock,flags); itamar> > itamar> } itamar> > itamar> Index: test/dapl_server.c itamar> > itamar> itamar> > =================================================================== itamar> > itamar> --- test/dapl_server.c (revision 2509) itamar> > itamar> +++ test/dapl_server.c (working copy) itamar> > itamar> @@ -49,7 +49,7 @@ itamar> > itamar> unsigned char *buffp = NULL; itamar> > itamar> unsigned char *module = "DT_cs_Server"; itamar> > itamar> int status = 0; itamar> > itamar> - itamar> > itamar> + unsigned long flags; itamar> > itamar> DAT_DTO_COOKIE dto_cookie; itamar> > itamar> struct dat_dto_completion_event_data dto_stat; itamar> > itamar> u32 ret; itamar> > itamar> @@ -616,9 +616,9 @@ itamar> > itamar> itamar> > itamar> itamar> > itamar> /* Count this new client and get ready for the next */ itamar> > itamar> - DT_Mdep_Lock (&ps_ptr->num_clients_lock); itamar> > itamar> + spin_lock_irqsave (&ps_ptr->num_clients_lock,flags); itamar> > itamar> ps_ptr->num_clients++; itamar> > itamar> - DT_Mdep_Unlock (&ps_ptr->num_clients_lock); itamar> > itamar> + spin_unlock_irqrestore itamar> > (&ps_ptr->num_clients_lock,flags); itamar> > itamar> itamar> > itamar> /* we passed the pt_ptr to the thread and must itamar> > now 'forget' it */ itamar> > itamar> pt_ptr = NULL; itamar> > itamar> Index: test/dapl_thread.c itamar> > itamar> itamar> > =================================================================== itamar> > itamar> --- test/dapl_thread.c (revision 2509) itamar> > itamar> +++ test/dapl_thread.c (working copy) itamar> > itamar> @@ -83,6 +83,7 @@ itamar> > itamar> unsigned int stacksize) itamar> > itamar> { itamar> > itamar> Thread *thread_ptr; itamar> > itamar> + unsigned long flags; itamar> > itamar> thread_ptr = (Thread *) DT_MemListAlloc (pt_ptr, itamar> > "thread.c", THREAD, sizeof (Thread)); itamar> > itamar> if (thread_ptr == NULL) itamar> > itamar> { itamar> > itamar> @@ -93,9 +94,9 @@ itamar> > itamar> thread_ptr->thread_handle = 0; itamar> > itamar> thread_ptr->stacksize = stacksize; itamar> > itamar> itamar> > itamar> - DT_Mdep_Lock (&pt_ptr->Thread_counter_lock); itamar> > itamar> + spin_lock_irqsave (&pt_ptr->Thread_counter_lock,flags); itamar> > itamar> pt_ptr->Thread_counter++; itamar> > itamar> - DT_Mdep_Unlock (&pt_ptr->Thread_counter_lock); itamar> > itamar> + spin_unlock_irqrestore itamar> > (&pt_ptr->Thread_counter_lock,flags); itamar> > itamar> itamar> > itamar> DT_Mdep_Thread_Init_Attributes (thread_ptr); itamar> > itamar> itamar> > itamar> @@ -108,11 +109,12 @@ itamar> > itamar> void itamar> > itamar> DT_Thread_Destroy (Thread * thread_ptr, itamar> > Per_Test_Data_t * pt_ptr) itamar> > itamar> { itamar> > itamar> + unsigned long flags; itamar> > itamar> if (thread_ptr) itamar> > itamar> { itamar> > itamar> - DT_Mdep_Lock (&pt_ptr->Thread_counter_lock); itamar> > itamar> + spin_lock_irqsave (&pt_ptr->Thread_counter_lock,flags); itamar> > itamar> pt_ptr->Thread_counter--; itamar> > itamar> - DT_Mdep_Unlock (&pt_ptr->Thread_counter_lock); itamar> > itamar> + spin_unlock_irqrestore itamar> > (&pt_ptr->Thread_counter_lock,flags); itamar> > itamar> itamar> > itamar> DT_Mdep_Thread_Destroy_Attributes (thread_ptr); itamar> > itamar> DT_MemListFree (pt_ptr, thread_ptr); itamar> > itamar> Index: test/dapl_test_data.c itamar> > itamar> itamar> > =================================================================== itamar> > itamar> --- test/dapl_test_data.c (revision 2509) itamar> > itamar> +++ test/dapl_test_data.c (working copy) itamar> > itamar> @@ -27,7 +27,7 @@ itamar> > itamar> itamar> > itamar> #include "dapl_proto.h" itamar> > itamar> itamar> > itamar> -DT_Mdep_LockType g_PerfTestLock; itamar> > itamar> +/*DT_Mdep_LockType g_PerfTestLock;*/ itamar> > itamar> /* itamar> > itamar> * check memory leaking int alloc_count; itamar> > DT_Mdep_LockType itamar> > itamar> * Alloc_Count_Lock; itamar> > itamar> Index: test/dapl_transaction_test.c itamar> > itamar> itamar> > =================================================================== itamar> > itamar> --- test/dapl_transaction_test.c (revision 2509) itamar> > itamar> +++ test/dapl_transaction_test.c (working copy) itamar> > itamar> @@ -99,7 +99,7 @@ itamar> > itamar> unsigned int i; itamar> > itamar> DT_Tdep_Print_Head *phead; itamar> > itamar> int status = 0; itamar> > itamar> - itamar> > itamar> + unsigned long flags; itamar> > itamar> phead = pt_ptr->Params.phead; itamar> > itamar> itamar> > itamar> pt_ptr->Countdown_Counter = cmd->num_threads; itamar> > itamar> @@ -129,9 +129,9 @@ itamar> > itamar> } itamar> > itamar> DT_Thread_Destroy (pt_ptr->thread, pt_ptr); itamar> > /* destroy Master thread */ itamar> > itamar> itamar> > itamar> - DT_Mdep_Lock (&pt_ptr->ps_ptr->num_clients_lock); itamar> > itamar> + spin_lock_irqsave itamar> > (&pt_ptr->ps_ptr->num_clients_lock,flags); itamar> > itamar> pt_ptr->ps_ptr->num_clients--; itamar> > itamar> - DT_Mdep_Unlock (&pt_ptr->ps_ptr->num_clients_lock); itamar> > itamar> + spin_unlock_irqrestore itamar> > (&pt_ptr->ps_ptr->num_clients_lock,flags); itamar> > itamar> itamar> > itamar> /* NB: Server has no pt_ptr->remote_netaddr */ itamar> > itamar> DT_PrintMemList (pt_ptr); /* check if itamar> > we return all space allocated */ itamar> > itamar> @@ -239,7 +239,7 @@ itamar> > itamar> enum dat_event_number event_num; itamar> > itamar> DT_Tdep_Print_Head *phead; itamar> > itamar> int status = 0; itamar> > itamar> - itamar> > itamar> + unsigned long flags; itamar> > itamar> pt_ptr = test_ptr->pt_ptr; itamar> > itamar> thread = test_ptr->thread; itamar> > itamar> phead = pt_ptr->Params.phead; itamar> > itamar> @@ -502,7 +502,7 @@ itamar> > itamar> */ itamar> > itamar> if (pt_ptr->local_is_server) itamar> > itamar> { itamar> > itamar> - DT_Mdep_Lock (&pt_ptr->Thread_counter_lock); itamar> > itamar> + spin_lock_irqsave (&pt_ptr->Thread_counter_lock,flags); itamar> > itamar> pt_ptr->Countdown_Counter--; itamar> > itamar> /* Deliberate pre-decrement. Post decrement won't itamar> > itamar> * work here, so don't do it. itamar> > itamar> @@ -512,7 +512,7 @@ itamar> > itamar> DT_Mdep_wait_object_wakeup itamar> > (&pt_ptr->synch_wait_object); itamar> > itamar> } itamar> > itamar> itamar> > itamar> - DT_Mdep_Unlock (&pt_ptr->Thread_counter_lock); itamar> > itamar> + spin_unlock_irqrestore itamar> > (&pt_ptr->Thread_counter_lock,flags); itamar> > itamar> } itamar> > itamar> itamar> > itamar> for (i = 0; i < test_ptr->cmd->eps_per_thread; i++) itamar> > itamar> Index: test/dapl_performance_server.c itamar> > itamar> itamar> > =================================================================== itamar> > itamar> --- test/dapl_performance_server.c (revision 2509) itamar> > itamar> +++ test/dapl_performance_server.c (working copy) itamar> > itamar> @@ -36,7 +36,7 @@ itamar> > itamar> Performance_Test_t *test_ptr = NULL; itamar> > itamar> DT_Tdep_Print_Head *phead; itamar> > itamar> int status = 0; itamar> > itamar> - itamar> > itamar> + unsigned long flags; itamar> > itamar> phead = pt_ptr->Params.phead; itamar> > itamar> DT_Tdep_PT_Debug (1,(phead,"Server: Starting itamar> > performance test\n")); itamar> > itamar> itamar> > itamar> @@ -75,9 +75,9 @@ itamar> > itamar> DT_Mdep_Thread_Detach (DT_Mdep_Thread_SELF ()); itamar> > /* AMM */ itamar> > itamar> DT_Thread_Destroy (pt_ptr->thread, pt_ptr); itamar> > /* destroy Master thread */ itamar> > itamar> itamar> > itamar> - DT_Mdep_Lock (&pt_ptr->ps_ptr->num_clients_lock); itamar> > itamar> + spin_lock_irqsave itamar> > (&pt_ptr->ps_ptr->num_clients_lock,flags); itamar> > itamar> pt_ptr->ps_ptr->num_clients--; itamar> > itamar> - DT_Mdep_Unlock (&pt_ptr->ps_ptr->num_clients_lock); itamar> > itamar> + spin_unlock_irqrestore itamar> > (&pt_ptr->ps_ptr->num_clients_lock,flags); itamar> > itamar> itamar> > itamar> DT_PrintMemList (pt_ptr); /* check if itamar> > we return all space allocated */ itamar> > itamar> DT_Mdep_LockDestroy (&pt_ptr->Thread_counter_lock); itamar> > itamar> Index: test/dapl_memlist.c itamar> > itamar> itamar> > =================================================================== itamar> > itamar> --- test/dapl_memlist.c (revision 2509) itamar> > itamar> +++ test/dapl_memlist.c (working copy) itamar> > itamar> @@ -42,6 +42,7 @@ itamar> > itamar> { itamar> > itamar> void *buffptr; itamar> > itamar> MemListEntry_t *entry_ptr; itamar> > itamar> + unsigned long flags; itamar> > itamar> buffptr = NULL; itamar> > itamar> entry_ptr = NULL; itamar> > itamar> itamar> > itamar> @@ -64,10 +65,10 @@ itamar> > itamar> entry_ptr->MemType = t; itamar> > itamar> entry_ptr->mem_ptr = buffptr; itamar> > itamar> itamar> > itamar> - DT_Mdep_Lock (&pt_ptr->MemListLock); itamar> > itamar> + spin_lock_irqsave (&pt_ptr->MemListLock,flags); itamar> > itamar> entry_ptr->next = pt_ptr->MemListHead; itamar> > itamar> pt_ptr->MemListHead = entry_ptr; itamar> > itamar> - DT_Mdep_Unlock (&pt_ptr->MemListLock); itamar> > itamar> + spin_unlock_irqrestore (&pt_ptr->MemListLock,flags); itamar> > itamar> itamar> > itamar> return buffptr; itamar> > itamar> } itamar> > itamar> @@ -76,12 +77,13 @@ itamar> > itamar> DT_MemListFree (Per_Test_Data_t * pt_ptr, void *ptr) itamar> > itamar> { itamar> > itamar> MemListEntry_t *pre, *cur; itamar> > itamar> + unsigned long flags; itamar> > itamar> if (pt_ptr == 0) /* not use mem_list */ itamar> > itamar> { itamar> > itamar> DT_Mdep_Free (ptr); itamar> > itamar> return; itamar> > itamar> } itamar> > itamar> - DT_Mdep_Lock (&pt_ptr->MemListLock); itamar> > itamar> + spin_lock_irqsave (&pt_ptr->MemListLock,flags); itamar> > itamar> pre = NULL; itamar> > itamar> cur = pt_ptr->MemListHead; itamar> > itamar> while (cur) itamar> > itamar> @@ -106,7 +108,7 @@ itamar> > itamar> cur = cur->next; itamar> > itamar> } itamar> > itamar> unlock_and_return: itamar> > itamar> - DT_Mdep_Unlock (&pt_ptr->MemListLock); itamar> > itamar> + spin_unlock_irqrestore (&pt_ptr->MemListLock,flags); itamar> > itamar> } itamar> > itamar> itamar> > itamar> void itamar> > itamar> @@ -119,9 +121,9 @@ itamar> > itamar> }; itamar> > itamar> DT_Tdep_Print_Head *phead; itamar> > itamar> MemListEntry_t *cur; itamar> > itamar> - itamar> > itamar> + unsigned long flags; itamar> > itamar> phead = pt_ptr->Params.phead; itamar> > itamar> - DT_Mdep_Lock (&pt_ptr->MemListLock); itamar> > itamar> + spin_lock_irqsave (&pt_ptr->MemListLock,flags); itamar> > itamar> cur = pt_ptr->MemListHead; itamar> > itamar> if (cur != 0) itamar> > itamar> { itamar> > itamar> @@ -133,5 +135,5 @@ itamar> > itamar> cur->filename, type[cur->MemType]); itamar> > itamar> cur = cur->next; itamar> > itamar> } itamar> > itamar> - DT_Mdep_Unlock (&pt_ptr->MemListLock); itamar> > itamar> + spin_unlock_irqrestore (&pt_ptr->MemListLock,flags); itamar> > itamar> } itamar> > itamar> Index: include/dapl_test_data.h itamar> > itamar> itamar> > =================================================================== itamar> > itamar> --- include/dapl_test_data.h (revision 2509) itamar> > itamar> +++ include/dapl_test_data.h (working copy) itamar> > itamar> @@ -42,7 +42,7 @@ itamar> > itamar> * problems on the server side occasionally itamar> > itamar> * the server will reject connections. itamar> > itamar> */ itamar> > itamar> -extern DT_Mdep_LockType g_PerfTestLock; itamar> > itamar> +/*extern DT_Mdep_LockType g_PerfTestLock; */ itamar> > itamar> itamar> > itamar> /* itamar> > itamar> * check memory leaking extern int itamar> > alloc_count ; extern itamar> > itamar> Index: kdapl/kdapl_module.c itamar> > itamar> itamar> > =================================================================== itamar> > itamar> --- kdapl/kdapl_module.c (revision 2509) itamar> > itamar> +++ kdapl/kdapl_module.c (working copy) itamar> > itamar> @@ -95,10 +95,10 @@ itamar> > itamar> break; itamar> > itamar> } itamar> > itamar> itamar> > itamar> - DT_Mdep_Lock(&kdapltest_lock); itamar> > itamar> + spin_lock_irq(&kdapltest_lock); itamar> > itamar> kdapltest_num++; itamar> > itamar> instance = kdapltest_num; itamar> > itamar> - DT_Mdep_Unlock(&kdapltest_lock); itamar> > itamar> + spin_unlock_irq(&kdapltest_lock); itamar> > itamar> itamar> > itamar> local_params->phead = itamar> > KDT_Print_Alloc(instance); itamar> > itamar> if (!local_params->phead) { itamar> > itamar> @@ -201,7 +201,7 @@ itamar> > itamar> static __init int kdapltest_init(void) itamar> > itamar> { itamar> > itamar> itamar> > itamar> - DT_Mdep_LockInit(&g_PerfTestLock); /* For itamar> > uDAPL, this is done in Tdep_init() */ itamar> > itamar> + /* DT_Mdep_LockInit(&g_PerfTestLock); For itamar> > uDAPL, this is done in Tdep_init() */ itamar> > itamar> kdapltest_major = register_chrdev(0, itamar> > "kdapltest", &kdapltest_fops); itamar> > itamar> DT_Mdep_LockInit(&kdapltest_lock); itamar> > itamar> kdapltest_num = 0; itamar> > itamar> @@ -218,7 +218,7 @@ itamar> > itamar> KDT_Print_Destroy(); itamar> > itamar> KDT_Evd_Destroy(); itamar> > itamar> DT_Mdep_LockDestroy(&kdapltest_lock); itamar> > itamar> - DT_Mdep_LockDestroy(&g_PerfTestLock); /* For itamar> > uDAPL, this is done in Tdep_end() */ itamar> > itamar> + /* DT_Mdep_LockDestroy(&g_PerfTestLock); For itamar> > uDAPL, this is done in Tdep_end() */ itamar> > itamar> } itamar> > itamar> itamar> > itamar> module_init(kdapltest_init); itamar> > itamar> Index: kdapl/kdapl_tdep_print.c itamar> > itamar> itamar> > =================================================================== itamar> > itamar> --- kdapl/kdapl_tdep_print.c (revision 2509) itamar> > itamar> +++ kdapl/kdapl_tdep_print.c (working copy) itamar> > itamar> @@ -106,7 +106,7 @@ itamar> > itamar> va_list args; itamar> > itamar> int len; itamar> > itamar> Tdep_Print_Entry *entry; itamar> > itamar> - itamar> > itamar> + unsigned long flags; itamar> > itamar> entry = DT_Mdep_Malloc (sizeof (Tdep_Print_Entry)); itamar> > itamar> itamar> > itamar> va_start (args, fmt); itamar> > itamar> @@ -120,7 +120,7 @@ itamar> > itamar> } itamar> > itamar> entry->next = NULL; itamar> > itamar> itamar> > itamar> - DT_Mdep_Lock (&phead->lock); itamar> > itamar> + spin_lock_irqsave (&phead->lock,flags); itamar> > itamar> itamar> > itamar> if (phead->head == NULL) itamar> > itamar> { itamar> > itamar> @@ -133,7 +133,7 @@ itamar> > itamar> phead->tail = entry; itamar> > itamar> } itamar> > itamar> DT_Mdep_wait_object_wakeup (&phead->wait_object); itamar> > itamar> - DT_Mdep_Unlock (&phead->lock); itamar> > itamar> + spin_unlock_irqrestore (&phead->lock,flags); itamar> > itamar> } itamar> > itamar> itamar> > itamar> int itamar> > itamar> @@ -143,10 +143,10 @@ itamar> > itamar> DT_Tdep_Print_Head *phead; itamar> > itamar> Tdep_Print_Entry *entry; itamar> > itamar> int rc; itamar> > itamar> - itamar> > itamar> + unsigned long flags; itamar> > itamar> buffer[0] = '\0'; itamar> > itamar> itamar> > itamar> - DT_Mdep_Lock (&DT_Print_List_Lock); itamar> > itamar> + spin_lock_irqsave (&DT_Print_List_Lock,flags); itamar> > itamar> phead = DT_Print_List; itamar> > itamar> while (phead) itamar> > itamar> { itamar> > itamar> @@ -156,7 +156,7 @@ itamar> > itamar> } itamar> > itamar> phead = phead->next; itamar> > itamar> } itamar> > itamar> - DT_Mdep_Unlock (&DT_Print_List_Lock); itamar> > itamar> + spin_unlock_irqrestore (&DT_Print_List_Lock,flags); itamar> > itamar> itamar> > itamar> if (!phead) itamar> > itamar> { itamar> > itamar> @@ -172,7 +172,7 @@ itamar> > itamar> return 0; itamar> > itamar> } itamar> > itamar> itamar> > itamar> - DT_Mdep_Lock (&phead->lock); itamar> > itamar> + spin_lock_irqsave (&phead->lock,flags); itamar> > itamar> entry = phead->head; itamar> > itamar> if (entry) itamar> > itamar> { itamar> > itamar> @@ -184,7 +184,7 @@ itamar> > itamar> strncpy (buffer, entry->buffer, PRINT_MAX); itamar> > itamar> DT_Mdep_Free (entry); itamar> > itamar> } itamar> > itamar> - DT_Mdep_Unlock (&phead->lock); itamar> > itamar> + spin_unlock_irqrestore (&phead->lock,flags); itamar> > itamar> } itamar> > itamar> itamar> > itamar> len = strlen (buffer); itamar> > itamar> Index: kdapl/kdapl_tdep_evd.c itamar> > itamar> itamar> > =================================================================== itamar> > itamar> --- kdapl/kdapl_tdep_evd.c (revision 2509) itamar> > itamar> +++ kdapl/kdapl_tdep_evd.c (working copy) itamar> > itamar> @@ -105,7 +105,7 @@ itamar> > itamar> u32 dat_status; itamar> > itamar> struct dat_upcall_object upcall; itamar> > itamar> Tdep_Evd *evd_ptr; itamar> > itamar> - itamar> > itamar> + unsigned long flags; itamar> > itamar> evd_ptr = NULL; itamar> > itamar> dat_status = DAT_SUCCESS; itamar> > itamar> itamar> > itamar> @@ -139,10 +139,10 @@ itamar> > itamar> DT_Mdep_wait_object_init (&evd_ptr->wait_object); itamar> > itamar> itamar> > itamar> /* add evd_ptr to front of evd list */ itamar> > itamar> - DT_Mdep_Lock (&DT_Evd_Lock); itamar> > itamar> + spin_lock_irqsave (&DT_Evd_Lock, flags); itamar> > itamar> evd_ptr->evd_next = DT_Evd_List; itamar> > itamar> DT_Evd_List = evd_ptr; itamar> > itamar> - DT_Mdep_Unlock (&DT_Evd_Lock); itamar> > itamar> + spin_unlock_irqrestore (&DT_Evd_Lock, flags); itamar> > itamar> itamar> > itamar> error: itamar> > itamar> if (dat_status != DAT_SUCCESS) itamar> > itamar> @@ -161,7 +161,7 @@ itamar> > itamar> u32 dat_status; itamar> > itamar> Tdep_Evd *evd_ptr; itamar> > itamar> Tdep_Event *event; itamar> > itamar> - itamar> > itamar> + unsigned long flags; itamar> > itamar> dat_status = DAT_SUCCESS; itamar> > itamar> itamar> > itamar> /* find the evd_ptr associated with evd_handle */ itamar> > itamar> @@ -181,7 +181,7 @@ itamar> > itamar> DAT_INTERNAL_ERROR; itamar> > itamar> } itamar> > itamar> /* Get event */ itamar> > itamar> - DT_Mdep_Lock (&DT_Evd_Lock); itamar> > itamar> + spin_lock_irqsave (&DT_Evd_Lock, flags); itamar> > itamar> event = evd_ptr->event_next; /* event present in itamar> > evd_ptr corresponding to the given evd_handle */ itamar> > itamar> if (event) itamar> > itamar> { itamar> > itamar> @@ -207,7 +207,7 @@ itamar> > itamar> { itamar> > itamar> dat_status = DAT_QUEUE_EMPTY; itamar> > itamar> } itamar> > itamar> - DT_Mdep_Unlock (&DT_Evd_Lock); itamar> > itamar> + spin_unlock_irqrestore (&DT_Evd_Lock, flags); itamar> > itamar> itamar> > itamar> return dat_status; itamar> > itamar> } itamar> > itamar> @@ -261,7 +261,7 @@ itamar> > itamar> } itamar> > itamar> itamar> > itamar> /* Got event */ itamar> > itamar> - DT_Mdep_Lock (&DT_Evd_Lock); itamar> > itamar> + spin_lock_irq (&DT_Evd_Lock); itamar> > itamar> event = evd_ptr->event_next; itamar> > itamar> if (event) itamar> > itamar> { itamar> > itamar> @@ -286,7 +286,7 @@ itamar> > itamar> } itamar> > itamar> } itamar> > itamar> itamar> > itamar> - DT_Mdep_Unlock (&DT_Evd_Lock); itamar> > itamar> + spin_unlock_irq (&DT_Evd_Lock); itamar> > itamar> return dat_status; itamar> > itamar> } itamar> > itamar> itamar> > itamar> @@ -297,7 +297,7 @@ itamar> > itamar> Tdep_Evd *last; itamar> > itamar> itamar> > itamar> last = NULL; itamar> > itamar> - DT_Mdep_Lock (&DT_Evd_Lock); itamar> > itamar> + spin_lock_irq (&DT_Evd_Lock); itamar> > itamar> next = DT_Evd_List; itamar> > itamar> if (next->evd_handle == evd_handle) itamar> > itamar> { itamar> > itamar> @@ -315,7 +315,7 @@ itamar> > itamar> last->evd_next = next->evd_next; itamar> > itamar> } itamar> > itamar> } itamar> > itamar> - DT_Mdep_Unlock (&DT_Evd_Lock); itamar> > itamar> + spin_unlock_irq (&DT_Evd_Lock); itamar> > itamar> itamar> > itamar> DT_Mdep_Free (next); itamar> > itamar> return dat_evd_free(evd_handle); itamar> > itamar> @@ -331,7 +331,7 @@ itamar> > itamar> itamar> > itamar> evd_ptr = (Tdep_Evd *) instance_data; itamar> > itamar> itamar> > itamar> - DT_Mdep_Lock (&DT_Evd_Lock); itamar> > itamar> + spin_lock_irq (&DT_Evd_Lock); itamar> > itamar> if (DT_Event_Free_List) itamar> > itamar> { itamar> > itamar> event = DT_Event_Free_List; itamar> > itamar> @@ -356,6 +356,6 @@ itamar> > itamar> evd_ptr->event_last = event; itamar> > itamar> } itamar> > itamar> } itamar> > itamar> - DT_Mdep_Unlock (&DT_Evd_Lock); itamar> > itamar> + spin_unlock_irq (&DT_Evd_Lock); itamar> > itamar> DT_Mdep_wait_object_wakeup (&evd_ptr->wait_object); itamar> > itamar> } itamar> > itamar> -- itamar> > itamar> Itamar itamar> > itamar> itamar> > itamar> From roland at topspin.com Tue May 31 08:52:35 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 31 May 2005 08:52:35 -0700 Subject: [openib-general] Problem compiling userspace driver. In-Reply-To: <20050531145917.GB15647@minantech.com> (Gleb Natapov's message of "Tue, 31 May 2005 17:59:17 +0300") References: <20050531145917.GB15647@minantech.com> Message-ID: <521x7nbdfg.fsf@topspin.com> Sorry, I had some uncommitted changes left in my tree. So of course I didn't see any problems. I just checked in the required libibverbs changes. - R. From itamar at mellanox.co.il Tue May 31 09:16:40 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Tue, 31 May 2005 19:16:40 +0300 Subject: [openib-general] Re: [PATCH] kDAPL: remove typedef DAT_CONTEX T Message-ID: <91DB792C7985D411BEC300B40080D29CC35BF1@mtvex01.mtv.mtl.com> hi all, I have tried to use DAT_UPCALL_NULL and I got compile error and I don't think that it is good to try to make comparator ( = ) between structs if we want to check for DAT_UPCALL_NULL we need to check that the CB function pointer is NULL. I mean that if you want to use DAT_UPCALL_NULL you need to have real dat_upcall struct and to set the CB function to NULL. instead of casting NULL to be a struct. Currently dat_evd_modify_upcall() is not implemented according the spec. Itamar > -----Original Message----- > From: James Lentini [mailto:jlentini at netapp.com] > Sent: Tuesday, May 31, 2005 6:39 PM > To: Tom Duffy > Cc: openib-general at openib.org > Subject: [openib-general] Re: [PATCH] kDAPL: remove typedef > DAT_CONTEXT > > > > Mostly committed in revision 2515. > > I didn't remove DAT_UPCALL_NULL and DAT_UPCALL_SAME. DAT_UPCALL_NULL > is provided as a convenience to the consumer. I think it is > useful, but > I'm willing to hear other opinions. The provider's implementation of > dat_evd_modify_upcall() should check for the DAT_UPCALL_SAME value. > The fact that it doesn't is a bug. > > james > > On Fri, 27 May 2005, Tom Duffy wrote: > > tduffy> Get rid of the typedef DAT_CONTEXT. > tduffy> > tduffy> Signed-off-by: Tom Duffy > tduffy> > tduffy> Index: linux-kernel/test/dapltest/include/dapl_common.h > tduffy> > =================================================================== > tduffy> --- linux-kernel/test/dapltest/include/dapl_common.h > (revision 2506) > tduffy> +++ linux-kernel/test/dapltest/include/dapl_common.h > (working copy) > tduffy> @@ -42,7 +42,7 @@ typedef enum > tduffy> typedef struct > tduffy> { > tduffy> DAT_RMR_CONTEXT rmr_context; > tduffy> - DAT_CONTEXT mem_address; > tduffy> + union dat_context mem_address; > tduffy> } RemoteMemoryInfo; > tduffy> #pragma pack() > tduffy> > tduffy> Index: linux-kernel/dat-provider/dapl_get_consumer_context.c > tduffy> > =================================================================== > tduffy> --- > linux-kernel/dat-provider/dapl_get_consumer_context.c (revision 2506) > tduffy> +++ > linux-kernel/dat-provider/dapl_get_consumer_context.c (working copy) > tduffy> @@ -48,7 +48,7 @@ > tduffy> * DAT_SUCCESS > tduffy> * DAT_INVALID_PARAMETER > tduffy> */ > tduffy> -u32 dapl_get_consumer_context(DAT_HANDLE dat_handle, > DAT_CONTEXT *context) > tduffy> +u32 dapl_get_consumer_context(DAT_HANDLE dat_handle, > union dat_context *context) > tduffy> { > tduffy> u32 dat_status = DAT_SUCCESS; > tduffy> struct dapl_header *header; > tduffy> Index: linux-kernel/dat-provider/dapl_set_consumer_context.c > tduffy> > =================================================================== > tduffy> --- > linux-kernel/dat-provider/dapl_set_consumer_context.c (revision 2506) > tduffy> +++ > linux-kernel/dat-provider/dapl_set_consumer_context.c (working copy) > tduffy> @@ -47,7 +47,7 @@ > tduffy> * DAT_SUCCESS > tduffy> * DAT_INVALID_HANDLE > tduffy> */ > tduffy> -u32 dapl_set_consumer_context(DAT_HANDLE dat_handle, > DAT_CONTEXT context) > tduffy> +u32 dapl_set_consumer_context(DAT_HANDLE dat_handle, > union dat_context context) > tduffy> { > tduffy> u32 dat_status = DAT_SUCCESS; > tduffy> struct dapl_header *header; > tduffy> Index: linux-kernel/dat-provider/dapl.h > tduffy> > =================================================================== > tduffy> --- linux-kernel/dat-provider/dapl.h (revision 2506) > tduffy> +++ linux-kernel/dat-provider/dapl.h (working copy) > tduffy> @@ -177,7 +177,7 @@ struct dapl_header { > tduffy> enum dat_handle_type handle_type; > tduffy> struct dapl_ia *owner_ia; > tduffy> struct dapl_llist_entry ia_list_entry; > tduffy> - DAT_CONTEXT user_context; /* user > context - opaque to DAPL */ > tduffy> + union dat_context user_context; /* user > context - opaque to DAPL */ > tduffy> spinlock_t lock; > tduffy> unsigned long flags; /* saved lock > flag values */ > tduffy> }; > tduffy> @@ -423,9 +423,11 @@ extern u32 dapl_ia_query(DAT_IA_HANDLE, > tduffy> > tduffy> /* helper functions */ > tduffy> > tduffy> -extern u32 dapl_set_consumer_context(DAT_HANDLE > handle, DAT_CONTEXT context); > tduffy> +extern u32 dapl_set_consumer_context(DAT_HANDLE handle, > tduffy> + union dat_context context); > tduffy> > tduffy> -extern u32 dapl_get_consumer_context(DAT_HANDLE > handle, DAT_CONTEXT *context); > tduffy> +extern u32 dapl_get_consumer_context(DAT_HANDLE handle, > tduffy> + union dat_context > *context); > tduffy> > tduffy> extern u32 dapl_get_handle_type(DAT_HANDLE handle, > tduffy> enum dat_handle_type *type); > tduffy> Index: linux-kernel/dat/dat.h > tduffy> > =================================================================== > tduffy> --- linux-kernel/dat/dat.h (revision 2506) > tduffy> +++ linux-kernel/dat/dat.h (working copy) > tduffy> @@ -361,14 +361,14 @@ typedef enum { > tduffy> TRUE = 1 > tduffy> } boolean_t; > tduffy> > tduffy> -typedef union dat_context { > tduffy> +union dat_context { > tduffy> void *as_ptr; > tduffy> u64 as_64; > tduffy> unsigned long long as_index; > tduffy> -} DAT_CONTEXT; > tduffy> +}; > tduffy> > tduffy> -typedef DAT_CONTEXT DAT_DTO_COOKIE; > tduffy> -typedef DAT_CONTEXT DAT_RMR_COOKIE; > tduffy> +typedef union dat_context DAT_DTO_COOKIE; > tduffy> +typedef union dat_context DAT_RMR_COOKIE; > tduffy> > tduffy> enum dat_completion_flags { > tduffy> /* Completes with notification > */ > tduffy> @@ -920,13 +920,6 @@ struct dat_upcall_object { > tduffy> DAT_UPCALL_FUNC upcall_func; > tduffy> }; > tduffy> > tduffy> -/* Define NULL upcall */ > tduffy> - > tduffy> -#define DAT_UPCALL_NULL \ > tduffy> - ((struct dat_upcall_object) { (void *) NULL, > (DAT_UPCALL_FUNC) NULL }) > tduffy> - > tduffy> -#define DAT_UPCALL_SAME ((struct > dat_upcall_object *) NULL) > tduffy> - > tduffy> /* Provider/registration info */ > tduffy> > tduffy> struct dat_provider_info { > tduffy> @@ -1098,9 +1091,9 @@ typedef u32 (*DAT_IA_QUERY_FUNC)(DAT_IA_ > tduffy> struct dat_ia_attr *, > tduffy> struct dat_provider_attr *); > tduffy> > tduffy> -typedef u32 > (*DAT_SET_CONSUMER_CONTEXT_FUNC)(DAT_HANDLE, DAT_CONTEXT); > tduffy> +typedef u32 > (*DAT_SET_CONSUMER_CONTEXT_FUNC)(DAT_HANDLE, union dat_context); > tduffy> > tduffy> -typedef u32 > (*DAT_GET_CONSUMER_CONTEXT_FUNC)(DAT_HANDLE, DAT_CONTEXT *); > tduffy> +typedef u32 > (*DAT_GET_CONSUMER_CONTEXT_FUNC)(DAT_HANDLE, union dat_context *); > tduffy> > tduffy> typedef u32 (*DAT_GET_HANDLE_TYPE_FUNC)(DAT_HANDLE, > enum dat_handle_type *); > tduffy> > tduffy> @@ -1387,14 +1380,14 @@ extern u32 > dat_registry_list_providers(i > tduffy> DAT_HANDLE_TO_PROVIDER(handle)->func(handle, > ##__VA_ARGS__) > tduffy> > tduffy> static inline u32 dat_set_consumer_context(DAT_HANDLE handle, > tduffy> - DAT_CONTEXT context) > tduffy> + union > dat_context context) > tduffy> { > tduffy> return > DAT_CALL_PROVIDER_FUNC(set_consumer_context_func, handle, > tduffy> context); > tduffy> } > tduffy> > tduffy> static inline u32 dat_get_consumer_context(DAT_HANDLE handle, > tduffy> - DAT_CONTEXT *context) > tduffy> + union > dat_context *context) > tduffy> { > tduffy> return > DAT_CALL_PROVIDER_FUNC(get_consumer_context_func, handle, > tduffy> context); > tduffy> > _______________________________________________ > openib-general mailing list > openib-general at openib.org > http://openib.org/mailman/listinfo/openib-general > > To unsubscribe, please visit > http://openib.org/mailman/listinfo/openib-general > From iod00d at hp.com Tue May 31 09:38:51 2005 From: iod00d at hp.com (Grant Grundler) Date: Tue, 31 May 2005 09:38:51 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <469958e005052816262c8ff2b4@mail.gmail.com> References: <20050528121755.44489.qmail@web51907.mail.yahoo.com> <20050528134746.GA6277@lst.de> <469958e005052816262c8ff2b4@mail.gmail.com> Message-ID: <20050531163851.GB20793@esmail.cup.hp.com> On Sat, May 28, 2005 at 04:26:43PM -0700, Caitlin Bestler wrote: ... > if so what the best strategy for > achieving it is (try to plan an IB/iWARP merge immediately > or wait until there is an iWARP code base). If there is no iWARP code base, I fail to see how one can merge. Having a specification is one basis for communication. Linux developers normally use existing code as the basis. Committees submit CRs (Change Requests) to update specs. The CRs get voted on by the committee. Linux developers submit patches. The Linux subsystems maintainer(s) decide if patches are ok or not. > Claiming that an InfiniBand-specific interface is somehow > thinking "long term" is just plain ludicrous. "It Works" is worth 10x more to *any* customer than a transport neutral API that only exists as a spec. The specs are guides to how something *should* work and linux tries to comply with them (e.g. 802.3 or T10) where HW implementations actually follow the spec. That doesn't mean linux has to implement every brain damaged spec that some committee comes up with....OTOH, rdmaconsortium.org does have a fair shot given I2O made it into the kernel. :^/ (I'm willing to have a conversation about why I think I2O is brain damaged if someone else is buying drinks. It's not total crap, but it certainly has it's downside.) > Now it may be that the short term interest of the InfiniBand > vendors is such that they cannot commit resources to > helping build a transport neutral API. That is always a > legitimate tradeoff, but it is "short term corporate thinking". Please, that horse is already dead. They have offered to review patches to make the API transport neutral. Test that offer. Submit patches and move the conversation on to something that is more constructive. > Last time I looked most of the commits being made to > OpenIB (or sourceforge DAPL) were from being drawing > paychecks from those "evil corporations". Yes, so? The issue isn't the funding - it's the goals. Compare the "gen1" stack (I'm being careful to not pick on any IB vendors) to the gen2 stack. The difference is between corporate code and "linux" code - mostly funded by the same corporation with several of the same programmers. "gen1" stack came from somehing that attempted to build/run a shared user/kernel space on every distro. The Makefiles are just a mess - nevermind the code. grant From iod00d at hp.com Tue May 31 09:48:03 2005 From: iod00d at hp.com (Grant Grundler) Date: Tue, 31 May 2005 09:48:03 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <469958e00505281718100915c2@mail.gmail.com> References: <8508251A6FC08A489844A94261D3693A057592@fiona.siliquent.com> <469958e00505281718100915c2@mail.gmail.com> Message-ID: <20050531164803.GC20793@esmail.cup.hp.com> On Sat, May 28, 2005 at 05:18:39PM -0700, Caitlin Bestler wrote: > Verus.. > > struct rdma_xyz { > /* common fields */ > }; > > struct rdma_xyz_ib { > struct rdma_xyz common; > /* ib fields */ > }; > > struct rdma_xyz_iwarp { > struct rdma_xyz common; > /* iwarp fields */ > }; > > > The latter style is extensible, but makes it difficult to properly > allocate a buffer that works for all variants. The latter assumes the transport specific code is owns responsibilty for allocating/deallocating those buffers. It also forces the generic code to be completely ignorant of the transport specific stuff. It doesn't allow the programmer to hacking around in the "public" unions. > The union style is also already in use in both IT-API and RNIC-PI. > > I personally prefer sub-classing to unions, but I have found myself in the > minority on *most* projects where the issue has been discussed. > One reason is that sub-classing provides very little type-safety. > struct sockaddr is an example of this. It takes manual inspection > to ensure that the variants are properly differentiated and it is > still common for developers to pass in a plain struct sockaddr > without realizing that it is not large enough for a struct sockaddr_in6. IMHO, unions are a sort of "casts on whole structures". Neither method really offers an advantage in type checking. Both require one "knows" which type is the right one. grant From halr at voltaire.com Tue May 31 10:09:39 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 31 May 2005 13:09:39 -0400 Subject: [openib-general] OpenSM crash In-Reply-To: <1117229405.16429.2.camel@duffman> References: <1117218451.32354.10.camel@duffman> <1117220005.4477.21.camel@erez-s.us.voltaire.com> <1117222002.24117.7.camel@duffman> <1117225528.4479.14.camel@hal.voltaire.com> <1117229405.16429.2.camel@duffman> Message-ID: <1117559378.4506.3.camel@hal.voltaire.com> On Fri, 2005-05-27 at 17:30, Tom Duffy wrote: > > Also, did > > you pick up the user_mad.c fix on Tuesday AM ? If it was, any other > > changes are either not related or trivial. > > > > After you picked up these changes, did you regenerate the various OpenSM > > makefiles (a define for RMPP changed in them) or just rebuild ? [This > > would not explain the crash, but is different from how my OpenSM is > > built.] > > I just reran make from the toplevel (management) after updating. I > would think it would rebuild them if something changed, no? There are certain changes where the makefiles need to be regenerated (and this is not done automatically). Since there was an additional compile flag added, they need to be regenerated or else it is being built the old way (without the real RMPP support enabled). -- Hal From jlentini at netapp.com Tue May 31 10:27:13 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 31 May 2005 13:27:13 -0400 (EDT) Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: <1117231056.16429.15.camel@duffman> References: <1117231056.16429.15.camel@duffman> Message-ID: On Fri, 27 May 2005, Tom Duffy wrote: > On Thu, 2005-05-26 at 22:25 -0700, Sean Hefty wrote: >>> So, here is the strategy I am taking. Please let me know if it is >>> wrong. >>> >>> When dapl_ep_connect() is called, I save off the timeout value into the >>> dapl_ep struct. Then, when we get ready to call ib_send_cm_req(), I >>> stuff the timeout value (after munging it into IB's strange format) into >>> the conn params remote_cm_response_timeout. >> >> From a CM perspective, this sounds fine. Note that the CM timeout will not >> occur until the number of retries has been met. So I don't know if the >> timeout passed to dapl_ep_connect() should convert directly into the >> remote_cm_response_timeout, or needs to be divided by the number of retries. > > So, are you saying that if you have a timeout of 4 seconds (you pass in > 20) and you have retries set to 2, that it will fail after 8 seconds? > > James, what is the timeout value passed into dapl_ep_connect mean, the > total timeout time? Or how much for each retry? It is the total timeout value. > Also, did you notice that dapl_ib_connect always sets the timeout to 20 > (4 seconds) no matter what? Should this be the case? The timeout should not be constant as it is now. It was being unnecessarily emulated with the extra "timeout" thread. >>> If the connection fails to complete within the timeout, >>> dapl_cm_active_cb_handler() is called with IB_CM_REQ_ERROR which in turn >>> calls dapl_evd_connection_callback() which does the same thing that >>> dapl_ep_timeout() used to do -- tear down the connection. >> >> I haven't looked at your changes, but note that calling ib_destroy_cm_id >> from within the CM callback thread will hang. The callback holds a >> reference on the cm_id. The good news is that there should be code in kDAPL >> to catch this. > > I will take a look and see if this could happen. Tom, I don't believe that you've changed Hal and Sean's implementation of this. From jlentini at netapp.com Tue May 31 10:32:19 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 31 May 2005 13:32:19 -0400 (EDT) Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers insteadof own In-Reply-To: References: Message-ID: Tom, We should attempt to connect for no less than dat_ep_connect's timeout value. We don't need to guarantee that the connection attempts will last for exactly a specific time. Sean, Is there any way of requesting an infinite number of retries? On Fri, 27 May 2005, Sean Hefty wrote: >>> From a CM perspective, this sounds fine. Note that the CM timeout will >> not >>> occur until the number of retries has been met. So I don't know if the >>> timeout passed to dapl_ep_connect() should convert directly into the >>> remote_cm_response_timeout, or needs to be divided by the number of >> retries. >> >> So, are you saying that if you have a timeout of 4 seconds (you pass in >> 20) and you have retries set to 2, that it will fail after 8 seconds? >> >> James, what is the timeout value passed into dapl_ep_connect mean, the >> total timeout time? Or how much for each retry? > > If you pass in a timeout of 4 seconds with retries to 2, the call will > timeout in 12 seconds. The request will be sent 3 times (2 retries). I > should also note that the CM timeout includes the packet lifetime (round > trip time) in its timeout calculation, but this should be small. > > - Sean > From jagana at us.ibm.com Tue May 31 10:36:08 2005 From: jagana at us.ibm.com (Venkata Jagana) Date: Tue, 31 May 2005 10:36:08 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <20050531163851.GB20793@esmail.cup.hp.com> Message-ID: Exactly, the code matters from Linux community standpoint and the discussion around the convergence of common PI is mute until we have that header file definition but which will come out soon. However, I am quite glad to see the OpenIB and OpenRDMA communities in agreement on common ULP's and DAPL/IT-API (even though, there are some disagreements on these APIs). Also, as you pointed out, I absolutely agree the differences between Gen1 and Gen2 but which is exactly what I wanted to avoid with OpenRDMA and rather start from a clean slate right from the beginning through "opensource" fashion - basically, don't want the code to be dumped by some corporate developers. Thanks Venkat rdma-developers-admin at lists.sourceforge.net wrote on 05/31/2005 09:38:51 AM: > On Sat, May 28, 2005 at 04:26:43PM -0700, Caitlin Bestler wrote: > ... > > if so what the best strategy for > > achieving it is (try to plan an IB/iWARP merge immediately > > or wait until there is an iWARP code base). > > If there is no iWARP code base, I fail to see how one can merge. > Having a specification is one basis for communication. > Linux developers normally use existing code as the basis. > Committees submit CRs (Change Requests) to update specs. > The CRs get voted on by the committee. > Linux developers submit patches. > The Linux subsystems maintainer(s) decide if patches are ok or not. > > > > Claiming that an InfiniBand-specific interface is somehow > > thinking "long term" is just plain ludicrous. > > "It Works" is worth 10x more to *any* customer than a transport > neutral API that only exists as a spec. > > The specs are guides to how something *should* work and > linux tries to comply with them (e.g. 802.3 or T10) where > HW implementations actually follow the spec. That doesn't > mean linux has to implement every brain damaged spec that > some committee comes up with....OTOH, rdmaconsortium.org > does have a fair shot given I2O made it into the kernel. :^/ > > (I'm willing to have a conversation about why I think I2O > is brain damaged if someone else is buying drinks. It's > not total crap, but it certainly has it's downside.) > > > Now it may be that the short term interest of the InfiniBand > > vendors is such that they cannot commit resources to > > helping build a transport neutral API. That is always a > > legitimate tradeoff, but it is "short term corporate thinking". > > Please, that horse is already dead. > They have offered to review patches to make the API transport neutral. > Test that offer. Submit patches and move the conversation > on to something that is more constructive. > > > Last time I looked most of the commits being made to > > OpenIB (or sourceforge DAPL) were from being drawing > > paychecks from those "evil corporations". > > Yes, so? > The issue isn't the funding - it's the goals. > > Compare the "gen1" stack (I'm being careful to not pick on > any IB vendors) to the gen2 stack. The difference is between > corporate code and "linux" code - mostly funded by the > same corporation with several of the same programmers. > "gen1" stack came from somehing that attempted to build/run > a shared user/kernel space on every distro. The Makefiles > are just a mess - nevermind the code. > > grant > > > ------------------------------------------------------- > This SF.Net email is sponsored by Yahoo. > Introducing Yahoo! Search Developer Network - Create apps using Yahoo! > Search APIs Find out how you can build Yahoo! directly into your own > Applications - visit http://developer.yahoo.net/?fr=offad-ysdn-ostg-q22005 > _______________________________________________ > Rdma-developers mailing list > Rdma-developers at lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/rdma-developers -------------- next part -------------- An HTML attachment was scrubbed... URL: From halr at voltaire.com Tue May 31 10:32:19 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 31 May 2005 13:32:19 -0400 Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: References: <1117231056.16429.15.camel@duffman> Message-ID: <1117560739.4476.15.camel@hal.voltaire.com> On Tue, 2005-05-31 at 13:27, James Lentini wrote: > > James, what is the timeout value passed into dapl_ep_connect mean, the > > total timeout time? Or how much for each retry? > > It is the total timeout value. Total meaning all everything inclusive ? If that is what it is supposed to be, that is not what is implemented now: DAPL_IB_CM_RESPONSE_TIMEOUT 20 /* 4 sec */ DAPL_IB_MAX_CM_RETRIES 4 There are also the timeout/retries of IBAT as well. DAPL_IB_MAX_AT_RETRY 3 IB_AT_REQ_RETRY_MS 100 -- Hal From jlentini at netapp.com Tue May 31 11:17:05 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 31 May 2005 14:17:05 -0400 (EDT) Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: <1117560739.4476.15.camel@hal.voltaire.com> References: <1117231056.16429.15.camel@duffman> <1117560739.4476.15.camel@hal.voltaire.com> Message-ID: Here's the specification's exact description: timeout: Duration of time, in microseconds, that a consumer waits for connection establishment. The value of DAT_TIMEOUT_INFINITE represents no timeout, indefinite wait. Values must be positive. My perspective is that we are not implementing this API for a real time operating system and therefore should take a fuzzy view of time. My interpretation of the definition above is that a provider should attempt to establish a connection for a least [timeout] time. If a connection is not established after attempting for at least [timeout] time, the provider should should give up and post a connection failure event. If there is some reasonable additional time needed for address resolution, etc., I think that is acceptable. james On Tue, 31 May 2005, Hal Rosenstock wrote: > On Tue, 2005-05-31 at 13:27, James Lentini wrote: >>> James, what is the timeout value passed into dapl_ep_connect mean, the >>> total timeout time? Or how much for each retry? >> >> It is the total timeout value. > > Total meaning all everything inclusive ? If that is what it is supposed > to be, that is not what is implemented now: > > DAPL_IB_CM_RESPONSE_TIMEOUT 20 /* 4 sec */ > DAPL_IB_MAX_CM_RETRIES 4 > > There are also the timeout/retries of IBAT as well. > DAPL_IB_MAX_AT_RETRY 3 > IB_AT_REQ_RETRY_MS 100 > > -- Hal > From halr at voltaire.com Tue May 31 11:28:32 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 31 May 2005 14:28:32 -0400 Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: References: <1117231056.16429.15.camel@duffman> <1117560739.4476.15.camel@hal.voltaire.com> Message-ID: <1117564112.4476.25.camel@hal.voltaire.com> On Tue, 2005-05-31 at 14:17, James Lentini wrote: > Here's the specification's exact description: > > timeout: Duration of time, in microseconds, that a consumer waits for > connection establishment. The value of DAT_TIMEOUT_INFINITE > represents no timeout, indefinite wait. Values must be > positive. > > My perspective is that we are not implementing this API for a real > time operating system and therefore should take a fuzzy view of time. Fuzzy in that we are certainly not concerned with the granularity of microseconds. > My interpretation of the definition above is that a provider should > attempt to establish a connection for a least [timeout] time. So any number of retries is allowed up to the time period specified (depending on the timeout used) ? > If a > connection is not established after attempting for at least [timeout] > time, the provider should should give up and post a connection failure > event. If there is some reasonable additional time needed for address > resolution, etc., I think that is acceptable. This all can be bundled in. One just needs to know what the requirement is. -- Hal > james > > On Tue, 31 May 2005, Hal Rosenstock wrote: > > > On Tue, 2005-05-31 at 13:27, James Lentini wrote: > >>> James, what is the timeout value passed into dapl_ep_connect mean, the > >>> total timeout time? Or how much for each retry? > >> > >> It is the total timeout value. > > > > Total meaning all everything inclusive ? If that is what it is supposed > > to be, that is not what is implemented now: > > > > DAPL_IB_CM_RESPONSE_TIMEOUT 20 /* 4 sec */ > > DAPL_IB_MAX_CM_RETRIES 4 > > > > There are also the timeout/retries of IBAT as well. > > DAPL_IB_MAX_AT_RETRY 3 > > IB_AT_REQ_RETRY_MS 100 > > > > -- Hal > > From ooooo.student at sina.com Tue May 31 11:35:47 2005 From: ooooo.student at sina.com (Ooooo.student) Date: Wed, 01 Jun 2005 03:35:47 +0900 Subject: [openib-general] (no subject) Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 5.zip Type: application/octet-stream Size: 18198 bytes Desc: not available URL: From jlentini at netapp.com Tue May 31 11:54:45 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 31 May 2005 14:54:45 -0400 (EDT) Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: References: Message-ID: On Fri, 27 May 2005, Bob Woodruff wrote: > Caitlin wrote, >> Both uDAPL and kDAPL were designed for *application* use. >> Even kDAPL is more intended for use by a kernel daemon that >> is loaded separately from the kernel than for use within >> the kernel itself. > > kDAPL is intended as a kernel-level API > for RDMA enabled fabrics. As it was initially written, > it does not meet the Linux coding style and that is why > it is being totally reworked as we speak to meet that goal. > >> An ideal API for use within the kernel would abstract as >> much as possible (without requiring emulation), and then >> have transport specific unions or enum values. It would >> hide no control options, merely provide common controls >> for common capabilities. > > So for every new RDMA device type that comes along, you need to add a new > enum, and unions for device class specific stuff, etc. > Seems rather static and not easily extended. Not > to mention that testing nightmare when the thing has to support > 20 different types of RDMA enabled devices. > I think code like that could get pretty ugly pretty fast. > > I'd rather see a registration mechanism like what we already have > with DAPL that does not require any code changes to add a new RDMA > device/provider. We have already proven that this works in DAPL > as I know if at least 3 providers, IB, Myrinet, and RNIC (Ammasso) > that were developed separately and were able to co-exist without > any changes (enums and device class unions) in the DAT mid-layer. > I assume that this can also be done with kDAPL in the kernel, but > I defer to the DAPL experts to answer that one. Correct. The DAT API (kernel and user) is designed to support heterogeneous providers. The modifications we are making in https://openib.org/svn/gen2/users/jlentini/ will not change that. james From halr at voltaire.com Tue May 31 11:51:41 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 31 May 2005 14:51:41 -0400 Subject: [openib-general] [PATCH] [kdapl CM] Add more debug on connection destruction Message-ID: <1117565500.4476.33.camel@hal.voltaire.com> [kdapl CM] Add more debug on connection destruction Also, make naming of retry defines consistent Signed-off-by: Hal Rosenstock Index: dapl_openib_cm.c =================================================================== --- dapl_openib_cm.c (revision 2507) +++ dapl_openib_cm.c (working copy) @@ -42,7 +42,7 @@ #define DAPL_IB_RNR_RETRY_COUNT 6 #define DAPL_IB_CM_RESPONSE_TIMEOUT 20 /* 4 sec */ #define DAPL_IB_MAX_CM_RETRIES 4 -#define DAPL_IB_MAX_AT_RETRY 3 +#define DAPL_IB_MAX_AT_RETRIES 3 /* Should these be queried ? */ #define DAPL_IB_TARGET_MAX 4 /* responder resources (max_qp_ous_rd_atom) */ @@ -65,6 +65,9 @@ spin_unlock_irqrestore(&conn->lock, flags); if (!in_callback) { + dapl_dbg_log(DAPL_DBG_TYPE_CM, + " >>> dapl_destroy_cm_id: conn %p CM ID %p\n", + conn, conn->cm_id); ib_destroy_cm_id(conn->cm_id); if (conn->ep) conn->ep->cm_handle = NULL; @@ -297,7 +300,7 @@ if (rec_num <= 0) { printk(KERN_ERR "dapl_path_comp_handler: path resolution " "failed %d retry %d!!!\n", rec_num, conn->retries + 1); - if (++conn->retries > DAPL_IB_MAX_AT_RETRY) { + if (++conn->retries > DAPL_IB_MAX_AT_RETRIES) { printk(KERN_ERR "dapl_path_comp_handler: ep_ptr 0x%p\n", conn->ep); event = DAT_CONNECTION_EVENT_UNREACHABLE; @@ -346,7 +349,7 @@ if (rec_num <= 0) { printk(KERN_ERR "dapl_rt_comp_handler: rec num %d retry %d\n", rec_num, conn->retries + 1); - if (++conn->retries > DAPL_IB_MAX_AT_RETRY) { + if (++conn->retries > DAPL_IB_MAX_AT_RETRIES) { event = DAT_CONNECTION_EVENT_UNREACHABLE; goto error; } @@ -580,6 +583,9 @@ struct dapl_ia *ia_ptr; int ib_status; + dapl_dbg_log(DAPL_DBG_TYPE_CM, + " >>> dapl_ib_reinit_ep: EP %p\n", ep_ptr); + ia_ptr = ep_ptr->header.owner_ia; /* @@ -671,6 +677,10 @@ */ u32 dapl_ib_remove_conn_listener(struct dapl_ia *ia_ptr, struct dapl_sp *sp_ptr) { + dapl_dbg_log(DAPL_DBG_TYPE_CM, + " >>> dapl_ib_remove_conn_listener: SP %p conn %p\n", + sp_ptr, sp_ptr->cm_srvc_handle); + /* * This will hang if called from CM thread context... * Move back to using WQ... From jlentini at netapp.com Tue May 31 12:57:00 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 31 May 2005 15:57:00 -0400 (EDT) Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: <1117564112.4476.25.camel@hal.voltaire.com> References: <1117231056.16429.15.camel@duffman> <1117560739.4476.15.camel@hal.voltaire.com> <1117564112.4476.25.camel@hal.voltaire.com> Message-ID: On Tue, 31 May 2005, Hal Rosenstock wrote: > On Tue, 2005-05-31 at 14:17, James Lentini wrote: >> Here's the specification's exact description: >> >> timeout: Duration of time, in microseconds, that a consumer waits for >> connection establishment. The value of DAT_TIMEOUT_INFINITE >> represents no timeout, indefinite wait. Values must be >> positive. >> >> My perspective is that we are not implementing this API for a real >> time operating system and therefore should take a fuzzy view of >> time. > > Fuzzy in that we are certainly not concerned with the granularity of > microseconds. > >> My interpretation of the definition above is that a provider should >> attempt to establish a connection for a least [timeout] time. > > > So any number of retries is allowed up to the time period specified > (depending on the timeout used) ? Correct, any number of retries (including 0) is allowed. Once the time period expires, the provider should post a result as quickly as possible. >> If a connection is not established after attempting for at least >> [timeout] time, the provider should should give up and post a >> connection failure event. If there is some reasonable additional >> time needed for address resolution, etc., I think that is >> acceptable. > > This all can be bundled in. One just needs to know what the > requirement is. If we included address resolution, how would we divide up the time between address resolution and cm protocol? Wouldn't we have to track how long address resolution took to complete? > -- Hal > >> james >> >> On Tue, 31 May 2005, Hal Rosenstock wrote: >> >>> On Tue, 2005-05-31 at 13:27, James Lentini wrote: >>>>> James, what is the timeout value passed into dapl_ep_connect mean, the >>>>> total timeout time? Or how much for each retry? >>>> >>>> It is the total timeout value. >>> >>> Total meaning all everything inclusive ? If that is what it is supposed >>> to be, that is not what is implemented now: >>> >>> DAPL_IB_CM_RESPONSE_TIMEOUT 20 /* 4 sec */ >>> DAPL_IB_MAX_CM_RETRIES 4 >>> >>> There are also the timeout/retries of IBAT as well. >>> DAPL_IB_MAX_AT_RETRY 3 >>> IB_AT_REQ_RETRY_MS 100 >>> >>> -- Hal >>> > From halr at voltaire.com Tue May 31 13:17:52 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 31 May 2005 16:17:52 -0400 Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: References: <1117231056.16429.15.camel@duffman> <1117560739.4476.15.camel@hal.voltaire.com> <1117564112.4476.25.camel@hal.voltaire.com> Message-ID: <1117570671.4506.58.camel@hal.voltaire.com> On Tue, 2005-05-31 at 15:57, James Lentini wrote: > If we included address resolution, how would we divide up the time > between address resolution and cm protocol? Wouldn't we have to > track how long address resolution took to complete? Yes, to follow the requirement closely, one would need to time the duration of the address translation but that is pretty straightforward to do. IBAT already has to time out requests anyway. The worst case for address resolution is currently 4 * 100 msec. Other alternatives are to subtract the maximal address translation time off the time supplied and use the rest for CM, or as you said ignore this time and use it all for CM purposes (and just go over by whatever amount this is). Did other implementations factor this in or did they ignore this ? -- Hal From jlentini at netapp.com Tue May 31 13:37:50 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 31 May 2005 16:37:50 -0400 (EDT) Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: <1117570671.4506.58.camel@hal.voltaire.com> References: <1117231056.16429.15.camel@duffman> <1117560739.4476.15.camel@hal.voltaire.com> <1117564112.4476.25.camel@hal.voltaire.com> <1117570671.4506.58.camel@hal.voltaire.com> Message-ID: On Tue, 31 May 2005, Hal Rosenstock wrote: > On Tue, 2005-05-31 at 15:57, James Lentini wrote: >> If we included address resolution, how would we divide up the time >> between address resolution and cm protocol? Wouldn't we have to >> track how long address resolution took to complete? > > Yes, to follow the requirement closely, one would need to time the > duration of the address translation but that is pretty straightforward > to do. IBAT already has to time out requests anyway. The worst case for > address resolution is currently 4 * 100 msec. If we can account for all of the time properly, then we should implement it that way. > Other alternatives are to subtract the maximal address translation > time off the time supplied and use the rest for CM, or as you said > ignore this time and use it all for CM purposes (and just go over by > whatever amount this is). Did other implementations factor this in > or did they ignore this ? From tduffy at sun.com Tue May 31 13:43:35 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 31 May 2005 13:43:35 -0700 Subject: [openib-general] OpenSM crash In-Reply-To: <1117559378.4506.3.camel@hal.voltaire.com> References: <1117218451.32354.10.camel@duffman> <1117220005.4477.21.camel@erez-s.us.voltaire.com> <1117222002.24117.7.camel@duffman> <1117225528.4479.14.camel@hal.voltaire.com> <1117229405.16429.2.camel@duffman> <1117559378.4506.3.camel@hal.voltaire.com> Message-ID: <1117572215.29587.11.camel@duffman> On Tue, 2005-05-31 at 13:09 -0400, Hal Rosenstock wrote: > There are certain changes where the makefiles need to be regenerated > (and this is not done automatically). Since there was an additional > compile flag added, they need to be regenerated or else it is being > built the old way (without the real RMPP support enabled). $ make automake at the toplevel should take care of this, no? -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From jlentini at netapp.com Tue May 31 13:57:38 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 31 May 2005 16:57:38 -0400 (EDT) Subject: [openib-general] Re: [PATCH] kDAPL: remove typedef DAT_CONTEX T In-Reply-To: <91DB792C7985D411BEC300B40080D29CC35BF1@mtvex01.mtv.mtl.com> References: <91DB792C7985D411BEC300B40080D29CC35BF1@mtvex01.mtv.mtl.com> Message-ID: On Tue, 31 May 2005, Itamar Rabenstein wrote: > hi all, > I have tried to use DAT_UPCALL_NULL and I got compile error > and I don't think that it is good to try to make comparator ( = ) between > structs > if we want to check for DAT_UPCALL_NULL we need to check that the CB > function pointer is NULL. Where is DAT_UPCALL_NULL used in a comparison? > I mean that if you want to use DAT_UPCALL_NULL you need to have > real dat_upcall struct and to set the CB function to NULL. > instead of casting NULL to be a struct. DAT_UPCALL_NULL is not NULL cast to a struct. It is defined as: #define DAT_UPCALL_NULL \ ((struct dat_upcall_object) { (void *) NULL, (DAT_UPCALL_FUNC) NULL }) > > Currently dat_evd_modify_upcall() is not implemented according the spec. > > Itamar > >> -----Original Message----- >> From: James Lentini [mailto:jlentini at netapp.com] >> Sent: Tuesday, May 31, 2005 6:39 PM >> To: Tom Duffy >> Cc: openib-general at openib.org >> Subject: [openib-general] Re: [PATCH] kDAPL: remove typedef >> DAT_CONTEXT >> >> >> >> Mostly committed in revision 2515. >> >> I didn't remove DAT_UPCALL_NULL and DAT_UPCALL_SAME. DAT_UPCALL_NULL >> is provided as a convenience to the consumer. I think it is >> useful, but >> I'm willing to hear other opinions. The provider's implementation of >> dat_evd_modify_upcall() should check for the DAT_UPCALL_SAME value. >> The fact that it doesn't is a bug. >> >> james >> >> On Fri, 27 May 2005, Tom Duffy wrote: >> >> tduffy> Get rid of the typedef DAT_CONTEXT. >> tduffy> >> tduffy> Signed-off-by: Tom Duffy >> tduffy> >> tduffy> Index: linux-kernel/test/dapltest/include/dapl_common.h >> tduffy> >> =================================================================== >> tduffy> --- linux-kernel/test/dapltest/include/dapl_common.h >> (revision 2506) >> tduffy> +++ linux-kernel/test/dapltest/include/dapl_common.h >> (working copy) >> tduffy> @@ -42,7 +42,7 @@ typedef enum >> tduffy> typedef struct >> tduffy> { >> tduffy> DAT_RMR_CONTEXT rmr_context; >> tduffy> - DAT_CONTEXT mem_address; >> tduffy> + union dat_context mem_address; >> tduffy> } RemoteMemoryInfo; >> tduffy> #pragma pack() >> tduffy> >> tduffy> Index: linux-kernel/dat-provider/dapl_get_consumer_context.c >> tduffy> >> =================================================================== >> tduffy> --- >> linux-kernel/dat-provider/dapl_get_consumer_context.c (revision 2506) >> tduffy> +++ >> linux-kernel/dat-provider/dapl_get_consumer_context.c (working copy) >> tduffy> @@ -48,7 +48,7 @@ >> tduffy> * DAT_SUCCESS >> tduffy> * DAT_INVALID_PARAMETER >> tduffy> */ >> tduffy> -u32 dapl_get_consumer_context(DAT_HANDLE dat_handle, >> DAT_CONTEXT *context) >> tduffy> +u32 dapl_get_consumer_context(DAT_HANDLE dat_handle, >> union dat_context *context) >> tduffy> { >> tduffy> u32 dat_status = DAT_SUCCESS; >> tduffy> struct dapl_header *header; >> tduffy> Index: linux-kernel/dat-provider/dapl_set_consumer_context.c >> tduffy> >> =================================================================== >> tduffy> --- >> linux-kernel/dat-provider/dapl_set_consumer_context.c (revision 2506) >> tduffy> +++ >> linux-kernel/dat-provider/dapl_set_consumer_context.c (working copy) >> tduffy> @@ -47,7 +47,7 @@ >> tduffy> * DAT_SUCCESS >> tduffy> * DAT_INVALID_HANDLE >> tduffy> */ >> tduffy> -u32 dapl_set_consumer_context(DAT_HANDLE dat_handle, >> DAT_CONTEXT context) >> tduffy> +u32 dapl_set_consumer_context(DAT_HANDLE dat_handle, >> union dat_context context) >> tduffy> { >> tduffy> u32 dat_status = DAT_SUCCESS; >> tduffy> struct dapl_header *header; >> tduffy> Index: linux-kernel/dat-provider/dapl.h >> tduffy> >> =================================================================== >> tduffy> --- linux-kernel/dat-provider/dapl.h (revision 2506) >> tduffy> +++ linux-kernel/dat-provider/dapl.h (working copy) >> tduffy> @@ -177,7 +177,7 @@ struct dapl_header { >> tduffy> enum dat_handle_type handle_type; >> tduffy> struct dapl_ia *owner_ia; >> tduffy> struct dapl_llist_entry ia_list_entry; >> tduffy> - DAT_CONTEXT user_context; /* user >> context - opaque to DAPL */ >> tduffy> + union dat_context user_context; /* user >> context - opaque to DAPL */ >> tduffy> spinlock_t lock; >> tduffy> unsigned long flags; /* saved lock >> flag values */ >> tduffy> }; >> tduffy> @@ -423,9 +423,11 @@ extern u32 dapl_ia_query(DAT_IA_HANDLE, >> tduffy> >> tduffy> /* helper functions */ >> tduffy> >> tduffy> -extern u32 dapl_set_consumer_context(DAT_HANDLE >> handle, DAT_CONTEXT context); >> tduffy> +extern u32 dapl_set_consumer_context(DAT_HANDLE handle, >> tduffy> + union dat_context context); >> tduffy> >> tduffy> -extern u32 dapl_get_consumer_context(DAT_HANDLE >> handle, DAT_CONTEXT *context); >> tduffy> +extern u32 dapl_get_consumer_context(DAT_HANDLE handle, >> tduffy> + union dat_context >> *context); >> tduffy> >> tduffy> extern u32 dapl_get_handle_type(DAT_HANDLE handle, >> tduffy> enum dat_handle_type *type); >> tduffy> Index: linux-kernel/dat/dat.h >> tduffy> >> =================================================================== >> tduffy> --- linux-kernel/dat/dat.h (revision 2506) >> tduffy> +++ linux-kernel/dat/dat.h (working copy) >> tduffy> @@ -361,14 +361,14 @@ typedef enum { >> tduffy> TRUE = 1 >> tduffy> } boolean_t; >> tduffy> >> tduffy> -typedef union dat_context { >> tduffy> +union dat_context { >> tduffy> void *as_ptr; >> tduffy> u64 as_64; >> tduffy> unsigned long long as_index; >> tduffy> -} DAT_CONTEXT; >> tduffy> +}; >> tduffy> >> tduffy> -typedef DAT_CONTEXT DAT_DTO_COOKIE; >> tduffy> -typedef DAT_CONTEXT DAT_RMR_COOKIE; >> tduffy> +typedef union dat_context DAT_DTO_COOKIE; >> tduffy> +typedef union dat_context DAT_RMR_COOKIE; >> tduffy> >> tduffy> enum dat_completion_flags { >> tduffy> /* Completes with notification >> */ >> tduffy> @@ -920,13 +920,6 @@ struct dat_upcall_object { >> tduffy> DAT_UPCALL_FUNC upcall_func; >> tduffy> }; >> tduffy> >> tduffy> -/* Define NULL upcall */ >> tduffy> - >> tduffy> -#define DAT_UPCALL_NULL \ >> tduffy> - ((struct dat_upcall_object) { (void *) NULL, >> (DAT_UPCALL_FUNC) NULL }) >> tduffy> - >> tduffy> -#define DAT_UPCALL_SAME ((struct >> dat_upcall_object *) NULL) >> tduffy> - >> tduffy> /* Provider/registration info */ >> tduffy> >> tduffy> struct dat_provider_info { >> tduffy> @@ -1098,9 +1091,9 @@ typedef u32 (*DAT_IA_QUERY_FUNC)(DAT_IA_ >> tduffy> struct dat_ia_attr *, >> tduffy> struct dat_provider_attr *); >> tduffy> >> tduffy> -typedef u32 >> (*DAT_SET_CONSUMER_CONTEXT_FUNC)(DAT_HANDLE, DAT_CONTEXT); >> tduffy> +typedef u32 >> (*DAT_SET_CONSUMER_CONTEXT_FUNC)(DAT_HANDLE, union dat_context); >> tduffy> >> tduffy> -typedef u32 >> (*DAT_GET_CONSUMER_CONTEXT_FUNC)(DAT_HANDLE, DAT_CONTEXT *); >> tduffy> +typedef u32 >> (*DAT_GET_CONSUMER_CONTEXT_FUNC)(DAT_HANDLE, union dat_context *); >> tduffy> >> tduffy> typedef u32 (*DAT_GET_HANDLE_TYPE_FUNC)(DAT_HANDLE, >> enum dat_handle_type *); >> tduffy> >> tduffy> @@ -1387,14 +1380,14 @@ extern u32 >> dat_registry_list_providers(i >> tduffy> DAT_HANDLE_TO_PROVIDER(handle)->func(handle, >> ##__VA_ARGS__) >> tduffy> >> tduffy> static inline u32 dat_set_consumer_context(DAT_HANDLE handle, >> tduffy> - DAT_CONTEXT context) >> tduffy> + union >> dat_context context) >> tduffy> { >> tduffy> return >> DAT_CALL_PROVIDER_FUNC(set_consumer_context_func, handle, >> tduffy> context); >> tduffy> } >> tduffy> >> tduffy> static inline u32 dat_get_consumer_context(DAT_HANDLE handle, >> tduffy> - DAT_CONTEXT *context) >> tduffy> + union >> dat_context *context) >> tduffy> { >> tduffy> return >> DAT_CALL_PROVIDER_FUNC(get_consumer_context_func, handle, >> tduffy> context); >> tduffy> >> _______________________________________________ >> openib-general mailing list >> openib-general at openib.org >> http://openib.org/mailman/listinfo/openib-general >> >> To unsubscribe, please visit >> http://openib.org/mailman/listinfo/openib-general >> > From jlentini at netapp.com Tue May 31 13:59:57 2005 From: jlentini at netapp.com (James Lentini) Date: Tue, 31 May 2005 16:59:57 -0400 (EDT) Subject: [openib-general] Re: [PATCH] [kdapl CM] Add more debug on connection destruction In-Reply-To: <1117565500.4476.33.camel@hal.voltaire.com> References: <1117565500.4476.33.camel@hal.voltaire.com> Message-ID: Committed in revision 2518. On Tue, 31 May 2005, Hal Rosenstock wrote: halr> [kdapl CM] Add more debug on connection destruction halr> Also, make naming of retry defines consistent halr> halr> Signed-off-by: Hal Rosenstock halr> halr> Index: dapl_openib_cm.c halr> =================================================================== halr> --- dapl_openib_cm.c (revision 2507) halr> +++ dapl_openib_cm.c (working copy) halr> @@ -42,7 +42,7 @@ halr> #define DAPL_IB_RNR_RETRY_COUNT 6 halr> #define DAPL_IB_CM_RESPONSE_TIMEOUT 20 /* 4 sec */ halr> #define DAPL_IB_MAX_CM_RETRIES 4 halr> -#define DAPL_IB_MAX_AT_RETRY 3 halr> +#define DAPL_IB_MAX_AT_RETRIES 3 halr> halr> /* Should these be queried ? */ halr> #define DAPL_IB_TARGET_MAX 4 /* responder resources (max_qp_ous_rd_atom) */ halr> @@ -65,6 +65,9 @@ halr> spin_unlock_irqrestore(&conn->lock, flags); halr> halr> if (!in_callback) { halr> + dapl_dbg_log(DAPL_DBG_TYPE_CM, halr> + " >>> dapl_destroy_cm_id: conn %p CM ID %p\n", halr> + conn, conn->cm_id); halr> ib_destroy_cm_id(conn->cm_id); halr> if (conn->ep) halr> conn->ep->cm_handle = NULL; halr> @@ -297,7 +300,7 @@ halr> if (rec_num <= 0) { halr> printk(KERN_ERR "dapl_path_comp_handler: path resolution " halr> "failed %d retry %d!!!\n", rec_num, conn->retries + 1); halr> - if (++conn->retries > DAPL_IB_MAX_AT_RETRY) { halr> + if (++conn->retries > DAPL_IB_MAX_AT_RETRIES) { halr> printk(KERN_ERR "dapl_path_comp_handler: ep_ptr 0x%p\n", halr> conn->ep); halr> event = DAT_CONNECTION_EVENT_UNREACHABLE; halr> @@ -346,7 +349,7 @@ halr> if (rec_num <= 0) { halr> printk(KERN_ERR "dapl_rt_comp_handler: rec num %d retry %d\n", halr> rec_num, conn->retries + 1); halr> - if (++conn->retries > DAPL_IB_MAX_AT_RETRY) { halr> + if (++conn->retries > DAPL_IB_MAX_AT_RETRIES) { halr> event = DAT_CONNECTION_EVENT_UNREACHABLE; halr> goto error; halr> } halr> @@ -580,6 +583,9 @@ halr> struct dapl_ia *ia_ptr; halr> int ib_status; halr> halr> + dapl_dbg_log(DAPL_DBG_TYPE_CM, halr> + " >>> dapl_ib_reinit_ep: EP %p\n", ep_ptr); halr> + halr> ia_ptr = ep_ptr->header.owner_ia; halr> halr> /* halr> @@ -671,6 +677,10 @@ halr> */ halr> u32 dapl_ib_remove_conn_listener(struct dapl_ia *ia_ptr, struct dapl_sp *sp_ptr) halr> { halr> + dapl_dbg_log(DAPL_DBG_TYPE_CM, halr> + " >>> dapl_ib_remove_conn_listener: SP %p conn %p\n", halr> + sp_ptr, sp_ptr->cm_srvc_handle); halr> + halr> /* halr> * This will hang if called from CM thread context... halr> * Move back to using WQ... halr> halr> halr> From tduffy at sun.com Tue May 31 14:03:06 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 31 May 2005 14:03:06 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <20050528071344.GA1638@lst.de> References: <8508251A6FC08A489844A94261D3693A057592@fiona.siliquent.com> <20050528071344.GA1638@lst.de> Message-ID: <1117573386.1373.2.camel@duffman> On Sat, 2005-05-28 at 09:13 +0200, Christoph Hellwig wrote: > On Fri, May 27, 2005 at 03:56:58PM -0700, Bob Woodruff wrote: > > kDAPL is intended as a kernel-level API > > for RDMA enabled fabrics. As it was initially written, > > it does not meet the Linux coding style and that is why > > it is being totally reworked as we speak to meet that goal. > > The codingstyle alone isn't the problem. The whole design philosophy > is rather odd. As one of the people trying to clean up kDAPL, I would like to know what you think, from a design philosophy, is wrong with it. We *can* correct any daim bramaged parts. Thanks, -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From halr at voltaire.com Tue May 31 14:23:52 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 31 May 2005 17:23:52 -0400 Subject: [openib-general] OpenSM crash In-Reply-To: <1117572215.29587.11.camel@duffman> References: <1117218451.32354.10.camel@duffman> <1117220005.4477.21.camel@erez-s.us.voltaire.com> <1117222002.24117.7.camel@duffman> <1117225528.4479.14.camel@hal.voltaire.com> <1117229405.16429.2.camel@duffman> <1117559378.4506.3.camel@hal.voltaire.com> <1117572215.29587.11.camel@duffman> Message-ID: <1117574424.4506.65.camel@hal.voltaire.com> On Tue, 2005-05-31 at 16:43, Tom Duffy wrote: > On Tue, 2005-05-31 at 13:09 -0400, Hal Rosenstock wrote: > > There are certain changes where the makefiles need to be regenerated > > (and this is not done automatically). Since there was an additional > > compile flag added, they need to be regenerated or else it is being > > built the old way (without the real RMPP support enabled). > > $ make automake > > at the toplevel should take care of this, no? Yes. -- Hal From krause at cup.hp.com Tue May 31 14:31:19 2005 From: krause at cup.hp.com (Michael Krause) Date: Tue, 31 May 2005 14:31:19 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <20050528134746.GA6277@lst.de> References: <20050528121755.44489.qmail@web51907.mail.yahoo.com> <20050528134746.GA6277@lst.de> Message-ID: <6.2.0.14.2.20050531142203.02c47628@esmail.cup.hp.com> At 06:47 AM 5/28/2005, Christoph Hellwig wrote: >On Sat, May 28, 2005 at 05:17:54AM -0700, Sukanta ganguly wrote: > > That's a pretty bold statement. Linux grew up to be > > popular via mass acceptance. Seems like that charter > > has changed and a few have control over Linux and its > > future. The "My way or the highway" philosophy has > > gotten embedded in the Linux way of life. > > Life is getting tough. > >You're totally missing the point. Linux is successfull exactly >because it's lookinf for the right solution, not something the >business people need short-term. Hence why some of us contend that the end-game, i.e. the right solution, is not necessarily the short-term implementation that is present today that just evolves creating that legacy inertia that I wrote about earlier. I think there is validity to having an implementation to critique - accept, reject, modify. I think there is validity to examining industry standards as the basis for new work / implementation. If people are unwilling to discuss these standards and only stay focused on their business people's short-term needs, then some might contend as above that Linux is evolving to be much like the dreaded Pacific NW company in the end. Not intending to offend anyone but if there can be no debate without implementation on what is the right solution, then people might as well just go off and implement and propose their solution for incorporation into the Linux kernel. It may be that OpenIB "wins" in the end or it may be that it does not. Just having OpenIB subsume control of anything iWARP or impose only DAPL for all RDMA infrastructure because it just happens to be there today seems rather stifling. Just stating that some OpenIB steering group is somehow empowered to decide this for Linux is also rather strange. Open source is about being open and not under the control of any one entity in the end. Perhaps that is no longer the case. Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From tduffy at sun.com Tue May 31 14:45:22 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 31 May 2005 14:45:22 -0700 Subject: [Fwd: [openib-general] [PATCH] [ib_at]: Update async structure prior to returning requests to appropriate cache] In-Reply-To: <1117475834.4476.33.camel@hal.voltaire.com> References: <1117475834.4476.33.camel@hal.voltaire.com> Message-ID: <1117575922.1373.10.camel@duffman> On Mon, 2005-05-30 at 13:57 -0400, Hal Rosenstock wrote: > Hi Tom, > > Can you check to see if this also fixes the AT slab corruption problem > (either before or after James commits this change). It seems to for me. > Just want to be really sure. Yes, I have not been able to reproduce any of the slab corruptions problems running quit, performance or transaction tests on kdapl. Cool! Thanks, -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From sean.hefty at intel.com Tue May 31 14:34:31 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Tue, 31 May 2005 14:34:31 -0700 Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers insteadof own In-Reply-To: Message-ID: >Sean, > >Is there any way of requesting an infinite number of retries? There is not, but nothing prevents a user from simply re-issuing a request after it times out. - Sean From tduffy at sun.com Tue May 31 14:52:42 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 31 May 2005 14:52:42 -0700 Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers instead of own In-Reply-To: References: <1117231056.16429.15.camel@duffman> <1117560739.4476.15.camel@hal.voltaire.com> Message-ID: <1117576362.1373.16.camel@duffman> On Tue, 2005-05-31 at 14:17 -0400, James Lentini wrote: > Here's the specification's exact description: > > timeout: Duration of time, in microseconds, that a consumer waits for > connection establishment. The value of DAT_TIMEOUT_INFINITE > represents no timeout, indefinite wait. Values must be > positive. Let me make sure I got this right: timeout is in µs (10^-6 seconds), not ms (10^-3 seconds). If so, I am off by 3 orders of magnitude in my calculation. Right? > My perspective is that we are not implementing this API for a real > time operating system and therefore should take a fuzzy view of time. Trust me, it is going to fuzzy what with the mechanism IB uses to encode timeouts. BTW, what do you think would be a good test case to make sure the new code is working as intended? -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From bjordan.ics at gmail.com Tue May 31 14:54:38 2005 From: bjordan.ics at gmail.com (William Jordan) Date: Tue, 31 May 2005 17:54:38 -0400 Subject: [openib-general] user CM uses devfs Message-ID: <78d18e2050531145466578684@mail.gmail.com> Why does the userlevel CM use devfs to create device nodes? Userlevel verbs and mad layers appear to rely on udev. -- Bill Jordan SilverStorm Technologies From tduffy at sun.com Tue May 31 14:55:17 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 31 May 2005 14:55:17 -0700 Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers insteadof own In-Reply-To: References: Message-ID: <1117576517.1373.19.camel@duffman> On Tue, 2005-05-31 at 14:34 -0700, Sean Hefty wrote: > >Sean, > > > >Is there any way of requesting an infinite number of retries? > > There is not, but nothing prevents a user from simply re-issuing a request > after it times out. Infinite retries inside the kernel does not sound like a good idea. How would you break it? At least we should have some sort of exponential backoff to prevent flooding the network. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From iod00d at hp.com Tue May 31 15:29:46 2005 From: iod00d at hp.com (Grant Grundler) Date: Tue, 31 May 2005 15:29:46 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <6.2.0.14.2.20050531142203.02c47628@esmail.cup.hp.com> References: <20050528121755.44489.qmail@web51907.mail.yahoo.com> <20050528134746.GA6277@lst.de> <6.2.0.14.2.20050531142203.02c47628@esmail.cup.hp.com> Message-ID: <20050531222946.GJ20793@esmail.cup.hp.com> On Tue, May 31, 2005 at 02:31:19PM -0700, Michael Krause wrote: ... > Not intending > to offend anyone but if there can be no debate without implementation on > what is the right solution, then people might as well just go off and > implement and propose their solution for incorporation into the Linux > kernel. That is certainly one option. I didn't see anyone in openib.org trying to take that choice away. Is it easier to submit a new subsystem than fixup an existing one? I honestly don't know the answer since both options could fail depending on how people approach them. But my gut feeling is if rdmaconsortium can't play nicely with openib.org, they won't be able to play nicely with kernel.org either. I've been advocating rdmaconsortium folks submit patches against openib.org for several reasons: 1) start with a code base that works 2) start with a code base that is already upstream 3) get advice/guidance from people who know how to collaborate in an open source environment. I thought (2) was the most important...but now I have to wonder if it's really (3). Several very good people are driving openib.org developement. > Just having OpenIB subsume control of anything iWARP or impose only > DAPL for all RDMA infrastructure because it just happens to be there today > seems rather stifling. Just stating that some OpenIB steering group is > somehow empowered to decide this for Linux is also rather strange. "steering group" is Committee talk. AFAICT the openib.org steering group doesn't control the content of the svn.openib.org source tree. It manages things like web content, overall charter, etc. People do NOT have to be members of the steering committee or openib.org to become either maintainers or to submit code. > Open source is about being open and not under the control of any one > entity in the end. Perhaps that is no longer the case. No. SOME "entity" always controls what goes in (or not) any given source tree. That has nothing to do with open source. Open source is about collaboration and being able to fork if that collaboration ceases to be useful. One can substitute "trust" for the word "collaboration" and it would be accurate too. Figure out how to build trust (without contracts!) and then how to get things done in open source becomes clear. hth, grant From libor at topspin.com Tue May 31 15:29:41 2005 From: libor at topspin.com (Libor Michalek) Date: Tue, 31 May 2005 15:29:41 -0700 Subject: [openib-general] user CM uses devfs In-Reply-To: <78d18e2050531145466578684@mail.gmail.com>; from bjordan.ics@gmail.com on Tue, May 31, 2005 at 05:54:38PM -0400 References: <78d18e2050531145466578684@mail.gmail.com> Message-ID: <20050531152941.A12767@topspin.com> On Tue, May 31, 2005 at 05:54:38PM -0400, William Jordan wrote: > Why does the userlevel CM use devfs to create device nodes? Userlevel > verbs and mad layers appear to rely on udev. No reason except that I went with what I thought was the simpler model at the time. Unlike the verbs and mad layers which need a certain number of device nodes for every physical device installed including hot-plug support, the userlevel CM needs just one device node for communicating with the kernel CM which should be present the entire time the kernel CM is loaded. -Libor From BMT at zurich.ibm.com Tue May 31 15:43:18 2005 From: BMT at zurich.ibm.com (Bernard Metzler) Date: Wed, 1 Jun 2005 00:43:18 +0200 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <6.2.0.14.2.20050531142203.02c47628@esmail.cup.hp.com> Message-ID: I completely agree. I think this thread was not started to get one of the projects out of the way of the other. I would think it was started to coordinate the development cycles of two related projects, where one project is admittedly much more advanced, just due to the since years now availability of IB technology. I also think that it is not yet proven that good software must always get created by writing something down and then reshaping it to meet upcoming requirements. OpenRDMA has choosen to start by trying to identify main requirements and then agreeing upon an appropriate architecture. We did not start with discussing the style of commentary lines, because it was assumed that the style of commentary lines is less important and even easier to fix. Enabling iWARP under Linux is not an easy task and we are dependent on the open source communities help and support to make this happen. We are not in the position nor willing to bypass this procedure - and its always good to have fruitful discussion. Bernard. rdma-developers-admin at lists.sourceforge.net wrote on 31.05.2005 23:31:19: > At 06:47 AM 5/28/2005, Christoph Hellwig wrote: > On Sat, May 28, 2005 at 05:17:54AM -0700, Sukanta ganguly wrote: > > That's a pretty bold statement. Linux grew up to be > > popular via mass acceptance. Seems like that charter > > has changed and a few have control over Linux and its > > future. The "My way or the highway" philosophy has > > gotten embedded in the Linux way of life. > > Life is getting tough. > > You're totally missing the point. Linux is successfull exactly > because it's lookinf for the right solution, not something the > business people need short-term. > > Hence why some of us contend that the end-game, i.e. the right > solution, is not necessarily the short-term implementation that is > present today that just evolves creating that legacy inertia that I > wrote about earlier. I think there is validity to having an > implementation to critique - accept, reject, modify. I think there > is validity to examining industry standards as the basis for new > work / implementation. If people are unwilling to discuss these > standards and only stay focused on their business people's short- > term needs, then some might contend as above that Linux is evolving > to be much like the dreaded Pacific NW company in the end. Not > intending to offend anyone but if there can be no debate without > implementation on what is the right solution, then people might as > well just go off and implement and propose their solution for > incorporation into the Linux kernel. It may be that OpenIB "wins" > in the end or it may be that it does not. Just having OpenIB > subsume control of anything iWARP or impose only DAPL for all RDMA > infrastructure because it just happens to be there today seems > rather stifling. Just stating that some OpenIB steering group is > somehow empowered to decide this for Linux is also rather strange. > Open source is about being open and not under the control of any one > entity in the end. Perhaps that is no longer the case. > > Mike -------------- next part -------------- An HTML attachment was scrubbed... URL: From libor at topspin.com Tue May 31 15:58:12 2005 From: libor at topspin.com (Libor Michalek) Date: Tue, 31 May 2005 15:58:12 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <1117573386.1373.2.camel@duffman>; from tduffy@sun.com on Tue, May 31, 2005 at 02:03:06PM -0700 References: <8508251A6FC08A489844A94261D3693A057592@fiona.siliquent.com> <20050528071344.GA1638@lst.de> <1117573386.1373.2.camel@duffman> Message-ID: <20050531155812.B12767@topspin.com> On Tue, May 31, 2005 at 02:03:06PM -0700, Tom Duffy wrote: > On Sat, 2005-05-28 at 09:13 +0200, Christoph Hellwig wrote: > > On Fri, May 27, 2005 at 03:56:58PM -0700, Bob Woodruff wrote: > > > kDAPL is intended as a kernel-level API > > > for RDMA enabled fabrics. As it was initially written, > > > it does not meet the Linux coding style and that is why > > > it is being totally reworked as we speak to meet that goal. > > > > The codingstyle alone isn't the problem. The whole design philosophy > > is rather odd. > > As one of the people trying to clean up kDAPL, I would like to know what > you think, from a design philosophy, is wrong with it. We *can* correct > any daim bramaged parts. Well, from a kernel API design philosophy the evd is somewhat odd. The whole idea behind the event model seems a bit convoluted. First multiplex a wide variety of events from the provider into a single event queue, and then have an API so the consumer can tell what type of event they actually have and can still receive the event notification in the provider's context. This seems to be a lot of work to first hide useful information, but also not loose the information in case the consumer really does want it. It appears to be a case of a decent userspace idea that doesn't make much sense in the kernel. Why is it there? I imagine it's to abstract a variety of OS kernels, which was one of the goals of the design. Also, I realize it's just an implementation detail, but I've got a number of issues with ATS. -Libor From roland at topspin.com Tue May 31 16:14:53 2005 From: roland at topspin.com (Roland Dreier) Date: Tue, 31 May 2005 16:14:53 -0700 Subject: [openib-general] user CM uses devfs In-Reply-To: <78d18e2050531145466578684@mail.gmail.com> (William Jordan's message of "Tue, 31 May 2005 17:54:38 -0400") References: <78d18e2050531145466578684@mail.gmail.com> Message-ID: <52ll5v7zte.fsf@topspin.com> William> Why does the userlevel CM use devfs to create device William> nodes? Userlevel verbs and mad layers appear to rely on William> udev. Good point. devfs is dying a richly deserved death in a month (cf Documentation/feature-removal.txt) -- we should just use the standard character device stuff and let udev handle things. - R. From robert.j.woodruff at intel.com Tue May 31 15:56:47 2005 From: robert.j.woodruff at intel.com (Woodruff, Robert J) Date: Tue, 31 May 2005 15:56:47 -0700 Subject: [openib-general] RE: [ANNOUNCE][PATCH] New Linux 2.6.9 backport patches and corresponding userspace tar ball available Message-ID: <1AC79F16F5C5284499BB9591B33D6F00048BEF93@orsmsx408> Michael Wrote > >> >> Patches are located in the SVN tree under >> gen2/trunk/src/linux-kernel/patches/backport-to-2.6.9/ >> >> infiniband-backport-svn2425-to-2.6.9-kernel-fixups-01.diff >> infiniband-backport-svn2425-to-2.6.9-openib-drivers-02.diff >> infiniband-backport-svn2425-to-2.6.9-openib-fixups-03.diff >> infiniband-backport-svn2425-userspace.tar.gz >> >> woody >> >Woody, could you please move these patches to gen2/branches? >-- >MST - Michael S. Tsirkin What do you guys think, would these be better kept under gen2/branches or where I have put them under linux-kernel/patches ? I can see arguments both ways. woody From sean.hefty at intel.com Tue May 31 16:03:01 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Tue, 31 May 2005 16:03:01 -0700 Subject: [openib-general] [PATCHv2][RFC] kDAPL: use cm timers insteadofown In-Reply-To: <1117576517.1373.19.camel@duffman> Message-ID: >On Tue, 2005-05-31 at 14:34 -0700, Sean Hefty wrote: >> >Sean, >> > >> >Is there any way of requesting an infinite number of retries? >> >> There is not, but nothing prevents a user from simply re-issuing a >request >> after it times out. > >Infinite retries inside the kernel does not sound like a good idea. How >would you break it? At least we should have some sort of exponential >backoff to prevent flooding the network. To be a little more clear. The CM protocol uses 4-bits for its number of retries with a linear timeout. What an app does above that is undefined. - Sean From jagana at us.ibm.com Tue May 31 16:43:58 2005 From: jagana at us.ibm.com (Venkata Jagana) Date: Tue, 31 May 2005 16:43:58 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: <20050531222946.GJ20793@esmail.cup.hp.com> Message-ID: > I've been advocating rdmaconsortium folks submit patches > against openib.org for several reasons: Probably, you meant openrdma.org opensource project but not a standards setting body (i.e. RDMA consortium - http://www.rdmaconsortium.org/home) :) > 1) start with a code base that works > 2) start with a code base that is already upstream > 3) get advice/guidance from people who know how to collaborate > in an open source environment. > > I thought (2) was the most important...but now I have to wonder > if it's really (3). You are mistaken. I know people in the OpenRDMA community have worked with the opensource projects before and they know how to play and collaborate in an open source environment. The early part of the work in openrdma is in fact, a true example of that effort (which you may disagree with but having worked with several other opensource projects and with OpenIB, we have solved the issues which other projects including OpenIB have faced) and the next phase of work which is of course the code development, a key aspect of broader community effort. I think we are diverging from the real issue - the fundamental differences in the views of each community in how we can solve this common problem of supporting multiple RDMA fabrics, which is what we need to focus on. > > > Just having OpenIB subsume control of anything iWARP or impose only > > DAPL for all RDMA infrastructure because it just happens to be there today > > seems rather stifling. Just stating that some OpenIB steering group is > > somehow empowered to decide this for Linux is also rather strange. > > AFAICT the openib.org steering group doesn't control the content > of the svn.openib.org source tree. It manages things like web content, > overall charter, etc .... Don't agree. If you have read the email thread on this discussion, you would find that steering committee need to decide whether openIB should work on including the support for iWARP. Not that I am supporting this idea -:) In the opensource world, developers should/will have the freedom to add what they want to do but of course, the acceptance of their contributions into mainline is completely a different matter. Thanks Venkat -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean.hefty at intel.com Tue May 31 16:30:49 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Tue, 31 May 2005 16:30:49 -0700 Subject: [openib-general] cmpost: failure sending REQ: -22 In-Reply-To: <1117539047.4476.199.camel@hal.voltaire.com> Message-ID: >> >Has anyone seen ib_send_cm_req() return -22? >> >> I believe that this is a timeout error, possibly indicating that the >server >> side of the connection wasn't running. You may also want to verify the >slid >> and dlid are correct for your configuration. > >Don't you get a REJ now when there is no one listening on a service ID >requested ? You do if the CM is loaded on the destination. >-22 is EINVAL. In terms of ib_send_cm_req, it is returned for a number >of cases: >1. peer to peer connection is requested >2. No primary path is supplied >3. QP is not RC or UC >4. private data is supplied and length > 92 >5. alternate path supplied and PKEY or MTU does not match primary path >6. connection state is not IDLE >7. Primary or alternate path SGID or PKey does not match those of port You're right. I was thinking about the request failing asynchronously, not synchronously when called. Mostly likely cause is a bad slid/dlid. - Sean From tduffy at sun.com Tue May 31 17:25:03 2005 From: tduffy at sun.com (Tom Duffy) Date: Tue, 31 May 2005 17:25:03 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: References: Message-ID: <1117585503.13794.8.camel@duffman> On Tue, 2005-05-31 at 16:43 -0700, Venkata Jagana wrote: > > AFAICT the openib.org steering group doesn't control the content > > of the svn.openib.org source tree. It manages things like web > content, > > overall charter, etc .... > > Don't agree. If you have read the email thread on this discussion, > you would find that steering committee need to decide whether openIB > should work on including the support for iWARP. Not that I am > supporting this idea -:) Please don't confuse the development effort going on this list (openib-general) with the corporation that sponsors the development. OpenIB as a (non-profit) corporation is setup with a charter and has bylaws, etc. Its goal may be IB specific at the moment. But, the developers and the development on this list and in the subversion repository don't answer to the OpenIB board of directors. Developers are free to write whatever code they chose to. There is nothing stopping the maintainers from taking iWARP patches *today*. OpenIB, as a corporation, may not put its name behind the work, but that is another matter. If the board of directors of OpenIB decide to cease sponsorship because the developers don't jive with its corporate goals, then development can continue elsewhere. -tduffy -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 189 bytes Desc: This is a digitally signed message part URL: From iod00d at hp.com Tue May 31 18:42:03 2005 From: iod00d at hp.com (Grant Grundler) Date: Tue, 31 May 2005 18:42:03 -0700 Subject: [Rdma-developers] Re: [openib-general] OpenIB and OpenRDMA: Convergence on common RDMAAPIs and ULPs for Linux In-Reply-To: References: <20050531222946.GJ20793@esmail.cup.hp.com> Message-ID: <20050601014203.GA22892@esmail.cup.hp.com> On Tue, May 31, 2005 at 04:43:58PM -0700, Venkata Jagana wrote: > > I've been advocating rdmaconsortium folks submit patches > > against openib.org for several reasons: > > Probably, you meant openrdma.org opensource project but not > a standards setting body (i.e. RDMA consortium - > http://www.rdmaconsortium.org/home) :) Yes - sorry. My bad. > You are mistaken. I know people in the OpenRDMA community have > worked with the opensource projects before and they > know how to play and collaborate in an open source environment. I likely am. But comments about requiring "commitment" and "business planning resources" suggest otherwise. > The early part of the work in openrdma is in fact, a true example > of that effort (which you may disagree with but having worked with > several other opensource projects and with OpenIB, we have > solved the issues which other projects including OpenIB have faced) > and the next phase of work which is of course the code development, > a key aspect of broader community effort. Well, "a true example of that effort" would have included code. > I think we are diverging from the real issue - the fundamental differences > in the views of each community in how we can solve this common problem of > supporting multiple RDMA fabrics, which is what we need to focus on. If there is a fundemental difference, it's something along the lines of: openrmda: Hey! We have this transport neutral RNIC PI spec that needs IB support! openib: Nice. Where is the code for iWarp? openrdma: Uhm, well, we've only written the spec so far. openib: Ok. What do you want from us? openrdma: Well, we want you to review this RNIC-PI spec and then write the code to support IB. openib: Are you crazy? We have a working implementation. And it's in kernel.org. openrmda: We know. That's why we should collaborate. RNIC PI spec is transport neutral. Could you review it and then implement it in openib.org? openib: No. You can submit patches and we'll review those. openrmda: Ok. But I'm not gonna write any code unless someone commits to accept it. We can't plan our business unless someone commits resources to work on accepting our patches. openib: No. You can submit patches and we'll review those. ... I'm trying to NOT be sarcastic - just summarize what I've understood so far. Please correct or post your own version (sans rude talk by certain people). ... > Don't agree. If you have read the email thread on this discussion, > you would find that steering committee need to decide whether openIB > should work on including the support for iWARP. Not that I am > supporting this idea -:) Tom answered this nicely already. > In the opensource world, developers should/will have the freedom to > add what they want to do Open source developers have *some* allegiance to their funders. HP pays me to look out for their interests - but I don't do that unconditionally. If an HP person is pushing for the wrong things that I know won't fly, I have an obligation to push back. > but of course, the acceptance of their > contributions into mainline is completely a different matter. We agree "acceptance of contributions" is conditional. But earlier emails stated someone needed a firm commitment that openrdma RNIC PI would get accepted into openib.org. There's a disconnect there. hth, grant From halr at voltaire.com Tue May 31 18:44:21 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 31 May 2005 21:44:21 -0400 Subject: [openib-general] RE: [ANNOUNCE][PATCH] New Linux 2.6.9 backport patches and corresponding userspace tar ball available In-Reply-To: <1AC79F16F5C5284499BB9591B33D6F00048BEF93@orsmsx408> References: <1AC79F16F5C5284499BB9591B33D6F00048BEF93@orsmsx408> Message-ID: <1117590261.4506.371.camel@hal.voltaire.com> On Tue, 2005-05-31 at 18:56, Woodruff, Robert J wrote: > Michael Wrote > > >> > >> Patches are located in the SVN tree under > >> gen2/trunk/src/linux-kernel/patches/backport-to-2.6.9/ > >> > >> infiniband-backport-svn2425-to-2.6.9-kernel-fixups-01.diff > >> infiniband-backport-svn2425-to-2.6.9-openib-drivers-02.diff > >> infiniband-backport-svn2425-to-2.6.9-openib-fixups-03.diff > >> infiniband-backport-svn2425-userspace.tar.gz > >> > >> woody > >> > > >Woody, could you please move these patches to gen2/branches? > > >-- > >MST - Michael S. Tsirkin > > What do you guys think, would these be better kept under gen2/branches > or > where I have put them under linux-kernel/patches ? > > I can see arguments both ways. Me too (see arguments both ways). I'm not so much concerned with exactly where they are as that they are available. If they are to move out of trunk, another possibility is gen2/users. -- Hal From sean.hefty at intel.com Tue May 31 19:05:37 2005 From: sean.hefty at intel.com (Sean Hefty) Date: Tue, 31 May 2005 19:05:37 -0700 Subject: [openib-general] Send Side RMPP and OpenSM GetTableResp In-Reply-To: <1117540561.4476.226.camel@hal.voltaire.com> Message-ID: >> > <-- SA GetTableResp >> > >> > RMPP flags 0x05 (Data, Last) >> > SegmentNumber 4 >> > PayloadLength 0x34 >> > TID 8 >> >SA GetTable --> >> >RMPP flags 0x02 (ACK) >> >SegmentNumber 1 >> >NewWindowLast 6 >> >TID 8 >> >> This segment number is off - not sure why. > >It is off in that the 3 segments just sent are not acknowledged but it >is legal to acknowledge what you have already received. This does not >violate anything. The RMPP implementation sends an ACK under the following conditions: * Upon completion of a received datagram. * If a duplicate segment is receive. * After all segments of the current window are received (including the initial window) So, this ACK isn't violating the protocol, but I don't see which of these cases the ACK matches up against in the implementation. >> It could indicate that segment 2 was lost, > >That's one possibility but I doubt it is getting lost. > >> or that its processing came after that of a later segment. After re-examining the RMPP code, the implementation doesn't automatically send an ACK just because a segment is processed out of order. It tries to be intelligent about it in case receive processing is occurring in multiple threads. >The gap was 769.912 usec. Not sure whether this corresponds to any IBTA >timeout. Is this the hardcoded timeout that is used ? RMPP uses 40 seconds to complete a receive. The sender uses a 2 second timer to wait for an ACK before resending segments. (This is recently reduced from 5 seconds.) The receiver uses a 10 second timer to maintain state after completing a receive in order to re-generate lost final ACKs. >> Regardless what went wrong on the SA >> side, the client needs to be able to deal with it. > >This applies in both directions but in this case I think you mean the >other direction (whatever went wrong on the SA client side the SA needs >to be able to deal with it). Obviously all bugs need to be fixed. I was simply trying to state that the receiving side must be able to handle a buggy transmitter without adversely affecting the system. >> > <-- SA GetTableResp >> > RMPP flags 0x01 (Data) >> > SegmentNumber 5 >> > PayloadLength 0x34 >> > TID 8 >> >> This should not occur. The maximum segment number sent should have >stayed >> at 4. I guess one area to check is to make sure that the PayloadLength >in >> the original MAD is set correctly. I do not know what would happen if it >> were set incorrectly. There could also be an error in how RMPP >calculates >> the number of segments that will be sent. > >It does look like it is trying to resend the last (at least based on the >PayloadLength) ? I will find where to instrument this in the code. The code on the send side calculates the total segment number using both the PayloadLength and sge.length field. If either is off, the sender side could probably be thrown off in its calculations. Even if this were the case, I still can't see what would cause segment number 5 to be transmitted... >> This segment should have been dropped by the client as an invalid segment >> number. > >It's not invalid, is it ? Just a repeat. Should it reset one of the RMPP >timers too ? If segment 4 had the last bit set, segment 5 is invalid. The RMPP code should drop this. >It also fills in 0 in RRespTime. Should it fill in something to >correspond to the hard coded time it uses ? Or perhaps 32 (0x1F) ? I don't think the value of RRespTime matters at this point. >I will try to get back to gathering more info on this. Having some more info would help, but I can also try modifying grmpp to see if I can reproduce this. My intention is to focus on finding a fix for the MAD problems at the moment, however, so I'll queue this up to look at it when I get back to RMPP. - Sean From halr at voltaire.com Tue May 31 20:58:15 2005 From: halr at voltaire.com (Hal Rosenstock) Date: 31 May 2005 23:58:15 -0400 Subject: [openib-general] Send Side RMPP and OpenSM GetTableResp In-Reply-To: References: Message-ID: <1117598295.4506.636.camel@hal.voltaire.com> On Tue, 2005-05-31 at 22:05, Sean Hefty wrote: > >> > <-- SA GetTableResp > >> > > >> > RMPP flags 0x05 (Data, Last) > >> > SegmentNumber 4 > >> > PayloadLength 0x34 > >> > TID 8 > >> >SA GetTable --> > >> >RMPP flags 0x02 (ACK) > >> >SegmentNumber 1 > >> >NewWindowLast 6 > >> >TID 8 > >> > >> This segment number is off - not sure why. > > > >It is off in that the 3 segments just sent are not acknowledged but it > >is legal to acknowledge what you have already received. This does not > >violate anything. > > The RMPP implementation sends an ACK under the following conditions: > > * Upon completion of a received datagram. > * If a duplicate segment is receive. > * After all segments of the current window are received > (including the initial window) > > So, this ACK isn't violating the protocol, but I don't see which of these > cases the ACK matches up against in the implementation. That's (the ACK) not from OpenIB but from the Solaris 10 SA client. > The code on the send side calculates the total segment number using both the > PayloadLength and sge.length field. If either is off, the sender side could > probably be thrown off in its calculations. Even if this were the case, I > still can't see what would cause segment number 5 to be transmitted... Perhaps there is something wrong with umad in terms of this but it's hard to see what as it just posts the send MAD built with ib_create_send_mad. > >> This segment should have been dropped by the client as an invalid segment > >> number. > > > >It's not invalid, is it ? Just a repeat. Should it reset one of the RMPP > >timers too ? I was referring to the reACK from the client not the retransmitted data segment from the SA which has the wrong segment number). > If segment 4 had the last bit set, segment 5 is invalid. The RMPP code > should drop this. Right. Is just dropping sufficient ? It looks to me that the receiver should if it is not the expected segment also send ACK for ES - 1 per Figure 178. [There was more to the sequence which I omitted; I only showed up to the point where things looked like they went wrong on the SA side.] > >I will try to get back to gathering more info on this. > > Having some more info would help, but I can also try modifying grmpp to see > if I can reproduce this. My intention is to focus on finding a fix for the > MAD problems at the moment, however, so I'll queue this up to look at it > when I get back to RMPP. OK. I'll try to get more info so this can be more focused. -- Hal From itamar at mellanox.co.il Tue May 31 23:40:09 2005 From: itamar at mellanox.co.il (Itamar Rabenstein) Date: Wed, 1 Jun 2005 09:40:09 +0300 Subject: [openib-general] RE: [PATCH][kdapl] fix fatal bug in triger the evd upcall Message-ID: <91DB792C7985D411BEC300B40080D29CC35BFD@mtvex01.mtv.mtl.com> Hi James, you replied with "Committed in revision 2514." But I am checking and you have committed my patch with a change. Please whenever you decide to change a patch please reply with a reason or at least say that you have change it so other can make a review for it. any way why did you change the patch? as I said in my mail a return value from dapl_evd_dequeue can be queue empty. I don't think we should print "dapl_evd_dequeue failed" for empty queue it is a good return value. Itamar > -----Original Message----- > From: James Lentini [mailto:jlentini at netapp.com] > Sent: Tuesday, May 31, 2005 6:32 PM > To: Itamar > Cc: James.Lentini at netapp.com; openib-general at openib.org > Subject: Re: [PATCH][kdapl] fix fatal bug in triger the evd upcall > > > > Committed in revision 2514. > > On Sun, 29 May 2005, Itamar wrote: > > itamar> Hi James, > itamar> > itamar> This patch fix a fatal bug that exist in current > lastet bits in kdapl (svn rev 2507) > itamar> As you can see we need to triger the upcall when > dapl_evd_dequeue return with good status > itamar> and quit the method when dapl_evd_dequeue return with > non zero status which mean queue is empty. > itamar> In the current bits no kdapltest can run even the > simple quit test. > itamar> > itamar> Please in the future before you commit changes to the > svn run a simple regression. > itamar> Any way with this patch the code is working again. > itamar> > itamar> fix fatal bug in triger the evd upcall > itamar> Signed-off-by: Itamar Rabenstein > itamar> > itamar> Index: dapl_cno_util.c > itamar> > =================================================================== > itamar> --- dapl_cno_util.c (revision 2509) > itamar> +++ dapl_cno_util.c (working copy) > itamar> @@ -115,12 +115,8 @@ > itamar> > itamar> for (;;) { > itamar> status = > dapl_evd_dequeue((DAT_EVD_HANDLE)evd, &event); > itamar> - if (DAT_SUCCESS == status) { > itamar> - dapl_dbg_log(DAPL_DBG_TYPE_ERR, > itamar> - "dapl_evd_dequeue > failed: %x\n", status); > itamar> + if (DAT_SUCCESS != status) > itamar> return; > itamar> - } > itamar> - > itamar> > cno->cno_upcall.upcall_func(cno->cno_upcall.instance_data, > itamar> &event, FALSE); > itamar> } > itamar> -- > itamar> Itamar > itamar> >