<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<HTML xmlns="http://www.w3.org/TR/REC-html40" xmlns:w =
"urn:schemas-microsoft-com:office:word" xmlns:o =
"urn:schemas-microsoft-com:office:office" xmlns:v =
"urn:schemas-microsoft-com:vml"><HEAD><!--[if !mso]>
<META http-equiv=Content-Type content="text/html; charset=us-ascii">
<STYLE>v\:* {
BEHAVIOR: url(#default#VML)
}
o\:* {
BEHAVIOR: url(#default#VML)
}
w\:* {
BEHAVIOR: url(#default#VML)
}
.shape {
BEHAVIOR: url(#default#VML)
}
</STYLE>
<![endif]-->
<STYLE>@font-face {
font-family: Tahoma;
}
@page Section1 {size: 8.5in 11.0in; margin: 1.0in 1.25in 1.0in 1.25in; }
P.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"
}
LI.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"
}
DIV.MsoNormal {
FONT-SIZE: 12pt; MARGIN: 0in 0in 0pt; FONT-FAMILY: "Times New Roman"
}
A:link {
COLOR: blue; TEXT-DECORATION: underline
}
SPAN.MsoHyperlink {
COLOR: blue; TEXT-DECORATION: underline
}
A:visited {
COLOR: purple; TEXT-DECORATION: underline
}
SPAN.MsoHyperlinkFollowed {
COLOR: purple; TEXT-DECORATION: underline
}
SPAN.EmailStyle17 {
COLOR: navy; FONT-FAMILY: Arial; mso-style-type: personal-reply
}
DIV.Section1 {
page: Section1
}
</STYLE>
<META content="MSHTML 6.00.2900.2873" name=GENERATOR></HEAD>
<BODY lang=EN-US vLink=purple link=blue>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN class=833220519-03062006>I'll
try this fix, but I think that it will not help.</SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=833220519-03062006></SPAN></FONT> </DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN class=833220519-03062006>The
main reason for this is that according to the specs (if I remember correctly)
the RNR timeout can be set to somewhere between x and 7 (7 is hard coded in the
spec) x.</SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN class=833220519-03062006>What
we are looking right now is the correct value of x. Since the receives are
posted by a user mode thread from the switch, we need to try and understand the
amount of time that a user mode thread can get delayed. The correct answer is
probably that on a very busy system this time is almost for ever. (at least
there is no way to give a maximum to this time). Assuming that the system is
never that busy, we should give it at least the needed time for a thread context
switch which is 60ms. (On the worst case we might want to give other times, such
as 2,3, or n times this period.) But even being very conservative and assuming
that we wish to allow only twice this time this brings us to 120ms at
maximum. 120/7 = 17, and there for the RNR retry time should be 17ms.
1000/17 =~60 and this is an upper limit to our connection rate, which is not
that nice. (actually it is somewhat bigger as I saw that only one connection out
of 5 reaches to RNR on a non busy system).</SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=833220519-03062006></SPAN></FONT> </DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN class=833220519-03062006>That
said, I'll give your code a chance and see how this works.</SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=833220519-03062006></SPAN></FONT> </DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN class=833220519-03062006>As for
the patch that I have sent that tries to solve the same problem (delay CM until
there is a recv posted). Considerations there are similar but we have more
freedom in choosing the constants so I believe that it should work
better.</SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=833220519-03062006></SPAN></FONT> </DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN class=833220519-03062006>As for
the latencies that will be introduced by your patch: It seems that the current
code already takes the lock at the beginning of the recv, so another check
if this is the first buffer shouldn't take more than a single if statement
which is less than 10ns to my understanding.</SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=833220519-03062006></SPAN></FONT> </DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN class=833220519-03062006>In any
case I'll test your fix and see how things are working.</SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=833220519-03062006></SPAN></FONT> </DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=833220519-03062006>Thanks</SPAN></FONT></DIV>
<DIV><FONT face=Arial color=#0000ff size=2><SPAN
class=833220519-03062006>Tzachi </SPAN></FONT></DIV><FONT face=Arial
color=#0000ff size=2></FONT><BR>
<BLOCKQUOTE dir=ltr
style="PADDING-LEFT: 5px; MARGIN-LEFT: 5px; BORDER-LEFT: #0000ff 2px solid; MARGIN-RIGHT: 0px">
<DIV class=OutlookMessageHeader lang=en-us dir=ltr align=left>
<HR tabIndex=-1>
<FONT face=Tahoma size=2><B>From:</B> Tillier, Fabian
[mailto:ftillier@silverstorm.com] <BR><B>Sent:</B> Thursday, June 01, 2006
8:57 PM<BR><B>To:</B> Tzachi Dar; Leonid Keller<BR><B>Cc:</B> Erez
Cohen<BR><B>Subject:</B> RE: Connection rate of WSD<BR></FONT><BR></DIV>
<DIV></DIV>
<DIV class=Section1>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial">Hi
Tzachi,<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"><o:p> </o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial">I think I know why
the RNR delay is so large – the retry count is set, but the RNR timeout
doesn’t get set since the HCA driver expects RNR timer to be specified as an
option in the RTR transition. This is actually a required field for RTR,
so the option field is not set, resulting in a value of 0 being used, which
means 655 milliseconds.<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"><o:p> </o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial">Here’s a patch that
sets the timer to a more aggressive value, which should hopefully help, as
well as fixing the two HCA drivers to set the RNR timer field in the RTR
transition. If it does, this is a far simpler fix. The RNR
solution, while a bit crude, eliminates a lot of synchronization overhead that
would be required to buffer the first receive and report it properly.
This synchronization overhead would also impact latencies, as it would be part
of all receive and completion processing for the life of the connection, an
overhead required only because of the first receive
buffer.<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"><o:p> </o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial">Let me know how it
works.<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"><o:p> </o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial">Thanks,<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"><o:p> </o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial">-
Fab<o:p></o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"><o:p> </o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"><o:p> </o:p></SPAN></FONT></P>
<P class=MsoNormal><FONT face=Arial color=navy size=2><SPAN
style="FONT-SIZE: 10pt; COLOR: navy; FONT-FAMILY: Arial"><o:p> </o:p></SPAN></FONT></P>
<DIV>
<DIV class=MsoNormal style="TEXT-ALIGN: center" align=center><FONT
face="Times New Roman" size=3><SPAN style="FONT-SIZE: 12pt">
<HR tabIndex=-1 align=center width="100%" SIZE=2>
</SPAN></FONT></DIV>
<P class=MsoNormal><B><FONT face=Tahoma size=2><SPAN
style="FONT-WEIGHT: bold; FONT-SIZE: 10pt; FONT-FAMILY: Tahoma">From:</SPAN></FONT></B><FONT
face=Tahoma size=2><SPAN style="FONT-SIZE: 10pt; FONT-FAMILY: Tahoma"> Tzachi
Dar [mailto:tzachid@mellanox.co.il] <BR><B><SPAN
style="FONT-WEIGHT: bold">Sent:</SPAN></B> Tuesday, May 30, 2006 9:11
AM<BR><B><SPAN style="FONT-WEIGHT: bold">To:</SPAN></B> Tillier, Fabian;
openib-windows@openib.org<BR><B><SPAN
style="FONT-WEIGHT: bold">Subject:</SPAN></B> Connection rate of
WSD</SPAN></FONT><o:p></o:p></P></DIV>
<P class=MsoNormal><FONT face="Times New Roman" size=3><SPAN
style="FONT-SIZE: 12pt"><o:p> </o:p></SPAN></FONT></P>
<DIV>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Hi
Fab,</SPAN></FONT><o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal><FONT face="Times New Roman" size=3><SPAN
style="FONT-SIZE: 12pt"> <o:p></o:p></SPAN></FONT></P></DIV>
<DIV>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">While doing tests of connecting
and disconnecting to WSD I have found out that the connection rate is
low.</SPAN></FONT><o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">When I say low, I mean that some
times it took 2 seconds to create a connection. After checking for the reason
of the problem, I have found out that the main reason is that RNR nack signal
is sent.</SPAN></FONT><o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal><FONT face="Times New Roman" size=3><SPAN
style="FONT-SIZE: 12pt"> <o:p></o:p></SPAN></FONT></P></DIV>
<DIV>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">It seems that the current flow is
that one side sends CM_REQ. The received notifies the switch on a new accepted
socket. After that when the switch calls accept, a CM_REP message is sent. No
recves are posted, and they will only be posted when the switch wants which
might be too late. Once the CM_REP was sent, the remote side sends RTU and
might start to send data.</SPAN></FONT><o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal><FONT face="Times New Roman" size=3><SPAN
style="FONT-SIZE: 12pt"> <o:p></o:p></SPAN></FONT></P></DIV>
<DIV>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">To my understanding, the best way
to solve this problem is to dely the sending of the CM_REP until the first
recv is posted. This makes sure that once data arrives, things work
fine.</SPAN></FONT><o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal><FONT face="Times New Roman" size=3><SPAN
style="FONT-SIZE: 12pt"> <o:p></o:p></SPAN></FONT></P></DIV>
<DIV>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">I have created an experimental
patch that implements this idea and it seems that things are working well with
it.</SPAN></FONT><o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal><FONT face="Times New Roman" size=3><SPAN
style="FONT-SIZE: 12pt"> <o:p></o:p></SPAN></FONT></P></DIV>
<DIV>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">What is your
opinion?</SPAN></FONT><o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal><FONT face="Times New Roman" size=3><SPAN
style="FONT-SIZE: 12pt"> <o:p></o:p></SPAN></FONT></P></DIV>
<DIV>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Do you want me to prepare a "real"
patch for this?</SPAN></FONT><o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal><FONT face="Times New Roman" size=3><SPAN
style="FONT-SIZE: 12pt"> <o:p></o:p></SPAN></FONT></P></DIV>
<DIV>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Thanks</SPAN></FONT><o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Tzachi</SPAN></FONT><o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal><FONT face="Times New Roman" size=3><SPAN
style="FONT-SIZE: 12pt"> <o:p></o:p></SPAN></FONT></P></DIV>
<DIV>
<P class=MsoNormal><FONT face=Arial size=2><SPAN
style="FONT-SIZE: 10pt; FONT-FAMILY: Arial">Index:
ib_cm.c<BR>===================================================================<BR>---
ib_cm.c (revision 1372)<BR>+++ ib_cm.c (working copy)<BR>@@ -92,6
+92,7 @@<BR> IBSP_TRACE2(
IBSP_DBG_NEV,<BR> ("Signaling eventHandle %p at time
%I64d.\n",<BR> h_event, cl_get_time_stamp() )
);<BR>+ IBSP_ERROR(("Setting the
event\n"));<BR> SetEvent( h_event
);<BR> }<BR> <BR>@@ -208,6 +209,7
@@<BR> ib_api_status_t status;<BR> <BR> IBSP_ENTER(
IBSP_DBG_CM );<BR>+ IBSP_ERROR(("cm_rep_callback
called\n"));<BR> <BR> memset( &cm_rtu, 0, sizeof(cm_rtu)
);<BR> <BR>@@ -290,6 +292,8 @@<BR> <BR> IBSP_ENTER(
IBSP_DBG_CM );<BR> <BR>+ IBSP_ERROR(("cm_rtu_callback
called\n"));<BR>+<BR> cl_spinlock_acquire(
&socket_info->mutex );<BR> <BR> if(
socket_info->socket_state == IBSP_DUPLICATING_REMOTE )<BR>@@ -877,9 +881,11
@@<BR> IN struct
ibsp_socket_info *socket_info,<BR> IN ib_cm_req_rec_t *cm_req_received
)<BR> {<BR>- ib_cm_rep_t cm_rep;<BR>- ib_api_status_t
status;<BR>+// ib_cm_rep_t cm_rep;<BR>+// ib_api_status_t
status;<BR> <BR>+ UNREFERENCED_PARAMETER(cm_req_received);<BR>+<BR> IBSP_ENTER(
IBSP_DBG_CM );<BR> <BR> /* Insert into the connection map.
*/<BR>@@ -888,7 +894,60 @@<BR> IBSP_EXIT( IBSP_DBG_CM
);<BR> return WSAEADDRINUSE;<BR> }<BR>+#if
0<BR>+ memset( &cm_rep, 0, sizeof(cm_rep)
);<BR> <BR>+ cm_rep.qp_type =
IB_QPT_RELIABLE_CONN;<BR>+ cm_rep.h_qp =
socket_info->qp;<BR>+ cm_rep.access_ctrl = IB_AC_RDMA_READ |
IB_AC_RDMA_WRITE | IB_AC_LOCAL_WRITE;<BR>+#if 0<BR>+ // Bug in
TAVOR<BR>+ cm_rep.sq_depth =
QP_ATTRIB_SQ_DEPTH;<BR>+ cm_rep.rq_depth =
QP_ATTRIB_RQ_DEPTH;<BR>+#endif<BR>+ cm_rep.init_depth =
QP_ATTRIB_INITIATOR_DEPTH;<BR>+ cm_rep.target_ack_delay =
10;<BR>+ cm_rep.failover_accepted =
IB_FAILOVER_ACCEPT_UNSUPPORTED;<BR>+ cm_rep.flow_ctrl =
cm_req_received->flow_ctrl;<BR>+ cm_rep.rnr_nak_timeout =
QP_ATTRIB_RNR_NAK_TIMEOUT;<BR>+ cm_rep.rnr_retry_cnt =
cm_req_received->rnr_retry_cnt;<BR>+ cm_rep.pfn_cm_mra_cb =
cm_mra_callback;<BR>+ cm_rep.pfn_cm_rej_cb =
cm_rej_callback;<BR>+ cm_rep.pfn_cm_rtu_cb =
cm_rtu_callback;<BR>+ cm_rep.pfn_cm_lap_cb =
cm_lap_callback;<BR>+ cm_rep.pfn_cm_dreq_cb =
cm_dreq_callback;<BR>+<BR>+ fzprint(("%s():%d:0x%x:0x%x: flow_ctrl=%d
rnr_retry_cnt=%d\n", __FUNCTION__,<BR>+ __LINE__,
GetCurrentProcessId(),<BR>+ GetCurrentThreadId(),
cm_rep.flow_ctrl, cm_rep.rnr_retry_cnt));<BR>+<BR>+ status = ib_cm_rep(
cm_req_received->h_cm_req, &cm_rep );<BR>+ if( status !=
IB_SUCCESS )<BR>+ {<BR>+ /* Remove from connection map.
*/<BR>+ ibsp_conn_remove( socket_info
);<BR>+<BR>+ IBSP_ERROR_EXIT(<BR>+ ("ib_cm_rep
failed (0x%d) at time %I64d\n",<BR>+ ib_get_err_str( status
), cl_get_time_stamp()) );<BR>+ return
WSAEACCES;<BR>+ }<BR>+#endif<BR>+ IBSP_EXIT( IBSP_DBG_CM
);<BR>+ return
0;<BR>+}<BR>+<BR>+<BR>+int<BR>+ib_accept1(<BR>+ IN struct
ibsp_socket_info *socket_info,<BR>+ IN ib_cm_req_rec_t *cm_req_received
)<BR>+{<BR>+ ib_cm_rep_t cm_rep;<BR>+ ib_api_status_t
status;<BR>+<BR>+ IBSP_ENTER( IBSP_DBG_CM );<BR>+<BR> memset(
&cm_rep, 0, sizeof(cm_rep) );<BR> <BR> cm_rep.qp_type =
IB_QPT_RELIABLE_CONN;<BR>Index:
ibspdll.c<BR>===================================================================<BR>---
ibspdll.c (revision 1372)<BR>+++ ibspdll.c (working copy)<BR>@@
-325,6 +325,8 @@<BR> /* Update the state of the socket context
*/<BR> IBSP_CHANGE_SOCKET_STATE( new_socket_info, IBSP_CONNECTED
);<BR> <BR>+ new_socket_info->cm_req_received =
p_incoming->cm_req_received;<BR>+<BR> *lpErrno = ib_accept(
new_socket_info, &p_incoming->cm_req_received );<BR> if(
*lpErrno )<BR> {<BR>@@ -346,6 +348,8
@@<BR> deref_socket_info( new_socket_info
);<BR> return
INVALID_SOCKET;<BR> }<BR>+ CL_ASSERT(new_socket_info->cm_rep_waiting
== FALSE);<BR>+ new_socket_info->cm_rep_waiting =
TRUE;<BR> <BR> cl_spinlock_acquire(
&g_ibsp.socket_info_mutex );<BR> cl_qlist_insert_tail(<BR>@@
-369,6 +373,7 @@<BR> * of the user supplied callback
so you can trigger that once your<BR> * substituted
function is triggered).<BR> */<BR>+<BR> static SOCKET
WSPAPI<BR> IBSPAccept(<BR> IN SOCKET s,<BR>@@
-724,6 +729,8 @@<BR> <BR> IBSP_ENTER( IBSP_DBG_CONN
);<BR> <BR>+ IBSP_ERROR(("IBSPConnect
called\n"));<BR>+<BR> UNUSED_PARAM( lpCalleeData
);<BR> UNUSED_PARAM( lpSQOS );<BR> UNUSED_PARAM( lpGQOS
);<BR>@@ -855,6 +862,7 @@<BR> done:<BR> cl_spinlock_release(
&socket_info->mutex );<BR> IBSP_EXIT( IBSP_DBG_CONN
);<BR>+ IBSP_ERROR(("IBSPConnect returning\n"));<BR> return
SOCKET_ERROR;<BR> }<BR> <BR>@@ -1455,6 +1463,12 @@<BR>
* handle and then make the receive call. If called with overlap,
post the operation<BR> * to our IOCP or completion
routine.<BR> */<BR>+<BR>+int<BR>+ib_accept1(<BR>+ IN struct
ibsp_socket_info *socket_info,<BR>+ IN ib_cm_req_rec_t *cm_req_received
);<BR>+<BR> static int
WSPAPI<BR> IBSPRecv(<BR> SOCKET s,<BR>@@
-1496,6 +1510,12
@@<BR> }<BR> <BR> cl_spinlock_acquire(
&socket_info->mutex );<BR>+<BR>+ if
(socket_info->cm_rep_waiting == TRUE)
{<BR>+ ib_accept1(socket_info,&socket_info->cm_req_received);<BR>+ socket_info->cm_rep_waiting
= FALSE;<BR>+ }<BR>+<BR> switch( socket_info->socket_state
)<BR> {<BR> case IBSP_CONNECTED:<BR>Index:
ibspstruct.h<BR>===================================================================<BR>---
ibspstruct.h (revision 1372)<BR>+++ ibspstruct.h (working
copy)<BR>@@ -326,6 +326,9
@@<BR> long recv_log_idx;<BR> long send_log_idx;<BR> #endif<BR>+<BR>+ boolean_t cm_rep_waiting;<BR>+ ib_cm_req_rec_t cm_req_received
;<BR> };<BR> <BR> </SPAN></FONT><o:p></o:p></P></DIV>
<DIV>
<P class=MsoNormal><FONT face="Times New Roman" size=3><SPAN
style="FONT-SIZE: 12pt"> <o:p></o:p></SPAN></FONT></P></DIV>
<DIV>
<P class=MsoNormal><FONT face="Times New Roman" size=3><SPAN
style="FONT-SIZE: 12pt"> <o:p></o:p></SPAN></FONT></P></DIV></DIV></BLOCKQUOTE></BODY></HTML>