[ofw] RE: [IPoIB CM] [Patch] ARP REP should be send in UD mode (connectivity issue)

Alex Estrin alex.estrin at qlogic.com
Thu Jan 22 06:43:43 PST 2009


> I believe that reducing connection timeout will partially resolve the
> issue.
> But RFC defines ARP REP to be sent through UD QP, and it solves the
> problem.

All ARPs are sent through UD QP. Please see __send_mgr_filter() and look for:

	case ETH_PROT_TYPE_ARP:
		cl_perf_start( FilterArp );
		status = __send_mgr_filter_arp(
			p_port, p_eth_hdr, p_buf, buf_len, p_desc );
		p_desc->send_dir = SEND_UD_QP;
		cl_perf_stop( &p_port->p_adapter->perf, FilterArp );
		break;

Then, later in __build_send_desc()

		if( p_desc->send_dir == SEND_UD_QP )
		{
			p_desc->send_qp = p_port->ib_mgr.h_qp; // UD QP
...

> So why do we try to debug RC flow ?

Connection request is issued in context of send ARP REPLY. 
While processing of ARP REPLY packet itself get delayed until connection succeed or timed out.

Thanks,
Alex.

> -----Original Message-----
> From: Alex Estrin [mailto:alex.estrin at qlogic.com]
> Sent: Thursday, January 22, 2009 4:14 PM
> To: Alex Naslednikov; ofw at lists.openfabrics.org
> Subject: RE: [IPoIB CM] [Patch] ARP REP should be send in UD mode
> (connectivity issue)
> 
> Hello,
> 
> When Responder generates ARP REP packet and endpoint is in
> IPOIB_CM_DISCONNECTED state, it will move endpoint to transition state
> IPOIB_CM_CONNECT, initiate connect request for that endpoint and queue
> ARP REP packet.
> [Xalex] 
> When connection is established (endpoint in state IPOIB_CM_CONNECTED)
> ARP REP will resume through UD QP.
> TCP applications start sending TCP packets immediately after 
> it received
> ARP REP, so delaying it would make sure all TCP packets go through
> connected QP.
> In your case I think host didn't get connect reply back.
> Not sure why, but it is likely something went wrong with a path record
> either locally or on remote side.
> We need to look deeper in this.
> Also we could probably reduce connect timeout, or connect retries so
> host can reinit endpoint(on connection timeout) and on next 
> ARP REP will
> retry connect again.
> Please see more notes inline.
> 
> Thanks,
> Alex.
> 
> > Hello,Alex,
> > Recently, I found the following problem:
> > 1. Connect 2 machines B2B, run opensm, set static IPoIB adresses,
> > verify ping.
> > 2. Then disconnect a cable for 10-15 seconds, and connect it back 3.
> > Wait for a couple of seconds for opensm to indicate that 
> the links is
> > UP, then try to ping again.
> > 4. The ping now will not work
> >
> > Why this happens:
> > 1. On the sender side, ping (ARP REQ) packet will be generated and
> > sent to the responder size 2. Responder will generate ARP 
> REP packet,
> > but it will be not sent:
> > in recv_mgr_filter_arp, when getting to IPOIB_CM_DISCONNECTED or
> > IPOIB_CM_DISCONNECTED, the code wil return NDIS_STATUS_PENDING, and
> > these ARP REPs will be queued 3. Now, CEP manager will not 
> be able to
> > restore the communication, because of no response for ARP packets :)
> > 4. Sending ARP REP in UD mode will resolve this issue
> >
> > Patch: ARP REP should be send in UD mode Signed-off by: Alexander
> > Naslednikov (xalex at mellanox.co.il)
> > Index: ipoib_port.c
> > ===================================================================
> > --- ipoib_port.c      (revision 3775)
> > +++ ipoib_port.c      (working copy)
> > @@ -4098,70 +4101,12 @@
> >                       return status;
> >               }
> >               ipoib_addr_set_qpn( &p_ib_arp->dst_hw, qpn );
> > -
> > -             if( p_arp->op == ARP_OP_REP &&
> > -                     p_port->p_adapter->params.cm_enabled &&
> > -                     p_desc->p_endpt->cm_flag == IPOIB_CM_FLAG_RC )
> > -             {
> > -                     cm_state_t      cm_state;
> > -                     cm_state =
> > -                             ( cm_state_t
> > )InterlockedCompareExchange( (volatile LONG
> > *)&p_desc->p_endpt->conn.state,
> > -
> > IPOIB_CM_CONNECT, IPOIB_CM_DISCONNECTED );
> > -                     switch( cm_state )
> > -                     {
> > -                     case IPOIB_CM_DISCONNECTED:
> > -                                     IPOIB_PRINT(
> > TRACE_LEVEL_INFORMATION, IPOIB_DBG_INIT,
> > -                                             ("ARP REPLY pending
> > Endpt[%p] QPN %#x MAC %02x:%02x:%02x:%02x:%02x:%02x\n",
> > -                                             p_desc->p_endpt,
> > -                                             cl_ntoh32(
> > ipoib_addr_get_qpn( &p_ib_arp->dst_hw )),
> > -
> > p_desc->p_endpt->mac.addr[0], p_desc->p_endpt->mac.addr[1],
> > -
> > p_desc->p_endpt->mac.addr[2], p_desc->p_endpt->mac.addr[3],
> > -
> > p_desc->p_endpt->mac.addr[4], p_desc->p_endpt->mac.addr[5] ) );
> > -                                     ipoib_addr_set_sid(
> > &p_desc->p_endpt->conn.service_id,
> > -
> > ipoib_addr_get_qpn( &p_ib_arp->dst_hw ) );
> > -
> > -                                     ExFreeToNPagedLookasideList(
> > -
> > &p_port->buf_mgr.send_buf_list, p_desc->p_buf );
> > -                                     cl_qlist_insert_tail(
> > &p_port->send_mgr.pending_list,
> > -
> > IPOIB_LIST_ITEM_FROM_PACKET( p_desc->p_pkt ) );
> > -                                     NdisInterlockedInsertTailList(
> > &p_port->endpt_mgr.pending_conns,
> > -
> > &p_desc->p_endpt->list_item,
> > -
> > &p_port->endpt_mgr.conn_lock );
> > -                                     cl_event_signal(
> > &p_port->endpt_mgr.event );
> 
> Here we add endpoint to the connecting queue and signal cm management
> thread to process.
> Please see __endpt_cm_mgr_thread().
> 
> > -                                     return NDIS_STATUS_PENDING;
> > -                    
> > -                     case IPOIB_CM_CONNECT:
> > -                             /* queue ARP REP packet until 
> connected
> > */
> > -                                     ExFreeToNPagedLookasideList(
> > -                                     
> &p_port->buf_mgr.send_buf_list,
> > p_desc->p_buf );
> > -                                     cl_qlist_insert_tail(
> > &p_port->send_mgr.pending_list,
> > -
> > IPOIB_LIST_ITEM_FROM_PACKET( p_desc->p_pkt ) );
> > -                                     return NDIS_STATUS_PENDING;
> > -                     default:
> > -                             break;
> > -                     }
> > -             }
> >       }
> >       else
> >       {
> >               cl_memclr( &p_ib_arp->dst_hw, 
> sizeof(ipoib_hw_addr_t) );
> >       }
> > -
> > -#if DBG
> > -     if( p_port->p_adapter->params.cm_enabled )
> > -     {
> > -             IPOIB_PRINT( TRACE_LEVEL_INFORMATION, IPOIB_DBG_INIT,
> > -             (" ARP SEND to ENDPT[%p] State: %d flag: %#x, QPN: %#x
> > MAC %02x:%02x:%02x:%02x:%02x:%02x\n",
> > -                     p_desc->p_endpt,
> > -                     endpt_cm_get_state( p_desc->p_endpt ),
> > -                     p_desc->p_endpt->cm_flag,
> > -                     cl_ntoh32( ipoib_addr_get_qpn( 
> &p_ib_arp->dst_hw
> > )),
> > -                     p_desc->p_endpt->mac.addr[0],
> > p_desc->p_endpt->mac.addr[1],
> > -                     p_desc->p_endpt->mac.addr[2],
> > p_desc->p_endpt->mac.addr[3],
> > -                     p_desc->p_endpt->mac.addr[4],
> > p_desc->p_endpt->mac.addr[5] ));
> > -     }
> > -#endif
> > -
> > +    
> >       p_ib_arp->dst_ip = p_arp->dst_ip;
> > 
> >       p_desc->send_wr[0].local_ds[1].vaddr = 
> cl_get_physaddr( p_ib_arp
> );
> >
> 
> 
> 


More information about the ofw mailing list