[ofw] [PATCH] IBAL: Only create CEPs for REQs that matched to a listen

[Alex Estrin]

Hi Fab,

Do you think that scenario could cause REQ sender to timeout, i.e. listener would destroy CEP instead of sending reject?
Thanks,
Alex.

> -----Original Message-----
> From: ofw-bounces at lists.openfabrics.org [mailto:ofw-bounces at lists.openfabrics.org] On Behalf Of Fab
> Tillier
> Sent: Wednesday, February 25, 2009 6:18 PM
> To: ofw at lists.openfabrics.org
> Subject: [ofw] [PATCH] IBAL: Only create CEPs for REQs that matched to a listen
> 
> Currently, the CM creates a CEP for an incoming REQ before matching to either a listen or a Peer-to-
> Peer connection request.  As Peer-to-Peer is not supported, this means that CEPs are created before
> matching the incoming REQ to a listen.  In the case where there is no listen, this will result in CEPs
> being created, and then queued for destruction to the timewait timer callback.  Because the CEPs have
> a timewait time of zero, the timer callback runs as fast as REQs are received, severely affecting
> system responsiveness.
>
> This patch moves the CEP creation after matching against listens, avoiding creation altogether if
> there is no listening endpoint.
> 
> Signed-off-by: Fab Tillier <ftillier at microsoft.com>
> 
> Index: core/al/kernel/al_cm_cep.c
> ===================================================================
> --- core/al/kernel/al_cm_cep.c  (revision 1978)
> +++ core/al/kernel/al_cm_cep.c  (working copy)
> @@ -1108,29 +1108,32 @@ __req_handler(
>         }
> 
>         /*
> -        * Allocate a new CEP for the new request.  This will
> -        * prevent multiple identical REQs from queueing up for processing.
> -        */
> -       p_new_cep = __create_cep();
> -       if( !p_new_cep )
> -       {
> -               /* Reject the request for insufficient resources. */
> -               reason = IB_REJ_INSUF_RESOURCES;
> -               AL_PRINT_EXIT( TRACE_LEVEL_ERROR, AL_DBG_ERROR,
> -                       ("al_create_cep failed\nREJ sent for insufficient resources.\n") );
> -               goto reject;
> -       }
> -
> -       __save_wire_req( p_new_cep, p_req );
> -
> -       /*
>          * Match against listens using SID and compare data, also provide the receiving
>          * MAD service's port GUID so we can properly filter.
>          */
>         p_cep = __lookup_listen( p_req->sid, p_port_cep->port_guid, p_req->pdata );
>         if( p_cep )
>         {
> +               /*
> +                * Allocate a new CEP for the new request.  This will
> +                * prevent multiple identical REQs from queueing up for processing.
> +                */
> +               p_new_cep = __create_cep();
> +               if( !p_new_cep )
> +               {
> +                       /* Reject the request for insufficient resources. */
> +                       reason = IB_REJ_INSUF_RESOURCES;
> +                       AL_PRINT_EXIT( TRACE_LEVEL_ERROR, AL_DBG_ERROR,
> +                               ("al_create_cep failed\nREJ sent for insufficient resources.\n") );
> +                       goto reject;
> +               }
> +
> +               __save_wire_req( p_new_cep, p_req );
> +
>                 __bind_cep( p_new_cep, p_cep->p_cid->h_al, p_cep->pfn_cb, NULL );
> +               AL_PRINT( TRACE_LEVEL_VERBOSE, AL_DBG_CM,
> +                       ("Created CEP with CID = %d, h_al %p, remote = %d\n",
> +                       p_new_cep->cid, p_cep->p_cid->h_al, p_new_cep->remote_comm_id) );
> 
>                 /* Add the new CEP to the map so that repeated REQs match up. */
>                 p_stale_cep = __insert_cep( p_new_cep );
> @@ -1197,7 +1200,7 @@ __req_handler(
>                         /* No match found.  Reject. */
>                         reason = IB_REJ_INVALID_SID;
>                         AL_PRINT( TRACE_LEVEL_INFORMATION, AL_DBG_CM, ("REQ received but no match
> found.\n") );
> -                       goto cleanup;
> +                       goto reject;
>                 }
>         }
> 
> @@ -1213,7 +1216,6 @@ __req_handler(
>  unbind:
>         __unbind_cep( p_new_cep );
> 
> -cleanup:
>         /*
>          * Move the CEP in the idle state so that we don't send a reject
>          * for it when cleaning up.  Also clear the RQPN and RCID so that