[openib-general] [PATCH for-2.6.18] IB/cma: option to limitMTU to 1K

[Hal Rosenstock]

Hi Fab,

On Wed, 2006-09-13 at 13:23, Fabian Tillier wrote:
> Hi Hal,
> 
> On 13 Sep 2006 12:35:58 -0400, Hal Rosenstock <halr at voltaire.com> wrote:
> > Hi Fab,
> >
> > On Wed, 2006-09-13 at 12:39, Fabian Tillier wrote:
> > > On 9/13/06, Michael S. Tsirkin <mst at mellanox.co.il> wrote:
> > > > Quoting r. Hal Rosenstock <halr at voltaire.com>:
> > > > > Subject: Re: [openib-general] [PATCH for-2.6.18] IB/cma: option to limitMTU to 1K
> > > > >
> > > > > On Wed, 2006-09-13 at 11:57, Michael S. Tsirkin wrote:
> > > > > > Tavor systems get better performance with 1K MTU. Since there does
> > > > > > not seem to be any way to find out whether the remote system uses Tavor,
> > > > > > add an option to limit the MTU globally.
> > > > >
> > > > > Can't Tavor be determined locally ?
> > > >
> > > > It can, but we need this for remote tavor as well, anyway.
> > > >
> > > > > And couldn't the remote end negotiate the MTU down (if Tavor) as well ?
> > > >
> > > > The way to do this is would be for SA to select 1K MTU if it detects Tavor on one side
> > > > and if this does not conflict with MTU selector.
> > >
> > > You can't do this because the SA doesn't have a way to tell if a path
> > > query is going to be used for RC or UD, and IPoIB needs paths with 2K
> > > MTU.
> >
> > Are you referring to IPoIB-CM ?
> >
> > The patch appears to be for the SA PR request prior to the CM REQ. I
> > don't think it affects IPoIB SA PR requests.
> 
> I interpreted Michael's comment as suggesting the SA return paths with
> a 1K MTU when it detects that either endpoint is Tavor.  The SA has
> access to this information based on the vendor ID/device ID in the
> node record.

That's the part I missed.

> If I understood Michael's comment properly, this will have the side
> effect that IPoIB won't work since IPoIB requires 2K MTUs.  As far as
> I know, there is no way to specify whether a path is needed for UD vs.
> RC in the path query.

I don't know how either. I don't think it can be done (at least
currently per the standard).

> I like your suggestion to reject with a smaller MTU.  Seems like the
> proper way to handle this, as well as allowing for the retry logic to
> be put in the CMA itself so clients don't have to deal with it.

But a penalty is paid for connect setup (more connection setup latency)
in more round trips until the right MTU is achieved so as most
engineering "solutions" it is a tradeoff with pros and cons.

-- Hal


> - Fab