[ofa-general] Configuring a 4 KB InfniBand link MTU

[Hal Rosenstock]

James,

On Wed, Jan 7, 2009 at 5:11 PM, James Lentini <jlentini at netapp.com> wrote:
>
> I have a Mellanox ConnectX HCA connected to a Cisco SFS 7000D switch.
> There are no other devices connected to the switch.
>
> According to their specifications, both the ConnectX chip (MT25418) in
> my HCA and the InfiniScale III chip (MT47396) in my switch support a 4
> KB InfiniBand link MTU.
>
> In practice, I'm getting a 2 KB MTU. Is there some configuration
> setting I need to change to enable a 4 KB MTU? I'd like to see how a 4
> KB MTU affects protocol performance.
>
> My topology looks like this:
>
>  [HCA lid 8, port 1] <------> [SWITCH lid 2, port 1]
>
> smpquery shows a MtuCap of 4 KB for the HCA:
>
> # smpquery portinfo 8 1
> # Port info: Lid 8 port 1
> Mkey:............................0x0000000000000000
> GidPrefix:.......................0xfe80000000000000
> Lid:.............................0x0008
> SMLid:...........................0x0002
> CapMask:.........................0x2500868
>                                IsTrapSupported
>                                IsAutomaticMigrationSupported
>                                IsSLMappingSupported
>                                IsSystemImageGUIDsupported
>                                IsVendorClassSupported
>                                IsCapabilityMaskNoticeSupported
>                                IsClientRegistrationSupported
> DiagCode:........................0x0000
> MkeyLeasePeriod:.................15
> LocalPort:.......................1
> LinkWidthEnabled:................1X or 4X
> LinkWidthSupported:..............undefined (0) (IBA extension)
> LinkWidthActive:.................4X
> LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps
> LinkState:.......................Active
> PhysLinkState:...................LinkUp
> LinkDownDefState:................Polling
> ProtectBits:.....................0
> LMC:.............................0
> LinkSpeedActive:.................5.0 Gbps
> LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps
> NeighborMTU:.....................2048
> SMSL:............................0
> VLCap:...........................VL0-7
> InitType:........................0x00
> VLHighLimit:.....................4
> VLArbHighCap:....................8
> VLArbLowCap:.....................8
> InitReply:.......................0x00
> MtuCap:..........................4096
> VLStallCount:....................0
> HoqLife:.........................31
> OperVLs:.........................VL0-7
> PartEnforceInb:..................0
> PartEnforceOutb:.................0
> FilterRawInb:....................0
> FilterRawOutb:...................0
> MkeyViolations:..................0
> PkeyViolations:..................0
> QkeyViolations:..................0
> GuidCap:.........................32
> ClientReregister:................0
> SubnetTimeout:...................8
> RespTimeVal:.....................16
> LocalPhysErr:....................0
> OverrunErr:......................5
> MaxCreditHint:...................0
> RoundTrip:.......................0
>
> but smpquery reports only a 2 KB MTU capability for the switch

If the switch ports support a 4K MTU cap, it needs to be advertised
(via the PortInfo response). That's what needs fixing if possible
rather than some SM policy. Without an MTUCap of 4K on both sides of
the link, an SM cannot set NeighborMTU to 4K.

-- Hal

> # smpquery portinfo 2 1
> # Port info: Lid 2 port 1
> Mkey:............................0x0000000000000000
> GidPrefix:.......................0x0000000000000000
> Lid:.............................0x0000
> SMLid:...........................0x0000
> CapMask:.........................0x0
> DiagCode:........................0x0000
> MkeyLeasePeriod:.................0
> LocalPort:.......................1
> LinkWidthEnabled:................4X
> LinkWidthSupported:..............1X or 4X
> LinkWidthActive:.................4X
> LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps
> LinkState:.......................Active
> PhysLinkState:...................LinkUp
> LinkDownDefState:................Polling
> ProtectBits:.....................0
> LMC:.............................0
> LinkSpeedActive:.................5.0 Gbps
> LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps
> NeighborMTU:.....................2048
> SMSL:............................0
> VLCap:...........................VL0-7
> InitType:........................0x00
> VLHighLimit:.....................0
> VLArbHighCap:....................8
> VLArbLowCap:.....................8
> InitReply:.......................0x00
> MtuCap:..........................2048
> VLStallCount:....................7
> HoqLife:.........................18
> OperVLs:.........................VL0-7
> PartEnforceInb:..................0
> PartEnforceOutb:.................0
> FilterRawInb:....................0
> FilterRawOutb:...................0
> MkeyViolations:..................0
> PkeyViolations:..................0
> QkeyViolations:..................0
> GuidCap:.........................0
> ClientReregister:................0
> SubnetTimeout:...................0
> RespTimeVal:.....................0
> LocalPhysErr:....................15
> OverrunErr:......................15
> MaxCreditHint:...................0
> RoundTrip:.......................0
>
> and I confirmed that the switch had a MT47396:
>
> # smpquery nodeinfo 2
> # Node info: Lid 2
> BaseVers:........................1
> ClassVers:.......................1
> NodeType:........................Switch
> NumPorts:........................24
> SystemGuid:......................0x0005ad03011df356
> Guid:............................0x0005ad00001df356
> PortGuid:........................0x0005ad00001df356
> PartCap:.........................8
> DevId:...........................0xb924
> Revision:........................0x000001a1
> LocalPort:.......................1
> VendorId:........................0x0005ad
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit http://openib.org/mailman/listinfo/openib-general
>