[ofa-general] Configuring a 4 KB InfniBand link MTU

[James Lentini]

I have a Mellanox ConnectX HCA connected to a Cisco SFS 7000D switch. 
There are no other devices connected to the switch.

According to their specifications, both the ConnectX chip (MT25418) in 
my HCA and the InfiniScale III chip (MT47396) in my switch support a 4 
KB InfiniBand link MTU.

In practice, I'm getting a 2 KB MTU. Is there some configuration 
setting I need to change to enable a 4 KB MTU? I'd like to see how a 4 
KB MTU affects protocol performance.

My topology looks like this:

 [HCA lid 8, port 1] <------> [SWITCH lid 2, port 1]

smpquery shows a MtuCap of 4 KB for the HCA:

# smpquery portinfo 8 1
# Port info: Lid 8 port 1
Mkey:............................0x0000000000000000
GidPrefix:.......................0xfe80000000000000
Lid:.............................0x0008
SMLid:...........................0x0002
CapMask:.........................0x2500868
                                IsTrapSupported
                                IsAutomaticMigrationSupported
                                IsSLMappingSupported
                                IsSystemImageGUIDsupported
                                IsVendorClassSupported
                                IsCapabilityMaskNoticeSupported
                                IsClientRegistrationSupported
DiagCode:........................0x0000
MkeyLeasePeriod:.................15
LocalPort:.......................1
LinkWidthEnabled:................1X or 4X
LinkWidthSupported:..............undefined (0) (IBA extension)
LinkWidthActive:.................4X
LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps
LinkState:.......................Active
PhysLinkState:...................LinkUp
LinkDownDefState:................Polling
ProtectBits:.....................0
LMC:.............................0
LinkSpeedActive:.................5.0 Gbps
LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps
NeighborMTU:.....................2048
SMSL:............................0
VLCap:...........................VL0-7
InitType:........................0x00
VLHighLimit:.....................4
VLArbHighCap:....................8
VLArbLowCap:.....................8
InitReply:.......................0x00
MtuCap:..........................4096
VLStallCount:....................0
HoqLife:.........................31
OperVLs:.........................VL0-7
PartEnforceInb:..................0
PartEnforceOutb:.................0
FilterRawInb:....................0
FilterRawOutb:...................0
MkeyViolations:..................0
PkeyViolations:..................0
QkeyViolations:..................0
GuidCap:.........................32
ClientReregister:................0
SubnetTimeout:...................8
RespTimeVal:.....................16
LocalPhysErr:....................0
OverrunErr:......................5
MaxCreditHint:...................0
RoundTrip:.......................0

but smpquery reports only a 2 KB MTU capability for the switch

# smpquery portinfo 2 1
# Port info: Lid 2 port 1
Mkey:............................0x0000000000000000
GidPrefix:.......................0x0000000000000000
Lid:.............................0x0000
SMLid:...........................0x0000
CapMask:.........................0x0
DiagCode:........................0x0000
MkeyLeasePeriod:.................0
LocalPort:.......................1
LinkWidthEnabled:................4X
LinkWidthSupported:..............1X or 4X
LinkWidthActive:.................4X
LinkSpeedSupported:..............2.5 Gbps or 5.0 Gbps
LinkState:.......................Active
PhysLinkState:...................LinkUp
LinkDownDefState:................Polling
ProtectBits:.....................0
LMC:.............................0
LinkSpeedActive:.................5.0 Gbps
LinkSpeedEnabled:................2.5 Gbps or 5.0 Gbps
NeighborMTU:.....................2048
SMSL:............................0
VLCap:...........................VL0-7
InitType:........................0x00
VLHighLimit:.....................0
VLArbHighCap:....................8
VLArbLowCap:.....................8
InitReply:.......................0x00
MtuCap:..........................2048
VLStallCount:....................7
HoqLife:.........................18
OperVLs:.........................VL0-7
PartEnforceInb:..................0
PartEnforceOutb:.................0
FilterRawInb:....................0
FilterRawOutb:...................0
MkeyViolations:..................0
PkeyViolations:..................0
QkeyViolations:..................0
GuidCap:.........................0
ClientReregister:................0
SubnetTimeout:...................0
RespTimeVal:.....................0
LocalPhysErr:....................15
OverrunErr:......................15
MaxCreditHint:...................0
RoundTrip:.......................0

and I confirmed that the switch had a MT47396:

# smpquery nodeinfo 2
# Node info: Lid 2
BaseVers:........................1
ClassVers:.......................1
NodeType:........................Switch
NumPorts:........................24
SystemGuid:......................0x0005ad03011df356
Guid:............................0x0005ad00001df356
PortGuid:........................0x0005ad00001df356
PartCap:.........................8
DevId:...........................0xb924
Revision:........................0x000001a1
LocalPort:.......................1
VendorId:........................0x0005ad