[Users] kernel: scsi host8: ib_srp: SRP_LOGIN_REJ: requested max_it_iu_len too large
Tom Robinson
tom.robinson at motec.com.au
Thu Jan 16 14:16:14 PST 2020
On Thu, 16 Jan 2020 at 16:44, Bart Van Assche <bvanassche at acm.org> wrote:
>
> On 2020-01-15 18:04, Tom Robinson wrote:
> > I hope this is the right list to ask this question. Please let me know
> > if otherwise.
> >
> > We are trying to configure the following cards on Arch Linux. This is
> > part of a reconfiguration of our infrastructure and change of OS from
> > CentOS to Arch. These cards were working well under CentOS 7 but I can't
> > seem to get them to work under Arch Linux.
> >
> > 05:00.0 InfiniBand: Mellanox Technologies MT25408A0-FCC-QI ConnectX,
> > Dual Port 40Gb/s InfiniBand / 10GigE Adapter IC with PCIe 2.0 x8 5.0GT/s
> > In... (rev b0)
> > 06:00.0 InfiniBand: Mellanox Technologies MT25408A0-FCC-QI ConnectX,
> > Dual Port 40Gb/s InfiniBand / 10GigE Adapter IC with PCIe 2.0 x8 5.0GT/s
> > In... (rev b0)
> >
> > Currently I am getting the following error on startup of the srp_daemon:
> >
> > kernel: scsi host8: ib_srp: SRP_LOGIN_REJ: requested max_it_iu_len too large
> >
> > Has anyone seen this before? I have done a lot of searching on the
> > internet but as yet have not come across a solution. Can someone please
> > help and shed some light on this?
>
> Which kernel version is running on the Arch Linux system? How have the
> SRP daemon and the ib_srp driver been configured on the initiator
> system? (cat /etc/srp_daemon.conf; cat /etc/modprobe.d/ib_srp.conf)? Is
> the target system a vendor-supplied SRP system or a self-configure Linux
> system? In the latter case, which kernel version has been installed on
> the SRP target system (address fe80:0000:0000:0000:0002:c903:002c:d311)?
> If it is Linux, is the target system using the in-tree (LIO) ib_srpt
> driver or the SCST ib_srpt driver?
>
> Thanks,
>
> Bart.
Hi Bart,
Thanks for looking at this.
On the Arch system:
# uname -a
Linux daytona.motec.com.au 5.4.11-arch1-1 #1 SMP PREEMPT Sun, 12 Jan
2020 12:15:27 +0000 x86_64 GNU/Linux
The /etc/srp_daemon.conf holds only comments. This is consistent with
the original CentOS system.
Originally I had no configuration for the ib_srp module. I adjusted
cmd_sg_entries=128 after reading this:
https://www.spinics.net/lists/linux-rdma/msg29124.html. That change
didn't help resolve this issue. The module options are currently set
as follows:
# cat /etc/modprobe.d/ib_srp.conf
options ib_srp cmd_sg_entries=128
The target system is OmniOS running COMSTAR. This configuration has
been working with CentOS 7 for a few years.
As an aside, we have originally set up two CentOS 7 systems with
exactly the same hardware/software configuration affording us the
flexibility to do maintenance and upgrades while keeping our
production systems up and running. Currently one system is still
configured with CentOS 7 and is up and running with iSCSI boot, SRP
and Multipathing. The Arch system with the issues is our 'upgrade
path'. Our plan is to replace the CentOS systems with Arch.
Hopefully this ASCII art renders correctly for you:
----------- -----------
| CentOS 7 | | CentOS 7 | (<-replaced with Arch)
------------ ------------
| | | |
| -------\ /------ |
| X |
| -------/ \------ |
| | | |
------------- -------------
| IB Switch | | IB Switch |
------------- -------------
| |
---------- --------
| |
| |
-----------
| OminiOS |
-----------
| |
-----------
| Storage |
-----------
On OmniOs target system:
# cfgadm | egrep '(hca|ib)'
hca:2C903002CD310 IB-HCA connected configured ok
ib IB-Fabric connected configured ok
# dladm show-phys | grep ib
ibp0 Infiniband up 32000 unknown ibp0
ibp1 Infiniband up 32000 unknown ibp1
# dladm show-phys | grep ib
ibp0 Infiniband up 32000 unknown ibp0
ibp1 Infiniband up 32000 unknown ibp1
# dladm show-link | grep ib
ibp0 phys 65520 up -- --
ibp1 phys 65520 up -- --
pFFFF.ibp0 part 65520 up -- ibp0
pFFFF.ibp1 part 65520 up -- ibp1
# ipadm show-addr | grep ipmp
ipmp0/? static ok 10.0.2.2/24
Also including ibnetdiscover output from the Arch Linux system for reference:
# ibnetdiscover
#
# Topology file: generated on Fri Jan 17 07:58:01 2020
#
# Initiated from node f452140300cea42c port f452140300cea42e
vendid=0x2c9
devid=0xbd36
sysimgguid=0x2c9020048df63
switchguid=0x2c9020048df60(2c9020048df60)
Switch 8 "S-0002c9020048df60" # "Infiniscale-IV Mellanox Technologies"
base port 0 lid 5 lmc 0
[1] "H-0002c903002cd2f4"[2](2c903002cd2f6) # "atom mlx4_0" lid 6 4xQDR
[3] "H-0002c903002cd310"[2](2c903002cd312) # "MT25408 ConnectX
Mellanox Technologies" lid 8 4xQDR
[5] "H-0002c903002cd2f8"[2](2c903002cd2fa) # "daytona ibp5s0" lid 12 4xQDR
[6] "S-0002c9020048df58"[6] # "Infiniscale-IV Mellanox Technologies" lid 2 4xQDR
[7] "H-f452140300cea310"[2](f452140300cea312) # "lemans mlx4_0" lid 10 4xQDR
[8] "S-0002c9020048df58"[8] # "Infiniscale-IV Mellanox Technologies" lid 2 4xQDR
vendid=0x2c9
devid=0xbd36
sysimgguid=0x2c9020048df5b
switchguid=0x2c9020048df58(2c9020048df58)
Switch 8 "S-0002c9020048df58" # "Infiniscale-IV Mellanox Technologies"
base port 0 lid 2 lmc 0
[1] "H-0002c903002cd2f4"[1](2c903002cd2f5) # "atom mlx4_0" lid 1 4xQDR
[3] "H-0002c903002cd310"[1](2c903002cd311) # "MT25408 ConnectX
Mellanox Technologies" lid 3 4xQDR
[5] "H-f452140300cea42c"[2](f452140300cea42e) # "daytona ibp6s0" lid 11 4xQDR
[6] "S-0002c9020048df60"[6] # "Infiniscale-IV Mellanox Technologies" lid 5 4xQDR
[7] "H-0002c903002cd30c"[2](2c903002cd30e) # "lemans mlx4_1" lid 9 4xQDR
[8] "S-0002c9020048df60"[8] # "Infiniscale-IV Mellanox Technologies" lid 5 4xQDR
vendid=0x2c9
devid=0x673c
sysimgguid=0xf452140300cea313
caguid=0xf452140300cea310
Ca 2 "H-f452140300cea310" # "lemans mlx4_0"
[2](f452140300cea312) "S-0002c9020048df60"[7] # lid 10 lmc 0
"Infiniscale-IV Mellanox Technologies" lid 5 4xQDR
vendid=0x2c9
devid=0x673c
sysimgguid=0x2c903002cd2fb
caguid=0x2c903002cd2f8
Ca 2 "H-0002c903002cd2f8" # "daytona ibp5s0"
[2](2c903002cd2fa) "S-0002c9020048df60"[5] # lid 12 lmc 0
"Infiniscale-IV Mellanox Technologies" lid 5 4xQDR
vendid=0x2c9
devid=0x673c
sysimgguid=0x2c903002cd30f
caguid=0x2c903002cd30c
Ca 2 "H-0002c903002cd30c" # "lemans mlx4_1"
[2](2c903002cd30e) "S-0002c9020048df58"[7] # lid 9 lmc 0
"Infiniscale-IV Mellanox Technologies" lid 2 4xQDR
vendid=0x2c9
devid=0x673c
sysimgguid=0x2c903002cd313
caguid=0x2c903002cd310
Ca 2 "H-0002c903002cd310" # "MT25408 ConnectX Mellanox Technologies"
[1](2c903002cd311) "S-0002c9020048df58"[3] # lid 3 lmc 0
"Infiniscale-IV Mellanox Technologies" lid 2 4xQDR
[2](2c903002cd312) "S-0002c9020048df60"[3] # lid 8 lmc 0
"Infiniscale-IV Mellanox Technologies" lid 5 4xQDR
vendid=0x2c9
devid=0x673c
sysimgguid=0x2c903002cd2f7
caguid=0x2c903002cd2f4
Ca 2 "H-0002c903002cd2f4" # "atom mlx4_0"
[1](2c903002cd2f5) "S-0002c9020048df58"[1] # lid 1 lmc 0
"Infiniscale-IV Mellanox Technologies" lid 2 4xQDR
[2](2c903002cd2f6) "S-0002c9020048df60"[1] # lid 6 lmc 0
"Infiniscale-IV Mellanox Technologies" lid 5 4xQDR
vendid=0x2c9
devid=0x673c
sysimgguid=0xf452140300cea42f
caguid=0xf452140300cea42c
Ca 2 "H-f452140300cea42c" # "daytona ibp6s0"
[2](f452140300cea42e) "S-0002c9020048df58"[5] # lid 11 lmc 0
"Infiniscale-IV Mellanox Technologies" lid 2 4xQDR
Kind regards,
Tom Robinson
IT Manager/System Administrator
MoTeC Pty Ltd
MoTeC Research Centre
121 Merrindale Drive
Croydon South 3136
Victoria Australia
T: 61 3 9761 5050
F: 61 3 9761 5051
W: www.motec.com
Disclaimer Notice:
This message, including any attachments, contains confidential
information intended for a specific individual and purpose and is
protected by law. If you are not the intended recipient you should
delete this message. Any disclosure, copying, or distribution of this
message or the taking of any action based on it is strictly
prohibited.
More information about the Users
mailing list