[ofa-general] problems using smpdump

Or Gerlitz ogerlitz at voltaire.com
Wed Jan 28 06:44:29 PST 2009 

Hi Sasha,

I'm having some problems with smpdump when used with the Mellanox IS4
switch, for example for nodeinfo (0x11), both smpquery and smpdump when
run against HCA in this form

	$ smpquery nodeinfo LID
	$ smpdump LID 0x11

produce the --same-- response mad, but when run against IS4 smpdump doesn't
return anything. Invoking both with -ddd I noticed these two lines:

ibwarn: [30549] umad_set_addr: umad 0xae0c010 dlid 12 dqp 0 sl 65535, qkey 0
ibwarn: [30549] umad_addr_dump: qpn 0 qkey 0x0 lid 0xc sl 255

so the SLs used by smpdump are wierd (255 and 65535), is it just
print-errors or can suggest a possible explanation for the failure?

I am using the latest mng git, did all the runs from a host, see the
runs info and also the IS4 FW/etc info (mstflint q) below.

Or.


$ /home/ogerlitz/ib-mng/sbin/ibnetdiscover -P 2

#
# Topology file: generated on Wed Jan 28 16:22:10 2009
#
# Max of 2 hops discovered
# Initiated from node 0002c90300026be2 port 0002c90300026be4

vendid=0x2c9
devid=0xbd36
sysimgguid=0x8f100010c0063
switchguid=0x8f100010c0062(8f100010c0062)
Switch	36 "S-0008f100010c0062"		# "Infiniscale-IV Mellanox Technologies" base port 0 lid 12 lmc 0
[27]	"H-0002c90300026be6"[2](2c90300026be8) 		# "linux-cto-1 HCA-1" lid 7 4xDDR
[6]	"H-0002c90300026be2"[2](2c90300026be4) 		# " HCA-1" lid 3 4xDDR

vendid=0x2c9
devid=0x6732
sysimgguid=0x2c90300026be9
caguid=0x2c90300026be6
Ca	2 "H-0002c90300026be6"		# "linux-cto-1 HCA-1"
[2](2c90300026be8) 	"S-0008f100010c0062"[27]		# lid 7 lmc 0 "Infiniscale-IV Mellanox Technologies" lid 12 4xDDR

vendid=0x2c9
devid=0x6732
sysimgguid=0x2c90300026be5
caguid=0x2c90300026be2
Ca	2 "H-0002c90300026be2"		# " HCA-1"
[2](2c90300026be4) 	"S-0008f100010c0062"[6]		# lid 3 lmc 0 "Infiniscale-IV Mellanox Technologies" lid 12 4xDDR

$ /home/ogerlitz/ib-mng/sbin/smpquery -P 2 nodeinfo 12

# Node info: Lid 12
BaseVers:........................1
ClassVers:.......................1
NodeType:........................Switch
NumPorts:........................36
SystemGuid:......................0x0008f100010c0063
Guid:............................0x0008f100010c0062
PortGuid:........................0x0008f100010c0062
PartCap:.........................8
DevId:...........................0xbd36
Revision:........................0x000000a0
LocalPort:.......................6
VendorId:........................0x0002c9

$ /home/ogerlitz/ib-mng/sbin/smpdump -P 2 12 0x11

0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000

$ /home/ogerlitz/ib-mng/sbin/smpdump -ddd -P 2 12 0x11

before send:
0101 0101 0000 0000 0000 0000 0000 0123
0011 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
ibwarn: [30549] umad_init: umad_init
ibwarn: [30549] umad_open_port: ca (null) port 2
ibwarn: [30549] umad_get_cas_names: max 20
ibwarn: [30549] umad_get_cas_names: return 1 cas
ibwarn: [30549] resolve_ca_name: checking ca 'mlx4_0'
ibwarn: [30549] resolve_ca_port: checking ca 'mlx4_0'
ibwarn: [30549] umad_get_ca: ca_name mlx4_0
ibwarn: [30549] umad_get_ca: opened mlx4_0
ibwarn: [30549] resolve_ca_name: found ca mlx4_0 with port 2 type 1
ibwarn: [30549] resolve_ca_name: found ca mlx4_0 with active port 2
ibwarn: [30549] umad_open_port: opening mlx4_0 port 2
ibwarn: [30549] dev_to_umad_id: mapped mlx4_0 2 to 1
ibwarn: [30549] umad_open_port: opened /dev/infiniband/umad1 fd 3 portid 1
ibwarn: [30549] umad_register: fd 3 mgmt_class 1 mgmt_version 1 rmpp_version 0 method_mask (nil)
ibwarn: [30549] umad_register: fd 3 registered to use agent 0 qp 0
ibwarn: [30549] umad_set_addr: umad 0xae0c010 dlid 12 dqp 0 sl 65535, qkey 0
ibwarn: [30549] umad_send: fd 3 agentid 0 umad 0xae0c010 timeout 1000
ibwarn: [30549] umad_dump: agent id 0 status 0 timeout 1000
ibwarn: [30549] umad_addr_dump: qpn 0 qkey 0x0 lid 0xc sl 255
grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x0
Gid 0x00000000000000000000000000000000
ibwarn: [30549] umad_recv: fd 3 umad 0xae0c010 timeout 4294967295
ibwarn: [30549] umad_recv: mad received by agent 0 length 88
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000


$ /home/ogerlitz/ib-mng/sbin/smpquery -ddd -P 2 nodeinfo 12

send buf
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 000c 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0101 0101 0000 0000 4b6e eb31 6545 c29d
0011 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
rcv buf
0101 0181 0000 0000 0000 019d 6545 c29d
0011 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0101 0224 0008 f100 010c 0063 0008 f100
010c 0062 0008 f100 010c 0062 0008 bd36
0000 00a0 0600 02c9 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
mad data
0101 0224 0008 f100 010c 0063 0008 f100
010c 0062 0008 f100 010c 0062 0008 bd36
0000 00a0 0600 02c9 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
ibwarn: [30756] umad_init: umad_init
ibwarn: [30756] umad_open_port: ca (null) port 2
ibwarn: [30756] umad_get_cas_names: max 20
ibwarn: [30756] umad_get_cas_names: return 1 cas
ibwarn: [30756] resolve_ca_name: checking ca 'mlx4_0'
ibwarn: [30756] resolve_ca_port: checking ca 'mlx4_0'
ibwarn: [30756] umad_get_ca: ca_name mlx4_0
ibwarn: [30756] umad_get_ca: opened mlx4_0
ibwarn: [30756] resolve_ca_name: found ca mlx4_0 with port 2 type 1
ibwarn: [30756] resolve_ca_name: found ca mlx4_0 with active port 2
ibwarn: [30756] umad_open_port: opening mlx4_0 port 2
ibwarn: [30756] dev_to_umad_id: mapped mlx4_0 2 to 1
ibwarn: [30756] umad_open_port: opened /dev/infiniband/umad1 fd 3 portid 1
ibwarn: [30756] umad_register: fd 3 mgmt_class 1 mgmt_version 1 rmpp_version 0 method_mask (nil)
ibwarn: [30756] umad_register: fd 3 registered to use agent 0 qp 0
ibwarn: [30756] umad_register: fd 3 mgmt_class 129 mgmt_version 1 rmpp_version 0 method_mask (nil)
ibwarn: [30756] umad_register: fd 3 registered to use agent 1 qp 0
ibwarn: [30756] umad_register: fd 3 mgmt_class 3 mgmt_version 2 rmpp_version 1 method_mask (nil)
ibwarn: [30756] umad_register: fd 3 registered to use agent 2 qp 1
ibwarn: [30756] smp_query_via: attr 0x11 mod 0x0 route Lid 12
ibwarn: [30756] umad_set_addr: umad 0x7fffe43460c0 dlid 12 dqp 0 sl 0, qkey 0
ibwarn: [30756] _do_madrpc: >>> sending: len 256 pktsz 320
ibwarn: [30756] umad_send: fd 3 agentid 0 umad 0x7fffe43460c0 timeout 1000
ibwarn: [30756] umad_dump: agent id 0 status 0 timeout 1000
ibwarn: [30756] umad_addr_dump: qpn 0 qkey 0x0 lid 0xc sl 0
grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x0
Gid 0x00000000000000000000000000000000
ibwarn: [30756] umad_recv: fd 3 umad 0x7fffe4345cc0 timeout 1000
ibwarn: [30756] umad_recv: mad received by agent 0 length 320
ibwarn: [30756] _do_madrpc: rcv buf:
ibwarn: [30756] mad_rpc: data offs 64 sz 64
# Node info: Lid 12
BaseVers:........................1
ClassVers:.......................1
NodeType:........................Switch
NumPorts:........................36
SystemGuid:......................0x0008f100010c0063
Guid:............................0x0008f100010c0062
PortGuid:........................0x0008f100010c0062
PartCap:.........................8
DevId:...........................0xbd36
Revision:........................0x000000a0
LocalPort:.......................6
VendorId:........................0x0002c9

$ /home/ogerlitz/ib-mng/sbin/smpquery -ddd -P 2 nodeinfo 7

send buf
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0007 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0101 0101 0000 0000 0c26 8ddd 34d6 34b1
0011 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
rcv buf
0101 0181 0000 0000 0000 01a6 34d6 34b1
0011 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0101 0102 0002 c903 0002 6be9 0002 c903
0002 6be6 0002 c903 0002 6be8 0080 6732
0000 00a0 0200 02c9 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
mad data
0101 0102 0002 c903 0002 6be9 0002 c903
0002 6be6 0002 c903 0002 6be8 0080 6732
0000 00a0 0200 02c9 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
ibwarn: [32714] umad_init: umad_init
ibwarn: [32714] umad_open_port: ca (null) port 2
ibwarn: [32714] umad_get_cas_names: max 20
ibwarn: [32714] umad_get_cas_names: return 1 cas
ibwarn: [32714] resolve_ca_name: checking ca 'mlx4_0'
ibwarn: [32714] resolve_ca_port: checking ca 'mlx4_0'
ibwarn: [32714] umad_get_ca: ca_name mlx4_0
ibwarn: [32714] umad_get_ca: opened mlx4_0
ibwarn: [32714] resolve_ca_name: found ca mlx4_0 with port 2 type 1
ibwarn: [32714] resolve_ca_name: found ca mlx4_0 with active port 2
ibwarn: [32714] umad_open_port: opening mlx4_0 port 2
ibwarn: [32714] dev_to_umad_id: mapped mlx4_0 2 to 1
ibwarn: [32714] umad_open_port: opened /dev/infiniband/umad1 fd 3 portid 1
ibwarn: [32714] umad_register: fd 3 mgmt_class 1 mgmt_version 1 rmpp_version 0 method_mask (nil)
ibwarn: [32714] umad_register: fd 3 registered to use agent 0 qp 0
ibwarn: [32714] umad_register: fd 3 mgmt_class 129 mgmt_version 1 rmpp_version 0 method_mask (nil)
ibwarn: [32714] umad_register: fd 3 registered to use agent 1 qp 0
ibwarn: [32714] umad_register: fd 3 mgmt_class 3 mgmt_version 2 rmpp_version 1 method_mask (nil)
ibwarn: [32714] umad_register: fd 3 registered to use agent 2 qp 1
ibwarn: [32714] smp_query_via: attr 0x11 mod 0x0 route Lid 7
ibwarn: [32714] umad_set_addr: umad 0x7fff2ae3abb0 dlid 7 dqp 0 sl 0, qkey 0
ibwarn: [32714] _do_madrpc: >>> sending: len 256 pktsz 320
ibwarn: [32714] umad_send: fd 3 agentid 0 umad 0x7fff2ae3abb0 timeout 1000
ibwarn: [32714] umad_dump: agent id 0 status 0 timeout 1000
ibwarn: [32714] umad_addr_dump: qpn 0 qkey 0x0 lid 0x7 sl 0
grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x0
Gid 0x00000000000000000000000000000000
ibwarn: [32714] umad_recv: fd 3 umad 0x7fff2ae3a7b0 timeout 1000
ibwarn: [32714] umad_recv: mad received by agent 0 length 320
ibwarn: [32714] _do_madrpc: rcv buf:
ibwarn: [32714] mad_rpc: data offs 64 sz 64
# Node info: Lid 7
BaseVers:........................1
ClassVers:.......................1
NodeType:........................Channel Adapter
NumPorts:........................2
SystemGuid:......................0x0002c90300026be9
Guid:............................0x0002c90300026be6
PortGuid:........................0x0002c90300026be8
PartCap:.........................128
DevId:...........................0x6732
Revision:........................0x000000a0
LocalPort:.......................2
VendorId:........................0x0002c9


$  /home/ogerlitz/ib-mng/sbin/smpdump -ddd -P 2 7 0x11

before send:
0101 0101 0000 0000 0000 0000 0000 0123
0011 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
ibwarn: [305] umad_init: umad_init
ibwarn: [305] umad_open_port: ca (null) port 2
ibwarn: [305] umad_get_cas_names: max 20
ibwarn: [305] umad_get_cas_names: return 1 cas
ibwarn: [305] resolve_ca_name: checking ca 'mlx4_0'
ibwarn: [305] resolve_ca_port: checking ca 'mlx4_0'
ibwarn: [305] umad_get_ca: ca_name mlx4_0
ibwarn: [305] umad_get_ca: opened mlx4_0
ibwarn: [305] resolve_ca_name: found ca mlx4_0 with port 2 type 1
ibwarn: [305] resolve_ca_name: found ca mlx4_0 with active port 2
ibwarn: [305] umad_open_port: opening mlx4_0 port 2
ibwarn: [305] dev_to_umad_id: mapped mlx4_0 2 to 1
ibwarn: [305] umad_open_port: opened /dev/infiniband/umad1 fd 3 portid 1
ibwarn: [305] umad_register: fd 3 mgmt_class 1 mgmt_version 1 rmpp_version 0 method_mask (nil)
ibwarn: [305] umad_register: fd 3 registered to use agent 0 qp 0
ibwarn: [305] umad_set_addr: umad 0x16174010 dlid 7 dqp 0 sl 65535, qkey 0
ibwarn: [305] umad_send: fd 3 agentid 0 umad 0x16174010 timeout 1000
ibwarn: [305] umad_dump: agent id 0 status 0 timeout 1000
ibwarn: [305] umad_addr_dump: qpn 0 qkey 0x0 lid 0x7 sl 255
grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x0
Gid 0x00000000000000000000000000000000
ibwarn: [305] umad_recv: fd 3 umad 0x16174010 timeout 4294967295
ibwarn: [305] umad_recv: mad received by agent 0 length 320
0101 0102 0002 c903 0002 6be9 0002 c903
0002 6be6 0002 c903 0002 6be8 0080 6732
0000 00a0 0200 02c9 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000

--> here's the IS4 info

# mstflint -d 81:00.0 q
Image type:      FS2
FW Version:      7.1.0
Device ID:       48438
Chip Revision:   A0
Description:     Node             Sys image
GUIDs:           0008f100010c0062 0008f100010c0063
Board ID:         (MT_0C20110003)
VSD:
PSID:            MT_0C20110003

More information about the general mailing list