[ofa-general] can't use perfquery from git against late kernels

[Or Gerlitz]

Sasha,

I just case across a problem where with the latest management git - I can't use
perfquery against latest kernel (I also see the problem if running against the
drivers provided by ofed 1.4.x, but not against 1.3.y !)

The problem doesn't exist for smpquery - and also it doesn't exist if I use
perfquery from 1.4.x / 1.3.y - so it seems something was broken for user space
GMPs between late kernels and latest mng git.

--> run perfquery of 1.4.x

# rpm -qf /usr/sbin/perfquery
infiniband-diags-1.4.3_20081127_a92a5e5-1.ofed1.4.rc6

# /usr/sbin/perfquery 4 1
# Port counters: Lid 4 port 1
PortSelect:......................1
[... and so on all counters - works fine ...]

--> run perfquery built from latest git

# /home/ogerlitz/ib-mng/sbin/perfquery 4 1
ibwarn: [8278] mad_rpc: _do_madrpc failed; dport (Lid 4)
/home/ogerlitz/ib-mng/sbin/perfquery: iberror: failed: classportinfo query

running with -ddd (attached the two runs) - it seems that the
there's no response on the classportinfo PM mad.

Or.
-------------- next part --------------
send buf
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0001 8001 0000 0004 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0104 0101 0000 0000 2d21 6022 24ff 756e
0001 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0001 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
rcv buf
0104 0101 0000 0000 0000 005c 24ff 756e
0001 0000 0000 0000 0100 0000 0000 0000
001f a004 ff7f 0000 0f25 5dfc f17f 0000
1025 5dfc f17f 0000 0000 0000 0000 0000
c71d a004 ff7f 0000 5e26 44d3 3e00 0000
203c 3cfc f17f 0000 7834 5f30 2f70 6f72
c01e a004 ff7f 0000 7973 2f64 0000 0000
e019 a004 ff7f 0000 d01b a004 ff7f 0000
101b a004 ff7f 0000 60b9 74d3 3e00 0000
101f a004 ff7f 0000 f024 5dfc f17f 0000
60a7 74d3 3e00 0000 0000 0000 0100 0000
593f 3cfc f17f 0000 0800 0000 0000 0000
0000 0000 0000 0000 ffff ffff ffff ffff
1800 0000 2b00 0000 f924 5dfc f17f 0000
0000 0000 0000 0000 c81d a004 ff7f 0000
e644 3cfc 0300 0000 0000 0000 0000 0000
rcv buf
0104 0101 0000 0000 0000 005c 24ff 756e
0001 0000 0000 0000 0100 0000 0000 0000
001f a004 ff7f 0000 0f25 5dfc f17f 0000
1025 5dfc f17f 0000 0000 0000 0000 0000
c71d a004 ff7f 0000 5e26 44d3 3e00 0000
203c 3cfc f17f 0000 7834 5f30 2f70 6f72
c01e a004 ff7f 0000 7973 2f64 0000 0000
e019 a004 ff7f 0000 d01b a004 ff7f 0000
101b a004 ff7f 0000 60b9 74d3 3e00 0000
101f a004 ff7f 0000 f024 5dfc f17f 0000
60a7 74d3 3e00 0000 0000 0000 0100 0000
593f 3cfc f17f 0000 0800 0000 0000 0000
0000 0000 0000 0000 ffff ffff ffff ffff
1800 0000 2b00 0000 f924 5dfc f17f 0000
0000 0000 0000 0000 c81d a004 ff7f 0000
e644 3cfc 0300 0000 0000 0000 0000 0000
rcv buf
0104 0101 0000 0000 0000 005c 24ff 756e
0001 0000 0000 0000 0100 0000 0000 0000
001f a004 ff7f 0000 0f25 5dfc f17f 0000
1025 5dfc f17f 0000 0000 0000 0000 0000
c71d a004 ff7f 0000 5e26 44d3 3e00 0000
203c 3cfc f17f 0000 7834 5f30 2f70 6f72
c01e a004 ff7f 0000 7973 2f64 0000 0000
e019 a004 ff7f 0000 d01b a004 ff7f 0000
101b a004 ff7f 0000 60b9 74d3 3e00 0000
101f a004 ff7f 0000 f024 5dfc f17f 0000
60a7 74d3 3e00 0000 0000 0000 0100 0000
593f 3cfc f17f 0000 0800 0000 0000 0000
0000 0000 0000 0000 ffff ffff ffff ffff
1800 0000 2b00 0000 f924 5dfc f17f 0000
0000 0000 0000 0000 c81d a004 ff7f 0000
e644 3cfc 0300 0000 0000 0000 0000 0000
ibwarn: [8304] umad_init: umad_init
ibwarn: [8304] umad_open_port: ca (null) port 0
ibwarn: [8304] umad_get_cas_names: max 20
ibwarn: [8304] umad_get_cas_names: return 1 cas
ibwarn: [8304] resolve_ca_name: checking ca 'mlx4_0'
ibwarn: [8304] resolve_ca_port: checking ca 'mlx4_0'
ibwarn: [8304] umad_get_ca: ca_name mlx4_0
ibwarn: [8304] umad_get_ca: opened mlx4_0
ibwarn: [8304] resolve_ca_port: checking port 0
ibwarn: [8304] resolve_ca_port: checking port 1
ibwarn: [8304] resolve_ca_port: found active port 1
ibwarn: [8304] resolve_ca_name: found ca mlx4_0 with port 1 type 1
ibwarn: [8304] resolve_ca_name: found ca mlx4_0 with active port 1
ibwarn: [8304] umad_open_port: opening mlx4_0 port 1
ibwarn: [8304] dev_to_umad_id: mapped mlx4_0 1 to 0
ibwarn: [8304] umad_open_port: opened /dev/infiniband/umad0 fd 3 portid 0
ibwarn: [8304] umad_register: fd 3 mgmt_class 1 mgmt_version 1 rmpp_version 0 method_mask (nil)
ibwarn: [8304] umad_register: fd 3 registered to use agent 0 qp 0
ibwarn: [8304] umad_register: fd 3 mgmt_class 129 mgmt_version 1 rmpp_version 0 method_mask (nil)
ibwarn: [8304] umad_register: fd 3 registered to use agent 1 qp 0
ibwarn: [8304] umad_register: fd 3 mgmt_class 3 mgmt_version 2 rmpp_version 1 method_mask (nil)
ibwarn: [8304] umad_register: fd 3 registered to use agent 2 qp 1
ibwarn: [8304] umad_register: fd 3 mgmt_class 4 mgmt_version 1 rmpp_version 0 method_mask (nil)
ibwarn: [8304] umad_register: fd 3 registered to use agent 3 qp 1
ibwarn: [8304] pma_query_via: lid 4 port 1
ibwarn: [8304] umad_set_addr: umad 0x7fff04a01bc0 dlid 4 dqp 1 sl 0, qkey 80010000
ibwarn: [8304] _do_madrpc: >>> sending: len 256 pktsz 320
ibwarn: [8304] umad_send: fd 3 agentid 0 umad 0x7fff04a01bc0 timeout 1000
ibwarn: [8304] umad_dump: agent id 0 status 0 timeout 1000
ibwarn: [8304] umad_addr_dump: qpn 1 qkey 0x80010000 lid 4 sl 0
grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x0
Gid 0x00000000000000000000000000000000
ibwarn: [8304] umad_recv: fd 3 umad 0x7fff04a017c0 timeout 1000
ibwarn: [8304] umad_recv: mad received by agent 0 length 88
ibwarn: [8304] _do_madrpc: rcv buf:
ibwarn: [8304] _do_madrpc: retry 1 (timeout 1000 ms)
ibwarn: [8304] umad_send: fd 3 agentid 0 umad 0x7fff04a01bc0 timeout 1000
ibwarn: [8304] umad_dump: agent id 0 status 0 timeout 1000
ibwarn: [8304] umad_addr_dump: qpn 1 qkey 0x80010000 lid 4 sl 0
grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x0
Gid 0x00000000000000000000000000000000
ibwarn: [8304] umad_recv: fd 3 umad 0x7fff04a017c0 timeout 1000
ibwarn: [8304] umad_recv: mad received by agent 0 length 88
ibwarn: [8304] _do_madrpc: rcv buf:
ibwarn: [8304] _do_madrpc: retry 2 (timeout 1000 ms)
ibwarn: [8304] umad_send: fd 3 agentid 0 umad 0x7fff04a01bc0 timeout 1000
ibwarn: [8304] umad_dump: agent id 0 status 0 timeout 1000
ibwarn: [8304] umad_addr_dump: qpn 1 qkey 0x80010000 lid 4 sl 0
grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x0
Gid 0x00000000000000000000000000000000
ibwarn: [8304] umad_recv: fd 3 umad 0x7fff04a017c0 timeout 1000
ibwarn: [8304] umad_recv: mad received by agent 0 length 88
ibwarn: [8304] _do_madrpc: rcv buf:
ibwarn: [8304] _do_madrpc: timeout after 3 retries, 3000 ms
ibwarn: [8304] mad_rpc: _do_madrpc failed; dport (Lid 4)
/home/ogerlitz/ib-mng/sbin/perfquery: iberror: [pid 8304] main: failed: classportinfo query
-------------- next part --------------
send buf
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0001 8001 0000 0004 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0104 0101 0000 0000 4866 a330 42b8 9877
0001 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0001 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
rcv buf
0104 0181 0000 0000 0000 004b 42b8 9877
0001 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0101 1000 0000 0014 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
mad data
0101 1000 0000 0014 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
send buf
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0001 8001 0000 0004 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0104 0101 0000 0000 4866 a330 42b8 9878
0012 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0101 1000 0000 0014 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
rcv buf
0104 0181 0000 0000 0000 004b 42b8 9878
0012 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0101 1000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0006 1e23 0005 048b
0000 15c1 0000 11d0 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
mad data
0101 1000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0006 1e23 0005 048b
0000 15c1 0000 11d0 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
0000 0000 0000 0000 0000 0000 0000 0000
ibwarn: [8273] umad_init: umad_init
ibwarn: [8273] umad_open_port: ca (null) port 0
ibwarn: [8273] umad_get_cas_names: max 20
ibwarn: [8273] umad_get_cas_names: return 1 cas
ibwarn: [8273] resolve_ca_name: checking ca 'mlx4_0'
ibwarn: [8273] resolve_ca_port: checking ca 'mlx4_0'
ibwarn: [8273] umad_get_ca: ca_name mlx4_0
ibwarn: [8273] umad_get_ca: opened mlx4_0
ibwarn: [8273] resolve_ca_port: checking port 0
ibwarn: [8273] resolve_ca_port: checking port 1
ibwarn: [8273] resolve_ca_port: found active port 1
ibwarn: [8273] resolve_ca_name: found ca mlx4_0 with port 1 type 1
ibwarn: [8273] resolve_ca_name: found ca mlx4_0 with active port 1
ibwarn: [8273] umad_open_port: opening mlx4_0 port 1
ibwarn: [8273] dev_to_umad_id: mapped mlx4_0 1 to 0
ibwarn: [8273] umad_open_port: opened /dev/infiniband/umad0 fd 3 portid 0
ibwarn: [8273] umad_register: fd 3 mgmt_class 1 mgmt_version 1 rmpp_version 0 method_mask (nil)
ibwarn: [8273] umad_register: fd 3 registered to use agent 0 qp 0
ibwarn: [8273] umad_register: fd 3 mgmt_class 129 mgmt_version 1 rmpp_version 0 method_mask (nil)
ibwarn: [8273] umad_register: fd 3 registered to use agent 1 qp 0
ibwarn: [8273] umad_register: fd 3 mgmt_class 3 mgmt_version 2 rmpp_version 1 method_mask (nil)
ibwarn: [8273] umad_register: fd 3 registered to use agent 2 qp 1
ibwarn: [8273] umad_register: fd 3 mgmt_class 4 mgmt_version 1 rmpp_version 0 method_mask (nil)
ibwarn: [8273] umad_register: fd 3 registered to use agent 3 qp 1
ibwarn: [8273] pma_query_via: lid 4 port 1
ibwarn: [8273] umad_set_addr: umad 0x7fffe8c9a620 dlid 4 dqp 1 sl 0, qkey 80010000
ibwarn: [8273] _do_madrpc: >>> sending: len 256 pktsz 320
ibwarn: [8273] umad_send: fd 3 agentid 3 umad 0x7fffe8c9a620 timeout 1000
ibwarn: [8273] umad_dump: agent id 3 status 0 timeout 1000
ibwarn: [8273] umad_addr_dump: qpn 1 qkey 0x80010000 lid 0x4 sl 0
grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x0
Gid 0x00000000000000000000000000000000
ibwarn: [8273] umad_recv: fd 3 umad 0x7fffe8c9a220 timeout 1000
ibwarn: [8273] umad_recv: mad received by agent 3 length 320
ibwarn: [8273] _do_madrpc: rcv buf:
ibwarn: [8273] mad_rpc: data offs 64 sz 192
ibwarn: [8273] pma_query_via: lid 4 port 1
ibwarn: [8273] umad_set_addr: umad 0x7fffe8c9a1d0 dlid 4 dqp 1 sl 0, qkey 80010000
ibwarn: [8273] _do_madrpc: >>> sending: len 256 pktsz 320
ibwarn: [8273] umad_send: fd 3 agentid 3 umad 0x7fffe8c9a1d0 timeout 1000
ibwarn: [8273] umad_dump: agent id 3 status 0 timeout 1000
ibwarn: [8273] umad_addr_dump: qpn 1 qkey 0x80010000 lid 0x4 sl 0
grh_present 0 gid_index 0 hop_limit 0 traffic_class 0 flow_label 0x0 pkey_index 0x0
Gid 0x00000000000000000000000000000000
ibwarn: [8273] umad_recv: fd 3 umad 0x7fffe8c99dd0 timeout 1000
ibwarn: [8273] umad_recv: mad received by agent 3 length 320
ibwarn: [8273] _do_madrpc: rcv buf:
ibwarn: [8273] mad_rpc: data offs 64 sz 192
# Port counters: Lid 4 port 1
PortSelect:......................1
CounterSelect:...................0x1000
SymbolErrors:....................0
LinkRecovers:....................0
LinkDowned:......................0
RcvErrors:.......................0
RcvRemotePhysErrors:.............0
RcvSwRelayErrors:................0
XmtDiscards:.....................0
XmtConstraintErrors:.............0
RcvConstraintErrors:.............0
LinkIntegrityErrors:.............0
ExcBufOverrunErrors:.............0
VL15Dropped:.....................0
XmtData:.........................400931
RcvData:.........................328843
XmtPkts:.........................5569
RcvPkts:.........................4560