----- Demesg report -----

ib0: ipoib_ib_post_receive failed for buf 111
ib0: failed to allocate receive buffer
ib0: ipoib_ib_post_receive failed for buf 112
ib0: failed to allocate receive buffer
ib0: ipoib_ib_post_receive failed for buf 113
ib0: failed to allocate receive buffer
ib0: ipoib_ib_post_receive failed for buf 114
ib0: failed to allocate receive buffer
ib0: ipoib_ib_post_receive failed for buf 115
ib0: failed to allocate receive buffer
ib0: ipoib_ib_post_receive failed for buf 116
ib0: failed to allocate receive buffer
ib0: ipoib_ib_post_receive failed for buf 117
ib0: failed to allocate receive buffer
ib0: ipoib_ib_post_receive failed for buf 118
LustreError: 3838:0:(ost_handler.c:735:ost_brw_write()) @@@ timeout on bulk
GET req@d89d2230 x1529751/t0 o4-><?>@<?>:-1 lens 328/288 ref 0 fl
Interpret:/0/0 rc 0/0
LustreError: 3853:0:(ost_handler.c:735:ost_brw_write()) @@@ timeout on bulk
GET req@f351cc00 x1553961/t0 o4-><?>@<?>:-1 lens 328/288 ref 0 fl
Interpret:/0/0 rc 0/0
LustreError: 3865:0:(ost_handler.c:822:ost_brw_write()) on6-ost2: bulk IO
comm e rror evicting 252c7_lov2_62a0a4e38b@NET_0xc0a8024e_UUID id
192.168.2.78-12345Lus
tre: A connection with 192.168.2.72 timed out; the network or that node may
be d own.
LustreError: 3541:0:(socknal_cb.c:2264:ksocknal_check_peer_timeouts())
Timeout o ut conn->0xc0a80248 ip 192.168.2.72:1021
Lustre: A connection with 192.168.2.79 timed out; the network or that node
may b e down.
LustreError: 3541:0:(socknal_cb.c:2264:ksocknal_check_peer_timeouts())
Timeout o ut conn->0xc0a8024f ip 192.168.2.79:1021
Lustre: A connection with 192.168.2.73 timed out; the network or that node
may b e down.
Lustre: previously skipped 2 similar messages
LustreError: 3541:0:(socknal_cb.c:2264:ksocknal_check_peer_timeouts())
Timeout o ut conn->0xc0a80249 ip 192.168.2.73:1022
LustreError: 3541:0:(socknal_cb.c:2264:ksocknal_check_peer_timeouts())
previousl y skipped 2 similar messages
LustreError: 3541:0:(socknal.c:1329:ksocknal_destroy_conn()) Completing
partial receive from 0xc0a80249, ip 192.168.2.73:1021, with error
LustreError: 3541:0:(events.c:320:server_bulk_callback()) event type 5,
status 1 9, desc f37e1000
LustreError: 3838:0:(ost_handler.c:822:ost_brw_write()) on6-ost2: bulk IO
comm e rror evicting e5c2f_lov2_58076a2400@NET_0xc0a80249_UUID id
192.168.2.73-12345Lus
treError: 3838:0:(ost_handler.c:822:ost_brw_write()) previously skipped 1
simila r messages
LustreError: 3838:0:(filter.c:1728:filter_grant_sanity_check())
filter_disconnec
t: tot_granted 58273792 != fo_tot_granted 59322368
LustreError: 3838:0:(filter.c:1731:filter_grant_sanity_check())
filter_disconnec
t: tot_pending 0 != fo_tot_pending 1048576
LustreError: 3838:0:(filter.c:1728:filter_grant_sanity_check())
filter_destroy_e
xport: tot_granted 38592512 != fo_tot_granted 39641088
LustreError: 3838:0:(filter.c:1731:filter_grant_sanity_check())
filter_destroy_e
xport: tot_pending 0 != fo_tot_pending 1048576
Lustre: A connection with 192.168.2.83 timed out; the network or that node
may b e down.
LustreError: 3541:0:(socknal_cb.c:2264:ksocknal_check_peer_timeouts())
Timeout o ut conn->0xc0a80253 ip 192.168.2.83:1021
Lustre: A connection with 192.168.2.78 timed out; the network or that node
may b e down.
LustreError: 3541:0:(socknal_cb.c:2264:ksocknal_check_peer_timeouts())
Timeout o ut conn->0xc0a8024e ip 192.168.2.78:1021
LustreError: 3541:0:(socknal.c:1329:ksocknal_destroy_conn()) Completing
partial receive from 0xc0a8024e, ip 192.168.2.78:1021, with error
LustreError: 3541:0:(events.c:320:server_bulk_callback()) event type 5,
status 1 9, desc d4e17000
LustreError: 3853:0:(ost_handler.c:822:ost_brw_write()) on6-ost2: bulk IO
comm e rror evicting 252c7_lov2_62a0a4e38b@NET_0xc0a8024e_UUID id
192.168.2.78-12345Lus
treError: 3834:0:(client.c:945:ptlrpc_expire_one_request()) @@@ timeout
(sent at  1128772410, 100s ago) req@e98d8000 x195/t0
o401->@NET_0xc0a80253_UUID:15 lens 296/64 ref 1 fl Rpc:/0/0 rc 0/0
LustreError: 3834:0:(recov_thread.c:410:log_commit_thread()) commit
ebb94000:e26 d8380 drop 7 cookies: rc -110


