[ewg] traffic jittery, send queue full reports from mthca driver

[Or Gerlitz]

I just opened case #897 on the below, it happens with last night snapshot.

Or

client MT25204 FW 1.2.0 two CPUs, four cores each
server MT25418 FW 2.3.0 two CPUs, four cores each

client : iperf -c $server -P 4 -d -t 3600 -i 1
server : iperf -s -i 1

[  5] 39.0-40.0 sec  29.4 MBytes    246 Mbits/sec
[  4] 39.0-40.0 sec  25.5 MBytes    214 Mbits/sec
[  3] 34.0-35.0 sec  88.0 KBytes    721 Kbits/sec
[  3] 35.0-36.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 36.0-37.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 37.0-38.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 38.0-39.0 sec  0.00 Bytes  0.00 bits/sec
[  3] 39.0-40.0 sec  0.00 Bytes  0.00 bits/sec
[  5] 40.0-41.0 sec  38.5 MBytes    323 Mbits/sec
[  8] 40.0-41.0 sec  36.2 MBytes    304 Mbits/sec
[  9] 40.0-41.0 sec  54.3 MBytes    456 Mbits/sec
[ 10] 40.0-41.0 sec  32.1 MBytes    270 Mbits/sec
[ 11] 40.0-41.0 sec  29.4 MBytes    247 Mbits/sec
[SUM] 40.0-41.0 sec    152 MBytes  1.28 Gbits/sec

ib_mthca 0000:03:00.0: SQ 000404 full (756910656 head, 756910592 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (756915376 head, 756915312 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (757146224 head, 757146160 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (757146336 head, 757146272 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (757317104 head, 757317040 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (757361808 head, 757361744 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (757361920 head, 757361856 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (757515760 head, 757515696 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (757515872 head, 757515808 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (757515984 head, 757515920 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (757516112 head, 757516048 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (757516224 head, 757516160 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (757516352 head, 757516288 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (757516448 head, 757516384 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (757516576 head, 757516512 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (757523168 head, 757523104 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (757531472 head, 757531408 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (757531568 head, 757531504 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (757548064 head, 757548000 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (757582992 head, 757582928 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (758082528 head, 758082464 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (758162208 head, 758162144 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (758232720 head, 758232656 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (758232848 head, 758232784 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (758232960 head, 758232896 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (758233088 head, 758233024 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (758303696 head, 758303632 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (758303776 head, 758303712 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (758307744 head, 758307680 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (758307872 head, 758307808 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (758334928 head, 758334864 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (758335056 head, 758334992 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (758341744 head, 758341680 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (758341856 head, 758341792 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (758396784 head, 758396720 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (758445440 head, 758445376 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (758458000 head, 758457936 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (758458112 head, 758458048 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (758712016 head, 758711952 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (758877136 head, 758877072 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (758877248 head, 758877184 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (759188272 head, 759188208 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (759220784 head, 759220720 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (759220896 head, 759220832 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (759245296 head, 759245232 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (759258512 head, 759258448 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (759258640 head, 759258576 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (759261856 head, 759261792 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (759274880 head, 759274816 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (759447312 head, 759447248 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (759447424 head, 759447360 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (759484464 head, 759484400 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (759543808 head, 759543744 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (759548400 head, 759548336 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (759548512 head, 759548448 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (759618768 head, 759618704 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (759618880 head, 759618816 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (759683216 head, 759683152 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (759708720 head, 759708656 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (759717040 head, 759716976 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (759754576 head, 759754512 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (759976944 head, 759976880 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (759977056 head, 759976992 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (759996448 head, 759996384 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (760297536 head, 760297472 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (760297648 head, 760297584 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (760309488 head, 760309424 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (760322688 head, 760322624 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (760331808 head, 760331744 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (760331920 head, 760331856 tail, 64 max, 0 nreq)
ib0: failed to post zlen send
ib_mthca 0000:03:00.0: SQ 000404 full (760359936 head, 760359872 tail, 64 max, 0 nreq)
ib0: failed to post zlen send