[openib-general] [PATCH/RFC 1/2] IB: Return "maybe_missed_event" hint from ib_req_notify_cq()

[Shirley Ma]

Roland Dreier <rdreier at cisco.com> wrote on 11/16/2006 11:26:31 AM:

>  > What I have found in ehca driver, n! = t, does't mean it's empty. If
poll
>  > again, there are still some packets in cq. IB_CQ_REPORT_mISSED_EVENTS
most
>  > of the time reports 1. It relies on netif_rx_reschedule() returns0 to
exit
>  > napi poll. That might be the reason in poll routine for a long time? I
will
>  > rerun my test to use n! = 0 to see any difference here.
>
> Maybe there's an ehca bug in poll CQ?  If n != t then it should mean
> that the CQ was indeed drained.  I would expect a missed event would
> be rare, because it means a completion occurs between the last poll CQ
> and the request notify, and that shouldn't be that common...
>
> My rough estimate is that even at a higher throughput than what you're
> seeing, IPoIB should only generate ~ 500K completions/sec, which means
> the average delay between completions is 2 microseconds.  So I
> wouldn't expect completions to hit the window between poll and request
> notify that often.
>
>  - R.

I have tried low_latency is 1 to disable TCP prequeue, the throughput was
increased from 1XXMb/s to 4XXMb/s. If I delayed net_skb_receive() a little
bit, I could get around 1700Mb/s. If I totally disable
netif_rx_reschedule(), then there is no repoll and return 0, I could get
around 2900Mb/s throughout without packet seeing out of order issues. I
have tried to add a spin lock in ipoib_poll(). And I still see packets out
of orders.

disable prequeue: 2XXMb/s to 4XXMb/s (packets out of order)
slowdown netif_receive_skb: 17XXMb/s (packets out of order)
don't handle missed event: 28XXMb/s (no packets out of order)
handler missed envent later: 7XXMb/s to 11XXMb/s (packets out of order)

Maybe it is ehca driver deliver packets much faster?  Which makes me think
user processes tcp backlogqueue, prequeue might be out of order?

Thanks
Shirley Ma
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/general/attachments/20061116/141b1609/attachment.html>