[openib-general] CQ peek

[Roland Dreier]

First, a general point about peek CQ.  For all RDMA device
architectures that I know of, doing "peek CQ" is just as expensive as
doing "poll CQ," since the driver needs to do the same locking and
incur the same cache misses either way.  And then the app will always
have to do "poll CQ" later anyway.

    Kevin> Peeks are often used to determine what processing to
    Kevin> perform next in code that doesn't want to do the processing
    Kevin> itself.  A simple example would be if your in a send thread
    Kevin> was sending lots on packets every so often it may peek at a
    Kevin> CQ used for receive completions and if their are receives
    Kevin> ready then either signal the receive thread or just put a
    Kevin> marker on the wall and exit to allow receive processing to
    Kevin> happen. Usually ULP that support polling modes give
    Kevin> priority to receives over sends so peek is an easy way for
    Kevin> the send thread to cool it and switch gears to received.

It seems better to use a completion event on the receive CQ to wake
up the receive thread, or just poll the receive CQ and pass any
completions you find to the receive queue.  Doing a peek in one thread
and a poll in another is almost guaranteed to be terrible for performance.

    Kevin> Peek may also be useful to batch together completion
    Kevin> processing into a bigger therefore potentially more
    Kevin> efficient unit. For example, you have a send fifo where the
    Kevin> completions are steered into a completion queue.  You may
    Kevin> only want to process the send completions after some number
    Kevin> of send fifo slots have completed therefore batching up the
    Kevin> processing into big units.

Again, I think peek is the wrong interface here.  Either continue
polling and accumulating completions until you get the number of
completions you want to batch, or have only every Nth send generate a
completion at all.

    Kevin> My last and least convincing reason for peek is if we want
    Kevin> the gen2 user space libraries to have a broad set of ULP
    Kevin> layer ontop of it.  Some of those ULPs may have implemented
    Kevin> peek in their interface and not having it in the lower
    Kevin> layer makes it difficult.

This is moderately compelling, and if someone has a concrete use for
poll CQ I'll probably end up implementing it someday (unless someone
beats me to it).  But I'd be somewhat surprised if a ULP uses a
primitive that is not defined in either the IB or iWARP verbs.

 - R.