[openib-general] question about poll_cq()

[Roland Dreier]

    Joerg> hi, i hope it's ok to ask this question here.  i just want
    Joerg> to know more about poll_cq().  the standard seems to define
    Joerg> only the input/ouput-params.  after reading the code i
    Joerg> figured out that there is some kind of "mapping" from
    Joerg> libibverbs to the device specific "plugin" (mthca).  so it
    Joerg> looks like that the ibv_poll_cq() call from the userspace
    Joerg> (e.g.  ibv_xx_pingpong) finally ends in a call of:
    Joerg> mthca_poll_cq() in mthca_cq.c

Yes, if the underlying device is a Mellanox HCA.  PathScale HCAs will
end up in a function in libipathverbs, and IBM eHCAs will end up in libehca.

    Joerg> so my question is: can someone give me a brief summary what
    Joerg> mthca_poll_cq (or mthca_poll_one()) is really doing?  i
    Joerg> want to know how polling the completion queue really works,
    Joerg> there must be some kind of low-level acknowledge/response-
    Joerg> message if a new entry enters the cq?  where is the
    Joerg> completion queue really located (on a rdma operation) -
    Joerg> local or remote?

A very brief sketch of what happens is that the device-specific
implementation of CQs for Mellanox HCAs allocates a circular buffer in
memory and passes the address to the hardware.  The buffer is divided
into fixed-size chunks, each of which represents one completion entry.
Initially the buffer is cleared out, and every time the hardware adds
an entry onto the completion queue, it sets a bit in that chunk to
show that the entry is now valid.  The driver polls the CQ by looking
to see if the next chunk has the bit set.  If it does, then the driver
translates the entry from hardware format into standard struct ibv_wc
format; if it doesn't, then the driver returns status indicating that
the CQ is empty.

Completion queues are always located in local system memory.

 - R.