[ofa-general] [PATCH 0/ 8] XRC patch series (including xrc receive-only QPs)
jackm at dev.mellanox.co.il
Wed Jan 23 01:59:30 PST 2008
This patch series is the updated XRC implementation
(kernel and user (libibverbs and libmlx4)).
Please give feedback -- I'm still reviewing the locking in this implementation.
The kernel patches are all based on
In addition, the kernel patch series requires that Eli Cohen's
patch 7/16, posted January 16, be applied first
([ofa-general] [PATCH 7/16] ib/core: Add creation flags to QPs )
The patches should be applied in the order posted.
- Added creation of XRC receive-only QPs for userspace, which
reside in kernel space (user cannot post-to or poll these QPs).
Motivation: MPI community required XRC receive QPs which would
not be destroyed when the creating process terminated.
Solution: Userspace requests that a QP be created in kernel space.
Each userspace process using that QP (i.e. receiving
packets on an XRC SRQ via the qp), registers with
that QP (-- the creator is also registered, whether
or not it is a user of the QP). When the last userspace user
unregisters with the QP, it is destroyed. Unregistration
is also part of userspace cleanup, so there is no leakage.
API for this:
Creating process workflow:
ibv_create_xrc_rcv_qp -- to create
ibv_modify_xrc_rcv_qp -- to move QP to INIT
ibv_modify_xrc_rcv_qp -- to move QP to RTR
(to RTS is not needed for receive-only QPs)
ibv_unreg_xrc_rcv_qp -- instead of destroy.
Using process workflow
ibv_create_xrc_srq -- to create an SRQ
ibv_reg_xrc_rcv_qp -- to register with the QP as a user
ibv_unreg_xrc_rcv_qp -- to "unregister" with the QP. If no
user process remain registered, the
QP is destroyed.
1. Since there is no userspace object for the QP, the API uses
the XRC domain object and qp number instead.
2. Registration needs to be performed only once per process
(multiple registrations count as a single registration).
3. Async events for the receive QP are delivered to all registered
processes. The event ID is "OR'ed" with 0x80000000, to indicate
that this is an XRC receive-only QP event. The element field
union value "xrc_qp_num" is set to the QP number which generated the
More information about the general