[ofa-general] [PATCH 0/ 8] XRC patch series (including xrc receive-only QPs)

[Tang, Changqing]

Jack:
        In order to open a new XRC domain, all processes on a node open a file descriptor using the
same pathname, and pass the fd to ibv_open_xrc_domain().

        When can I close the fd ? when can I remove the temp file ?  Can I close the fd and unlink the
temp file right after ibv_open_xrc_domain() returns ?

        Does ibv_open_xrc_domain() increase the fd reference count and ibv_close_xrc_domain() decrease the
fd reference count ?

        Thanks.

--CQ


> -----Original Message-----
> From: general-bounces at lists.openfabrics.org
> [mailto:general-bounces at lists.openfabrics.org] On Behalf Of
> Jack Morgenstein
> Sent: Wednesday, January 23, 2008 4:00 AM
> To: Roland Dreier
> Cc: general at lists.openfabrics.org
> Subject: [ofa-general] [PATCH 0/ 8] XRC patch series
> (including xrc receive-only QPs)
>
> This patch series is the updated XRC implementation (kernel
> and user (libibverbs and libmlx4)).
>
> Please give feedback -- I'm still reviewing the locking in
> this implementation.
>
> The kernel patches are all based on
> git: //git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband.git
> branch: for-2.6.25
> commit: 5e8a3c6041ded7e306607bb6c96a0e68ca4dd2b4
>
> *****
> In addition, the kernel patch series requires that Eli
> Cohen's patch 7/16, posted January 16, be applied first
> ([ofa-general] [PATCH 7/16] ib/core: Add creation flags to QPs )
> *****
>
> The patches should be applied in the order posted.
>
> Changes:
> - Added creation of XRC receive-only QPs for userspace, which
>   reside in kernel space (user cannot post-to or poll these QPs).
>   Motivation:  MPI community required XRC receive QPs which would
>                not be destroyed when the creating process terminated.
>
>   Solution:  Userspace requests that a QP be created in kernel space.
>              Each userspace process using that QP (i.e. receiving
>              packets on an XRC SRQ via the qp), registers with
>              that QP (-- the creator is also registered, whether
>              or not it is a user of the QP). When the last
> userspace user
>              unregisters with the QP, it is destroyed.  Unregistration
>              is also part of userspace cleanup, so there is
> no leakage.
>
>   API for this:
>         ibv_create_xrc_rcv_qp
>         ibv_modify_xrc_rcv_qp
>         ibv_query_xrc_rcv_qp
>         ibv_reg_xrc_rcv_qp
>         ibv_unreg_xrc_rcv_qp
>
>   Creating process workflow:
>         ibv_create_xrc_rcv_qp -- to create
>         ibv_modify_xrc_rcv_qp -- to move QP to INIT
>         ibv_modify_xrc_rcv_qp -- to move QP to RTR
>               (to RTS is not needed for receive-only QPs)
>
>         ibv_unreg_xrc_rcv_qp -- instead of destroy.
>
>   Using process workflow
>         ibv_create_xrc_srq -- to create an SRQ
>         ibv_reg_xrc_rcv_qp -- to register with the QP as a user
>
>         ibv_destroy_srq
>         ibv_unreg_xrc_rcv_qp -- to "unregister" with the QP.  If no
>                                 user process remain registered, the
>                                 QP is destroyed.
>
> NOTES:
>   1. Since there is no userspace object for the QP, the API uses
>      the XRC domain object and qp number instead.
>
>   2. Registration needs to be performed only once per process
>      (multiple registrations count as a single registration).
>
>   3. Async events for the receive QP are delivered to all registered
>      processes.  The event ID is "OR'ed" with 0x80000000, to indicate
>      that this is an XRC receive-only QP event.  The element field
>      union value "xrc_qp_num" is set to the QP number which
> generated the
>      event.
>
> - Jack
>
> _______________________________________________
> general mailing list
> general at lists.openfabrics.org
> http://lists.openfabrics.org/cgi-bin/mailman/listinfo/general
>
> To unsubscribe, please visit
> http://openib.org/mailman/listinfo/openib-general
>