[ewg] RFCv2: SRC API
Michael S. Tsirkin
mst at dev.mellanox.co.il
Mon Aug 6 06:35:00 PDT 2007
This is version 2 of the proposal, addressing comments
from version 1.
Changelog:
- Use oflags to make API smaller
- Clarify sharing semantics
- Add documentation
This is the API proposal for support of the SRC
(scalable reliable connected) protocol extension in libibverbs.
This adds APIs to:
- manage SRC domains
- share SRC domains between processes,
by means of creating a 1:1 association
between an SRC domain and an inode.
Notes:
- The inode is specified by means of a file descriptor,
this makes it possible for the user to manage file
creation/deletion in the most flexible manner
(e.g. tmpfile can be used).
- I envision implementing this sharing mechanism in kernel by means
of a per-device tree, with inode as a key and domain object
as a value.
Please comment.
Signed-off-by: Michael S. Tsirkin <mst at dev.mellanox.co.il>
----
diff --git a/SRC.txt b/SRC.txt
new file mode 100644
index 0000000..3881477
--- /dev/null
+++ b/SRC.txt
@@ -0,0 +1,133 @@
+Here's some documentation on Scalable Reliable Connections.
+
+ * * *
+
+SRC is an extension supported by recent Mellanox hardware
+which is geared toward reducing the number of QPs
+required for all-to-all communication on systems
+with a high number of jobs per node.
+
+===================================================================
+Motivation:
+===================================================================
+Given N nodes with J jobs per node, number of QPs required
+for all-to-all communication is:
+
+With RC:
+ O((N * J) ^ 2)
+
+ Since each job out of O(N * J) jobs must create a single QP
+ to communicate with each one of O(N * J) other jobs.
+
+With SRC:
+ O(N ^ 2 * J)
+
+ This is achived by using a single send queue (per job, out of O(N * J) jobs)
+ to send data to all J jobs running on a specific node (out of O(N) nodes).
+ Hardware uses new "SRQ number" field in packet header to
+ multiplex receive WRs and WCs to private memory of each job.
+
+This is similiar idea to IB RD.
+Q: Why not use RD then?
+A: Because no hardware supports it.
+
+Details:
+
+===================================================================
+Verbs extension:
+===================================================================
+
+- There is a new transport/QP type "SRC".
+- There is a new object type "SRC domain"
+- Each SRQ gets new (optional) attributes:
+ SRC domain
+ SRC SRQ number
+ SRC CQ
+ SRQ must have either all 3 of these or none of these attributes
+
+- QPs of type SRC have all the same attributes as regular RC QPs
+ connected to SRQ, except that:
+ A. Each SRC QP has a new required attribute "SRC domain"
+ B. SRC QPs do *not* have "SRQ" attribute
+ (do not have a specific SRQ associated with them)
+
+===================================================================
+Protocol extension:
+===================================================================
+SRC QP behaviour: Requestor
+- Post send WR for this QP type is extended with SRQ number field
+ This number is sent as part of packet header
+- SRC Packets follow rules for RC packets on the wire, exactly
+ What is different is their handling at the responder side
+
+SRC QP behaviour: Responder
+Each incoming packet passes transport checks with respect
+to the SRC QP, following RC rules, exactly.
+
+After this, SRQ number in packet header is used to look up
+a specific SRQ. SRC domain of the resulting SRQ must be equal
+to SRC domain of the QP, otherwise a NAK is sent,
+and QP moves to error state.
+
+If the SRC domains match, receive WR and receive WC processing
+are as follows:
+
+- RC Send
+ - Rather than using SRQ to which the QP is attached,
+ SRQ is looked up by SRQ number in the packet.
+ Receive WR is taken from this SRQ.
+ - Completions are generated on the CQ specified in the SRQ
+
+- RDMA/Atomic
+ - Rather than using PD to which the QP is attached,
+ SRQ is looked up by SRQ number in the packet.
+ PD of this SRQ is used for protection checks.
+
+===================================================================
+Pseudo code:
+===================================================================
+
+Consider again a setup where there are N nodes with J jobs per node.
+All N * J jobs need to perform all-to-all communication.
+Using RC QPs, this would call for O((N * J) ^ 2) QPs.
+Here is how SRC can be used to reduce the number of QPs to O(N ^ 2 * J).
+
+At startup:
+1. All jobs on each node share a single SRC domain
+2. Each job creates a CQ for receive WCs
+3. Each job creates a SRQ attached to this CQ and to the shared domain
+
+When job j1 needs to transmit to job j2 on remote node n for the first time:
+1. Test: does job j1 have an existing connection to some job on node n?
+ - If no:
+ j1 creates an SRC QP qp1 (send QP)
+ qp1 is only used to post send WRs
+ j2 creates an SRC QP qp2
+ qp2 is part of SRC domain
+ qp2 is only used to do transport checks:
+ neither send nor receive WRs are posted on qp2
+ j1 and j2 create a connection between qp1 and qp2
+ - If yes:
+ let qp1 be the QP which belongs to j1 and is connected
+ to some qp on node n
+
+2. j1 gets SRQ number from j2
+3. j1 can now use QP qp2 from step 1
+ and SRQ number from step 3 to send data to j2
+
+Cleanup:
+When job j1 does not need to communicate to any jobs on node n,
+it disconnects qp1 from qp2, and asks j2 to destroy qp2.
+
+===================================================================
+
+Resources used (CQs are ignored below):
+Each node:
+- An SRC domain - to the total of n domains
+- A Receive QP for each (remote) job - to the total of N * (N * J) recv QPs
+
+Each job:
+- A SRQ - to the total of N * J SRQs
+- A send QP for each (remote) node - to the total of N * (N * J) send QPs
+
+===================================================================
diff --git a/include/infiniband/verbs.h b/include/infiniband/verbs.h
index acc1b82..d18475a 100644
--- a/include/infiniband/verbs.h
+++ b/include/infiniband/verbs.h
@@ -142,6 +142,7 @@ struct ibv_device_attr {
uint16_t max_pkeys;
uint8_t local_ca_ack_delay;
uint8_t phys_port_cnt;
+ int max_src_domain;
};
enum ibv_mtu {
@@ -370,6 +371,11 @@ struct ibv_ah_attr {
uint8_t port_num;
};
+struct ibv_src_domain {
+ struct ibv_context *context;
+ uint32_t handle;
+};
+
enum ibv_srq_attr_mask {
IBV_SRQ_MAX_WR = 1 << 0,
IBV_SRQ_LIMIT = 1 << 1
@@ -389,7 +395,8 @@ struct ibv_srq_init_attr {
enum ibv_qp_type {
IBV_QPT_RC = 2,
IBV_QPT_UC,
- IBV_QPT_UD
+ IBV_QPT_UD,
+ IBV_QPT_SRC
};
struct ibv_qp_cap {
@@ -408,6 +415,7 @@ struct ibv_qp_init_attr {
struct ibv_qp_cap cap;
enum ibv_qp_type qp_type;
int sq_sig_all;
+ struct ibv_src_domain *src_domain;
};
enum ibv_qp_attr_mask {
@@ -526,6 +534,7 @@ struct ibv_send_wr {
uint32_t remote_qkey;
} ud;
} wr;
+ uint32_t src_remote_srq_num;
};
struct ibv_recv_wr {
@@ -553,6 +562,10 @@ struct ibv_srq {
pthread_mutex_t mutex;
pthread_cond_t cond;
uint32_t events_completed;
+
+ uint32_t src_srq_num;
+ struct ibv_src_domain *src_domain;
+ struct ibv_cq *src_cq;
};
struct ibv_qp {
@@ -570,6 +583,8 @@ struct ibv_qp {
pthread_mutex_t mutex;
pthread_cond_t cond;
uint32_t events_completed;
+
+ struct ibv_src_domain *src_domain;
};
struct ibv_comp_channel {
@@ -912,6 +927,25 @@ struct ibv_srq *ibv_create_srq(struct ibv_pd *pd,
struct ibv_srq_init_attr *srq_init_attr);
/**
+ * ibv_create_src_srq - Creates a SRQ associated with the specified protection
+ * domain and src domain.
+ * @pd: The protection domain associated with the SRQ.
+ * @src_domain: The SRC domain associated with the SRQ.
+ * @src_cq: CQ to report completions for SRC packets on.
+ *
+ * @srq_init_attr: A list of initial attributes required to create the SRQ.
+ *
+ * srq_attr->max_wr and srq_attr->max_sge are read the determine the
+ * requested size of the SRQ, and set to the actual values allocated
+ * on return. If ibv_create_srq() succeeds, then max_wr and max_sge
+ * will always be at least as large as the requested values.
+ */
+struct ibv_srq *ibv_create_src_srq(struct ibv_pd *pd,
+ struct ibv_src_domain *src_domain,
+ struct ibv_cq *src_cq,
+ struct ibv_srq_init_attr *srq_init_attr);
+
+/**
* ibv_modify_srq - Modifies the attributes for the specified SRQ.
* @srq: The SRQ to modify.
* @srq_attr: On input, specifies the SRQ attributes to modify. On output,
@@ -1074,6 +1108,42 @@ int ibv_detach_mcast(struct ibv_qp *qp, union ibv_gid *gid, uint16_t lid);
*/
int ibv_fork_init(void);
+/**
+ * ibv_open_src_domain - open an SRC domain
+ * Returns a reference to an SRC domain.
+ *
+ * @context: Device context
+ * @fd: descriptor for inode associated with the domain
+ * If fd == -1, no inode is associated with the domain; in this case,
+ * the only legal value for oflag is O_CREAT
+ *
+ * @oflag: oflag values are constructed by OR-ing flags from the following list
+ *
+ * O_CREAT
+ * If a domain belonging to device named by context is already associated
+ * with the inode, this flag has no effect, except as noted under O_EXCL
+ * below. Otherwise, a new SRC domain is created and is associated with
+ * inode specified by fd.
+ *
+ * O_EXCL
+ * If O_EXCL and O_CREAT are set, open will fail if a domain associated with
+ * the inode exists. The check for the existence of the domain and creation
+ * of the domain if it does not exist is atomic with respect to other
+ * processes executing open with fd naming the same inode.
+ */
+struct ibv_src_domain *ibv_open_src_domain(struct ibv_context *context,
+ int fd, int oflag);
+
+/**
+ * ibv_close_src_domain - close an SRC domain
+ * If this is the last reference, destroys the domain.
+ *
+ * @d: reference to SRC domain to close
+ *
+ * close is implicitly performed at process exit.
+ */
+int ibv_close_src_domain(struct ibv_src_domain *d);
+
END_C_DECLS
# undef __attribute_const
--
MST
More information about the ewg
mailing list