[ofa-general] [RFC] Receive Core Afinnity (aka RSS) support

[Eli Cohen]

I plan to implement RCA for IPoIB and would like your comments. Using
RCA allows better utilization of available CPU cores power by evenly
distributing received SKBs between a few receive queues. Since each such
queue is connected to a different CQ, which in turn is connected to an
EQ which triggers a distinct IRQ vector, we can have more power to
process received buffers.
The implementation involves extending the ib core API, make
modifications in mlx4 and implementing in IPoIB. The planned work relies
on VPI patches sent to the general list for review to provide means for
allocating a group of QPs with consecutive QP numbers and alignment
requirements. The patch bellow describes some of the required changes to
the IB core.

diff --git a/include/rdma/ib_verbs.h b/include/rdma/ib_verbs.h
index 4c3c76b..4ca946c 100644
--- a/include/rdma/ib_verbs.h
+++ b/include/rdma/ib_verbs.h
@@ -104,6 +104,7 @@ enum ib_device_cap_flags {
 	IB_DEVICE_UD_IP_CSUM		= (1<<18),
 	IB_DEVICE_UD_TSO		= (1<<19),
 	IB_DEVICE_MEM_MGT_EXTENSIONS	= (1<<21),
+	IB_DEVICE_IPOIB_RCA		= (1<<22),
 };
 
 enum ib_atomic_cap {
@@ -505,6 +506,7 @@ enum ib_qp_type {
 
 enum ib_qp_create_flags {
 	IB_QP_CREATE_IPOIB_UD_LSO	= 1 << 0,
+	IB_QP_CREATE_IPOIB_RCA		= 1 << 1,
 };
 
 struct ib_qp_init_attr {
@@ -989,6 +991,9 @@ struct ib_device {
 	int                        (*post_srq_recv)(struct ib_srq *srq,
 						    struct ib_recv_wr *recv_wr,
 						    struct ib_recv_wr **bad_recv_wr);
+	int 			   (*create_qp_range)(struct ib_pd *pd,
+						      struct ib_qp_init_attr *qp_init_attr,
+						      int nqps, int align, struct ib_qp *list[]);
 	struct ib_qp *             (*create_qp)(struct ib_pd *pd,
 						struct ib_qp_init_attr *qp_init_attr,
 						struct ib_udata *udata);
@@ -1318,6 +1323,20 @@ static inline int ib_post_srq_recv(struct ib_srq *srq,
 }
 
 /**
+ * ib_create_qp_range - Creates a range of QPs associated with the
+ * specified protection domain.
+ * @pd: The protection domain associated with the QP.
+ * @qp_init_attr: A list of initial attributes required to create the
+ *   QPs.  If QP creation succeeds, then the attributes are updated to
+ *   the actual capabilities of the created QP.
+ * @nqps: The number of required QPs
+ * @align: Required alignment for the allocated QP numbers.
+ * @list: pointer to an array to hold allocated QPs
+ */
+int ib_create_qp_range(struct ib_pd *pd, struct ib_qp_init_attr *qp_init_attr,
+		       int nqps, int align, struct ib_qp *list[]);
+
+/**
  * ib_create_qp - Creates a QP associated with the specified protection
  *   domain.
  * @pd: The protection domain associated with the QP.


In IPoIB support will be added only for UD mode. For sending we will use
the same QP used today. For receive we will create a an RCA QP (by using
IB_QP_CREATE_IPOIB_RCA in the create flags) and a group of QPs. The base
QPN, number of QPs and a default QPN have to be passed to the RCA QP --
I am not sure what is the best way to do that but I am thinking of
passing these values with modify QP to init using new fields in struct
ib_qp_attr. The default QPN will be the current QP used for UD. We will
have multiple receive rings (as the number of QPs) so we don't need to
add spinlocks.


--
Eli Cohen
Mellanox Technologies LTD.
SW- Linux
Phone: +972 (4) 909 7200 (ext 276)
Mobile: +972 (54) 650 1206
E-mail: eli at mellanox.co.il

----------------------------------------------------------------------
Emails belong on computers, trees belong in forests; if you must print
this, do it on recycled paper.
http://www.greenpeace.org/international/
----------------------------------------------------------------------


Disclaimer added by CodeTwo Exchange Rules
http://www.codetwo.com