[ofa-general] Re: [RFC] Receive Core Afinnity (aka RSS) support

[Eli Cohen]

On Thu, 2008-06-19 at 15:11 -0700, Roland Dreier wrote:
> >  /**
>  > + * ib_create_qp_range - Creates a range of QPs associated with the
>  > + * specified protection domain.
>  > + * @pd: The protection domain associated with the QP.
>  > + * @qp_init_attr: A list of initial attributes required to create the
>  > + *   QPs.  If QP creation succeeds, then the attributes are updated to
>  > + *   the actual capabilities of the created QP.
>  > + * @nqps: The number of required QPs
>  > + * @align: Required alignment for the allocated QP numbers.
>  > + * @list: pointer to an array to hold allocated QPs
>  > + */
>  > +int ib_create_qp_range(struct ib_pd *pd, struct ib_qp_init_attr *qp_init_attr,
>  > +		       int nqps, int align, struct ib_qp *list[]);
> 
> Well, RSS for IPoIB is OK in theory but I think we really are getting to
> the point of adding a lot of extra complexity to the RDMA core where
> IPoIB is the only user.  And this new interface for a QP range seems
> like the wrong level of abstraction -- it is exposing the mlx4
> implementation details of RSS.  If the next HCA that comes out has a
> different RSS scheme then we end up having to change all this.
> 
> So the details of how RSS works should be in low-level driver code.
I still need to create a bunch of QPs for this so I have to convey that
information to mlx4. We can get rid of the align parameter and use
another creation flag to let mlx4 know that we need the QPs aligned to
the size of required number of QPs.

We also need to let mlx4 know which QP is used as a default QP -- the
one QP that receives all rx packets that did not map to any of the other
QPs. I thought of doing this through modify QP by putting something like
this

+struct rca_attr {
+       int     base_qpn;
+       int     num_qps;
+       int     default_qpn;
+};
+
 struct ib_qp_attr {
        enum ib_qp_state        qp_state;
        enum ib_qp_state        cur_qp_state;
@@ -621,6 +630,7 @@ struct ib_qp_attr {
        u8                      rnr_retry;
        u8                      alt_port_num;
        u8                      alt_timeout;
+       struct rca_attr         rca;
 };


Also, after all this IPoIB so we need to handle posting of RX buffers to
receive queues at the IPoIB level so we need to assume that multiple RX
rings implies multiple QPs and it is likely that any implementation will
need similar things.

Do you have in mind something specific as for how you would like to see
the IB core API modified to support receive core affinity?



> 
> By the way, Michael Chan from Broadcom just posted multi-rx-ring support
> for bnx2 to netdev.

I did not find this. Is it in a published git tree? Can you send a
pointer?


--
Eli Cohen
Mellanox Technologies LTD.
SW- Linux
Phone: +972 (4) 909 7200 (ext 276)
Mobile: +972 (54) 650 1206
E-mail: eli at mellanox.co.il

----------------------------------------------------------------------
Emails belong on computers, trees belong in forests; if you must print
this, do it on recycled paper.
http://www.greenpeace.org/international/
----------------------------------------------------------------------


Disclaimer added by CodeTwo Exchange Rules
http://www.codetwo.com