[ofa-general] How much pinned memory each QP needs ?

[Tang, Changqing]

So if I have 16K QP for a node, the QPs will use 16K*64K = 1G. It is fairly large memory.

If I change max_send_wr=16, inline_data=16, the memory per QP will
be 16*16 = 256 bytes ?

I am asking Mellanox engineer, is there any document or will document the formula  ?


--CQ

> -----Original Message-----
> From: Roland Dreier [mailto:rdreier at cisco.com]
> Sent: Tuesday, October 30, 2007 4:14 PM
> To: Tang, Changqing
> Cc: general at lists.openfabrics.org
> Subject: Re: [ofa-general] How much pinned memory each QP needs ?
>
>  >         We are running a big system with some memory
> issue(16K QPs), I want to know how much
>  > pinned memory each QP needs, the settings when creating QP
> is follows:
>  >
>  >         qp_init_attr.cap.max_send_wr = 136;
>  >         qp_init_attr.cap.max_recv_wr = 1;
>  >         qp_init_attr.cap.max_send_sge = 1;
>  >         qp_init_attr.cap.max_recv_sge = 1;
>  >         qp_init_attr.cap.max_inline_data = 128;
>
> Not sure really without tracing through the code.  The
> easiest way to find out would probably be to add a print
> statement into the low-level driver library to find out how
> big a buffer it allocates for the QP.
>
> Naively, the max_send_wr is going to be rounded up to a power
> of 2, so 256, and so will the work request size, which has to
> be over the inline data size of 128, so that will be 256
> bytes.  So I would guess you'll end up using about 256 * 256
> or 64 KB per QP.
>
>  - R.
>