[ofa-general] How many processes on a node can open IB device ?

[Tang, Changqing]

Roland:
        Thank you very much for the info.

        I hope Mellanox can tell me what to do next, Our project needs to run 2048 ranks
on a node, every rank has IB communication(most of them are sleeping, only a few are active).

--CQ

> -----Original Message-----
> From: Roland Dreier [mailto:rdreier at cisco.com]
> Sent: Tuesday, August 19, 2008 2:53 PM
> To: Tang, Changqing
> Cc: general at lists.openfabrics.org
> Subject: Re: [ofa-general] How many processes on a node can
> open IB device ?
>
>  > I have simple IBV code, which only open the device and create PD.
>  > (attached below), then the code sleep there.
>  >
>  >  When I start as many processes as I could, it fails at
> 895 copies, it fails with error:
>
> That sounds right for mlx4 with default firmware on a 4KB
> page size (ie
> x86) system.  There are 1024 pages of user access registers
> available, but 128 + 1 = 129 are reserved for internal driver
> use.  So that would leave 895 available for userspace use,
> exactly as you found.
>
> You should be able to build firmware that supports more
> processes, but I believe there may be some
> performance/stability tradeoffs related to that -- Mellanox
> could tell you more.
>
>  - R.
>