[ofa-general] How many processes on a node can open IB device ?
changquing.tang at hp.com
Tue Aug 19 13:08:31 PDT 2008
Thank you very much for the info.
I hope Mellanox can tell me what to do next, Our project needs to run 2048 ranks
on a node, every rank has IB communication(most of them are sleeping, only a few are active).
> -----Original Message-----
> From: Roland Dreier [mailto:rdreier at cisco.com]
> Sent: Tuesday, August 19, 2008 2:53 PM
> To: Tang, Changqing
> Cc: general at lists.openfabrics.org
> Subject: Re: [ofa-general] How many processes on a node can
> open IB device ?
> > I have simple IBV code, which only open the device and create PD.
> > (attached below), then the code sleep there.
> > When I start as many processes as I could, it fails at
> 895 copies, it fails with error:
> That sounds right for mlx4 with default firmware on a 4KB
> page size (ie
> x86) system. There are 1024 pages of user access registers
> available, but 128 + 1 = 129 are reserved for internal driver
> use. So that would leave 895 available for userspace use,
> exactly as you found.
> You should be able to build firmware that supports more
> processes, but I believe there may be some
> performance/stability tradeoffs related to that -- Mellanox
> could tell you more.
> - R.
More information about the general