[ofa-general] Issues to scale to 64K ranks.

[Dotan Barak]

Hi.

Tang, Changqing wrote:
> When talking to run 64K processes, I noticed that, on connectX with 2.2
> firmware,
> ibv_devinfo only shows: max_qp: 65472, that means we can not create
> 65536 QPs on
> this HCA, Is this max_qp on per process basis, or per HCA basis ?
>   
This number of QPs (and any other resource) is per HCA basis.

The HCA itself support much more QPs (and more elements from any other 
resource),
but the driver have limited the number of the QPs to consume less memory.

The mthca low level driver support changing the number of resources with 
module parameters,
this need to be done with the connectX low level driver as well.
> How to increase this number ? any hardware/firmwire change needed ?
>
>   
Until those module parameters will be added, the only way to do is to 
hack the low level driver.

Dotan