[openib-general] ibv_reg_mr failure with pvfs on ehca?

[Christoph Raisch]

> (I am taking this back to the openib list because I think the list
> needs to hear about real applications that are hitting memory
> registration limits)
>
> What are the limits on the ehca memory registrations?
> Is there a limit to the number of regions that can be registered?

The numbe rof regions should not be limited,
The total size of regions is limited, all user applications together can
only register the complete available physical memory once.
The rationale behind that is that you can give away physical memory only
once to a application.
Registering shared memory regions on a "physical" memory region should be
unlimited as well.

> Is
> there any way (with kernel hacks) that we can register the entire
> address space of the application?
I'd guess you mean physical available memory space. Would be definetly hard
to "pin" virtual memory provided by swapping.

> We would like to be able to do RDMA
> sends and receives from anywhere in the application address space
> eventually, and only register it once.

Yes, that's the fastest way to use IB. But keep in mind that registered
memory is pinned and can't be given to "helper" tasks, like sshd.
So you have to restrict you application to max memory minus the memory
needed by base kernel+ daemons+bash+... to be able to "breathe".

>
> What is the point of RDMA for memory-intensive applications if you
> have to copy the data to a registered buffer before sending it anyway?
>

Regards . . . Christoph Raisch