[ofiwg] [libfabric-users] mmap'ed kernel memory in fi_mr_reg

[Jörn Schumacher]

On 11/26/2018 07:44 PM, Jason Gunthorpe wrote:
> On Mon, Nov 26, 2018 at 04:42:49PM +0100, Jörn Schumacher wrote:
>> On 11/19/2018 08:42 PM, Hefty, Sean wrote:
>>>>> The only alternative I can think of is to try a normal registration
>>>> call, and if that fails, try again using the physical flag.  Would
>>>> this work, or does the normal registration call succeed, but produce
>>>> an unusable MR?
>>>>
>>>> This would not work because of a subtlety of the physical memory
>>>> registration. The reason is that actually NULL is passed as address in
>>>> the call. Check the github link to my patch in the other E-Mail, there
>>>> is a line that replaces the address with NULL.
>>>>
>>>> If a user passes an illegal virtual address the call should fail. But
>>>> if the libfabric call falls back to the physical address registration,
>>>> this would then actually succeed as the address is replaced with NULL.
>>>
>>> I looked back at the patches and related documentation.  IMO, the verbs physical memory registration interface is just weird.  There is no association between the actual pages and the region AFAICT.
>>
>> Indeed this is a rather strange extension.
>>
>> I came across a potential solution to adjust our driver to produce memory
>> that is compatible with the RDMA stack in the kernel. Supposedly there is an
>> alternative to remap_pfn_range. In that case we would not need the physical
>> memory registration in libfabric anymore and the overall solution would be
>> cleaner (not dependent on the verbs provider)
> 
> Several other people have been interested in this, I think many would
> appreciate it if you share your solution to the linux-rdma mailing
> list.

Yes, I was thinking of writing this up for the list, will do that in the 
next couple of days after running some tests.

Jörn