[ofiwg] [libfabric-users] mmap'ed kernel memory in fi_mr_reg

[Jörn Schumacher]

On 11/19/2018 08:42 PM, Hefty, Sean wrote:
>>> The only alternative I can think of is to try a normal registration
>> call, and if that fails, try again using the physical flag.  Would
>> this work, or does the normal registration call succeed, but produce
>> an unusable MR?
>>
>> This would not work because of a subtlety of the physical memory
>> registration. The reason is that actually NULL is passed as address in
>> the call. Check the github link to my patch in the other E-Mail, there
>> is a line that replaces the address with NULL.
>>
>> If a user passes an illegal virtual address the call should fail. But
>> if the libfabric call falls back to the physical address registration,
>> this would then actually succeed as the address is replaced with NULL.
> 
> I looked back at the patches and related documentation.  IMO, the verbs physical memory registration interface is just weird.  There is no association between the actual pages and the region AFAICT.

Indeed this is a rather strange extension.

I came across a potential solution to adjust our driver to produce 
memory that is compatible with the RDMA stack in the kernel. Supposedly 
there is an alternative to remap_pfn_range. In that case we would not 
need the physical memory registration in libfabric anymore and the 
overall solution would be cleaner (not dependent on the verbs provider)

I will validate this and report back here. Thanks so far for the help.


Cheers,
Jörn