[ofiwg] [libfabric-users] mmap'ed kernel memory in fi_mr_reg
joern.schumacher at cern.ch
Tue Nov 27 00:08:30 PST 2018
On 11/26/2018 07:44 PM, Jason Gunthorpe wrote:
> On Mon, Nov 26, 2018 at 04:42:49PM +0100, Jörn Schumacher wrote:
>> On 11/19/2018 08:42 PM, Hefty, Sean wrote:
>>>>> The only alternative I can think of is to try a normal registration
>>>> call, and if that fails, try again using the physical flag. Would
>>>> this work, or does the normal registration call succeed, but produce
>>>> an unusable MR?
>>>> This would not work because of a subtlety of the physical memory
>>>> registration. The reason is that actually NULL is passed as address in
>>>> the call. Check the github link to my patch in the other E-Mail, there
>>>> is a line that replaces the address with NULL.
>>>> If a user passes an illegal virtual address the call should fail. But
>>>> if the libfabric call falls back to the physical address registration,
>>>> this would then actually succeed as the address is replaced with NULL.
>>> I looked back at the patches and related documentation. IMO, the verbs physical memory registration interface is just weird. There is no association between the actual pages and the region AFAICT.
>> Indeed this is a rather strange extension.
>> I came across a potential solution to adjust our driver to produce memory
>> that is compatible with the RDMA stack in the kernel. Supposedly there is an
>> alternative to remap_pfn_range. In that case we would not need the physical
>> memory registration in libfabric anymore and the overall solution would be
>> cleaner (not dependent on the verbs provider)
> Several other people have been interested in this, I think many would
> appreciate it if you share your solution to the linux-rdma mailing
Yes, I was thinking of writing this up for the list, will do that in the
next couple of days after running some tests.
More information about the ofiwg