[libfabric-users] FI_MR_SCALABLE on GNI

[Biddiscombe, John A.]

Howard


Thanks for that link and your reply. I did waste a bit of time experimenting earlier, but soon found that leaving the mr_mode empty resulted in an error as well. I will stick to the MR_BASIC version and leave my memory management code in place. For a while I was thinking I could remove it all, but I guess it'll have to wait for a few more releases - or a new machine.


Cheers


JB


________________________________
From: Howard Pritchard <hppritcha at gmail.com>
Sent: 31 August 2020 17:57:37
To: Biddiscombe, John A.
Cc: libfabric-users at lists.openfabrics.org
Subject: Re: [libfabric-users] FI_MR_SCALABLE on GNI

Hi John,

Interesting you ask about this.  The answer is that the matrix is only partially correct.  The Aries hardware doesn't really support Mr. Scalable - it supports a very constrained sort of Mr. Scalable using FI_MR_MMU_NOTIFY.
There are other problems with the way it was implemented as well.  For GNI provider, you'll do better sticking with FI_MR_BASIC.  See the patches we're applying to Open MPI for this issue.


See https://github.com/ofiwg/libfabric/issues/6194

Howard


Am Mo., 31. Aug. 2020 um 05:00 Uhr schrieb Biddiscombe, John A. <biddisco at cscs.ch<mailto:biddisco at cscs.ch>>:

Dear List

libfabric:16321:gni:core:ofi_check_mr_mode():545<info> Invalid memory registration mode
libfabric:16321:gni:core:ofi_check_mr_mode():546<info> Expected: FI_MR_BASIC, FI_MR_MMU_NOTIFY
libfabric:16321:gni:core:ofi_check_mr_mode():546<info> Given: FI_MR_SCALABLE
libfabric:16321:gni:domain:_gnix_ep_getinfo():597<info> [16321:1] failed ofi_check_mr_mode, ret=-61


I'm using the libfabric 1.11 tag from the git repo and according to the feature matrix, scalable memory registration is supported on gni, however, I get the above error in the logs when I ask for it in using

    fabric_hints_->domain_attr->mr_mode = FI_MR_SCALABLE;


Is the matrix incorrect, or should I keep playing with flags in case I've messed something up elsewhere?


Thanks


JB

_______________________________________________
Libfabric-users mailing list
Libfabric-users at lists.openfabrics.org<mailto:Libfabric-users at lists.openfabrics.org>
https://lists.openfabrics.org/mailman/listinfo/libfabric-users
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20200831/e5388789/attachment.htm>