<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=UTF-8">
</head>
<body>
<div class="moz-cite-prefix"><br>
</div>
<blockquote type="cite"
cite="mid:MW3PR11MB4651BADD8EFE97CEE76CE4079E020@MW3PR11MB4651.namprd11.prod.outlook.com">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">I have some additional questions regarding the usage of fi_rma,
especifically fi_read:
Firstly: Is it needed to register the target local memory buffer as well
(and if so why, since it is created by the same application)?
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">
This is determined by checking the FI_MR_LOCAL mr_mode bit. If that bit is set, then the provider requires that local data buffers be registered. The reason for the registration is to provide a mapping from the virtual address used by the application to the physical memory address that the hardware will access. Registration pins that mapping, so that the virtual address does not migrate to a different physical address.
The tcp provider won't need this, however, existing RDMA based hardware does. You can always register local buffers; there's just a performance cost.
</pre>
</blockquote>
<p><br>
</p>
<p>Since at the moment I am using sockets and tcp provider, no
receiving memory registration is not needed.</p>
<p>What about verbs provider? I d like to tesk the program via
infiband.</p>
<p><br>
</p>
<blockquote type="cite"
cite="mid:MW3PR11MB4651BADD8EFE97CEE76CE4079E020@MW3PR11MB4651.namprd11.prod.outlook.com">
<blockquote type="cite">
<pre class="moz-quote-pre" wrap="">
Is it the local file descriptor of the locally registered memory buffer
(case 1) or the "local" file descriptor of the remote memory region
(case 2).
So far I get "No such file or directory" from the call. There are 2
connected endpoints, the access key was transfered from read target to
calling part of the application.
Both memory regions are registered.
I tried case 1 and setting the file descriptor to NULL so far. The first
one produced said result, the latter one too except for local testing.
So I assume it is case 2, but wanted to confirm whether there is
something I dont see.
for fi_mr_reg:
No special flags are used
modes are set to FI_READ | FI_REMOTE_READ | FI_WRITE
for fi_read:
a memory buffer is allocated and the length of the remote memory buffer
is known to the calling side (thus the buffer is of sufficient length)
Offset stays at 0 (since all data of the target buffer is wanted)
</pre>
</blockquote>
<pre class="moz-quote-pre" wrap="">
Note that the peer's buffer is identified by the memory key. The peer obtains this by calling fi_mr_key() after registering the memory. Any memory accessed remotely through an RMA operation (read or write) must be registered -- consider it opt-in security permission. </pre>
</blockquote>
<p><br>
</p>
<p>And the memory key is determined by the key it is given at
creation. <br>
</p>
<p>Is it possible another key than the given one is chosen
constantly if a 64 bit key is generated (so close to 0 cahnce of
collision)?<br>
</p>
<blockquote type="cite"
cite="mid:MW3PR11MB4651BADD8EFE97CEE76CE4079E020@MW3PR11MB4651.namprd11.prod.outlook.com">
<pre class="moz-quote-pre" wrap="">The address (addr parameter) passed into fi_read() is the offset into the peer's buffer. The address may either be 0-based (the default), or based on the virtual address that the peer uses to access the memory. In the latter case, the FI_MR_VIRT_ADDR mr_mode bit will be set.</pre>
</blockquote>
<p>I assumed that the mr_modes are set by user interaction. Since I
dont set FI_MR_VIRT_ADDR (in fact none) so far the mr_mode should
be FI_MR_SCALABLE<em>.</em></p>
Thus it should be a 0-based offset (which is set to 0 at my
application).<br>
<blockquote type="cite"
cite="mid:MW3PR11MB4651BADD8EFE97CEE76CE4079E020@MW3PR11MB4651.namprd11.prod.outlook.com">
<pre class="moz-quote-pre" wrap="">
As before, with tcp, it can use a 0-based offset. But RDMA hardware decided that having a base address start at the peer's virtual address made sense to them. So an offset of 0 is indicated by specifying the peer's virtual address associated with the start of the buffer.
</pre>
</blockquote>
<p>In the target system there is no special RDMA hardware I know of.<br>
</p>
<blockquote type="cite"
cite="mid:MW3PR11MB4651BADD8EFE97CEE76CE4079E020@MW3PR11MB4651.namprd11.prod.outlook.com">
<pre class="moz-quote-pre" wrap="">
One way to handle this is for the peer to always exchange a base address with the mr key. If FI_MR_VIRT_ADDR is set, the base address is set to 0. If FI_MR_VIRT_ADDR is 0, the base address should equal the virtual address of the memory buffer. The process that initiates the RMA then just uses the provided key and base address that it was given.
</pre>
</blockquote>
<p>What I can do is:</p>
<p>Verify that the mr_mode is indeed set to FI_MR_SCALABLE or
another mr_mode. <br>
</p>
<p>Send the key returned by the fi_mr_key function and not directly
the generated ones. <em> <br>
</em></p>
<p>Any other idea what I could be doing wrong/check? What additional
information could I provide to help in the search for my
misunderstanding?</p>
<p>At least to me it seems that I am doint the steps right.<br>
</p>
<p><br>
</p>
<p><br>
<em></em></p>
<p>greetings, Arne.<br>
</p>
<p><em><br>
</em></p>
</body>
</html>