[libfabric-users] fi_read questions

[Hefty, Sean]

> 		I have some additional questions regarding the usage of fi_rma,
> 		especifically fi_read:
> 
> 		Firstly: Is it needed to register the target local memory buffer as well
> 		(and if so why, since it is created by the same application)?
> 
> 
> 	This is determined by checking the FI_MR_LOCAL mr_mode bit.  If that bit is set,
> then the provider requires that local data buffers be registered.  The reason for the
> registration is to provide a mapping from the virtual address used by the application
> to the physical memory address that the hardware will access.  Registration pins that
> mapping, so that the virtual address does not migrate to a different physical address.
> 
> 	The tcp provider won't need this, however, existing RDMA based hardware does.
> You can always register local buffers; there's just a performance cost.
> 
> 
> 
> 
> Since at the moment I am using sockets and tcp provider, no receiving memory
> registration is not needed.
> 
> What about verbs provider? I d like to tesk the program via infiband.

Yes, verbs (the RDMA based hardware I mentioned above) will need this.


> 		Is it the local file descriptor of the locally registered memory buffer
> 		(case 1) or the "local" file descriptor of the remote memory region
> 		(case 2).
> 
> 
> 		So far I get "No such file or directory" from the call. There are 2
> 		connected endpoints, the access key was transfered from read target to
> 		calling part of the application.
> 
> 		Both memory regions are registered.
> 
> 		I tried case 1 and setting the file descriptor to NULL so far. The first
> 		one produced said result, the latter one too except for local testing.
> 
> 		So I assume it is case 2, but wanted to confirm whether there is
> 		something I dont see.
> 
> 
> 		for fi_mr_reg:
> 
> 		No special flags are used
> 
> 		modes are set to FI_READ | FI_REMOTE_READ | FI_WRITE
> 
> 
> 		for fi_read:
> 
> 		a memory buffer is allocated and the length of the remote memory buffer
> 		is known to the calling side (thus the buffer is of sufficient length)
> 
> 		Offset stays at 0 (since all data of the target buffer is wanted)
> 
> 
> 	Note that the peer's buffer is identified by the memory key.  The peer obtains
> this by calling fi_mr_key() after registering the memory.  Any memory accessed remotely
> through an RMA operation (read or write) must be registered -- consider it opt-in
> security permission.
> 
> 
> 
> 
> And the memory key is determined by the key it is given at creation.
> 
> 
> Is it possible another key than the given one is chosen constantly if a 64 bit key is
> generated (so close to 0 cahnce of collision)?

Ah, I didn't mention the key details.  If the FI_MR_PROV_KEY mr_mode bit is set, the provider selects the key.  If FI_MR_PROV_KEY is 0, the app selects the key.  The tcp provider allows the app to select the key, but verbs will select the key.  If the app selects the key, it will either be used or registration will fail.


> 	The address (addr parameter) passed into fi_read() is the offset into the peer's
> buffer.  The address may either be 0-based (the default), or based on the virtual
> address that the peer uses to access the memory.  In the latter case, the
> FI_MR_VIRT_ADDR mr_mode bit will be set.
> 
> I assumed that the mr_modes are set by user interaction. Since I dont set
> FI_MR_VIRT_ADDR (in fact none) so far the mr_mode should be FI_MR_SCALABLE.
> 
> Thus it should be a 0-based offset (which is set to 0 at my application).

The app is supposed to set which mr_mode bits it _can_ support.  The provider will clear the bits that it doesn't require.


> 	As before, with tcp, it can use a 0-based offset.  But RDMA hardware decided that
> having a base address start at the peer's virtual address made sense to them.  So an
> offset of 0 is indicated by specifying the peer's virtual address associated with the
> start of the buffer.
> 
> In the target system there is no special RDMA hardware I know of.
> 
> 
> 
> 	One way to handle this is for the peer to always exchange a base address with the
> mr key.  If FI_MR_VIRT_ADDR is set, the base address is set to 0.  If FI_MR_VIRT_ADDR
> is 0, the base address should equal the virtual address of the memory buffer.  The
> process that initiates the RMA then just uses the provided key and base address that it
> was given.
> 
> What I can do is:
> 
> Verify that the mr_mode is indeed set to FI_MR_SCALABLE or another mr_mode.

If you're using libfabric v1.9 (or anything after 1.5), think of FI_MR_SCALABLE as no longer existing.  :)  Use the individual mr_mode bit flags.


> Send the key returned by the fi_mr_key function and not directly the generated ones.

fi_mr_key() is always usable.  Even if the app specifies the key, it will just return the app's value.


> Any other idea what I could be doing wrong/check? What additional information could I
> provide to help in the search for my misunderstanding?

You mentioned that you were receiving errno ENOENT (no such file or directory).  How were you receiving this?  Error code returned from a call?  As part of a completion?  Which provider were you using?

- Sean