[ofiwg] input on intra-node implementation

[Jeff Squyres (jsquyres)]

On Feb 9, 2016, at 3:54 PM, Hefty, Sean <sean.hefty at intel.com> wrote:
> 
> I want to provide an intra-node communication (i.e. loopback) utility to libfabric.  The loopback utility could be part of a stand-alone provider, or incorporated into other providers.  For this, I'm looking at selecting a single, easily maintained implementation.  These are my choices so far:
> 
> 1. Control messages transfer over shared memory
>   Data transfers use shared memory bounce buffers
> 2. Control messages transfer over shared memory
>   Data transfers occur over CMA
>   (small transfers go with control messages)
> 3. Use XPMEM in some TBD way

FWIW, I think you and I chatted about some shared memory pitfalls here in a separate (off list?) email thread: there is trickiness in terms of ensuring that shared memory is cleaned up if processes die abnormally.  We just need to make sure to handle those kinds of cases properly.

> Some of these options are only available on Linux.  Does the portability of this solution matter?  FreeBSD and Solaris would fall back to using the network loopback device.

FWIW: MPI implementations have typically used pure shared memory for portable solutions (i.e., copy in/copy out), and used OS/hardware-specific shared memory mechanisms where relevant.  More specifically:

1. most control messages go over portable shared memory (i.e., copy in/copy out): mmap, POSIX, etc.
2. data messages can go over portable shared memory or some other OS-specific one-copy shared memory functionality (e.g., Linux CMA)

So I think it's fine to use Linux-specific technologies for #2 (e.g., CMA).  If someone else wants to contribute+maintain some non-Linux-OS shared memory technologies, that would be great.

> How much concern needs to be given to security?  

Again in the FWIW category: MPI hasn't worried too much about this because a single MPI job is typically given reign over an entire server.  This may not be true for all libfabric scenarios.

That being said, Linux CMA probably gives us enough of a process-to-process security model...?

> Should the loopback utility enforce RMA registrations?  

You can easily argue both ways on this:

1. Memory registration is a PITA for the developer.  It's a LOT easier to just say "yes, you can write anywhere in process X's address space."  Opening up the entire process also eliminates all the management of registrations.

2. ...but then you can write anywhere in process X's address space -- that's super-scary (and error/bug-prone)!

> Do we require processes to share a certain level of access, such as ptrace capability?

I don't have much of an opinion here.

-- 
Jeff Squyres
jsquyres at cisco.com
For corporate legal information go to: http://www.cisco.com/web/about/doing_business/legal/cri/