[ofiwg] the problems with atomics

[Hefty, Sean]

> > FI_LONG_DOUBLE_80BITS_ALIGNED_96,
> > FI_LONG_DOUBLE_80BITS_ALIGNED_128
> > FI_LONG_DOUBLE_96BITS_ALIGNED_96,
> > FI_LONG_DOUBLE_96BITS_ALIGNED_128,
> > FI_LONG_DOUBLE_128BITS_ALIGNED_128,
> > FI_LONG_DOUBLE_WEIRD_PPC_ALIGNED_128
> 
> You expect app writers to provide this value? That sounds very hard.

Long double seems ridiculous.  But from what I can tell, 2 apps on the same system could in theory have different alignments based on how they were compiled.  I don't want what I listed above as a solution at all, but I do need something.  At the moment, what I need to fix is the lack of any specific definition for 'FI_LONG_DOUBLE', which was added under the assumption that it had a sane IEEE definition...

Maybe the best option is to just define the equivalent of:

FLOAT32/64/96/128

And map DOUBLE -> FLOAT64, and LONG_DOUBLE -> FLOAT128

Figuring out if FLOAT96 is implemented using 80 bits would be left as an exercise to the app.

> There is no implicit conversion when doing a RDMA READ/WRITE or SEND,
> so the app is going to be exposed to all of the differences if it
> wants to run homogeneously.

RDMA and sends only deal with bytes.  They don't attempt to interpret data at the remote side.

> I'd probably also say that the integer atomic types should have the
> same endianness as the local machine (why should ATOMIC X vs RDMA READ
> return different answers?)

Even with RDMA, an app should care about endianness.  E.g. rsockets exchanges endianness format as part of its protocol, so that RDMA writes end up in a format that the remote side can read.  I'm guessing most apps ignore this.

> Generally speaking, changing the alignment and size breaks all memory
> layouts, I'd be amazed if a MPI or PGAS app could run across hardware
> with different FPU and different integer representations.

I agree, but that doesn't mean that I think the libfabric API should be broken.  Apps should be able to run between different CPU architectures, though it will come with a cost. 

- Sean