[libfabric-users] no perf effect from FI_MORE with verbs; ofi_rxm provider?

[Hefty, Sean]

> Is there a way to tell from the information returned by fi_getinfo() whether one should
> expect any benefit from FI_MORE with the provider one is using?  I don't see anything,

No - this flag is merely an optimization that a provider _may_ act upon.

> and none of the provider man pages seem to mention it, at least in the v1.8.x branch
> I've built from.  I'd prefer to avoid doing the effective equivalent of 'if
> (!usingVerbsProvider) { /* do FI_MORE stuff */ ...}', unless I absolutely have to.
> (I'm writing middleware, a multinode communication implementation for the Chapel
> language runtime, which I want to work over as wide a range of networks and providers
> as possible.)

My recommendation is to write the app to always use FI_MORE where it makes sense, or just always ignore the flag.  There should not be a performance penalty if it is set but ignored by the provider.

> Sure can, although there will be a delay because right now the initiator for my
> fi_writemsg() calls seems to be concluding that all of them have arrived at the target
> before that's actually happened, so I'll have to fix that first.  I don't know if
> that's because I'm mis-counting transactions (doesn't look like it) or have
> misunderstood the completion semantics.  I'm throwing
> FI_DELIVERY_COMPLETE|FI_COMPLETION in my  hints->tx_attr->op_flags, and believe that
> means that once the initiator sees the CQ event from a fi_writemsg() the processor on
> the target side must see the written bytes if it loads from the target memory.  (In
> some cases it's seeing the old memory contents.)

Your understanding looks correct.

This could be a bug in the provider implementing DELIVERY_COMPLETE correctly.  Note that delivery complete isn't supported natively by most hardware, and requires software based acks in order to meet that guarantee.

Are you using RDM endpoints?

- Sean