[libfabric-users] no perf effect from FI_MORE with verbs; ofi_rxm provider?

[Titus, Greg]

> I'm trying out FI_MORE with the verbs;ofi_rxm provider, but I'm not seeing any
> performance benefit from it.  ...

The verbs provider ignores the FI_MORE flag.  With FI_MORE, you would lose overlapping the transfer of the first write with the posting of the next, with a potential gain of limiting PCI transactions.  The GNI and EFA providers support FI_MORE, and might have details of the benefits they see.

Is there a way to tell from the information returned by fi_getinfo() whether one should expect any benefit from FI_MORE with the provider one is using?  I don't see anything, and none of the provider man pages seem to mention it, at least in the v1.8.x branch I've built from.  I'd prefer to avoid doing the effective equivalent of 'if (!usingVerbsProvider) { /* do FI_MORE stuff */ ...}', unless I absolutely have to.  (I'm writing middleware, a multinode communication implementation for the Chapel language runtime, which I want to work over as wide a range of networks and providers as possible.)

I would expect to see a benefit batching writes, and waiting for completions at the end, versus delaying each write until the previous one completed.  If you're not seeing this, can you try decreasing the size of the writes and see if the results are unchanged?
Sure can, although there will be a delay because right now the initiator for my fi_writemsg() calls seems to be concluding that all of them have arrived at the target before that's actually happened, so I'll have to fix that first.  I don't know if that's because I'm mis-counting transactions (doesn't look like it) or have misunderstood the completion semantics.  I'm throwing FI_DELIVERY_COMPLETE|FI_COMPLETION in my  hints->tx_attr->op_flags, and believe that means that once the initiator sees the CQ event from a fi_writemsg() the processor on the target side must see the written bytes if it loads from the target memory.  (In some cases it's seeing the old memory contents.)

greg

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/libfabric-users/attachments/20200129/10bc7352/attachment.htm>