[libfabric-users] Is this a reasonable verbs; ofi_rxm warm-up time on an IB-based Cray CS system?
gregory.titus at hpe.com
Wed Jun 10 09:52:29 PDT 2020
I'm seeing what looks like a significant "warm-up" time for a concentrated-load stress test on an IB-based Cray CS system using the 'verbs;ofi_rxm' provider, and am wondering if it's expected/reasonable. The test does fi_send() operations from each of 36 threads on 14 nodes, with each transmitting thread having its own tx endpoint. The sends are all directed at a single rx endpoint on a 15th node, where they land in a multi-receive buffer and a single thread consumes them. I don't normally set any FI_* environment variables when running the test, though see the last paragraph below for an exception.
I've recently discovered that it takes a fair amount of wall time, around 4 seconds, to get through the first fi_send() from all the source threads. I'm assuming this is due to RxM dynamic endpoint connection, perhaps exacerbated by contention due to the concentrated load at the single target endpoint. Under the circumstances I've described here, is 4s reasonable for dynamically establishing 504 connections?
There may be some other overheads present as well -- as I raise the value of FI_OFI_RXM_MSG_RX_SIZE from its default of 128 to higher values I can eventually see (around 1024 or so) the warm-up time start to become proportional to my specified msg_rx_size value. The implication is that some aspect of the endpoint creation is space-dependent. That said, reducing msg_rx_size below the default 128 doesn't correspondingly reduce the time needed for the first fi_send()s below 4s, so my guess is that at smaller values like that such space-related overhead, if there is any, is not significant.
-------------- next part --------------
An HTML attachment was scrubbed...
More information about the Libfabric-users