[libfabric-users] Queue size question
sean.hefty at intel.com
Mon Mar 22 16:08:38 PDT 2021
> I have a test that seems to run fine on tcp;ofi_rxm - though this test is two ranks on
> the same laptop, so it isn't really a very good test - however, I can throw anything at
> it and it seems to reliably complete.
> On GNI, I get lockups and after much head scratching, I am wondering what the
> significance of the tx/rx attribute size may be.
> On tcp/ofi_rxm the size reports as "size: 65536" and I can have 16 threads each sending
> up to 128 messages in flight on one thread per endpoint, and a single receive endpoint
> handling all receives - possibly 16*128 messages with posted receives = 2048.
> When I run on GNI, using two nodes, each reports tx/rx attr "size: 500" - and I find
> that when many messages are in flight, things can lock up because some posted sends are
> never received. This seems to happen even when I drop down to 16 threads with 8 in
> flight messages which ought to be 128 at a time - and I would have suspected that a
> size of 500 (cq size limitation?) would handle this.
> Question 1 - what is the tx/rx attr size really telling me?
Unfortunately, this is provider dependent, and there's very little that can be done to define it crisper without forcing an implementation. In some cases it's related to a HW queue size. I suspect that may be the case with gni. However, the HW queue size doesn't necessarily mean that the number is equal to the number of operations that can be queue. For example, it's possible for a send that requires 2 iovecs to consume 2 entries in the queue. But each operation consuming 1 entry is usually a safe assumption.
Someone familiar with gni will need to chime in on how it maps to their HW.
> Question 2 - if I post more than the allowed receives or sends, should I not receive
> some kind of error? (I have enabled resource management, so I might expect a retry code
> when I attempt the send/recv)
Yes, you should see -FI_EAGAIN when trying to post more operations that the queues support. There are checks like this in some providers -- I think rxm, verbs, and tcp all do, and rxm is actually forgiving about it by allowing queues to overflow. (Because it's easy to swamp a receiver, even with a reasonably well-written app.)
> Ideally, I'd like to throttle the number of messages in flight according to what the
> hardware reports its capabilities - which vars should I use from the fi_info to do
Resource management is the correct setting. Manually limited your application to the tx/rx sizes, and sizing the CQ appropriately should have done the trick.
It sounds like this is a problem likely restricted to gni.
More information about the Libfabric-users