[ofiwg] wait sets vs poll sets vs application

[Dardo D Kleiner - CONTRACTOR]

Hey folks!  Finally looking in depth at porting my verbs code to libfabric.

I have a general topic I thought I'd post here, and a couple more specific issues I'll open on github.

So, my verbs SOP makes extensive use of the underlying fd in rdma_event_channel and ibv_comp_channel since I usually have some other things I need to watch and to avoid spinning (at the possible expense of latency), blocking, and/or multithreading constructs.

Clearly, I can use the "native operating system constructs for applications to use when waiting for events" described in the OFI introduction document and that would give me pretty much the same as my current implementation.

My interest was piqued by the wait and poll set abstractions, however - as possible provider-specific optimizations can be leveraged and also *perhaps* towards my holy-grail of a cross-platform event interface (thinking of e.g. a NetworkDirect provider and Windows IOCompletion stuff)

I'm thinking an application might first check for provider support at the fabric level for wait sets, then at the domain for poll sets, and finally fall back to queue-specific wait objects.

However, the subtle differences in the interfaces seems to make it very difficult to achieve kind of a consistent usage approach.  In particular:

1) wait sets don't return context
2) poll sets don't support timeouts
3) poll sets don't support access to their underlying "wait" object

So my question is, is the design of these object interfaces intentional in this regard?  Should apps just pick one approach and work with what's there?

I also went ahead and put epoll-based wait and poll implementations into my fork of the verbs provider and it seems of value (disregarding issues above) - though it doesn't really make sense there as its not verbs specific.  Perhaps any provider that uses fd's for internal waits can leverage a common epoll implementation?

Open to thoughts and suggestions...

- Dardo

PS - Good fun last week in Monterey!