[ofiwg] OFIWG 3/24/2026 Minutes

[Xiong, Jianxin]

3/24/2026

Minutes generated by AI.

** Participants **

Alexander Oganezov (HPE)
Alexia Ingerson (Intel)
Bob Cernohous (Cornelis Networks)
Howard Pritchard (LANL)
Jerome Soumagne (HPE)
Jianxin Xiong (Intel) (Work Group Chair)
Shi Jin (AWS)
Stephen Oost (Intel)
[AWS]Seth Zegelstein
Rajalaxmi (Intel)


** Overview **

The meeting, led by the Working Group Chair with significant contributions from Shi Jin of AWS, focuses on addressing challenges with the FI more flag used in message batching and error handling in network providers. The group identifies problems with dynamic flag setting and infinite retry loops caused by unflushed queues. To resolve these, a new flush API is proposed to allow explicit flushing of pending operations, improving error handling and efficiency.

Shi Jin shares AWS customer experiences highlighting the practical difficulties and discusses RDMA core semantics relevant to the flush operation. The group agrees that the flush API should not return errors directly but rely on asynchronous error reporting mechanisms to avoid application confusion. After considering design options, the group favors implementing the flush as a new endpoint operation member. AWS volunteers to draft the initial pull request, and the chair will document the approach to guide development. The meeting concludes with no further open issues.


** Detailed Summary **

* FI More Flag Challenges

The group discusses the complexities and issues related to the FI more flag, which allows providers to delay processing certain messages for batch handling. Challenges include difficulty in deciding when to set the flag dynamically and handling errors that can cause infinite retry loops if the queue is not flushed properly. The current requirement for providers to flush the queue on errors imposes strict constraints on implementations.

- Working Group Chair explains FI more flag delays processing to batch messages for efficiency.
- Working Group Chair notes error handling issues where unflushed queues cause infinite retry loops.
- Working Group Chair mentions updated requirements for providers to flush queues on errors to mitigate these issues.

* Proposal for New Flush API

A proposal is introduced to create a new flush API that allows applications to explicitly flush all pending operations. This API aims to give applications better control over error handling and improve efficiency by signaling the end of delayed operations without the overhead of sending additional messages. The API is expected to be a local operation with minimal overhead, benefiting both applications and providers.

- Working Group Chair introduces the flush API proposal to explicitly flush pending operations.
- The flush API provides better error handling control to applications.
- The API is designed to be low overhead and beneficial for provider implementations.

* AWS Experience and RDMA Core Semantics

Shi Jin from AWS shares customer experiences where improper handling of the FI more flag caused hangs and bugs, leading to complex workaround logic. He discusses RDMA core semantics, noting differences in how flush and completion operations are implemented and the potential for errors during these operations. AWS currently supports two code paths, one using RDMA core APIs and another bypassing them for direct fabric control, each with different flush implementations.

- Shi Jin (AWS) describes customer issues with FI more flag causing hangs due to improper clearing.
- Shi Jin explains RDMA core semantics and differences in flush and completion operations.
- AWS uses two code paths: one with RDMA core APIs and one bypassing them for direct fabric control.

* Error Handling in Flush Operations

The group examines how errors should be handled and reported by the flush API. There is consensus that the flush operation should be void, with providers responsible for propagating errors asynchronously through completion queues rather than returning errors directly from the flush call. This approach avoids ambiguity for applications and aligns with existing semantics where errors are reported per work request.

- Shi Jin (AWS) suggests flush API should be void with error propagation via completion queues.
- Working Group Chair agrees that direct error returns from flush would confuse applications.
- Consensus is that providers must handle error propagation internally to maintain clarity.

* API Design and Implementation Approach

Two design approaches for the flush API are considered: adding a new member to endpoint operations or introducing a new control operation. The group favors adding the flush as an endpoint operation due to its specificity and simplicity. AWS volunteers to draft a pull request based on this approach, and the working group chair agrees to document the approach details in the issue tracker to facilitate development.

- Working Group Chair presents two design options for flush API implementation.
- Group agrees to add flush as a new endpoint operation member for simplicity.
- Shi Jin (AWS) volunteers to create a pull request for the proposal.
- Working Group Chair commits to documenting approach details in the issue tracker.

Jianxin Xiong
Fabric Software
Intel Corporation
Jianxin.xiong at intel.com