[Ofvwg] OpenFabrics Verbs Working Group meeting minutes 4/21/15

[Gilad Shainer]

Hasan Abbasi (ORNL) presented ADIOS, a system for data staging and asynch IO.

Data staging and IO pipelines may use additional nodes in the system, which are separate from the application compute nodes, to form an IO pipeline between the application and the final storage.

The pipeline may provide various IO post-processing stages, as well as in-memory shared storage between applications.
Internally, data staging makes intensive use of RDMA.

Requirements from the RDMA layer
==============================

Hasan said that ADIOS needs a mechanism to achieve congestion isolation between ADIOS and the application. In addition, prioritization is required between data staging flows and critical low-latency control flows.

Liran mentioned that in InfiniBand, Virtual Lanes (VLs) may provide a good match for both requirements.

Multicast RDMA was raised as a useful feature for distributing data from a source to multiple consumers. However, it is not clear how to manage reliability, as the set of consumers is often dynamic.

Hasan asked how can the application be signaled once data has been fetched by ADIOS? Liran commented that this is not a communication service per-se, since ADIOS manages the communication end-point for IO rather than the applications. In this context, the notion of a "sleepable PollCQ()" was as a useful concept.

The question of indicating progress during very large RDMA data transfers (100s of MBs to GBs) was discusses. One way to tackle this issue is to pipeline huge data transfers as smaller ones. This way, ordered completions provide progress indication during the transfer, while pipelining ensures that these transfers occur without RTT delays. In addition, for such sizes, the CPU overhead is not an issue.

Generic unstructured scatter-gather capabilities is a useful construct for ADIOS. The size of S/G vectors may vary widely depending on the apps, from a few entries to 100s and even 1000s within a single IO transaction.

RDMA to NVRAM will likely be an important building block for ADIOS, since future clusters are envisioned to have a NVRAM at every node.


Regards,
Gilad Shainer
Vice President, Marketing
Mellanox Technologies
350 Oakmead Parkway, Suite 100, Sunnyvale CA, 94085
Office: 408-916-0048, Mobile: 408-421-0048, Fax: 408-585-0348

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.openfabrics.org/pipermail/ofvwg/attachments/20150422/d012a435/attachment.html>