[ofiwg] DS/DA Re-sending Chet Douglas' use case slides from a few weeks ago (EOM)

[Atchley, Scott]

Hi Paul,

I apologize for missing most of the NVM_usage discussion and joining late. Hopefully, I was not too disruptive.

Looking through your NVM_usage slides this morning, I have a few comments.

* Slide 6

I imagine some vendors may provide block access on top of NVDIMMs to allow traditional file systems to use these devices even though the devices natively support byte-addressable (i.e. load/store) semantics.

I do not know if vendors will provide byte-addressable access to NVMe devices (without doing the equivalent of mapping the file), although we and everyone else would like that.

* Slide 7

NVMe-over-Fabrics (NVMeOF) provides the block access of remote storage. It provides similar functionality to SRP, iSER, iSCSI, AOE, etc. The difference, I believe, is that with proper NIC support, the target is not involved in the I/O path at all.

* Slide 8

This slide represents NAS only including NFS. Note, the file server can use locally attached NVMe or remotely attached NVMeOF (or SRP, iSER, iSCSI, etc.) block devices.

Parallel (or distributed) file systems (PFS) such as Lustre, GPFS, Ceph, etc. are object stores with  metadata servers to handle the POSIX interface. All metadata updates go only to the metadata server and only object requests go directly to the object servers. The object server case is identical to slide 7 expect that the protocols between the server and the client are proprietary to the PFS (Lustre’s LNET, GPFS’ NSD protocol, Ceph’s RADOS protocol, etc.). As with NAS servers, the object server can use locally attached NVMe or remote attached NVMeOF (or SRP, iSER, iSCSI, etc.) block devices. Each of the object servers (except possibly GPFS) creates a traditional file system on the block device (Lustre uses a modified ext4, Ceph can use ext4, xfs, btrfs) to manage the objects (although Ceph is discussion ditching the local file system in order to manage the blocks directly).

To beat a dead horse, a Lustre client _never_ talks to a Object Storage Target (OST) directly. The client talks to an Object Storage Server (OSS) using LNET. The OSS manages multiple block devices that may be locally attached or remotely attached using any of the protocols above.

* Slide 9

This is wide open. Vendors are trying multiple things in this area (Key/Value stores, object stores, high-performance caches, etc.). The protocols are all custom and may use sockets (over Ethernet, IPoIB, etc.), OFA Verbs (over IB, RoCE), or other transports.

Scott

On Oct 27, 2015, at 10:58 AM, Paul Grun <grun at cray.com> wrote:

> And including a skeleton slide deck on summarizing use cases.  We can discuss this today.  CAUTION!  These are Work In Progress, meaning that the drawings are not currently correct!
>  
> From: ofiwg-bounces at lists.openfabrics.org [mailto:ofiwg-bounces at lists.openfabrics.org] On Behalf Of Paul Grun
> Sent: Tuesday, October 27, 2015 7:49 AM
> To: ofiwg at lists.openfabrics.org
> Subject: [ofiwg] DS/DA Re-sending Chet Douglas' use case slides from a few weeks ago (EOM)
>  
>  
>  
> Cray Inc.
> Office:    (503) 620-8757
> Mobile:  (503) 703-5382
>  
> <NVM_usage_2015_1026.pptx>_______________________________________________
> ofiwg mailing list
> ofiwg at lists.openfabrics.org
> http://lists.openfabrics.org/mailman/listinfo/ofiwg