NVMe Introduction

Introduction to Non-Volatile Memory Express (NVMe) Interface for Beginners.

IDC, Worldwide Solid State Drive Forecast Update, 2020-2024, Doc #US4590920, December 2020

This article serves as a beginner’s guide to the Non-Volatile Memory Express (NVMe) interface. A significant portion of the content is carefully selected from various sources on the internet, as I’m relatively new to this field. Nonetheless, the article includes thorough discussions and emphasizes key points to offer an enhanced learning path for newcomers.

Learning Resources

• NVMe Official Specification and Education resources.
• Flash Memory Summit Proceedings.
• An introduction to NVMe from Seagate.
• SNIA.

Conventions

• NVME-MI: NVM Express Management Interface

The NVM Express Management Interface is the command set for in-band and out-of-band management of NVMe storage systems. These management functions include, but are not limited to, discovering, monitoring, and updating firmware on NVMe devices. NVMe-MI provides an industry-standard way to manage NVMe drives and devices.

• NVME-oF: NVM Express over Fabric

The NVMe-oF protocol enables NVMe commands to be transmitted over RDMA, Fibre Channel, and TCP. NVMe-oF extends the NVMe deployment from local host to remote host(s) for a scale-out NVMe storage system.

Specification

NVMe Roadmap from this presentation

NVMe Specification Family

NVMe Specification Family in Group

NVM Express (NVMe) is a protocol for the transport of data over different me- dia and for optimized storage in NAND flash. Peripheral Component Intercon- nect Express (PCIe) is currently the most used transport medium. Other media, like NVMe over Fabrics, are currently being standardized. NVMe is optimized for NAND flash chips. The protocol provides a high-bandwidth and low-latency framework to the storage protocol, but with flash-specific improvements.

It could be seen that NVMe plays as a higher level protocol, which is usually implemented in software. The PCIe is mainly utilized as a transmission layer protocol, where the physical connector could be in M.2, SATA Express or PCIe form.

A more direct illustration is drawn below.

A basic Memory-Based Transport Model is shown below. Queue Pair of NVMe

For a description of the queueing interface and command arbitration details, refer to the introduction (early 2013 version) available here.

Command Set

Command set is a feature that NVMe evolves with the need changes of storage.

The individual command set specifications allow NVM Express to isolate and independently evolve command sets for emerging technologies such as Zoned Namespaces, Key Value and eventually computational storage. The NVMe specifications before NVMe 2.0 included all command set details. NVMe 2.0 separates these command sets into different specifications.

NVMe 2.0 introduced two additional command sets in addition to those in NVMe 1.x. For more information, you can refer to this presentation.

Key-Value Command Set

Block Storage

• Data stored in blocks of a fixed size
• Data addressed by a Logical Block Address (LBA)
• LBA is a fixed number of bytes
• Storage space allocated in integer multiples of block size
• Logical Blocks are associated 1to1 with physical blocks

KV Storage

• Data is stored as unstructured data
• Data addressed by a key
• Key is variable length
• Storage space is allocated in increments of bytes
• Value is associated with amount of physical storage necessary

Zoned Namespace Command Set

By dividing an NVMe namespace into zones, which are required to be sequentially written, ZNS offers essential benefits to hyper-scale organizations, all-flash array vendors and large storage-system vendors to take advantage of storage devices optimized for sequential write workloads. ZNS reduces device-side write amplification, over-provisioning and DRAM while improving tail latency, throughput and drive capacity.

Summary

A minimum set of only 13 commands, as listed in Table 1, could achieve the NVMe interface.

Storage Model

The NVM storage model includes the following entities:

• NVM subsystems
• Domains
• Endurance Groups
• NVM Sets
• Namespaces

Further Reading

• NVME device libraries for multi-platform (xNVME) and Linux (NVME Cli). Full knowledge about other drivers could be found here.
• MQSim: A Simulator for Modern NVMe and SATA SSDs
• Open-source Key-Value SSD Driver
• Low-level NVM Express Mastery Galore