Database / RocksDB Basics Interview Questions

RocksDB is a high-performance, embedded key-value store, a storage engine library that applications link directly into their own process, developed by Facebook (Meta) and optimized for fast storage like SSDs and RAM.

• Written in C++, and built on a log-structured merge-tree (LSM-tree)
• Runs inside the calling application's process rather than as a separate database server
• Optimized for high write throughput while keeping reads reasonably efficient

Because it's embeddable rather than a standalone server, RocksDB is commonly used as the storage layer inside larger databases and systems, rather than being the end-user-facing database itself.

RocksDB was developed by Facebook, now Meta, as an evolution of Google's earlier LevelDB project.

• Originally forked from LevelDB and heavily extended to meet Facebook's own performance and scale requirements
• Released as an open-source project, widely adopted well beyond its original creator
• Actively maintained and used as a foundational storage engine inside many other well-known systems

Its origin as an internal Facebook project explains much of its design focus, handling extremely high write throughput at scale on fast storage hardware.

RocksDB is based on LevelDB, an earlier embedded key-value store originally created by Google.

• Kept LevelDB's core design: a log-structured merge-tree with memtables and SSTables
• Added significant extensions, including column families, additional compaction styles, backup and checkpoint support, and deep performance tuning options
• Rewrote much of the internals for better performance on modern, fast storage devices like SSDs

RocksDB can be thought of as LevelDB's more feature-rich, heavily optimized successor, rather than a completely unrelated design.

A key-value store is a type of database that stores data as simple pairs, a unique key mapped to an associated value, without the fixed rows, columns, or schema of a relational database.

• Supports basic operations like storing a value under a key, retrieving a value by its key, and deleting a key
• Values are typically treated as opaque byte sequences, the store itself doesn't need to understand their internal structure
• Trades the rich querying of relational systems for simplicity and speed on straightforward lookups

RocksDB is this kind of store, and its simple interface is exactly what makes it a convenient, flexible building block for other systems to build more complex data models on top of.

A Log-Structured Merge-Tree is a data structure designed to make writes fast by turning random writes into sequential ones, at the cost of some added complexity on the read path.

• New writes go into an in-memory structure first, rather than being written directly to their final location on disk
• That in-memory data is periodically flushed to disk as an immutable, sorted file
• A background compaction process merges these files over time to keep the overall structure efficient to read from

This design is what lets RocksDB and similar systems sustain very high write throughput, since sequential writes are dramatically faster than scattered random writes on most storage hardware.

A MemTable is RocksDB's in-memory buffer that absorbs all new writes before they're ever touched by disk.

• Every Put, Delete, or Merge operation is applied to the active MemTable first
• Keeps data in sorted order, so it can be flushed directly into a sorted file later
• Has a configurable maximum size, with a default around 64 megabytes, after which it's swapped out for a fresh one

Because writes only need to touch memory to be considered durable-in-progress, once combined with the write-ahead log, the MemTable is central to why RocksDB can sustain such high write throughput.

RocksDB's default MemTable implementation uses a skip list to keep incoming writes ordered by key.

• A skip list supports efficient, roughly logarithmic-time insertion and lookup, similar in spirit to a balanced tree
• Keeping the MemTable sorted means it can be flushed directly into an already-sorted SSTable with no extra sorting step needed
• RocksDB is also pluggable here, alternative memtable implementations exist for specific workload patterns

The skip list's combination of simplicity and efficient ordered access is a big part of why it was chosen as RocksDB's default in-memory structure.

The Write-Ahead Log is a sequential, on-disk log that RocksDB appends every write to, in addition to applying it to the in-memory MemTable.

• Provides durability, if the process crashes before a MemTable is flushed to disk, the WAL can be replayed to recover those writes
• Written sequentially, which is fast even on spinning disks, unlike the random writes a naive design might require
• Can optionally be disabled for extra write speed, at the cost of losing durability guarantees on crash

The WAL is what lets RocksDB keep writes primarily in fast memory while still guaranteeing they won't simply vanish if the process crashes unexpectedly.

An SSTable, or Sorted String Table, is the immutable, on-disk file format RocksDB uses to store key-value pairs once they're flushed out of memory.

• Stores keys in sorted order, which enables efficient binary search and range scans
• Once written, an SSTable is never modified in place, only replaced or removed through compaction
• Typically includes an index block and, optionally, a Bloom filter to speed up lookups

This immutability is a deliberate design choice, it simplifies concurrency and makes it straightforward to cache and share these files safely across reads.

Flushing is the process of writing a full, immutable MemTable's contents out to disk as a new SSTable.

• Triggered once the active MemTable reaches its configured size limit
• The full MemTable is marked immutable, and a new, empty MemTable takes over for future writes
• A background thread performs the actual write to disk, so it doesn't block new writes from continuing to arrive

This is the step that moves data from RocksDB's fast, memory-resident write path into its durable, disk-resident storage as a Level-0 SSTable.

Compaction is the background process that merges multiple SSTables together, removing outdated or deleted data and keeping the overall storage structure efficient to read from.

• Combines SSTables with overlapping key ranges, producing a smaller number of more organized files
• Physically removes data that was overwritten or explicitly deleted, reclaiming disk space
• Runs automatically in the background, without requiring the application to trigger it manually

Without compaction, the number of SSTables would keep growing indefinitely, and reads would get progressively slower as more and more files needed to be checked for a given key.

RocksDB supports a few different strategies for how it merges SSTables together over time.

Leveled compaction is RocksDB's default and most commonly used style, generally offering the best balance of read performance and space efficiency for most workloads.

Leveled compaction organizes SSTables into a series of levels, L0 through Ln, where each level is typically several times larger than the one above it.

• Level 0 holds SSTables flushed directly from MemTables, and their key ranges may overlap
• From Level 1 onward, SSTables within a level cover non-overlapping key ranges, kept strictly sorted
• When a level exceeds its size limit, some of its SSTables are merged into the level below, gradually pushing data downward over time

This structure keeps reads efficient, since only a bounded number of SSTables per level need to be checked, at the cost of some write amplification from the repeated merging.

Universal compaction, sometimes called tiered compaction, keeps SSTables organized by age rather than by strict, non-overlapping key ranges across levels.

• New SSTables are added to a sorted run, and periodically a group of runs gets merged together into a larger one
• Tends to produce lower write amplification than leveled compaction, since data is rewritten less often
• Trades that write efficiency for higher space amplification and potentially more SSTables to check on a read

This style is often chosen for extremely write-heavy workloads where minimizing the total amount of data rewritten matters more than optimal read performance.

A Bloom filter is a compact, probabilistic data structure attached to each SSTable that can quickly tell whether a key is definitely not present, without needing to actually read the file.

• Can produce false positives, occasionally saying a key might be present when it isn't, but never false negatives
• Lets RocksDB skip reading SSTables entirely when the filter confirms a key definitely isn't there
• Dramatically reduces unnecessary disk reads, especially important when a key lookup would otherwise have to check many SSTables

Bloom filters are one of the main reasons RocksDB's read performance stays reasonable despite data being spread across many separate, immutable files.

An Index Block is a section within an SSTable that maps key ranges to the specific data block containing them, letting RocksDB jump directly to the right location instead of scanning the whole file.

• Acts like a table of contents for the data blocks inside that SSTable
• Loaded into memory, often cached, so lookups within it are fast
• Used together with a Bloom filter, the filter first rules out files that definitely don't contain the key, then the index block locates exactly where to look within a file that might

Without an index block, finding a specific key inside an SSTable would require scanning through data sequentially rather than jumping straight to the relevant section.

The Block Cache is an in-memory cache that holds recently accessed data and index blocks from SSTables, reducing how often RocksDB needs to read from disk.

• Frequently accessed, or "hot," blocks stay in memory rather than being re-read from disk on every lookup
• Configurable in size, letting operators trade memory usage against read performance
• Shared across the database instance, so blocks from different SSTables can all benefit from the same cache

For read-heavy workloads, a well-sized block cache can be one of the single biggest levers for improving overall read latency.

The Manifest is a metadata file that tracks the current set of SSTables and their organization into levels, essentially recording the database's structural state over time.

• Records every change to which SSTables exist and which level they belong to
• Used during startup to reconstruct exactly what the database's on-disk structure looked like
• Critical for consistency, since it's the authoritative record of the database's actual layout, independent of the individual SSTable files themselves

If the Manifest were lost or corrupted, RocksDB would have no reliable way to know which SSTables belong to the current, valid version of the database.

Column Families let a single RocksDB database instance maintain multiple, independently configured, logically separate key spaces within the same database.

• Each column family can have its own compaction style, comparator, and other tuning options
• All column families in a database share the same underlying write-ahead log, keeping writes across them consistent
• Commonly used to store logically distinct kinds of data, like an index versus the actual records, within a single database instance

This feature is what lets an application avoid running multiple separate RocksDB instances just to keep different categories of data configured and organized differently.

A Comparator defines how RocksDB orders keys, determining what "sorted" actually means for a given database or column family.

• The default comparator sorts keys as raw bytes, in straightforward lexicographic order
• Applications can supply a custom comparator to sort keys differently, for example treating a numeric portion of the key numerically rather than as raw bytes
• Once a database is created with a particular comparator, that choice generally can't be changed without rebuilding the database

Because so much of RocksDB's internal structure, from MemTables to SSTables, depends on maintaining sorted order, the comparator is a foundational, largely fixed choice made early in a database's life.

A Snapshot is a consistent, read-only view of the database's state at a specific point in time, unaffected by writes that happen after it was taken.

• Lets a read operation see the database exactly as it looked at snapshot creation, even while other writes continue concurrently
• Doesn't physically copy any data, it's implemented efficiently using RocksDB's existing versioned, immutable SSTable structure
• Commonly used for things like consistent backups or long-running reads that shouldn't be affected by concurrent writes

Because SSTables are already immutable, RocksDB can offer this kind of point-in-time view relatively cheaply, without needing to duplicate the underlying data.

An Iterator lets an application traverse a range of keys in sorted order, rather than looking up one key at a time.

• Can be positioned at the beginning, at a specific key via a seek operation, or moved forward and backward through the sorted keyspace
• Transparently merges data from the MemTable and multiple SSTables into one logically sorted sequence
• Can be created against a Snapshot, giving a consistent, ordered view of the data as it existed at that point in time

Iterators are RocksDB's primary tool for range scans, listing all keys with a given prefix, or walking through data in sorted order, tasks a simple key lookup alone can't handle.

Put is the core write operation in RocksDB, used to insert a new key-value pair or overwrite the value of an existing key.

• Writes go to both the active MemTable in memory and the write-ahead log on disk
• If the key already exists, its previous value is logically superseded by the new one, though the old value may still physically remain in older SSTables until compaction removes it
• Considered committed once it's durably recorded in the write-ahead log, unless WAL writes have been explicitly disabled

Put is the fundamental building block nearly every other higher-level operation in an application built on RocksDB ultimately relies on.

Get is the core read operation in RocksDB, used to retrieve the value currently associated with a specific key.

• Checks the active MemTable first, then any immutable MemTables waiting to be flushed, then SSTables from Level 0 downward
• Uses Bloom filters and index blocks along the way to avoid unnecessary reads from files that don't contain the key
• Returns not found if the key doesn't exist, or if its most recent operation was a delete

This read path, checking progressively older layers of data until the key is found or ruled out, is exactly what makes an LSM-tree's write speed possible without sacrificing correctness on reads.

Delete marks a key as removed, rather than immediately erasing any existing data for that key from disk.

• Writes a special marker, often called a tombstone, associated with that key, rather than searching for and physically removing prior values right away
• The tombstone is what later reads use to know the key should be treated as absent
• The actual underlying data is only physically removed later, during compaction

This deferred approach fits the LSM-tree's append-only write philosophy, deletion becomes just another kind of write, rather than requiring an immediate, potentially expensive search-and-remove operation.

A Tombstone is the marker RocksDB writes to record that a key has been deleted, without immediately removing any of that key's prior data from disk.

• Behaves like any other write, it goes through the MemTable and WAL just like a Put would
• Causes reads for that key to correctly report it as not found, even though older SSTables might still physically contain earlier values
• Gets physically removed, along with the data it was masking, once compaction processes the SSTables containing it

If too many tombstones accumulate without being compacted away, they can noticeably slow down reads, since the read path has to skip past them to determine a key's true current state.

Merge is a special write operation that lets an application apply an incremental update to a key's value without first reading its current value.

• Common example: incrementing a counter, where you want to add a value rather than overwrite the whole thing
• Requires the application to supply a Merge Operator, custom logic defining how multiple pending merge operations should eventually be combined into a single value
• RocksDB defers actually applying the merge logic until a read happens or compaction processes the relevant data, rather than computing it immediately on every write

This operation avoids the classic read-modify-write pattern's overhead and potential race conditions, letting an application record "add 1" as its own lightweight write instead.

Write amplification is the ratio between the total amount of data actually written to physical storage and the amount of data the application logically intended to write.

• Arises because a single logical write can end up being rewritten multiple times as it moves through compaction across levels
• Higher write amplification means more wear on storage hardware and more background I/O competing with the application's own workload
• Different compaction styles trade write amplification against read and space amplification differently

This is one of the three core trade-offs, alongside read and space amplification, that shapes almost every tuning decision made when configuring an LSM-tree-based store like RocksDB.

Read amplification is the ratio between the amount of data actually read from storage to answer a query and the amount of data logically needed to answer it.

• Happens because a single key lookup may need to check the MemTable and multiple SSTables across several levels before finding, or ruling out, that key
• Reduced significantly by Bloom filters and index blocks, which let RocksDB skip files that clearly don't contain the key
• Tends to be higher under compaction styles, like universal compaction, that favor write throughput over keeping data tightly organized

Read amplification is the primary cost an LSM-tree design accepts in exchange for its much faster write path compared to structures like a B-tree.

Space amplification is the ratio between the actual disk space RocksDB's data files consume and the true logical size of the data currently stored.

• Occurs because outdated versions of a key, and tombstones marking deletions, can persist on disk until compaction removes them
• Compaction styles that delay merging more aggressively, like universal compaction, tend to have higher space amplification while that data waits to be cleaned up
• Can be reduced through more frequent or more aggressive compaction, at the cost of increased write amplification

Like the other two amplification metrics, this reflects a genuine trade-off rather than something that can simply be eliminated, tuning RocksDB means choosing which of these costs matters most for a given workload.

RocksDB's leveled compaction organizes SSTables into a numbered hierarchy, from Level 0 up to a configurable maximum level, with each level generally holding more data than the one above it.

• Level 0 holds SSTables freshly flushed from MemTables, and their key ranges can overlap each other
• Level 1 and deeper levels hold SSTables with non-overlapping key ranges within that level, kept strictly sorted
• Data gradually moves from higher, smaller levels down to lower, larger levels through repeated compaction over time

This structure bounds how many files a read might need to check, since at most one SSTable per level 1-and-below needs inspecting for a given key, keeping lookups efficient even as total data volume grows.

1. An application calls Put, Delete, or Merge with a key and, if applicable, a value
2. The write is appended to the Write-Ahead Log on disk, for durability
3. The write is also applied to the active MemTable in memory
4. Once the MemTable reaches its size limit, it's marked immutable, and a new MemTable takes over for future writes
5. A background thread flushes the immutable MemTable to disk as a new Level-0 SSTable
6. Background compaction later merges that SSTable with others as part of RocksDB's ongoing housekeeping

Because the application-facing part of this path, steps one through three, only touches memory and a sequential log, writes can complete quickly without waiting on the slower, more complex compaction process happening in the background.

1. A Get call for a key first checks the active MemTable, since that holds the most recent writes
2. If not found there, it checks any immutable MemTables still waiting to be flushed
3. It then checks SSTables starting at Level 0, using each file's Bloom filter to quickly skip files that definitely don't contain the key
4. For files the Bloom filter can't rule out, the index block locates the right data block, and a binary search finds the exact entry
5. This continues down through successive levels until the key is found, or every relevant level has been checked and the key is reported as not found

This layered search, from freshest to oldest data, is exactly why a single Get can, in the worst case, touch several different structures before returning an answer.

An Immutable MemTable is a MemTable that has reached its size limit and stopped accepting new writes, but hasn't yet been flushed to disk as an SSTable.

• Created the moment an active MemTable fills up, at which point a brand new, empty MemTable takes over for future writes
• Still readable, so Get calls and iterators continue to check it until it's flushed
• Removed once a background thread finishes flushing its contents to a new Level-0 SSTable

This intermediate state is what lets RocksDB keep accepting new writes into a fresh MemTable without pausing to wait for the previous one to finish being written to disk.

A Skip List is a probabilistic data structure that maintains sorted data with multiple layers of "express lane" links, letting it support fast search, insertion, and deletion without the complexity of a balanced tree.

• The bottom layer contains every element in sorted order, like a regular linked list
• Higher layers contain progressively fewer elements, acting as shortcuts that let a search skip over large portions of the list
• Achieves roughly logarithmic-time operations on average, comparable to a balanced tree, but with a notably simpler implementation

This is the data structure RocksDB's default MemTable implementation is built on, chosen for its efficient ordered access combined with straightforward concurrent-access properties.

RocksDB is written in C++, and its core is distributed as a native library that other software links directly into its own process.

• Being embeddable in C++ is what makes it usable as an internal storage engine inside other systems, rather than a standalone server application
• Official bindings and wrappers exist for other languages, including Java, allowing non-C++ applications to use RocksDB as well
• Its C++ foundation is part of why it's well suited to performance-critical, resource-constrained environments

Choosing C++ gave RocksDB's developers fine-grained control over memory management and performance, both essential for a storage engine expected to handle extremely high throughput.

RocksDB is commonly used as the underlying storage engine inside larger, higher-level database and infrastructure systems, rather than being the end-user-facing database itself.

• Distributed SQL and NoSQL databases that need a fast, reliable local storage layer on each node
• Stream processing and messaging systems that need durable, high-throughput local storage
• Caching layers and other infrastructure components that benefit from an embeddable, high-performance key-value store

This pattern, building a distributed or higher-level system on top of an embedded storage engine like RocksDB, lets those systems focus their own development effort on distribution, querying, or protocol logic instead of reinventing a low-level storage engine from scratch.

RocksDB offers transaction support that lets multiple operations be grouped so they either all succeed together or none of them take effect, along with mechanisms to handle conflicting concurrent writes.

• Supports both optimistic and pessimistic concurrency control, letting an application choose the conflict-handling strategy that fits its workload
• Optimistic transactions check for conflicts at commit time, assuming conflicts are rare
• Pessimistic transactions instead lock keys upfront, preventing conflicting writes from happening concurrently in the first place

This transactional layer is what lets applications built on RocksDB implement multi-key atomic updates, rather than being limited to single-key operations with no consistency guarantees across them.

RocksDB includes built-in utilities for backing up a database's data to a separate location and later restoring it, without needing to write this logic from scratch.

• Backups can be taken incrementally, reusing unchanged SSTable files rather than copying the entire database every time
• Because SSTables are immutable, unchanged files can safely be shared between successive backups instead of being duplicated
• Restoring reconstructs a working database directory from a chosen backup, ready to be reopened

This built-in support saves application developers from having to reimplement careful, consistent backup logic themselves on top of RocksDB's file-based storage.

A Checkpoint creates a lightweight, point-in-time copy of a RocksDB database, typically by hard-linking its current SSTable files rather than physically copying them.

• Extremely fast to create, since it mostly avoids duplicating the underlying immutable data files
• Produces a fully independent, openable database directory, distinct from ongoing backup utilities designed for longer-term storage
• Commonly used for things like taking a quick, consistent snapshot of the data directory before a risky operation

Checkpoints take advantage of the same immutability that makes SSTables safe to share elsewhere in RocksDB's design, letting a nearly free point-in-time copy be created without duplicating gigabytes of data.

RocksDB can compress the data blocks inside SSTables to reduce disk space usage and the amount of data that needs to be read from storage.

• Supports multiple compression algorithms, letting an operator choose a trade-off between compression ratio and CPU cost
• Can be configured differently per level, for example using faster, lighter compression on frequently-accessed upper levels and heavier compression on colder, lower levels
• Reduces I/O for reads that touch compressed blocks, at the cost of CPU time spent decompressing them

This per-level flexibility lets an operator tune compression separately for hot, frequently-touched data versus cold, rarely-accessed data sitting in the lowest levels.

A Data Block is the actual unit of storage inside an SSTable that holds a contiguous, sorted group of key-value pairs.

• An SSTable is made up of many data blocks, each covering a small portion of that file's overall sorted key range
• The SSTable's index block records where each data block starts, letting RocksDB jump directly to the relevant one
• Data blocks are the unit that gets compressed, cached, and read from disk, rather than the SSTable file as a whole

Splitting an SSTable into many smaller data blocks is what makes it practical to read just the small piece of a large file that's actually relevant to a given lookup.

Prefix Seek is an iteration technique that lets RocksDB efficiently find and scan all keys sharing a common prefix, without needing to scan through unrelated keys.

• Relies on a configured prefix extractor, which tells RocksDB how to derive a prefix from a full key
• Can use a specialized prefix Bloom filter to quickly skip SSTables that don't contain any keys with the requested prefix
• Commonly used for tasks like listing all keys belonging to a particular category or namespace stored under a shared prefix

This is a practical optimization for a very common access pattern, since without it, finding all keys under a given prefix would otherwise require scanning much more of the keyspace than necessary.

A Merge Operator is custom logic an application supplies that defines how RocksDB should combine a base value with one or more pending Merge operations into a final result.

• Applied lazily, RocksDB doesn't necessarily compute the merged result immediately on every write, only when a read or compaction actually needs the final value
• Lets an application express operations like incrementing a counter or appending to a list without a read-modify-write round trip
• Must be associative and consistent, since merges can be applied in different groupings depending on when compaction happens to run

Without a properly implemented Merge Operator, the Merge operation itself has no way to know how to actually combine values, it's the piece of application-specific logic that makes Merge meaningful.

• MemTable size: a larger MemTable absorbs more writes before flushing, trading memory usage for fewer, larger flush operations
• Compaction style: leveled versus universal versus FIFO, each shifting the balance between write, read, and space amplification
• Block cache size: a larger cache keeps more hot data in memory, directly improving read latency
• Bloom filter settings: affects how effectively RocksDB can skip SSTables that don't contain a requested key
• Compression settings: trades CPU time for reduced disk space and I/O
• Number of background threads: controls how much parallelism is available for flush and compaction work

Tuning RocksDB well generally means understanding a specific workload's read/write mix and adjusting these options together, rather than treating any single setting in isolation.

This distinction is about whether a write waits for its write-ahead log entry to be physically confirmed on durable storage before returning to the caller.

• Sync write: waits for the WAL entry to be flushed all the way to physical storage before the write call returns, guaranteeing durability even against a full power loss
• Async write: returns as soon as the write is applied to memory and the OS's write buffer, without waiting for a physical disk sync, which is faster but risks losing very recent writes on a hard crash

This is a direct trade-off between write latency and durability guarantees, and RocksDB lets applications choose per-write which side of that trade-off they need.

The Options object is the central configuration structure passed when opening a RocksDB database, controlling nearly every tunable aspect of its behavior.

• Covers settings like MemTable size, compaction style, compression, block cache configuration, and the number of background threads
• Some settings apply database-wide, while others can be set per column family for finer-grained control
• Sensible defaults exist for most options, letting a simple application get started without tuning everything manually

Because so much of RocksDB's performance behavior is configurable, the Options object is effectively where an operator translates their understanding of a workload into RocksDB's actual runtime behavior.