Database / ScyllaDB Interview questions
How do you troubleshoot compaction backlog in ScyllaDB?
- Check pending compaction metrics in the Monitoring Stack - a steadily growing backlog means compaction can't keep pace with incoming writes.
- Review the compaction strategy fit - STCS under heavy, uneven write patterns can lag; LCS or ICS may handle the shape of the workload better.
- Check disk I/O saturation - compaction is I/O-heavy, and a disk that's already near its throughput limit from foreground traffic will starve compaction.
- Look for oversized partitions - a few very large partitions can dominate compaction time disproportionately.
- Review compaction throughput throttling settings - an overly conservative bandwidth cap can be safe for foreground latency but let backlog build up over time; it may need raising if the cluster has I/O headroom.
- Check for repair-driven compaction spikes - large repairs stream a lot of new data that then needs compacting, so backlog can correlate with a recent repair window.
A persistent backlog usually means the table is fundamentally under-provisioned for its write rate rather than a one-time blip, so alongside these checks it's worth evaluating whether the table needs more nodes or a schema change (like reducing tombstone-heavy delete patterns) rather than only tuning compaction settings.
More Related questions...