What is chaos engineering and how does it relate to observability?

Chaos engineering is the practice of intentionally injecting failures into a system in a controlled way to discover weaknesses before they cause unplanned outages. The discipline was pioneered by Netflix with their Chaos Monkey tool (now part of the Simian Army), which randomly terminated EC2 instances in production to verify that services could survive node failures.

Chaos engineering tools include Chaos Monkey (instance termination), Gremlin (CPU, memory, network, and disk failure injection as a SaaS platform), Chaos Mesh (Kubernetes-native chaos experiments), and Litmus Chaos (another CNCF project for Kubernetes chaos).

The connection to observability is direct and bidirectional. Chaos experiments are only safe and useful if you have strong observability in place first:

• You cannot run chaos safely without observability. If you cannot detect the blast radius of an experiment in real time, you risk turning a controlled test into a real incident. You need dashboards, alerts, and SLO tracking active before any experiment starts.
• Chaos experiments validate your observability. Running a chaos experiment and checking whether your alerts fired, your runbooks worked, and your dashboards showed the failure is a direct test of whether your monitoring would catch the same failure if it happened unexpectedly.
• Chaos reveals observability gaps. If an experiment causes a failure that goes undetected by your monitoring until you look at a dashboard manually, that is an observability gap to close.