bhup-github/chaos-eng

chaos-eng is a small, working example of chaos engineering for Kubernetes, a practice where you deliberately break parts of a running system to see whether it can recover on its own. The idea behind it is simple: instead of finding out during a real outage whether your system heals itself, you test that in a calm, controlled way ahead of time by killing something on purpose and watching what happens. The project has four parts working together. First, a Go controller called the chaos monkey runs on a timer, picks a random pod in Kubernetes, and deletes it. Second, several safety measures keep this contained: the monkey only touches pods that carry a specific label marking them as opted in, it defaults to a dry run mode where it only logs what it would kill instead of actually killing anything, and its permissions are limited to listing and deleting pods in a single namespace. A minimum alive setting also stops it from ever killing a pod if doing so would drop the service below a set number of running replicas. Third, a small sample web service is deployed with three copies running behind a load balancing service, so when the chaos monkey removes one, Kubernetes notices and automatically starts a replacement. Fourth, a watch script continuously sends requests to the web service and prints a live success rate along with the current pod count, so you can see in real time whether the killings are actually affecting users. To try it, you need Docker, kind, and kubectl installed, then run the setup script, which builds a local cluster and deploys everything. Watching the logs and the live success rate side by side lets you see pods being killed and replaced every sixty seconds while the success rate stays steady, which is the actual proof that the system is resilient. The author is upfront that this is a simplified, single instance version of chaos engineering, meant as a first step before exploring more advanced tools that can automatically abort experiments or expand testing to larger parts of a system.