major101x/nimbus-data

NimbusData is a learning project for practicing distributed systems concepts using a made up weather data pipeline. A Rust program acts as a producer, generating fake sensor readings and publishing them to Kafka, a message streaming system, while a separate Rust consumer program reads those messages back and tracks how many it has processed. The whole thing runs inside Kubernetes, a system for managing groups of containers, using a tool called kind to create a small local cluster for testing and an operator called Strimzi to run Kafka on top of it. Prometheus and Grafana are included for monitoring, collecting metrics like how many events were produced and consumed, and showing them on dashboards, with an alert that fires if the consumer falls too far behind the producer. The project also demonstrates how Kafka spreads work across multiple consumers: the data topic is split into three partitions, and each partition can only be read by one consumer in a group at a time, so adding more consumers, up to three, increases how fast messages get processed, while a fourth consumer would sit idle. Messages are grouped by a weather station identifier so that all readings from one station always land on the same partition, keeping their order consistent. Setting it up requires Docker, the kind tool for local Kubernetes clusters, kubectl, Helm, and a Rust toolchain along with some C build tools, since one of the Rust libraries used compiles from C source code. The README includes step by step setup commands for the cluster, Kafka, and the monitoring stack, plus a section on real mistakes the author ran into, such as a producer getting stuck after a Kafka broker restarts. No license is stated in the README.