ngaut/benthos

Benthos is a tool for moving and transforming data streams between different services. Think of it as a reliable middleman that sits between your data sources and destinations, reading messages from one place, optionally modifying them, and sending them somewhere else, all while being rock-solid about not losing anything. The typical use case: you have messages flowing through your system (maybe from Kafka, an HTTP endpoint, or RabbitMQ), and you need to clean them up, filter them, or route them to multiple places. Benthos handles all that plumbing. It connects to a wide variety of popular services, Kafka, AWS S3, Redis, Elasticsearch, MQTT, and many others, so you can plug it into almost any existing pipeline. You write a simple configuration file describing where data comes from, what to do with it, and where it should go, then Benthos runs that configuration reliably. What makes Benthos stand out is resilience. By default, it's built to handle crashes gracefully. If your application restarts, Benthos won't lose messages, it ensures everything gets delivered at least once, even without extra infrastructure. If you need even stronger guarantees, you can add a buffer layer. You also get monitoring built in, with the ability to send metrics to Statsd or Prometheus so you can track what's happening in your stream. The project is straightforward to deploy. You can run it as a standalone binary, drop it in a Docker container, or use it as a framework if you're building custom stream processing in Go. Configuration is done through YAML files that you can parameterize with environment variables, making it easy to adjust settings for different environments without changing code. The README includes concrete examples and even a cookbook section with real-world use cases like deduplicating Twitter data into Kafka.