glassflow/clickhouse-etl

GlassFlow is an open source stream processing engine built to move and reshape large amounts of data into ClickHouse, a database designed for fast analytics. It is meant for pipelines that need to handle huge volumes of data, described in the README as terabyte scale, moving information in real time from sources like Kafka into ClickHouse tables. The tool offers two kinds of transformations. Stateless transformations use an expression engine to apply straightforward changes to each piece of data as it passes through, such as removing null values or filling in missing timestamps. Stateful transformations rely on a built in state store to handle more complex logic, like removing duplicate records or joining data that arrives within a certain time window of each other. Beyond transformations, GlassFlow can also filter out events before they reach ClickHouse, or simply move data across without changing it at all. Operationally, the project includes built in metrics with OpenTelemetry support so teams can monitor their pipelines, along with a dead letter queue that catches events which fail to process, so a single bad record does not stop the whole pipeline. Instead, those failed events can be inspected and reprocessed later. GlassFlow is self hosted and runs on Kubernetes, and the README points to a Helm based installation guide for setting it up on services like AWS EKS, GKE or AKS. There is also a live demo environment where visitors can see it running against a real cluster with a Grafana dashboard, plus written documentation covering installation, everyday usage and its overall architecture. The project is released under the Apache License 2.0.