el10savio/duckwatch

duckWatch gives you real-time visibility into what is happening inside a DuckLake, a data lake format that uses a live Postgres database as its catalog instead of a plain file. Because the catalog is an actual running database, it can show things a file-based catalog simply cannot capture, such as transactions that are still in progress, lock contention between concurrent writers, and how often writers collide and have to retry when they race to commit changes at the same time. The project watches this catalog in two ways at once. First, it uses a tool called Debezium to tail the Postgres write-ahead log through change data capture, which means it picks up every committed change to the catalog, like new snapshots and file statistics, as it happens. Second, a small Go program regularly polls Postgres system views that report on active queries, current locks, and database activity, which surfaces the live, in-progress state that committed-change tracking alone would miss. Both streams of information flow through Kafka and land in ClickHouse, a database built for fast analytical queries, where they populate separate sets of tables for the committed-change data and the live-activity data. From there, a Superset dashboard lets you explore the results visually, for example watching how contention and conflict rates change as you add more concurrent writers. To try it out, you need Docker Compose v2 and Go 1.25 installed. A set of make commands handles the whole lifecycle: one command sets up the full pipeline end to end, another brings up the Superset dashboard on your local machine, another lets you generate a configurable load of requests and writers to see contention build up, and a final command tears everything down and removes the volumes when you are finished. This is a young, small project with no stated license or broader documentation beyond the setup steps and architecture diagram in the README, so treat it as an experimental tool for exploring DuckLake internals rather than a finished product.