tj/statsite

Statsite is a high-performance metrics aggregation server that collects, summarizes, and forwards application metrics to various destinations. It is a C-based reimplementation of Etsy's StatsD, meaning it speaks the same protocol but is built for speed and lower resource usage. At its core, it listens for incoming metrics over TCP, UDP, or standard input. Applications send simple text messages like "api.requests:142|c" to increment a counter or "api.response_time:45|ms" to record a timer. Statsite collects these data points during a configurable flush interval (default 10 seconds), then aggregates them, computing sums for counters, means and percentiles for timers, and unique counts for sets. When the interval expires, it pipes the summarized results to an external command (a "sink") via standard input, which then forwards the data to a storage or visualization system. The project ships with ready-made sinks for popular destinations like Graphite, InfluxDB, Ganglia, and Librato. Because the sink interface is simply any executable that reads from standard input, you can write your own in any language to send metrics wherever you need. It would appeal to engineering teams running high-traffic services who need to track performance metrics like request rates, response times, or error counts, and who want to feed that data into a dashboarding or alerting tool. If you are already sending metrics in StatsD format but find your aggregator becoming a bottleneck, this project offers a drop-in replacement designed to handle millions of metrics across hundreds of connections. A notable design choice is how it manages memory. Rather than storing every individual data point, it uses algorithms that trade a small, configurable amount of accuracy for significant memory savings. For example, timer percentiles are estimated using a streaming quantile algorithm, and large sets of unique values are estimated using HyperLogLog. This means reported percentiles are approximately correct rather than exact, but it allows the server to handle massive throughput without exhausting memory.