netflix/concurrency-limits

Netflix's concurrency-limits is a Java library that helps online services automatically figure out how many requests they can handle at the same time without slowing down or crashing. Instead of setting a fixed cap on requests per second, which quickly becomes outdated as systems scale up or down, the library continuously measures and adjusts the limit in real time based on actual conditions. The core idea borrows from how internet traffic control works. Rather than counting total requests, it focuses on concurrent in-flight requests. When too many requests pile up, latency increases and queues form. The library detects this by watching latency measurements and rejection signals, then adjusts the allowed concurrency up or down. It includes algorithms like Vegas, which estimates queue buildup by comparing current latency to the minimum observed, and Gradient2, which smooths out spikes using two averaging windows to spot real trends versus noise. This is useful for teams running distributed services that auto-scale, where nobody can manually track every server's capacity. For example, a service handling both live user traffic and batch jobs could use this to guarantee live traffic gets 90% of capacity while batch gets 10%, so real users aren't starved during heavy background processing. It works with common communication frameworks including gRPC and standard web servlets, and can be deployed on either the server side to protect a service or the client side to prevent one service from overwhelming another. What stands out is the shift from static configuration to adaptive, measurement-based limits. Each node in a system makes its own local decisions based on what it observes, which is more practical than trying to coordinate limits across a large fleet of machines. The library also supports priority-based enforcement, so critical traffic can be protected while less important work gets throttled first.