anishathalye/unblock

Unblock is a small command-line utility that sits between two programs connected by a "pipe" (the | symbol) and makes the pipe behave as if it has unlimited buffering capacity. Normally, when you chain two programs together so that one's output feeds into the other's input, there's a strict limit on how much data can wait in the transfer space. Once that limit is hit, the producing program pauses until the receiving program catches up. Unblock removes that bottleneck by absorbing everything the first program spits out, holding it in memory, and passing it along as the second program is ready for it. The classic example is a program that slowly generates thousands of lines of data, which you want to browse using a pager like less. Normally, if you run slow | less, the slow program gets paused once the pipe fills up, meaning when you scroll down you're stuck waiting for more data to be computed. By inserting Unblock in the middle (slow | unblock | less), the slow program runs to completion without ever pausing, dumping everything into Unblock's memory buffer. You can then scroll through the results freely in less at your own pace. This tool is for developers or power users who work in a terminal and run into situations where the natural pause-and-wait behavior of pipes gets in their way. It's useful when you have a program that produces a lot of output slowly or in bursts, and you want to separate the production of that data from the consumption of it without writing it to an intermediate file on disk. The README is upfront about a significant trade-off: this tool is usually not what you want. The standard pipe behavior exists for a good reason, as it prevents a fast producer from overwhelming a slow consumer and eating up all your computer's memory. Because Unblock's buffer is truly unlimited, if the producing program generates a massive amount of data and the reader is slow, Unblock will keep consuming memory until your system runs out.