davidpdrsn/swirl

Swirl is a tool for Rust developers who need to handle slow or heavy tasks in the background, away from their main web application. Imagine a user uploads a photo to your website. Instead of making them wait while the server resizes that image, you can hand the task off to this tool. It saves the task, and a separate background worker picks it up and processes it later, keeping your website fast and responsive. It works by storing your queued tasks directly inside your existing PostgreSQL database. When you want to run a task in the background, you simply call a function and tell it to enqueue. The tool writes that task and its details into your database. Then, a separate background process continuously checks the database for new tasks. When it finds one, it grabs the task, spins up a worker, and runs the job across multiple threads. This approach was originally built for and extracted from crates.io, the official package registry for the Rust programming language. They use it to update their search index without slowing down their main web servers. Any developer building an application in Rust with the Diesel database library could use it for similar slow tasks, like sending batch emails, processing uploaded files, or doing heavy data calculations. One important thing to know is that it guarantees a task will run at least once, but it might occasionally run the same task twice if something goes wrong. Because of this, developers need to design their tasks so that running them multiple times causes no harm. The project is still in its early stages and lacks some conveniences like configurable retry behavior or multiple priority queues, but it is successfully running in production today.