tinyfish-io/bigset

BigSet is a tool that lets you describe a dataset you want in plain English, and it builds that dataset for you, keeps it up to date automatically, and lets you query it using SQL, a standard language for asking questions about structured data. Think of it like a self-filling spreadsheet: instead of manually collecting data from different sources or writing your own scraper, you tell BigSet what you want, such as companies currently hiring, insurance quotes in your area, or restaurants near you that serve a specific brand, and it goes out and gathers that data for you. Once the dataset exists, BigSet keeps it fresh on a schedule you choose, refreshing every 30 minutes, every hour, or whatever interval fits your needs. A built in healer feature watches for data sources that break, for example when a website changes its layout, and tries to patch the collection process automatically before you notice a problem. The collected data lives in a database, so both you and any automated agents you build can query it with SQL at any time. Under the hood, BigSet is built on TinyFish's own data collection APIs for search, fetching pages, and browser automation. The rest of the stack uses Next.js and React for the web interface, Fastify and TypeScript for the backend API, and PostgreSQL through the Drizzle ORM for storage, with a self-hosted email and password login system. Getting it running locally just requires Docker and Make: cloning the repository and running one command starts the database, backend, and frontend together. BigSet is aimed at developers and teams who are tired of building and maintaining web scrapers and data pipelines by hand, especially those building AI agents that need a steady, queryable supply of fresh data. It is an open-source, self-hosted project that is still being actively built, and the project welcomes contributions and feedback from anyone using it.