lllyasviel/singlefiledb

Single File Database (SFDB) is a lightweight Python tool for storing key-value data, think of it like a dictionary that saves to disk. The author created it to manage millions of image metadata records in machine learning projects, particularly for storing Stable Diffusion prompts. The entire thing is one Python file with under 150 lines of code, no external dependencies, and you install it by just copying the file into your project. Under the hood, it stores data using SQLite, a widely-supported database format. Your values are saved as human-readable JSON text, which means you can open the database file in any SQLite viewer and inspect or edit your data directly, even outside your code. For small datasets, you can load everything into memory with a single line. For massive datasets, it lets you read and write items one at a time without loading the whole file, so you could theoretically process 10TB of data using only 10MB of memory. This would appeal to someone building a Python app who needs persistent storage but doesn't want the overhead of setting up a full database server or learning a complex ORM. A solo developer prototyping an AI tool, a researcher tracking experiment results, or a startup founder scraping and storing web data could drop this in and start saving records immediately. The thread-safety and automatic crash recovery mean you don't have to worry much about data corruption if your app dies mid-write. A notable design choice is how it handles disk writes. Updates appear instant to your code, but the actual file writes are batched, either every 16,384 updates, every 60 seconds, or when the program exits. This reduces wear on your storage drive and improves performance, though it means a sudden power loss could lose up to a minute of recent changes (application crashes, however, are handled safely by SQLite's built-in recovery).