beastmastergrinder/turbopuffer-engine-opensource

This repository is an educational clone of a real-world search engine called turbopuffer. The project is built in Go and shows how a vector search system can use object storage (like Amazon S3 or the S3-compatible MinIO) as its primary data store rather than relying on fast SSD arrays, then achieve low query latency through smart caching and indexing on top. The central insight is that search is not a transactional database. Writes can be slow and batched, like a data warehouse, while reads need to come back quickly (the project targets under 100 milliseconds). The code demonstrates three ideas that make this work: treating object storage as the single source of truth, coordinating concurrent writes with a conditional-write pattern on a single JSON file instead of a distributed consensus system like Raft or Kafka, and keeping unindexed data searchable by scanning a write-ahead log for recent additions. The project includes a full Go library and a command-line tool with commands for creating namespaces, adding data, building indexes, querying, and branching. Queries support vector similarity search (finding items nearest to a given vector), full-text BM25 search, and hybrid combinations of both. The codebase also demonstrates a two-tier cache using RAM and NVMe storage to reduce how often the system needs to reach back to object storage. The documentation folder has sourced explanations for every architectural decision. The readme explicitly flags anything the original turbopuffer has not publicly confirmed, so the educational claims stay honest. You need Docker and Go to run it locally: Docker runs MinIO as the S3-compatible backend, and the Go toolchain builds the CLI and runs the test suite. This is for developers who want to understand how modern cloud-native search engines are built, particularly the ideas behind tiered storage, object-storage-backed indexes, and approximate nearest-neighbor search.