reyneill/turbosearch

TurboSearch is a TypeScript command line tool for running local semantic search over documents that an AI agent needs to look through, such as a client's files. It is built with Bun and organizes its work into separate layers for ingesting files, storing data, generating embeddings, and running the actual vector search, so each piece can be swapped out later. The tool can ingest plain text and code files, extract text from PDFs (using a layout aware parser first, then pdftotext, then an OCR model for scanned pages), run OCR on images, and transcribe audio and video using either the Cohere Transcribe service or a self hosted alternative. Video files can also have their frames sampled and read with OCR at fixed time intervals. By default TurboSearch uses local hash based embeddings, which are private and consistent but not true semantic embeddings, for higher quality search, a user can add an OpenAI API key and ingest documents with real embeddings instead. For storage, SQLite is the default, using its FTS5 feature for keyword search and an in process vector index for similarity search, which keeps the tool simple to run without extra setup. For larger, shared, or production use, TurboSearch can instead use PostgreSQL with the pgvector and psql_bm25s extensions, letting the database itself combine keyword and vector matching rather than pulling every chunk into the application. A Docker based command is included to spin up a temporary Postgres instance for local testing of that mode. Search itself supports several backends: an exact cosine similarity search, a compressed TurboQuant style search meant to be faster on large collections, a plain keyword BM25 search, and a hybrid mode that merges vector and keyword results together. The command line interface also has a JSON output mode and a streaming JSON Lines mode, both intended for other programs or AI agents to call directly rather than for a human reading plain text output. The README frames this as infrastructure for agent facing search rather than a finished consumer product, and lists formats like DOCX, PPTX, and XLSX as not yet supported.