matthiasnordwig/pdf-struct-chunker

pdf-struct-chunker is a Rust library and command-line tool for splitting PDF documents into meaningful text chunks based on the document's actual structure rather than a fixed character count. It is aimed at developers building systems that answer questions over documents using AI, a pattern commonly called RAG (Retrieval-Augmented Generation). The problem it addresses is that most tools split PDFs by a fixed number of characters or tokens, which can cut headings away from their content or break sentences in the middle. This tool reads the position and font size of text on each PDF page to understand where headings start, where sections end, and how the document is organized. Each chunk of text it produces includes the section name, heading, and page number as structured data alongside the text itself, so downstream tools know exactly where each chunk came from. No AI model or internet connection is needed for the chunking process. The tool runs entirely on the CPU and processes a 100-page PDF in under one second on a standard laptop. You can use it as a command-line tool to produce JSONL or JSON output, or as a Rust library where you pass raw PDF bytes and get structured chunks back. Custom rules for how to detect headings, definitions, and lines to ignore can be written as JSON configuration files using regular expressions. The built-in defaults are tuned for legal and regulatory documents that use numbered sections. The library is available on crates.io and can be added to a Rust project with one command. The license is MIT.