ckl75/semantic-manifest-specification

The Semantic Manifest Specification is an open data standard for describing a website's content and structure in a format that AI crawlers, language models, and retrieval systems can read efficiently. It defines a streamable file format called NDJSON, where each line is its own small piece of JSON, so a crawler can read a site's content types, how pages relate to each other, and which pages have plain text markdown versions, one line at a time without loading everything into memory at once. The project argues that existing web standards were not built for this job. Sitemaps only list raw page addresses without saying how those pages relate to each other. JSON-LD only describes one page at a time and cannot show the shape of an entire site. Plain text files meant to guide AI crawlers use up a lot of a language model's limited context space once a site has thousands of pages. The Semantic Manifest is meant to close that gap by giving crawlers a compact, line by line map of the whole site. The repository includes the written specification itself, version 0.1, along with a reference example file showing what a real manifest looks like. According to the project, after linking a manifest on a roughly 58,000 page site, an AI crawler called ClaudeBot was able to read the entire site at about seven pages per second, and a demo video of that crawl is linked in the project. The specification is already used as a built in part of a larger content publishing system called the High-Velocity Content Engine, and two live example manifests from real sites are linked for reference. The specification and its reference file are released under the CC0 public domain license, meaning anyone can use them freely without restriction.