explaingit

jl037/doc-lok

Analysis updated 2026-05-18

★ 2TypeScriptAudience · developerComplexity · 2/5LicenseSetup · easy

TLDR

A Node.js tool that shrinks Markdown files before sending them to AI models by replacing unchanged external links with tiny placeholders, cutting token usage by up to 99.5%.

Mindmap

mindmap
  root((doc-lok))
    What it does
      Compresses Markdown for LLMs
      Caches external link hashes
      Restores links on demand
    How it works
      SHA-256 content hashing
      ETag fast-path checks
      Local lockfile storage
    Usage modes
      CLI pipe-friendly
      TypeScript library
      JSON output for agents
    Setup
      npm install doc-lok
      Node.js 18+
      Zero runtime dependencies
Click or tap to explore — scroll the page freely

Code map

Detail Auto

An interactive map of this repo's files and how they connect — its source is parsed live in your browser. Click Visualize to build it.

filefunction / class

What do people build with it?

USE CASE 1

Pipe a large Markdown file through doc-lok before sending it to an LLM to cut token costs on unchanged external links.

USE CASE 2

Integrate doc-lok as a TypeScript library in an AI agent workflow to condense documents programmatically before each prompt.

USE CASE 3

Run doc-lok --check --json to inspect which URLs in a document have changed before deciding whether to re-run an AI analysis.

What is it built with?

TypeScriptNode.js

How does it compare?

jl037/doc-lokarashthr/hugo-flowargeneau12e/kairos-tx
Stars222
LanguageTypeScriptTypeScriptTypeScript
Setup difficultyeasymoderatehard
Complexity2/53/54/5
Audiencedeveloperdeveloperdeveloper

Figures from each repo's GitHub metadata at analysis time.

How do you get it running?

Difficulty · easy Time to first run · 5min
Use freely for any purpose, including commercial use, as long as you keep the copyright notice.

In plain English

doc-lok is a Node.js library and command-line tool that reduces the number of tokens your Markdown documents consume when you feed them to an AI model. It works by detecting every external URL in a Markdown file, checking whether the content at that URL has changed since you last ran it, and replacing any unchanged links with a tiny HTML comment placeholder. Only links that have actually changed are left visible to the model. The token savings come from the fact that URLs and the text around them take up space in the AI's context window. If a documentation link or external reference has not changed since your last prompt, the model does not need to see it again. doc-lok caches the content hash and HTTP ETag of each URL in a local lockfile, then checks freshness with a lightweight network request before deciding whether to compress a link. The README claims up to 99.5% reduction per cached link. The restore command reverses the operation, turning the compact placeholders back into full links. This round-trip is lossless because each placeholder embeds the hash needed to look up the original URL in the lockfile. You can use it from the command line by piping a Markdown file through it before sending the result to an LLM, or as a TypeScript library inside a larger automation. It also outputs JSON for machine-readable integration with AI agent frameworks. A Windsurf workflow skill file is included. The library has zero runtime dependencies and requires Node.js 18 or later. The license is MIT.

Copy-paste prompts

Prompt 1
I'm building a pipeline that reads Markdown docs and sends them to Claude. Show me how to use doc-lok's condenseMarkdown() function to pre-process files in TypeScript before the API call.
Prompt 2
How does doc-lok's lockfile track whether a cached URL has changed? Walk me through the SHA-256 hashing and ETag validation logic.
Prompt 3
I want to add doc-lok to a GitHub Actions workflow so it compresses my documentation before a nightly LLM summary job. Show me the YAML step.
Prompt 4
Can I configure doc-lok to skip compression for certain domains (like internal URLs) and only cache public external links?

Frequently asked questions

What is doc-lok?

A Node.js tool that shrinks Markdown files before sending them to AI models by replacing unchanged external links with tiny placeholders, cutting token usage by up to 99.5%.

What language is doc-lok written in?

Mainly TypeScript. The stack also includes TypeScript, Node.js.

What license does doc-lok use?

Use freely for any purpose, including commercial use, as long as you keep the copyright notice.

How hard is doc-lok to set up?

Setup difficulty is rated easy, with roughly 5min to a first successful run.

Who is doc-lok for?

Mainly developer.

Open on GitHub → Explain another repo

This repo across BitVibe Labs

Scan in gitsafehub Deploy in gitdeployhub jl037 on gitmyhub

Verify against the repo before relying on details.

Alternatives & similar repos