germondai/trawl

Analysis updated 2026-05-18

★ 218TypeScriptAudience · ops devopsComplexity · 3/5LicenseSetup · moderate

Mindmap

mindmap
  root((TRAWL))
    What it does
      Bypasses Cloudflare
      Solves CAPTCHA
      Caches sessions
    How it works
      4-tier escalation
      Camoufox Firefox
      Redis session cache
      Residential proxy
    Tech stack
      TypeScript
      Bun
      Elysia
      Docker
    Use cases
      arr stack scraping
      Anti-bot bypass
      Fast repeat fetch
    Setup
      Docker Compose
      Baseline for old HW
      AGPL license

mindmap root((TRAWL)) What it does Bypasses Cloudflare Solves CAPTCHA Caches sessions How it works 4-tier escalation Camoufox Firefox Redis session cache Residential proxy Tech stack TypeScript Bun Elysia Docker Use cases arr stack scraping Anti-bot bypass Fast repeat fetch Setup Docker Compose Baseline for old HW AGPL license

Click or tap to explore — scroll the page freely

What do people build with it?

USE CASE 1

Replace FlareSolverr in a Sonarr/Radarr/Prowlarr setup to scrape torrent indexers protected by Cloudflare.

USE CASE 2

Scrape data from websites that block bots with Cloudflare challenges or CAPTCHA using your own self-hosted server.

USE CASE 3

Cache Cloudflare clearance cookies so repeat requests to the same domain are served in under 500ms.

USE CASE 4

Run a scraping server on a Synology NAS or older CPU hardware using the baseline Docker image.

What is it built with?

TypeScriptBunElysiaRedisDockerCamoufox Firefox

How does it compare?

	germondai/trawl	suyancc/openai-plus-vxt	texsellix/polymarket-trading-bot
Stars	218	217	244
Language	TypeScript	TypeScript	TypeScript
Setup difficulty	moderate	moderate	moderate
Complexity	3/5	3/5	4/5
Audience	ops devops	general	developer

Figures from each repo's GitHub metadata at analysis time.

How do you get it running?

Difficulty · moderate Time to first run · 30min

Requires Docker and Docker Compose, first boot takes 15-30s for browser pool warmup. Residential proxy is optional for Tier 4 escalation.

Free to use and modify, but if you distribute it or run it as a public service you must publish your source code under the same AGPL-3.0 license.

In plain English

Many websites use anti-bot protection services like Cloudflare to block automated software from accessing them. TRAWL is a self-hosted web scraping tool you run on your own server that can get through these protections, including Cloudflare challenges, CAPTCHA puzzles from reCAPTCHA and hCaptcha, and other bot detection systems. When you send a URL to TRAWL, it tries to fetch it using four progressively heavier methods. First it tries a plain web request. If that gets blocked, it uses a saved browser session from a previous successful visit. If that fails, it fires up a full Firefox browser specially patched to appear indistinguishable from a real person's browser and solves the challenge fresh. As a last resort, it can route the request through a residential IP address, a real home internet connection, if you supply one. Results from successful sessions are stored in Redis, an in-memory database, so repeat requests to the same site return in under half a second. TRAWL is designed as a drop-in replacement for FlareSolverr, a popular tool used with home media server software like Sonarr, Radarr, Prowlarr, and Jackett (collectively called the arr stack). You point those tools at TRAWL's address instead of FlareSolverr and they work without any other changes. Setup is Docker-based: clone the repo, copy the sample configuration file, and run one command. The first startup takes 15 to 30 seconds while the browser pool warms up. There is also a baseline Docker image for older hardware like Synology NAS devices that do not support modern Linux kernel features. The license is AGPL-3.0, which requires you to share source code of any modified version you distribute or run as a public service.

Copy-paste prompts

Prompt 1

Show me how to set up TRAWL with Docker Compose and connect it to Prowlarr as a FlareSolverr replacement.

Prompt 2

How do I configure TRAWL to use a residential proxy URL for Tier 4 escalation when its IP gets flagged by Cloudflare?

Prompt 3

Write a curl request to TRAWL's native /scrape endpoint to fetch a Cloudflare-protected page and parse the tier and timing from the response.

Prompt 4

How do I tune TRAWL's BROWSER_POOL_SIZE and SESSION_TTL_SECONDS for a high-volume scraping workload?

Prompt 5

Can I run TRAWL on a Synology DS920+ NAS, and which Docker image tag should I use?

Frequently asked questions

What is trawl?

TRAWL is a self-hosted web scraping server that bypasses Cloudflare, reCAPTCHA, and other anti-bot systems, with FlareSolverr-compatible API for use in Sonarr/Radarr/Prowlarr stacks.

What language is trawl written in?

Mainly TypeScript. The stack also includes TypeScript, Bun, Elysia.

What license does trawl use?

Free to use and modify, but if you distribute it or run it as a public service you must publish your source code under the same AGPL-3.0 license.

How hard is trawl to set up?

Setup difficulty is rated moderate, with roughly 30min to a first successful run.

Who is trawl for?

Mainly ops devops.

Open on GitHub → Explain another repo

This repo across BitVibe Labs

Scan in gitsafehub Deploy in gitdeployhub germondai on gitmyhub

Verify against the repo before relying on details.