h4ckf0r0day/awesome-ai-web-scraping

This is an awesome list repository, a curated catalogue of tools and services that combine AI or large language models with web scraping. The point is to help someone find ready-made options for turning the web into clean text for LLMs, retrieval pipelines, or agent workflows. The README is clear about what does not belong here. General-purpose scrapers like Scrapy or BeautifulSoup are pointed at a different awesome list, and autonomous browser agents are pointed at yet another list. The catalogue is split into sections that follow the typical scraping stack. Frameworks and Libraries is the self-hosted, open-source layer. The list calls out Crawl4AI, Scrapling, ScrapeGraphAI, llm-scraper, Jina Reader, Stagehand, Browser-Use, Skyvern, LaVague, CyberScraper 2077, ScraperAI, SpiderCreator, and PulsarRPA, each with a one-line description of the approach and the languages or models supported. Hosted APIs is the managed equivalent. The README lists Firecrawl, Jina Reader, Diffbot, Apify, Bright Data, Zyte, ScrapingBee, ZenRows, Oxylabs, Spider, WebScraping.AI, Scrapeless, Kadoa, Expand.ai, and Reworkd, with notes on pricing tiers and what each is best at. Some target LLM-ready Markdown output, others sit closer to traditional scraping APIs with anti-bot bypass and proxy networks. Several supporting sections cover infrastructure pieces around those tools. Browser Infrastructure for AI covers Steel.dev, Browserbase, Hyperbrowser, Anchor Browser, Browserless, Obscura, and Browserable for the headless browser layer. No-Code AI Scrapers covers point-and-click tools like Browse AI and Bardeen. Further sections listed in the table of contents are MCP Servers for Scraping, Web Search APIs for LLMs, Proxy and Anti-Bot Infrastructure, Datasets, Benchmarks and Research, and Tutorials and Guides, plus a contributing section at the end. The repository itself contains no code; the language is reported as unknown. It exists as a single Markdown file with the standard Awesome badge, and it acts as a starting reference for someone evaluating which AI scraping tool fits their workflow.