d4vinci/scrapling

Scrapling is a Python web scraping framework built to solve two very real problems that developers face when extracting data from websites: websites keep changing their structure (so your scraper breaks), and many modern websites actively try to block automated access using anti-bot protection systems like Cloudflare Turnstile. The framework's standout feature is its adaptive element-finding capability. When you scrape a page and save the elements you found, Scrapling remembers them. If the website later changes its layout, say a CSS class gets renamed or the HTML structure shifts, Scrapling can still locate the same data by recognizing the element from its previous position and context. This means scrapers stay working longer without constant maintenance. For getting past anti-bot defenses, Scrapling provides multiple "fetcher" modes. A standard fetcher handles simple, unprotected websites quickly. A "stealthy" fetcher mimics a real browser by disguising request headers and browser fingerprints. A "dynamic" fetcher uses actual browser automation under the hood for JavaScript-heavy pages, capable of waiting until all content has loaded before extracting anything. When someone needs to scale beyond a single page, Scrapling includes a spider framework where developers define starting URLs and parsing logic, and the system handles crawling multiple pages concurrently. It supports pause-and-resume crawls, automatic proxy rotation, and real-time progress reporting. You would use Scrapling when building any kind of data pipeline that pulls information from websites, price monitoring, research aggregation, lead generation, or content archiving, especially on sites that require browser-like behavior or change their layouts frequently. The tech stack is Python, with optional browser automation powered by Playwright internally for dynamic page rendering. It is installable via pip and supports Python 3.9 and above.