binux/pyspider

pyspider is a web crawling framework written in Python. A web crawler, also called a spider, is a program that automatically visits websites, reads their content, and extracts data from them. pyspider makes it easier to build these programs by handling the scheduling, retrying, and storage of crawl jobs while letting you focus on writing the logic for what to collect. The framework comes with a web-based interface where you can write and edit your crawl scripts, monitor running tasks, manage projects, and view results, all from a browser. This is unlike most crawlers that are purely command-line tools. The sample code in the README shows the basic pattern: you define a handler class with methods for different types of pages. One method handles the starting page, finds links, and queues them for crawling. Another method extracts the specific data you want from each page, in the example, the URL and title. pyspider supports multiple database backends for storing results (MySQL, MongoDB, PostgreSQL, SQLite, and others) and multiple message queue systems for coordinating work across distributed machines. It supports crawling JavaScript-heavy pages and can be configured with task priorities, automatic retries on failure, and scheduled re-crawls on a time interval. You would use pyspider if you need to regularly scrape data from websites at scale, for example, monitoring prices, aggregating content, or building a dataset. It is installed via pip and starts with a single command.