ruipgil/scraperjs

Scraperjs is a Node.js library for extracting data from web pages. You point it at a URL, write a short function that picks out the content you want using jQuery-style selectors, and it returns the results. The whole thing is promise-based, so you chain steps together in a readable sequence. The library offers two scraper types. The StaticScraper is lightweight and fast: it downloads the raw HTML and lets you query it with cheerio, a jQuery-compatible library that runs server-side. It works well for pages whose content is already present in the HTML source. The DynamicScraper uses PhantomJS, a headless browser, which means it can run JavaScript on the page and see content that only appears after scripts execute, similar to what you would see in a real browser. The DynamicScraper is heavier and requires PhantomJS installed separately. Both scrapers share nearly the same API, so switching between them is straightforward. The scraping function you write is almost identical for both, with one key difference: in the DynamicScraper, the function runs inside the page's sandboxed environment, which means it cannot reference variables from the surrounding Node.js code. For scraping multiple sites or many different URL patterns, scraperjs includes a Router class. You define URL patterns with named parameters (similar to route patterns in web frameworks), attach a scraper to each pattern, and then feed URLs to the router. It matches each URL to the right scraper and handler automatically. The API also exposes controls for status code handling, delays, timeouts, async steps, and error catching within the promise chain. The library is installed via npm with a single command. PhantomJS is optional and only required for the DynamicScraper.