gao-pooh/paper-scraper

Paper-scraper is a Python tool for searching and batch-downloading academic papers from two platforms: ScienceDirect (Elsevier) and INFORMS PubsOnLine. It is aimed at researchers who have institutional access to these journals through a university or library and want to collect large numbers of PDFs without clicking through each article manually. You can search by keyword, journal name, author, ISSN, or a combination of criteria. Results come back as spreadsheet or JSON files containing paper titles, authors, abstracts, and links. If you have an active institutional login in Chrome, the tool can read your browser cookies automatically so you do not have to copy credentials by hand. Downloading PDFs from ScienceDirect is technically more involved because Cloudflare blocks normal web requests. The tool handles this by launching Chrome in a special debug mode, navigating to each article page inside your existing browser session, and intercepting the PDF bytes before they reach the screen. This bypasses the block without requiring you to install extra browser automation software beyond Chrome itself. For INFORMS, the PDF endpoints are simpler and a valid session cookie is sufficient. All output files are saved to a local results folder, organized by search query and timestamp. PDFs are named by author and year. The project is written in Python 3.8 and above. Running it requires installing a few Python packages (curl_cffi, websocket-client, browser-cookie3, and openpyxl). The scripts currently hardcode the Chrome path for macOS, though Linux and Windows users can point the script to their own Chrome binary with a small config change. An interactive wizard mode is available for first-time users who prefer prompts over command-line flags.