just-spider/spiderfor91

SpiderFor91 is a reference guide for writing Python web crawler scripts that plug into a specific video platform called Project 91. The repository does not ship any finished crawlers. What it provides instead are two detailed AI prompt templates: paste the first into any AI assistant to generate code that finds direct video and thumbnail URLs on a target site, then paste the second to add all the structural requirements the Project 91 backend expects. Those structural requirements are precise. Each crawler must be a single Python file with a static name declared at the top. It reads a job description from a JSON file passed on the command line. That job file specifies how many candidate videos to collect, a proxy address for outbound requests, an output directory, and a path to a text file listing video IDs the backend has already processed. The script skips any video whose ID appears in that seen list without re-fetching it. Output works differently from a typical batch scraper. Instead of collecting all results and writing them at the end, each video record must be printed to standard output immediately as it is found, one JSON object per line. Required fields are a stable source ID, a title, and a direct media URL. Thumbnail URLs and request headers for sites that check the referring page are recommended. The backend handles all downloading, deduplication, and storage separately. The project is aimed at developers who want to contribute new site-specific crawlers to the Project 91 platform, or who need to adapt an existing script to the streaming output format the Go backend requires. The README acts as a spec sheet rather than a runnable program. New site requests can go through issues, and finished scripts can be submitted as pull requests.