cianmcnally/youtuberecs-neo4j

YoutubeRecs-Neo4j, whose command line tool is called raquel, is a crawler written in Rust that explores YouTube's related-videos sidebar and saves what it finds into a Neo4j graph database. Every YouTube watch page shows a list of related videos chosen by YouTube's recommendation system. By visiting enough of these pages and following the related videos they suggest, the tool builds up a map of an entire topic area, showing which channels and videos cluster together and which sit on the edges. The crawler calls the same internal API that a real YouTube watch page uses, so it does not need an official API key or quota, and it does not scrape the page's HTML directly. For each video it visits, it records the video's title, view count, likes, publish date, and its channel's name and subscriber count, then creates a link in the graph to every video shown in that sidebar, keeping track of the position each one held in the list. Videos that show up in a sidebar but have not been visited yet are stored as unscraped placeholders, and the database itself acts as the queue of what to visit next: there is no separate task list, no cache, and no memory that could be lost if the program stops. You can stop the crawler at any time and restart it, and it will pick up exactly where it left off. The walk is guided by keywords and includes some randomness: at each step it picks an unvisited video whose title matches a chosen topic, whose channel is not blocked, and that was ranked highly in some sidebar, then waits a random five to fifteen seconds before the next request to stay polite to the site. To run it, you need Rust installed and a Neo4j database, which can be started quickly with Docker or Neo4j Desktop. Once set up, you can fetch a single video, do a dry run with no database, or start a full crawl seeded with one video and a set of keywords. A separate stats command reports how large the graph has grown and how much unexplored frontier remains. The finished graph can be explored visually in Neo4j Bloom or exported for other graph visualization tools.