yehiabrkt46-arch/cs2-market-scrapers

This project documents the architecture of a system that collects live pricing data for Counter-Strike 2 skins from four separate online marketplaces and combines it into one unified feed. The marketplaces covered are Skinport, CSFloat, Buff163, and the Steam Community Market, each of which protects itself differently against automated data collection, for example through Cloudflare challenges, login tokens, or strict rate limits. The README explains that this repo shares the system's design and some sample data, but the actual code that collects the data is kept closed source and instead powers a paid subscription product at a separate website. What is shown here is a detailed explanation of the infrastructure choices: a pool of 20 separate residential internet connections is used so requests do not all appear to come from one place, each connection is checked to make sure it has a unique identity before use, and traffic is slowed down automatically as any single connection gets close to a marketplace's rate limit, so it never crosses that limit and gets blocked. If a connection starts failing repeatedly, the system pauses the whole pool briefly rather than pushing through errors. Once data is collected, everything flows into a local REST API with over 30 endpoints that lets someone query prices across all four markets at once, look for profitable trade-up opportunities, or scan for price differences between markets on the same item. The project is written in Python and uses a PostgreSQL database to cache results and reduce how often it needs to re-fetch data from each marketplace. Because the actual scraping and anti-bot bypass code is not included in this repository, someone browsing it can understand the overall system design and see example code snippets, but cannot run the full pipeline themselves without the private implementation.