explaingit

jhao104/proxy_pool

📈 Trending★ 23,348PythonAudience · developerComplexity · 3/5ActiveLicenseSetup · moderate

TLDR

Automatically collects, verifies, and serves working proxy IP addresses for web scraping via a simple API, storing them in Redis.

Mindmap

mindmap
  root((proxy_pool))
    What it does
      Gathers free proxies
      Verifies they work
      Serves via API
    How it works
      Scheduler collects
      Redis storage
      Web endpoints
    Use cases
      Web scraping
      Avoid rate limits
      Data collection
    Tech stack
      Python
      Redis
      Docker
    Key features
      Custom sources
      Bad proxy removal
      Random selection

Things people build with this

USE CASE 1

Build a web crawler that makes thousands of requests without getting blocked by rotating through different IP addresses.

USE CASE 2

Collect data from websites that rate-limit or ban repeated requests from the same source.

USE CASE 3

Test your application's behavior when accessed from different geographic locations using proxy rotation.

Tech stack

PythonRedisDocker

Getting it running

Difficulty · moderate Time to first run · 30min

Requires Docker to run Redis and the proxy service; API key or proxy source configuration may be needed.

Use freely for any purpose including commercial, as long as you keep the copyright notice.

In plain English

proxy_pool is a Python tool that automatically gathers and maintains a pool of free proxy IP addresses for use in web scraping. A proxy is an intermediary server that lets your program fetch web pages using a different IP address, which helps avoid being blocked by websites that restrict repeated requests from the same source. The tool works in two main parts running at once: a scheduler that continuously collects proxy addresses from several free proxy listing websites, verifies whether each one actually works, and stores the valid ones in a Redis database (a fast in-memory data store); and a small web API server that your scraper can query to get a working proxy on demand. You interact with it through simple web endpoints, ask for a random proxy, fetch all proxies, check the count, or delete a bad one. When a proxy stops working, your scraper can report it for removal and request a fresh one. The system also lets you add your own proxy sources if the built-in free ones are not reliable enough for your needs. You would use this when building a web crawler or data collection script that needs to make many requests without triggering rate limits or bans. It is written in Python and uses Redis for storage, and can be run directly or via Docker.

Copy-paste prompts

Prompt 1
How do I set up proxy_pool with Redis and start collecting proxies from free sources?
Prompt 2
Show me how to query the proxy_pool API to get a random working proxy for my web scraper.
Prompt 3
How can I add my own custom proxy sources to proxy_pool instead of using the built-in free ones?
Prompt 4
What's the best way to report a dead proxy to proxy_pool and request a fresh one in my scraper?
Open on GitHub → Explain another repo

Generated 2026-05-18 · Model: sonnet-4-6 · Verify against the repo before relying on details.