gocolly/colly

Colly is a web scraping and crawling framework written in Go (also called Golang). Web scraping means automatically visiting websites and pulling out structured information from their pages, the same information you would otherwise copy manually. The problem it solves is that writing code to crawl websites from scratch involves a lot of repetitive work: handling cookies, managing how fast you visit pages, dealing with different text encodings, and respecting a site's robots.txt rules (a file that tells bots which pages they may access). Colly handles all of that for you. You give Colly a starting URL and write callback functions, small pieces of code that run when Colly finds a link, downloads a page, or encounters an HTML element matching a selector (like all anchor tags or all product prices). Colly then follows links, manages sessions, and can run multiple requests in parallel. It can cache pages locally so you don't re-download them, and it supports distributed scraping across multiple machines. You would use Colly if you need to gather data from websites for analysis, build a search index, archive content, or monitor pages for changes. It works for anything from a quick one-off script to a large-scale data pipeline. The language is Go, no other runtime or framework is required.