devdaim6/seo-crawler-triage-engine

This is a self-hosted web crawler and SEO analysis tool aimed at teams and agencies who manage Google indexing for large numbers of pages. The core idea: instead of blindly submitting thousands of URLs to Google and hoping they get indexed, you run them through this pipeline first, and an AI evaluates each page's quality before you waste your indexing budget on thin or low-value content. When you submit a list of URLs, the tool fetches each one, parses the page content, runs a Google PageSpeed technical audit, then sends the extracted content to an AI model (either Groq in the cloud or a local Ollama model running on your own machine). The AI categorizes the content, identifies the search intent behind the page, and scores it against Google's E-E-A-T framework (Experience, Expertise, Authoritativeness, Trustworthiness). That score acts as a triage signal: pages with strong scores are worth submitting to Google, pages with weak scores need improvement first. Beyond the E-E-A-T analysis, the tool converts every crawled page into vector embeddings stored in a Postgres database using the pgvector extension. This turns your entire crawl history into a semantic search engine, so a content team can search for concepts like "articles about sustainable finance" rather than exact keywords, helping them spot topical gaps in their content library. The dashboard shows real-time crawl progress using live streaming, so you can watch each URL move through fetching, DOM audit, AI scan, and storage phases. Client workspaces let agencies keep each client's data separate, and results can be exported to CSV. Deployment is via Docker Compose, which sets up the Next.js frontend, Node.js backend, Postgres, and Redis in one command. A Groq API key is optional but recommended for speed, Ollama supports fully local and private processing.