aaronrzh/llm-gateway

LLM Gateway is a traffic router that sits between your application and multiple AI providers such as OpenAI, Anthropic, DeepSeek, and a few Chinese providers. Instead of writing separate code for each provider's API format, you send requests in either OpenAI or Anthropic style, and the gateway automatically translates the request to whatever format the chosen upstream provider expects, including streaming responses. The gateway picks which provider and model to use through configurable routing strategies: by priority order, round robin, lowest latency, or lowest cost. Models can also be grouped into tiers like premium, standard, or economy, so a request can automatically fall back to a similar tier if the first choice is unavailable. If a provider starts failing repeatedly, a circuit breaker temporarily stops sending it traffic and reroutes to healthy alternatives, then tests it again after a cooldown period. Other built-in features include rate limiting to control how many requests go through per second, token usage estimation and logging so you can track spend across providers, and API key management with support for both keys stored in config files and keys added dynamically through Redis. A web-based admin panel lets you manage providers, API keys, and model settings without editing configuration files directly, protected by login authentication. Under the hood it is built in Go using the Gin web framework, with PostgreSQL for storing usage data (with a file-based fallback if the database is unavailable), Redis for caching and dynamic key storage, and separate libraries for token counting and circuit breaking. Getting started involves copying an example environment file with your provider API keys, starting PostgreSQL and Redis with Docker, and running the gateway with a single command. This project would suit developers or small teams who call multiple AI providers and want one consistent endpoint, automatic failover, and centralized cost tracking, rather than building that plumbing themselves for every provider they use.