reezanahamed/nano-llm-gateway

This project is a small, deliberately simple example of how an LLM gateway works. In plain terms, an LLM gateway is a piece of software that sits between your app and multiple AI providers like Claude, GPT, Gemini, or DeepSeek, so your app only needs one key instead of juggling a separate key and format for each provider. Popular gateways such as LiteLLM or OpenRouter do this at large scale, but their codebases are big and hard to learn from. This project packs the same core idea into a single Python file of about 300 lines, meant to be read from top to bottom rather than deployed to production. Once running, the gateway starts a local server, and any tool built to talk to OpenAI's API can point at it instead, just by changing one setting. Behind the scenes it looks at the model name in the request, figures out which provider that model belongs to, and translates the request into that provider's own format, then translates the answer back into the common format your app expects. If a provider call fails, it automatically tries the next one on a fallback list. It also tracks how many tokens each request used and calculates the cost in dollars for every call, using a built in price table. Getting started involves cloning the repository, installing one dependency, copying an example environment file and adding at least one API key you already have, then running the gateway script, which starts a local server you leave running. A separate demo script then sends the same question to several providers side by side so you can compare their answers. The author is upfront about what this project is not. It has no streaming responses, no rate limiting, no retry logic, and no request queueing. Its price table is hardcoded, so costs shown may go stale as providers change pricing, and everything runs in a single process with nothing saved between runs. It is meant purely to teach how a gateway works. For actual production use, the README points readers toward LiteLLM instead. The project is released under the MIT license.