gysho/llmrpro

LLMrPro is a self hosted router that lets you use your own computers to run AI language models instead of always paying a cloud provider. It gives you one address to send requests to, shaped exactly like the format OpenAI uses, so any tool or app built for OpenAI can point at LLMrPro instead without changes. Behind that one address, LLMrPro spreads the actual work across a pool of machines you own, such as desktops, workstations, or servers. Each of your machines runs a small agent program that connects outward to the central router over a websocket connection, so you do not need to open any firewall ports. That agent then runs the request on a local AI engine already installed on the machine, such as LM Studio, Ollama, vLLM, llama.cpp, or a bundled engine for Mac computers. Requests are grouped into three tiers named frontier, mini, and nano, and each tier can be linked to a specific cloud provider, like OpenAI, Anthropic, Google, or Azure OpenAI, as a backup. If none of your own machines can handle a request for a given tier, LLMrPro automatically sends it to that tier's cloud provider instead, so a request still gets answered even when your hardware pool is busy or offline. The project includes a web based admin panel for managing which devices are paired to the pool, revoking access from an agent, configuring tiers, and viewing a log of dispatched requests. It also builds in security features such as rate limiting per source, protection against requests being redirected to internal addresses, encrypted storage of provider API keys, and signed tokens with expiration for each connected agent. Setting it up involves running a central balancer program on a server, which needs Node.js, MongoDB, and Meteor, along with a reverse proxy for secure connections in production. Each worker machine then installs the desktop agent, either through a one line installer script or a downloadable app for macOS or Linux, and pairs itself to the balancer using a code generated by an administrator. The README does not state a software license for this project.