getumbrel/llama-gpt

LlamaGPT is a self-hosted chatbot you can run on your own computer or home server. It works like a conversational AI assistant, but unlike cloud services, all the processing happens on your device and no data leaves your machine. It is powered by Llama 2, an open source language model from Meta, and also supports Code Llama models for programming-related questions. The interface looks similar to ChatGPT. You type a message, the model generates a response, and you continue the conversation from there. Because everything runs locally, there is no account to create and no data sent to external servers. The tradeoff is that your hardware does the work, so slower machines produce text more slowly. A Raspberry Pi with 8GB of RAM generates roughly 0.9 words per second with the smallest model, while an M1 Max MacBook Pro generates about 54 words per second. You can choose from several model sizes. Larger models generally give better answers but require more memory and disk space. The smallest option needs about 6GB of RAM and a 3.79GB download. The largest needs 41GB of RAM and a 38.87GB download. The README includes a full benchmark table showing generation speeds across different hardware. Installation options cover a few scenarios: one-click install through the Umbrel home server platform, a shell script for M1 and M2 Macs, a Docker-based setup for any other machine, and a Kubernetes deployment for more advanced infrastructure. The project also exposes an API compatible with the OpenAI format, meaning tools built for that API can connect to it instead. The project was built by Umbrel, a company that makes home server software. It is aimed at people who want a private AI assistant without relying on hosted services.