eburgueno/chat-rk1

This project sets up a browser based chat interface for talking to an AI language model running entirely on a small piece of home hardware called a Turing RK1, a compact Kubernetes cluster board. The point of the repository is to make use of a chip on that board called an NPU, which speeds up the part of running an AI model that reads and processes your prompt before it starts writing an answer. It does not make the model type faster once it starts replying, but it noticeably cuts how long you wait before the reply begins, especially with long prompts like pasted documents or code files. On the hardware the author tested, a long prompt that took over a minute and a half to get a first response using only the CPU dropped to about a minute with the NPU turned on, and larger models see an even bigger improvement. Everything needed to reproduce this setup is included: configuration for the operating system running on the board, an optional patched driver for a further speed boost, the container image that runs the AI model, and the Kubernetes configuration files to deploy the chat interface itself. Setting it up involves several steps: preparing the operating system image so the NPU is recognized, labeling the correct node in your cluster, either using a prebuilt container image or building your own, and then applying the provided Kubernetes configuration files to deploy both the model server and the chat website. The first time it starts, it downloads the default AI model automatically, which takes a few minutes. After that, you open the chat page in a browser with no login required by default, though the author notes you should turn login on before exposing it beyond your home network. The README is upfront that its documentation and scripts were largely written with AI assistance, though the performance numbers themselves come from real hardware tests rather than being invented.