ozymand1as/guppy

This repository is a port of an existing AI inference engine called Colibri, adapted to run a specific large language model known as Laguna S 2.1. The Laguna model, created by Poolside AI, has roughly 100 billion parameters. The original Colibri engine was built for a different, larger model called GLM-5.2 with 744 billion parameters. This fork adjusts the engine so it works with Laguna's smaller size, different memory requirements, and distinct architectural details. The project involves several technical differences from the original engine. Laguna uses fewer layers, a smaller hidden size, and a varying number of attention heads per layer ranging from 48 to 72. It also uses a mix of global and sliding window attention, a different type of positional encoding called YaNE RoPE, and a softplus gating mechanism in its router rather than sigmoid. The model weights are quantized to int4 format, taking up about 56 GB of disk space, while embeddings use int8 format. RAM usage sits around 20 GB. To get it running, a user downloads the model files, converts them to the int4 format using a provided Python script, compiles the main engine written in C, and then launches the model. There are three ways to interact with it: a simple chat script, an OpenAI-compatible API server, or direct execution of the compiled binary. An optional web frontend from the upstream Colibri project can also be built and connected to the API server. The fork includes several bug fixes that were necessary to make Laguna work correctly. These fixes address issues like incorrect gate application, config parsing for per-layer head counts, a segfault related to expert cache allocation, a buffer over-read in the router bias, and a sliding window mask problem where softmax was using zero instead of a large negative number. The README lists eight specific fixes in total. Performance on Apple Silicon Macs ranges from 0.4 to 1.5 tokens per second. The expert cache hit rate falls between 50 and 90 percent depending on prompt similarity. The project is licensed under Apache 2.0, with the model weights themselves under MIT.