zengxiao-he/tessera

Tessera is a project that builds the full pipeline for training a small language model and then running it efficiently, all from scratch. The central idea is knowledge distillation: you start with a larger, more capable model (the teacher) and train a smaller model (the student) to reproduce its behavior. The result is a model that runs much faster and cheaper while trying to preserve as much of the teacher's quality as possible. On the training side, the project implements its own version of the distributed training technique that splits a model's parameters across multiple processors, along with the matching optimizer. This lets training run across multiple machines without needing a pre-built framework to manage the coordination. The README notes this implementation is verified to produce numerically identical results to single-machine training. The inference (serving) side of Tessera is where much of the complexity lives. It includes a paged key-value cache, which is a memory management technique borrowed from operating systems that lets the server handle many simultaneous conversations without wasting GPU memory. There is also a continuous batching scheduler that keeps the hardware busy by constantly recombining active requests, and speculative decoding, which speeds up generation by having the model draft several tokens at once and then verify them in a single pass. For raw speed, the project includes custom GPU kernels written in Triton and CUDA, which are low-level languages for programming graphics cards directly. These replace standard library functions for attention, normalization, and matrix multiplication with hand-tuned versions. On hardware without a compatible GPU, the code falls back to standard PyTorch operations automatically. The network-facing layer is written in Rust and connects to the Python engine, handling incoming HTTP requests and managing back-pressure when the system is near capacity. There is also a separate reimplementation of the model's core logic in JAX, used purely as an independent check that the PyTorch version produces correct numbers. The project is licensed under Apache 2.0.