lightseekorg/tokenspeed

TokenSpeed is a tool for running large language models as fast as possible, specifically tuned for "agentic workloads", meaning AI agents that make multiple back-to-back calls, reason through steps, and generally need quick responses to keep working autonomously. The project's goal is to deliver top-tier performance while staying easy to use, aiming to match the raw speed of high-end systems like NVIDIA's TensorRT-LLM while keeping the developer-friendly feel of vLLM, a popular open-source serving framework. Under the hood, it handles the heavy lifting of inference through a few key pieces. The modeling layer lets developers describe how a model should be split across multiple GPUs without writing complex parallelism code by hand, you annotate where things go, and a compiler figures out the communication. The scheduler, which manages incoming requests and memory for intermediate results, is built partly in C++ for speed and partly in Python for flexibility, using a state-machine approach to safely juggle resources. It also includes custom-optimized math kernels, including a fast implementation of Multi-head Latent Attention (MLA) for NVIDIA's latest Blackwell GPUs. The target users are teams building production AI applications, especially agents, where latency and throughput directly impact product quality and cost. If you're running something like a coding assistant, a multi-step research agent, or a customer service bot that chains together many model calls, shaving milliseconds off each response compounds into a much snappier experience. The project claims a notable benchmark: 580 tokens per second on a massive 397-billion-parameter model (Qwen3.5), which they highlight as a speed record for agentic workloads on GPU. The main tradeoff here is that this is a specialized, performance-focused engine. It's built for teams who need to squeeze maximum speed out of expensive GPU hardware and are willing to work with a newer tool to get there, rather than relying on more established but potentially slower general-purpose serving options.