sgl-project/sglang

SGLang is a serving framework for large language models and multimodal models, meaning it is the piece of infrastructure that sits between your application and a model and is responsible for actually running the model and answering requests. The README describes it as high-performance and aimed at low-latency, high-throughput inference across setups ranging from a single graphics card on one machine up to large distributed clusters. The README highlights a fast runtime built around a long list of optimizations: a prefix-cache mechanism called RadixAttention so repeated parts of prompts do not have to be recomputed, a zero-overhead CPU scheduler, splitting the prefill and decode stages across machines, speculative decoding, continuous batching, paged attention, several forms of parallelism, structured outputs, and serving many fine-tuned adapters in a single batch. It supports a broad set of model families, including Llama, Qwen, DeepSeek, GLM, Gemma, and Mistral, plus embedding and reward models and some diffusion image and video models. It is compatible with most Hugging Face models and exposes an interface modeled on the OpenAI API, so existing client code often works without changes. Someone would use SGLang when they need to host a model themselves and care about cost and speed, powering a chatbot, an internal AI service, or the rollout step during reinforcement-learning post-training. The README reports it powers over 400,000 GPUs and is used as a backend by several training frameworks. It is written in Python, distributed via PyPI, and runs on NVIDIA, AMD, Intel CPUs, Google TPUs, and other accelerators.