sgl-project/sglang

SGLang is a high-performance serving framework for large language models (LLMs) and multimodal models, that is, the engine that takes a trained AI model and lets applications send it requests and get answers back quickly. The problem it tackles is that running an LLM in production is very different from running one on a laptop: many users may be asking questions at once, prompts get long and repetitive, the hardware is expensive, and every millisecond matters. SGLang focuses on low latency and high throughput across setups from a single GPU to large distributed clusters. The README describes a runtime aimed at making serving efficient: a prefix-caching system called RadixAttention that reuses work across requests with shared beginnings, a low-overhead scheduler, continuous batching to keep hardware busy, support for structured JSON outputs, and several quantization formats that shrink models so they need less memory. It also supports splitting work across multiple GPUs in several parallelism styles, and serving many fine-tuned adapters at once. On the model side, the README lists broad compatibility with popular open language models like Llama, Qwen, DeepSeek, GLM, Gemma, and Mistral, plus embedding, reward, and diffusion models. It is compatible with most Hugging Face models and exposes OpenAI-style APIs. Hardware support spans NVIDIA, AMD, Intel CPUs, Google TPUs, and Ascend NPUs. You would use SGLang when you are deploying an LLM behind your own product or service and want to maximize requests-per-second and minimize delay, instead of paying a hosted API. It is written primarily in Python. The full README is longer than what was provided.