alicankiraz1/gemma-4-31b-mtp-vllm-server

This project is a small Python server that sits in front of a larger AI model server and makes it easier and safer to use. The larger server is vLLM, which is the program that actually loads Google's Gemma 4 31B language model into GPU memory and answers requests. The wrapper, built with FastAPI, exposes two HTTP interfaces shaped like the ones from OpenAI and Anthropic, so existing client code written against those services can point at this server with little change. The headline feature is something called Multi-Token Prediction, or MTP. Normally a language model produces one word piece at a time. MTP uses a smaller helper model, an assistant drafter, to guess several pieces ahead, then the main model verifies them in a single pass. The README's measured numbers, taken on a machine with two NVIDIA RTX 5090 cards, show throughput rising from around 63 tokens per second without MTP to around 130 to 136 with it, roughly two times faster across runs of 250, 500, and 1000 tokens. Beyond raw speed, the wrapper adds the practical pieces that the raw vLLM process does not include. There is API-key authentication, rate limiting, controls on cross-origin requests, a limit on how many requests can be in flight at once, and rules about which network addresses the process is allowed to bind to. Health endpoints (/livez, /readyz, /health), a version endpoint, and Prometheus-style metrics make it possible to watch the service from outside. The project ships two profiles. The default, safe80, is sized for a single 80 GB-class GPU, sets tensor parallel size to 1, and aims for a 32k context window. A second profile, tp2, splits the model across two smaller GPUs. The Gemma 4 MTP feature requires vLLM version 0.21.0 or newer, since that release was the first to support it officially. vLLM itself is an optional install extra because it pulls in heavy CUDA or ROCm wheels. Getting started involves cloning the repo, creating a Python 3.12 virtual environment, installing the package, then running two commands: vllm-mtp launch to start the underlying vLLM serve process with the right speculative-decoding flags, and vllm-mtp serve to start the gateway in front of it. A vllm-mtp doctor command checks that vLLM is reachable, new enough, and serving the expected target model. The current release is described as an alpha for local or private GPU serving.