weicj/vllm-2080ti-definitive

This is a specialized fork of vLLM, an open-source tool for running large AI language models locally. The fork is built specifically to get the most out of a pair of older NVIDIA RTX 2080 Ti graphics cards connected together via NVLink, with custom 22GB memory upgrades. The author's argument is that two such cards can be bought on the secondhand market for roughly half the price of a single RTX 3090 Ti, while providing more total memory, more processing cores, and competitive throughput for running large AI models. The two primary models this setup targets are Qwen3.6 27B and Gemma4 31B, which are both large language models in the 27-31 billion parameter range. On the described hardware, the fork reportedly reaches over 100 tokens per second for single-request decode on both models. The fork achieves this through a collection of optimizations: compressed weight formats (FP8, INT4, INT8), a key-value cache compression technique, GPU graph capture to avoid repeated overhead, and a faster attention kernel called FlashInfer adapted for the Turing GPU architecture. The fork is designed for a single user running one serious workload at a time, described as a personal-agent style use case. It is not intended for serving many users in parallel. The launcher presents three choices: which model checkpoint to load, which performance profile to use, and which port and network scope to serve on. A successful start exposes an OpenAI-compatible API endpoint, so any tool that can talk to OpenAI's API can send requests to it instead. Setup requires a source checkout, running the provided build script, and then starting the launcher. The README documents which combinations of model, quantization format, and context length are fully supported, partially supported, or known to cause problems on this hardware, based on the author's own testing.