frankkk96/flashqwen

FlashQwen is a C++ and CUDA inference engine for running Qwen3-8B, a large language model, directly on an NVIDIA GPU. The entire project is built from scratch with no external library dependencies beyond the CUDA toolkit and the C++ standard library. There is no PyTorch, no cuBLAS, no tokenizer library, and no third-party JSON parser. Every component, including the tokenizer, the tensor file reader, the matrix math kernels, and the JSON parser, is written in this codebase. The project was built for learning purposes and totals roughly 1,940 lines of code. The intended use is running Qwen3-8B in interactive multi-turn chat on a local GPU, or running a built-in benchmark mode that measures how fast the engine processes different prompt lengths. You point the tool at a directory containing the model's weight files, which you download separately from Hugging Face using Git LFS, and it loads them at startup. The engine quantizes the matrix multiplication weights from 16-bit to 8-bit integers in memory at load time, which reduces the memory footprint from roughly 16 gigabytes to about 9 gigabytes and improves decode speed. On an RTX 4090 GPU, the benchmarks show the engine decoding at around 100 tokens per second for short to medium contexts. The README walks through an optimization study across eight stages, starting from a scalar math baseline at 49.7 tokens per second and tracing each gain: tensor-core prefill, vectorized matrix-vector multiply for decode, an in-GPU argmax step, INT8 weight quantization, and a CUDA graph to reduce per-step overhead. Each stage is a tagged commit on a separate branch so any version can be checked out and measured. The project requires a CUDA toolkit (version 12.x recommended) and an NVIDIA GPU with at least 20 gigabytes of VRAM. It builds in about 9 seconds with CMake and produces a binary under 2 megabytes. FlashQwen supports any dense Qwen3 model from 0.6 billion to 32 billion parameters, mixture-of-experts variants, multimodal variants, and non-Qwen architectures are not supported.