zimingttkx/quantumflow

QuantumFlow is a Python framework for scheduling AI inference workloads across a cluster of machines, aimed at running large language models efficiently across many GPUs. The README describes it as a next generation distributed inference platform that treats AI inference tasks the way Kubernetes treats pods, deciding automatically where each task should run instead of requiring someone to assign it to a machine by hand. The project's core feature is intelligent scheduling. It supports multiple scheduling strategies: Gang scheduling, which reserves an entire block of GPUs at once for very large models that need every GPU working together with minimal communication overhead, Pack scheduling, which is better suited to smaller models that can be packed efficiently onto fewer resources, and an adaptive strategy that picks the best approach automatically based on the workload. QuantumFlow also supports multiple inference backends behind one unified interface, including Hugging Face and vLLM, which the README marks as already working, alongside TGI and SGLang, which are listed as planned. Support for domestic Chinese AI accelerator hardware, specifically Ascend NPUs, is also listed as planned rather than finished. The system architecture has four layers: a gateway layer offering REST and gRPC APIs plus a Python SDK and command line interface, a scheduler layer implementing the Gang, Pack, and adaptive strategies alongside a priority queue, a cluster management layer handling node registration, service discovery, and health monitoring, and a worker pool layer that actually runs the models on hardware such as A100, 4090, and H100 GPUs. To get started, you clone the repository and run pip install to install it in editable mode, then start the service either with a bundled script or by running the CLI's serve command, after which a web interface becomes available locally. The command line tool also supports checking cluster status, listing available models, loading a specific model, and chatting with or generating text from a loaded model directly from the terminal. The project is released under the Apache 2.0 license, a permissive open source license.