fxyz666/logicpipe

LogicPipe is a Python project for running large language models (AI text-generation models) across multiple GPUs or multiple edge devices working together. The README is written in Chinese. The core problem it addresses is that large AI models often do not fit on a single GPU, so the model's layers need to be split across several devices and each device handles its assigned portion. LogicPipe automates that splitting and the coordination between devices during inference. The project divides a model into stages, where each stage is a slice of the model's transformer layers. A tool called model_partition.py cuts the full model weights into separate files, one per device. When running inference, each device loads only its own file. The devices communicate through PyTorch's distributed computing library, passing intermediate results (called activations) from one stage to the next until the final output is produced. A key challenge with this kind of pipeline is that devices can sit idle waiting for the previous stage to finish. LogicPipe addresses this by breaking a complex request into smaller sub-tasks with dependencies tracked as a directed acyclic graph (a DAG, which is a flowchart where tasks point to the next tasks they unlock). A scheduler watches which sub-tasks are ready to run and feeds them into the pipeline. When a sub-task finishes, its intermediate computation (KV cache) is saved so that follow-on sub-tasks can reuse that context without reprocessing earlier steps. Other features include an offline planning step that estimates compute, communication, and memory costs before a run begins and saves the plan for reuse, a speculative decoding approach called Medusa/MBSD that can generate multiple candidate tokens at once to speed up output, and support for 4-bit and 8-bit quantization to reduce memory usage on devices with limited GPU memory. The project is aimed at researchers experimenting with multi-device LLM pipeline inference and developers studying DAG-based scheduling, context reuse, and parallel decoding. It is not packaged as a user-facing product but rather as research scaffolding that requires familiarity with distributed PyTorch setups. No license is mentioned in the README.