hsy23/clif-co-orchestrating-llm-inference-serving-and-fine-tuning.

CLIF is a research system for running two things on the same GPU infrastructure at once: serving live requests to a large language model (LLM), and continuing to fine tune that model in the background using a technique called PEFT, short for parameter efficient fine tuning, which updates only a small part of the model instead of retraining it fully. The idea is to use spare capacity on inference servers to keep improving the model without interrupting the requests it is already serving. The system keeps a pool of model replicas, each of which can be in a serving state, an idle state, or a combined state where it does both serving and training at once. A dispatcher routes incoming requests to available replicas while keeping request batches within safe limits. A separate launcher watches how much spare capacity, or slack, the serving replicas have, and only starts a new fine tuning round when there is enough room to do so without hurting response times. A coordinator then balances how much of that spare capacity goes to training versus serving for replicas running in the combined state. Each replica running in combined mode keeps a shadow copy of its adapter, the small file holding the fine tuned changes, training it in the background while still serving from the original active adapter. This project targets researchers and engineers working with GPU clusters who want to study or build systems that combine LLM inference and continual learning. It requires an NVIDIA GPU setup with a CUDA compatible PyTorch build, and depending on the model chosen, one or more GPUs mapped explicitly through a configuration flag. A smoke test script is included to verify the wiring with a small model before attempting larger, paper scale experiments, and results are written out as spreadsheet files covering serving, training, and GPU usage metrics.