benmagnifico/smartfed

SmartFed is the companion repository for a paper accepted as a spotlight at the ICML 2026 conference. The work is about adapting large language models to new tasks in a setting called federated fine-tuning, where many devices each hold their own private data and the model has to be improved without that data ever leaving the device. Normally each device would train its own small adapter, then send it to a central server to be averaged in. SmartFed is built on the observation that the open-source community has already published thousands of these small adapters, called LoRA modules, for many different tasks. The slogan in the README is do not reinvent the wheel, just realign the spokes you already have. The core idea has three parts. First, an existing LoRA module is broken apart along its rank into a pool of tiny experts. Second, each device trains only a lightweight router that decides which of these experts to mix together for a given input, while the experts themselves stay frozen on disk. Third, a method called Elastic Expert Quota Allocation reshuffles how many experts each weight matrix is allowed to use, based on which matrices are pulling more weight in the current task. Because only the router is trained and sent across the network, and routers are less than 0.1 percent of the size of the full model, the cost in computation, bandwidth, and energy drops sharply. The README reports up to 10.21 percentage points higher average accuracy than competing methods, convergence about 3.95 times faster, communication volume about 31 times smaller, and energy use about 3.6 times lower. The method is also more data efficient: with only 10 percent of the training data it already beats a baseline called FedIT trained on the full set. The repository layout includes folders for the federated training loop, the core algorithm components, a pool that loads reusable LoRA modules from LoRAHub-style sources, shell scripts for three skill combinations (Chinese plus math, Chinese plus code, math plus code), and evaluation suites for the MGSM, DoIT, and GSM-Hard benchmarks. The README notes that the code, processed datasets, and pre-trained LoRA modules are not yet released, only the paper and the repository skeleton. The provided installation and launch commands will become runnable when the team publishes the official release. The license is MIT.