moonshotai/moonep

MoonEP is a specialized software library used when training or running very large AI models that are split across many graphics cards, or GPUs, at once. Modern large language models often use a technique called mixture of experts, where instead of every part of the model processing every piece of input, the model routes each piece of data to only a handful of specialized sub networks, called experts, spread across different GPUs. The problem this creates is that some experts naturally end up receiving far more work than others depending on what the data looks like, which slows the whole system down because everything has to wait for the busiest GPU to finish. MoonEP solves this by making sure every GPU always receives exactly the same, predictable amount of work no matter how unevenly the AI model's routing decisions turn out to be. It does this by planning, in real time, a small number of duplicate copies of the busiest experts and prefetching them onto less busy GPUs before computation starts, so the load evens out automatically. It also avoids unnecessary copying of data between memory buffers, moving information as directly as possible between GPUs to keep communication overhead low. The project includes benchmark comparisons against a similar existing library, showing that MoonEP keeps communication time and training speed stable even as the imbalance between experts grows, while the comparison library slows down and can eventually run out of memory under the same conditions. It currently supports NVIDIA GPUs, with support for another type of hardware described as still under review. Using MoonEP requires directly integrating its API into an existing AI training or inference framework, working with concepts like GPU memory buffers, weight tensors, and gradient reduction, which assumes a strong existing background in distributed machine learning systems. The README does not state a license, so it is unclear under what terms the code can be reused. This project is intended for AI researchers and engineers building or scaling very large language models across multiple GPUs, not for general developers or non-technical users.