ahapbean/xhc

xHC stands for Expanded Hyper Connections. It is the code and paper release for a research project about how large language models pass information between their internal layers. Normally a transformer model, the type of neural network behind most modern AI, moves information forward through a single main pathway called the residual stream. Earlier research called Hyper Connections showed that splitting this pathway into several parallel streams could help the model learn better, but past attempts topped out at four streams because adding more ran into cost and diminishing returns. This project pushes that idea further. It expands the residual pathway to sixteen parallel streams while only updating four of them at each step, keeping the extra cost manageable. To make the wider setup actually useful, the authors add two techniques: a way to enrich the information written back to the streams using nearby token context, and a sparse update method that only touches a handful of active streams instead of doing an expensive full update across all of them. A lighter variant called xHC Flash further reduces memory use by sharing computation across nearby layers. The README reports that models trained with this method score higher on a set of language, reasoning, code, and Chinese language benchmarks compared to the earlier four stream approach, while needing less extra training compute to get there. The project links to a preprint on arXiv and includes a technical report PDF, along with charts showing training loss curves, benchmark scores, and memory usage comparisons. This repository is aimed at people working on model architecture research rather than end users looking for a tool to install. The README does not include setup instructions, a code walkthrough, or example commands, only the paper, citation information, and contact details for one of the authors. Anyone interested in trying the method would need to read the technical report to understand the implementation details.