orange2019220/relupruner

ReluPruner is a research tool that reduces the number of a specific type of operation inside a trained neural network, called a ReLU activation, while keeping the network's accuracy roughly the same. The reason to do this is that ReLU operations are expensive to handle when running a neural network privately using cryptographic techniques. Two of those techniques are secure multi-party computation and homomorphic encryption, which let someone run AI model predictions on sensitive data without revealing the data to the server. Both methods become much faster when the model uses fewer ReLU operations. The approach works in two stages. First, a full-size teacher model is trained normally. Second, a smaller student model is trained to imitate the teacher while gradually cutting away the least important ReLU activations. To decide which ones to remove, the method uses a scoring formula based on Taylor expansion, a mathematical way to estimate how much each activation contributes to the model's output. The pruning happens progressively during training rather than all at once, so the student model can adjust at each step. The README reports that this method can remove 90 to 95 percent of ReLU operations while keeping accuracy competitive on standard image classification benchmarks. The code is written in Python and uses PyTorch. It works with the CIFAR-10, CIFAR-100, and Tiny ImageNet image datasets and the ResNet18 model architecture. The repository includes scripts for training the teacher, training the pruned student, and validating the result. Parameters for controlling the pruning target ratio and the balance between distillation loss and task loss are passed as command-line flags. The work is associated with a paper published at the AAAI 2026 conference.