zpao/captum

Captum is a tool that helps you understand why a machine learning model made a specific prediction. When complex models like deep neural networks produce an output, it is often a mystery which pieces of input data actually drove that decision. This library lifts the hood on those models, showing you exactly which features contributed to the result and how strongly they pushed the outcome one way or another. At a high level, the library analyzes your model and assigns "attribution scores" to each piece of input data. A positive score means that specific feature helped drive the prediction, while a negative score means it pushed against it. The larger the number, the stronger the influence. It works by comparing the actual input against a neutral baseline, like an empty or zeroed-out version of the input, and mathematically tracing how the model's output changes as it moves from that baseline to the real data. It can also peek inside the model to show which internal components, or "neurons," are doing the heavy lifting for a given prediction. Machine learning developers and researchers are the primary users. A developer building a movie recommendation system, for instance, could use this to figure out why the system keeps suggesting a specific film, was it the user's past viewing history, the genre, or the actor? If a model starts making unexpected or biased decisions, engineers can use these tools to pinpoint exactly which inputs are causing the problem and fix the underlying logic. Researchers can also use it to test and compare different interpretability algorithms against each other. The project also includes a web-based visualization interface called Insights, which lets users interactively explore these attribution scores. It is worth noting that the library is still in beta and under active development, meaning features are still evolving. It also requires a specific machine learning framework called PyTorch to function, as it was built specifically to work with PyTorch models.