lamm-mit/substrates

This repository contains the research code and data for a project that studies how an open-weight language model, Gemma, represents materials science concepts internally. The researchers apply a technique called the Jacobian Lens to examine and influence what happens inside the model when it processes prompts about materials and their properties. The project covers three areas: readable concepts that the model encodes, relational state changes that show how representations shift, and causal interventions that test whether changing internal signals alters the model's output. The code is organized into a Python package with tools for fitting the lens, reading out model states, analyzing results, generating reports, and running causal swap experiments. It includes frozen prompt manifests that specify exactly which questions the model is asked, experiment scripts with fixed protocols for reproducing the study, and offline tests. The package also bundles the Jacobian Lens runtime so the installation is self-contained. The authors note that the package does not expose a model's private chain of thought, it measures model activations and behavior. Companion resources on Hugging Face hold the fitted lens checkpoints, extracted state vectors, and the full prompt corpus organized into 21 named configurations. These are kept separate from this GitHub release, which is source only. The prompt corpus is divided into development, evaluation, falsification, and exploratory groups, and the models used may require accepting a license before downloading. Installation requires Python 3.10 or newer and supports both virtual environments and Conda. A small demonstration command downloads GPT-2, fits a tiny lens on eight records, and runs basic mechanics examples as a smoke test. The full paper protocol is much more demanding: it fixes 1,000 fitting records, targets specific model layers, requires at least 50 evaluation items per distribution, and expects both model and dataset revisions to be pinned. The authors describe this as a compute-intensive workflow, and each fitted lens file requires a provenance sidecar recording its metadata to be valid for any paper-level claim. The README is detailed and includes installation instructions, workflow descriptions, and references to additional documentation files for prompt schemas and CLI conventions. The full README is longer than what was shown.