dobriban/bh

This repository is a reproducibility bundle backing a statistics research result about the Benjamini-Hochberg procedure, a widely used method for controlling the rate of false discoveries when running many statistical tests at once. The claim being supported is a counterexample: under a specific two-sided Gaussian testing setup, the long run false discovery rate can actually exceed the 0.01 target the procedure is supposed to guarantee, contrary to what many people assume about it. The project backs this claim with two separate kinds of evidence. The first is a rigorous mathematical certificate, produced with a library called Arb that does interval arithmetic with guaranteed error bounds, proving that the false discovery rate is provably above 0.01 for the described model. The second is a large scale Monte Carlo simulation, meaning the procedure is run many times on randomly generated data at three different problem sizes, to show the same effect empirically. The README is explicit that the mathematical proof does not depend on the simulation results, the simulation is supporting evidence, not the actual disproof. Everything here is built for reproducibility. The exact Python and library versions are pinned, a script called verify_reproducibility.py checks every file's hash against a manifest and reruns the certificate, and another script can regenerate the entire experiment from scratch, including all seven simulation chunks and the resulting research paper's PDF. Random number generation is carefully seeded so that every run is deterministic and can be checked against the recorded seeds. A table of retained results shows the measured false discovery rate at each problem size along with confidence intervals and p-values, illustrating how the effect becomes more statistically clear as the problem size grows. This repository is aimed at researchers and readers who want to independently verify the paper's numerical and mathematical claims rather than at general software users.