Analysis updated 2026-05-18
Check whether a trading strategy's backtest results are statistically meaningful or just noise.
Learn how to properly split data into training and test halves to avoid fooling yourself with a strategy.
Reuse the permutation and bootstrap testing code to validate a different trading idea.
See a worked example of debunking a viral trading claim with real data.
| mykola-quant/volatility-breakout-falsification | 0xhassaan/nn-from-scratch | 3ks/embedoc | |
|---|---|---|---|
| Stars | 0 | 0 | — |
| Language | Python | Python | Python |
| Last pushed | — | — | 2023-06-08 |
| Maintenance | — | — | Dormant |
| Setup difficulty | moderate | moderate | hard |
| Complexity | 3/5 | 4/5 | 1/5 |
| Audience | researcher | developer | developer |
Figures from each repo's GitHub metadata at analysis time.
Requires installing yfinance and fetching market data before running the validation scripts.
This project tests a popular stock trading idea called volatility breakout trading, where a trader buys or sells when the price moves a certain distance away from yesterday's closing price. The idea went viral online after someone claimed they had tested thousands of versions of this strategy and found a way to remove the risk of the results being a fluke, simply by changing how trades are exited rather than how they are entered. This repo checks that claim honestly on real market data, using a strict process: pick the best settings on one half of the data, then test those exact settings on the other half without changing anything. The conclusion is that the strategy does not hold up. On two different Nasdaq-100 data sources, the strategy looked profitable when tested on new data, but that apparent profit was not reliably different from random noise once the numbers were checked with statistical significance tests. In other words, a seller of a trading course might stop at the promising-looking result, but a more careful check shows the gain could easily have happened by chance. The project also separately tests the claim that avoiding trades over the weekend saves money. It finds that holding positions over the weekend actually made a small amount of money on average in this data, not lost money, but that result also was not statistically reliable. A third claim, that adjusting exits instead of entries removes the risk of accidentally overfitting a strategy to past data, is also rejected. The project shows that moving the tuning to exits does not remove the overfitting problem, it just moves it somewhere else. The code includes a strategy engine, a data loader for fetching Nasdaq futures and index prices, and a validation script that runs permutation tests, bootstrap confidence intervals, and a random-walk control test to confirm the testing method itself is not inventing effects that are not there. It is written in Python and depends on the yfinance library for downloading market data. The full README also lists honest trading cost assumptions used in the test.
A rigorous statistical test showing a popular stock trading strategy that looked profitable does not actually beat random chance.
Mainly Python. The stack also includes Python, yfinance, pandas.
Setup difficulty is rated moderate, with roughly 30min to a first successful run.
Mainly researcher.
This repo across BitVibe Labs
Verify against the repo before relying on details.