openforecaster/futuresim

Futuresim is a simulation framework for testing how well AI language models do at forecasting, meaning predicting the outcomes of real world questions like whether something will happen by a given date. Multiple AI agents each produce their own prediction on the same question, and the system scores them against the actual resolved answer, letting you compare how different models or agent designs perform at forecasting tasks. The simulator works by loading a set of questions, by default from a Hugging Face dataset called OpenForesight, presenting them to a set of agents while controlling what information each agent can access and when, to avoid leaking future information, collecting the agents' predictions, and scoring the results against ground truth. Agents can optionally be given a search tool backed by LanceDB, a vector database used for semantic search, so they can look up relevant news articles before answering. You configure runs through YAML files and an environment file that holds API keys, OpenRouter is used to call model providers, along with storage paths. You can swap in your own question datasets in CSV, JSONL, JSON, or Parquet format, and plug in alternative search backends by implementing a defined interface. This is aimed at researchers or developers who want to systematically benchmark how AI models perform at predicting future events, or who want to build and test new forecasting agent designs. It is written in Python. The full README is longer than what was shown.