neutralityproject/political-compass-benchmark

This project is a tool for measuring where an AI language model sits politically, producing a detailed profile rather than a single label like left or right. It works by having the model answer nearly four thousand real public opinion survey questions taken from established sources like Pew Research and the World Values Survey. Each possible answer has a pre assigned political weight across eleven different dimensions, such as economic views, social views, foreign policy, and attitudes toward religion, and the pattern of a model's answers is turned into a position on each of those scales. To make the results meaningful rather than arbitrary numbers, the tool runs each model three times: once acting neutrally, once instructed to answer as a far left persona, and once as a far right persona. The scale for each dimension is then stretched so that model's own far left and far right answers define the two ends, meaning a score is always measured relative to that specific model's own extremes rather than some universal standard. The project also builds its reference answer key using three separate AI models from three different countries and companies, so no single company or country's assumptions define what counts as a left leaning or right leaning answer. Running the benchmark requires only Python 3.9 or newer with no extra libraries, and it connects to any model server that speaks the common OpenAI style API format, including tools like vLLM, LiteLLM, or OpenAI itself. A single command runs the full process, from asking the survey questions through calculating the final calibrated scores, and results are saved as a JSON file. The README is upfront about the tool's limits, noting that results are relative to each model's own extremes rather than an absolute political scale, that testing one of the reference judge models against itself introduces some circular bias, and that the exact size of scores should be trusted less than their direction and ranking. Several example results from real models are included in the repository's results folder for reference.