atharva-kanherkar/advaita-vedanta-benchmark

AdvaitaBench is a research benchmark that tests whether a large language model can teach, interpret, and defend classical Advaita Vedanta, a school of Hindu philosophy associated with the teacher Shankara. It is not a general trivia quiz about Hinduism and not a loose test of spiritual tone. Every task is graded against reference answers, required distinctions the model must make, and specific claims the model is forbidden from making, with automatic score penalties when a model falls into a known doctrinal mistake. The project already ran a pilot in July 2026 across ten frontier language models on 84 tasks, using two AI judges to score each response and a correction for the fact that judges tend to rate their own family of models more favorably. The pilot found that the top few models were close enough that their ranking sits within the judges' disagreement, that the most useful signal came from multi turn conversations where the model faced escalating contradicting scripture, and that models did not do noticeably better on well known material than on unfamiliar material, suggesting the benchmark tests reasoning rather than memorized answers. The benchmark covers eight task families, including telling apart closely related concepts, correctly describing different levels of reality without contradicting itself, distinguishing this school of thought from other related philosophical schools, grounding claims in supplied source passages, correcting a misconception after being pushed back on, staying consistent across different framings of a question, answering an open ended question with no technical vocabulary, and holding up under a scripted style of intense philosophical questioning. To use it, you need Python 3.11 or later. After installing dependencies and adding API keys for the model providers you want to test, a small set of command line tools lets you validate the task bank, run models against the tasks, have them judged, and generate a final report with score normalization. Model providers are routed automatically based on the model name, and the results record how many tokens and dollars each run cost. The project is released under the MIT license.