ghandhitechnology/munbench

MunBench is a benchmark that tests whether AI language models can write naturally in Korean, not just answer multiple-choice questions about Korean facts. The creators found that existing Korean benchmarks focus on knowledge tests and fail to measure whether a model can actually produce good Korean prose. MunBench fills this gap by evaluating emotional intelligence, creative writing, and cultural understanding through generative tasks. All 158 test items were written natively in Korean, never translated, and were reviewed by critics to ensure each one separates strong models from weak ones. The benchmark has three tracks. The first track presents 50 roleplay scenarios where the model plays a character in emotionally charged, culturally specific situations, like navigating office politics after a team dinner excluded the boss. After the conversation, the model must analyze what the other person was actually feeling. The second track gives 48 creative writing prompts across forms like short fiction, essays, and poetry, targeting known weaknesses like melodramatic overwriting or fake child voices. The third track has 60 tasks testing Korean honorific shifts, idiom usage, cultural defaults, and reading subtext, including trap items that catch models hallucinating emotional undertones that are not present. Scoring works three independent ways. A rubric system uses three judges scoring each criterion on a 0 to 10 scale, three times each, with per-judge disagreement visible in the results. An Elo rating system compares pairs of anonymized outputs head to head, with three anchor models pinned at fixed ratings so adding new models later does not shift existing scores. Judge-free metrics act as tripwires, counting cliches against a curated 184-phrase list, detecting English leaking into Korean text, and measuring repetition and length compliance without relying on any LLM opinions. Running the benchmark requires a single API key routed through OpenRouter, or you can use direct provider APIs instead. The pipeline runs validation, generation, rubric judging, pairwise comparison, Elo fitting, and reporting, with each stage independently resumable if something fails partway through. A full run produces roughly 191 outputs per model. The full README is longer than what was shown.