dhellmann/sacrebleu

If you're building a machine translation system, you need a way to measure how good the translations actually are. SacreBLEU is a tool that computes BLEU scores, the standard metric the research community uses to grade translation quality. The core problem it solves is that different teams often calculate BLEU slightly differently, making it nearly impossible to compare results fairly across papers or products. This tool standardizes that process so everyone is on the same page. The tool works by taking your system's translated text and comparing it against a known, high-quality reference translation. The standout convenience is that it automatically downloads and manages those standard reference test sets for you. Instead of hunting down files on your computer, you just tell it to score against "wmt17," and it handles the rest. It also applies the correct text processing rules (like how to split words and punctuation) and generates a short version string that documents exactly how the score was calculated, so others can verify your work. Machine learning researchers, NLP engineers, and anyone evaluating translation systems would use this. For example, if your team just built a new English-to-German translator, you'd run your output through the tool to get a definitive score. It also supports additional metrics like chrF and Translation Error Rate, and includes specialized tokenizers for languages like Japanese, Korean, and Chinese, which require different text-splitting rules than English. A notable design choice is its emphasis on reproducibility. The version signature it attaches to every score acts as a receipt, showing the exact settings used. As of version 2.0.0, it also outputs results in JSON format by default, making it much easier for other software to parse the scores programmatically. This focus on standardized, hassle-free computation makes it a trusted standard in the translation community.