dharmamitra/dharmamitra-lexicon

Dharmamitra Lexicon is a dataset that maps individual words and phrases from ancient Buddhist texts in one language to their equivalent spans in another language. The three languages covered are Sanskrit, Tibetan, and Chinese, and the dataset captures all three pairings: Sanskrit to Tibetan, Sanskrit to Chinese, and Chinese to Tibetan. Each record pinpoints exactly where in the target sentence the translation of a given source word appears, down to the character position. This data powers a live lookup tool at lexicon.dharmamitra.org. The records come from aligned parallel texts, meaning pairs of sentences where a Buddhist canonical text in one language is matched against its translation in another. To build the dataset, each source sentence was processed to identify individual word boundaries, which is especially complex in Sanskrit and Classical Chinese. A machine learning model trained for this specific task then inserted markers into the target sentence to show which stretch of text corresponds to each source word or phrase. Any proposed match that is not a literal substring of the target sentence is rejected, which keeps the dataset free of invented alignments. Each JSONL file in the repository covers one source text paired with one target text. A file might contain, for example, all the Sanskrit-to-Tibetan span alignments for one specific Buddhist sutra. The format is plain text with one JSON object per line, so it can be read by any programming language without a special library. The license is Creative Commons Attribution 4.0, which allows you to use and adapt the data in any project, including commercial ones, as long as you credit the Dharmamitra project.