albertaworlds/japanese-corpus-syntactic-analysis-agent

This project is an AI agent for analyzing the structure of Japanese text. It takes raw Japanese writing and automatically runs it through several stages: cleaning up the text, breaking it into its smallest meaningful word parts, mapping out how those parts relate to each other grammatically, calculating numeric measures of complexity, and producing charts and diagrams from the results. The system uses a hybrid design. A large language model handles the higher level orchestration, deciding which analysis steps to run and helping interpret the results, while two established linguistics tools, MeCab with the UniDic dictionary and spaCy with a Japanese grammar model, do the precise technical work of splitting words and identifying grammatical relationships. This split matters because the language model can be swapped out for a different one through an environment variable without affecting the underlying linguistic analysis, which runs independently. The metrics it produces cover four areas: counts and variety of word forms, sentence and clause length as a measure of complexity, how far related words sit from each other in a sentence, and side by side comparison of these numbers across multiple texts. Results can be turned into dependency tree diagrams, radar charts, and bar charts, saved as image or PDF files. The authors validated their word counting method against a well known Japanese literary text, Natsume Soseki's Kokoro, and report exact matches with reference figures on several measures. Setting it up requires Python 3.12 or newer, the uv package manager, and an API key for an OpenAI compatible language model service, since the project is built to work with several different model providers rather than one fixed model. It also needs a large Japanese dictionary file, UniDic, which is roughly 500 megabytes and can be installed either as a Python package or downloaded and placed manually into the project folder. The project is released under the MIT license.