fxsjy/jieba

Jieba is a Chinese text segmentation library for Python. The challenge it solves is fundamental to processing Chinese text on computers: unlike English, written Chinese does not place spaces between words, so before you can analyze or search text you need software to figure out where one word ends and the next begins. Jieba does that work reliably and quickly. The library offers several segmentation modes suited to different tasks. Precise mode tries to split a sentence into the most accurate set of words, which is best for text analysis. Full mode scans the sentence and extracts every possible word combination very quickly, though it cannot resolve ambiguous cases. Search engine mode works on top of precise mode by further splitting long words, which is useful when building search indexes where you want to match partial phrases too. There is also a deep-learning mode based on PaddlePaddle for higher accuracy and part-of-speech tagging. Under the hood, jieba builds a directed acyclic graph of all possible word combinations in a sentence, then uses dynamic programming and a Hidden Markov Model (a statistical technique for inferring hidden states from sequences) to find the most probable segmentation. Beyond splitting text into words, jieba supports keyword extraction using two approaches: TF-IDF, which scores words by how distinctive they are to the document, and TextRank, which scores words by how centrally they connect to other words. It also returns the part of speech (noun, verb, location name, etc.) for each word, and can tell you the exact character positions where each word starts and ends. You would use jieba in any Python project that processes Chinese text: search engines, chatbots, sentiment analysis pipelines, document classifiers, or natural language processing research. The entire library is Python, compatible with Python 2 and 3, and installable via pip.