facebookresearch/laser

LASER is a research library from Meta AI that converts sentences into numerical representations called embeddings, with the distinguishing property that it works across more than 200 languages. The name stands for Language-Agnostic Sentence Representations. The practical consequence is that a sentence in English and its translation in French will produce embeddings that are numerically close to each other, even though the two sentences share no words. This property makes it useful for a set of tasks that require matching text across languages without a human translator involved. The library includes tools for mining parallel sentences from large text sources like Wikipedia and the web, meaning it can automatically find pairs of sentences across different languages that say the same thing. Those mined pairs can then be used to train translation systems. The simplest way to use it is through a pip-installable package called laser_encoders, which supports two families of models called LASER-2 and LASER-3. LASER-2 uses one encoder for all supported languages, while LASER-3 provides 147 language-specific encoders. A few lines of Python code are enough to load a model and turn a list of sentences into numerical vectors. The full kit includes more dependencies for advanced use cases, including tools for fast nearest-neighbor search and Chinese and Japanese text segmentation. The repository also contains several research tasks showing how the embeddings have been applied, such as cross-language document classification and speech-to-speech translation mining.