nvlabs/vila

VILA is a family of AI models from NVIDIA Labs that can understand both images and text together. These are called vision language models: you give them a picture (or multiple pictures, or a video) and a question or instruction in plain text, and the model responds in text. The project covers a range of model sizes and is designed to work not just on large data center servers but also on smaller devices like the NVIDIA Jetson Orin, which is a compact computer used in robotics and edge computing. The project has evolved through several versions. The early VILA models introduced the ability to handle multiple images at once and showed strong in-context learning, meaning you could give the model a few examples of a task and it would follow the pattern without any retraining. Later versions, grouped under the NVILA name, focused on making the models faster and cheaper to run while keeping accuracy high. There are also specialized variants: LongVILA handles very long videos, VILA-HD processes high-resolution images in more detail, and VILA-M3 is fine-tuned for medical image analysis. Installing and running VILA requires a Python environment and a compatible NVIDIA GPU. The repository includes training scripts, evaluation scripts, and instructions for running the models through different backends. There are also pre-trained model weights available on Hugging Face. For users who want faster inference on consumer hardware, a quantized version of the models is available that trades a small amount of accuracy for a significant speed improvement. The code is open source under an Apache 2.0 license, but the model weights use a Creative Commons Non-Commercial license, so they cannot be used in commercial products. NVIDIA researchers and collaborators continue to publish new variants and extensions from this codebase.