haotian-liu/llava

LLaVA (Large Language and Vision Assistant) is a research project and open-source AI model that can understand and discuss both images and text together. In simple terms, you can show it a picture and ask questions about it in plain language, and it will respond conversationally, describing what it sees, answering questions, and following instructions related to the image. The core idea is "visual instruction tuning", training an AI so it can follow human instructions when those instructions involve visual content, not just text. It connects a vision encoder (a system that understands images) to a large language model (LLM, the type of AI behind ChatGPT), allowing the combined system to reason about images and language together. The project was accepted as an oral presentation at NeurIPS 2023, one of the most competitive AI research conferences. Later versions (LLaVA-1.5, LLaVA-NeXT) improved on the original by achieving top benchmark scores while using only publicly available training data and completing training in about one day on a standard cluster of eight high-end GPUs (A100s). The LLaVA-NeXT version also added video understanding and support for newer language models including LLaMA-3 and Qwen-1.5. Researchers and developers use LLaVA as a foundation for building multimodal AI applications, things like visual question answering, image captioning, or AI assistants that can look at screenshots and explain them. It is built in Python and weights are distributed via Hugging Face.