bradyfu/awesome-multimodal-large-language-models

★ 17,784Audience · researcherComplexity · 1/5Setup · easy

Mindmap

mindmap
  root((repo))
    What it does
      Curated paper list
      Dataset index
      Benchmark tracker
    Research Topics
      Instruction tuning
      Hallucination
      Chain-of-thought
    Benchmarks
      MME
      Video-MME
    Audience
      AI researchers
      ML practitioners

mindmap root((repo)) What it does Curated paper list Dataset index Benchmark tracker Research Topics Instruction tuning Hallucination Chain-of-thought Benchmarks MME Video-MME Audience AI researchers ML practitioners

Click or tap to explore — scroll the page freely

Things people build with this

USE CASE 1

Track the latest research papers on multimodal AI topics including instruction tuning, hallucination, and chain-of-thought visual reasoning.

USE CASE 2

Find datasets for training or evaluating vision-language models from a structured, frequently updated table.

USE CASE 3

Discover benchmarks like MME and Video-MME to measure how well a multimodal model understands images and video.

Getting it running

Difficulty · easy Time to first run · 5min

In plain English

This repository is a curated list of research papers, datasets, and benchmarks focused on multimodal large language models, AI systems that can understand and reason about more than just text. Multimodal means the model can work with multiple types of input, most commonly combining text with images or video. Standard large language models (LLMs) only process written language. Multimodal versions can also interpret what is in a photo, analyze a video, or listen to speech. The research area is moving fast and tracking which papers, models, and benchmarks exist is difficult. This repository maintains a structured, frequently updated table of notable research papers organized by topic. Categories include multimodal instruction tuning (teaching models to follow instructions involving images), multimodal hallucination (when models incorrectly describe what they see), in-context learning (learning from examples shown in the prompt), chain-of-thought reasoning (having the model explain its visual reasoning step by step), and evaluation benchmarks for measuring how well models understand images and video. The repository is maintained by a research group and also links to their own benchmark projects, including MME (for evaluating multimodal LLMs) and Video-MME (focused on video understanding). It also lists datasets used for training and evaluating these models. You would use this as a research reference if you are working in the AI field and want to track progress in multimodal AI, or if you need to find relevant papers or datasets for a specific aspect of vision-language model development. The full README is longer than what was provided.

Copy-paste prompts

Prompt 1

I am building a multimodal AI app that takes images and text as input. Based on the awesome-multimodal-large-language-models repo, which recent papers on multimodal instruction tuning should I read first and why?

Prompt 2

What are the most widely used benchmarks for evaluating multimodal LLMs listed in awesome-multimodal-large-language-models? Help me choose the right one for a model focused on image question answering.

Prompt 3

Summarize the key research directions in multimodal hallucination from awesome-multimodal-large-language-models. What causes models to incorrectly describe what they see in an image?

Prompt 4

I want to evaluate a new vision-language model on standard benchmarks. Which datasets from awesome-multimodal-large-language-models cover both image understanding and video understanding?

Open on GitHub → Explain another repo

← bradyfu on gitmyhub — every repo by this author, as a profile.

Verify against the repo before relying on details.