opengvlab/llama-adapter

★ 5,923PythonAudience · researcherComplexity · 4/5Setup · hard

Mindmap

mindmap
  root((repo))
    What it does
      Instruction fine-tuning
      1.2M extra params
      One hour training
    Variants
      LLaMA Adapter
      V2 image and text
      ImageBind multimodal
    Inputs
      Text instructions
      Photos and images
      Audio video depth
    Research
      ICLR 2024 paper
      Hosted demos

mindmap root((repo)) What it does Instruction fine-tuning 1.2M extra params One hour training Variants LLaMA Adapter V2 image and text ImageBind multimodal Inputs Text instructions Photos and images Audio video depth Research ICLR 2024 paper Hosted demos

Click or tap to explore — scroll the page freely

Things people build with this

USE CASE 1

Fine-tune a LLaMA model to follow custom instructions in about one hour instead of a multi-day full training run.

USE CASE 2

Build a multimodal AI that answers questions about images using LLaMA-Adapter V2 with vision input.

USE CASE 3

Extend a language model to handle audio, video, or depth inputs using the ImageBind-LLM variant.

Tech stack

PythonPyTorch

Getting it running

Difficulty · hard Time to first run · 1day+

Requires a GPU and pre-downloaded LLaMA model weights, obtaining and setting up model weights can take more than a day.

In plain English

LLaMA-Adapter is a method for customizing a large language model called LLaMA so that it follows instructions given in plain text, rather than just completing text in a general way. Large language models in their base form are trained to predict what comes next in text, but they need additional training to reliably act on instructions like "summarize this" or "translate this into Spanish." That additional training is called fine-tuning, and it normally requires a lot of time and computing resources. The key idea behind LLaMA-Adapter is that instead of retraining the full model, you only add a small set of extra parameters, about 1.2 million, into the model's existing structure. Those extra parameters learn to steer the model's behavior toward following instructions. Because so little is being trained, the process takes roughly one hour on appropriate hardware, compared to many hours for a full fine-tune of the same base model. The paper describing this method was accepted at the ICLR 2024 research conference. A second version, LLaMA-Adapter V2, extends the approach to handle both images and text together. With this, the model can take in a photo alongside a question and generate a relevant response based on the image content. ImageBind-LLM, a further extension included in the repository, widens this to additional input types such as audio, video, and depth data. The repository provides the code needed to run and fine-tune the different model variants, along with links to hosted demos where you can try the models without setting anything up locally. The training data used for each model variant is described in a comparison table in the README.

Copy-paste prompts

Prompt 1

I want to fine-tune LLaMA using LLaMA-Adapter on my own instruction dataset. Walk me through the training script, what GPU I need, and roughly how long it will take.

Prompt 2

Show me how to load a pre-trained LLaMA-Adapter V2 checkpoint and run inference on an image-question pair to get a text answer.

Prompt 3

Explain the difference between LLaMA-Adapter, V2, and ImageBind-LLM and help me pick the right variant for a project combining text and audio.

Open on GitHub → Explain another repo

← opengvlab on gitmyhub — every repo by this author, as a profile.

Verify against the repo before relying on details.