huggingface/text-generation-inference

★ 10,855PythonAudience · ops devopsComplexity · 4/5Setup · moderate

Mindmap

mindmap
  root((text-generation-inference))
    What it does
      Serve LLMs as API
      Batch requests
      Stream tokens
    Performance
      Multi-GPU splitting
      Continuous batching
      Fast TTFT
    Supported models
      Llama Falcon
      Other open models
    API
      OpenAI compatible
      REST endpoint
    Hardware
      Nvidia AMD Intel GPUs
      Docker deployment
    Audience
      DevOps engineers
      AI platform teams

mindmap root((text-generation-inference)) What it does Serve LLMs as API Batch requests Stream tokens Performance Multi-GPU splitting Continuous batching Fast TTFT Supported models Llama Falcon Other open models API OpenAI compatible REST endpoint Hardware Nvidia AMD Intel GPUs Docker deployment Audience DevOps engineers AI platform teams

Click or tap to explore — scroll the page freely

Things people build with this

USE CASE 1

Self-host a Llama or Falcon model on your own GPU server and expose it as an OpenAI-compatible API for your applications.

USE CASE 2

Spread a large model across multiple GPUs to serve it when it does not fit on a single card.

USE CASE 3

Stream LLM responses token by token to users so they see output appearing in real time rather than waiting.

Tech stack

PythonDockerCUDA

Getting it running

Difficulty · moderate Time to first run · 30min

Quickest start is Docker: pull the official image, pass the model name, and the server starts. Requires a compatible GPU, note the project is in maintenance mode.

In plain English

Text Generation Inference (TGI) is a server toolkit from Hugging Face for running large language models and making them available as an API. Large language models are the kind of AI that generates text, answers questions, and carries on conversations. TGI is the software that Hugging Face used internally to power its own chat and API products. The main purpose is speed and efficiency. Running a large AI model is computationally expensive, and TGI includes several techniques to serve many users at once without wasting resources. It can split a model across multiple graphics cards to handle models too large for a single one, process many incoming requests together in batches, and stream responses token by token so users see output appearing in real time rather than waiting for the full response. It is designed to work with popular open-source models including Llama, Falcon, and others. You start a server by pointing TGI at a model, and it exposes a web API that other programs can call to get text generated. The API format is compatible with OpenAI's chat format, so software already written for OpenAI can switch to a self-hosted model without major changes. The quickest way to start is with Docker: you pull the official container, tell it which model to load, and it handles everything else. Hardware support covers Nvidia GPUs, AMD GPUs, Intel GPUs, and some specialized accelerators. The README notes that TGI is now in maintenance mode. The Hugging Face team recommends newer inference engines like vLLM or SGLang for new projects going forward.

Copy-paste prompts

Prompt 1

I want to self-host the Llama 3 model using text-generation-inference on a machine with two NVIDIA GPUs. Give me the Docker command to start the server.

Prompt 2

How do I call a text-generation-inference server from Python code using the OpenAI client library?

Prompt 3

What environment variables do I set in text-generation-inference to enable tensor parallelism across 4 GPUs?

Prompt 4

TGI is now in maintenance mode. What drop-in replacement should I migrate to and what changes are needed?

Open on GitHub → Explain another repo

← huggingface on gitmyhub — every repo by this author, as a profile.

Verify against the repo before relying on details.