oobabooga/textgen

TextGen is a desktop application for running large language models (LLMs), the same type of AI that powers ChatGPT, entirely on your own computer, with no internet connection required and no data sent to any external service. The problem it solves is that most people who want to use powerful AI assistants depend on cloud services, which cost money per token, require sharing your data with a provider, and can be censored or restricted. TextGen lets you download an open-source language model once and run it locally as many times as you want, privately. The application provides a chat interface similar to what you see in consumer AI products, you type a message and the model responds in real time. It supports vision (attaching images to your messages for the model to analyze), file attachments like PDFs and Word documents, tool-calling (where the model can run small Python scripts to perform actions like web searches or calculations during a conversation), and fine-tuning (training a model on your own data to customize its behavior). Technically, it runs as a local web server that you access in your browser, or as a standalone desktop app for quick startup. It supports several inference backends including llama.cpp, which is optimized for running models efficiently on consumer hardware including CPU-only machines. It also exposes an API that is compatible with the OpenAI and Anthropic API formats, so existing applications built for those commercial services can point to your local TextGen instance instead. You would use this if you want to experiment with AI models without paying per use, if you have privacy requirements that preclude sending data to the cloud, if you want to use uncensored open-source models, or if you are a developer building applications who wants a local test environment. It is written in Python and supports Linux, Windows, and macOS.