jzhang38/tinyllama

TinyLlama is a research project that trained a small but capable AI language model from scratch. The model has 1.1 billion parameters, which is much smaller than most modern AI systems, and it was trained on 3 trillion pieces of text using 16 powerful GPUs over roughly 90 days. Training started in September 2023 and completed in late December 2023, with checkpoint releases posted throughout the process so researchers could track progress. The model follows the same design as Meta's Llama 2, which means it can slot into many existing tools and projects that were already built to work with that architecture. Its small size is the main selling point: because it requires less memory and processing power than larger models, it can run on devices with limited resources. The README mentions specific uses such as helping larger models generate text faster through a technique called speculative decoding, running translation or conversation features on phones or embedded hardware without needing an internet connection, and powering character dialogue in video games. The 4-bit compressed version of TinyLlama weighs only about 637 megabytes, which is small enough to fit on most consumer devices. A chat-tuned version was also released alongside the base model, trained further on conversation data so it responds more naturally to questions and instructions. Both the base model and chat versions are available for download through Hugging Face. The training code itself is designed to be fast and is offered as a reference for anyone who wants to study how to train a smaller language model from scratch without needing a massive cluster. It uses several optimization techniques to speed up training on multiple GPUs working together. The project is open source and the model weights are publicly available, though the README notes it is primarily aimed at researchers and developers comfortable with machine learning workflows rather than general end users.