xai-org/grok-1

This repository is the open-weights release of Grok-1, a very large AI language model developed by xAI (Elon Musk's AI company). It contains the model's weights, the numerical parameters learned during training, along with minimal example code to load and run the model. Grok-1 is a 314-billion-parameter model, making it one of the largest publicly released language models. It uses an architecture called Mixture of Experts (MoE), which means the model has 8 specialized sub-networks (experts), but only 2 of them are activated for any given piece of input text. This design makes the model more computationally efficient to run than a dense model of equivalent parameter count, since not all 314 billion parameters are used simultaneously. The repository provides a short Python script that loads a checkpoint, a saved snapshot of the model's learned weights, and generates sample text output. The code is built on JAX, a numerical computing framework developed by Google that is commonly used for machine learning research, particularly for its ability to run efficiently on GPU and TPU hardware. Running this model requires an enormous amount of GPU memory due to its size, the README notes that the model needs a machine with sufficient GPU memory, which in practice means server-grade multi-GPU hardware. You would use this repository if you are an AI researcher or engineer who wants to study the architecture of a large Mixture of Experts language model, experiment with inference code, or fine-tune the model for specific applications, and you have access to the necessary hardware. The tech stack is Python with JAX for tensor computation. Model weights are downloaded via BitTorrent or the Hugging Face Hub. The license is Apache 2.0.