mjsabby/gemma4-intel-serve

This repository provides a way to run Google's Gemma 4 artificial intelligence models on Intel Arc graphics cards. It is built with a C++ engine at its core, paired with a Python frontend that adds features like chat formatting and streaming responses. The project is designed to work with one or two Intel Arc Pro B70 graphics cards and supports several model sizes, from smaller versions like E2B up to a larger 31B model, including text, vision, and audio inputs. A key focus of this project is running these models more efficiently than other existing tools on Intel hardware. It splits the computation work directly across multiple graphics cards to speed up processing, a method that causes performance to scale well when a second card is added. It also handles very long text inputs through specialized memory management, allowing it to process up to 262,000 tokens of context on certain models, which is significantly more than the limits found in some other common tools. The system also includes speculative decoding, a technique where a smaller helper model guesses parts of the response to speed up text generation. The project guarantees that when using a specific setting called temperature 0, the output is exactly identical to what the model would produce without this speed boost. To ensure accuracy, every computational step is checked against a highly precise reference built directly into the repository, making the results reproducible across multiple runs and different numbers of graphics cards. Users can start the server and interact with the models through a command line interface. The server supports three different communication protocols, allowing it to work with software designed for OpenAI, OpenAI's Codex-style agents, or Anthropic's Claude tools. This means existing chat interfaces and agent programs can connect to the server without requiring custom integrations. The full README is longer than what was shown.