quixiai/embeddinggemma.c

This repository is a small, specialized server for generating embeddings from a specific model called EmbeddingGemma 300M. Embeddings are lists of numbers that capture the meaning of text, letting software compare passages by similarity. The server is written in C and runs on a wide range of hardware: standard CPUs, Apple Metal, NVIDIA CUDA, AMD ROCm, and Intel XPU SYCL. Each platform gets its own native executable, and the CPU version is roughly 100 kilobytes in size. The creators describe performance as the core selling point. The README benchmarks the project against llama.cpp, a well-known inference tool. On an Apple M5 Max, using identical model files and settings, this server produced embeddings faster in every tested scenario, with an average speedup of 1.25x and peaks reaching 2.01x. The benchmarks cover combinations of token counts and concurrency levels. The repository links to its methodology and raw results, and a command is provided so others can reproduce the comparison. Setup is straightforward. Running the executable starts the server, and the model downloads automatically on first launch to a cache directory. The server listens on port 42666 and exposes an HTTP API. Users send text to an endpoint and receive a list of floating-point numbers back. There is also an OpenAI-compatible endpoint, which means existing tools and SDKs built for OpenAI's API can talk to this server with minimal changes. The server supports Matryoshka embeddings, meaning you can request outputs at 768, 512, 256, or 128 dimensions. Lower dimensions trade some accuracy for smaller storage and faster comparison. Text is embedded exactly as provided, so the README points users to specific prompt prefixes for search queries and documents. An installer script detects the host platform and picks the right backend automatically, falling back to CPU if a specialized accelerator is not available. Performance tables list absolute throughput numbers across several hardware configurations. For example, on an AMD Instinct MI300X the server reaches 206,011 input tokens per second for 2,048-token requests, and a packed batch of 72 single-token requests hits 26,014 requests per second. The README notes that a fair cross-project comparison against other inference servers is planned for a future publication.