cactus-compute/cactus-hybrid

Cactus Hybrid is a project about making small AI models that run on your own device better at knowing when they are wrong. A small model is fast and private, but it sometimes gives incorrect answers. Cactus trains models to recognize their own uncertainty. Every answer the model produces comes with a confidence score between 0 and 1, returned as structured data rather than extracted from the answer text. The idea is simple in practice. When the model gives an answer with high confidence, you trust it. When the confidence is low, you can send the question to a larger model instead. The README shows this as a basic conditional: if confidence is below a threshold like 0.85, ask a bigger model. This lets you keep most queries on device while only paying for cloud compute on the uncertain ones. The first model released is Gemma 4 E2B Hybrid, the smallest Gemma model. The README claims it matches Gemini 3.1 Flash-Lite on most benchmarks by routing only 15 to 55 percent of queries to the larger model, depending on the benchmark and how much the model is compressed. Compression at lower bit rates requires more handoffs to maintain quality. For example, MMLU-Pro needs 45 to 55 percent handoff at full precision and roughly 90 percent at 4-bit. The project supports several runtimes. You can use it through a Python package called cactus-compute, through MLX for Apple Silicon, through Hugging Face Transformers, or through llama.cpp with a compiled patch. Each path returns the same confidence field alongside the answer. The Transformers instructions include a note about loading weights manually, because the confidence probe reads from a path that breaks with automatic device offloading. The README also reports routing quality using a metric called AUROC, which measures how well the confidence score separates correct answers from wrong ones. A score of 0.5 is random guessing and 1.0 is perfect. Cactus Hybrid averages 0.814 across thirteen benchmarks covering text, vision, and audio tasks. Notably, the probe was trained on no audio data yet still scored 0.79 to 0.88 on audio benchmarks, which the authors argue means it is reading a general correctness signal rather than memorizing patterns.