drewcsillag/qwen36ba3b-m1macbook

This repository provides Shell scripts to run Qwen3.6-35B-A3B, a large AI language model, as a local HTTP server on Apple Silicon Macs, with an API compatible with the OpenAI API format. The goal is to make it easy to use a capable open model locally without a cloud service. The model has an unusual architecture. GatedDeltaNet is used for 30 of its 40 layers: instead of a growing memory cache that expands with longer conversations, it maintains a fixed size recurrent state, so memory usage stays constant regardless of context length. The other layers use Mixture of Experts, where only 8 of 256 specialized sub networks are activated per token. The result is a model with 35 billion total parameters but roughly 3 billion active at a time. At 3 bit quantization it fits in about 16 GB of unified memory and runs at around 37 tokens per second on an M1 Pro with 32 GB RAM. Setup runs a single script that creates a Python virtual environment, installs MLX, Apple's machine learning framework for Apple Silicon, applies a small patch so the server accepts any model name, and downloads the model weights. The server then exposes endpoints at /v1/chat/completions and /v1/completions, so any tool configured for an OpenAI compatible API works without changes. A thinking mode is on by default, where the model produces a chain of thought reasoning section before its final answer. This can be disabled for faster responses. Requirements are an Apple Silicon Mac, M1, M2, M3, or M4, with at least 24 GB unified memory, macOS 13 Ventura or later, and Python 3.10 to 3.12.