rbadillap/ai-gateways-benchmark

This repository is a benchmarking tool that measures how quickly AI gateways respond when you send them requests. An AI gateway is a service that sits between your application and an AI provider like OpenAI or Anthropic, and this tool helps you understand the latency each step of the process adds. It runs from your own machine using only Python's built-in libraries, with no extra packages to install. The tool breaks latency down into distinct phases. It measures DNS resolution, which is looking up the gateway's address. It measures TCP connection setup and the TLS handshake, which is the secure connection negotiation. Then it tracks time to first byte, when the server starts responding, and time to first token, when the first actual content arrives. It also distinguishes between cold runs, where a fresh connection is made each time, and warm runs, where a second request reuses an already open connection. Setup is straightforward. You copy example configuration files, add your API keys, and run a single Python script. The script tests each gateway in a round-robin pattern so that timing variations throughout the run cancel out. It captures request identifiers from services like Vercel and Cloudflare as receipts. Results appear as a table showing median values for each metric, and raw per-run data is saved to a timestamped JSON file for deeper analysis. The configuration supports any OpenAI-compatible chat endpoint, with per-gateway overrides for authentication headers and custom headers. Environment variables can be referenced in the config, so sensitive values stay in a separate environment file. The example config includes a direct-to-provider baseline row plus two Cloudflare configurations, one using stored keys and one passing keys per request. The author stresses that results depend heavily on where and when you measure. They are not a global ranking. A run from a different region or day can produce entirely different conclusions. The README links to a separate methodology document covering baselines and honest presentation of results.