michelecampi/inferscope

Inferscope is a command-line profiler written in Rust for measuring how well a local AI language model inference server is actually performing. If you run an LLM server yourself, the kind that exposes an OpenAI-compatible API, inferscope tells you not just whether it responded, but exactly how long each step took and how much system memory and CPU it consumed while doing so. The tool works by sending requests to any inference server that speaks the standard OpenAI API format (the README names llama.cpp's server, vLLM, Ollama, and others as compatible targets), then recording the timing of each individual token as it is generated. From this it derives: how long until the first token appeared (time to first token), the average and distribution of the gap between tokens (inter-token latency), the overall speed in tokens per second, and the total request duration. Simultaneously, it samples the server process's memory (RSS, how much RAM is actually in use), CPU percentage, and thread count from the Linux /proc filesystem, and correlates all of that on a shared clock so you can see resource usage alongside generation speed. Output is available as readable plain text or as a JSON document that includes the raw per-token timestamps, allowing further analysis without re-running the probe. The project is built as a Cargo workspace (Rust's multi-crate project format) split into five purpose-specific crates. It requires Rust 1.83 or newer and is currently in alpha. GPU resource monitoring is planned for a future release.