aglio-lab/llm-observability-tools

This project is a curated reference list rather than software to run. It gathers 120 tools used to trace, monitor, debug, and evaluate how AI language model applications and agents behave once they are in development or live use. The README explains that observability here covers several related jobs: capturing traces of what a model or agent did step by step, monitoring quality and cost over time, catching security or policy problems such as prompt injection, and connecting all of that into existing monitoring systems teams may already run for other software. Entries are grouped into eleven categories. These include full platforms that combine tracing with evaluation and prompt management in one product, open-source tools you can self-host to keep telemetry inside your own infrastructure, the OpenTelemetry standard for instrumenting once and exporting to different backends, gateways that route model traffic while adding retries and logging, tools that score live outputs for quality or safety, tools focused on debugging multi-step agents and tool calls, tools for tracking token spend and latency, and tools for monitoring security and compliance risks. A section also covers general cloud and APM suites that have added LLM-specific monitoring, plus classic machine learning monitoring tools for things like data drift. Every listed tool links to its original source, such as a repository or product page, so readers can verify details themselves, and each is marked as open source, open weights, open core, or commercial. The project also publishes the same catalog as machine readable JSON and CSV files, is reviewed monthly, and keeps a section for discontinued tools so readers know when a recommendation is no longer maintained. The project is released under the CC0 license, placing it in the public domain, and welcomes pull requests to add or update entries.