agentops-ai/agentops

AgentOps is a Python toolkit for monitoring and debugging AI agents: programs that use large language models to take sequences of actions or make decisions autonomously. When you build something with a framework like CrewAI, LangChain, or OpenAI's Agents SDK, it can be hard to understand why it behaved a certain way, how much it cost, or where it went wrong. AgentOps records everything that happens during a run so you can review it afterward. Adding it to an existing project takes two lines of code: one to initialize it with an API key and one to mark when a session ends. From there, AgentOps automatically captures each call made to AI model providers, what was sent and received, how long it took, and what it cost. You can also add decorators to your own functions and classes to mark them as agents, tasks, or operations, which lets the dashboard show a structured breakdown of the full execution flow. The web dashboard lets you replay a session step by step, see a graph of how different parts of the program called each other, and view cost summaries across many sessions. This is useful for catching cases where an agent looped unnecessarily, where a tool call failed silently, or where spending spiked unexpectedly. AgentOps integrates with most major AI agent frameworks and model providers out of the box, including CrewAI, AutoGen, LangChain, Cohere, and LlamaIndex. The dashboard and backend are open-source under the MIT license and can be self-hosted on your own infrastructure if you do not want to send data to AgentOps' cloud service. It installs via pip and works with Python. Documentation is at docs.agentops.ai.