raga-ai-hub/ragaai-catalyst

RagaAI Catalyst is a Python toolkit for developers who build applications powered by large language models (LLMs, AI systems like ChatGPT that understand and generate text). It helps you observe, monitor, and evaluate how those AI applications are actually performing in practice. The core idea is visibility: when you deploy an AI-powered app, things can go wrong in subtle ways, the AI might give inaccurate answers, make up facts (called "hallucination"), or behave unpredictably across different requests. Catalyst helps you catch and debug these issues by recording detailed traces of what happened during each interaction, including which tools the AI called, how many tokens were used, and what the AI decided at each step. Beyond monitoring, it includes tools for evaluating AI output quality against metrics like Faithfulness (did the response match the source material?) and Hallucination (did the AI invent information?). You can upload datasets, run evaluations, and compare results over time through a self-hosted dashboard. It also includes prompt management (storing and versioning the instructions you send to the AI), synthetic data generation for testing, and guardrails to block unsafe or off-topic responses. The toolkit is installed as a Python package and connects to a RagaAI platform account via API keys. It is aimed at developers and ML engineers building and maintaining multi-agent AI systems who need structured tooling to understand and improve their applications' behavior.