finaspirant/llm-wearable-agentic-eval-pipeline

This project is an end to end pipeline for curating, annotating, and evaluating agentic AI systems, meaning AI agents that take a sequence of actions to reach a goal rather than just answering a single question. It is demonstrated using wearable device data such as health alerts, ambient conversations, location triggers, and calendar reminders, but it is built to work with any AI system whose behavior is not perfectly predictable. The project targets a specific gap: many teams can watch what their AI agents do, but have no rigorous way to judge whether the agent did it correctly. The pipeline covers five parts. First, a synthetic data generator creates realistic wearable sensor logs while applying differential privacy through calibrated noise. Second, an inter-rater reliability calculator measures how much human or AI annotators agree with each other, using several standard statistics plus a new metric the project introduces called Path-Invariant Agreement. That metric is meant for cases where an agent can reach a correct outcome through more than one valid sequence of steps, situations where ordinary agreement scoring would wrongly count different valid paths as disagreements. Third, a step-level annotation pipeline scores each individual step in an agent's reasoning chain rather than only judging the final result. Fourth, a benchmark runner compares four different agent frameworks, LangGraph, CrewAI, AutoGen, and OpenAI Agents SDK, running the same tasks through each one and measuring things like token usage and how errors are recovered from. Fifth, an evaluation tool breaks an agent's run down into five stages: understanding the intent, planning, calling the right tools, recovering from mistakes, and the final outcome, scoring each stage separately. The codebase is mostly Jupyter notebooks with supporting Python modules, and it needs Python 3.11 or newer. A local annotation setup is included using the Argilla tool and Docker, for teams that want humans reviewing agent trajectories directly. The full README is longer than what was shown.