cs-whh/skilltrain-langfuse

SkillTrain Langfuse is a Python project that provides a structured workflow for improving AI agent behaviors using measurable evidence rather than guesswork. It targets developers who are building or maintaining "AI skills," which are sets of instructions or configured behaviors that tell an AI agent how to handle a particular type of task. The core problem it addresses is that AI skills often need tuning as real users expose failures and edge cases, but changing a skill without a way to measure the effect makes it hard to know whether the change actually helped. This project pairs each round of changes with a formal test run using Langfuse, an external platform that records the inputs, outputs, and scores for each test case, so a "before" and "after" can be compared side by side. The workflow follows a fixed loop. First, a test dataset is prepared with representative inputs and examples of expected behavior. An evaluator is configured, using rules, an AI-as-judge approach, or a human reviewer, to score each output. The current skill is run against all test cases to establish a baseline. Low-scoring cases and evaluator comments are then analyzed to understand exactly what is failing. A minimal change is made to the skill, and the same dataset and evaluator are run again. A change is kept only if scores improve without introducing new problems. The project is structured as a single Codex skill, meaning it is designed to be invoked from within an AI coding assistant. It includes a Python experiment template script for running test batches, reference documents for the Langfuse API, and notes for using Claude Code CLI as the agent runtime being tested against the dataset. The README is written primarily in Chinese with an English version provided separately. The project is licensed under MIT.