amap-ml/roleagent

Role-Agent is a research project for training AI agents to perform multi-step tasks in text-based environments. It builds on top of an existing training framework called verl-agent and adds two new training techniques that the authors call World-In-Agent and Agent-In-World. The World-In-Agent component asks the agent to predict what feedback it will receive from its environment before each action. If the prediction matches what actually happens, the agent gets an extra reward signal. The idea is that an agent which can anticipate consequences should make better decisions. The Agent-In-World component tracks which tasks the agent repeatedly fails at and steers future training to include more examples of those difficult cases. Rather than sampling training tasks evenly, it keeps a record of failure patterns and increases how often similar situations appear in new training batches. This is meant to stop the agent from ignoring problems it consistently struggles with. Both components are optional. You enable them with two configuration flags, and they layer on top of the existing training pipeline without replacing it. The repository includes ready-to-run training scripts for three established benchmark environments: ALFWorld, a text-based household task simulator, WebShop, a simulated online shopping environment, and Search-R1, a search and reasoning task. The code is aimed at researchers working on language model training and reinforcement learning from interaction. Running it requires setting up the upstream verl-agent dependencies, which are substantial. The project is released under the Apache 2.0 license.