Feature Request: Expand ChatEnv with Popular Agent Environment Integrations

The feature: Agent env integration

Building on top of torchrl.envs.llm.ChatEnv, we should expand the environment integrations to support a wider range of agent-tool interactions. This would enable training models that can interact with various tools (browsers, shells, games) through LLMs.

## Proposed envs

Here are the key environments and frameworks to consider integrating:

• MLGym (facebookresearch/mlgym) for structured ML experimentation
• BrowserGym for web interaction environments
• MineDojo for rich Minecraft-based tasks with natural language specifications
• AgentVerse for multi-agent interactions and tool usage
• WebArena\
• SWE-bench

The implementation would extend ChatEnv with a standardized interface for tool integration, consistent reward transformations, and unified observation spaces. This would greatly benefit research in tool-using LLM agents and provide standardized benchmarking across different environments.

Start with torchrl.envs.llm.ChatEnv and review how the checked MLGym integration is represented. Then compare the proposed BrowserGym, MineDojo, AgentVerse, WebArena, and SWE-bench integrations and define the scope of a standardized tool interface, reward transformations, and observation spaces. Done is not yet precisely defined beyond supporting the selected environments consistently.