OpenEnv in Practice: Evaluating Tool-Using Agents in Real-World Environments

This post introduces OpenEnv, an open-source framework for evaluating AI agents in real-world environments, using a calendar management benchmark called the Calendar Gym.

• •OpenEnv uses a gym-oriented API (reset, step, action, observations) and standard MCP tool call interface to connect agents to real tools and APIs
• •The Calendar Gym exposes agents to realistic constraints: Access Control Lists, limited visibility into other users' state, and multi-step workflows requiring correct action chaining
• •Agents achieved ~90% success on tasks with explicit calendar identifiers, but only ~40% success when tasks used natural language descriptions
• •Over half of errors in failed interactions came from malformed tool arguments or incorrect ordering, not wrong tool selection
• •