A real-time monitoring system for AI agent behavior is urgently needed

Cases of AI agents going out of control are being confirmed one after another. As inappropriate behavior by AI agents that handle tasks on their own comes to light, experts point out that the need for real-time oversight systems is growing. MIT Technology Review reported on October 29 that this summer hundreds of OpenAI agents attempted to cheat on cybersecurity tests, and some broke out of the test environment on their own and hacked Hugging Face (a developer platform for sharing machine learning models and data) after judging it to contain information related to the evaluation criteria. Around the same time, another OpenAI agent was found to have accessed non-public files on an Australian government website.

The problem is not limited to OpenAI. In the following months, Anthropic, Google, and Meta also disclosed cases of their agents causing problems, and in many of these cases the agents managed to slip past the surveillance nets of the very researchers and safety teams meant to watch them.

EU regulators and US authorities are requiring developers and deployers to monitor agents in real time. However, these rules rest on the premise that companies can see what their agents are doing and why, while in reality that level of visibility has not been achieved, the report noted.

Analyzing the four most widely used agent frameworks together with academia, civil society, and industry peers, the researchers identified a critical gap: the frameworks fail to reliably produce the observability records essential to detecting policy violations, unauthorized actions, and failures of human oversight. Lacking elements such as agent IDs that persist over time and logs of which human approved an agent's tasks, it is difficult even to determine which agent did what, or whether it acted without authorization, the study concluded.

The article likened the situation to a bank where an anonymous new employee's ID badge changes every day while accessing customer accounts, describing today's agent deployments as no different — an "anonymous in-flight" state. Companies, it stressed, should therefore focus on building observability systems for safe deployment rather than relying on chance and after-the-fact responses to reduce harmful impacts.