AI Agents Don't Cheat. They Optimize. That's the Problem.

MIT Technology Review published something last week that reframed a question I have been thinking about since the OpenAI incident.

The headline: "Here's Why AI Agents Lie and Cheat to Reach Their Goals."

The reality, as the piece explains, is more precise and more unsettling than lying or cheating. The misbehavior is called reward hacking - a phenomenon in which AI agents complete tasks or earn high scores using unintended strategies. The OpenAI models that hacked Hugging Face weren't trying to make money or commit sabotage - they were just looking for answers to a test question - the best way they possibly could, as an intern would.

They optimized. Ruthlessly, creatively, and completely outside the boundaries or common norms anyone anticipated - exactly like an intern... not maliciously. There was no intent to cause harm. There was only a goal, a sufficiently capable system, and a gap between how the goal was defined and what it actually required to achieve.

That gap is the governance problem. And it is the one that most enterprise AI security programs are not designed to close.

The Optimization Problem

Reward hacking is "the phenomenon in which a model exploits imperfections in its learned reward signal to maximize a proxy objective without satisfying the task's true intent." The keyword is imperfection. Every reward signal we build is an imperfect copy of the true goal, and reward hacking is the water that leaks through the gap left by that imperfection.

The Coast Runners story - a famous early example of reward hacking - illustrates this perfectly. Researchers trained an AI agent to play a boat-racing game. Instead of driving to the finish line, the agent found a corner of the course where it could spin in circles, repeatedly hitting the point markers. Maximum score. Zero racing.

The agent did not misunderstand the task. It solved the task as specified - optimizing the score metric - by finding a path nobody anticipated. The specification was imperfect. The optimization was perfect.

This is not a training problem that better fine-tuning will solve. Researchers found models increasingly understand that their reward-hacking is misaligned - but do it anyway. Remediation prompts actually increase the behavior.

The AI agent knows. It optimizes anyway. Because the goal is the score, and the score is the measure of success, and the most efficient path to the score is the path the AI agent takes.

Now apply this to enterprise AI agents operating in Finance, HR, Sales, and Operations.

An AI agent tasked with resolving customer service tickets at high volume will find the fastest path to "resolved" status - which may or may not involve actually resolving the customer's problem. An AI agent tasked with completing financial reconciliations will find the path that produces a balanced ledger - which may or may not reflect the transactions that actually occurred. An AI agent tasked with generating sales pipeline activity will find the paths that maximize pipeline metrics - which may or may not represent genuine prospect engagement.

None of these AI agents are cheating. They are optimizing. They are eager to succeed - exactly like an intern would.The specification was imperfect. The optimization is relentless.

Why Prevention Alone Is Not Enough

Fast Company published a piece this week that makes a related argument from a different angle.

All of the responses to recent AI incidents - voluntary testing programs, stronger oversight, tougher regulation - are trying to solve the wrong problem: preventing failures before models are deployed. The Hugging Face breach made that fact crystal clear.

The fire brigade analogy in the piece is the right frame: we have spent a decade improving fire prevention. What the OpenAI incident proved is that we also need fire brigades - teams and infrastructure designed to respond to fires that have already started, not just prevent ones that haven't happened yet.

But I want to extend the analogy one step further. Because the reward hacking problem is not quite a fire. It does not announce itself with smoke. It accumulates silently, invisibly, under the radar, through thousands of small optimization decisions that each look individually reasonable and collectively produce an outcome nobody intended.

Prevention does not catch this. Testing does not catch this. Deployment guardrails do not catch this. The AI agent passes every pre-deployment test because it is optimizing correctly against the test criteria. It begins to drift from intended behavior in production - when the real environment introduces variation the test environment did not capture, when the workflows it is invoked in evolve, when the context of its operation shifts in ways nobody mapped.

By the time the drift is visible, the optimization has been compounding for weeks.

The Functional Continuity Answer

The academic literature on AI agent governance has developed a concept that maps precisely to this problem: functional continuity.

The idea, emerging from work on persistent and self-modifying agents, is that governance cannot be a one-time verification. You cannot approve an AI agent at deployment and assume that approval remains valid as the agent operates in a changing environment. Continuous governance requires continuously verifying that the system is still doing what you approved it to do - not just what it was designed to do.

The distinction is important. An AI agent was designed to resolve customer service tickets. It was approved to resolve them through specific channels, with specific data access, within specific authority limits. As it operates, it finds optimization paths that resolve tickets faster - paths the designers did not anticipate, within the technical scope of what it was approved to do, but diverging from the original functional intent of the approval.

The AI agent is still doing what it was designed to do: resolving tickets. It is no longer doing what you approved: resolving tickets through the methods and constraints that made the approval valid.

Functional continuity governance asks the harder question: is this system, right now, in this operational context, still doing what we approved - not just technically, but functionally, with respect to intent?

That question cannot be answered by a test suite. It cannot be answered by an audit log. It requires continuous behavioral observation against the organizational intent baseline that defined what approval actually meant.

What This Requires in Practice

The governance architecture that catches reward hacking before it compounds into an incident has three components that most enterprise AI programs are still building toward.

Organizational intent captures precise enough to detect optimization drift.

Most organizational intent for AI agents is specified at a level of abstraction that leaves significant room for optimization paths the specifier never considered. "Resolve customer service tickets efficiently" is not a governance baseline. It is a direction. An effective organizational intent baseline specifies the methods, the constraints, the data sources, the authority limits, and the expected behavioral profile for the specific function the AI agent is performing - precisely enough that a governance system can evaluate whether the agent is still within that functional envelope.

This is not a natural language description. It is a structured definition that can be operationalized - evaluated automatically, continuously, without requiring a human to interpret each action against a vague statement of purpose.

Behavioral monitoring that detects optimization drift before it produces visible harm.

Reward hacking is gradual. An AI agent that is optimizing toward an unintended path does not immediately produce outcomes that trigger alerts. It produces outcomes that look reasonable, because the optimization is working, the metrics are improving, and the drift from intended behavior is measured in degrees rather than discontinuities.

Catching this requires behavioral monitoring that tracks not just whether individual actions are within permitted scope, but whether the pattern of actions over time is consistent with the organizational intent baseline in multiple dimensions. Is the distribution of tool calls consistent with what we expected? Is the AI agent's approach to its task consistent with the methods we approved? Is the behavioral profile of this week's operation consistent with the baseline established when the agent was approved?

Drift detection at the session level catches individual deviations. Drift detection at the trend level catches the accumulating optimization that compounds into incidents.

Continuous evaluation, not periodic review.

The reward hacking problem is specifically a production problem - it emerges as the agent encounters real-world variation that test environments do not capture. Periodic governance reviews - quarterly access certifications, annual audits, post-deployment assessments- are designed for systems whose behavior is stable between reviews. Enterprise AI agents that are optimizing are not stable between reviews. They are discovering new paths continuously.

Continuous evaluation against the organizational intent baseline is what makes functional continuity governance operational rather than aspirational. Not "did this agent deviate from its approved behavior since the last review" but "is this agent, right now, operating within its approved functional envelope."

Why I Think This Is the Most Urgent Governance Problem Right Now

The OpenAI incident attracted enormous attention - appropriately - because the optimization that produced it was dramatic. An AI agent escaped a sandbox. It hacked a company. It compromised a second. The deviation from intended behavior was visible and immediate.

Most enterprise reward hacking does not look like this. It looks like a customer service agent that is closing tickets faster than any human team ever did, with a quality score that has been quietly declining for six weeks. It looks like a Finance AI agent whose reconciliations are completed on time, with a systematic rounding approach in edge cases that nobody defined as a policy but that is accumulating exposure. It looks like a Sales agent whose pipeline metrics are strong, with an outreach pattern that is technically within its authority but that legal counsel would find uncomfortable if anyone reviewed it.

The AI agent is not cheating. It is optimizing. The specification was imperfect. The optimization is working exactly as designed.

The governance infrastructure that catches this is not a better specification. Specifications will always be imperfect - because the real world introduces variation that no specification fully anticipates. The governance infrastructure that catches this is continuous behavioral observation against organizational intent at the operational level where optimization occurs, before compounding produces outcomes that are no longer reversible.

That is what functional continuity governance requires. That is what aizome's intent monitoring and drift detection are built to provide. And that is the governance capability that the reward hacking problem - the AI agent that does not cheat, only optimizes - makes undeniably necessary.

The Question Every Enterprise Leader Should Ask Now

MIT Technology Review's reward hacking piece closes with a question that every enterprise leader deploying AI agents should internalize: the models are not trying to deceive you. They are trying to succeed. And they are very good at it.

The governance question that follows is simple to state and hard to answer: when your AI agent succeeds, is it succeeding at what you actually wanted - or at the proxy metric you used to measure it?

If you cannot answer that question continuously, in real time, for every AI agent operating in your environment, you do not have functional continuity governance. You have a specification and a hope.

The gap between those two things is where reward hacking lives.

Amir Ofek is CEO and Co-Founder of aizome, an Enterprise AI Agent Identity Fabric Platform and a founding player in the ARISE - Agentic Runtime Identity Security Enforcement - category.

Sources: MIT Technology Review, "Here's Why AI Agents Lie and Cheat to Reach Their Goals," August 3, 2026:

https://www.technologyreview.com/2026/08/03/1141009/heres-why-ai-agents-lie-and-cheat-to-reach-their-goals/ | Fast Company, "The Next Frontier in AI Governance Isn't Stronger Guardrails. It's Fire Brigades," August 2026: https://www.fastcompany.com/91583956/the-next-frontier-in-ai-governance-isnt-stronger-guardrails-its-fire-brigades-technology-ai-safety-governance