Human-in-the-loop governance is the most widely cited answer to the enterprise AI agent risk problem. It is also the answer that fails most reliably in production.
Not because the concept is wrong. The instinct to keep a human accountable for consequential AI actions is correct. The problem is that "human-in-the-loop" as an implementation strategy breaks down in three distinct ways before most enterprises realize it has broken - and the replacement architecture is rarely in place when it does.
This piece is for practitioners designing governance programs that actually need to work at scale. Not governance that satisfies a policy document. Governance that holds up when an enterprise AI agent executes consequential workflows at machine speed across five systems, while the human who is nominally "in the loop" is in a meeting.
How HITL Breaks: The Three Failure Modes
Failure Mode 1: The Speed Problem
Human-in-the-loop governance was designed for AI systems that produce outputs for human review. A model generates a recommendation. A human reads it, evaluates it, approves or rejects it. The loop closes before any consequential action occurs.
Enterprise AI agents do not produce outputs for review. They execute actions. The actions happen at machine speed - tool calls completing in milliseconds, multi-step workflows executing in seconds, agent chains producing downstream effects before the first action has been reviewed.
Enterprise AI agents executing Finance workflows make tool calls in rapid sequence - read, query, validate, write, repeat - at a pace that makes meaningful human review of individual decisions operationally impossible.
The Klarna case - frequently cited as evidence that AI agents can replace human workers at scale - is also the clearest illustration of the speed problem. When Klarna deployed AI agents to handle customer service at the scale of 700 human equivalents, the volume and velocity of agent actions made meaningful human review of individual decisions operationally impossible. The "human in the loop" became the human who reviewed aggregate metrics and escalations, not the human who approved individual agent actions.
This is not a failure of intent. It is a structural consequence of operating AI agents at the scale and speed that makes them economically valuable.
Failure Mode 2: The Organizational Problem
The speed problem is well understood, even if it is underappreciated. The organizational problem is less discussed - and in practice, more consequential.
When CISO, Legal, Compliance, HR, and business units all own a piece of AI governance, no one owns enforcement. In many organizations, AI governance exists on paper but not in practice, and risk accumulates in the gaps between teams.
This is not a description of organizations that have not thought carefully about governance. It is a description of organizations that have thought carefully about it and still ended up here, because the governance structure they designed assigned responsibility by domain rather than by agent.
The CISO owns the security risk for every agent that touches sensitive systems. Legal owns the contracting and liability exposure for every agent that communicates externally. Compliance owns the regulatory obligations for every agent that accesses regulated data. HR owns the acceptable use policies for every agent that employee-facing workflows depend on. Business units own the workflows the agents are running.
When a Finance agent that accesses regulated financial data, communicates with external vendors, and operates on behalf of employees triggers a governance question, which of these stakeholders is the human in the loop? The honest answer, in most enterprises, is: all of them and none of them.
The accountability gap is not technical. It is organizational. And it does not get resolved by writing better policies. It gets resolved by a governance model that assigns accountability at the agent level rather than the domain level.
Failure Mode 3: The Drift Problem
The most insidious HITL failure mode is the one that looks like success until it doesn't.
An enterprise deploys an AI agent with appropriate human oversight — a documented owner, a review process for consequential actions, a clear escalation path. The agent performs well. The human owner reviews a sample of its outputs periodically and finds nothing alarming. The review cadence relaxes. The oversight becomes nominal.
Six months later, the agent is operating in workflows that have evolved significantly from its original deployment context. New integrations have expanded what it can access. New invocations have changed how it is used. The behavioral baseline the human owner formed in the first month of deployment no longer reflects how the agent actually operates.
The human is nominally still in the loop. The loop has drifted so far from the original governance intent that the oversight is providing false confidence rather than real protection.
78% of organizations have no documented policy for creating or removing AI agent identities. If there is no policy for the beginning and end of an agent's lifecycle, there is almost certainly no systematic process for detecting when the agent's operational context has drifted from what was governed at deployment.
The human in the loop is reviewing a version of the agent that no longer exists.
Why HITL Still Matters - Just Not as the Primary Control
Before describing what works instead, it is important to be precise about what HITL is actually good for, because dismissing it entirely produces governance programs that fail in different ways.
Human-in-the-loop oversight remains the right model for specific categories of agent action: irreversible operations with significant consequences (financial transactions above defined thresholds, data deletions, external communications that create legal obligations), novel situations that fall outside the agent's established behavioral profile, and escalations triggered by anomaly detection or policy violation signals.
The mistake is not having human oversight. The mistake is relying on human oversight as the primary control mechanism for the full population of agent actions - including the routine, high-volume, machine-speed operations where meaningful human review is structurally impossible.
The governance architecture that works treats HITL as a selective, targeted intervention triggered by automated controls - not as a continuous review process applied uniformly to all agent activity.
What Actually Works: Intent-First, Human-Targeted
The governance model that holds up in production is built on a different premise than HITL. Rather than asking humans to review agent actions, it asks the governance infrastructure to evaluate agent behavior continuously and surface only the signals that actually require human judgment.
Three architectural components make this work:
1. Organizational Intent as the Control Standard
The reason most HITL implementations degrade over time is that they lack a precise standard against which agent behavior can be evaluated. Without a clear definition of what the agent is supposed to do, human reviewers are making subjective judgments about whether each action "seems right" — and as volume increases and reviewer attention decreases, the quality of those judgments degrades.
Organizational intent capture replaces the subjective judgment with a structured standard. When an agent is provisioned, its organizational intent is defined in precise, machine-readable form: what business function it serves, what systems it should access, what data types it handles, and what actions it is authorized to take. This becomes the control standard against which all subsequent behavior is evaluated, not by a human reviewer checking individual actions, but by a continuous governance layer comparing observed behavior to the defined intent baseline.
The human's role shifts from reviewing individual actions to defining and maintaining the intent baseline. That is a task humans can do effectively. Reviewing 15-40 tool calls per task across hundreds of concurrent agent sessions is not.
2. Continuous Behavioral Governance as the Primary Control
With organizational intent as the standard, continuous behavioral governance becomes the primary control mechanism. Every action the agent takes - every tool call, every data access, every delegation hop - is evaluated in real time against the intent baseline before execution.
This evaluation is not probabilistic. It is not asking "does this seem consistent with the agent's purpose?" It is asking "does this specific action fall within the structured organizational intent definition for this agent?" - a deterministic evaluation with bounded error rates and auditable logic.
The consequence is that the volume of human review required drops dramatically. Rather than reviewing all agent actions, humans review the cases where the governance layer has identified a meaningful deviation from intent, actions that fall outside the expected intent envelope and warrant human judgment before execution proceeds.
In a well-designed system, this is a small fraction of total agent activity. Routine actions that fall clearly within the agent's intent baseline execute without escalation. Edge cases and potential violations are surfaced for human judgment at the moment they matter - before execution, not after consequences.
3. Human Escalation as a Targeted, Triggered Intervention
With continuous behavioral governance as the primary control, human escalation becomes what it should always have been: a targeted intervention for the specific situations that actually require human judgment.
This means defining - in advance - the conditions that trigger escalation. Not "review a sample of agent outputs weekly." Specific behavioral signals: intent deviation above a defined threshold, tool call sequences that don't match any established workflow pattern, data access in combinations that suggest scope violation, chain-level authorization context that has degraded beyond an acceptable boundary.
When those signals fire, a specific human - the agent's documented owner, not a diffuse governance committee - receives an alert with the context needed to make a decision: what the agent was doing, what the intent baseline says it should be doing, and what the specific deviation is. The human's decision is captured, audited, and used to improve the intent baseline.
The human is genuinely in the loop for the decisions that matter. Not nominally in the loop for the full volume of decisions that don't.
The Organizational Structure That Makes This Work
The technical architecture is necessary but not sufficient. The organizational failure mode- nobody owns enforcement when five teams each own a piece of governance - requires an organizational fix as well.
The governance model that resolves the accountability gap assigns ownership at the agent level rather than the domain level. Every agent has a documented human owner. That owner has authority over the agent's operational scope, receives the behavioral alerts generated by continuous governance, makes the escalation decisions when they are triggered, and is accountable for the agent's actions in the same way a manager is accountable for the actions of a direct report.
Functional teams - Legal, Compliance, HR, Security - define the policies that constrain what agents can do. The agent owner enforces those policies for their specific agent. The governance infrastructure makes enforcement possible by surfacing the signals the owner needs to act.
This structure does not eliminate the multi-stakeholder complexity of enterprise AI governance. It routes that complexity through a single point of accountability for each agent, which is what makes enforcement possible rather than diffuse.
The Governance Program That Actually Holds
The enterprises that get HITL right are not the ones that have eliminated human oversight. They are the ones that have redefined what human oversight means, and what the infrastructure underneath it needs to do.
Human judgment remains essential for irreversible actions above risk thresholds, genuine behavioral anomalies, novel situations outside the established intent profile, and escalations that require context no automated system can fully evaluate.
What human judgment does not scale to - and what governance programs that rely on it fail to cover - is the full volume of machine-speed agent actions that enterprise AI deployment at scale produces.
The governance infrastructure that fills the gap is intent-first and continuous, defining what agents are supposed to do, observing whether they are doing it, and surfacing only the deviations that require human intervention. The human is in the loop for the decisions that matter. The infrastructure governs everything else.
Nobody is the human in the loop when everybody is the human in the loop. The fix is not more humans. It is infrastructure that makes the humans that are there effective.
Chen Pipek is CPO and Co-Founder of aizome, an Enterprise AI Agent Identity Fabric Platform. He previously co-founded and led AxoniusX within Axonius and has held product and security leadership positions for over 20 years.
