I have been spending time lately with AI agents. Reading about them, following the discussions, and running some myself in the background to see what they actually do. The honest verdict after a few weeks of this: the automation feels impressive on first contact. You set something in motion, walk away, and come back to find work completed. That is genuinely new in terms of feel.
But once you look under the hood, most of what gets called an agent today is structurally simpler than the name implies. A markup file that orchestrates a sequence of API calls to a large language model, with some tool access layered on top. Not rocket science. In many cases, a thoughtful set of chained prompts would produce something similar. The impression of autonomous intelligence is real. The underlying architecture is considerably more modest.
I am not saying this to dismiss agents. I am saying it because the gap between how agents feel and what they actually are matters for the question I want to ask. Because whether the technology is genuinely autonomous or just cleverly automated, the organizational consequence is the same: something is now acting inside your business without a human making each individual decision. And that changes something important.
It changes who is responsible when the action goes wrong.
The difference that actually matters
A chatbot suggests. A copilot assists. A dashboard informs. An agent acts. It does not generate language for a human to evaluate and then decide. It pursues a goal, uses tools, interacts with systems, makes intermediate choices, and changes the state of the business. That is not a marginal technical improvement. It is a shift in delegation. And delegation always moves accountability somewhere.
The comforting story organizations tell themselves about agents is that they will handle routine tasks, coordinate between systems, and free people for higher-value judgment. Some of that is true and the value is real. But the productivity story skims over the deeper question. Once software starts acting inside the organization, work no longer moves only through people. Decisions no longer sit only in meetings, approvals, and managerial routines. Authority starts migrating into systems. At first this feels harmless - the agent schedules something, drafts something, summarizes something. Then it prepares actions. Then it executes within rules. Then it escalates only exceptions. Then it coordinates across systems. The change does not arrive all at once. It arrives through convenience, and because convenience feels like progress, very few people stop to ask what has actually moved.
What has moved is judgment. And with judgment, accountability becomes harder to locate.
The data is already telling the story
Deloitte’s 2026 State of AI in the Enterprise research, based on a survey of 3,235 IT and business leaders across 24 countries, found that by 2027, 74 percent of companies expect to use AI agents at least moderately, with 23 percent expecting extensive use and 5 percent planning full integration into core operations. That is a dramatic acceleration from where most organizations sit today. What makes the finding striking is what sits alongside it: only 21 percent of those same organizations report having a mature governance model for agentic AI in place right now.
That gap between deployment intent and governance readiness is the actual story. The organization discovers what the technology can do before it decides who is responsible for what the technology does. That sequence is dangerous not because agents are inherently dangerous, but because organizations were already struggling with accountability before agents arrived.
McKinsey’s 2026 AI Trust research found progress in AI trust maturity overall, but persistent gaps specifically in strategy, governance, and agentic AI controls. Only about a third of organizations report meaningful maturity in those dimensions. McKinsey Partner Rich Isenberg put the core shift cleanly: “Agency isn’t a feature -- it’s a transfer of decision rights. The question shifts from ‘Is the model accurate?’ to ‘Who’s accountable when the system acts?’”
That is the right question. Most organizations are not yet answering it.
Accountability was already blurred before agents arrived
Even before AI agents, many companies struggled to give honest answers to basic questions. Who owns this decision? Who approved this exception? Who saw the risk and chose to continue anyway? Who can stop the process when it starts producing the wrong outcome?
In theory the answers exist. In practice they dissolve into committees, handoffs, vendor relationships, dashboards, policy documents, approval rituals, and matrix structures. A bad outcome happens and everyone can explain why their part was reasonable. The process followed the policy. The system behaved as configured. The data was available. The approval was documented. And the result was still bad. This is how accountability disappears in modern organizations - not through malice, but through fragmentation.
AI agents do not fix this fragmentation. They can accelerate it, because they add speed to ambiguity.
Consider a customer service agent that can refund a customer, apply a retention offer, update the CRM, trigger a shipping exception, and flag the account for follow-up. On paper this is efficient. The customer gets an answer faster. The employee handles less repetitive work. The company reduces friction. But if the agent makes the wrong call, who owns it? The service team? The product team that defined the policy? The data team that integrated the model? The vendor that supplied the system? The manager who approved the workflow? The employee who never saw the case because the system was designed not to require their review?
Now move the same pattern into procurement, claims management, compliance, hiring, credit decisions, or cybersecurity. The question becomes sharper very quickly. McKinsey’s research found that 80 percent of organizations have already encountered risky behavior from AI agents, including improper data exposure and unauthorized system access. That is not a warning about a future problem. It is a description of the present.
Policy will not be enough
Most organizations will try to solve this with policy. Acceptable-use rules, agent registries, risk committees, monitoring dashboards, escalation procedures. These are necessary. But they are not sufficient.
The problem is not only whether the agent follows policy. The problem is where the agent sits in the work, what it is allowed to change, who genuinely understands the downstream consequences, and who has the real authority to stop it. That is not a compliance problem. It is work architecture. Agents do not sit neatly inside a single function. They cross boundaries. They touch data, systems, customers, third parties, and operational decisions simultaneously. The governance problem cannot be solved by legal, compliance, IT, or data science working in isolation. It belongs to leadership, because accountability is not a technical feature. It is a design choice.
The human-in-the-loop illusion
The phrase “human in the loop” gets used too easily and explains too little. It sounds responsible. But which human? At what moment? With what information and how much time? With what authority and what genuine obligation to intervene?
A human who receives an alert after the action is complete is not in the loop. A human who approves hundreds of low-context recommendations per day is not meaningfully in control. A human who cannot understand why the agent acted is not supervising -- they are rubber-stamping. And a human who faces implicit pressure not to slow the process will eventually stop questioning it altogether. Oversight can become theater. A person appears somewhere in the workflow, so the organization tells itself that responsibility remains human. But the real decision has already moved into the system. The question worth asking is not whether a human is present in the workflow. It is whether human judgment is placed at a point where it can still change the outcome. That is a considerably higher bar.
Why modular ownership fails with agents
Most organizations assign ownership by component. The model belongs to data science. The platform belongs to IT. The workflow belongs to operations. The policy belongs to compliance. The customer impact belongs to the business unit. That structure already creates tension in normal circumstances. With agents operating across all of those seams, it becomes a genuine failure mode.
When something goes wrong, each function can plausibly demonstrate its part worked. The model produced a valid output. The system executed correctly. The policy existed. The data was available. The workflow behaved as configured. And the business outcome was still bad. This is modular correctness. Each part performs. The whole fails.
For agentic workflows, there has to be one accountable business owner for the consequence of the workflow. Not a committee. Not an AI governance working group. Not the platform team. One person who understands what the agent is allowed to do, what it is not allowed to do, what success looks like, what risk is unacceptable, when human intervention is mandatory, and who can stop the system. Technical teams own infrastructure. Legal owns legal interpretation. Compliance owns regulatory obligations. But the business owner owns the outcome. Without that, agents become organizational orphans: useful when they work, nobody’s child when they fail.
The right to stop
Every serious agentic workflow needs clearly defined stop conditions. This sounds obvious. Most organizations will skip it because stopping feels like failure, or because nobody wants to be the one who slowed the system down. But if an agent is changing prices, what drift requires intervention? If it handles customer complaints, what pattern signals harm? If it manages procurement, what behavior indicates it is optimizing the wrong objective?
Who can pause it? Who can restart it, on what evidence, with what review? These are not bureaucratic questions. No serious company runs a production line without emergency stops. No serious company should run autonomous digital workflows without them either.
The decision that belongs to leadership
The leadership decision is not whether to use agents. That decision is largely already made across the industry, and the productivity case is real enough to drive adoption regardless. The decision is where autonomy belongs and who remains accountable for its consequences.
That decision cannot be delegated to technology teams alone. It sits at the intersection of trust, authority, risk, and organizational design. McKinsey’s framing holds: the question has shifted from whether the model is accurate to who is accountable when the system acts. The companies that answer that question clearly before scaling will move faster and more safely, because they will know where agents can operate without constant oversight and where human judgment must remain in the chain.
The companies that skip the question will also move fast. Until something breaks and the search for accountability leads nowhere useful.
That is the real cost of deploying agents without redesigning accountability first. Not that machines acted. That leaders delegated action before they had decided who owned the consequence.
