How Much Autonomy Does an AI Agent Actually Need?
Every grant of autonomy to an AI agent should buy a better business outcome than the process you already have. Autonomy can create real value: an agent can find a route a fixed workflow would miss, recover from an unexpected error, or handle a case nobody scripted. It can also add cost, add risk, and quietly move work onto the humans who review its output. So the useful question for a product team is how much autonomy a given outcome requires, and how much your verification can safely support.
That leads to the principle this post is built on.
The Autonomy Principle: Grant an agent as much decision-making freedom as the outcome requires and your verification system can safely support.
Call it outcome-governed autonomy. Autonomy becomes a product parameter, set by the outcome, the evidence that proves it, and how reversible each action is. Design thinking is the working method for setting that parameter well.
Better models expand what agents can attempt. Better design decides what they should be allowed to do.
Why do agentic AI projects struggle to show value?
Gartner forecasts that more than 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value, and inadequate risk controls. That is a prediction, not an observed failure rate. Its stated causes are still telling. Cost, value, and risk control are all decided by how a team defines the outcome and checks it, long before model choice enters the picture.
How is task success different from outcome success?
An agent can execute every instruction correctly and still fail the business. The gap shows up when you set what the agent does next to what the business needed.
| Agent task | Business outcome |
|---|---|
| Respond to a customer ticket | The customer's problem is actually resolved |
| Generate a sales proposal | The proposal advances a qualified opportunity |
| Reconcile an invoice | Records are accurate and exceptions are resolved |
| Schedule an appointment | The customer attends a correctly booked appointment |
| Launch a marketing campaign | The campaign produces profitable conversions |
Agents get optimized around the left column because it is observable in their own logs. The right column usually lives in another system, weeks later. Evaluate at three layers:
- Execution success: did the agent complete the requested actions?
- Outcome success: did those actions produce the intended result?
- Business success: was the result worth it after costs, risks, and side effects?
Most agent dashboards stop at layer one.
Why is perceived productivity a weak signal?
In a 2025 randomized controlled trial, METR had 16 experienced open-source developers work through 246 real tasks in codebases they knew well. They expected AI tools to make them 24% faster, took 19% longer, and afterwards still believed they had been about 20% faster. The setting was specific (early-2025 tools, experts, large mature repositories), and METR said results could differ as tools improve. The durable lesson is that people can feel faster while the measured outcome moves the other way.
A convincing demo proves an agent can perform. Proving it creates value takes a baseline.
Measure value as a comparison:
Incremental agent value = net value with the agent − net value without it
where each side's net value is the value of outcomes delivered minus execution cost, human supervision, error recovery, and operational or compliance cost. An agent producing $100,000 of value at $80,000 of cost looks fine until you learn the current process produces the same value for $40,000.
The operational metric that falls out of this is cost per verified outcome. Read it alongside outcome quality, customer experience, and risk, because on its own it rewards an agent for steering around difficult cases.
What is an outcome contract?
An outcome contract is a one-page definition, written before any code, that connects the business requirement to architecture, evaluation, and autonomy decisions. It is the core tool in this approach.
The hardest line to write is "success evidence," and customer support pricing shows why. Intercom charges $0.99 per Fin outcome, counted when the customer confirms resolution, does not ask for more help after Fin responds, or Fin completes a configured procedure, including handoffs. Its help center specifies that a customer who leaves after Fin's answer without asking for more help counts as an assumed resolution and is billed.
That is a reasonable commercial definition. It also counts a frustrated customer who gives up after an unhelpful answer the same way as a satisfied one. A vendor's billable resolution, a customer-confirmed resolution, and a business-verified resolution are three different things. Outcome-based pricing gives you a contract about a metric; alignment depends on who defines success, who verifies it, and who absorbs the cost when the definition is wrong.
Klarna shows what that cost looks like. Its OpenAI-powered assistant handled about two-thirds of customer service requests, doing work equivalent to 700 full-time agents. Then CEO Sebastian Siemiatkowski told Bloomberg that cost had been "a too predominant evaluation factor", and Klarna began hiring humans again to protect service quality. Cost per conversation was easy to count. The customer outcome was what the business depended on.
Here is a worked contract for a support resolution agent. The values are illustrative.
| Component | Definition |
|---|---|
| Desired outcome | Resolve eligible customer issues without human intervention |
| Success evidence | Resolution recorded in the CRM, with no repeat contact on the same issue within 7 days |
| Failure condition | Incorrect resolution, repeat contact, or any action outside policy |
| Escalation trigger | Low confidence, a sensitive request, or ambiguity that remains after one clarifying question |
| Cost ceiling | Total cost per verified resolution below the current human-handled process |
| Verification | CRM state checks, policy rules, customer confirmation, follow-up signals |
| Reversibility | Informational answers and status lookups run freely; refunds and account changes need approval |
| Autonomy threshold | Reliability above a set bar on representative cases before scope widens |
Reversibility earns its own row. An incorrect draft sent to an internal reviewer costs a minute. An unauthorized refund costs money and trust.
How does design thinking apply to autonomous agents?
The five classic stages give you a sequence for filling in the contract.
1. Empathize: map the job, especially the handoffs
Shadow the people doing the work today. Record every point where they hesitate, ask a follow-up, check another system, or escalate. Those moments predict agent failures.
Salesforce's CRMArena-Pro benchmark makes the point with numbers: leading agents scored about 58% on single-turn business tasks and about 35% in multi-turn settings, and in a review of failed multi-turn tasks, nearly half failed because the agent never asked for crucial missing information. These are mid-2025 results and models have improved since. The design requirement holds: real users reveal needs in stages, so clarification behaviour has to be specified and tested.
2. Define: write the outcome contract
Use the template above. If it runs past one page, the scope is too wide.
3. Ideate: choose the least autonomy that meets the contract
Anthropic's guidance recommends [finding the simplest solution possible and only increasing complexity when needed](https://www.anthropic.com/engineering/building-ef
Frequently asked questions
- How much autonomy should an AI agent have?
- Give an agent the decision-making freedom its outcome requires and your verification can safely support. Actions that are easy to check and easy to reverse can run independently. Actions that are hard to verify or hard to undo, such as refunds or account changes, need tighter limits or human approval until the agent proves reliable on representative cases.
- What is the difference between task completion and outcome achievement?
- Task completion means the agent performed the requested actions, such as replying to a ticket. Outcome achievement means those actions produced the intended result, such as the customer's problem staying solved. Business success goes one step further and asks whether that result was worth its full cost, including supervision, error recovery, and risk.
- What is an outcome contract for an AI agent?
- An outcome contract is a one-page definition written before building. It names the desired outcome, the evidence that proves it, failure conditions, escalation triggers, a cost ceiling, how each risky step is verified, how reversible each action is, and the reliability threshold at which the agent's autonomy can be widened.
- What is an autonomy budget?
- An autonomy budget sets the boundaries within which an agent acts on its own: spending limits, number of external actions, time without a human check, data access, decision rights, retry limits, and which actions need approval. It turns autonomy into an adjustable product parameter that teams can expand as reliability is demonstrated.
- Why do agentic AI projects struggle to deliver value?
- Gartner forecasts that over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value, and inadequate risk controls. Those causes point to outcome definition, verification, and fit with real work, which are product design problems more than model capability problems.
Read next
Loop Engineering: The Verifier Is the Product
Loop engineering replaced prompt engineering as the core agent skill. The loop is twenty lines of code. The verifier inside it is the actual system.
SaaS Isn't Dying. It's Being Demoted.
AI won't kill your CRM. It will demote it into a back-end capability, and the real fight is over who owns the layer that runs the business.
Jev by TypeSafe: 8 Practical Use Cases for AI Teams
TypeSafe's Jev makes typed decisions instead of writing text. Here are eight places it pays off in real products, with cost math and the traps to avoid.
Get new posts by email
No spam. Just the occasional note when I publish something worth your time.