Cheap Mistakes Go to AI. Expensive Ones Wait.

The task I would worry about first isn't always the difficult one. Sometimes it is the routine job that everyone agreed was safe to automate, because nobody is looking at it closely anymore.

A complaint about service gets tagged as a billing question, the system sends it to the wrong queue, and the people responsible for the account never see it. That sounds easy to correct, until you ask how anyone would know it needed correcting.

I have seen that happen in the franchise network's delivery data. A complaint from a major account sat unseen by the people who needed to act for 11 days, then the client called the consultant directly, asked to cancel and subsequently left.

I cannot say the delay was the only reason they left. I can say the complaint was missed for 11 days and cancellation followed, which was enough to change how that kind of message was handled.

Within 24 hours, cancellation language, threats to leave and strategic accounts were set to bypass normal classification. Ordinary billing questions kept going through the automation, because the fix wasn't to put every message back in front of a person.

The part we had priced badly was the exception, a message that looked routine to the classifier and mattered a great deal to the client.

That is how I would decide which tasks to give AI, look at the thinking the work needs and the worst realistic consequence if a wrong result gets through. Then look at the checks, how likely they are to catch it and how long the mistake could live if they don't.

Before automation, a person doing a simple task often noticed other things while they were there. The consultant writing a follow-up could notice that nobody had created the project, the account manager reading a message could see that the customer was getting ready to leave.

Those checks weren't necessarily written into the process, they happened because a person was already reading. When you automate the task, that person may no longer be there to notice.

And the system can look healthy while the work is wrong. In one recurring, rule-bound email-to-task workflow, ten consecutive runs recorded zero failed attempts and 10 out of 10 first-pass technical completions.

An automated reviewer checked three of those runs. One had processed an email outside its allowed time window, missed a required project, and left no evidence of a requested draft, while the other seven runs hadn't been reviewed at all.

That is not a measured failure rate for the ten runs, because we don't know what was in the seven unchecked ones. It is a concrete example of a run finishing successfully in the system while failing to do the whole job.

The check now looks for the expected project, tasks and drafts after each meeting and compares them with what was promised. A draft doesn't pass just because something called a draft exists, it has to be the draft the meeting called for.

We had a harder version of this with a client status update. The creator saw a closed task and treated it as completed work, then the validator checked the message against the task system and approved it.

Both agents agreed with the record, and then the client replied, “This hasn’t been delivered.”

So the check had to move beyond whether the task was closed. It needed the delivered file, sent message, published item, whatever actual completion meant for that work, and without that evidence the status stayed unresolved.

A second AI agreeing with the first didn't solve a bad definition of done. We had to change what counted as proof.

Most owners start with a different test, can I undo the mistake? That is a useful question, and Amazon's one-way and two-way doors make the same distinction, reversible decisions can have a lighter process because you can correct them quickly.

But correcting quickly requires knowing something went wrong. The complaint could have been moved out of the billing queue on any of those 11 days, the problem was that nobody responsible knew it was there.

So I would keep the undo question and add the one it doesn't answer, what tells us we need to undo it?

Across the cases I've watched get pulled back from automation, three things kept showing up. The failure was silent, the ordinary case hid a much more expensive exception, or a judgment call was buried inside what looked like routine work.

A payment reminder is a good example of the second one. The amount can be small, but if the message goes to a strategic account that already paid, the cost is in the relationship rather than the invoice.

And the status update shows the third, pulling data and writing a message sounds mechanical until someone has to decide whether the closed task means the work was delivered. That decision was inside the job even though we hadn't named it.

Those are observed patterns, not a complete list of everything that can go wrong. They are enough to show why I wouldn't sort work by the average mistake alone.

The cost can change after the workflow is running too. Meeting summaries started as a convenient way to skim a call, then people began opening the summary instead of the recording, proposals quoted it and scopes relied on it.

The summary had become evidence for other work without anyone changing the task that produced it. You need to look at the risk again when that happens, because a wrong sentence now travels further than it used to.

Sort the thinking and the consequence separately

The first question is whether the task mostly follows rules or needs experience and judgment. Sequoia's Julien Bek uses the terms intelligence and judgment for that distinction, with writing code to a specification, testing and debugging on one side, and deciding what to build or when to ship on the other.

That is an investor's way of framing the work, not a measured boundary. Applied to a service firm, I would put tagging, matching quotes to sources and some summary work on the rules-heavy side, while pricing, choosing a direction or deciding what quality to put your name on needs judgment.

The second question is what happens if a wrong result escapes the normal checks. A job can be easy for AI to perform and still be expensive to get wrong.

If it is rules-heavy and the consequence is low, let the system run and sample the results. Internal drafts and routine tagging can fit there, provided they aren't quietly becoming evidence for something more important.

If the rules are clear but the consequence is high, let AI do the work and require proof or approval before it leaves. Invoices, quotes, client messages and status updates need a check that matches what could go wrong.

Where judgment matters but the consequence is low, AI can give you options, shortlists or a first pass at a plan, and you choose. Where both judgment and consequence are high, you decide and AI prepares the brief, as with pricing or ending a client relationship.

Better checks can earn more freedom for the system, but they don't make the underlying mistake harmless. A false invoice doesn't become cheap because a validator is good at catching it, the validator changes how often the false invoice gets through.

That is why I keep consequence separate from the quality of the controls. Otherwise we can start calling dangerous work safe because we've forgotten the condition that was keeping it safe.

An earlier piece uses checkability and undo cost to choose how much freedom to give an agent inside one task. Jina Yoon's autonomy levels at PostHog use those same two tests.

This sort comes before that, across the company's task list, what thinking does the work need and what happens if an error gets through? Once you know that, you can design the check and decide how much the agent may do on its own.

Make the check earn its place

I have watched meeting summaries move from human review to agent review, research checks start matching quotes to references automatically, and data analysis move away from heavy human checking. Competitor analysis and content creation followed a similar path as the checking layer improved.

That doesn't mean errors became impossible. It means the work no longer needed the same full human read to be useful, and the checks had taken over part of that job.

The strongest implementation I have seen uses three roles, an orchestrator assigning work, a creator producing it and a validator checking it. The creator fixes what the validator rejects, and the loop runs until the work passes or an attempt limit stops it.

But remember the status update, that loop can miss two agents agreeing with a bad source. The correction had to demand evidence of delivery, not another opinion about the same closed task.

Attempt limits are useful for another reason, they show you where the work repeatedly fails to settle. If a task keeps reaching its limit, the inputs may be broken or there may be a judgment call nobody wrote down.

Fix the input and the task may stay automated. Find a judgment call and the better job for AI may be preparing the information for a person rather than trying repeatedly to make the decision.

I don't have a universal acceptable hit rate to give you. The pattern is a reason to investigate, not a percentage another firm should copy.

And building these checks costs real time. We estimated as much as 200 hours of senior people's rework over six months, using material marked as reworked, while the system was being tuned.

That cost was small next to the manual labor removed, but it wasn't zero and it wasn't a standard budget for every firm. The time to build the checking belongs beside the time it eventually saves.

Putting a senior person in front of every output can be necessary while you learn what fails, but it isn't a free permanent answer. At volume, you either stop doing the full review or turn your senior people into proofreaders.

The work with serious consequences keeps the proof or human signoff it needs. Routine work needs a less expensive way to discover what passed through incorrectly.

Check work the system says is finished

If you only count reported failures, you're counting what somebody happened to notice. You don't know what is still out there looking successful.

The approach is to audit a random sample of completed outputs every week, comparing them with the source material and what actually happened. An escape is an incorrect result that passed all the normal checks before the audit found it.

Then keep two numbers, sampled escape rate and time to detection. The sampled escape rate is the incorrect outputs that escaped divided by the number you audited, not divided by everything the system produced.

Time to detection is how long each escaped error lived before somebody found it. A wrong update found before the client acts on it and the same update found after other work relies on it can have very different costs.

The first number tells you what passed through in the sample, the second tells you how long it had to cause trouble. Neither makes the unchecked work known, and a small clean sample doesn't prove that every output was correct.

Start with one weekly sample and follow the work all the way to reality. A status update needs delivery evidence, a meeting follow-up needs the project, tasks and draft it promised, and a quote needs the current cost table.

When you find an escape, add the smallest check that would have caught that kind of error, then keep sampling. Your normal checks cover the mistakes you know about, the audit gives you a chance to find the ones you haven't named yet.

And change the route for the expensive exception rather than assuming the entire task must go back to a person. The complaint classifier can keep handling ordinary messages while cancellation language bypasses it, payment reminders can keep running while top accounts and larger amounts get another check.

The status update can still be written automatically too, but it doesn't get to call something delivered without evidence. That is a narrower and more useful correction than declaring that AI can't write status updates.

Start with ten tasks

Write down ten tasks AI already touches in your firm and go through them with the person who used to do the work. That person often knows the checks you lost because they were the one making them without being asked.

For each task, decide whether it follows rules or needs judgment, what the worst realistic escaped mistake could cost, and what actual output proves the job is complete. Then ask how long an error could survive and which finished outputs you will sample.

If the honest answer is that you would only find out when a client complained, give that task a stronger control before you give it more freedom.

You probably won't have clean numbers at first. Make the first estimate, start the sample and improve the decision from what you find, rather than wait for a perfect picture before admitting the blind spot exists.

Sorting ten tasks can take an afternoon. Building the checks takes as long as the work requires, and the person who owns each check needs the authority and time to act on what it finds.

As that improves, consultants can spend less time copying information and writing routine first drafts, while senior people spend more time on the ambiguous cases and decisions that need their judgment. Work sold with senior signoff should keep that promise, not quietly turn into routine AI output with a different label.

This approach needs something you can check and a named person responsible for the decision to release it. If the work lives in verbal handoffs and a shared inbox with no reliable record of completion, sorting the tasks correctly won't give the system evidence it doesn't have.

While you still read most of what goes out, you are already the checking layer, although an expensive one. Before you stop reading everything, decide what will notice the mistakes you used to catch just by being there.