The 23× AI Cost Cut We Couldn't Call a Win

I moved one of our AI workflows to a cheaper model, the model bill fell by about 23 times, and I still couldn't tell you whether we had made the work cheaper to deliver.

Every run finished, the dashboard stayed green, and the saving was real. But we hadn't recorded whether a person accepted the work without fixing it, or how much time those fixes took, so the part that could have changed the answer wasn't in the record.

That is the mistake I want you to avoid before you switch a whole team to a cheaper model. You need to compare what it costs to get work you can use, not just what it costs to make the model finish a run.

Our example was one recurring, rule-bound email-to-task workflow. The average model cost per attempt went from $1.5474 to $0.0659, and those numbers came from a live production database on August 5, 2026.

The earlier model completed all 11 recorded attempts and the cheaper one completed all ten. Neither period recorded a failed attempt, retry or regeneration, so if you were only watching for technical errors, the switch looked finished.

Here is the full comparison, because the unknowns matter as much as the numbers.

Measure	Before	After
Model	gemini-3.5-flash	gemini-3.1-flash-lite
Period observed	July 7–21, 2026	July 22–August 4, 2026
Workflow attempts	11	10
Average model cost per attempt	$1.5474	$0.0659
Cost range per attempt	$0.0975–$3.2411	$0.0039–$0.1277
Average LLM calls per attempt	12.82	7.50
Failed attempts	0	0
Recorded retries or regenerations	0	0
First-pass technical completion	11/11	10/10
First-pass human acceptance	Unknown	Unknown
Human review or correction time	Unknown	Unknown

Calls per attempt fell too, which was another reason to look more closely. But the last two rows stop me from saying the cheaper model delivered the same quality or saved us money across the whole job.

A system finishing without an error means it reached the end of its instructions without reporting a problem. It doesn't mean it created everything the person needed, and it doesn't mean the output was correct.

Think about a report with a required section missing, or an email draft with a claim that nobody can support. The AI workflow can finish while a person still has to repair the result before it goes to a client.

That repair is work your firm pays for. If the cheaper model leaves more of it behind, you can lower the AI token cost and move the saving into a senior person's calendar without noticing.

And I am not saying that happened in this comparison. We didn't record the time, so I can't honestly say whether review got better, worse or stayed the same.

Reviewing AI work needs to be in the cost either way. The client doesn't get a lower standard because the model was cheaper, and the person doing the correction doesn't become free because their salary was already in the budget.

OpenAI makes the same distinction in its July 2026 guidance on cost per accepted outcome, a lower-priced model can need more attempts, correction or human review. Its separate AI scorecard includes employee time, review, retries and rework in the full cost of successful work.

The calculation I would use is the whole cost of the workflow divided by the number of tasks that met the written standard. Model usage, tools, infrastructure, review and correction all go in, including the cost of failed attempts.

In a sheet, that is:

Cost per accepted task = total model, tool, infrastructure, retry, review and correction cost / accepted tasks

Count each cost once, so if your model total already includes retries, don't add those model charges again under another column. You want the actual bill for the work, not a formula that accidentally counts part of it twice.

An accepted task has met the standard. First-pass acceptance means it got there without a person correcting it, which is useful to track separately because it tells you how much repair the model leaves behind.

If a task passes after correction, it can enter the accepted count, but the correction time stays in the cost. If it never passes, it stays out of the accepted count while everything spent on it stays in the total.

That is how you avoid making a weak workflow look cheap by counting all the attempts as useful work.

There was another problem with our comparison, the two models hadn't done the same jobs. Each daily run processed different emails and different amounts of work.

One run could contain a short email, another could contain several with many actions to create. So the averages included both the model change and whatever work arrived in each period.

The earlier model also wasn't judged with the same acceptance test. Two different sets of production days, with no human acceptance or correction time, cannot tell us whether the quality held.

Our records weren't useless, they did the job they were built for, showing whether the workflow ran, how many calls it made and what those calls cost. We were asking them a different question when we tried to use them to choose a model.

So I would keep the saving as a reason to test the switch, not as proof that the decision was already made.

Put both models on the same work

I would save ten jobs from this workflow, enough to include ordinary work and the edge cases it already has to handle. Ten is a practical first comparison for this repeated task, not a sample that proves the model is safe on everything.

Use saved copies, and remove identifying details where needed without changing the rules or difficulty of the job. Running the comparison on new live work would give each model a different day again, and you don't need that confusion in the test.

Run both models on every saved input with the same prompt, tools, context, limits and workflow version. Then mix the outputs and hide the model names from the person reviewing them, so the reviewer doesn't know which result is supposed to be the cheap one.

Write the acceptance checklist before either model runs. Otherwise a missing output can start looking like a small problem after you see an otherwise impressive result, and the standard changes around the work you wanted to like.

For this kind of rule-bound workflow, the first-pass rule is strict, follow all the input and exclusion rules, create every required output, add nothing incorrect or unsupported, and need no human correction.

The standard should describe the job, not how polished the result looks. A well-written draft doesn't compensate for a required project that was never created.

For every attempt, record what the model cost, how many calls it made, whether it retried and why it stopped. Then record whether the work passed on the first try, what was missing or wrong, how long review took and how long correction took.

Keep elapsed time too, because a result can meet the standard and still arrive after the team needed it. A model that is cheap overnight may not fit a job that needs an answer while the client is waiting.

Keep the record small enough to use

You don't need to build a reporting system before making this decision. Ten saved jobs run on two models gives you twenty rows, and a sheet is enough to hold them.

Each row needs the job and model, model cost, tool and infrastructure costs, calls, retries, stop reason, technical completion, first-pass acceptance, acceptance after correction, review minutes, correction minutes, missing or incorrect work, elapsed time and checklist version.

That sounds like a lot when I say it all together, but it is one record of what happened to one job. Most of the fields describe either the system's work or the person's work, and you need both to compare delivery cost.

Use one hourly labor cost that includes pay and employer costs, and keep the labor amount separate from the model bill. You will be able to see whether the cheaper model saved money or simply handed more work to the reviewer.

Don't build a permanent ledger just because you can. Keep this as a small comparison for the decision in front of you, and add the fields to regular reporting when model changes happen often enough to make that useful.

Then compare acceptance before comparing price. If one model misses a requirement, the lower bill doesn't make the requirement go away.

When both meet the same standard, compare total cost per accepted task and whether they finished in time. Set that decision rule before looking at which model won the cheaper column.

If the cheaper model doesn't meet the bar, you have more than one option. Keep the stronger model, fix the weak part of the workflow, or use the cheaper one only on a narrower task you can check properly.

The test is allowed to end without a switch. Its job is to help you make the decision, not to justify the saving you hoped to get.

A stronger model can cost more and leave less work for a person, while a cheaper one can be the better choice on a bounded job. That balance belongs to each workflow on your model rate card, not to one claim about the cheapest option for the whole company.

This method needs repeated work you can save safely and a standard someone can write down. Ten jobs won't settle a rare, high-consequence decision or work that depends on broad judgment, and I would keep human approval on those tasks.

For this workflow, the next move is still the same, save the ten jobs, write the standard, run both models blind and count the human work. Until that is done, the 23-times drop is a model-cost result, not a delivery-cost win.