# The 23× AI Cost Cut We Couldn't Call a Win Published: 2026-08-10T13:00:00.000Z · Updated: 2026-08-23T14:29:06.000Z · Author: Rod Amora · Canonical URL: https://rodamora.com/blog/the-23-ai-cost-cut-we-couldnt-call-a-win > I moved one live AI workflow to a cheaper model and cut average AI token cost from $1.5474 to $0.0659 per run, about 23×. The dashboard stayed green. That still did not prove delivery got cheaper. I moved one live [AI workflow](https://rodamora.com/glossary/ai-workflow) to a cheaper model. The average model bill fell from $1.5474 to $0.0659 per run, about 23×. The old model finished 11 of 11 runs. The new one finished 10 of 10. The dashboard made the switch look done. Every run reached the end without an error. Our records did not show whether a person accepted the work on the first pass, how long corrections took, or whether the two models handled jobs of the same difficulty. The model bill got smaller. That is all the record could prove. The rest of delivery was still missing from the record. Before I call the switch a win, I need both models on the same 10 jobs, one written standard, and a count of the time people spend checking and fixing the work. ## What did the 23× drop in AI token cost prove? The records cover one recurring, rule-bound email-to-task workflow. They prove one narrow result: [AI token cost](https://rodamora.com/glossary/token) per run fell by about 23×. Measure Before After Model `gemini-3.5-flash` `gemini-3.1-flash-lite` Period observed July 7–21, 2026 July 22–August 4, 2026 Workflow attempts 11 10 Average model cost per attempt $1.5474 $0.0659 Cost range per attempt $0.0975–$3.2411 $0.0039–$0.1277 Average LLM calls per attempt 12.82 7.50 Failed attempts 0 0 Recorded retries or regenerations 0 0 First-pass technical completion 11/11 10/10 First-pass human acceptance Unknown Unknown Human review or correction time Unknown Unknown The numbers came from a live production database on August 5, 2026. Calls per run also fell. Neither period recorded a failed run, retry, or regeneration. The two unknown rows carry the decision. I do not know whether the cheaper model met the same standard. I also do not know whether someone spent more time checking and fixing its work. The switch gave me a result worth testing. It did not give me evidence of equal quality or lower delivery cost. That boundary matters before a firm changes every workflow. A large cost cut can deserve a test and still be too weak to support a rollout. ## Why isn't a completed run the same as accepted work? A green check says the system reached its last step without reporting an error. It says nothing about the work a person still has to do. A report can reach the last step with a required section missing. An email draft can follow the workflow and still contain a claim nobody can support. The green check stays green while someone repairs the work before a client sees it. The hidden cost shows up later in the week. A report returns to the review queue, or a senior person checks it line by line because nobody can tell which parts are safe. That is where [reviewing AI work](https://rodamora.com/blog/the-four-stages-of-reviewing-ai-work) eats the gain the cheaper model just created. That repair belongs in the cost. OpenAI's July 14, 2026 guidance recommends tracking [cost per accepted outcome](https://openai.com/index/managing-ai-investments-in-agentic-era/) for important workflows. It says a cheaper model may need more attempts, correction, or human review. OpenAI's July 17, 2026 [AI scorecard](https://openai.com/index/a-scorecard-for-the-ai-age/) counts employee time, review, retries, and rework inside the full cost of successful work. For this decision, I would calculate: `Cost per accepted task = (model + tools + infrastructure + retries + review + correction) / accepted tasks` An accepted task has cleared the written standard. First-pass acceptance means it cleared that standard without a person fixing it. If the work passes after a correction, the task can enter the accepted count. The correction time stays in the cost. Work that never passes stays out of the accepted count. Every dollar and minute spent on it stays in the total. ## Why can't I compare the two production weeks? The two models did different work. Each daily run handled different emails and a different amount of work. Can I still compare the averages? One run may contain one short email. Another may contain several emails with many required actions. The cost difference includes the model change and whatever work arrived that day. The earlier model was not judged with the same acceptance test. Neither period recorded human acceptance or correction time. Those two changing weeks cannot tell me whether quality fell, held, or improved. The records were built to run the workflow, count calls and cost, and flag crashes. They did that job. The daily question was whether the workflow ran. A model comparison asks which model handles the same work more cheaply. That needs a different record. ## How should I compare the two models? Both models need to do the same work under the same rules. I would save 10 jobs from this workflow, including normal work and the edge cases it already has to handle. Then I would run this test: 1. Save 10 input snapshots from the repeated workflow. 2. Run both models on every snapshot with the same prompt, tools, context, limits, and workflow version. 3. Mix the outputs and hide the model name from the reviewer. 4. Score every output against one written checklist. Write the checklist before either model runs. To pass on the first try, the work must follow every input and exclusion rule, create every required output, add nothing incorrect or unsupported, and need no human correction. Why write the rule first? A missing output can look minor after you see an otherwise good result. The same mistake can then pass for one model and fail for the other. For every run, record the model cost, model calls, retries, stop reason, first-pass acceptance, review minutes, correction minutes, missing or incorrect work, elapsed time, and checklist version. Time needs its own field. Work can meet the standard and still arrive after the team needed it. Use saved or scrubbed copies instead of live client work. When a saved input contains identifying details, remove them without changing the job's rules or difficulty. Live work would give each model a different day again. ## What should I write down? Twenty rows are enough for this decision. Each saved job gets one row for each model. A wider reporting system can wait. Job Model Model cost Tool and infrastructure cost Calls Retries Finished Accepted first pass Accepted after correction Review minutes Correction minutes Missing or incorrect work Elapsed time Checklist 01 Model A v1 01 Model B v1 Use one hourly labor cost that includes pay and employer costs. Keep it separate from the model bill. That split shows whether the cheaper model cut delivery cost or moved the cost into someone's week. If the cheaper model adds correction time, the sheet puts that time beside the smaller API bill. The saving cannot disappear inside someone's calendar. Every attempt stays in the total cost because the firm paid for it. Only work that clears the checklist enters the accepted count. Compare first-pass acceptance before comparing price. When both models clear the same bar, compare total cost and elapsed time. Keep this sheet temporary for a one-time choice. Add these fields to regular reporting only when model changes happen often enough to reuse the record. ## When does the cheaper model win? The cheaper model wins when it meets the same written standard on the same 10 jobs, lowers total cost per accepted task, and finishes on time. Set that rule before looking at the results. One workflow's model bill is one line inside [the four numbers](https://rodamora.com/four-numbers) that decide whether delivery actually got cheaper. What if it misses the standard? Keep the more capable model, fix the weak step, or give the cheaper model only work that is narrow and easy to check. The test does not have to end in a switch. A more capable model can cost more and leave less work for a person to review. A cheaper model can move cost into that review. Each workflow has its own balance on the [model rate card](https://rodamora.com/blog/your-ai-employee-has-a-rate-card-you-set-it). One pass rate or cost ratio for the whole company would be made up. This test fits repeated work that can be saved or safely scrubbed and judged with a written checklist. Ten jobs cannot settle model choice for rare work that needs wide judgment or carries a high cost when it fails. Keep human approval on that work. Before the next model change, save 10 representative jobs, write the acceptance rule, run both models blind, and count the time people spend checking and correcting the work.