Eval

An eval is a repeatable test that scores AI output against a written standard. It gives you a way to check whether the work is good enough, rather than judging it by how quickly it arrived or how much people used the tool.

For a business, that can start with a few real deliverables and checks your team agrees on. You don’t need a new platform before you can ask whether a proposal used the right price.

What an eval looks like in a business

Take 20 proposals made with AI help and write down what makes one correct at your firm. The approved price is used, the scope matches the call, every claim has support, an owner is named, and the client details are right.

Score each proposal against those five checks, then use the same checks next month. Keep a fixed set of cases too, so when you change a prompt or model you can compare it on the same work.

That is enough to start, though it doesn’t prove you’ve covered every kind of mistake. The first useful part is getting the people who own delivery to agree on what correct means.

Write that agreement down, because otherwise two reviewers can quietly apply different rules. Use your own cases too, because a model doing well on generic test data doesn’t tell you how it handles your client exceptions.

And keep the score close to a decision somebody can make. A failed price check tells the owner what to fix, while one number blending speed, tone, completeness, and retention makes it harder to see what went wrong.

Hours saved answers a different question. The Four Numbers explains why usage and saved time are weak substitutes for business results, an eval checks the work itself.

Why a feeling is not a measurement

If your first quality signal is a complaint, the work has already reached a client. Quality can also slide slowly, as more output goes through the same review time and small errors turn into rework or a lost renewal later.

Without a record of the earlier standard, you have little to compare with when someone says the work seems worse. The point of the eval is to make that comparison possible while you can still do something about it.

The company-level evidence is another reason to measure rather than assume. In a survey of nearly 6,000 senior executives across the US, UK, Germany, and Australia, 89% reported no effect from AI on labor productivity over the prior three years.

That was what executives reported, not an audit of their output, and it doesn’t prove evals would have changed the result. It does mean you shouldn’t take tool use as evidence that the company is getting more useful work done.

Gartner also predicts that more than 40% of agentic AI projects will be canceled by the end of 2027 because of rising costs, unclear value, or weak risk controls. That is a forecast, not a measured result for your firm, so I would use it as a reason to establish a starting measure, not as odds for your project.

As output grows, human in the loop review takes more attention because someone has to look at each item. Automated eval checks can cover a larger set and send failures to a person, but only for the things those checks can actually judge.

Who should grade the output?

Start with the simplest grader that can answer the check. Usually that means a person with a checklist, partly because writing it exposes the places where senior people haven’t agreed on what right looks like.

A person still needs the evidence and enough time to use it. Human grading isn’t automatically correct just because the reviewer knows the work.

For a narrow fact, a rule may be enough. Ordinary software can check whether a price matches the price list, whether a figure appears in the source record, or whether a required section exists, without asking another model to judge it.

Another model can help where a rule struggles, such as whether the tone fits a client or a conclusion follows from the evidence. In a 2023 study of LLM judges, strong models agreed with human preferences more than 80% of the time in the tested comparisons, about the same level of agreement reported between people.

But that study also found position, verbosity, and self-enhancement bias, meaning the judge could favor an answer because of where it appeared, how long it was, or whether the model produced it. The judges also had limits in reasoning, so that agreement rate isn’t a promise about grading your proposals.

Give the grader outputs people have already rated and compare the answers, then check it again on a schedule. Otherwise you’ve put a score on another opinion you haven’t tested.

What does an eval cost?

A small first eval may take an afternoon, including a conversation among senior people about what makes a deliverable correct. That’s a starting example, not a fixed budget, because agreeing on the standard can take more work than running the test.

After that, somebody has to own it, run the checks, read the score, and act when it changes. If nobody has that job, the test can get run once and forgotten.

The checks also need to change with the work. A new price list, client requirement, or delivery process can make yesterday’s expected answer wrong, so review the standard when those things change.

Keep each failed case and the reason it failed. Then you can see whether the same mistake returns after a prompt or tool change, and the team has real examples to learn from.

People need time to learn what the result asks them to do. In the delivery data I’ve reviewed, training that sticks takes 8 to 10 hours per person over about 3 weeks, and a score on its own doesn’t do that teaching.

Where this sits on the Delivery Model Ladder

An eval sits between Stage 2 and Stage 3 on the Delivery Model Ladder. Stage 3, AI-native delivery, needs work you can divide into pieces and check, and the eval supplies that check.

That’s why an AI employee needs a way to validate the result as well as scope, access, and a trigger. The first three describe what it may do, the test tells you whether the work passed.

Across the 150+ franchise units in the delivery data I track, roughly 80% still run chatbots, 15% run rigid workflows on triggers, and 5% run autonomous agents defined by conditions. Those figures describe that network, not the market as a whole.

In that smaller group, the firms have a written answer to what a correct output looks like. That’s the pattern in the delivery data I track, not something the adoption percentages alone prove about another firm.

Quick answers

Do I need an eval before I start using AI? You can experiment before you have one, but put checks in place when the output is going to reach a client. Treat an eval as a requirement before removing a human check or expanding the workflow.

What should the first eval measure? Pick one failure that would matter, a wrong number, unsupported claim, or missing disclosure. “Overall quality” is too broad to tell someone what to fix.

How does an eval affect what an agent can run alone? Work should be checked and undoable before you grant more autonomy, and an eval makes the checking part concrete. Revisit that decision when the task changes, even if you’re using the same model.

Does an eval appear in the Four Numbers? Its early signal is quality, before a problem shows up as clients not renewing. That’s the ROI question firms often answer too late, when the client has already decided to leave.