Fine-tuning

Fine-tuning is extra training that changes how an existing model responds to a repeated kind of work. You give it examples of the answers you want, so it becomes more likely to follow that pattern on the next job.

I would consider it when the task runs often, the standard stays stable, and ordinary instructions keep failing in the same way on your tests. If the real problem is missing client data, an old policy, or a process nobody has written down, training the model isn’t the first thing to fix.

What does fine-tuning change?

Fine-tuning starts with a model that already exists. In supervised fine-tuning, each example pairs a request with the answer you wanted, and training adjusts the model to make answers like that more likely.

OpenAI lists classification, nuanced translation, fixed-format content, and repeated failures to follow instructions as suitable uses. Those are patterns of work you can show through examples.

Imagine you turn call notes into project briefs, and the model understands the notes but keeps putting risks under scope and leaving out the named owner. Approved briefs can teach it where those things belong, provided your team agrees on the structure.

Now imagine it uses last quarter’s price or an old cancellation rule. Those facts need to come from current records, because teaching them through examples would leave you changing the dataset and training again when the facts change.

So use fine-tuning to help the model repeat a stable behavior, and use current context for the facts this job depends on. A model can absorb facts during training, but that doesn’t make training the best way to keep them up to date.

When is fine-tuning the right tool?

First, the task needs to repeat enough to repay the work. One unusual proposal is a poor reason to train a custom model, while hundreds of similar classifications, summaries, or structured drafts give you more room to recover the cost.

The standard also needs to hold still. If three senior people grade the same answer differently, the examples will carry that disagreement into training, so settle what correct means before making the dataset.

Then you need approved examples that look like the actual work. OpenAI allows as few as 10, reports improvements around 50 to 100, and recommends starting with 50 well-crafted demonstrations.

Google’s guidance points to roughly 100 or more labeled examples for tasks where advanced prompting hasn’t been enough. Those are starting guides, not a guarantee that a particular number will solve your problem.

And you need a test before you train. OpenAI puts it plainly, “Good evals first,” because without the ordinary model’s starting score you can’t tell whether the tuned version improved or merely became more consistent in how it gets things wrong.

Most of the useful work happens before the training run, collecting real inputs, approving answers, and writing the checks. Document the process first, because training cannot settle a standard your firm hasn’t settled itself.

When should you use prompts, context, or retrieval instead?

I would start with the smallest change that fixes the failure you measured. If the missing piece is a clear instruction or format, try a prompt and a few examples before changing the model.

Google recommends starting there too, so you have the best prompt you can produce and a result to compare against. When the model follows the instruction reliably, you may not need to train anything.

If the answer depends on the signed agreement, today’s price list, approved scope, or a client’s last three decisions, supply context or retrieve those records. They can change tomorrow, and the system needs tomorrow’s version when it does tomorrow’s work.

If the missing piece is the order of the job, build an AI workflow. Pull the record, check the required fields, draft the response, send uncertain cases to a person, and write the result back, because that route belongs in the process.

Fine-tuning can improve one step inside that route, but it doesn’t create the route for you. It becomes a candidate when those pieces work and the model still repeats the same mistake across a test set, with a failure worth fixing beyond the next policy update.

How do you know fine-tuning worked?

Run the ordinary model and the tuned model on examples neither saw during training, then score both with the same eval. Keep those test examples separate, because doing well on answers the model trained on doesn’t show it can handle a new case.

For a project brief, check whether the scope matches the call, risks appear in the right section, decisions have owners, and the model hasn’t added promises the sources don’t support. Correctness comes before a better-looking draft.

Then count the work around it, how often someone edits the draft, how long checking takes, what an accepted brief costs, and how often it comes back for correction. That’s how you find out whether better model behavior helped delivery.

Training doesn’t guarantee an improvement. An ICML 2025 paper found that an OLMo-1B checkpoint pretrained on 3 trillion tokens performed more than 2% worse on several downstream benchmarks after fine-tuning than its 2.3-trillion-token counterpart.

That’s a comparison between those checkpoints on those benchmarks, not a prediction for your model. It is a reason to measure the result rather than assume more training made it better.

A smaller tuned model may also need a shorter prompt and cost less per run than a larger general model. Compare accepted work at the required quality, because your AI employee’s rate card includes the model calls and the person checking them.

What does a tuned model cost to own?

After the training bill, you still have a dataset to look after. Someone needs to remove weak examples, add failures worth learning from, keep private data in the right place, and record which examples produced each model.

That person also needs to rerun the eval when the model or work changes. A new base model may beat your tuned model without extra training, or it may need tuning of its own, and the comparison tells you which path is worth taking.

Your delivery standard can change too. A new proposal structure, approval rule, or definition of an acceptable risk can turn yesterday’s approved answer into a bad training example.

That is why I would reserve fine-tuning for stable, repeated work with enough volume to repay that upkeep. Five custom deliverables a month, a policy library changing every week, or a process the team still argues about are poor starting points.

Where does fine-tuning sit on the Delivery Model Ladder?

Fine-tuning belongs at Stage 1, Enhanced, on the Delivery Model Ladder. It can improve one AI-assisted task while a person still starts it and owns the surrounding work.

Stage 2 begins when the firm builds the trigger, current context, tool access, approval point, failure route, and final record around that task. A tuned model can be part of that workflow, but training alone doesn’t move the firm up a stage.

That’s part of the Production Gap, because a model producing one clean sample is only part of what you need. Somebody still has to supply current information, test the work, handle exceptions, and keep it running after the original builder moves on.

Quick answers

Is fine-tuning the same as training a model from scratch? No, it starts with an existing model and trains it further for a narrower task or behavior, rather than building a model from a large initial dataset.

Can I fine-tune a model on company documents? You can, but keep changing facts in sources the system can retrieve. I would fine-tune stable behavior and fetch the current policy, price, contract, or client record when the job runs.

How many examples do I need? Vendor starting guidance is around 50 to 100 strong examples that represent the work. Fifty approved examples following one standard are more useful than a larger pile of conflicting answers.

Does fine-tuning replace an eval? No, the eval comes first and tells you whether tuning helped. Keep running it when the model, dataset, or delivery standard changes.