Fine-tune only when the measurements say so.
Model adaptation covers everything that changes how a model behaves on your task — instructions, examples, context, output schemas, retrieval and, sometimes, further training. Fine-tuning is the most expensive of these to build and to maintain, so it is the last lever to pull, not the first.
We start from a measured baseline and add complexity only when a representative evaluation shows a gap that simpler changes did not close.
01Four levers
What each kind of adaptation changes — and what it costs to change it.
Teams often reach for fine-tuning when the real gap is missing context or an unclear task definition. Comparing the levers side by side makes the cheaper options visible.
| Lever | Prompting & examples | Structured outputs | Retrieval (RAG) | Fine-tuning |
|---|---|---|---|---|
| What changes | Instructions and worked examples sent with each request | The format the model must return, enforced by a schema | The information available at answer time | The model's weights, through further training |
| Good for | Almost every task — the first thing to get right | Extraction and hand-off to other systems | Private or frequently changing knowledge | Consistent style, labels or patterns at volume |
| Cost to change | Low: edit and re-run the evaluation | Low: schema and validation code | Moderate: pipeline and index upkeep | High: data preparation, training runs, re-evaluation |
| Watch for | Long prompts raise cost and latency | A valid format is not a correct answer | Retrieval that misses the right passage | Regressions on behaviour you did not train |
02How it works
A decision ladder, measured at every rung.
- InputTask & baselineRepresentative cases, agreed metrics
- ToolPrompt & examples↳ gap closed: stop
- ToolStructured output↳ gap closed: stop
- ToolAdd retrievalIf knowledge is missing↳ gap closed: stop
- CheckGap persists?Same cases, same metrics
- ModelFine-tuneCurated, permitted examples
- CheckCompare & releaseAgainst baseline; versioned
- Input
- Tool
- Check
- Model
Define the task and baseline
Describe the task as inputs and expected outputs, build an evaluation set with the people who know the work, and measure a capable general-purpose model before changing anything.Pull the cheaper levers
Clarify instructions, add worked examples, enforce an output schema and, where knowledge is missing, add retrieval — re-measuring after each change so its effect is known.Prepare training data
If a gap remains, assemble examples that are correct, consistent, representative of real inputs and that you are permitted to use for training. How many are needed depends on the task and method; consistency matters more than volume.Keep the test set apart
Training, validation and test examples are separated, and the test set never informs tuning decisions, so reported results describe data the model has not seen.Train and compare
Train with the chosen provider's service or with open-weight tooling, then compare against the baseline on the held-out set and on a regression set covering behaviour you did not intend to change.Version, release and roll back
Each model version is recorded with its data and settings; the previous version stays deployable, and production inputs are watched for drift.
03Implementation considerations
Providers, licences, running cost and upkeep.
- Provider availability
- Which models can be tuned, by which methods and in which regions changes frequently, and options for new customers can be restricted or withdrawn. Availability is verified for the specific model when the decision is made — never assumed from earlier projects.
- Licences and terms
- Open-weight licences differ in what they permit commercially; hosted providers differ in how long they keep training data and what they may do with it. Both are read before any data is shared.
- Portability
- A tuned model is tied to its base model. Keeping the evaluation set, training data and process portable makes it possible to re-tune on a different base later — at the cost of repeating the work.
- Running cost
- Training is paid per version; serving a tuned model may be priced differently from the base model, and self-hosting adds infrastructure. A smaller tuned model is sometimes cheaper per request than a larger general one — that is tested, not assumed.
- Maintenance
- When the base model is updated or retired, or the task and its inputs change, the tuned model may need retraining and a full re-evaluation.
04Limits and data handling
What tuning cannot do, and what it does to your data.
Limitations
- Tuning does not reliably add new facts; a tuned model can still state things that are wrong.
- It narrows behaviour: gains on the trained task can come with losses elsewhere.
- Inconsistent labels teach inconsistency — results are bounded by example quality.
Data and security
- Only data you are permitted to use for training is used, with personal data removed or minimised first.
- Training sets and model checkpoints are confidential assets, stored and deleted under agreed rules.
- A tuned model can sometimes reproduce fragments of its training data, so sensitive material stays out of training sets.
- Unlike retrieval, a tuned model cannot vary what it knows per user: anything learned is available to everyone who can query it.
05How success is measured
Better than the baseline, without new regressions.
The approach mirrors the iterative loop of evaluate, prompt and tune described in OpenAI’s model optimization guide (external site), applied whichever provider or open model is chosen.
- Task metrics
- Chosen per task: field-level accuracy for extraction, per-label precision and recall for classification, reviewer-scored rubrics for writing tasks.
- Format validity
- Share of outputs that parse against the schema without any repair.
- Regression set
- Behaviour outside the trained task, compared with the baseline model.
- Operational
- Latency and cost per request against the untuned alternative.
06Related
Often considered alongside.
Next step
Bring the task and a handful of real examples.
With examples of real inputs and the outputs you expect, we can set a baseline and tell you whether tuning is likely to be worth its cost.