Evidence before release, and monitoring after it.
An AI system’s quality cannot be summed up in a single accuracy figure. Whether it is good enough depends on the task, on what its mistakes cost, and on what it is being compared with.
Evaluation and governance run through every Ramerion engagement: the measures are agreed before building, tested before release, and watched once real users arrive.
01What to measure
The metrics follow the kind of system, not a universal score.
Most real systems combine several of these parts, so most evaluations combine several of these measures — each with its own threshold.
| Component | What is measured | How |
|---|---|---|
| Retrieval | Whether passages that hold the answer are found and ranked near the top | Labelled question sets; recall and ranking metrics |
| Generated answers | Groundedness in sources, correctness, relevance, completeness | Reference answers, reviewer rubrics, calibrated automated judges |
| Agents and tools | Correct tool, arguments and order; safe handling of errors and refusals | Expected traces, step-level checks, adversarial cases |
| Extraction | Field-level accuracy and schema validity | Reviewed ground-truth documents; per-field precision and recall |
| Classification | Correct labels, especially on rare but important classes | Per-class precision and recall; confusion analysis |
| Operation | Latency, cost, failure and escalation rates | Load tests before release; monitoring afterwards |
02How it works
A release gate before launch, and a feedback loop after it.
Before release
- InputRepresentative test setReal, difficult and adversarial cases
- ToolBaselineToday's process or a simpler system
- ToolMeasureTask-specific metrics
- CheckRelease gateAgreed thresholds met?↳ no: fix and re-run
After release
- ToolMonitorSampled quality, drift, cost, latency
- CheckDetectAlerts and user reports
- PersonRespondTriage, contain, communicate
- ToolRoll back or fixPrevious version kept deployable
- StoreAdd to the test set
- Input
- Tool
- Check
- Person
- Store
Build the test set with the people who do the work
Real inputs, the answers experts would give, and deliberately hard cases — ambiguous requests, poor scans, out-of-scope questions, attempts to misuse the system.Set a baseline
Measure the current process or a simpler alternative on the same cases, so improvement is shown rather than asserted.Agree thresholds before seeing results
Decide what “good enough” means for each metric, and which individual errors are unacceptable whatever the average.Automate regression testing
Any change to a model, prompt, retrieval setting or tool re-runs the suite before it reaches users.Red-team before launch
Probe for prompt injection, sensitive-information disclosure, unsafe actions and harmful output, using public threat lists such as the OWASP Top 10 for LLM Applications (external site) as a checklist.Monitor and review traces
Sample live interactions for human review, watch for drift in inputs and outputs, and track cost and latency against expectations.Handle incidents
A named owner, a way to switch features off, a rehearsed rollback, and a written record of what happened and what changed.
03Governance
Controls in proportion to what is at stake.
A drafting aid and a system that changes customer records should not carry the same overhead. Frameworks such as the NIST AI RMF Generative AI Profile (external site) help structure the risks. Using them as a guide is not certification, and none is claimed.
| Risk level | Typical uses | Typical controls |
|---|---|---|
| Lower | Internal search; drafts a colleague edits before use | Usage guidance, basic logging, periodic quality sampling, a named owner for feedback |
| Moderate | Customer-facing replies; suggestions that inform decisions | Documented evaluation and thresholds, review before outputs take effect, monitoring, an incident process, and telling users when AI is involved |
| Higher | Actions on records, money or access; outputs affecting people's rights or safety | Everything above, plus approval gates, access reviews, audit trails, pre-release red-teaming, formal sign-off and a review of applicable legal and regulatory requirements |
04Limits and security
What evaluation can and cannot tell you.
Limitations
- A test set covers the cases someone thought of; production will bring cases nobody did.
- Automated judges — models scoring other models — need calibrating against human ratings and can share the blind spots of the system they judge.
- Averages hide failures: a high overall score can sit beside poor results on a rare but important case.
- Passing an evaluation shows readiness for a monitored pilot, not a guarantee.
Security and data implications
- Test sets are often built from sensitive real cases and are stored and accessed like production data.
- Traces kept for monitoring are themselves sensitive; who may see them and for how long is agreed up front.
- Red-team findings are fixed, or recorded as accepted risks with an owner — never quietly dropped.
05Related
Where these measures are applied.
Next step
Ask how you would know it works.
Whether a system is planned or already running, start with the decisions it supports and the mistakes it must not make. The evaluation plan follows from those.