Definitions, plus what it means for you

Total cost of serving divided by tasks the user actually completed.

The single most useful AI economics metric. Token cost alone misleads.

Change in input or output distribution over time that degrades performance.

Monitor input drift; it gives you warning before users notice.

The tolerated amount of failure within a period before change is frozen in favour of reliability.

Extend the concept to quality and cost, not just availability.

An automated assessment of system output against expected behaviour, run on every change.

Treat evals like tests: in CI, release-blocking, owned by the team.

A curated, version-controlled set of labelled cases used as the stable basis for quality claims.

If you have no golden set, you have no quality baseline — only opinions. Fund it in the Prove stage.

The degree to which generated claims are traceable to retrieved source material.

The metric that separates a useful assistant from a confident liar.

A design where a person reviews or approves output before it takes effect.

Only real if reviewers have the time and skill to genuinely review. Otherwise it is rubber-stamping.

A structured analysis of a system's effects on people, covering fairness, safety and transparency.

Start it at Prove, not at launch. Late assessments become launch blockers.

Pre-agreed conditions under which a programme stops, decided before emotional investment builds.

Write them at Frame. They are the only reliable defence against sunk-cost momentum.

Directing requests to different models based on complexity, cost or latency needs.

Usually the fastest route to fixing a broken unit-economics case.

The chosen level of human control: shadow, advisory, human-review or autonomous.

Match it to consequence severity and write it into the charter at Frame.

Untrusted content that the model treats as instructions, subverting intended behaviour.

Assume any retrieved or user-supplied content is hostile. Red-team before external exposure.

Retrieval-augmented generation: grounding model output in documents fetched at request time.

Most 'model quality' problems are retrieval quality problems. Check recall before changing models.

Deliberate adversarial testing to find harmful, unsafe or exploitable behaviour.

Book external or cross-team red-teamers; the build team cannot surprise itself.

Running the system on live traffic without acting on its output, to measure real-world quality safely.

The cheapest way to de-risk a consequential use case. Use it more than you think you need to.

A deployed capability nobody uses that still consumes budget and attention.

Review every quarter. Retiring well is a delivery skill, not a failure.