Metrics library
Measure the thing, not the noise
Sixteen metrics with their formula, cadence and the specific way each one gets gamed. Pick four — one per category — and put them on the wall. Reporting more than that is how a programme hides.
Outcome
— The only metrics a steering committee should open with.| Metric | Formula | How it gets gamed | Cadence |
|---|---|---|---|
| Cycle time reduction | (baseline_time − current_time) / baseline_time | Compare like-for-like case mix, or easy cases will flatter you. | Monthly |
| Cost per successful task | (inference + retrieval + human review cost) / successful tasks | Exclude failed attempts from the denominator, not the numerator. | Weekly |
| Value realisation | realised benefit / business-case benefit | Re-baseline honestly; do not quietly move the target. | Quarterly |
| Deflection rate | tasks completed without human handoff / total tasks | High deflection with low satisfaction means users gave up. | Weekly |
Quality
— Evidence that the thing actually works, measured the same way every time.| Metric | Formula | How it gets gamed | Cadence |
|---|---|---|---|
| Task success rate | cases rated pass on rubric / golden set size | Rubric drift between raters — calibrate monthly. | Per change (CI) |
| Groundedness | claims supported by retrieved source / total claims | A confident unsupported claim is worse than a refusal. | Per change (CI) |
| Refusal rate on valid requests | valid requests refused / valid requests | Safety tuning silently raises this; test both directions. | Weekly |
| Regression delta | current eval score − previous release score | Any negative delta needs a named accept/reject decision. | Per release |
Trust & adoption
— Whether people actually choose to use it when nobody is watching.| Metric | Formula | How it gets gamed | Cadence |
|---|---|---|---|
| Weekly active ratio | weekly active users / licensed users | Mandated usage is not adoption. Segment voluntary use. | Weekly |
| Override rate | outputs edited or rejected / outputs accepted | The sharpest trust signal you have. Track the trend, not the level. | Weekly |
| Escalation rate | sessions escalated to human / total sessions | Rising escalation before falling usage is your early warning. | Weekly |
| Retention at 30 days | users active in week 4 / users onboarded in week 1 | The honest answer to whether the pilot worked. | Per cohort |
Operations
— Can you run this at 3am without heroics?| Metric | Formula | How it gets gamed | Cadence |
|---|---|---|---|
| p95 end-to-end latency | 95th percentile request duration | Measure user-perceived time, including retrieval and reranking. | Continuous |
| Time to rollback | decision → previous version fully serving | Unrehearsed rollback is an assumption, not a capability. | Per drill |
| Input drift index | divergence of input distribution vs reference window | Drift precedes quality loss — alert on it, not after it. | Daily |
| Eval coverage | production failure modes represented in golden set | Every incident should permanently grow this number. | Monthly |
A workable reporting set
Cost per successful task
Economics that scale or do not
Task success rate
Quality with a stable definition
Override rate
Trust, measured not assumed
Time to rollback
Operational confidence