Why Do AI Benchmarks Fail in Production?

Why do AI benchmarks break at the production boundary?

An AI benchmark breaks when its score no longer represents the decision the live system must make. The metric may reward the wrong property, the harness may call a different generation path, the judge may introduce a conflict, or the aggregate may hide missing verdicts and sparse replication.

Nazr encountered each failure while evaluating generated plans. A proposed metric was guaranteed by database structure. A favorable whole-plan score came from a path that was not yet in production. A model appeared to win until its family left the judge panel. Later release numbers survived in committed prose while their row-level outputs did not.

These were useful failures because the evaluation system was allowed to contradict the product story. Google researchers made a similar production distinction in their ML Test Score paper: real systems face a “host of issues not found” in research experiments, so testing and monitoring must cover the system around the model. NIST states the same boundary directly: how a component is evaluated can change with the context in which the AI system operates. A planning-system eval therefore begins with the production decision, not a convenient score.

How can a benchmark metric be true by construction?

A metric is true by construction when the data model already guarantees the property it claims to test. It can report a perfect result while measuring no model capability. Before running an expensive evaluation, trace every metric through the schema, fixtures, and scorer and ask what output could possibly fail it.

An early Nazr benchmark proposal included an orphan-work metric. The product's foreign keys already required every generated task to belong to a work stream and every work stream to belong to a priority. Under that definition, the flagship integrity measure could not reveal a generator difference. A committed review gave the design a needs-rethink verdict before it became public evidence.

The historical record does not support a more dramatic story that a completed public benchmark was run and rejected. The proposal itself is absent, and there is no formal founder rejection record. The defensible lesson is better: schema review caught a metric whose success was guaranteed. A real orphan test would need to measure semantic alignment, such as whether a task advances its stated parent, not merely whether a parent identifier exists. This is why whole-plan coherence uses cross-branch relationships rather than database validity as its main outcome.

What happens when the harness tests the wrong path?

The benchmark can credit production for an architecture that users do not receive. Nazr's early holistic harness recorded about 2.99/5, but the live per-priority path still scored about 2.71. The report corrected the result before treating the holistic improvement as a shipped production gain.

The gap came from generation shape. The harness let a generator see sibling priorities and coordinate work across them. The live system was still making separate calls, so it could not use that information in the same way. Both calls produced valid trees, and both used familiar prompt machinery, but only one exercised the architecture behind the attractive score.

The corrected chronology was 1.72 at the original whole-tree baseline, roughly 2.71 on the historical live path, and 3.21 after the holistic path and dependency annotations were actually measured together. A later one-shot generator was then tested separately. Its model-by-architecture matrix prevented another false attribution by testing both factors. A harness should import or call the production function when possible and state every remaining substitution.

Can the judge panel create a false win?

Yes. Nazr recorded one candidate at 4.05/5 when the generator's model family joined the panel and 3.72/5 under an excluding panel. The 0.33 panel-associated shift reversed the implied model decision. A second risk was quieter: the aggregator could return a composite after one of two intended judges failed.

The first issue is a conflict control. Model-family overlap does not prove causal self-preference, but it creates a result that should be rerun under a neutral panel. The second issue is a completion control. A report labeled as a two-judge result should either require both verdicts or state that only one completed.

Averaging successful responses after silently dropping failures makes a brittle panel look complete. It also prevents the missingness from informing the release decision. The benchmark contract should name every judge, generator, attempt, and failure, then fail or visibly downgrade an incomplete cell. Nazr's LLM-judge controls cover position, model family, calibration, and raw-ledger requirements in detail.

Why do sample size and raw artifacts matter?

Sample size determines how narrowly a result must be stated, and raw artifacts determine whether anyone can reconstruct it. A six-tree gate can support an internal release decision. It cannot become a universal quality rating when the plan required deeper replication and the per-run judge ledger is unavailable.

Nazr's final one-shot gate used three fixtures and two runs per fixture, recording a 3.97/5 mean and six valid first attempts. The earlier written plan had required at least six runs per fixture. The numeric threshold passed; the replication condition did not. Several later artifact directories were gitignored and absent from all inspected checkouts, leaving committed summaries but no saved outputs or judge splits.

That does not make the summaries useless. Contemporaneous code and reports establish what the team measured and why it changed the product. It does constrain the claim to a documented internal result. The next run should archive a redacted ledger containing fixture versions, outputs, model and judge identities, errors, and every input to the aggregate. The nine-tree paired critic null is credible because its design and boundary are explicit, but raw cells would make it independently checkable.

What does production-shaped really mean?

Production-shaped means the harness shares important contracts with the live path while retaining named differences. It is a stronger label than a toy benchmark and a weaker label than the exact production call. The input context, prompt, model resolution, graph mode, persistence, and failure behavior all belong in that comparison.

Nazr's dependency grid made 198 successful calls: three models, two prompt variants, 11 fixtures, and three runs. It used the production edge schema and prompt machinery over a fixed final priority set. It did not use the same team context as the live path, and its graph pass differed in mode. Calling it exact production would hide those differences.

The selected configuration accounted for 33 calls. Its headline 1.00 precision result came from three authored over-demotion controls across three runs, nine trials, not 198 perfect real portfolios. The wider grid tested stability and alternatives; the narrow controls supported the safety gate. Production-shaped language keeps both facts visible and avoids turning fixture precision into a field error rate.

How do you build an eval that can change what ships?

Start with a decision and its most costly failure. Define fixtures that can expose it, run the real path where feasible, freeze the scorer and judge rules, require complete cells, and archive the ledger. Then state the result at the scope of its sample and preserve findings that oppose the desired product story.

Public benchmark history shows why maintenance matters. In its account of retiring SWE-bench Verified for frontier evaluation, OpenAI says it created the set by reviewing 1,699 problems and selecting 500. In a later audit of 138 problems that one model did not solve consistently, OpenAI reported material test or description issues in 59.4 percent. It also found evidence of benchmark exposure and stopped reporting the score for frontier coding capability. This is a vendor report about its own evaluation, but it demonstrates that a repaired benchmark can still age out of its construct.

Nazr's most useful evals did not create a clean ascending chart. They rejected a schema-guaranteed metric, corrected a non-production result, reversed a judge-conflicted model win, recorded a critic null, and qualified a release gate whose replication was too shallow. The instrument earned trust by changing what the team could claim as well as what it chose to ship.