How Do You Evaluate an AI Planning System?

What is the right unit of evaluation?

Evaluate the production system that creates and acts on the plan: capture, model, generation architecture, judge, whole-plan rubric, decision policy, and human boundary. A model-only score cannot tell you whether the system received the necessary facts, preserved context across branches, or handled uncertainty safely.

NIST's measurement guidance says that “context is crucial” and that evaluation changes with the setting in which an AI component operates. Google's ML Test Score paper likewise argues that production systems face issues missing from research experiments and need tests and monitoring around the model. Planning makes that boundary especially visible because a plausible sentence is not the same thing as a valid sequence of work.

The peer-reviewed PlanBench paper warns that common-sense tasks can make it hard to tell whether a model is planning or retrieving familiar patterns. An organization plan has an analogous trap: polished branches can look intelligent while duplicating, contradicting, or blocking one another. Nazr's evaluation program therefore treats each score as product instrumentation. The benchmark is useful when it can embarrass the builder, reject a claim, and alter the system that ships.

How do you test whether the model has enough context?

Hold the model and prompt template fixed, then vary only information the production system can genuinely collect. Use fixtures with known current state, goals, and constraints, and let the judge see that truth. The test should distinguish a model reasoning failure from a prompt that never contained the answer.

Nazr tested three input variants across three fixtures and three runs. The recorded composite moved from 2.61/5 with the original answers to 3.17 with current state and 3.99 with goals and constraints added. Aggregate high-severity findings moved from 97 to 70 to 3. These are documented internal aggregates rather than independently reproducible public results, but they changed which facts the generation path receives.

The test did not show that two fields always improve an AI system by a fixed percentage. It showed that private operating facts were load-bearing in this task. The complete input and model comparison also found that richer capture beat a separate frontier-model swap at the root-priority layer. Before changing models, test whether the current model is being asked to infer information that only the user knows.

How do you separate model quality from system architecture?

Run a factor matrix over the model and generation shape, then inspect interactions rather than isolated averages. A stronger model may be constrained by a decomposed call, while a weaker model may fail under the longer context of one-shot generation. Include invalid outputs and fixture-level collapses beside the mean.

Nazr's 2 x 2 whole-plan matrix recorded 3.36 for a Flash-class model in the decomposed shape, 3.44 for the same class single-shot, 3.48 for an Opus-class model decomposed, and 3.98 for the Opus-class model single-shot. The gain belonged to the model-shape pair. One cheaper single-shot fixture collapsed to 2.50, and one stronger decomposed generation returned an empty tree.

A separate six-tree release run for the selected pair recorded 3.97 against a 3.76 gate and six valid first attempts. It also used only two runs per fixture despite an earlier plan requiring six or more. The result supported an internal ship decision, not a timeless model recommendation. Model identity, prompt shape, sample count, invalid outputs, and the historical date all belong in the release record.

How do you know whether the judge is trustworthy?

Calibrate the judge against human or deterministic labels, blind model identity, reverse pairwise order, repeat close comparisons, and require every intended verdict. A panel is only stronger than one judge when its members make meaningfully different errors and the harness exposes incomplete cells.

Nazr recorded a candidate at 4.05/5 when its model family sat on the panel and 3.72 under an excluding panel, a 0.33 panel-associated shift. The raw cells are missing, so the result does not identify a universal self-preference effect. It establishes that panel composition changed the product decision and should be controlled.

The code audit found a second issue: a nominal two-judge composite could be calculated after one verdict failed. That turns missing evidence into an apparently complete result unless the harness exposes successful verdicts out of intended verdicts. The full LLM-judge analysis recommends generator-family exclusion, order swaps, gold-label checks, repeated runs, and explicit successful-verdict counts. Consequential claims should carry the judge configuration and completion rate beside the score.

How do you score the plan as one portfolio?

Render the complete priority, work-stream, and task tree, then score relationships across it. Keep non-redundancy, consistency, sequencing, coverage, granularity, and local branch quality separate. Weight the cross-branch constraints intentionally and require findings to identify the nodes responsible for each deduction.

Nazr's original whole-tree benchmark recorded a 1.72/5 composite while local branch quality stayed at 4.0. The early production chronology later moved through roughly 2.71 and 3.21, but the reports also corrected a roughly 2.99 harness result that had not yet reached production. The later one-shot experiment used a separate reference and must not be spliced into one smooth progress chart.

Sequencing remained weak even in the higher-scoring release run, with one fixture's runs between 1.5 and 2.0 on that metric. Nazr's six-part coherence rubric preserves that residual instead of allowing the composite to declare the plan solved. A release gate should include critical metric floors as well as an average.

When should code or humans make the decision?

Use deterministic code when the desired answer can be specified exactly, an LLM when perception or subjective comparison must scale, and a human when ambiguity is strategic or consequential. Do not describe a pipeline as deterministic when a model supplies the graph that the code later classifies.

Nazr's dependency stage uses an LLM to propose edges, deterministic graph policy to classify source and downstream components, and humans to choose how to handle cycles. In the production-shaped grid, the selected configuration achieved 1.00 precision on nine authored over-demotion control trials. That is a fixture gate, not a measured field error rate.

The critic experiment found another human boundary. Automatically applied changes moved the nine-tree composite from 3.31 to 3.31, while the strongest duplication remedy remained in human review. The critic null showed that workflow plumbing and output-quality improvement were different claims. Test code on exact policies, judges on subjective output, and humans on the decisions the system deliberately leaves open.

What should a release record contain?

Record the decision, fixture versions, input variant, generation and judge identities, prompt and schema versions, run count, every failure, metric-level scores, aggregation rule, gate, and resulting product action. Archive redacted row-level outputs so the aggregate can be reconstructed without exposing customer or confidential fixture data.

Nazr's record failed its own durability test in places because later run directories were ignored. The benchmark-failure audit distinguishes committed summaries from independently reconstructable evidence. Future exact figures should not leave the release gate without a durable ledger.

Why does system-level evaluation matter for lean teams?

Lean teams use agents to carry more organization-scale work, which makes coordination quality a binding constraint. A locally plausible task can still duplicate another agent's work, rely on a blocked priority, or preserve a fact that changed. Evaluating the model alone leaves those system failures invisible.

Nazr is built around that coordination constraint. Agent reports enter with provenance, planning work sits in one hierarchy, and uncertain corrections can reach human review rather than silently rewriting the plan. Those product boundaries do not prove customer value, retention, or business outcomes. They define what the evaluation system must test: grounding, coherence, freshness, and governance across the work.

The useful evaluation is therefore one capable of stopping a flattering story. In this program, the instrument rejected a metric guaranteed by schema, corrected a score from the wrong path, reversed a conflicted model win, recorded a critic null, and exposed shallow replication in a passing release run. For a lean team, that is the standard: evidence must be strong enough to change what ships and narrow what gets claimed.