How Do You Measure Coherence in an AI-Generated Plan?

What does coherence mean in an AI-generated plan?

Coherence means that the plan works as one portfolio. Its priorities do not duplicate or contradict each other, together cover the objective, follow a usable order, and break work into comparable units. Local writing quality is only one part. The decisive relationships often sit between branches that were generated separately.

A polished branch can describe sensible work while another branch quietly repeats it. Two priorities can assume incompatible product states. Every initiative can be marked urgent, leaving no critical path. None of those failures appears when an evaluator sees only one branch at a time.

This distinction also appears in public planning research. The peer-reviewed PlanBench paper argues that common-sense tasks make it difficult to tell whether a model is planning or retrieving familiar patterns. Its authors found that performance “falls quite short” on several planning capabilities even for state-of-the-art models tested in 2023. Nazr's benchmark addresses a different kind of planning, an organization's strategy tree, but uses the same principle: define the relationships a plan must satisfy, then test those relationships directly. The broader method for evaluating an AI planning system starts with that measurement boundary.

Why can good branches form a bad plan?

Branch quality and portfolio quality answer different questions. In Nazr's first internal coherence campaign, per-branch quality stayed at a recorded 4.0/5 while the whole-tree composite began at 1.72/5. The branches were specific and plausible; their overlap, contradictions, and ordering made the combined plan weak.

The initial system generated work beneath each priority independently. That gave every call a clear local problem, but hid sibling branches. Several priorities could independently rediscover the same customer research or shared infrastructure. The evaluator that chose a good model for one branch had not tested whether all branches added up.

Nazr deliberately weighted cross-priority non-redundancy at 30 percent and local branch quality at only 5 percent. The low control weight did not mean branch quality was unimportant. It prevented an already-high local score from drowning out the constraint the benchmark was built to detect. Later model and architecture tests kept branch quality in a 4.4 to 4.6 band while whole-tree results still moved materially. The model-size comparison explains why the capability difference became visible only after the model could see the whole plan.

Which metrics expose whole-plan failures?

Use separate metrics for overlap, factual consistency, sequencing, coverage, granularity, and local quality. Fix the weights before the comparison and retain every metric-level finding. A composite supports a decision, but the failing dimension tells engineers what to change and prevents one strong dimension from concealing another.

These weights are design judgments, not empirically proven constants. Nazr preregistered them for the comparison rather than tuning them after seeing a preferred result. A different planning product may need different weights, but it should still publish the choice. The judge must cite the offending nodes, not merely return six decimals. That produces an error inventory that can be checked against the controls in the LLM-judge evaluation.

Which changes actually improved coherence?

Whole-context generation and model capability improved coherence, but the experiments must remain separate. The early architecture moved from a recorded 1.72 to about 2.71 and then 3.21. A later controlled model-shape comparison used a 3.36 reference and reached 3.97 in its six-tree release run.

The early chronology contains a useful correction. A roughly 2.99 result initially associated with production came from a holistic harness path that was not yet live. The report corrected the production estimate to roughly 2.71. After holistic work-stream generation and dependency annotations entered the measured path, nine annotated trees across three fixtures averaged 3.21. A one-run-per-fixture comparison without the annotations averaged 3.02, too small a sample to isolate their effect.

The later experiment changed the initial generator more completely. A stronger model inside the decomposed path recorded 3.48, and the cheaper model in one-shot form recorded 3.44. The stronger model plus one-shot whole-plan context recorded 3.98 in the matrix and 3.97 in a separate six-tree release run. The accurate history is two related experiments, not one smooth 1.72 to 3.97 climb. Correcting an attractive benchmark story is part of the evidence, not an embarrassment to remove.

How can code and humans check plan dependencies?

Separate perception from policy. Nazr uses an LLM to propose dependency edges over the final priority set, deterministic graph code to classify sources and downstream work, and humans to handle cycles. Exact graph rules become repeatable without pretending that the model always perceives every relationship correctly.

The production-shaped benchmark tested three models, two prompt variants, 11 fixtures, and three runs, producing 198 successful calls. The selected configuration represented 33 of those calls. Its three over-demotion control fixtures, repeated three times, produced nine trials at 1.00 precision against authored answer keys. That is a narrow gate against hiding work incorrectly, not a field precision claim.

Once edges exist, deterministic code condenses strongly connected components. Source components stay active, downstream singletons can be demoted, and cycle members stay visible for human handling. The policy is fail-open: if extraction or persistence fails, no priority is hidden. The model can still miss an edge, and humans created the fixture truth, so the complete statement is model-perceived edges, deterministic graph policy, human cycle decisions. An agent report entering Nazr with its source attached follows the same boundary between evidence and the decision made from it.

What does the coherence benchmark still not measure?

It does not measure customer value, long-term plan freshness, agreement between multiple human leaders, or a current production success rate. The row-level artifacts for several historical runs are missing, and the six-metric rubric lacks a formal human inter-rater study. The scores can guide product work without proving those wider outcomes.

Sequencing remains the clearest residual. In the early annotated comparison, the sequencing metric moved from 2.33 to 2.28. In the later one-shot release run, both runs of one fixture recorded sequencing between 1.5 and 2.0 even while the overall mean reached 3.97. A better composite therefore did not mean the critical-path problem was solved.

The benchmark also relies on an LLM panel for subjective dimensions. Several committed reports survive, but later raw run directories were ignored and could not be recovered, preventing independent recomputation. A trustworthy next run would archive redacted outputs, require every judge verdict, add human agreement checks, and keep sequencing visible as its own release condition. The paired critic experiment shows why that discipline matters: a plausible correction mechanism can change output while leaving the measured outcome flat.