Do LLM Critics Improve Output Quality?

Did the LLM critic improve Nazr's plan quality?

Not on the measured whole-plan outcome. Across nine paired trees, Nazr's critic moved the recorded composite from 3.31/5 before the pass to 3.31/5 after it. Non-redundancy moved from 3.22 to 3.28, a 0.06 difference far below the instrument's documented single-run variation.

The null mattered because the critic had an appealing product story. It inspected a generated plan, added dependency information and cross-references, and flagged overlapping work. Those actions sounded like they should improve whole-plan coherence. The paired test asked whether the changes visible to the judge actually moved the metric the critic was supposed to improve.

They did not. Three individual trees moved by 0.50 on non-redundancy, one downward and two upward, while six were flat. The overall composite spread was identical before and after. The committed result therefore describes this critic, these automatically applied changes, and these nine trees. It does not support the broader claim that critique is ineffective.

Why was the critic test paired?

A paired test isolates the critic from generation variance. Nazr generated each plan once, judged it with the critic off, ran the real critic over that same persisted tree, and judged it again. Both arms used the same rendering path and judge panel, so the critic pass was the intended difference.

The design covered three fixtures and three runs, producing nine trees. Each tree passed through the production-shaped context composition, plan generation, persistence, dependency annotation, and context export. Two judges scored both states. Independent before-and-after generation would have mixed the critic's effect with ordinary variation between two different plans.

Pairing does not solve every measurement problem. The raw run directory was not recovered, so the audit verifies the design and committed aggregates rather than recomputing every cell. The two-judge aggregator can also accept one successful verdict if another fails. Those limitations are why the judge panel needs its own completion controls. Still, pairing was the right design for detecting a change expected to be smaller than generation noise.

What did the critic actually change?

The critic added 105 dependency edges and 16 cross-reference notes across nine plans, while creating four duplication signals for human review. The edges and notes changed the judged tree. The duplication signals did not, because merging strategic work required human approval and the test deliberately kept those signals out of both judged arms.

That boundary kept the experiment honest. Showing the judge a signal that explicitly said two branches overlapped would have handed it the intended answer without changing the plan. Automatically merging the branches would have tested a product behavior that Nazr did not permit. The experiment therefore measured only the critic's safe, automatically applied, rendered effects. A count of critic activity would have implied more than the score could establish about output quality.

Why did the score stay flat?

The critic's strongest anti-duplication action was outside the judged treatment. Cross-reference notes could describe a relationship without removing duplicated work, and dependency edges appeared as identifiers that were difficult for the judge to interpret. The pass changed execution metadata more than the plan structure scored by the rubric.

The lowest non-redundancy trees make the limit visible. One plan scored 2.00 before and after, and another stayed at 2.50. The critic correctly flagged a duplication in the first, but that recommendation waited for human review. The judged plan still contained the overlap, so a flat non-redundancy score was the expected measurement.

This is a treatment-boundary problem, not a reason to expose hidden signals to the judge. If the next hypothesis is that human-approved merges improve quality, the next test should render the approved merge and score that resulting tree. If the intended benefit is better task gating, use a deterministic dependency test instead of a prose-quality rubric. A benchmark fails when its measured treatment differs from the product claim.

Does a null result mean LLM critics do not work?

No. Published results vary by task, feedback source, training method, and whether revision is allowed. Nazr tested an untrained plan critic whose most substantial action was human-gated. Other studies test a same-model revision loop or train a specialist critic to maximize the corrected output.

The Self-Refine preprint evaluated one-model feedback and revision across seven tasks and reported about 20 percent absolute average improvement. A later preprint by Xie and colleagues trained code critics with reinforcement learning and reported up to 106.1 percent relative improvement across its code-generation benchmarks. Those treatments allowed feedback to drive a new answer.

Counter-evidence is equally important. Huang and colleagues' peer-reviewed ICLR 2024 paper found that models struggled to correct reasoning without external feedback and that performance “at times ... even degrades after self-correction.” These papers do not validate or contradict Nazr's exact 3.31 result. Together they show why “add a critic” is not a complete intervention. The task, critic training, available evidence, revision permission, and outcome measure define what was tested.

How should a team decide whether to keep a critic?

Keep a critic only for benefits the evidence supports. Define its job, expose the resulting change to the appropriate evaluator, compare the same output before and after where possible, and inspect failures beside the average. A critic that improves workflow metadata but not prose quality should be described and measured as workflow infrastructure.

Nazr kept the critic because its dependency edges support task handoff and its duplication signals create human-review work. The test also showed what Nazr must not claim: the automatically applied critic pass was not a measurable non-redundancy lever. That narrower role is useful precisely because the null prevented a broader product claim. It also preserves the boundary used for source-carrying agent reports in Nazr: evidence enters the system before a person or policy decides what to change.

The next tests should separate the two remaining hypotheses. Score execution gating with deterministic dependency fixtures. Separately, compare a duplicated tree with the tree produced after a human accepts a merge recommendation. Archive every output and judge verdict so the result can be reconstructed. Those steps fit the production-system evaluation method: measure the action the system is allowed to take and the consequence it is meant to change.